Gesture recognition method based on adas intelligent simulation training cockpit and interactive training platform thereof

Abstract

This invention provides a gesture recognition method and interactive training platform based on an ADAS intelligent simulation training cockpit. The method includes: detecting hands based on continuous video frames to obtain hand structure information; filtering hands located within a preset driver's seat operating area based on the hand structure information and palm orientation to obtain target hands; determining gesture morphological features based on the first coordinate difference in the vertical direction between the first spatial key point coordinates of each fingertip and the second spatial key point coordinates of the corresponding metacarpophalangeal joint in the target hand; if a gesture morphological template matching the gesture morphological features exists, comparing the gesture morphological features with those of the previous video frame to obtain the number of consecutively occurring frames; if the number of consecutively occurring frames reaches a preset continuous detection threshold, controlling the vehicle's actuators to perform the corresponding operation based on the operation command corresponding to the gesture morphological features. This invention improves driving safety and system reliability.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a gesture recognition method and interactive training platform based on an ADAS intelligent simulation training cockpit. Background Technology

[0002] In intelligent cockpit human-machine interaction systems, gesture recognition, as a non-contact control method, has been widely applied to the operation of in-vehicle infotainment systems. A typical existing method involves capturing images of the driver's hands using a camera installed in the cockpit, employing a pre-trained deep learning model (such as the YOLO series) to perform real-time object detection on the images, identifying specific gesture categories, and mapping the recognition results to corresponding vehicle control commands, thereby enabling the control of devices such as windows and sunroofs.

[0003] However, the main problem with existing methods is that in scenarios with multiple people present or watching, the system cannot effectively distinguish the gestures of the target operator from those of other people, leading to frequent false triggers and erroneous executions. Because the model relies solely on the appearance features of the hands in a single frame image for classification, it lacks a verification mechanism for the identity of the operator and the continuity of the operator's intention. When a non-driver makes a similar gesture, the system will still treat it as a valid command and drive the vehicle to perform the corresponding action, seriously threatening driving safety and system reliability. Summary of the Invention

[0004] This invention provides a gesture recognition method and its interactive training platform based on an ADAS intelligent simulation training cockpit, aiming to improve driving safety and system reliability.

[0005] In a first aspect, the present invention provides a gesture recognition method based on an ADAS intelligent simulation training cockpit, comprising: Hand detection is performed based on the continuous video frames of the target user, and the hand structure information of the hand to be detected in each video frame is obtained. Based on the hand structure information and palm orientation in each video frame, the hand located in the preset driver's seat operation area is selected to obtain the target hand. Based on the first coordinate difference in the vertical direction between the first spatial key point coordinates of the fingertip of each finger in the target hand and the second spatial key point coordinates of the corresponding metacarpophalangeal joint, the gesture morphology features of the current video frame are determined. If a gesture pattern template that matches the gesture pattern features exists, then the gesture pattern features of the current video frame are compared with the gesture pattern features of the previous video frame to obtain the number of consecutively appearing frames. If the number of consecutive frames reaches the preset continuous detection threshold, the vehicle actuator will be controlled to perform the corresponding operation based on the operation command corresponding to the gesture shape feature of the current video frame.

[0006] Secondly, the present invention also provides an interactive training platform for implementing the gesture recognition method based on the ADAS intelligent simulation training cockpit as described in the first aspect; the interactive training platform includes: The target hand recognition module is used to detect hands based on continuous video frames of the target user, obtain the hand structure information of the hand to be detected in each video frame, and filter out the hands located in the preset driver's seat operation area based on the hand structure information and palm orientation in each video frame to obtain the target hand. The gesture feature recognition module is used to determine the gesture morphology features of the current video frame based on the first coordinate difference in the vertical direction between the first spatial key point coordinates of the fingertip of each finger in the target hand and the second spatial key point coordinates of the corresponding metacarpophalangeal joint. The morphological feature comparison module is used to compare the morphological features of the current video frame with the morphological features of the previous video frame if a morphological template matching the morphological features of the gesture exists, and to obtain the number of consecutively occurring frames. The vehicle control module is used to control the vehicle actuator to perform corresponding operations based on the operation instructions corresponding to the gesture shape features of the current video frame if the number of consecutive frames reaches a preset continuous detection threshold.

[0007] Thirdly, the present invention also provides an electronic device, comprising: a memory for storing computer software programs; and a processor for reading and executing the computer software programs, thereby realizing the gesture recognition method based on the ADAS intelligent simulation training cockpit as described above.

[0008] Fourthly, the present invention also provides a non-transitory computer-readable storage medium storing a computer software program, which, when executed by a processor, implements the gesture recognition method based on the ADAS intelligent simulation training cockpit as described above.

[0009] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the gesture recognition method based on the ADAS intelligent simulation training cockpit described above.

[0010] The gesture recognition method based on an ADAS intelligent simulation training cockpit provided in this invention performs hand detection on continuous video frames of the target user, obtaining the hand structure information of the hand to be detected in each video frame. Combining the hand structure information with the palm orientation, the method filters out target hands located within a preset driver's seat operating area, thus eliminating interference from hands of non-drivers and solving the problem of being unable to distinguish the gestures of the target operator from those of other personnel. Based on the first coordinate difference in the vertical direction between the first spatial key point coordinates of the fingertips and the second spatial key point coordinates of the corresponding metacarpophalangeal joints of each fingertip in the target hand, the gesture morphology features of the current video frame are obtained, achieving accurate depiction of the target hand gestures. The gesture morphology features of the current video frame are matched with a gesture morphology template and compared with the gesture morphology features of the previous video frame to obtain the number of consecutively occurring frames, thus verifying the continuity of the target hand gestures and overcoming the shortcomings of relying solely on single-frame image features and lacking verification of the continuity of operational intent. The system determines whether a preset continuous detection threshold has been reached based on the number of consecutive frames. If so, it controls the vehicle's actuators to perform actions based on the corresponding operation command, thus accurately executing the target operation command. This effectively prevents similar gestures from non-drivers or accidental misgestions from being recognized as valid commands, reducing false triggering and execution. By accurately executing the target operation command, the system eliminates situations where non-target personnel's gestures lead to vehicle misoperation, thereby solving the problems of false triggering and execution in multi-person scenarios and improving driving safety and system reliability. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the gesture recognition method based on an ADAS intelligent simulation training cockpit provided in an embodiment of the present invention. Figure 2 This is one of the schematic diagrams of gesture morphology features provided in the embodiments of the present invention; Figure 3 This is the second schematic diagram of the gesture morphology features provided in the embodiments of the present invention; Figure 4 This is the third schematic diagram of the gesture morphology features provided in the embodiments of the present invention; Figure 5 This is the fourth schematic diagram of the gesture morphology features provided in the embodiments of the present invention; Figure 6 This is the fifth schematic diagram of the gesture morphology features provided in the embodiments of the present invention; Figure 7 This is the sixth schematic diagram of the gesture morphology features provided in the embodiments of the present invention; Figure 8 This is a schematic diagram of the structure of the interactive training platform provided in an embodiment of the present invention; Figure 9 An embodiment diagram of the electronic device provided in this invention; Figure 10An embodiment diagram of a computer-readable storage medium provided in accordance with the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] Optionally, see Figure 1 , Figure 1 This is a flowchart illustrating the gesture recognition method based on an ADAS intelligent simulation training cockpit provided by the present invention. In this embodiment of the invention, the execution subject of the gesture recognition method based on an ADAS intelligent simulation training cockpit is the vehicle control system. Therefore, the gesture recognition method includes: Step 10: Perform hand detection based on the collected continuous video frames of the target user to obtain the hand structure information of the hand to be detected in each video frame, and filter out the hands located in the preset driver's seat operation area based on the hand structure information and palm orientation in each video frame to obtain the target hand.

[0014] Optionally, the vehicle control system activates the hand detection function to continuously receive continuous video frames of the target user captured by a preset camera. This camera is fixedly installed in the ADAS intelligent simulation training cockpit, facing the driver's seat, to ensure that the hand movements of the driver can be clearly captured. The acquisition frequency is set to 30 frames per second to ensure the real-time performance of hand detection.

[0015] Furthermore, the vehicle control system preprocesses each frame of video image, including three steps: image denoising, grayscale conversion, and contrast adjustment. Image denoising employs a Gaussian filtering algorithm to remove noise caused by changes in lighting and device interference in video frames, ensuring the clarity of the hand area. Grayscale processing converts color video frames into grayscale images, reducing image data volume and improving hand detection efficiency. Contrast adjustment enhances the distinction between the hand and the background by stretching the image's grayscale range, preventing hand detection failure due to excessively dark or bright lighting.

[0016] Furthermore, the vehicle control system calls a hand detection algorithm to extract hand structure information from each frame of grayscale image. The hand structure information includes hand contour, coordinates of key hand points (a total of 21, covering the wrist, finger joints and fingertips), palm size and palm contour features. The 21 key hand points are defined in a fixed order, corresponding to 1 wrist, 4 thumbs, 4 index fingers, 4 middle fingers, 4 ring fingers and 4 little fingers, completely covering all key positions of the hand and ensuring the integrity of hand structure information.

[0017] Furthermore, based on the hand structure information and palm orientation extracted from each video frame, the vehicle control system filters out the hand located in the preset driver's seat operating area to obtain the target hand. The target hand determines whether the hand to be detected is the left or right hand, as detailed in steps 101 to 104.

[0018] For example, the left and right hand recognition operation can be performed by comparing the relative positional relationship of 21 key points of the target hand, analyzing the difference in horizontal position between the metacarpophalangeal joint of the index finger and the metacarpophalangeal joint of the little finger, and combining the ratio of the coordinates of the wrist key points to the width of the video frame to determine the left and right attributes of the target hand.

[0019] In one optional embodiment, if the horizontal coordinate of the index finger metacarpophalangeal joint is less than the horizontal coordinate of the little finger metacarpophalangeal joint, and the horizontal coordinate of the wrist key point is located in the left third of the video frame width, then the target hand is determined to be the left hand; if the horizontal coordinate of the index finger metacarpophalangeal joint is greater than the horizontal coordinate of the little finger metacarpophalangeal joint, and the horizontal coordinate of the wrist key point is located in the left third of the video frame width, then the target hand is determined to be the right hand; if the target hand does not meet the above positional relationship, it is determined to be a non-driver's seat valid hand and is not included in subsequent processing.

[0020] In one embodiment, the vehicle control system's camera continuously captures video of the driver's hand at a frequency of 30 frames per second. At a given moment, a single frame is captured, pre-processed to obtain a clear grayscale image containing a hand region. The vehicle control system then uses a hand detection algorithm to extract the structural information of this hand region: identifying 21 key hand points, with the wrist key point coordinates at (150, 300), the thumb tip key point coordinates at (120, 220), the index finger tip key point coordinates at (100, 180), the middle finger tip key point coordinates at (130, 170), the ring finger tip key point coordinates at (160, 180), and the little finger tip key point coordinates at (190, 200). The palm outline is complete, and the palm faces the camera.

[0021] Based on the preset driver's seat operating area (the left third of the video frame width, i.e., horizontal coordinates 0-200, vertical coordinates 150-400), the wrist key point (150, 300) of the hand is determined to be within this area, and the palm orientation meets the preset requirements. Therefore, this hand is selected as the target hand. Left and right hand recognition is then performed. Comparing the horizontal positions of the index finger metacarpophalangeal joint (coordinates 110, 250) and the little finger metacarpophalangeal joint (coordinates 180, 260), the horizontal coordinate (110) of the index finger metacarpophalangeal joint is smaller than the horizontal coordinate (180) of the little finger metacarpophalangeal joint, and the horizontal coordinate (150) of the wrist key point is located in the left third of the video frame width (0-200). Therefore, the target hand is determined to be the left hand.

[0022] Step 20: Determine the gesture morphology features of the current video frame based on the first coordinate difference in the vertical direction between the first spatial key point coordinates of the fingertips of each finger in the target hand and the second spatial key point coordinates of their corresponding metacarpophalangeal joints.

[0023] Optionally, the vehicle control system extracts the coordinates of the first spatial key point of each fingertip and the coordinates of the second spatial key point of the corresponding metacarpophalangeal joint of each finger based on the hand structure information of the target hand. Each finger corresponds to a unique fingertip key point and metacarpophalangeal joint key point, specifically as follows: thumb corresponds to the thumb fingertip key point and thumb metacarpophalangeal joint key point; index finger corresponds to the index fingertip key point and index finger metacarpophalangeal joint key point; middle finger corresponds to the middle fingertip key point and middle finger metacarpophalangeal joint key point; ring finger corresponds to the ring fingertip key point and ring finger metacarpophalangeal joint key point; and little finger corresponds to the little fingertip key point and little finger metacarpophalangeal joint key point, ensuring accurate correspondence between the two key points of each finger.

[0024] Furthermore, the vehicle control system calculates the first coordinate difference in the vertical direction between the first spatial key point coordinates and the second spatial key point coordinates of each finger. The vertical direction is defined as the up and down direction of the video frame, that is, in the coordinate system, vertical downward is the positive direction and vertical upward is the negative direction.

[0025] In this embodiment of the invention, the first coordinate difference is calculated as follows: the vertical coordinate of the second spatial key point is subtracted from the vertical coordinate of the first spatial key point. If the calculation result is positive, it means that the fingertip key point is above the metacarpophalangeal joint key point; if the calculation result is negative, it means that the fingertip key point is below the metacarpophalangeal joint key point; if the calculation result is zero, it means that the fingertip key point and the metacarpophalangeal joint key point are aligned in the vertical direction.

[0026] Furthermore, the vehicle control system determines the raised state of each finger based on the first coordinate difference of each of the five fingers, combined with a preset difference threshold (the preset threshold is calibrated through a large number of experiments, with a value range of 15-25 pixels; in this embodiment of the invention, 20 pixels can be used as the standard threshold): if the first coordinate difference of a finger is greater than 20 pixels, it means that the fingertip is significantly higher than the metacarpophalangeal joint, and the finger is determined to be in a raised state; if the first coordinate difference of a finger is less than -20 pixels, it means that the fingertip is significantly lower than the metacarpophalangeal joint, and the finger is determined to be in a downward extended state; if the first coordinate difference of a finger is between -20 pixels and 20 pixels, it means that the finger is not significantly raised or downward extended, and the finger is determined to be in a bent state.

[0027] Furthermore, the vehicle control system determines the gesture shape features of the current video frame based on the raised, extended downward, or bent state of the five fingers, combined with four preset gesture shape feature definitions, as described in steps 201 to 204. In this embodiment of the invention, four gesture shape features are included, specifically defined as follows: "All five fingers raised upward" (all five fingers are raised, neither bent nor extended downward); "All five fingers raised downward" (all five fingers are extended downward, neither bent nor raised); "Thumb raised upward" (only the thumb is raised, the other four fingers are bent); "Thumb raised downward" (only the thumb is extended downward, the other four fingers are bent).

[0028] Step 30: If a gesture pattern template that matches the gesture pattern features exists, then the gesture pattern features of the current video frame are compared with the gesture pattern features of the previous video frame to obtain the number of consecutively appearing frames.

[0029] Optionally, the vehicle control system pre-stores four gesture pattern templates. Each gesture pattern feature corresponds to a unique gesture pattern template. The core content of the gesture pattern template is the standard state of the five fingers raised, extended downward, or bent for that gesture, as well as the standard range of the first coordinate difference of each finger. This corresponds completely to the definition of the four gesture pattern features in step 20, ensuring the consistency of template matching. Specifically, the templates in this embodiment are defined as follows: the template for raising all five fingers upward corresponds to the first coordinate difference of all five fingers being greater than 20 pixels; the template for raising all five fingers downward corresponds to the first coordinate difference of all five fingers being less than -20 pixels; the template for raising only the thumb upward corresponds to only the first coordinate difference of the thumb being greater than 20 pixels, while the first coordinate differences of the other four fingers are between -20 and 20 pixels; and the template for raising only the thumb downward corresponds to only the first coordinate difference of the thumb being less than -20 pixels, while the first coordinate differences of the other four fingers are between -20 and 20 pixels.

[0030] Optionally, the vehicle control system compares the gesture features of the current video frame with four pre-stored gesture templates one by one. The comparison process uses a feature matching algorithm, focusing on comparing the consistency between the state of the five fingers in the current gesture (raised, extended downward, bent) and the corresponding finger state in the template. At the same time, it compares the deviation of the first coordinate difference of each finger from the standard range of the template. The deviation threshold is preset to 5 pixels. If the deviation is less than 5 pixels and the finger state is completely consistent, the current gesture features are determined to match the gesture template. If they do not match any of the four templates, the current gesture is determined to be an invalid gesture. Subsequent frame counting is not performed, and the system directly returns to step 10 to continue processing the next frame of video.

[0031] Furthermore, the vehicle control system retrieves the gesture morphology features and corresponding frame count statistics of the previous video frame stored internally. The gesture morphology features of the previous video frame are the valid gesture features output in the previous step 20 (if it is the first valid gesture, the features of the previous frame are recorded as none). The frame count statistics include the counter for the number of consecutive frames corresponding to the current matching template. The initial value of the counter is 0. For each frame that matches the template, the counter value is incremented by 1. If a frame that does not match the template is detected (including invalid gestures or gestures that match other templates), the counter value is cleared to zero and the statistics start again.

[0032] Furthermore, the vehicle control system compares the gesture morphology features of the current video frame with those of the previous video frame to determine if they are consistent: if the previous frame is an invalid gesture and the current frame is a valid gesture that matches a certain template, the consecutive occurrence frame counter is set to 1; if the previous frame is a valid gesture that matches the gesture morphology features of the current frame and both match the same template, the consecutive occurrence frame counter is incremented by 1; if the previous frame is a valid gesture but does not match the gesture morphology features of the current frame (or matches a different template), the consecutive occurrence frame counter corresponding to the current frame is set to 1, and the counter corresponding to the previous frame is cleared, thus obtaining the consecutive occurrence frame number corresponding to the current gesture morphology features.

[0033] In one embodiment, for example, the gesture shape feature of the current video frame (the first frame of valid gestures) is that all five fingers are raised upwards. The vehicle control system compares this feature with four preset templates: all five fingers of the current gesture are raised, and the first coordinate difference of each finger (40, 70, 70, 70, 60 pixels) is greater than 20 pixels, which is completely consistent with the standard of the five fingers raised upwards template. The deviation is 0 pixels, which is less than the preset deviation threshold of 5 pixels. Therefore, it is determined that the current gesture shape feature matches the five fingers raised upwards template.

[0034] Further, the gesture information of the previous video frame is retrieved. The previous video frame is an invalid gesture (no template is matched). Therefore, the vehicle control system assigns the consecutive occurrence frame counter corresponding to the five fingers raised upward to 1, and the current consecutive occurrence frame count is 1. Continue processing the next video frame (referred to as frame 2). Step 20 determines that the gesture shape feature of this frame is still five fingers raised upward, which is consistent with the gesture shape feature of the previous frame (frame 1), and both match the five fingers raised upward template. Therefore, the consecutive occurrence frame counter is incremented by 1 based on the original value (1), and the current consecutive occurrence frame count is 2. Similarly, the video frames from frame 3 to frame 15 are processed. The gesture shape feature of each frame is five fingers raised upward, and the consecutive occurrence frame count is accumulated sequentially. When the frame is frame 15, the consecutive occurrence frame count is 15.

[0035] Step 40: If the number of consecutive frames reaches the preset continuous detection threshold, the vehicle actuator is controlled to perform the corresponding operation based on the operation command corresponding to the gesture shape feature of the current video frame.

[0036] Optionally, the vehicle control system internally presets a continuous detection threshold. Through extensive experimental calibration, combined with the real-time and accuracy requirements of gesture recognition, the preset continuous detection threshold is 15 frames, corresponding to a time of approximately 0.5 seconds (based on a collection frequency of 30 frames per second). It can be adjusted according to the actual usage scenario, but the adjustment range is limited to 10-20 frames to ensure the reasonableness of the threshold setting and avoid misoperation caused by an excessively low threshold.

[0037] Optionally, the vehicle control system compares the number of consecutive frames corresponding to the gesture morphology features of the current video frame with a preset continuous detection threshold. The comparison process uses a numerical comparison method to determine two comparison results: the first result is that the number of consecutive frames is greater than or equal to the preset continuous detection threshold (15 frames), indicating that the gesture is a valid gesture made intentionally by the user and is not a momentary erroneous action or false detection, thus meeting the conditions for executing the operation command; the second result is that the number of consecutive frames is less than the preset continuous detection threshold (15 frames), indicating that the gesture may be a momentary erroneous action or an incomplete gesture, thus not meeting the conditions for executing the operation command. In this case, the vehicle control system does not perform any operation and directly returns to step 10 to continue processing the next video frame and continuously count the number of consecutive frames.

[0038] Furthermore, if the number of consecutive frames reaches a preset continuous detection threshold, the vehicle control system retrieves the internal preset correspondence between gesture features and operation commands. Each valid gesture feature corresponds to a unique operation command. The correspondence is pre-stored in the system and matched with the vehicle actuator control logic of the ADAS intelligent simulation training cockpit to ensure that the command correspondence is without deviation.

[0039] Optionally, the specific correspondence in this embodiment of the invention is as follows: raising all five fingers upward corresponds to the sunroof raising operation command, raising all five fingers downward corresponds to the sunroof lowering operation command, raising one thumb upward corresponds to the driver's side window raising operation command, and raising one thumb downward corresponds to the driver's side window lowering operation command.

[0040] Furthermore, based on the operation command corresponding to the gesture shape characteristics of the current video frame, the vehicle control system sends a control signal to the vehicle actuator to control the vehicle actuator to perform the corresponding operation, as described in steps 401 to 403. After receiving the control signal, the vehicle actuator performs the corresponding operation and simultaneously feeds back the operation execution status (success or failure) to the vehicle control system. Upon receiving the feedback information, if the operation is successful, the vehicle control system maintains the current operation status, resets the consecutive frame counter to zero, and restarts subsequent gesture detection and statistics; if the operation fails, the vehicle control system issues a prompt signal and returns to step 10 to continue detecting gestures until the operation is successfully executed or the gesture disappears.

[0041] Furthermore, during operation, if the vehicle control system detects a new gesture feature (which is inconsistent with the gesture feature corresponding to the currently executed operation), and the number of consecutive frames of the new gesture reaches a preset threshold, the vehicle control system immediately stops the currently executed operation and controls the vehicle actuator to execute a new operation according to the operation instruction corresponding to the new gesture feature, ensuring the flexibility and real-time performance of the operation control.

[0042] Continuing with the above embodiment, the number of consecutive frames corresponding to the current gesture shape feature (all five fingers raised upward) is 15. The vehicle control system has a preset continuous detection threshold of 15 frames. The number of consecutive frames (15 frames) is compared with the preset threshold (15 frames). The number of consecutive frames is equal to the preset threshold. Therefore, it is determined that the gesture is a valid gesture made intentionally by the user and has the conditions to execute the operation command.

[0043] The vehicle control system retrieves the internally preset correspondence between gesture features and operation commands, determining that raising all five fingers upward corresponds to the sunroof raising operation command. It then sends a sunroof raising control signal to the vehicle's sunroof actuator. Upon receiving the control signal, the sunroof actuator initiates the sunroof raising action and simultaneously sends a "successful execution" status message back to the vehicle control system. After receiving the feedback, the vehicle control system resets the counter for consecutive frames corresponding to the raised fingers and returns to step 10 to process the next video frame, continuing gesture detection. If the consecutive frame count obtained in step 30 is 10 frames, which is less than the preset consecutive detection threshold of 15 frames, the vehicle control system determines the gesture as a momentary erroneous action, performs no operation, and directly returns to step 10 to process the next video frame, continuously counting the consecutive frame counts until the frame count reaches 15 or the gesture disappears.

[0044] This invention performs hand detection on consecutive video frames of a target user, obtaining hand structure information for the hand to be detected in each video frame. Combining this hand structure information with palm orientation, it filters out target hands located within a preset driver's seat operating area. This eliminates interference from hands belonging to non-drivers, addressing the difficulty of distinguishing gestures between the target operator and other individuals. Based on the first coordinate difference in the vertical direction between the first spatial keypoint coordinates of the fingertips and the second spatial keypoint coordinates of the corresponding metacarpophalangeal joints in the target hand, the gesture morphology features of the current video frame are obtained, achieving accurate depiction of the target hand gestures. The gesture morphology features of the current video frame are matched with a gesture morphology template and compared with the gesture morphology features of the previous video frame to obtain the number of consecutive occurrences, verifying the continuity of the target hand gestures and overcoming the shortcomings of relying solely on single-frame image features and lacking continuous verification of operational intent. The number of consecutive occurrences is used to determine if a preset continuous detection threshold has been reached. If so, the corresponding operation command is used to control the vehicle's actuator, resulting in accurate execution of the target operation command. This effectively avoids non-driver similar gestures and accidental misgestions being judged as valid commands, reducing false triggering and execution. By accurately executing target operation commands, the system eliminates the possibility of vehicle misoperation caused by gestures from non-target personnel, thereby solving the problems of accidental triggering and execution in multi-person scenarios and improving driving safety and system reliability.

[0045] Optionally, the processes of steps 101 to 104 include: Step 101: Based on the position coordinates of the wrist key points in the current video frame in the horizontal direction of the hand structure information, determine the horizontal coordinate of the wrist of the hand to be detected, and determine the positional relationship between the horizontal coordinate of the wrist and the left and right boundary coordinates of the preset driver's seat operation area in the horizontal direction of the image to determine whether the hand to be detected is within the horizontal range of the preset driver's seat operation area.

[0046] Optionally, the vehicle control system extracts the position coordinates of the wrist key points in the horizontal direction of the image from the hand structure information of the current video frame, which is the wrist horizontal coordinate of the hand to be detected.

[0047] The horizontal direction of the image represents the left and right directions of the video frame. The top left corner of the video frame is taken as the origin of the coordinate system, and the rightward direction is the positive direction of the horizontal direction of the image. The value of the wrist's horizontal coordinate directly reflects the specific position of the wrist in the left and right directions of the video frame. The smaller the value, the closer the wrist is to the left side of the video frame, and the larger the value, the closer the wrist is to the right side of the video frame.

[0048] Furthermore, the vehicle control system retrieves the left and right boundary coordinates of the preset driver's seat operating area in the horizontal direction of the image. The preset driver's seat operating area refers to the video frame range corresponding to the area that the driver's hands can normally reach when operating the vehicle. Its left and right boundary coordinates in the horizontal direction are determined through extensive experimental calibration, taking into account the driver's posture, arm range of motion, and camera installation position. The left and right boundary coordinates are fixed values ​​to ensure they are not arbitrarily changed after calibration, and they cover the entire horizontal range of normal hand operation by the driver, while avoiding the horizontal range of the passenger seat and other irrelevant areas.

[0049] Furthermore, the vehicle control system determines the positional relationship between the wrist coordinate of the hand to be detected and the left and right boundary coordinates of the preset driver's seat operating area. Optionally, the judgment logic of this embodiment is as follows: if the wrist coordinate is greater than or equal to the left boundary coordinate and less than or equal to the right boundary coordinate, the hand to be detected is determined to be within the horizontal range of the preset driver's seat operating area, and the spatial position determination result is "compliant with the horizontal range"; if the wrist coordinate is less than the left boundary coordinate or greater than the right boundary coordinate, the hand to be detected is determined to be not within the horizontal range of the preset driver's seat operating area, and the spatial position determination result is "not compliant with the horizontal range".

[0050] Step 102: Based on the spatial arrangement features of key points of each finger in the hand structure information of the current video frame, calculate the second coordinate difference between the metacarpophalangeal joint of the index finger and the metacarpophalangeal joint of the little finger in the horizontal direction of the image to obtain the lateral span index between the index finger and the little finger of the hand to be detected.

[0051] Optionally, the vehicle control system extracts the spatial arrangement features of key points of each finger from the hand structure information of the hand to be detected in the current video frame. The spatial arrangement features of each finger key point refer to the relative positional relationship of all key points of the hand (including fingertips, metacarpophalangeal joints, etc.) in the image, which can fully reflect the overall shape of the hand and the extension state of the fingers.

[0052] Furthermore, the vehicle control system accurately locates and extracts the key coordinates of the index finger metacarpophalangeal joint and the little finger metacarpophalangeal joint from the spatial arrangement features of the finger key points. Specifically, the index finger metacarpophalangeal joint key point is the only key point at the connection between the base of the index finger and the palm, and the little finger metacarpophalangeal joint key point is the only key point at the connection between the base of the little finger and the palm. Both key points belong to the 21 key points of the hand to be detected, and their coordinates include positional information in both the horizontal and vertical directions of the image.

[0053] Furthermore, the vehicle control system extracts the coordinates of the key points of the index finger metacarpophalangeal joint in the horizontal direction of the image, and the coordinates of the key points of the little finger metacarpophalangeal joint in the horizontal direction of the image.

[0054] Further, the vehicle control system calculates the second coordinate difference between the metacarpophalangeal joint of the index finger and the metacarpophalangeal joint of the little finger in the horizontal direction of the image. Optionally, the calculation algorithm of this embodiment is as follows: subtract the coordinate of the metacarpophalangeal joint of the index finger from the coordinate of the metacarpophalangeal joint of the little finger in the horizontal direction of the image, and the obtained value is the second coordinate difference. The second coordinate difference is defined as the lateral span index between the index finger and the little finger of the hand to be detected. The value of this index reflects the horizontal extension width of the index finger and the little finger of the hand to be detected. The larger the value, the wider the extension of the two fingers, and the smaller the value, the narrower the extension of the two fingers.

[0055] Step 103: Based on the lateral span index and the preset frontal orientation threshold, determine the palm orientation state to obtain the palm orientation determination result of the hand to be detected.

[0056] Optionally, the vehicle control system retrieves a pre-set forward orientation threshold. This forward orientation threshold refers to the standard range threshold value of the lateral span between the index finger and little finger when the palm faces the camera. It is calibrated through numerous experiments. During the calibration process, images of the hands of different drivers facing the camera are collected, and the lateral span between the index finger and little finger in each image is calculated. The average value of all indicators is taken as the forward orientation threshold, ensuring that the threshold can accurately distinguish between a forward-facing and non-forward-facing hand, and that this threshold is a fixed value.

[0057] Furthermore, the vehicle control system determines the palm orientation state based on the lateral span index and a preset frontal orientation threshold. Optionally, the determination logic of this embodiment is as follows: if the lateral span index is greater than or equal to the preset frontal orientation threshold, it indicates that the index finger and little finger of the hand to be detected are in a relatively wide extended state, which meets the characteristic of the palm facing the camera. The palm orientation of the hand to be detected is determined to be frontal, and the palm orientation determination result is "frontal". If the lateral span index is less than the preset frontal orientation threshold, it indicates that the index finger and little finger of the hand to be detected are in a relatively narrow extended state, which does not meet the characteristic of the palm facing the camera. The palm orientation of the hand to be detected is determined to be non-frontal, and the palm orientation determination result is "non-frontal".

[0058] Step 104: Determine the target hand based on the spatial location determination result and the palm orientation determination result.

[0059] Optionally, the vehicle control system determines the target hand based on the spatial position determination result and the palm orientation determination result, as in steps 1041 to 1044.

[0060] This invention achieves precise screening of valid hands in the driver's seat through dual verification of horizontal spatial position and palm orientation. It can effectively eliminate interference from the hands of non-driver's seat personnel such as passengers in the front passenger seat and bystanders, while also eliminating invalid hands with abnormal palm orientation in the driver's seat. This ensures that subsequent gesture recognition is only performed on valid hands facing the driver's seat, reducing the risk of misidentification caused by non-target hands from the source and improving the accuracy and reliability of gesture control.

[0061] Optionally, the process of steps 1041 to 1044 includes: Step 1041: If the palm orientation determination result indicates that the palm face is facing upwards, and the spatial position determination result indicates that it is within the horizontal range, then the hand to be detected is determined to be a valid candidate hand.

[0062] Optionally, the vehicle control system comprehensively judges the palm orientation determination result and the spatial position determination result. Optionally, the judgment logic of this embodiment is as follows: the palm orientation determination result indicates the palm facing state, that is, the palm of the hand to be detected is facing the camera. This state ensures that the key points of the hand can be clearly identified, providing accurate coordinate data support for subsequent left and right hand recognition; the spatial position determination result indicates that it is within the horizontal range, that is, the horizontal coordinate of the wrist of the hand to be detected is between the left and right boundary coordinates of the preset driver's seat operation area, ensuring that the hand to be detected is within the horizontal space of the driver's seat for normal operation, excluding irrelevant hands from non-driver's seats.

[0063] Furthermore, if the hand to be detected simultaneously meets both of the above conditions, namely, the palm orientation determination result is a frontal orientation and the spatial position determination result is within the horizontal range, then the vehicle control system determines the hand to be detected as a valid candidate hand, stores the relevant structural information of the valid candidate hand (including the coordinate information of the wrist key points, the key points of each finger metacarpophalangeal joint, etc.) and transmits it to subsequent steps; if the hand to be detected does not simultaneously meet the above two conditions, that is, at least one determination result does not meet the preset requirements, then the vehicle control system determines the hand to be detected as an invalid hand, does not include it in the category of valid candidate hands, does not perform subsequent steps, and directly terminates the target hand determination process of the hand to be detected.

[0064] Step 1042: Based on the relative positional relationship between the horizontal coordinates of the wrist key points of the valid candidate hand and the horizontal centerline of the preset driver's seat operating area, determine the horizontal offset of the valid candidate hand. The horizontal centerline is set based on the image coordinate system corresponding to the structural features of the vehicle cabin, and is located at the projection position of the driver's seat centerline in the image.

[0065] Optionally, the vehicle control system extracts the coordinates of the wrist key points of the valid candidate hand in the horizontal direction of the image from the hand structure information of the valid candidate hand. These coordinates are the specific positions of the wrist of the valid candidate hand in the left-right direction of the video frame.

[0066] Furthermore, the vehicle control system retrieves the preset horizontal center line of the driver's seat operating area. The horizontal center line is set based on the image coordinate system corresponding to the structural features of the vehicle cabin. The image coordinate system is completely consistent with the image coordinate system used in steps 101, 102, and 103, that is, the upper left corner of the video frame is the origin of the coordinate system, the horizontal direction to the right is the positive direction of the horizontal direction of the image, and the vertical direction downward is the positive direction of the vertical direction of the image. The horizontal center line is located at the projection position of the driver's seat center axis in the image. The driver's seat center axis is the symmetrical center line of the driver's seat in the vehicle cabin structure. Its projection position in the image is determined through a large number of experiments, combined with the vehicle cabin structure dimensions, camera installation position, and standard position of the driver's seat. The horizontal center line is a virtual straight line in the horizontal direction of the image, corresponding to a unique horizontal coordinate value. This coordinate value is a fixed value to ensure the stability of the horizontal center line position.

[0067] Further, the vehicle control system calculates the horizontal offset of the valid candidate hand. Optionally, the calculation algorithm of this embodiment is as follows: subtract the horizontal coordinate value corresponding to the horizontal center line of the preset driver's seat operation area from the coordinate value of the wrist key point of the valid candidate hand in the horizontal direction of the image. The obtained value is the horizontal offset of the valid candidate hand. The sign of the horizontal offset value reflects the left and right position of the valid candidate hand relative to the horizontal center line, and the magnitude of the value reflects the degree of offset. When the horizontal offset is less than or equal to zero, it indicates that the valid candidate hand is located to the left of the horizontal center line or coincides with the horizontal center line. When the horizontal offset is greater than zero, it indicates that the valid candidate hand is located to the right of the horizontal center line.

[0068] Step 1043: If the horizontal offset is less than or equal to zero, the valid candidate hand is determined to be located to the left or above the center line. Verify whether the horizontal coordinate value of the index finger metacarpophalangeal joint of the valid candidate hand is greater than the horizontal coordinate value of the thumb metacarpophalangeal joint. If it matches, the target hand is confirmed to be the left hand.

[0069] Optionally, the vehicle control system makes a numerical judgment on the horizontal offset of the valid candidate hand. If the horizontal offset is less than or equal to zero, the vehicle control system determines that the valid candidate hand is located to the left or above the horizontal center line of the preset driver's seat operating area, which is the main area for normal activity of the driver's left hand.

[0070] Furthermore, the vehicle control system extracts the horizontal coordinate values ​​of the index finger metacarpophalangeal joint and the thumb metacarpophalangeal joint from the hand structure information of the valid candidate hand. The index finger metacarpophalangeal joint is the only key point at the connection between the base of the index finger and the palm, and the thumb metacarpophalangeal joint is the only key point at the connection between the base of the thumb and the palm. Both key points belong to the 21 key points of the valid candidate hand.

[0071] Furthermore, the vehicle control system verifies the positional relationship between the two extracted coordinate values. Specifically, the verification logic in this embodiment is as follows: It determines whether the horizontal coordinate value of the index finger metacarpophalangeal joint of the valid candidate hand is greater than the horizontal coordinate value of the thumb metacarpophalangeal joint. This verification logic is based on the physiological structural characteristics of the human hand. When the driver's left hand faces the camera, the index finger is located to the right of the thumb, and its metacarpophalangeal joint's horizontal coordinate is naturally greater than the thumb's. This feature is unique and stable, and can be used as the core basis for left-hand identification. If the verification result is positive, i.e., the horizontal coordinate value of the index finger metacarpophalangeal joint is greater than the horizontal coordinate value of the thumb metacarpophalangeal joint, the vehicle control system confirms that the valid candidate hand is the left hand among the target hands. If the verification result is negative, the vehicle control system determines that the valid candidate hand is an invalid hand and terminates the target hand determination process.

[0072] Step 1044: If the horizontal offset is greater than zero, the valid candidate hand is determined to be located to the right of the center line. Verify whether the horizontal coordinate value of the thumb metacarpophalangeal joint of the valid candidate hand is greater than the horizontal coordinate value of the index finger metacarpophalangeal joint. If they match, the target hand is confirmed to be the right hand.

[0073] Optionally, the vehicle control system makes a numerical judgment on the horizontal offset of the valid candidate hand. If the horizontal offset is greater than zero, the vehicle control system determines that the valid candidate hand is located to the right of the horizontal center line of the preset driver's seat operating area, which is the main area for normal right-hand movement of the driver in the vehicle.

[0074] Furthermore, the vehicle control system extracts the horizontal coordinate values ​​of the thumb metacarpophalangeal joint and the index finger metacarpophalangeal joint from the hand structure information of the valid candidate hands.

[0075] Furthermore, the vehicle control system verifies the positional relationship between the two extracted coordinate values. Optionally, the verification logic in this embodiment of the invention is as follows: determine whether the horizontal coordinate value of the thumb metacarpophalangeal joint of the valid candidate hand is greater than the horizontal coordinate value of the index finger metacarpophalangeal joint. This verification logic is also based on the physiological structural characteristics of the human hand. When the driver's right hand is facing the camera, the thumb is located to the right of the index finger, and the horizontal coordinate of its metacarpophalangeal joint is naturally greater than the horizontal coordinate of the index finger metacarpophalangeal joint. This feature has uniqueness and stability and can be used as the core judgment basis for right hand recognition. If the verification result is consistent, that is, the horizontal coordinate value of the thumb metacarpophalangeal joint is greater than the horizontal coordinate value of the index finger metacarpophalangeal joint, then the vehicle control system confirms that the valid candidate hand is the right hand among the target hands.

[0076] If the verification result is not compliant, the vehicle control system determines that the valid candidate hand is invalid and terminates the target hand determination process.

[0077] This invention achieves accurate left and right hand recognition of valid candidate hands, ensuring that the selected target hands are not only located in the driver's seat operating area and have the correct palm orientation, but also accurately distinguish between left and right hand attributes. It effectively eliminates invalid hands with abnormal palm orientation, unreasonable position offset, or incorrect left and right hand recognition within the driver's seat, ensuring that subsequent gesture morphology feature recognition and operation command matching are only performed on the valid left and right hand gestures of the driver's seat personnel. This improves the accuracy of target hand recognition from the source, eliminates interference from the hands of non-driver's seat personnel, and makes up for the shortcomings of only screening position and orientation without distinguishing between left and right hands, which can easily lead to confusion of gesture commands. It improves the accuracy and reliability of gesture control, thereby solving the problem of false triggering and false execution in multi-person scenarios, and improving vehicle safety and system stability.

[0078] Optionally, the processes of steps 201 to 204 include: Step 201: Construct a vertical difference vector for each finger based on the vertical coordinate difference between the first spatial key point coordinates of the fingertip of each finger in the target hand and the second spatial key point coordinates of the corresponding metacarpophalangeal joint.

[0079] Optionally, each fingertip corresponds one-to-one with a key point of the metacarpophalangeal joint, that is, the thumb corresponds to the first spatial key point of the thumb tip and the second spatial key point of the thumb metacarpophalangeal joint, the index finger corresponds to the first spatial key point of the index finger tip and the second spatial key point of the index finger metacarpophalangeal joint, the middle finger corresponds to the first spatial key point of the middle finger tip and the second spatial key point of the middle finger metacarpophalangeal joint, the ring finger corresponds to the first spatial key point of the ring finger tip and the second spatial key point of the ring finger metacarpophalangeal joint, and the little finger corresponds to the first spatial key point of the little finger tip and the second spatial key point of the little finger metacarpophalangeal joint.

[0080] Optionally, the vehicle control system calculates the vertical coordinate difference between the coordinates of the first spatial key point and the corresponding second spatial key point of each finger. The vertical direction is the up-down direction of the video frame, with the top left corner of the video frame as the origin. Vertical downward is the positive vertical direction, and vertical upward is the negative vertical direction. Optionally, the calculation algorithm of this embodiment is as follows: subtract the vertical coordinate value of the first spatial key point of the fingertip from the vertical coordinate value of the second spatial key point of the metacarpophalangeal joint corresponding to each finger. The obtained value is the vertical coordinate difference of the finger. This difference can accurately reflect the vertical position relationship of the fingertip relative to the metacarpophalangeal joint. A positive difference indicates that the fingertip is above the metacarpophalangeal joint, a negative difference indicates that the fingertip is below the metacarpophalangeal joint, and a difference of zero indicates that the fingertip and the metacarpophalangeal joint are aligned in the vertical direction.

[0081] Furthermore, the vehicle control system integrates the vertical coordinate differences of the five fingers according to a preset finger order to construct a finger vertical difference vector. The preset finger order is thumb, index finger, middle finger, ring finger, and little finger. This order is fixed and all subsequent steps follow this order for analysis to ensure data processing consistency. The finger vertical difference vector is an ordered data set containing five values, each corresponding to the vertical coordinate difference of the thumb, index finger, middle finger, ring finger, and little finger in sequence.

[0082] Step 202: Based on the differences between the vertical coordinates of the index finger and thumb, the middle finger and index finger, the ring finger and middle finger, and the little finger and ring finger, construct a finger extension gradient sequence, and analyze the number of sign changes of adjacent gradient values ​​from positive to negative or from negative to positive based on the finger extension gradient sequence.

[0083] Optionally, the vehicle control system extracts the vertical coordinate differences of the five fingers in the vertical difference vector. The extraction order is consistent with the vector construction order, that is, the vertical coordinate differences of the thumb, index finger, middle finger, ring finger and little finger are extracted in sequence to ensure that the extracted data completely corresponds to the data in the vector.

[0084] Furthermore, the vehicle control system calculates the difference between the vertical coordinates of adjacent fingers one by one in the order of adjacent fingers to construct a finger extension gradient sequence. Optionally, the specific calculation logic of this embodiment is as follows: first, calculate the difference between the vertical coordinates of the index finger and the thumb by subtracting the vertical coordinate difference of the thumb from the vertical coordinate difference of the index finger; then calculate the difference between the vertical coordinates of the middle finger and the index finger by subtracting the vertical coordinate difference of the index finger from the vertical coordinate difference of the middle finger; finally, calculate the vertical coordinates of the ring finger and the middle finger. The difference between the values ​​is calculated by subtracting the difference in the vertical coordinates of the middle finger from the difference in the vertical coordinates of the ring finger. Finally, the difference between the vertical coordinates of the little finger and the ring finger is calculated by subtracting the difference in the vertical coordinates of the ring finger from the difference in the vertical coordinates of the little finger. The differences obtained from the above four calculations are integrated in the order of calculation to obtain the finger extension gradient sequence. Therefore, the finger extension gradient sequence is an ordered data set containing four values, each value corresponding to a set of vertical differences between adjacent fingers, which can reflect the changing gradient of the vertical positional relationship between adjacent fingers.

[0085] Furthermore, the vehicle control system analyzes the constructed finger extension gradient sequence, counting the number of sign changes between adjacent gradient values, where the sign changes from positive to negative or vice versa. A sign change refers to a situation where the sign (positive or negative) of one gradient value in the finger extension gradient sequence differs from that of the next. Specifically, this includes two cases: the previous gradient value is positive and the next is negative (sign changes from positive to negative); or the previous gradient value is negative and the next is positive (sign changes from negative to positive). If two adjacent gradient values ​​have the same sign (both positive or both negative), or if one of the gradient values ​​is zero, then it is not counted as a sign change. The vehicle control system compares the signs of adjacent gradient values ​​in the finger extension gradient sequence one by one from left to right, counting the number of sign changes for each pair.

[0086] Step 203: If the number of symbol changes is higher than the preset threshold, then the gesture morphology feature of the current video frame is determined to be an invalid gesture morphology feature.

[0087] Optionally, the vehicle control system retrieves a preset threshold number of times. This threshold number is a critical value used to distinguish between valid and invalid gestures. Through extensive experimental calibration, samples of various valid gestures (all five fingers raised upwards, all five fingers raised downwards, thumb raised upwards, thumb raised downwards) and various invalid gestures (random finger extensions, irregular bending, etc.) from the driver are collected during the calibration process. The number of symbol changes for each type of sample is counted. Combined with the symbol change pattern of valid gestures, the specific value of the preset threshold number is determined. The value range is 2-3 times to ensure accurate distinction between valid and invalid gestures. The function of the preset threshold number is to define the orderliness of gestures. Valid gestures have regular finger extensions and fewer symbol changes, while invalid gestures have random finger extensions and more symbol changes.

[0088] Furthermore, the vehicle control system compares the statistically obtained number of symbol changes with a preset threshold value. Optionally, the comparison logic of this embodiment is as follows: if the number of symbol changes is higher than the preset threshold value, it indicates that the symbol changes of adjacent gradient values ​​in the finger extension gradient sequence are frequent, and the vertical position relationship of the corresponding fingers changes randomly, which does not conform to the extension law of a valid gesture. That is, the gesture has no clear operational intention. At this time, the vehicle control system determines that the gesture morphology feature of the current video frame is an invalid gesture morphology feature, terminates the determination process of the current gesture morphology feature, does not perform subsequent steps, and returns to step 10 to reprocess the target hand in the next video frame.

[0089] Step 204: If the number of symbol changes is lower than the preset threshold, the gesture shape features of the current video frame are determined based on the vertical coordinate difference of each finger in the vertical difference vector.

[0090] Optionally, the vehicle control system determines the gesture shape features of the current video frame by combining the vertical coordinate difference of each finger in the vertical difference vector of the fingers with four gesture shape features (all five fingers raised upward, all five fingers raised downward, thumb raised upward, and thumb raised downward) to perform a segmentation, as in steps 2041 to 2043.

[0091] This invention achieves accurate recognition of valid gesture morphological features through orderliness verification, effectively eliminating interference from messy and invalid gestures. It ensures that the extracted gesture morphological features can accurately reflect the operating intentions of the driver, improving the accuracy and reliability of gesture recognition. It overcomes the shortcomings of only judging the finger state without verifying the orderliness of the gesture, which easily leads to misjudging messy gestures as valid gestures. This indirectly reduces the phenomenon of false triggering and false execution of gesture control, and improves driving safety and system stability.

[0092] Optionally, the processes of steps 2041 to 2043 include: Step 2041: If the vertical coordinate difference of each finger, determined based on the vertical difference vector, is greater than or equal to a preset lift-off threshold, then determine whether the absolute value of any gradient value in the finger extension gradient sequence is less than a preset smoothing tolerance threshold. If so, generate a first gesture candidate with the palm flat and raised, and determine the pointing direction of the first finger based on the sign of the median of the vertical coordinate difference of each finger. Based on the first gesture candidate and the pointing direction of the first finger, determine the gesture morphology features of the current video frame.

[0093] Optionally, the vehicle control system retrieves a preset lift-up threshold, which is a critical value used to determine whether a finger is in a lifted state. This threshold ranges from 15 to 25 pixels and defines whether the fingertip is significantly higher than the metacarpophalangeal joint. A difference greater than or equal to this threshold indicates that the finger is in a lifted state. Further, the vehicle control system compares the vertical coordinate difference of each finger with the preset lift-up threshold one by one, determining whether the vertical coordinate difference of each finger is greater than or equal to the preset lift-up threshold. This comparison process must cover all five fingers without omission or deviation. Only when the vertical coordinate differences of all five fingers meet the condition of being greater than or equal to the preset lift-up threshold will the subsequent judgment process proceed; otherwise, no first gesture candidate will be generated.

[0094] If the above comparison conditions are met, the vehicle control system extracts the absolute value of each gradient value from the finger extension gradient sequence. The absolute value of the gradient value is used to reflect the change range of the vertical difference between adjacent fingers. The smaller the absolute value, the closer the lifting height of adjacent fingers is, and the more the palm tends to be flat. At the same time, the vehicle control system retrieves the preset smoothing tolerance threshold, which is a critical value used to determine whether the palm is flat. It is calibrated through a large number of experiments and has a value range of 3-8 pixels. It is used to define the degree of difference in the lifting height of adjacent fingers. The smaller the difference, the flatter the palm.

[0095] Furthermore, the vehicle control system determines whether the absolute value of any gradient value in the finger extension gradient sequence is less than a preset smoothing tolerance threshold. Here, any gradient value refers to the four gradient values ​​contained in the finger extension gradient sequence. It is necessary to verify one by one that the absolute value of each gradient value meets the condition of being less than the preset smoothing tolerance threshold, to ensure that the difference in the lifting height of adjacent fingers is small and the palm is in a flat state. If this condition is met, the vehicle control system generates a first gesture candidate with the palm flat and raised. The first gesture candidate is used to characterize the current gesture as a valid gesture of five fingers flat and raised.

[0096] Furthermore, the vehicle control system calculates the median based on the vertical coordinate difference of each finger and determines the sign of the median to determine the pointing direction of the first finger. Optionally, the algorithm for calculating the median in this embodiment of the invention is as follows: arrange the vertical coordinate differences of the five fingers in ascending order, and take the value in the middle position after arrangement as the median; the sign of the median is consistent with the sign definition of the vertical coordinate difference of the fingers, a positive value indicates that the fingertip is above the metacarpophalangeal joint, that is, the finger is pointing upward, and a negative value indicates that the fingertip is below the metacarpophalangeal joint, that is, the finger is pointing downward. The sign of the median can reflect the overall pointing trend of the five fingers, ensuring the accuracy of the pointing direction determination.

[0097] Furthermore, the vehicle control system determines the gesture morphology features of the current video frame based on the generated first gesture candidate and the determined first finger pointing direction. Specifically, a palm raised flat and pointing upward corresponds to a gesture with all five fingers raised upward, and a pointing downward corresponds to a gesture with all five fingers raised downward, ensuring the accuracy and uniqueness of the feature determination.

[0098] Optionally, for the first gesture candidate, if the direction the first finger points is positive (i.e., upward), then the gesture shape feature of the current video frame is determined to be all five fingers raised upward, specifically as follows: Figure 2 As shown, Figure 2 This is one of the schematic diagrams of gesture morphology features provided in the embodiments of the present invention. If the direction in which the first finger points is negative, that is, downward, then the gesture morphology feature of the current video frame is determined to be all five fingers raised downward, specifically as follows: Figure 3 As shown, Figure 3 This is the second schematic diagram of the gesture morphology features provided in the embodiments of the present invention.

[0099] Step 2042: If the vertical coordinate difference of the thumb is greater than or equal to a preset lift threshold, and the vertical coordinate differences of all non-thumb fingers are less than a preset close threshold, then finger trend analysis is performed based on the vertical coordinate difference of each non-thumb finger to obtain a finger trend judgment result. If the finger trend judgment result indicates a non-increasing monotonic trend or a non-decreasing monotonic trend, then a second gesture candidate for lifting the isolated main finger is generated, with the sign of the vertical coordinate difference of the thumb as the second pointing direction. Based on the second gesture candidate and the second finger pointing direction, the gesture morphology features of the current video frame are determined. The finger trend judgment result represents whether it exhibits a non-increasing monotonic trend or a non-decreasing monotonic trend.

[0100] Optionally, the vehicle control system retrieves internally preset lift judgment threshold and preset fold judgment threshold. The preset fold judgment threshold is a critical value used to determine whether the finger is in a folded (bent) state. It is calibrated through a large number of experiments and has a value range of -8 to 8 pixels. It is used to define whether the fingertip is aligned or nearly aligned with the metacarpophalangeal joint in the vertical direction, that is, the finger is in a folded or bent state. If the vertical coordinate difference of the non-thumb finger is less than the threshold, it indicates that the non-thumb finger is in a folded state.

[0101] Furthermore, the vehicle control system performs a dual condition comparison: first, it determines whether the vertical coordinate difference of the thumb is greater than or equal to the preset lift threshold, i.e., whether the thumb is in a lifted state; second, it determines whether the vertical coordinate difference of all non-thumb fingers is less than the preset close threshold, i.e., whether all non-thumb fingers are in a closed state. Both conditions must be met simultaneously without deviation or omission. Only when the thumb is lifted and all non-thumb fingers are closed will the subsequent judgment process begin; otherwise, no second gesture candidate will be generated.

[0102] Furthermore, if the above dual conditions are met, the vehicle control system performs finger trend analysis based on the vertical coordinate difference of each non-thumb finger to obtain a finger trend judgment result. Optionally, the algorithm for finger trend analysis in this embodiment of the invention is as follows: The vertical coordinate differences of the four non-thumb fingers (index, middle, ring, and little fingers) are arranged in a fixed order: index, middle, ring, and little fingers. The vertical coordinate differences between adjacent non-thumb fingers are compared one by one to determine the overall trend. The finger trend judgment result indicates whether the vertical coordinate differences of the non-thumb fingers exhibit a non-increasing monotonic trend or a non-decreasing monotonic trend. A non-increasing monotonic trend means that the vertical coordinate difference between two adjacent non-thumb fingers is less than or equal to the previous one, indicating an overall unchanged or decreasing trend. A non-decreasing monotonic trend means that the vertical coordinate difference between two adjacent non-thumb fingers is greater than or equal to the previous one, indicating an overall unchanged or increasing trend. This trend analysis is used to verify the consistency of the non-thumb finger's retraction state and avoid the appearance of disordered extension of the non-thumb fingers.

[0103] Furthermore, the vehicle control system determines whether the finger trend judgment result indicates a non-increasing monotonic trend or a non-decreasing monotonic trend. Only when this condition is met does it indicate that the closing state of the non-thumb fingers is consistent, without disorderly extension, and conforms to the gesture characteristics of an isolated thumb raised. At this time, the vehicle control system generates a second gesture candidate for an isolated thumb raised. The second gesture candidate is used to characterize the current gesture as a valid gesture of a single thumb raised, and the candidate result is stored. If the finger trend judgment result does not meet this condition, it indicates that the closing state of the non-thumb fingers is inconsistent and there is disorderly extension, and no second gesture candidate is generated.

[0104] Furthermore, the vehicle control system uses the sign of the vertical coordinate difference of the thumb as the second pointing direction. A positive vertical coordinate difference indicates that the thumb tip is above the metacarpophalangeal joint, i.e., the second pointing direction is upward; a negative value indicates that the thumb tip is below the metacarpophalangeal joint, i.e., the second pointing direction is downward. The pointing direction of the thumb directly reflects the operational intention of this type of gesture, ensuring the accuracy of the pointing direction judgment.

[0105] Furthermore, the vehicle control system determines the gesture morphology features of the current video frame based on the generated second gesture candidate and the determined second pointing direction. Among them, the isolated thumb raising combined with the pointing upward corresponds to the thumb raising single finger upward gesture, and the pointing downward corresponds to the thumb raising single finger downward gesture.

[0106] Optionally, for the second gesture candidate, if the direction the second finger points is positive, then the gesture shape feature of the current video frame is determined to be a single thumb raised upwards. If the target hand at this time is the left hand, then the gesture shape feature is as follows: Figure 4 As shown, Figure 4 This is the third schematic diagram of gesture morphology features provided in this embodiment of the invention. If the target hand is the right hand, the gesture morphology features are as follows: Figure 5 As shown, Figure 5 This is the fourth schematic diagram of the gesture morphology features provided in the embodiments of the present invention.

[0107] Optionally, for the second gesture candidate, if the direction of the second finger is negative, then the gesture shape feature of the current video frame is determined to be a single thumb raised and pointing downwards. If the target hand at this time is the left hand, then the gesture shape feature is as follows: Figure 6 As shown, Figure 6 This is the fifth schematic diagram of gesture morphology features provided in this embodiment of the invention. If the target hand is the right hand, the gesture morphology features are as follows: Figure 7 As shown, Figure 7 This is the sixth schematic diagram of the gesture morphology features provided in the embodiments of the present invention.

[0108] Step 2043: If neither the first gesture candidate nor the second gesture candidate is generated, then the gesture morphology feature of the current video frame is determined to be an invalid gesture morphology feature.

[0109] Optionally, if neither the first gesture candidate nor the second gesture candidate is generated, it means that the current gesture does not conform to the valid gesture feature of raising the palm flat, nor does it conform to the valid gesture feature of raising the isolated main finger. It is a gesture without a clear operational intention and cannot correspond to the four valid gesture morphological features. The vehicle control system determines that the gesture morphological feature of the current video frame is an invalid gesture morphological feature.

[0110] This invention achieves accurate recognition of four valid gesture morphological features through multi-dimensional condition verification and trend analysis. It can effectively distinguish different types of valid gestures while completely eliminating invalid gestures that do not meet the requirements. It further refines the judgment criteria for gesture morphological features, making up for the shortcomings of judging gestures solely by finger status, which is prone to misjudgment. It effectively reduces the phenomenon of false triggering and false execution caused by gesture judgment deviation, improves the accuracy and reliability of gesture control, solves the problem of misoperation in multi-person scenarios, and improves driving safety and system stability.

[0111] Optionally, the processes of steps 401 to 403 include: Step 401: For the target hand being the left hand, if the gesture characteristic of the current video frame is a single thumb raised upwards, then control the driver's side window actuator to perform an upward operation. If the gesture characteristic of the current video frame is a single thumb raised downwards, then control the driver's side window actuator to perform a downward operation.

[0112] Optionally, if the target hand attribute is determined to be a left hand, the vehicle control system determines whether the gesture shape feature is a thumb-raised single-finger gesture, i.e., thumb raised upwards or thumb raised downwards. This gesture shape feature is a valid result obtained after filtering and verification in steps 201 to 204 in step 20, and invalid gestures have been excluded. The vehicle control system does not need to repeat the verification of the gesture validity. Further, the vehicle control system performs a specific determination of the gesture shape feature. Optionally, the determination logic of this embodiment is as follows: if the gesture shape feature of the current video frame is a thumb-raised single-finger gesture, i.e., only the left thumb is raised and pointing upwards, while the other four fingers are closed, this gesture corresponds to the intention to raise the driver's side window. The vehicle control system immediately sends a raising control signal to the driver's side window actuator. The driver's side window actuator is the core actuator used to control the raising and lowering of the driver's side window. It can receive the control signal from the vehicle control system and convert it into mechanical action to realize the raising or lowering of the window.

[0113] If the gesture in the current video frame is characterized by a single thumb raised and pointing downwards (i.e., only the left thumb is raised and pointing downwards, while the other four fingers are folded), this gesture corresponds to the intention to lower the driver's side window. The vehicle control system immediately sends a lowering control signal to the driver's side window actuator.

[0114] Furthermore, after receiving the corresponding control signal sent by the vehicle control system, the driver's window actuator performs the corresponding raising or lowering operation, and at the same time feeds back the operation execution status to the vehicle control system, including whether the execution was successful or failed.

[0115] Upon receiving feedback, if the execution is successful, the vehicle control system maintains the current control state and resets the counter for consecutively occurring frames counted in step 30, restarting the subsequent gesture detection and statistics process. If the execution fails, a prompt signal is issued, and the system returns to step 10 to continue detecting the target hand gesture until the operation is successfully executed or the gesture disappears, ensuring the closed-loop nature and reliability of the operation control.

[0116] Step 402: If the target hand is the right hand, and the gesture characteristic of the current video frame is a single thumb raised upwards, then control the passenger-side window actuator to perform an upward operation. If the gesture characteristic of the current video frame is a single thumb raised downwards, then control the passenger-side window actuator to perform a downward operation.

[0117] Optionally, if the target hand attribute is determined to be the right hand, the vehicle control system determines whether the gesture shape feature is a single thumb raised gesture, that is, a single thumb raised upward or a single thumb raised downward.

[0118] Furthermore, the vehicle control system makes a specific determination of the gesture shape characteristics. Optionally, the determination logic of this embodiment is consistent with the determination logic of the single thumb gesture in step 401, that is: if the gesture shape characteristics of the current video frame are that the thumb is raised and pointing upward, that is, only the right thumb is raised and pointing upward, while the other four fingers are closed, the gesture corresponds to the intention to raise the passenger side window, and the vehicle control system immediately sends a raising control signal to the passenger side window actuator; the passenger side window actuator is the core actuator used to control the raising and lowering of the passenger side window, and has the same structure as the driver's side window actuator, and can receive control signals and perform corresponding mechanical actions.

[0119] If the gesture in the current video frame is characterized by a single thumb raised and pointing downwards (i.e., only the right thumb is raised and pointing downwards, while the other four fingers are folded), this gesture corresponds to the intention to lower the passenger-side window. The vehicle control system immediately sends a lowering control signal to the passenger-side window actuator.

[0120] Furthermore, after receiving the corresponding control signal, the passenger-side window actuator performs the corresponding raising or lowering operation and reports the operation execution status back to the vehicle control system. Upon receiving the feedback information, the vehicle control system processes it according to the closed-loop logic of step 401. If the execution is successful, the continuous frame count counter is reset to zero; if the execution fails, a prompt is issued and the system returns to step 10 for re-detection, ensuring the rigor and reliability of the operation control.

[0121] Step 403: For the target hand being the right hand, if the gesture characteristic of the current video frame is all five fingers raised upwards, then control the skylight actuator to perform an opening operation. If the gesture characteristic of the current video frame is all five fingers raised downwards, then control the skylight actuator to perform a closing operation.

[0122] Optionally, if the target hand attribute is determined to be the right hand, the vehicle control system determines whether the gesture shape feature is a five-finger raised gesture, that is, five fingers raised upward or five fingers raised downward.

[0123] Furthermore, the vehicle control system makes a specific determination of the gesture shape characteristics. Optionally, the determination logic of this embodiment of the invention is as follows: if the gesture shape characteristics of the current video frame are all five fingers raised upward, that is, all five fingers of the right hand are raised and the palm is flat and pointing upward, the gesture corresponds to the intention to open the sunroof, and the vehicle control system immediately sends an opening control signal to the sunroof actuator; the sunroof actuator is the core actuator used to control the opening or closing of the vehicle sunroof, and can receive the control signal of the vehicle control system and convert it into mechanical action to realize the smooth opening or closing of the sunroof.

[0124] If the gesture in the current video frame is characterized by all five fingers raised downwards, that is, all five fingers of the right hand are raised and the palm is flat and pointing downwards, this gesture corresponds to the intention to close the sunroof, and the vehicle control system immediately sends a closing control signal to the sunroof actuator.

[0125] Furthermore, after receiving the corresponding control signal, the sunroof actuator performs the corresponding opening or closing operation and feeds back the operation execution status to the vehicle control system. After receiving the feedback information, the vehicle control system processes it according to the closed-loop logic of steps 401 and 402. If the execution is successful, the continuous frame count counter is cleared; if the execution fails, a prompt is issued and the system returns to step 10 for re-detection, ensuring the accuracy and reliability of the sunroof control.

[0126] This invention, by clearly distinguishing the left and right hand attributes of the target hand and accurately matching gesture morphology features with corresponding actuators, achieves a one-to-one correspondence between operation commands and vehicle execution actions. This ensures that the gesture operations of the driver can be accurately translated into corresponding actions of the vehicle's actuators, effectively avoiding misoperations caused by confusion between gestures and actuators, and reducing accidental triggering and execution. At the same time, it conforms to the operating habits of the driver, enabling convenient control of windows and sunroof through simple and intuitive gestures. This not only improves operational convenience but also further eliminates vehicle misoperations caused by gestures from non-target personnel, solving the problem of accidental triggering and execution in multi-person scenarios, and improving driving safety and system reliability.

[0127] The interactive training platform provided by this invention is described below. The interactive training platform described below can be referred to in correspondence with the gesture recognition method based on ADAS intelligent simulation training cockpit described above.

[0128] Optionally, refer to Figure 8 , Figure 8This is a schematic diagram of the interactive training platform provided by the present invention. The interactive training platform includes: The target hand recognition module 810 is used to perform hand detection based on the continuous video frames of the target user, obtain the hand structure information of the hand to be detected in each video frame, and filter out the hand located in the preset driver's seat operation area based on the hand structure information and palm orientation in each video frame to obtain the target hand. The gesture feature recognition module 820 is used to determine the gesture morphology features of the current video frame based on the first coordinate difference in the vertical direction between the first spatial key point coordinates of the fingertip of each finger in the target hand and the second spatial key point coordinates of the corresponding metacarpophalangeal joint. The morphological feature comparison module 830 is used to compare the morphological features of the current video frame with the morphological features of the previous video frame if a morphological template matching the morphological features of the gesture exists, and to obtain the number of consecutively appearing frames. If the number of consecutive frames reaches a preset continuous detection threshold, the vehicle control module 840 controls the vehicle actuator to perform the corresponding operation based on the operation command corresponding to the gesture shape feature of the current video frame.

[0129] This invention performs hand detection on consecutive video frames of a target user, obtaining hand structure information for the hand to be detected in each video frame. Combining this hand structure information with palm orientation, it filters out target hands located within a preset driver's seat operating area. This eliminates interference from hands belonging to non-drivers, addressing the difficulty of distinguishing gestures between the target operator and other individuals. Based on the first coordinate difference in the vertical direction between the first spatial keypoint coordinates of the fingertips and the second spatial keypoint coordinates of the corresponding metacarpophalangeal joints in the target hand, the gesture morphology features of the current video frame are obtained, achieving accurate depiction of the target hand gestures. The gesture morphology features of the current video frame are matched with a gesture morphology template and compared with the gesture morphology features of the previous video frame to obtain the number of consecutive occurrences, verifying the continuity of the target hand gestures and overcoming the shortcomings of relying solely on single-frame image features and lacking continuous verification of operational intent. The number of consecutive occurrences is used to determine if a preset continuous detection threshold has been reached. If so, the corresponding operation command is used to control the vehicle's actuator, resulting in accurate execution of the target operation command. This effectively avoids non-driver similar gestures and accidental misgestions being judged as valid commands, reducing false triggering and execution. By accurately executing target operation commands, the system eliminates the possibility of vehicle misoperation caused by gestures from non-target personnel, thereby solving the problems of accidental triggering and execution in multi-person scenarios and improving driving safety and system reliability.

[0130] Please see Figure 9 , Figure 9 An embodiment diagram of an electronic device provided in accordance with the present invention. For example... Figure 9 As shown, an embodiment of the present invention provides an electronic device 900, including a memory 910, a processor 920, and a computer program 911 stored in the memory 910 and executable on the processor 920. When the processor 920 executes the computer program 911, it implements the processes of steps 10 to 40.

[0131] Figure 10 An embodiment diagram of a computer-readable storage medium provided in accordance with an embodiment of the present invention is shown. Figure 10 As shown, this embodiment provides a computer-readable storage medium 1000, on which a computer program 911 is stored. When the computer program 911 is executed by a processor, it implements the processes of steps 10 to 40.

[0132] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the gesture recognition method based on the ADAS intelligent simulation training cockpit provided by the above methods, which includes the process of steps 10 to 40.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A gesture recognition method based on an ADAS intelligent simulation training cockpit, characterized in that, include: Hand detection is performed based on the continuous video frames of the target user, and the hand structure information of the hand to be detected in each video frame is obtained. Based on the hand structure information and palm orientation in each video frame, the hand located in the preset driver's seat operation area is selected to obtain the target hand. Based on the first coordinate difference in the vertical direction between the first spatial key point coordinates of the fingertip of each finger in the target hand and the second spatial key point coordinates of the corresponding metacarpophalangeal joint, the gesture morphology features of the current video frame are determined. If a gesture pattern template that matches the gesture pattern features exists, then the gesture pattern features of the current video frame are compared with the gesture pattern features of the previous video frame to obtain the number of consecutively appearing frames. If the number of consecutive frames reaches the preset continuous detection threshold, the vehicle actuator will be controlled to perform the corresponding operation based on the operation command corresponding to the gesture shape feature of the current video frame.

2. The gesture recognition method based on an ADAS intelligent simulation training cockpit according to claim 1, characterized in that, The steps for determining the gesture morphological features of the current video frame include: Based on the vertical coordinate difference between the first spatial key point coordinates of each fingertip in the target hand and the second spatial key point coordinates of its corresponding metacarpophalangeal joint, a finger vertical difference vector is constructed; the finger vertical difference vector includes the vertical coordinate differences of the thumb, index finger, middle finger, ring finger, and little finger. Based on the differences between the vertical coordinates of the index finger and thumb, the middle finger and index finger, the ring finger and middle finger, and the little finger and ring finger, a finger extension gradient sequence is constructed, and the number of sign changes of adjacent gradient values ​​from positive to negative or from negative to positive is analyzed based on the finger extension gradient sequence. If the number of symbol changes exceeds a preset threshold, the gesture morphology feature of the current video frame is determined to be an invalid gesture morphology feature. If the number of symbol changes is less than a preset threshold, the gesture shape features of the current video frame are determined based on the vertical coordinate difference of each finger in the vertical difference vector.

3. The gesture recognition method based on an ADAS intelligent simulation training cockpit according to claim 2, characterized in that, The method of determining the gesture shape features of the current video frame based on the vertical coordinate difference of each finger in the vertical difference vector includes: If the vertical coordinate difference of each finger is determined to be greater than or equal to a preset lift-up threshold based on the vertical difference vector of the fingers, then it is determined whether the absolute value of any gradient value in the finger extension gradient sequence is less than a preset smoothing tolerance threshold; if so, a first gesture candidate with the palm flat and raised is generated, and the pointing direction of the first finger is determined based on the sign of the median of the vertical coordinate difference of each finger; based on the first gesture candidate and the pointing direction of the first finger, the gesture morphology features of the current video frame are determined. If the vertical coordinate difference of the thumb is greater than or equal to a preset lift threshold, and the vertical coordinate differences of all non-thumb fingers are less than a preset close threshold, then finger trend analysis is performed based on the vertical coordinate difference of each non-thumb finger to obtain a finger trend judgment result. If the finger trend judgment result indicates a non-increasing monotonic trend or a non-decreasing monotonic trend, then a second gesture candidate for lifting the isolated main finger is generated, and the sign of the vertical coordinate difference of the thumb is used as the second pointing direction. Based on the second gesture candidate and the second finger pointing direction, the gesture shape feature of the current video frame is determined. The finger trend judgment result represents whether it exhibits a non-increasing monotonic trend or a non-decreasing monotonic trend. If neither the first gesture candidate nor the second gesture candidate is generated, then the gesture morphology feature of the current video frame is determined to be an invalid gesture morphology feature.

4. The gesture recognition method based on the ADAS intelligent simulation training cockpit according to claim 3, characterized in that, The steps involved in determining gesture morphological features based on gesture candidates and finger pointing direction include: For the first gesture candidate, if the direction in which the first finger points is positive, then the gesture shape feature of the current video frame is determined to be all five fingers raised upwards; if the direction in which the first finger points is negative, then the gesture shape feature of the current video frame is determined to be all five fingers raised downwards. For the second gesture candidate, if the direction in which the second finger points is positive, then the gesture shape feature of the current video frame is determined to be a single thumb raised upwards; if the direction in which the second finger points is negative, then the gesture shape feature of the current video frame is determined to be a single thumb raised downwards.

5. The gesture recognition method based on an ADAS intelligent simulation training cockpit according to claim 4, characterized in that, Controlling the vehicle's actuators to perform corresponding operations, including: If the target hand is the left hand, and the gesture feature of the current video frame is that the thumb is raised upwards, then the driver's side window actuator is controlled to perform an upward operation; if the gesture feature of the current video frame is that the thumb is raised downwards, then the driver's side window actuator is controlled to perform a downward operation. If the target hand is the right hand, and the gesture feature of the current video frame is a single thumb raised upwards, then the passenger-side window actuator is controlled to perform an upward operation; if the gesture feature of the current video frame is a single thumb raised downwards, then the passenger-side window actuator is controlled to perform a downward operation. If the target hand is the right hand, and the gesture shape of the current video frame is all five fingers raised upwards, then the sunroof actuator is controlled to perform an opening operation; if the gesture shape of the current video frame is all five fingers raised downwards, then the sunroof actuator is controlled to perform a closing operation.

6. The gesture recognition method based on an ADAS intelligent simulation training cockpit according to claim 1, characterized in that, The steps for determining the target hand include: Based on the position coordinates of the wrist key points in the current video frame in the horizontal direction of the hand structure information, the horizontal coordinate of the wrist of the hand to be detected is determined. Based on the positional relationship between the horizontal coordinate of the wrist and the left and right boundary coordinates of the preset driver's seat operation area in the horizontal direction of the image, the spatial position determination result of whether the hand to be detected is within the horizontal range of the preset driver's seat operation area is determined. Based on the spatial arrangement features of key points of each finger in the hand structure information of the current video frame, the second coordinate difference between the metacarpophalangeal joint of the index finger and the metacarpophalangeal joint of the little finger in the horizontal direction of the image is calculated to obtain the lateral span index between the index finger and the little finger of the hand to be detected. Based on the lateral span index and the preset frontal orientation threshold, the palm orientation state is determined to obtain the palm orientation determination result of the hand to be detected. The target hand is determined based on the spatial location determination result and the palm orientation determination result.

7. The gesture recognition method based on an ADAS intelligent simulation training cockpit according to claim 6, characterized in that, Determining the target hand based on the spatial location determination result and the palm orientation determination result includes: If the palm orientation determination result indicates that the palm face is facing upwards, and the spatial position determination result indicates that it is within the horizontal range, then the hand to be detected is determined to be a valid candidate hand. Based on the relative positional relationship between the horizontal coordinates of the wrist key points of the valid candidate hand and the horizontal centerline of the preset driver's seat operating area, the horizontal offset of the valid candidate hand is determined; the horizontal centerline is set based on the image coordinate system corresponding to the structural features of the vehicle cabin, and the horizontal centerline is located at the projection position of the driver's seat centerline in the image. If the horizontal offset is less than or equal to zero, the valid candidate hand is determined to be located to the left or above the center line. It is then verified whether the horizontal coordinate value of the index finger metacarpophalangeal joint of the valid candidate hand is greater than the horizontal coordinate value of the thumb metacarpophalangeal joint. If the coordinates are met, the target hand is confirmed to be the left hand. If the horizontal offset is greater than zero, the valid candidate hand is determined to be located to the right of the center line. It is then verified whether the horizontal coordinate value of the thumb metacarpophalangeal joint of the valid candidate hand is greater than the horizontal coordinate value of the index finger metacarpophalangeal joint. If the coordinates are met, the target hand is confirmed to be the right hand.

8. An interactive training platform, characterized in that, The method for implementing the gesture recognition method based on the ADAS intelligent simulation training cockpit as described in any one of claims 1 to 7; the interactive training platform includes: The target hand recognition module is used to detect hands based on continuous video frames of the target user, obtain the hand structure information of the hand to be detected in each video frame, and filter out the hands located in the preset driver's seat operation area based on the hand structure information and palm orientation in each video frame to obtain the target hand. The gesture feature recognition module is used to determine the gesture morphology features of the current video frame based on the first coordinate difference in the vertical direction between the first spatial key point coordinates of the fingertip of each finger in the target hand and the second spatial key point coordinates of the corresponding metacarpophalangeal joint. The morphological feature comparison module is used to compare the morphological features of the current video frame with the morphological features of the previous video frame if a morphological template matching the morphological features of the gesture exists, and to obtain the number of consecutively occurring frames. The vehicle control module is used to control the vehicle actuator to perform corresponding operations based on the operation instructions corresponding to the gesture shape features of the current video frame if the number of consecutive frames reaches a preset continuous detection threshold.

9. An electronic device, comprising: Memory, used to store computer software programs; A processor for reading and executing the computer software program, characterized in that, when the processor executes the computer software program, it implements the gesture recognition method based on the ADAS intelligent simulation training cockpit as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, wherein a computer software program is stored therein, characterized in that, When the computer software program is executed by the processor, it implements the gesture recognition method based on the ADAS intelligent simulation training cockpit as described in any one of claims 1 to 7.

Images