A simulated driving posture evaluation method and a simulated driving device

Abstract

The application provides a simulation driving posture evaluation method and a simulation driving device, and belongs to the technical field of simulation driving training. The method adopts an image acquisition and recognition method, obtains a simulation driving training area image of a driving area where a trainee is located, recognizes a driver and a gesture area from the simulation driving training area image, the driver is the trainee who is performing simulation driving training, extracts human feature points of the driver, judges a hand performing a gesture according to the human feature points and the gesture area, judges a gesture direction, and evaluates a simulation driving posture according to the judged gesture direction. According to the application, the simulation driving posture of the trainee can be automatically evaluated in the process of simulation driving training of the trainee, the simulation driving postures of multiple trainees can be simultaneously evaluated by a teacher, the simulation driving posture of the trainee can be evaluated by the trainee when the trainee is training alone, and the efficiency of simulation driving training is improved.

Description

Technical Field

[0001] This invention provides a method for evaluating simulated driving posture and a simulated driving device, belonging to the field of simulated driving training technology. Background Technology

[0002] During simulated driving training, locomotive and EMU crew members or trainees must strictly adhere to the standard requirements of a single crew operation, performing visual and hand gesture actions to help them develop correct operating habits and demonstrate good driving posture in actual driving. Currently, the simulated driving devices used by various railway bureaus and educational institutions only evaluate trainees' actions based on automatically acquired operational information. Furthermore, during visual and hand gesture training, instructors still rely on individually checking and evaluating each trainee's hand gestures, lacking automated evaluation methods. Instructors cannot simultaneously guide multiple trainees, reducing teaching efficiency. Additionally, during self-training, trainees cannot determine whether their actions are correct, affecting training effectiveness. Summary of the Invention

[0003] The purpose of this invention is to provide a method and device for evaluating simulated driving posture, which solves the problem that it is impossible to accurately evaluate the hand gestures of trainees during simulated driving training.

[0004] To achieve the above objectives, the present invention provides a method for simulating driving posture evaluation, comprising the following steps:

[0005] S1. Acquire images of the simulated driving training area through an image acquisition device, and identify the driver and the corresponding driver's gesture area within the simulated driving training area image;

[0006] S2. Extract the driver's human body feature points. Based on the positional relationship between the human body feature points and the gesture area, determine whether the driver's hand performing the current gesture is the left or right hand.

[0007] S3. Determine the direction of the driver's gesture when performing the current gesture based on human body feature points or gesture area feature points, and evaluate the simulated driving posture based on the gesture direction.

[0008] The simulated driving posture evaluation method of this invention employs image acquisition and recognition. It acquires an image of the simulated driving training area where the trainee is located, identifying the driver and their gesture area. The driver is the trainee undergoing simulated driving training. Human feature points of the driver are extracted. Based on these feature points and the gesture area, the hand performing the gesture is first identified to determine whether it is the left or right hand. Then, the direction of the gesture is determined, and the simulated driving posture is evaluated based on this direction. This invention enables automated evaluation of trainees' simulated driving posture during training, facilitating simultaneous evaluation of multiple trainees by instructors and allowing trainees to evaluate their own posture during individual training, thus improving the efficiency of simulated driving training.

[0009] Furthermore, in the above method, the human body feature points include left wrist feature points, right wrist feature points, left elbow feature points, and right elbow feature points;

[0010] In step S2, the driver's hand performing the current gesture is determined as follows: if the left wrist feature point is in the gesture area and the right wrist feature point is not in the gesture area, then the driver's hand performing the current gesture is the left hand; if the right wrist feature point is in the gesture area and the left wrist feature point is not in the gesture area, then the driver's hand performing the current gesture is the right hand; if both the left and right wrist feature points are in the gesture area, the overlapping area of ​​the left arm direction with the gesture area and the overlapping area of ​​the right arm direction with the gesture area are calculated. If the overlapping area of ​​the left arm is greater than the overlapping area of ​​the right arm, then the driver's hand performing the current gesture is the left hand; otherwise, the driver's hand performing the current gesture is the right hand.

[0011] The direction of the left arm is obtained by using the feature points of the left wrist and the left elbow, and the direction of the right arm is obtained by using the feature points of the right wrist and the right elbow.

[0012] The hand performing the current gesture is determined based on the wrist feature points in the human body feature points. If only the left wrist feature point is present in the gesture area, it is the left hand; if only the right wrist feature point is present, it is the right hand; if both left and right wrist feature points are present, the determination is based on the area of ​​the left and right arms within the gesture area. Using wrist feature points combined with elbow feature points allows for accurate identification of the hand performing the gesture, with high accuracy and ease of implementation. Furthermore, in the above method, the gesture direction includes upward or downward, and the gesture area feature points include the center point of the gesture area. In step S3, the gesture direction when the driver performs the current gesture is determined as follows: based on the pre-acquired first and second design curves, a first distance is calculated between the wrist feature point / gesture area center point related to the current gesture and the first design curve; a second distance is also calculated between the wrist feature point related to the current gesture and the second design curve. If the first distance is greater than the second distance, the current gesture direction is upward; if the first distance is less than the second distance, the current gesture direction is downward.

[0013] The first and second design curves are obtained by fitting historical experimental data: historical experimental data is obtained, which includes wrist feature points or gesture area center points with the gesture direction determined in the simulated driving training area image. Two curves are fitted according to the distribution of wrist feature points or gesture area center points. One curve is used to show the distribution of each wrist feature point / gesture area center point with the gesture direction upward, and is called the first design curve. The other curve is used to show the distribution of each wrist feature point / gesture area center point with the gesture direction downward, and is called the second design curve.

[0014] A first design curve and a second design curve are fitted using historical experimental data. The first design curve represents the distribution of wrist feature points or the center point of the gesture area when the gesture direction is upward, and the second design curve represents the distribution of wrist feature points or the center point of the gesture area when the gesture direction is downward. The positions of the wrist feature points or the center point of the gesture area are obtained, and the first distance between them and the first design curve and the second distance between them and the second design curve are calculated. The magnitudes of the first and second distances are compared. If the first distance is greater than the second distance, the current gesture direction is upward; if the first distance is less than the second distance, the current gesture direction is downward. This method only requires calculating the distances between the wrist feature points or the center point of the gesture area and the two pre-acquired curves, and then making a judgment by comparing their magnitudes. It is easy to operate and has a small computational load.

[0015] Further, in the above method, the gesture direction includes upward or downward, and the gesture area feature point includes the gesture area center point; in step S3, the gesture direction when the driver performs the current gesture is determined by the following method: when the horizontal angle between the image acquisition device and the driver's position is less than a set angle, according to the pre-acquired first design curve and second design curve, the first distance between the wrist feature point / gesture area center point related to the current gesture and the first design curve is calculated, and the second distance between the wrist feature point / gesture area center point related to the current gesture and the second design curve is also calculated; the difference between the first distance and the second distance is calculated, and if the absolute value of the difference is greater than a set value, and the first distance is greater than the second distance, then the direction of the current gesture is upward; when the absolute value of the difference is less than a set value, if the wrist feature point or gesture area center point related to the current gesture is in a pre-set middle area, then judgment condition 1 is used to determine the gesture direction when the driver performs the current gesture; if the wrist feature point or gesture area center point related to the current gesture is in a pre-set left area or a pre-set right area, then judgment condition 2, judgment condition 3 or judgment condition 4 is used to determine the gesture direction when the driver performs the current gesture;

[0016] The first and second design curves are obtained by fitting historical experimental data: historical experimental data is obtained, which includes wrist feature points or gesture area center points with the gesture direction determined in the simulated driving training area image. Two curves are fitted according to the distribution of wrist feature points or gesture area center points. One curve is used to show the distribution of each wrist feature point / gesture area center point with the gesture direction upward, and is called the first design curve. The other curve is used to show the distribution of each wrist feature point / gesture area center point with the gesture direction downward, and is called the second design curve.

[0017] The judgment conditions 1, 2, 3, and 4, and their corresponding judgment results are as follows:

[0018] Judgment Condition 1: Calculate the horizontal and vertical distances between the wrist feature point and the corresponding shoulder feature point. When both the horizontal and vertical distances are greater than zero, if the horizontal distance is greater than a first horizontal threshold and the vertical distance is greater than a first vertical threshold, then the current gesture direction is upward. When both the horizontal and vertical distances are less than zero, if the absolute value of the horizontal distance is greater than a first horizontal threshold and the absolute value of the vertical distance is greater than a first vertical threshold, then the current gesture direction is downward. Both the first horizontal threshold and the first vertical threshold are greater than zero.

[0019] Judgment condition 2: The angle between the forearm and the horizontal direction of the simulated driving training area image is taken as the forearm angle. If the forearm angle is greater than the set forearm angle threshold, the current gesture direction is upward; otherwise, the current gesture direction is downward. The line connecting the wrist feature point related to the current gesture and the elbow feature point corresponding to the wrist feature point in the human body feature points is taken as the forearm.

[0020] Judgment condition 3: The angle between the long arm and the horizontal direction of the simulated driving training area image is taken as the long arm angle. If the long arm angle is greater than the set long arm angle threshold, the current gesture direction is upward; otherwise, the current gesture direction is downward. The line connecting the wrist feature point related to the current gesture and the shoulder feature point corresponding to the wrist feature point in the human body feature points is taken as the long arm.

[0021] Judgment condition 4: If the angle between the line connecting the wrist feature point and the head feature point related to the current gesture in the human body feature points and the horizontal direction of the simulated driving training area image is greater than the first set threshold, then the current gesture is upward; otherwise, the current gesture is downward.

[0022] Considering the influence of the angle between the image acquisition device and the driver's seat on gesture direction recognition, when the horizontal angle between the image acquisition device and the driver's seat is less than a set angle, the position of the wrist feature point or the center point of the gesture area is acquired, and the first distance between it and the first design curve and the second distance between it and the second design curve are calculated. The direction is determined based on the magnitude of the first and second distances. If the absolute value of the difference between the first and second distances is greater than a set value, and the first distance is greater than the second distance, the current gesture direction is upward. If the absolute value of the difference between the first and second distances is less than a set value, the position of the wrist feature point or the center point of the gesture area related to the current gesture is determined. If the wrist feature point or the center point of the gesture area related to the current gesture is located in a pre-set middle region, the gesture direction is determined based on the distance between the wrist feature point and the shoulder feature point. If the wrist feature point or the center point of the gesture area related to the current gesture is located in a pre-set left region or a pre-set right region, the gesture direction is determined based on the forearm feature formed by the combination of the wrist feature point and the elbow feature point, the long arm feature formed by the combination of the wrist feature point and the shoulder feature point, or the feature of the line connecting the wrist feature point and the neck feature point. It is simple to calculate, easy to implement, and accurate and reliable.

[0023] Furthermore, in the above method, the gesture direction includes upward and downward. In step S3, one of the following judgment conditions is used to determine the gesture direction when the driver performs the current gesture. Each judgment condition and the corresponding judgment result are as follows:

[0024] Judgment condition 1: The angle between the forearm and the horizontal direction of the simulated driving training area image is taken as the forearm angle. If the forearm angle is greater than the set forearm angle threshold, the current gesture direction is upward; otherwise, the current gesture direction is downward. The line connecting the wrist feature point related to the current gesture and the elbow feature point corresponding to the wrist feature point in the human body feature points is taken as the forearm.

[0025] Judgment condition 2: The angle between the long arm and the horizontal direction of the simulated driving training area image is taken as the long arm angle. If the long arm angle is greater than the set long arm angle threshold, the current gesture direction is upward; otherwise, the current gesture direction is downward. The line connecting the wrist feature point related to the current gesture and the shoulder feature point corresponding to the wrist feature point in the human body feature points is taken as the long arm.

[0026] Judgment condition 3: If the angle between the line connecting the wrist feature point and the head feature point related to the current gesture in the human body feature points and the horizontal direction of the simulated driving training area image is greater than the first set threshold, then the current gesture is upward; otherwise, the current gesture is downward.

[0027] Without considering the influence of the angle between the image acquisition device and the driver's seat on the recognition of gesture direction, the gesture direction can be determined by using one of the above judgment conditions one, two, and three. The judgment methods are diverse, can be freely selected, are easy to implement, and are simple and reliable.

[0028] Furthermore, in the above method, the human body feature points include wrist feature points, and the gesture direction includes the left direction, the center direction, or the right direction;

[0029] In step S3, the direction of the driver's gesture when performing the current gesture is determined as follows: if the wrist feature point of the hand performing the current gesture is in a pre-defined left region, the direction of the current gesture is left; if the wrist feature point of the hand performing the current gesture is in a pre-defined middle region, the direction of the current gesture is middle; if the wrist feature point of the hand performing the current gesture is in a pre-defined right region, the direction of the current gesture is right.

[0030] By using wrist feature points and the positional relationship of three pre-defined regions (left, center, and right), the direction of the current gesture can be determined as left, center, or right, requiring minimal computation.

[0031] Furthermore, in the above method, in step S1, the driver and the corresponding driver's gesture area within the simulated driving training area image are identified through the following steps:

[0032] S1.1 Input the simulated driving training area image into the pre-established target detection model, and output the personnel target and gesture area in the image;

[0033] S1.2. Based on the pre-set driver area, determine the driver and the corresponding driver's gesture area in the personnel target;

[0034] The training data for the target detection model includes images of simulated driving training areas labeled with personnel targets and gesture regions.

[0035] The area where trainees are located during simulated driving training is designated as the simulated driving training area. Images of this area are acquired and input into a pre-established target detection model to identify personnel targets and their gesture regions. Then, based on a pre-defined driver region, the driver is identified from the personnel targets, and gesture regions related to the driver are also determined from the gesture regions. Using a model for personnel recognition is easy to implement and allows for easy adjustment of the model.

[0036] Furthermore, in the above method, if there are multiple personnel targets and gesture regions in the simulated driving training area image, the number of human feature points of each personnel target in the driver region is obtained, the personnel target with the largest number of human feature points is taken as the driver, and the gesture region related to the personnel target with the largest number of human feature points is taken as the gesture region of the corresponding driver.

[0037] Since the driver area is mainly the area where the driver is driving, there are obviously fewer non-driver human feature points in this area. Therefore, the human target with the largest number of human feature points is regarded as the driver, and the gesture area related to the human target with the largest number of human feature points is regarded as the gesture area of ​​the corresponding driver. The judgment method is simple and reliable.

[0038] The present invention also provides a driving simulation device, including an image acquisition unit for acquiring images of a driving simulation training area and a controller communicatively connected to the image acquisition unit. The controller includes a processor and a memory, and the processor executes instructions in the memory to implement the above-described driving simulation posture evaluation method.

[0039] Furthermore, the aforementioned device also includes a display device connected to the controller for displaying the results of the driver's simulated driving posture evaluation.

[0040] Using a display device to show the results of the simulated driving posture evaluation makes it easy for instructors and trainees to view, which can effectively improve training efficiency. Attached Figure Description

[0041] Figure 1 This is a flowchart of the simulated driving posture evaluation method in an embodiment of the present invention;

[0042] Figure 2 This is a flowchart illustrating the process of determining the left and right hands in an embodiment of the method of the present invention.

[0043] Figure 3 This is a schematic diagram of the left arm movement in an embodiment of the method of the present invention;

[0044] Figure 4 This is a schematic diagram of the image coordinate system in an embodiment of the method of the present invention;

[0045] Figure 5 This is a schematic diagram of human feature points in an embodiment of the method of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0047] Method Implementation Examples:

[0048] The present invention discloses a simulated driving posture evaluation method, which uses a camera to capture images of the simulated driving device to obtain an image of the simulated driving training area as the test image, i.e., the image of the area where the trainee is conducting simulated driving. Then, the test image is processed and analyzed to determine whether the driver (i.e., the trainee) is on duty. If the driver is on duty, the left and right hands of the driver in the test image are identified to determine the hand making a gesture. Then, based on the driver's human body feature points, it is determined whether the gesture being performed by the hand meets the operational requirements.

[0049] In this embodiment, the camera can be installed in the upper left or upper right position inside the cabin of the driving simulator, or it can be installed on the control panel of the driving simulator. The camera is fixed to the driving simulator using a camera bracket, ensuring that the camera's shooting area can completely cover the driver's operating area when the driver is training in the driving simulator.

[0050] Taking a camera installed in the upper left corner of the cabin as an example, the simulated driving posture during a simulated driving operation is recognized, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0051] 1) Acquire images of the driver's seat and its surrounding environment as the simulated driving training area image, i.e., the driving area image. Using a pre-established target detection model, extract all personnel targets and their gesture regions from the driving area image. For example... Figure 3 As shown, the gesture area includes the palm and part of the arm.

[0052] In this embodiment, the target detection model adopts the YOLO target detection model, and the driving area image with labeled personnel targets and gesture regions is used to divide the training set and test set. The target detection model is trained and tested using the training set and test set. As another implementation, two target detection models can also be used, one for recognizing personnel targets in the driving area image and the other for recognizing gesture regions in the driving area image.

[0053] 2) Based on prior knowledge or region fitting, the area where the driver is located during the locomotive simulation is designated as the driver area. If the personnel target extracted from the driving area image is outside the driver area, the personnel target outside the driver area and the gesture area of ​​the personnel target outside the driver area are deleted. If the personnel target extracted from the driving area image is inside the driver area, the personnel target inside the driver area and the gesture area related to the personnel target inside the driver area are retained.

[0054] 3) It can also determine the driver's post status based on whether there is a person target in the driver's area as determined in step 2). If there is no person target in the driver's area, the driver is considered to be off-duty and the process returns to step 1) for re-detection; if there is a person target in the driver's area, the driver is considered to be on-duty and step 4) is executed.

[0055] 4) Human pose recognition algorithms are used to detect human feature points of individuals within the driver's area. The number of human feature points of each individual within the driver's area is calculated, and the individual with the highest number of human feature points is identified as the driver. If there is only one individual, that individual is identified as the driver. By comparing the number of human feature points of each individual, only the driver's feature point information can be retained, thus eliminating the influence of gestures from other unrelated individuals on the driver's gestures during gesture recognition. In this embodiment, the OpenPose algorithm is used for human pose recognition.

[0056] 5) Through such Figure 2The illustrated left / right hand determination process identifies the hand a driver uses to make gestures during simulated driving. It extracts left and right wrist feature points from the driver's personnel feature point information to determine whether the hand making the gesture within the gesture area is the left or right hand. If the left wrist feature point is within the gesture area but the right wrist feature point is not, the hand making the gesture is the left hand; if the right wrist feature point is within the gesture area but the left wrist feature point is not, the hand making the gesture is the right hand. If both the left and right wrist feature points are within the gesture area, the overlapping area S1 of the left arm (the line connecting the left wrist and left elbow feature points) and the gesture area is calculated, as well as the overlapping area S2 of the right arm (the line connecting the right wrist and right arm feature points) and the gesture area are also calculated. If S1 is greater than S2, the hand making the gesture is the left hand; otherwise, it is the right hand.

[0057] For example, such as Figure 3 As shown, if the feature point of the left wrist is within the gesture area and the feature point of the right wrist is not within the gesture area, then the hand currently making the gesture is the left hand.

[0058] 6) Based on prior knowledge, the gesture area is divided into three defined regions: left, middle, and right (represented by left_rect, middle_rect, and right_rect, respectively), to determine whether the hand making the gesture is facing left, right, or center. If the wrist feature point of the hand making the gesture is located in the defined left region, the hand is facing left, and the gesture direction is left; if the wrist feature point is located in the defined middle region, the hand is facing center, and the gesture direction is center; if the wrist feature point is located in the defined right region, the hand is facing right, and the gesture direction is right.

[0059] 7) Place the driver area image in Figure 4 In the image coordinate system shown, the angle between the arm and the coordinate axes is used to determine whether the hand making the gesture is facing up or down. The x-axis of the image coordinate system points to the right as the horizontal direction, and the y-axis points downward as the vertical direction.

[0060] like Figure 5 The diagram shows human body feature points. In this embodiment, feature points of the left arm, right arm, and head are mainly used to determine the driver's gestures. The feature points of the left arm include the left wrist, left elbow, and left shoulder; the feature points of the right arm include the right wrist, right elbow, and right shoulder; and the feature points of the head include the left and right eye feature points, left and right ear feature points, nose feature points, and neck feature points.

[0061] When determining the direction of a hand making a gesture, one method can be used. Method 1: Calculate the direction of the forearm relative to the direction of the hand. The included angle A of the axis is compared with a set forearm judgment threshold. In this embodiment, the line connecting the wrist feature point and the elbow feature point in the image coordinate system is used to represent the forearm. If the included angle A is greater than the forearm judgment threshold, the hand making the hand gesture is facing upwards, that is, the gesture direction is upwards; otherwise, the direction is downwards.

[0062] The forearm detection threshold is obtained using the following method: A large number of driver area images are acquired and placed in a... Figure 4 In the coordinate system shown, collect the angles A1, A2, ..., An between the forearm and the x-axis; perform histogram statistics on A1, A2, ..., An in the interval of 2°-5°, with the horizontal axis of the histogram representing the angle and the vertical axis representing the frequency of each angle; calculate the mean of the frequency of each angle in each angle interval, and sort the calculated mean values ​​according to the Gaussian distribution; take the mean of the middle three angles or the mean of two angles as the forearm judgment threshold.

[0063] As another implementation method, when determining the direction of the hand making the gesture, method two can also be used. Method two: Calculate the ratio of the long arm to the long arm. The included angle B of the axis is compared with a set long arm judgment threshold. In this embodiment, the line connecting the wrist feature point and the shoulder feature point in the image coordinate system is used to represent the long arm. If the included angle B is greater than the long arm judgment threshold, the hand making the gesture is facing upwards, i.e., the gesture direction is upwards; otherwise, the direction is downwards. When the gesture area is in the intersection area formed by the middle area and the left area, or the intersection area formed by the middle area and the right area, this implementation method can improve the accuracy of simulated driving posture recognition. The method for obtaining the long arm judgment threshold is similar to that forearm judgment threshold, specifically: acquiring a large number of driver area images and placing them in a coordinate system such as... Figure 4 In the coordinate system shown, collect the angles B1, B2, ..., Bn between the forearm and the x-axis; perform histogram statistics on B1, B2, ..., Bn in the interval of 2°-5°; calculate the mean frequency of each angle in each angle interval, and sort the calculated mean values ​​in the order of Gaussian distribution; take the mean of the middle three angles or the mean of two angles as the long arm judgment threshold.

[0064] As another implementation method, the direction of the hand making the gesture can also be determined using Method 3. Method 3: Calculate the line connecting the wrist feature point and the head feature point. Angle between axes It is compared with the set first judgment threshold. If the included angle... If the value exceeds the first judgment threshold, the hand making the gesture is pointing upwards; otherwise, the hand is pointing downwards. The method for obtaining the first judgment threshold is similar to that forearm judgment threshold, specifically: acquiring a large number of driver area images and placing them on a... Figure 4 In the coordinate system shown, collect the angles C1, C2, ..., Cn between the forearm and the x-axis; perform histogram statistics on C1, C2, ..., Cn in the interval of 2°-5°; calculate the mean frequency of each angle in each angle interval, and sort the calculated mean values ​​in the order of Gaussian distribution; take the mean of the middle three angles or the mean of two angles as the first judgment threshold.

[0065] In this embodiment, since the left and right eye feature points, left and right ear feature points, and nose feature points in the head feature points are easily affected by head turning, which can cause a large judgment error, the neck feature points are used as the head feature points.

[0066] As another implementation method, the direction of the hand making a gesture can also be determined using Method Four. Method Four: Calculate the horizontal and vertical distances between the feature points of the left and right wrists and the feature points of the left and right shoulders, and determine the orientation based on these distances. Taking the left hand making a gesture as an example, the specific determination process is as follows: In the image coordinate system, obtain the coordinates (x1, y1) of the left wrist feature point and the coordinates (x2, y2) of the left shoulder feature point. Calculate the difference d1 between x1 and x2 as the horizontal distance between the left wrist feature point and the left shoulder feature point (i.e., the distance between the left wrist feature point and the left shoulder feature point in the image coordinate system). The distance between the left wrist feature point and the left shoulder feature point in the vertical direction (i.e., the axis direction of the image coordinate system) is calculated. If x1 is greater than x2, the difference d1 is positive; if x1 is less than x2, the difference d1 is negative. If y1 is greater than y2, the difference d2 is positive; if y1 is less than 2, the difference d2 is negative. The difference d1 is compared with a set threshold D1 (D1 is positive), and the difference d2 is compared with a set threshold D2 (D2 is positive). When both d1 and d2 are positive, if d1 is greater than D1 and d2 is greater than D2, the left hand making the gesture is pointing upwards. When both d1 and d2 are negative, if the absolute value of d1 is greater than D1 and the absolute value of d2 is greater than D2, the left hand making the gesture is pointing downwards.

[0067] As another implementation method, the determination can also be made using Method 4. Method 4: Count a certain number of points, such as wrist feature points or the center point of the recognition box, and determine the up and down based on the horizontal and vertical coordinates of the center point of the recognition box of the wrist feature points or the gesture area.

[0068] Taking wrist feature points as an example, by fitting experimental data of wrist feature point coordinates, design curves S1 and S2 are obtained. Under the condition that the horizontal coordinates are the same, the vertical coordinates of each point on the design curve S1 are smaller than the vertical coordinates of each point on the design curve S2, and the wrist feature points are located in the region between the design curves S1 and S2.

[0069] In this embodiment, when fitting design curves S1 and S2, a large amount of historical experimental data can be obtained. The fitting is performed based on the distribution of wrist feature points and the center point of the gesture area in the historical experimental data, where the points in the historical experimental data are distributed on both sides of the fitted design curve S1. The same method is used to fit design curve S2.

[0070] In addition, when fitting the design curves S1 and S2, a large amount of historical experimental data can be obtained, and the fitting can be performed based on the distribution of each wrist feature point and the center point of the gesture area in the historical experimental data. The points in the historical experimental data are distributed in the middle area of ​​the fitted design curves S1 and S2.

[0071] The specific judgment method is as follows: calculate the distances s1 and s2 between the wrist feature point and the design curve S1 and design curve S2 respectively. If s1 is less than s2, the hand making the gesture is facing upward, that is, the gesture is facing upward; otherwise, the hand is facing downward.

[0072] Considering the impact of the camera's position on the recognition results, a fifth method can also be used for judgment. Method Five: If the horizontal angle between the camera and the driver's position is less than a set angle, such as 45°, then the following judgment is made based on s1 and s2 calculated in Method Four. Calculate the difference s3 between s1 and s2. If the absolute value of s3 is greater than or equal to 5, and s1 is less than s2, then the hand making the gesture is pointing upwards; otherwise, it is pointing downwards. If the absolute value of s3 is less than 5, it indicates that the wrist feature point is in the middle region between the designed curves S1 and S2. If the wrist feature point is in the set middle region, then Method Four is used for judgment. If the wrist feature point is in the set left or right region, then Method One, Method Two, or Method Three is used for judgment.

[0073] Combining the judgment results of steps 6) and 7), it can be determined that the direction of the hand gestures made by the trainee during simulated driving is upper left, lower left, upper center, lower center, upper right, or lower right. This makes it easier for the trainee or instructor to evaluate the hand gestures and form the final result of the simulated driving posture evaluation.

[0074] Device Example:

[0075] The present invention also provides a driving simulator, including a driving simulator cockpit and a control panel. The control panel is disposed inside the cockpit and is used for simulating driving operations. The driving simulator also includes an intelligent evaluation system for monitoring and evaluating the driver's simulated driving posture. The intelligent evaluation system includes an image acquisition device for monitoring. The image acquisition device uses a camera, which is installed in the upper left and upper right corners inside the cockpit, or fixedly installed on the control panel via a camera bracket. The camera's field of view covers the driver's operating area to prevent missed shots.

[0076] The intelligent evaluation system also includes a controller and a display device. The controller includes a processor, memory, and an internal bus, with the processor and memory interacting via the internal bus. The controller analyzes and processes images of the driving area captured by the camera to evaluate the student's simulated driving posture. The simulated driving posture evaluation method described in the method embodiment is used, and its steps have been clearly explained in the method embodiment and will not be repeated here. After evaluating the student's simulated driving posture, the controller sends the evaluation results to the display device for viewing by the instructor or student. The display device is an LCD screen.

Claims

1. A method for evaluating simulated driving posture, characterized in that, Includes the following steps: S1. Acquire images of the simulated driving training area through an image acquisition device, and identify the driver and the corresponding driver's gesture area within the simulated driving training area image; S2. Extract the driver's human body feature points. Based on the positional relationship between the human body feature points and the gesture area, determine whether the driver's hand performing the current gesture is the left or right hand. S3. Determine the direction of the driver's gesture when performing the current gesture based on human feature points or gesture area feature points, and evaluate the simulated driving posture based on the gesture direction; The gesture direction includes upward or downward, and the gesture area feature point includes the center point of the gesture area; in step S3, the gesture direction when the driver performs the current gesture is determined by the following method: when the horizontal angle between the image acquisition device and the driver's position is less than a set angle, according to the pre-acquired first design curve and second design curve, the first distance between the wrist feature point / gesture area center point related to the current gesture and the first design curve is calculated, and the second distance between the wrist feature point / gesture area center point related to the current gesture and the second design curve is also calculated; the difference between the first distance and the second distance is calculated, and if the absolute value of the difference is greater than a set value, and the first distance is greater than the second distance, then the direction of the current gesture is upward; when the absolute value of the difference is less than a set value, if the wrist feature point or gesture area center point related to the current gesture is in a pre-set middle area, then judgment condition 1 is used to determine the gesture direction when the driver performs the current gesture; if the wrist feature point or gesture area center point related to the current gesture is in a pre-set left area or a pre-set right area, then judgment condition 2, judgment condition 3 or judgment condition 4 is used to determine the gesture direction when the driver performs the current gesture; The first and second design curves are obtained by fitting historical experimental data: historical experimental data is obtained, which includes wrist feature points or gesture area center points with the gesture direction determined in the simulated driving training area image. Two curves are fitted according to the distribution of wrist feature points or gesture area center points. One curve is used to show the distribution of each wrist feature point / gesture area center point with the gesture direction upward, and is called the first design curve. The other curve is used to show the distribution of each wrist feature point / gesture area center point with the gesture direction downward, and is called the second design curve. The judgment conditions 1, 2, 3, and 4, and their corresponding judgment results are as follows: Judgment Condition 1: Calculate the horizontal and vertical distances between the wrist feature point and the corresponding shoulder feature point. When both the horizontal and vertical distances are greater than zero, if the horizontal distance is greater than a first horizontal threshold and the vertical distance is greater than a first vertical threshold, then the current gesture direction is upward. When both the horizontal and vertical distances are less than zero, if the absolute value of the horizontal distance is greater than a first horizontal threshold and the absolute value of the vertical distance is greater than a first vertical threshold, then the current gesture direction is downward. Both the first horizontal threshold and the first vertical threshold are greater than zero. Judgment condition 2: The angle between the forearm and the horizontal direction of the simulated driving training area image is taken as the forearm angle. If the forearm angle is greater than the set forearm angle threshold, the current gesture direction is upward; otherwise, the current gesture direction is downward. The line connecting the wrist feature point related to the current gesture and the elbow feature point corresponding to the wrist feature point is taken as the forearm; Judgment condition 3: The angle between the long arm and the horizontal direction of the simulated driving training area image is taken as the long arm angle. If the long arm angle is greater than the set long arm angle threshold, the current gesture direction is upward; otherwise, the current gesture direction is downward. The line connecting the wrist feature point related to the current gesture and the shoulder feature point corresponding to the wrist feature point in the human body feature points is taken as the long arm. Judgment condition 4: If the angle between the line connecting the wrist feature point and the head feature point related to the current gesture in the human body feature points and the horizontal direction of the simulated driving training area image is greater than the first set threshold, then the current gesture is upward; otherwise, the current gesture is downward.

2. The simulated driving posture evaluation method according to claim 1, characterized in that, The human body feature points include the left wrist feature point, the right wrist feature point, the left elbow feature point, and the right elbow feature point; In step S2, the driver's hand performing the current gesture is determined as follows: if the feature point of the left wrist is in the gesture area and the feature point of the right wrist is not in the gesture area, then the driver's hand performing the current gesture is the left hand. If the right wrist feature point is in the gesture area and the left wrist feature point is not in the gesture area, then the driver's right hand is the one performing the current gesture. If both the left and right wrist feature points are within the gesture area, calculate the overlap area between the left arm direction and the gesture area and the right arm direction and the gesture area. If the overlap area of ​​the left arm is greater than the overlap area of ​​the right arm, then the driver's hand performing the current gesture is the left hand; otherwise, the driver's hand performing the current gesture is the right hand. The direction of the left arm is obtained by using the feature points of the left wrist and the left elbow, and the direction of the right arm is obtained by using the feature points of the right wrist and the right elbow.

3. The simulated driving posture evaluation method according to claim 1, characterized in that, The gesture direction includes upward or downward, and the gesture area feature point includes the center point of the gesture area; in step S3, the gesture direction when the driver performs the current gesture is determined by the following method: based on the pre-acquired first design curve and second design curve, a first distance between the wrist feature point / gesture area center point related to the current gesture and the first design curve is calculated, and a second distance between the wrist feature point related to the current gesture and the second design curve is also calculated; if the first distance is greater than the second distance, the current gesture direction is upward; if the first distance is less than the second distance, the current gesture direction is downward. The first and second design curves are obtained by fitting historical experimental data: historical experimental data is obtained, which includes wrist feature points or gesture area center points with the gesture direction determined in the simulated driving training area image. Two curves are fitted according to the distribution of wrist feature points or gesture area center points. One curve is used to show the distribution of each wrist feature point / gesture area center point with the gesture direction upward, and is called the first design curve. The other curve is used to show the distribution of each wrist feature point / gesture area center point with the gesture direction downward, and is called the second design curve.

4. The simulated driving posture evaluation method according to claim 1, characterized in that, The gesture direction includes upward and downward. In step S3, one of the following judgment conditions is used to determine the gesture direction when the driver performs the current gesture. Each judgment condition and the corresponding judgment result are as follows: Judgment condition 1: The angle between the forearm and the horizontal direction of the simulated driving training area image is taken as the forearm angle. If the forearm angle is greater than the set forearm angle threshold, the current gesture direction is upward; otherwise, the current gesture direction is downward. The line connecting the wrist feature point related to the current gesture and the elbow feature point corresponding to the wrist feature point is taken as the forearm; Judgment condition 2: The angle between the long arm and the horizontal direction of the simulated driving training area image is taken as the long arm angle. If the long arm angle is greater than the set long arm angle threshold, the current gesture direction is upward; otherwise, the current gesture direction is downward. The line connecting the wrist feature point related to the current gesture and the shoulder feature point corresponding to the wrist feature point in the human body feature points is taken as the long arm. Judgment condition 3: If the angle between the line connecting the wrist feature point and the head feature point related to the current gesture in the human body feature points and the horizontal direction of the simulated driving training area image is greater than the first set threshold, then the current gesture is upward; otherwise, the current gesture is downward.

5. The simulated driving posture evaluation method according to claim 1, characterized in that, The human body feature points include wrist feature points, and the gesture direction includes left, center, or right. In step S3, the direction of the driver's gesture when performing the current gesture is determined by the following method: if the wrist feature point of the hand performing the current gesture is in the preset left region, then the direction of the current gesture is left. If the wrist feature point of the hand performing the current gesture is located in the pre-defined central region, then the direction of the current gesture is the central direction. If the wrist feature point of the hand performing the current gesture is located in a pre-defined right region, then the direction of the current gesture is to the right.

6. The simulated driving posture evaluation method according to claim 1, characterized in that, In step S1, the driver and the corresponding driver's gesture area within the simulated driving training area image are identified through the following steps: S1.1 Input the simulated driving training area image into the pre-established target detection model, and output the personnel target and gesture area in the image; S1.

2. Based on the pre-set driver area, determine the driver and the corresponding driver's gesture area in the personnel target; The training data for the target detection model includes images of simulated driving training areas labeled with personnel targets and gesture regions.

7. The simulated driving posture evaluation method according to claim 6, characterized in that, If there are multiple personnel targets and gesture regions in the simulated driving training area image, obtain the number of human feature points of each personnel target in the driver region, take the personnel target with the largest number of human feature points as the driver, and take the gesture region related to the personnel target with the largest number of human feature points as the gesture region of the corresponding driver.

8. A driving simulator, characterized in that, The method includes an image acquisition device for acquiring images of a simulated driving training area and a controller communicatively connected to the image acquisition device. The controller includes a processor and a memory, and the processor executes instructions in the memory to implement the simulated driving posture evaluation method according to any one of claims 1-7.

9. The driving simulator according to claim 8, characterized in that, It also includes a display device connected to the controller for displaying the results of the driver's simulated driving posture evaluation.

Images