Vehicle control method and device based on intervention gesture recognition, automobile and medium

Abstract

The application relates to a vehicle control method and device based on intervention gesture recognition, an automobile and a medium, wherein the method comprises the following steps: acquiring point cloud data of pedestrians and a road in a vehicle driving process, and identifying feature information of each pedestrian and road feature information based on the point cloud data; performing traffic intervener scoring calculation according to the feature information to obtain a scoring calculation result, and determining an initial traffic intervener in the pedestrians; performing pedestrian trajectory prediction based on the road feature information and the feature information of the initial traffic intervener to obtain a prediction result, and dividing a current drivable area of the vehicle; determining a target traffic intervener in the initial traffic intervener according to the prediction result, the drivable area and obstacle information; performing traffic instruction recognition according to a gesture of the target traffic intervener to obtain a target traffic instruction, and controlling the vehicle according to the target traffic instruction and traffic flow information. The application enables automatic driving to adapt to complex traffic scenes and improves the safety of automatic driving.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a vehicle control method and device based on intervention gesture recognition, a car and a medium. BACKGROUND

[0002] During automatic driving, a traffic intervention role usually appears in the middle of the road, which directly intervenes in the vehicle passing situation of the road in addition to the inherent traffic system, at this time, the vehicle needs to follow the traffic participant's gesture command to act. Common traffic interveners include traffic police, auxiliary police, volunteers, construction personnel, security guards, people directing reversing vehicles, and traffic jam relief command personnel. Although there are standard traffic police gestures for reference in the driving license examination, many times the interveners do not necessarily command traffic according to standard gestures, but rely on some more simple and easy-to-understand gestures to quickly understand their intentions. In addition, traffic interveners are not only traffic police, and the real situation is more complex. If the intervention gesture of the traffic interveners cannot be understood in time, it may cause traffic congestion and affect the passing of others, and even some dangerous situations may occur due to the failure to stop in time due to front landslides or accidents, so it is necessary to correctly understand the semantic information expressed by the traffic interveners to make correct response behaviors.

[0003] The existing method is to determine the traffic instruction corresponding to the traffic intervention gesture through gesture recognition, and then control the vehicle; however, there are multiple pedestrians and different gestures of pedestrians during the driving of the vehicle, which causes the existing method to be difficult to accurately identify the traffic interveners and the traffic instructions corresponding to the traffic interveners, and causes the automatic driving to be difficult to adapt to complex traffic scenes, and the safety of the automatic driving is low. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a vehicle control method and device based on intervention gesture recognition, a car and a medium, to accurately identify traffic interveners and traffic instructions corresponding to the traffic interveners, so that the automatic driving can adapt to complex traffic scenes and improve the safety of the automatic driving.

[0005] In order to solve the above technical problems, the embodiments of the present application provide a vehicle control method based on intervention gesture recognition, comprising:

[0006] Obtaining point cloud data of pedestrians and roads in the driving process of the vehicle, and identifying feature information of each pedestrian and road feature information based on the point cloud data, wherein the road feature information includes obstacle information and traffic flow information;

[0007] According to the feature information, a traffic intervention score is calculated to obtain a score calculation result, and an initial traffic interveners in the pedestrians is determined according to the score calculation result;

[0008] perform pedestrian trajectory prediction based on the road feature information and the feature information of the initial traffic intervenor, obtain a prediction result, and divide a current drivable area of the vehicle;

[0009] score the initial traffic intervenor according to the prediction result, the drivable area, and the obstacle information, and determine a target traffic intervenor from the initial traffic intervenors according to a score result;

[0010] perform traffic instruction recognition according to a gesture of the target traffic intervenor, obtain a target traffic instruction, and control the vehicle according to the target traffic instruction and the traffic flow information.

[0011] To solve the above technical problem, an embodiment of the present application provides a vehicle control device based on intervenor gesture recognition, comprising:

[0012] a point cloud data acquisition unit configured to acquire point cloud data of pedestrians and roads in a vehicle driving process, and identify feature information of each pedestrian and road feature information based on the point cloud data, wherein the road feature information comprises obstacle information and traffic flow information;

[0013] a traffic intervenor determination unit configured to perform traffic intervenor score calculation according to the feature information, obtain a score calculation result, and determine an initial traffic intervenor from the pedestrians according to the score calculation result;

[0014] a drivable area division unit configured to perform pedestrian trajectory prediction based on the road feature information and the feature information of the initial traffic intervenor, obtain a prediction result, and divide a current drivable area of the vehicle;

[0015] a confidence score unit configured to score the initial traffic intervenor according to the prediction result, the drivable area, and the obstacle information, and determine a target traffic intervenor from the initial traffic intervenors according to a score result;

[0016] a vehicle control unit configured to perform traffic instruction recognition according to a gesture of the target traffic intervenor, obtain a target traffic instruction, and control the vehicle according to the target traffic instruction and the traffic flow information.

[0017] To solve the above technical problem, an embodiment of the present application provides a vehicle control device based on intervenor gesture recognition, comprising:

[0018] To solve the above technical problems, one technical solution adopted by the present application is: a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the vehicle control method based on intervenor gesture recognition.

[0019] The embodiment of the present application provides a vehicle control method, device, automobile and medium based on intervenor gesture recognition. The method comprises the following steps: acquiring point cloud data of pedestrians and roads in the vehicle driving process, and identifying feature information of each pedestrian and road feature information based on the point cloud data, wherein the road feature information comprises obstacle information and traffic flow information; performing traffic intervenor scoring calculation according to the feature information to obtain a scoring calculation result, and determining an initial traffic intervenor among the pedestrians according to the scoring calculation result; performing pedestrian trajectory prediction based on the road feature information and the feature information of the initial traffic intervenor to obtain a prediction result, and dividing a current drivable area of the vehicle; performing confidence score calculation on the initial traffic intervenor according to the prediction result, the drivable area and the obstacle information, and determining a target traffic intervenor among the initial traffic intervenors according to the confidence score calculation result; performing traffic instruction recognition according to a gesture of the target traffic intervenor to obtain a target traffic instruction, and controlling the vehicle according to the target traffic instruction and the traffic flow information. The embodiment of the present application accurately identifies the traffic intervenor from a plurality of pedestrians in the vehicle driving process by extracting the feature information of the pedestrians and the road feature information, and then directly identifies the gesture corresponding to the traffic intervenor, thereby avoiding the interference of gestures of other pedestrians, accurately recognizing the traffic instruction of the traffic intervenor gesture, facilitating accurate control of the vehicle based on the traffic instruction, enabling the automatic driving to adapt to complex traffic scenes, and being beneficial to improving the safety of the automatic driving. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the scheme in the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0021] Figure 1 is the implementation flowchart of the vehicle control method based on intervenor gesture recognition provided by the embodiment of the present application;

[0022] Figure 2 is the implementation flowchart of the sub-process in the vehicle control method based on intervenor gesture recognition provided by the embodiment of the present application;

[0023] Figure 3is an implementation flowchart of a sub-process in a vehicle control method based on intervention gesture recognition provided by an embodiment of the present application;

[0024] Figure 4 is an implementation flowchart of a sub-process in a vehicle control method based on intervention gesture recognition provided by an embodiment of the present application;

[0025] Figure 5 is an implementation flowchart of a sub-process in a vehicle control method based on intervention gesture recognition provided by an embodiment of the present application;

[0026] Figure 6 is an implementation flowchart of a sub-process in a vehicle control method based on intervention gesture recognition provided by an embodiment of the present application;

[0027] Figure 7 is an implementation flowchart of a sub-process in a vehicle control method based on intervention gesture recognition provided by an embodiment of the present application;

[0028] Figure 8 is a vehicle control implementation process schematic diagram provided by an embodiment of the present application;

[0029] Figure 9 is a vehicle control device schematic diagram based on intervention gesture recognition provided by an embodiment of the present application;

[0030] Figure 10 is a schematic diagram of an automobile provided by an embodiment of the present application. DETAILED DESCRIPTION

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application; the description and claims of this application as well as the above abstract are intended to cover all alternatives, modifications, equivalents and variations of the present application falling within the scope of the application. The terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application. The singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0032] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in an embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all directed to the same embodiment, or to a single alternative implementation. It is explicitly contemplated that embodiments described herein can be combined with each other.

[0033] In order to better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be clearly and completely described below with reference to the drawings.

[0034] The present application will be described in detail below with reference to the drawings and embodiments.

[0035] It should be noted that the vehicle control method based on intervention gesture recognition provided in the embodiments of the present application is generally executed by a car, and accordingly, the vehicle control device based on intervention gesture recognition is generally configured in a car.

[0036] Please refer to Figure 1 , Figure 1 An embodiment of the vehicle control method based on intervention gesture recognition is shown.

[0037] It should be noted that the method of the present application is not limited to the flow sequence shown, and the method includes the following steps: Figure 1

[0038] S1: Obtain point cloud data of pedestrians and roads in the driving process of a vehicle, and identify feature information of each pedestrian and road feature information based on the point cloud data.

[0039] The road feature information includes obstacle information and traffic flow information.

[0040] Specifically, in the driving process of the vehicle, the point cloud data of pedestrians and roads around the vehicle is received by a radar, and then each pedestrian is identified from the point cloud data, and the information of the clothes, position, orientation, speed and gesture of the pedestrian is identified to obtain the feature information of each pedestrian. Finally, the corresponding obstacle information and traffic flow information are identified according to the shape and size of the point cloud data to obtain the road feature information.

[0041] The feature information of the pedestrian includes the clothes, position, orientation, speed and gesture of the pedestrian. The laser radar (LiDAR) or other radar equipment installed on the vehicle continuously scans the surrounding environment to generate 3D point cloud data. These data contain the position and shape information of all objects around the vehicle.

[0042] Please refer to Figure 2 , Figure 2 An embodiment of step S1 is shown, which is described in detail as follows:

[0043] S11: Receive the point cloud data of pedestrians and roads in the driving process of the vehicle by a radar.

[0044] S12: Perform pedestrian segmentation processing on the point cloud data by a pedestrian recognition model to obtain a plurality of pedestrian data.

[0045] ​Specifically, the point cloud data is processed using a pre-trained pedestrian recognition model. The pedestrian recognition model can identify which point cloud data belongs to pedestrians and perform preliminary segmentation to divide the point cloud into multiple pedestrian data blocks, each block representing an independent pedestrian. Pedestrian data refers to data composed of independent pedestrians.

[0046] S13: Identify the pedestrian information in the pedestrian data using image recognition technology.

[0047] Specifically, the camera mounted on the vehicle captures the vehicle and radar synchronously, and the surrounding environment is captured by the camera on the vehicle. Then the camera image is processed using image recognition technology to identify the orientation, clothing, gestures, etc. of the pedestrians in the pedestrian data, and the pedestrian information is obtained.

[0048] S14: Perform clothing recognition and gesture recognition based on the pedestrian information using a clothing recognition model and a gesture recognition model, respectively, to obtain the recognition results.

[0049] S15: Generate the feature information of each pedestrian based on the pedestrian information and the recognition results.

[0050] Specifically, the clothing recognition and gesture recognition are performed based on the pedestrian information using a pre-trained clothing recognition model and a gesture recognition model, respectively, to obtain the recognition results of the pedestrian clothing and gestures; and the feature information of each pedestrian is generated based on the pedestrian information and the recognition results.

[0051] S16: Perform vehicle and obstacle recognition on the point cloud data to obtain the road feature information.

[0052] Specifically, clustering algorithms such as DBSCAN, K-means, etc. are used to segment the point cloud data, and point clouds belonging to the same object are clustered into clusters. These clusters may represent vehicles, pedestrians, trees, buildings or other obstacles. For each cluster, calculate its shape (such as bounding box, convex hull, etc.) and size (such as volume, area, length, width, etc.) features. These features can distinguish different types of obstacles and cars. Therefore, according to the shape, size and other features of the clustering cluster, a set of predefined rules are applied to identify obstacles and vehicles, thereby obtaining the road feature information. For example, a clustering cluster with a specific aspect ratio and height may be considered as a vehicle.

[0053] Please refer to Figure 3 , Figure 3 A specific embodiment before step S1 is shown as follows:

[0054] S1A: receiving point cloud data in an initial vehicle driving process through radar, and generating first training data by clustering and labeling the point cloud data, training a model according to the first training data to obtain the pedestrian recognition model.

[0055] Specifically, during the driving of the vehicle, three-dimensional point cloud data of the surrounding environment is continuously collected using a laser radar (LiDAR) installed on the vehicle. These data contain the position and shape information of all objects around the vehicle. The collected point cloud data is preprocessed, including removing noise points, filling missing data, adjusting point cloud density, etc., to improve the quality of subsequent processing. The preprocessed point cloud data is processed using clustering algorithms (such as DBSCAN, K-means or their variants specially designed for point cloud) to cluster point cloud data belonging to the same object into clusters. The purpose of this step is to preliminarily distinguish pedestrians from other objects (such as vehicles, trees, buildings). Then the clustered point cloud data is annotated in detail. The annotator needs to identify whether each cluster represents a pedestrian and mark the precise boundary of the pedestrian (such as a bounding box or contour). This step is crucial to ensure the quality of the training data, because the accuracy of the annotation will directly affect the performance of the pedestrian recognition model. The annotated point cloud data (including pedestrian labels and boundary information) is used as the first training data. These data will be used to train the pedestrian recognition model.

[0056] The first training data is used to train the pedestrian recognition model. This model can be a deep learning model (such as convolutional neural network CNN, point cloud neural network PointNet, etc.), which can learn the features of pedestrians from point cloud data and predict whether new point cloud data contains pedestrians and the location of pedestrians. During training, the model will iteratively adjust its parameters to minimize the difference between the predicted results and the actual annotations (i.e. loss function), and when training is complete, the pedestrian recognition model is obtained and deployed to the car.

[0057] S1B: Collecting clothing images of the target group, and screening and labeling the clothing images to obtain second training data, training a model according to the second training data to obtain the clothing recognition model.

[0058] The target group includes road construction personnel, personnel wearing a vest, and personnel wearing a police uniform, etc., and the target group can be a traffic intervenor.

[0059] Collect the clothes images of the target group, and remove the clothes pictures that are blurred, low resolution, poorly angled, or contain a large amount of irrelevant background; then ensure that the clothes in the pictures are highly relevant to the target group and the identification needs; then use image hash technology to remove duplicate or highly similar pictures, and obtain the screened clothes images. Then use open source tools such as LabelImg and VGGImage Annotator (VIA) to assign one or more category labels (such as police uniform, waistcoat) to each screened clothes image, and obtain the labeled clothes images. Then adjust all the labeled clothes images to a uniform size required for model training, and increase data diversity through rotation, scaling, cropping, and flipping to improve the generalization ability of the model. Finally, scale the pixel values to a specific range (such as 0-1) to obtain the second training data.

[0060] Any model in ResNet, VGG, or MobileNet in the convolutional neural network (CNN) is used as the training model. The model is trained according to the second training data to obtain the clothes recognition model, which is deployed in the car.

[0061] S1 C: Obtain target gesture image data of different orientations, and label the gesture type, position, and orientation of the target gesture image data to obtain third training data. The gesture recognition model is obtained by training the model according to the third training data.

[0062] Specifically, the target gesture image data of different orientations is obtained, and the orientation of the gesture is focused on to ensure that the target gesture image data contains gesture images taken from various angles (such as front, side, back, etc.). Then, the target gesture image is cropped to only include the gesture part and scaled uniformly to the same size for subsequent processing. Then, the data diversity is increased through rotation, flipping, adjusting brightness / contrast, etc., which helps to improve the robustness of the model. Then, the enhanced target gesture image is labeled, which includes gesture type: labeling the type of gesture for each image, such as "fist", "open", "wave", etc.; position: labeling the position of the gesture in the image, which can be marked using a bounding box; orientation: for gestures that need to recognize the orientation, the orientation angle or direction (such as front, left, right, etc.) of the gesture also needs to be labeled. After labeling, the third training data is obtained.

[0063] A convolutional neural network is used as the training model, and then the model is trained according to the third training data to generate the gesture recognition model. The gesture recognition model is deployed in the car.

[0064] S2: Calculate the traffic intervenor score according to the feature information to obtain the score calculation result, and determine the initial traffic intervenor among the pedestrians according to the score calculation result.

[0065] Specifically, the judgment of traffic interveners is made by using the gestures, positions and clothes of pedestrians in the feature information. In the above trained model (including gesture recognition model, clothes recognition model), the similarity scores (values between 0 and 1) of the gestures and clothes of pedestrians are recognized, while the position is quantified according to the distance between the pedestrian and the lane line, whether it is in the pedestrian crossing area and other factors, which is a value between 0 and 1. The present application designs a weighted sum form to integrate the three factors to form a score calculation result (Score, S):

[0066] S = w C *C + w P *P + w G *G;

[0067] Wherein, w C , w P , w G are the weights of clothes, position and gesture respectively, and C, P, G are the scores of clothes, position and gesture. The specific weights are set according to the priority of the specific application scene. These weights can be fixed or dynamically adjusted according to the training data. If the score result S is greater than the preset threshold, the pedestrian is identified as the initial traffic intervener.

[0068] S3: predicting the pedestrian trajectory based on the road feature information and the feature information of the initial traffic intervener to obtain a prediction result and dividing the current drivable area of the vehicle.

[0069] Specifically, in order to further accurately identify the real traffic intervener, the present application embodiment will also predict the pedestrian action trajectory and divide the current drivable area of the vehicle, so as to further judge the traffic intervener by integrating the pedestrian trajectory and the drivable area.

[0070] Please refer to Figure 4 , Figure 4 A specific implementation of step S3 is shown as follows:

[0071] S31: projecting the road feature information into the environmental road network of the semantic map.

[0072] Specifically, this step is an important part of the autonomous vehicle's perception of the environment. The road feature information collected by the vehicle through sensors such as cameras, radars, lidars, etc. (such as lane lines, traffic signs, obstacles, etc.) needs to be integrated and projected into a high-precision semantic map. The semantic map not only contains the physical structure information of the road, but also contains the semantic information of the road (such as lane type, traffic rules, etc.). By projecting the road feature information into the semantic map, the vehicle can construct a three-dimensional or two-dimensional virtual world containing rich environmental information, providing a basis for subsequent decision-making and planning.

[0073] S32: Extract the motion features of each frame in the initial traffic intervenor's feature information, wherein the motion features include speed, orientation, acceleration, and angular acceleration.

[0074] Specifically, attention needs to be paid to pedestrians or other traffic participants (i.e. initial traffic intervenors) that may affect the vehicle's driving trajectory. By tracking the motion of these intervenors, the motion features of each frame can be extracted, including speed, orientation, acceleration, and angular acceleration. These features reflect the dynamic changes of the intervenors and are an important basis for predicting their future trajectories.

[0075] S33: Perform trajectory prediction based on the environment road network and the motion features through a pedestrian trajectory prediction model to obtain the prediction result.

[0076] Specifically, using the motion features extracted in the previous step, combined with the environment road network information in the semantic map, the future motion trajectory of the intervenor can be predicted through a pedestrian trajectory prediction model. This model can learn and understand the behavior patterns of intervenors in different scenarios, thus more accurately predicting their trajectories. The predicted trajectory will be used to assess whether the intervenor is likely to collide with the vehicle in the future and how likely the collision is.

[0077] Further, before step S33, the embodiments of the present application further include training a pedestrian trajectory prediction model.

[0078] The specific process is as follows: a large amount of driving data of social vehicles (non-autonomous vehicles) is collected through various sensors installed on the vehicle (such as GPS, IMU, camera, radar, lidar, etc.) and possible third-party data sources (such as traffic monitoring cameras, mobile phone APPs, etc.). These data include the position, speed, direction, acceleration, etc. of the vehicle, as well as detailed information about the surrounding environment (such as road type, intersection, pedestrian, other vehicles, etc.). The collected driving data trajectory is projected into a high-precision map environment road network to construct a comprehensive data set containing vehicle driving trajectory and road environment. The motion characteristics of each frame of the pedestrian, such as speed, direction, acceleration, angular acceleration, etc. are extracted from the processed data, and a training set is constructed based on these motion characteristics. A neural model is used for training, and the architecture of the neural network can be designed according to specific needs, such as using recurrent neural network (RNN) or long short-term memory network (LSTM) to process time series data, or using convolutional neural network (CNN) to process image data (such as pedestrian images obtained from cameras). When the training is completed, a pedestrian trajectory prediction model is generated, which can predict the possible future driving trajectory of the pedestrian or other traffic interveners according to the input motion characteristics and current environmental road network information. At the same time, the model can also evaluate the probability of different trajectories, so as to find out the most probable possible driving trajectory.

[0079] S34: Comparing the embankment position marked or perceived in the semantic map with the current position of the vehicle to divide the current drivable area of the vehicle.

[0080] Specifically, the embankment position marked or perceived in the semantic map needs to be compared with the current position of the vehicle to determine the current environmental context of the vehicle. Through this process, the current drivable area of the vehicle can be divided, i.e. the area that meets the traffic rules, has no obstacles and is suitable for vehicle driving. This drivable area will serve as the basis for subsequent driving decision and planning of the vehicle, ensuring that the vehicle can drive to the destination according to the optimal path under the premise of safety.

[0081] S4: According to the prediction result, the drivable area and the obstacle information, a confidence score is given to the initial traffic interveners, and a target traffic interveners is determined according to the confidence score result.

[0082] Please refer to Figure 5 , Figure 5 An embodiment of step S4 is shown as follows:

[0083] S41: Based on the prediction result, it is judged whether the initial traffic interveners are in the drivable area, and a first judgment result is obtained.

[0084] Specifically, the prediction result obtained by the pedestrian trajectory prediction model is used to determine whether the future trajectory of the initial traffic intervenor (such as a pedestrian) within the prediction time range will intersect with the drivable area of the vehicle. If the predicted trajectory overlaps with the drivable area, it is considered that the intervenor may enter or is already in the drivable area, and the first determination result is "yes"; otherwise, the first determination result is "no".

[0085] S42: Determine whether the initial traffic intervenor is in the drivable area and there is obstacle information behind based on the prediction result, and obtain a second determination result.

[0086] Specifically, this step further refines the evaluation of the traffic intervenor. Not only is it considered whether the traffic intervenor may enter the drivable area, but also whether there is obstacle information behind. Here, "behind" is relative to the direction of the initial traffic intervenor. If the prediction result shows that the intervenor may enter the drivable area and there is an obstacle (such as other vehicles, stationary objects, etc.) behind, the second determination result is "yes"; otherwise, the second determination result is "no".

[0087] S43: Perform confidence scoring on the initial traffic intervenor based on the first determination result and the second determination result, and obtain a confidence scoring result.

[0088] Specifically, the initial traffic intervenor is scored according to the first and second determination results. Confidence scoring is a quantitative indicator used to represent the prediction accuracy and potential risk assessment of the future behavior of the initial traffic intervenor. The specific algorithm of the score can be designed according to actual needs, but generally considers the following factors: the degree of overlap between the predicted trajectory and the drivable area; the uncertainty of the predicted trajectory (such as the confidence interval predicted by the model); the presence or absence of obstacle information behind and its type (such as the difference between dynamic and static obstacles); the relative distance and speed between the obstacle and the predicted trajectory. By considering these factors comprehensively, a confidence score result can be given to each initial traffic intervenor.

[0089] S44: Determine the target traffic intervenor among the initial traffic intervenors according to the confidence scoring result.

[0090] Specifically, the target traffic intervenor is selected from the initial traffic intervenors according to the confidence scoring result. The threshold or ranking of the confidence score can be used to determine which initial traffic intervenors are considered as target traffic intervenors.

[0091] S5: Perform traffic instruction recognition according to the gesture of the target traffic intervenor, obtain a target traffic instruction, and control the vehicle according to the target traffic instruction and the traffic flow information.

[0092] Specifically, it is necessary to identify the traffic instruction corresponding to the gesture of the target traffic intervenor, and then control the vehicle in combination with the current traffic flow information and the traffic instruction.

[0093] Please refer to Figure 6 , Figure 6 An embodiment of step S5 is shown as follows:

[0094] S51: Record the positions of multiple gestures of the target traffic intervenor in time sequence to obtain time sequence data of the gestures.

[0095] Specifically, it is necessary to continuously monitor and record the positions of the gestures of the target traffic intervenor, which is achieved by sensors such as cameras or radars installed on the vehicle. The sensors will capture image or point cloud data of the gestures, and then record the position information of the gestures in time sequence to obtain the time sequence data of the gestures. These data not only contain the positions of the gestures in space, but also contain their trajectories changing over time.

[0096] S52: Identify the target traffic instruction based on the time sequence data of the gestures by a traffic instruction identification model.

[0097] Specifically, a pre-trained traffic instruction identification model is used to process the time sequence data of the gestures. This model is a deep learning model, such as a convolutional neural network (CNN) combined with a recurrent neural network (RNN) or a long short-term memory network (LSTM), which is used to capture the spatiotemporal features of the gestures. The model analyzes the shape, direction, speed of the gestures and their changing patterns over time, and matches them with predefined traffic instruction templates. Finally, the model outputs a target traffic instruction, which represents the intention of the traffic intervenor to convey through gestures, such as stopping, slow driving, turning, etc.

[0098] Further, before step S53, the embodiments of the present application further include training the traffic instruction identification model.

[0099] Specifically, it is necessary to concatenate the gestures in time sequence to form a gesture sequence. This sequence reflects the dynamic change process of the gestures of the pedestrian. In order to construct such a sequence, it is necessary to be able to track the moving trajectory and changing pattern of the gestures, and to identify the transition and correlation between the gestures. Then the model is trained according to the gesture sequence to generate the traffic instruction identification model. This model will infer the intention or instruction that the pedestrian wants to convey according to the shape, sequence, duration and other characteristics of the gestures, that is, it can identify the corresponding traffic instruction.

[0100] S53: Control the vehicle according to the target traffic instruction and the traffic flow information.

[0101] Specifically, the target traffic instruction is combined with the current traffic flow information to formulate a control strategy for the vehicle. The traffic flow information includes the positions, speeds, driving directions, etc. of surrounding vehicles, which are usually obtained through sensors installed on the vehicle (such as radar, lidar, camera, etc.) and vehicle networking technology. According to the priority of the target traffic instruction and the real-time situation of the traffic flow information, the most appropriate driving path and speed are calculated, and these control instructions are executed through the vehicle control system (such as the braking system, steering system, power system, etc.). For example, if the target traffic instruction is to stop, the vehicle will be controlled to slow down and stop safely; if the target traffic instruction is to turn, the vehicle will be controlled to turn according to the specified route and speed.

[0102] Referring to Figure 7 , Figure 7 An embodiment of step S53 is shown as follows:

[0103] S531: If the vehicle is not the first vehicle and the traffic flow in the traffic flow information is passing on one side of the target traffic intervenor, the vehicle is controlled to follow the front vehicle in the traffic flow information.

[0104] S532: If the vehicle is the first vehicle and there is no reference vehicle in the traffic flow information, the vehicle is controlled based on the target traffic instruction.

[0105] S533: If there is a red light in front of the road and there is no reference vehicle in the traffic flow information, the vehicle is controlled to wait.

[0106] The embodiments of the present application provide vehicle control strategies in several cases. If the vehicle is not the first vehicle and the traffic flow in the traffic flow information is passing on one side of the target traffic intervenor, it is considered that the target traffic intervenor has no traffic flow passing through the risk side, and the vehicle needs to follow the traffic flow from the non-risk side; if the vehicle is the first vehicle and there is no reference vehicle in the traffic flow information, the vehicle is controlled based on the target traffic instruction. The reference vehicle refers to the vehicle driving around the autonomous vehicle. However, if a gesture is received but its clear meaning is not understood, the vehicle needs to be controlled to stop slowly and interact with the intervenor again to observe the subsequent instructions or have a conversation with the intervenor by the main driver in the vehicle or remotely. If there is a red light in front of the road and there is no reference vehicle in the traffic flow information, the vehicle is controlled to wait.

[0107] As Figure 8 shown, Figure 8Fig. 1 is a schematic diagram of a vehicle control implementation process provided by an embodiment of the present application. In the figure, ego refers to an autonomous vehicle. When a suspected traffic intervenor is encountered at the observation end, the execution end slightly decelerates and prepares; when a stop gesture signal is identified, the vehicle is stopped; when the gesture for ego disappears, the vehicle continues to stop and wait; when a go gesture is identified, the vehicle starts until the gesture disappears.

[0108] In an embodiment of the present application, point cloud data of pedestrians and roads in the vehicle driving process is acquired, and feature information of each pedestrian and road feature information is identified based on the point cloud data, wherein the road feature information includes obstacle information and traffic flow information; traffic intervenor scoring calculation is performed according to the feature information, a scoring calculation result is obtained, and an initial traffic intervenor in the pedestrians is determined according to the scoring calculation result; pedestrian trajectory prediction is performed based on the road feature information and the feature information of the initial traffic intervenor, a prediction result is obtained, and a current drivable area of the vehicle is divided; the initial traffic intervenor is scored for confidence according to the prediction result, the drivable area, and the obstacle information, and a target traffic intervenor in the initial traffic intervenor is determined according to the confidence score result; traffic instruction identification is performed according to the gesture of the target traffic intervenor, a target traffic instruction is obtained, and the vehicle is controlled according to the target traffic instruction and the traffic flow information. Through the extraction of the feature information of the pedestrians and the road feature information, and the combination of the feature information of the pedestrians and the road feature information, the traffic intervenor is accurately identified from the multiple pedestrians in the vehicle driving process, then the gesture corresponding to the traffic intervenor is directly identified, the interference of the gestures of other pedestrians is avoided, the traffic instruction of the traffic intervenor gesture can be accurately identified, the vehicle can be precisely controlled based on the traffic instruction, the autonomous driving can adapt to complex traffic scenes, and the safety of the autonomous driving is improved.

[0109] Please refer to Figure 9 , as an implementation of the method shown in Figure 1 , the present application provides an embodiment of a vehicle control device based on intervenor gesture identification, which corresponds to the method embodiment shown in Figure 1 , and the device can be specifically applied to a car.

[0110] As shown in Figure 9 , the vehicle control device based on intervenor gesture identification of the present embodiment includes a point cloud data acquisition unit 61, a traffic intervenor determination unit 62, a drivable area division unit 63, a confidence scoring unit 64, and a vehicle control unit 65, wherein:

[0111] The point cloud data acquisition unit 61 is configured to acquire point cloud data of pedestrians and a road during driving of a vehicle, and identify feature information of each pedestrian and road feature information based on the point cloud data, wherein the road feature information comprises obstacle information and traffic flow information.

[0112] The traffic intervenor determination unit 62 is configured to perform traffic intervenor scoring calculation according to the feature information, obtain a scoring calculation result, and determine an initial traffic intervenor among the pedestrians according to the scoring calculation result.

[0113] The drivable area division unit 63 is configured to perform pedestrian trajectory prediction based on the road feature information and the feature information of the initial traffic intervenor, obtain a prediction result, and divide a current drivable area of the vehicle.

[0114] The confidence score unit 64 is configured to perform confidence scoring on the initial traffic intervenor according to the prediction result, the drivable area, and the obstacle information, and determine a target traffic intervenor among the initial traffic intervenors according to the confidence scoring result.

[0115] The vehicle control unit 65 is configured to perform traffic instruction recognition according to a gesture of the target traffic intervenor, obtain a target traffic instruction, and control the vehicle according to the target traffic instruction and the traffic flow information.

[0116] Further, the point cloud data acquisition unit 61 comprises:

[0117] A point cloud data receiving unit is configured to receive point cloud data of pedestrians and a road during driving of a vehicle through a radar;

[0118] A pedestrian segmentation unit is configured to perform pedestrian segmentation processing on the point cloud data through a pedestrian recognition model to obtain a plurality of pedestrian data;

[0119] A pedestrian information recognition unit is configured to identify pedestrian information in the pedestrian data by using an image recognition technology;

[0120] An identification result generation unit is configured to perform clothing recognition and gesture recognition based on the pedestrian information through a clothing recognition model and a gesture recognition model respectively to obtain an identification result;

[0121] A feature information generation unit is configured to generate the feature information of each pedestrian based on the pedestrian information and the identification result;

[0122] A road feature information generation unit is configured to perform vehicle and obstacle recognition on the point cloud data to obtain the road feature information.

[0123] Further, the point cloud data acquisition unit 61 further comprises:

[0124] The pedestrian recognition model training unit is configured to receive point cloud data in an initial vehicle driving process by radar, and perform clustering and labeling processing on the point cloud data to generate first training data, and perform model training according to the first training data to obtain the pedestrian recognition model.

[0125] The clothes recognition model training unit is configured to collect clothes images of a target group, and perform screening and labeling on the clothes images to obtain second training data, and perform model training according to the second training data to obtain the clothes recognition model.

[0126] The gesture recognition model training unit is configured to obtain target gesture image data in different directions, and perform labeling on gesture types, positions, and directions of the target gesture image data to obtain third training data, and perform model training according to the third training data to obtain the gesture recognition model.

[0127] Further, the drivable area division unit 63 includes:

[0128] The information projection unit is configured to project the road feature information into an environmental road network of the semantic map.

[0129] The motion feature extraction unit is configured to extract motion features of each frame in the feature information of the initial traffic intervenor, wherein the motion features include speed, direction, acceleration, and angular acceleration.

[0130] The prediction result generation unit is configured to perform trajectory prediction based on the environmental road network and the motion features by the pedestrian trajectory prediction model to obtain the prediction result.

[0131] The position comparison unit is configured to compare a roadbed position labeled or perceived in the semantic map with a current position of the vehicle to divide the drivable area of the vehicle at present.

[0132] Further, the confidence score unit 64 includes:

[0133] The first judgment result generation unit is configured to judge whether the initial traffic intervenor is in the drivable area based on the prediction result to obtain a first judgment result.

[0134] The second judgment result generation unit is configured to judge whether the initial traffic intervenor is in the drivable area and whether there is the obstacle information behind based on the prediction result to obtain a second judgment result.

[0135] The confidence score result generation unit is configured to perform confidence scoring on the initial traffic intervenor based on the first judgment result and the second judgment result to obtain a confidence score result.

[0136] a target traffic intervenor determination unit configured to determine the target traffic intervenor from the initial traffic intervenors according to the confidence score result.

[0137] Further, the vehicle control unit 65 comprises:

[0138] a time sequence data generation unit configured to record positions of a plurality of gestures of the target traffic intervenor in time sequence to obtain time sequence data of the gestures;

[0139] a traffic instruction recognition unit configured to recognize a target traffic instruction based on the time sequence data of the gestures through a traffic instruction recognition model;

[0140] a first vehicle control unit configured to control the vehicle according to the target traffic instruction and the traffic flow information.

[0141] Further, the first vehicle control unit comprises:

[0142] a first control unit configured to control the vehicle to follow a preceding vehicle in the traffic flow information to pass through if the vehicle is not a leading vehicle and all vehicles in the traffic flow information pass through on one side of the target traffic intervenor;

[0143] a second control unit configured to control the vehicle based on the target traffic instruction if the vehicle is a leading vehicle and no reference vehicle exists in the traffic flow information;

[0144] a third control unit configured to control the vehicle to wait if a red light exists in front of the road and no reference vehicle exists in the traffic flow information.

[0145] In the embodiment of the present application, point cloud data of pedestrians and roads in the vehicle driving process is acquired, and feature information of each pedestrian and road feature information is identified based on the point cloud data, wherein the road feature information includes obstacle information and traffic flow information; traffic interveners are scored according to the feature information to obtain a scoring result, and an initial traffic interveners among the pedestrians is determined according to the scoring result; pedestrian trajectory prediction is performed based on the road feature information and the feature information of the initial traffic interveners to obtain a prediction result, and a current drivable area of the vehicle is divided; the initial traffic interveners is confidence scored according to the prediction result, the drivable area and the obstacle information, and a target traffic interveners among the initial traffic interveners is determined according to the confidence score result; traffic instructions are identified according to the gestures of the target traffic interveners to obtain target traffic instructions, and the vehicle is controlled according to the target traffic instructions and the traffic flow information. Through the extraction of the feature information of the pedestrians and the road feature information, and the combination of the feature information of the pedestrians and the road feature information, the traffic interveners are accurately identified from the multiple pedestrians in the vehicle driving process, then the gestures corresponding to the traffic interveners are directly identified, the interference of the gestures of other pedestrians is avoided, the traffic instructions of the gestures of the traffic interveners can be accurately identified, the vehicle can be precisely controlled based on the traffic instructions, the automatic driving can adapt to complex traffic scenes, and the safety of the automatic driving is improved.

[0146] To solve the above technical problems, the embodiment of the present application also provides an automobile. For details, please refer to Figure 10 , Figure 10 The automobile is a basic structure block diagram of the embodiment.

[0147] The automobile 7 includes a memory 71, a processor 72, and a network interface 73 which are connected to each other through a system bus. It should be noted that only the automobile 7 with three components, the memory 71, the processor 72, and the network interface 73 is shown in the figure, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented. Among them, those skilled in the art can understand that the automobile here is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0148] The memory 71 includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 71 can be an internal storage unit of the vehicle 7, such as a hard disk or a memory of the vehicle 7. In other embodiments, the memory 71 can also be an external storage device of the vehicle 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the vehicle 7. Of course, the memory 71 can also include both the internal storage unit and the external storage device of the vehicle 7. In the present embodiment, the memory 71 is generally used to store an operating system and various application software installed on the vehicle 7, such as program codes of the vehicle control method based on the intervenor gesture recognition, etc. In addition, the memory 71 can also be used to temporarily store various data that have been output or will be output.

[0149] The processor 72 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 72 is generally used to control the overall operation of the vehicle 7. In the present embodiment, the processor 72 is used to run program codes or process data stored in the memory 71, such as running program codes of the vehicle control method based on the intervenor gesture recognition, to implement various embodiments of the vehicle control method based on the intervenor gesture recognition.

[0150] The network interface 73 can include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the vehicle 7 and other electronic devices.

[0151] The present application also provides another embodiment, i.e., to provide a computer readable storage medium storing a computer program, which can be executed by at least one processor to make the at least one processor execute the steps of the vehicle control method based on the intervenor gesture recognition as described above.

[0152] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the method of each embodiment of the present application.

[0153] Obviously, the above-described embodiments are only some of the embodiments of the present application, not all the embodiments, and the drawings give the preferred embodiments of the present application, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing specific embodiments, or make equivalent replacements to some of the technical features. Any equivalent structure made by using the content of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the scope of the patent protection of the present application.

Claims

1. A vehicle control method based on interventioner gesture recognition, characterized in that: include: Acquire point cloud data of pedestrians and roads during vehicle travel, and identify characteristic information of each pedestrian and road characteristic information based on the point cloud data, wherein the road characteristic information includes obstacle information and traffic flow information; performing a traffic intervener score calculation based on the pedestrian characteristic information and the road characteristic information to obtain a score calculation result, and determining an initial traffic intervener among the pedestrians based on the score calculation result; Pedestrian trajectory prediction is performed based on the road feature information and the feature information of the initial traffic intervener to obtain a prediction result, and a current drivable area of ​​the vehicle is divided; performing a confidence score on the initial traffic interveners according to the prediction result, the drivable area, and the obstacle information, and determining a target traffic intervener among the initial traffic interveners according to the confidence score result; Traffic instructions are recognized according to the gesture of the target traffic intervener to obtain target traffic instructions, and the vehicle is controlled according to the target traffic instructions and the traffic flow information.

2. The vehicle control method based on interventioner gesture recognition according to claim 1, characterized in that: The step of acquiring point cloud data of pedestrians and roads during vehicle travel and identifying characteristic information of each pedestrian and road based on the point cloud data includes: receiving point cloud data of pedestrians and roads during the driving of the vehicle through a radar; Performing pedestrian segmentation processing on the point cloud data using a pedestrian recognition model to obtain multiple pedestrian data; Using image recognition technology to identify pedestrian information in the pedestrian data; Performing clothing recognition and gesture recognition based on the pedestrian information using a clothing recognition model and a gesture recognition model, respectively, to obtain recognition results; generating the feature information of each pedestrian based on the pedestrian information and the recognition result; Vehicle and obstacle identification is performed on the point cloud data to obtain the road feature information.

3. The vehicle control method based on interventioner gesture recognition according to claim 2, characterized in that: Before acquiring point cloud data of pedestrians and roads during vehicle travel and identifying characteristic information of each pedestrian and road characteristic information based on the point cloud data, the method further includes: receiving, via a radar, point cloud data during an initial vehicle driving process, clustering and labeling the point cloud data to generate first training data, and performing model training based on the first training data to obtain the pedestrian recognition model; Collecting clothing images of a target group, screening and labeling the clothing images to obtain second training data, and performing model training based on the second training data to obtain the clothing recognition model; Target gesture image data of different orientations are acquired, and the target gesture image data are labeled with gesture type, position, and orientation to obtain third training data. Model training is performed based on the third training data to obtain the gesture recognition model.

4. The vehicle control method based on interventioner gesture recognition according to claim 1, characterized in that: The pedestrian trajectory prediction based on the road feature information and the feature information of the initial traffic intervener is performed to obtain a prediction result, and the current drivable area of ​​the vehicle is divided, including: Projecting the road feature information into the environment road network of the semantic map; Extracting motion features of each frame from the feature information of the initial traffic intervener, wherein the motion features include speed, direction, acceleration, and angular acceleration; Performing trajectory prediction based on the environmental road network and the motion characteristics using a pedestrian trajectory prediction model to obtain the prediction result; The roadbed position marked or sensed in the semantic map is compared with the current position of the vehicle to demarcate the current drivable area of ​​the vehicle.

5. The vehicle control method based on interventioner gesture recognition according to claim 1, characterized in that: The performing confidence scoring on the initial traffic interveners according to the prediction result, the drivable area, and the obstacle information, and determining a target traffic intervener among the initial traffic interveners according to the confidence scoring result, includes: determining whether the initial traffic intervener is within the drivable area based on the prediction result, to obtain a first determination result; determining, based on the prediction result, whether the initial traffic intervener is within the drivable area and has the obstacle information behind it, to obtain a second determination result; Performing a confidence score on the initial traffic intervener based on the first judgment result and the second judgment result to obtain a confidence score result; The target traffic intervener among the initial traffic interveners is determined according to the confidence score result.

6. The vehicle control method based on interventioner gesture recognition according to any one of claims 1 to 5, characterized in that: The identifying a traffic instruction according to the gesture of the target traffic intervener to obtain a target traffic instruction, and controlling the vehicle according to the target traffic instruction and the traffic flow information includes: Recording multiple gesture positions of the target traffic intervener in chronological order to obtain gesture time series data; performing traffic command recognition based on the time series data of the gesture using a traffic command recognition model to obtain a target traffic command; The vehicle is controlled according to the target traffic instruction and the traffic flow information.

7. The vehicle control method based on interventioner gesture recognition according to claim 1, characterized in that: The controlling the vehicle according to the target traffic instruction and the traffic flow information includes: If the vehicle is not the leading vehicle and all the vehicles in the traffic flow information are passing on one side of the target traffic intervener, the vehicle is controlled to follow the leading vehicle in the traffic flow information to pass; If the vehicle is a leading vehicle and there is no reference vehicle in the traffic flow information, controlling the vehicle based on the target traffic instruction; If the traffic light ahead is red and there is no reference vehicle in the traffic flow information, the vehicle is controlled to wait for the other vehicle.

8. A vehicle control device based on interventioner gesture recognition, characterized in that: include: a point cloud data acquisition unit, configured to acquire point cloud data of pedestrians and roads during vehicle travel, and identify characteristic information of each pedestrian and road characteristic information based on the point cloud data, wherein the road characteristic information includes obstacle information and traffic flow information; a traffic intervener determination unit, configured to calculate a traffic intervener score based on the pedestrian characteristic information and the road characteristic information, obtain a score calculation result, and determine an initial traffic intervener among the pedestrians based on the score calculation result; a drivable area division unit, configured to perform pedestrian trajectory prediction based on the road feature information and the feature information of the initial traffic intervener, obtain a prediction result, and divide the current drivable area of ​​the vehicle; a confidence scoring unit, configured to perform a confidence score on the initial traffic intervener based on the prediction result, the drivable area, and the obstacle information, and determine a target traffic intervener among the initial traffic interveners based on the confidence score result; A vehicle control unit is used to identify traffic instructions according to the gesture of the target traffic intervener, obtain target traffic instructions, and control the vehicle according to the target traffic instructions and the traffic flow information.

9. An automobile, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the vehicle control method based on interventioner gesture recognition according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the vehicle control method based on interventioner gesture recognition according to any one of claims 1 to 7.

Images