Method and device for obtaining driving status of obstacle vehicle

Abstract

The present application provides a method and device for training a state recognition model, and a method and device for obtaining the driving state of an obstacle vehicle. The state recognition model in the present application can be an end-to-end model, so that it can be obtained through supervised training of large-scale data, which can improve the generalization ability of the state recognition model, and when identifying the driving state of an obstacle vehicle based on the state recognition model, in addition to using "vehicle information of the self-vehicle traveling on the road" and "obstacle information of the obstacle vehicle in the driving direction of the self-vehicle on the road", "participant information of traffic participants in the area where the self-vehicle is located" and "road topology information of the road" are also used, so that the state recognition model can learn more information related to the self-vehicle and obstacles, so as to assist in identifying the driving state of the obstacle vehicle, and can improve the accuracy of identifying the driving state of the obstacle vehicle based on the state recognition model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for training a state recognition model, and a method and device for obtaining the driving state of an obstacle vehicle. Background Art

[0002] As the application areas of autonomous vehicles (self-driving vehicles) continue to expand, the driving scenarios that need to be handled are also gradually increasing. For example, the self-driving vehicle is driving on the road, but there is an obstacle vehicle in front of the self-driving vehicle in the driving direction on the road. The obstacle vehicle prevents the self-driving vehicle from continuing to move forward in the driving direction on the road. If the self-driving vehicle continues to move forward in the driving direction on the road, it may collide with the vehicle, affecting traffic safety. In this case, it is necessary to obtain the driving state of the obstacle vehicle, and use the driving state of the obstacle vehicle as a constraint condition to plan the driving situation of the self-driving vehicle.

[0003] It can be seen that it is more important to obtain the driving state of the obstacle vehicle. Among them, the current method is to obtain the driving state of the obstacle vehicle by the on and off of the headlights of the obstacle vehicle.

[0004] However, the inventors have found that the accuracy of the driving state of the obstacle vehicle obtained in the above manner is low. Summary of the invention

[0005] The present application shows a method and device for training a state recognition model, and a method and device for obtaining the driving state of an obstacle vehicle.

[0006] In a first aspect, the present application provides a method for training a state recognition model, the method comprising:

[0007] Acquire at least one sample data set, the sample data set comprising: vehicle information of a sample self-vehicle traveling on a sample road, obstacle information of a sample obstacle vehicle in a traveling direction of the sample self-vehicle on the sample road, participant information of a sample traffic participant in an area where the sample self-vehicle is located, road topology information of the sample road, and annotated driving status of the sample obstacle vehicle;

[0008] Create a network structure for the state recognition model;

[0009] At least one sample data set is used to train network parameters in the network structure of the state recognition model until the network parameters converge, thereby obtaining the state recognition model.

[0010] In an optional implementation, the network structure includes at least a feature extraction network, a feature fusion network and a state recognition network;

[0011] The feature extraction network is used to extract features from the vehicle information of the sample ego vehicle to obtain the vehicle features of the sample ego vehicle, to extract features from the obstacle information of the sample obstacle vehicle and the participant information of the sample traffic participant to obtain mixed features for the sample obstacle vehicle and the sample traffic participant, and to extract features from the road topology information of the sample road to obtain the road features of the sample road;

[0012] The feature fusion network is used to obtain the interactive features of vehicle features, mixed features and road features;

[0013] The state recognition network is used to identify the driving state of the sample obstacle vehicle based on the interaction features.

[0014] In an optional implementation, the feature fusion network includes at least a vehicle attention layer, a road attention layer, a hybrid attention layer, and a splicing layer;

[0015] The road attention layer is used to obtain the road attention features of vehicle features and mixed features to road features, the vehicle attention layer is used to obtain the vehicle attention features of road features and mixed features to vehicle features, the mixed attention layer is used to obtain the mixed attention features of road features and vehicle features to mixed features, and the splicing layer is used to splice the road attention features, vehicle attention features and mixed attention features to obtain interactive features.

[0016] In a second aspect, the present application shows a method for obtaining the driving state of an obstacle vehicle, which is applied to a self-vehicle, and the method includes:

[0017] When the vehicle is traveling on a road, identifying whether there is an obstacle vehicle on the road in the driving direction of the vehicle;

[0018] In the case where the obstacle vehicle exists, obtaining vehicle information of the own vehicle, obstacle information of the obstacle vehicle, participant information of traffic participants in the area where the own vehicle is located, and road topology information of the road;

[0019] The driving state of the obstacle vehicle is acquired according to the vehicle information of the own vehicle, the obstacle information of the obstacle vehicle, the participant information of the traffic participants in the area where the own vehicle is located, and the road topology information of the road.

[0020] In an optional implementation, the acquiring the driving state of the obstacle vehicle according to the vehicle information of the own vehicle, the obstacle information of the obstacle vehicle, the participant information of the traffic participants in the area where the own vehicle is located, and the road topology information of the road includes:

[0021] Feature extraction is performed on the vehicle information of the ego vehicle to obtain the vehicle features of the ego vehicle; feature extraction is performed on the obstacle information of the obstacle vehicle and the participant information of the traffic participant to obtain mixed features for the obstacle vehicle and the traffic participant; and feature extraction is performed on the road topology information of the road to obtain the road features of the road;

[0022] Obtain interactive features of vehicle features, mixed features, and road features;

[0023] Identify the driving status of the obstacle vehicle based on the interaction features.

[0024] In an optional implementation, the step of obtaining interactive features under the combined effects of vehicle features, mixed features, and road features includes:

[0025] Obtaining road attention features of vehicle features and mixed features for road features, obtaining vehicle attention features of road features and mixed features for vehicle features, and obtaining mixed attention features of road features and vehicle features for mixed features;

[0026] The road attention features, vehicle attention features and mixed attention features are concatenated to obtain interactive features.

[0027] In an optional implementation, the method further includes:

[0028] When the state of the obstacle vehicle is not a parking state, the self-vehicle is stopped, and the self-vehicle is controlled to drive along the original path after the obstacle vehicle leaves;

[0029] or,

[0030] When the obstacle vehicle is in a parked state, a path is planned for bypassing the obstacle vehicle, and the self-vehicle is automatically controlled to travel based on the planned path; or, the planned path is displayed so that the driver of the self-vehicle manually controls the self-vehicle to travel based on the planned path.

[0031] In an optional implementation, the method further includes:

[0032] Generating driving prompt information on the road according to the driving state of the obstacle vehicle;

[0033] The driving prompt information is sent to the traffic participant so that the traffic participant can control driving according to the driving prompt information.

[0034] In a third aspect, the present application shows a device for training a state recognition model, the device comprising:

[0035] A first acquisition module is used to acquire at least one sample data set, wherein the sample data set includes: vehicle information of a sample self-vehicle traveling on a sample road, obstacle information of a sample obstacle vehicle in a traveling direction of the sample self-vehicle on the sample road, participant information of a sample traffic participant in an area where the sample self-vehicle is located, road topology information of the sample road, and annotated driving status of the sample obstacle vehicle;

[0036] Create a module for creating a network structure of a state recognition model;

[0037] The using module is used to train the network parameters in the network structure of the state recognition model using at least one sample data set until the network parameters converge, thereby obtaining the state recognition model.

[0038] In an optional implementation, the network structure includes at least a feature extraction network, a feature fusion network and a state recognition network;

[0039] The feature extraction network is used to extract features from the vehicle information of the sample ego vehicle to obtain the vehicle features of the sample ego vehicle, to extract features from the obstacle information of the sample obstacle vehicle and the participant information of the sample traffic participant to obtain mixed features for the sample obstacle vehicle and the sample traffic participant, and to extract features from the road topology information of the sample road to obtain the road features of the sample road;

[0040] The feature fusion network is used to obtain the interactive features of vehicle features, mixed features and road features;

[0041] The state recognition network is used to identify the driving state of the sample obstacle vehicle based on the interaction features.

[0042] In an optional implementation, the feature fusion network includes at least a vehicle attention layer, a road attention layer, a hybrid attention layer, and a splicing layer;

[0043] The road attention layer is used to obtain the road attention features of vehicle features and mixed features to road features, the vehicle attention layer is used to obtain the vehicle attention features of road features and mixed features to vehicle features, the mixed attention layer is used to obtain the mixed attention features of road features and vehicle features to mixed features, and the splicing layer is used to splice the road attention features, vehicle attention features and mixed attention features to obtain interactive features.

[0044] In a fourth aspect, the present application shows a device for obtaining the driving state of an obstacle vehicle, which is applied to a self-vehicle, and the device includes:

[0045] An identification module, used for identifying whether there is an obstacle vehicle in the driving direction of the vehicle on the road when the vehicle is driving on the road;

[0046] A second acquisition module is used to acquire vehicle information of the own vehicle, obstacle information of the obstacle vehicle, participant information of traffic participants in the area where the own vehicle is located, and road topology information of the road when the obstacle vehicle exists;

[0047] The third acquisition module is used to acquire the driving status of the obstacle vehicle based on the vehicle information of the own vehicle, the obstacle information of the obstacle vehicle, the participant information of the traffic participants in the area where the own vehicle is located, and the road topology information of the road.

[0048] In an optional implementation, the second acquisition module includes:

[0049] An extraction unit is used to extract features from the vehicle information of the own vehicle to obtain the vehicle features of the own vehicle; extract features from the obstacle information of the obstacle vehicle and the participant information of the traffic participant to obtain mixed features for the obstacle vehicle and the traffic participant; and extract features from the road topology information of the road to obtain the road features of the road;

[0050] An acquisition unit, used for acquiring interactive features under the joint action of vehicle features, mixed features and road features;

[0051] The recognition unit is used to recognize the driving state of the obstacle vehicle according to the interaction characteristics.

[0052] In an optional implementation, the acquiring unit includes:

[0053] An acquisition subunit, used to acquire road attention features of vehicle features and mixed features to road features, acquire vehicle attention features of road features and mixed features to vehicle features, and acquire mixed attention features of road features and vehicle features to mixed features;

[0054] The splicing subunit is used to splice the road attention features, vehicle attention features and mixed attention features to obtain interactive features.

[0055] In an optional implementation, the device further includes:

[0056] The first driving module is used to stop the driving of the self-vehicle when the state of the obstacle vehicle is not the parking state, and wait for the obstacle vehicle to drive away before controlling the self-vehicle to drive based on the original path;

[0057] or,

[0058] The second driving module is used to plan a path for bypassing the obstacle vehicle when the obstacle vehicle is in a parked state, and automatically control the self-vehicle to drive based on the planned path; or, the display module is used to display the planned path so that the driver of the self-vehicle can manually control the self-vehicle to drive based on the planned path.

[0059] In an optional implementation, the device further includes:

[0060] A generating module, used for generating driving prompt information on the road according to the driving state of the obstacle vehicle;

[0061] The sending module is used to send driving prompt information to the traffic participants so that the traffic participants can control their driving according to the driving prompt information.

[0062] In a third aspect, the present application shows an electronic device, the electronic device comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the method shown in any of the aforementioned aspects.

[0063] In a fourth aspect, the present application shows a non-temporary computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform a method as shown in any of the aforementioned aspects.

[0064] In a fifth aspect, the present application illustrates a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to perform a method as shown in any of the aforementioned aspects.

[0065] Compared with the prior art, this application has the following advantages:

[0066] Among them, the advantages of the first and third aspects are:

[0067] In the present application, at least one sample data set is obtained, and the sample data set includes: vehicle information of a sample self-vehicle traveling on a sample road, obstacle information of a sample obstacle vehicle in the driving direction of the sample self-vehicle on the sample road, participant information of a sample traffic participant in the area where the sample self-vehicle is located, road topology information of the sample road, and annotated driving status of the sample obstacle vehicle. A network structure of a state recognition model is created. The network parameters in the network structure of the state recognition model are trained using at least one sample data set until the network parameters converge, thereby obtaining a state recognition model.

[0068] Among them, the state recognition model in the present application can be an end-to-end model, which can support supervised training of large-scale data, thereby improving the generalization ability of the trained state recognition model, and further improving the accuracy of identifying the driving state of obstacle vehicles based on the state recognition model.

[0069] In addition, when training the state recognition model, more data is used. For example, in addition to using "vehicle information of the sample self-vehicle traveling on the sample road" and "obstacle information of the sample obstacle vehicle in the driving direction of the sample self-vehicle on the sample road", environmental information is also used, for example, including "participant information of the sample traffic participants in the area where the sample self-vehicle is located" and "road topology information of the sample road". In this way, the state recognition model can learn more information related to the self-vehicle and the obstacle. More information can be used to assist in identifying the driving state of the obstacle vehicle, thereby improving the accuracy of identifying the driving state of the obstacle vehicle based on the state recognition model.

[0070] For example, based on the state recognition model, the road attention features of "vehicle characteristics of the sample self-vehicle traveling on the sample road" and "mixed characteristics of sample obstacle vehicles in the traveling direction of the sample self-vehicle on the sample road and sample traffic participants in the area where the sample self-vehicle is located" versus "road characteristics of the sample road" can be learned, the vehicle attention features of "road characteristics of the sample road" and "mixed characteristics of sample obstacle vehicles in the traveling direction of the sample self-vehicle on the sample road and sample traffic participants in the area where the sample self-vehicle is located" versus "vehicle characteristics of the sample self-vehicle traveling on the sample road", and the mixed attention features of "road characteristics of the sample road" and "vehicle characteristics of the sample self-vehicle traveling on the sample road" versus "mixed characteristics of sample obstacle vehicles in the traveling direction of the sample self-vehicle on the sample road and sample traffic participants in the area where the sample self-vehicle is located". In order to obtain the influence relationship and correlation relationship among "vehicle characteristics of the sample self-vehicle traveling on the sample road", "mixed characteristics of sample obstacle vehicles in the driving direction of the sample self-vehicle on the sample road and sample traffic participants in the area where the sample self-vehicle is located", and "road characteristics of the sample road", the influence relationship and correlation relationship among "vehicle characteristics of the sample self-vehicle traveling on the sample road", "mixed characteristics of sample obstacle vehicles in the driving direction of the sample self-vehicle on the sample road and sample traffic participants in the area where the sample self-vehicle is located", and "road characteristics of the sample road", etc. can be used to assist in identifying the driving state of the obstacle vehicle, thereby improving the accuracy of identifying the driving state of the obstacle vehicle based on the state recognition model.

[0071] Among them, the advantages of the second and fourth aspects are:

[0072] In the present application, when the ego vehicle is traveling on a road, it is identified whether there is an obstacle vehicle in the driving direction of the ego vehicle on the road. When there is an obstacle vehicle in the driving direction of the ego vehicle on the road, vehicle information of the ego vehicle, obstacle information of the obstacle vehicle, participant information of traffic participants in the area where the ego vehicle is located, and road topology information of the road are obtained. Based on the vehicle information of the ego vehicle, obstacle information of the obstacle vehicle, participant information of traffic participants in the area where the ego vehicle is located, and road topology information of the road, the driving state of the obstacle vehicle is obtained.

[0073] Among them, the state recognition model in the present application can be an end-to-end model. In this way, the state recognition model can be obtained through supervised training of large-scale data, thereby improving the generalization ability of the state recognition model, and further improving the accuracy of identifying the driving state of the obstacle vehicle based on the state recognition model.

[0074] Furthermore, when identifying the driving state of an obstacle vehicle based on a state recognition model, more data is used. For example, in addition to using "vehicle information of the ego vehicle traveling on the road" and "obstacle information of obstacle vehicles in the driving direction of the ego vehicle on the road", environmental information is also used, for example, including "participant information of traffic participants in the area where the ego vehicle is located" and "road topology information of the road". This allows the state recognition model to learn more information related to the ego vehicle and obstacles. More information can be used to assist in identifying the driving state of the obstacle vehicle, thereby improving the accuracy of identifying the driving state of the obstacle vehicle based on the state recognition model.

[0075] For example, based on the state recognition model, the road attention features of "vehicle characteristics of the ego vehicle traveling on the road" and "mixed characteristics for obstacle vehicles in the driving direction of the ego vehicle on the road and traffic participants in the area where the ego vehicle is located" versus "road characteristics of the road" can be obtained, the vehicle attention features of "road characteristics of the road" and "mixed characteristics for obstacle vehicles in the driving direction of the ego vehicle on the road and traffic participants in the area where the ego vehicle is located" versus "vehicle characteristics of the ego vehicle traveling on the road", and the mixed attention features of "road characteristics of the road" and "vehicle characteristics of the ego vehicle traveling on the road" versus "mixed characteristics for obstacle vehicles in the driving direction of the ego vehicle on the road and traffic participants in the area where the ego vehicle is located". In order to obtain the influence relationship and correlation relationship among "vehicle characteristics of the ego vehicle traveling on the road", "mixed characteristics of obstacle vehicles in the driving direction of the ego vehicle on the road and traffic participants in the area where the ego vehicle is located", and "road characteristics of the road", the influence relationship and correlation relationship among "vehicle characteristics of the ego vehicle traveling on the road", "mixed characteristics of obstacle vehicles in the driving direction of the ego vehicle on the road and traffic participants in the area where the ego vehicle is located", and "road characteristics of the road" can be used to assist in identifying the driving state of the obstacle vehicle, thereby improving the accuracy of identifying the driving state of the obstacle vehicle based on the state recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a flowchart of the steps of a method for training a state recognition model of the present application.

[0077] Figure 2 This is a schematic diagram of the state recognition model structure of this application.

[0078] Figure 3 This is a schematic diagram of the state recognition model structure of this application.

[0079] Figure 4 This is a flowchart of the steps of a method for obtaining the driving status of an obstacle vehicle in the present application.

[0080] Figure 5 It is a structural block diagram of a device for training a state recognition model of the present application.

[0081] Figure 6 This is a structural block diagram of a device for obtaining the driving status of an obstacle vehicle in the present application.

[0082] Figure 7 It is a structural block diagram of a device of the present application. DETAILED DESCRIPTION

[0083] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0084] Specifically, see Figure 1 , showing a training state recognition model method of the present application, the method is applied to a self-driving vehicle, the method comprising:

[0085] In step S101, at least one sample data set is obtained, and the sample data set includes: vehicle information of a sample self-vehicle traveling on a sample road, obstacle information of a sample obstacle vehicle in the traveling direction of the sample self-vehicle on the sample road, participant information of sample traffic participants in the area where the sample self-vehicle is located, road topology information of the sample road, and annotated driving status of the sample obstacle vehicle.

[0086] Autonomous vehicles can include driverless cars, human-powered bicycles, electric bicycles, motorcycles, and cars.

[0087] The vehicle information of the self-vehicle may include the position, driving speed, driving acceleration, driving direction and vehicle profile of the self-vehicle.

[0088] The vehicle information of the own vehicle may include the vehicle information of the own vehicle at the current time, and may also include the vehicle information of the own vehicle within a period of time before the current time, etc.

[0089] The obstacle information of the obstacle vehicle may include the position, driving speed, driving acceleration, driving direction and vehicle profile of the obstacle vehicle.

[0090] Obstacle vehicles may include unmanned vehicles, human-powered bicycles, electric bicycles, motorcycles, and cars.

[0091] The vehicle information of the obstacle vehicle may include the obstacle information of the obstacle vehicle at the current moment, and may also include the obstacle information of the obstacle vehicle within a period of time before the current moment, etc.

[0092] Among them, the obstacle vehicle is a concept relative to the self-vehicle.

[0093] The self-vehicle can be the executor of the solution of the present application, and can be an autonomous driving vehicle or a vehicle manually driven by a driver. The self-vehicle can travel on the road. When the self-vehicle travels on the road, it has a driving direction. The self-vehicle moves along the driving direction on the road. However, there is a vehicle in front of it in the driving direction, and the vehicle hinders the self-vehicle from continuing to move forward in the driving direction on the road. If the self-vehicle continues to move forward in the driving direction on the road, it may collide with the vehicle, affecting traffic safety. Therefore, the vehicle can be regarded as an obstacle vehicle.

[0094] The area where the vehicle is located may include the area around the vehicle, for example, the area within a radius of N meters centered on the vehicle, where N may include positive integers such as 3, 5, 7 or 10. The specific area may be determined based on actual conditions and is not limited in this application.

[0095] Traffic participants in the area where the vehicle is located include people, animals, and vehicles.

[0096] The participant information of the traffic participant may include the location, forward speed, forward acceleration, forward direction, and outline of the traffic participant.

[0097] The participant information of the traffic participant may include the participant information of the traffic participant at the current moment, and may also include the participant information of the traffic participant within a period of time before the current moment, etc.

[0098] The road topology information of the road includes: the distribution information of traffic lights on the road, the distribution information of traffic signs, the location information of lane dividing lines on the road, the location information of zebra crossings, and the location information of stop lines.

[0099] The marked driving state of the obstacle vehicle may include: a stopped state and a non-stop state, etc.

[0100] The stopped state means that the obstacle vehicle is stationary at this moment and will continue to be stationary for a long time and keep the same position, for example, the obstacle vehicle is parked in a parking space for a long time.

[0101] In addition, the non-stop state indicates that the obstacle vehicle is moving, or the obstacle vehicle is stationary at this moment but will move in a short time. For example, the obstacle vehicle is currently stationary in the lane and waiting for the red light in the traffic indicator light corresponding to the lane to go out and the green light to turn on. When the red light in the traffic indicator light corresponding to the lane goes out and the green light to turn on, the obstacle vehicle will move. The time required to wait for the red light in the traffic indicator light corresponding to the lane to go out and the green light to turn on is often short, for example, tens of seconds or more than a minute.

[0102] The long time and short time mentioned above can be understood as duration, and the division between long time and short time can be divided according to a preset duration threshold, for example, less than the preset duration threshold is a short time, and greater than or equal to the preset duration threshold is a long time.

[0103] Among them, the preset time threshold may include: 5 seconds, 10 seconds, 20 seconds, 30 seconds, 40 seconds, 60 seconds, 80 seconds, 90 seconds, 120 seconds or 150 seconds, etc., which can be determined according to actual conditions and is not limited in this application.

[0104] In one embodiment of the present application, a sensor is installed on the self-vehicle, and the vehicle information of the self-vehicle can be collected by the sensor of the self-vehicle. For example, the position, driving speed, driving acceleration, driving direction and vehicle profile of the self-vehicle can be obtained by a positioning device built into the self-vehicle, and the positioning device includes a GPS (Global Positioning System, global positioning system) and the like.

[0105] A camera is provided on the vehicle, which is at least used to collect images in the driving direction of the vehicle. In this way, the obstacle information of the obstacle vehicle can be obtained by analyzing "the image collected by the camera provided on the vehicle and used to collect images in the driving direction of the vehicle".

[0106] The participant information of the traffic participants in the area where the ego vehicle is located may be obtained by analyzing “the images captured by the camera provided on the ego vehicle for capturing images in various directions of the ego vehicle”.

[0107] Alternatively, a radar is provided in the ego vehicle, and the participant information of the traffic participants in the area where the ego vehicle is located can be obtained by analyzing data collected by the radar provided in the ego vehicle.

[0108] The road topology information of the road can be obtained by the vehicle by retrieving map data from a third party and analyzing the map data.

[0109] The marked driving status of the obstacle vehicle can be manually marked according to actual conditions, etc.

[0110] In step S102, a network structure of a state recognition model is created.

[0111] In one embodiment of the present application, see Figure 2 ,The network structure of the state recognition model at least includes a feature extraction network, a feature fusion network and a state recognition network.

[0112] The feature extraction network is used to extract features from the vehicle information of the sample ego vehicle to obtain the vehicle features of the sample ego vehicle, to extract features from the obstacle information of the sample obstacle vehicle and the participant information of the sample traffic participants to obtain mixed features for the sample obstacle vehicle and the sample traffic participants, and to extract features from the road topology information of the sample road to obtain the road features of the sample road.

[0113] The feature fusion network is used to obtain the interactive features of vehicle features, mixed features and road features.

[0114] The state recognition network is used to identify the driving state of the sample obstacle vehicle based on the interaction features.

[0115] The input end of the state recognition model includes the input end of the feature extraction network.

[0116] The output of the feature extraction network is connected to the input of the feature fusion network.

[0117] The output of the feature fusion network is connected to the input of the state recognition network.

[0118] The output end of the state recognition model includes the output end of the state recognition network.

[0119] In another embodiment of the present application, see Figure 3 ,The feature fusion network at least includes a vehicle attention layer, a road attention layer, a mixed attention layer, and a splicing layer.

[0120] The road attention layer is used to obtain the road attention features of vehicle features and mixed features to road features.

[0121] The vehicle attention layer is used to obtain the vehicle attention features of road features and hybrid features to vehicle features.

[0122] The mixed attention layer is used to obtain the mixed attention features of road features and vehicle features to mixed features.

[0123] The concatenation layer is used to concatenate road attention features, vehicle attention features, and mixed attention features to obtain interactive features.

[0124] The input of the feature fusion network includes the input of the vehicle attention layer, the input of the road attention layer, and the input of the mixed attention layer.

[0125] The output of the vehicle attention layer, the output of the road attention layer, and the output of the mixed attention layer are respectively connected to the input of the concatenation layer.

[0126] The output of the feature fusion network includes the output of the concatenation layer.

[0127] In one embodiment of the present application, the feature extraction network includes: a convolutional neural network (Convolutional Neural Networks), and the convolutional neural network may include a self-vehicle convolution layer, a mixed convolution layer, and a road convolution layer.

[0128] The ego vehicle convolution layer is used to extract features of the vehicle information of the sample ego vehicle, obtain the vehicle features of the sample ego vehicle, and input the vehicle features of the sample ego vehicle into the vehicle attention layer, the mixed attention layer, and the road attention layer respectively.

[0129] The mixed convolutional layer is used to extract features of the obstacle information of the sample obstacle vehicle and the participant information of the sample traffic participants, obtain mixed features for the sample obstacle vehicle and the sample traffic participants, and input the mixed features into the vehicle attention layer, the mixed attention layer and the road attention layer respectively.

[0130] The road convolution layer is used to extract the road topology information of the sample road, obtain the road features of the sample road, and input the road features into the vehicle attention layer, the mixed attention layer and the road attention layer respectively.

[0131] Vehicle features, mixed features, and road features can all be vectors or matrices of the same dimension.

[0132] The feature fusion network may include a transformer network, etc.

[0133] The state recognition network may include a Softmax or fully connected layer, etc.

[0134] In the present application, the state recognition model finally obtained through training may be applied to different application scenarios later. Thus, in this step, the network structure of the state recognition model may be generated based on actual needs, and the network structure of the state recognition model applicable to different application scenarios may be different.

[0135] In this application, Figure 2 and 3 The network structure of the state recognition model shown is for illustration only and is not intended to limit the scope of protection of the present application.

[0136] In step S103, at least one sample data set is used to train network parameters in the network structure of the state recognition model until the network parameters converge, thereby obtaining a state recognition model.

[0137] In the present application, after the network structure of the state recognition model is created, the network parameters in the state recognition model can be trained according to at least one sample data set.

[0138] During the training process, vehicle information of a sample self-vehicle traveling on a sample road, obstacle information of a sample obstacle vehicle in the traveling direction of the sample self-vehicle on the sample road, participant information of sample traffic participants in the area where the sample self-vehicle is located, and road topology information of the sample road in a sample data set can all be input into the feature extraction network of the state recognition model.

[0139] Among them, the feature extraction network can perform feature extraction on the vehicle information of the sample ego vehicle to obtain the vehicle features of the sample ego vehicle, perform feature extraction on the obstacle information of the sample obstacle vehicle and the participant information of the sample traffic participants to obtain mixed features for the sample obstacle vehicle and the sample traffic participants, and perform feature extraction on the road topology information of the sample road to obtain the road features of the sample road, and then input the vehicle features, mixed features and road features into the feature fusion network.

[0140] The feature fusion network can process vehicle features, mixed features and road features to obtain interactive features under the joint action of vehicle features, mixed features and road features, and then input the interactive features into the state recognition network.

[0141] The state recognition network can identify the driving state of the sample obstacle vehicle according to the interaction features and output the driving state of the sample obstacle vehicle.

[0142] Then, the network parameters in the network structure in the state recognition model can be adjusted with the help of the loss function based on the output driving state of the sample obstacle vehicle and the labeled driving state of the sample obstacle vehicle.

[0143] For each other sample data set, the above operation is performed in the same way until the network parameters in the state recognition model converge, thereby completing the training and obtaining the state recognition model.

[0144] In the present application, at least one sample data set is obtained, and the sample data set includes: vehicle information of a sample self-vehicle traveling on a sample road, obstacle information of a sample obstacle vehicle in the driving direction of the sample self-vehicle on the sample road, participant information of a sample traffic participant in the area where the sample self-vehicle is located, road topology information of the sample road, and annotated driving status of the sample obstacle vehicle. A network structure of a state recognition model is created. The network parameters in the network structure of the state recognition model are trained using at least one sample data set until the network parameters converge, thereby obtaining a state recognition model.

[0145] Among them, the state recognition model in the present application can be an end-to-end model, which can support supervised training of large-scale data, thereby improving the generalization ability of the trained state recognition model, and further improving the accuracy of identifying the driving state of obstacle vehicles based on the state recognition model.

[0146] In addition, when training the state recognition model, more data is used. For example, in addition to using "vehicle information of the sample self-vehicle traveling on the sample road" and "obstacle information of the sample obstacle vehicle in the driving direction of the sample self-vehicle on the sample road", environmental information is also used, for example, including "participant information of the sample traffic participants in the area where the sample self-vehicle is located" and "road topology information of the sample road". In this way, the state recognition model can learn more information related to the self-vehicle and the obstacle. More information can be used to assist in identifying the driving state of the obstacle vehicle, thereby improving the accuracy of identifying the driving state of the obstacle vehicle based on the state recognition model.

[0147] For example, based on the state recognition model, the road attention features of "vehicle characteristics of the sample self-vehicle traveling on the sample road" and "mixed characteristics of sample obstacle vehicles in the traveling direction of the sample self-vehicle on the sample road and sample traffic participants in the area where the sample self-vehicle is located" versus "road characteristics of the sample road" can be learned, the vehicle attention features of "road characteristics of the sample road" and "mixed characteristics of sample obstacle vehicles in the traveling direction of the sample self-vehicle on the sample road and sample traffic participants in the area where the sample self-vehicle is located" versus "vehicle characteristics of the sample self-vehicle traveling on the sample road", and the mixed attention features of "road characteristics of the sample road" and "vehicle characteristics of the sample self-vehicle traveling on the sample road" versus "mixed characteristics of sample obstacle vehicles in the traveling direction of the sample self-vehicle on the sample road and sample traffic participants in the area where the sample self-vehicle is located". In order to obtain the influence relationship and correlation relationship among "vehicle characteristics of the sample self-vehicle traveling on the sample road", "mixed characteristics of sample obstacle vehicles in the driving direction of the sample self-vehicle on the sample road and sample traffic participants in the area where the sample self-vehicle is located", and "road characteristics of the sample road", the influence relationship and correlation relationship among "vehicle characteristics of the sample self-vehicle traveling on the sample road", "mixed characteristics of sample obstacle vehicles in the driving direction of the sample self-vehicle on the sample road and sample traffic participants in the area where the sample self-vehicle is located", and "road characteristics of the sample road", etc. can be used to assist in identifying the driving state of the obstacle vehicle, thereby improving the accuracy of identifying the driving state of the obstacle vehicle based on the state recognition model.

[0148] After the state recognition model is obtained, the state recognition model can be put into use online. For example, the driving state of the obstacle vehicle in the driving direction of the ego vehicle on the road can be identified based on the trained state recognition model, so that the ego vehicle can decide the driving strategy of the ego vehicle according to the driving state of the obstacle vehicle.

[0149] For example, when the self-vehicle is traveling on a road, if there is an obstacle in the driving direction of the self-vehicle on the road, in order to identify the driving state of the obstacle vehicle in the driving direction of the self-vehicle on the road based on the trained state recognition model, the self-vehicle can obtain the vehicle information of the self-vehicle, the obstacle information of the obstacle vehicle, the participant information of the traffic participants in the area where the self-vehicle is located, and the road topology information of the road, and then input the vehicle information of the self-vehicle, the obstacle information of the obstacle vehicle, the participant information of the traffic participants in the area where the self-vehicle is located, and the road topology information of the road into the trained state recognition model, so that the state recognition model processes the vehicle information of the self-vehicle, the obstacle information of the obstacle vehicle, the participant information of the traffic participants in the area where the self-vehicle is located, and the road topology information of the road, obtains the driving state of the obstacle vehicle, and outputs the driving state of the obstacle vehicle.

[0150] Specifically, refer to Figure 4 , showing a method for obtaining the driving state of an obstacle vehicle in the present application, characterized in that the method is applied to the self-vehicle and includes:

[0151] In step S201 , when the own vehicle is traveling on a road, it is identified whether there is an obstacle vehicle on the road in the traveling direction of the own vehicle.

[0152] In the present application, a camera is installed on the vehicle for collecting images, which can collect images in the driving direction of the vehicle.

[0153] In this way, when the self-vehicle is traveling on the road, the self-vehicle can control the camera to collect images in the driving direction of the self-vehicle on the road in real time, and then use image recognition technology to analyze the collected images to determine whether there are any obstacle vehicles in the driving direction of the self-vehicle on the road.

[0154] Among them, the image recognition technology can use currently existing technology, and this application does not limit the specific image recognition technology.

[0155] In step S202, when there is an obstacle vehicle in the driving direction of the own vehicle on the road, vehicle information of the own vehicle, obstacle information of the obstacle vehicle, participant information of traffic participants in the area where the own vehicle is located, and road topology information of the road are obtained.

[0156] In one embodiment of the present application, if there is no obstacle vehicle in the driving direction of the self-vehicle on the road, the self-vehicle can continue to travel along the driving direction on the road.

[0157] Alternatively, in another embodiment of the present application, if there is an obstacle vehicle on the road in the direction of travel of the ego vehicle, if the ego vehicle continues to travel in the direction of travel on the road, the ego vehicle may collide with the obstacle vehicle, affecting traffic safety.

[0158] Therefore, in order to avoid collision between the ego vehicle and the obstacle vehicle and thus avoid affecting traffic safety, the driving state of the obstacle vehicle can be determined, and then the driving strategy of the ego vehicle can be decided according to the driving state of the obstacle vehicle, for example, the ego vehicle can be stopped and controlled to drive based on the original path after the obstacle vehicle leaves. Alternatively, a path for bypassing the obstacle vehicle can be planned, and the ego vehicle can be controlled to drive based on the planned path, etc., so as to avoid collision between the ego vehicle and the obstacle vehicle and thus avoid affecting traffic safety.

[0159] In order to obtain the driving status of the obstacle vehicle, the vehicle information of the own vehicle, the obstacle information of the obstacle vehicle, the participant information of the traffic participants in the area where the own vehicle is located, and the road topology information of the road can be obtained.

[0160] In one embodiment of the present application, a sensor is installed on the self-vehicle, and the vehicle information of the self-vehicle can be collected by the sensor of the self-vehicle. For example, the position, driving speed, driving acceleration, driving direction and vehicle profile of the self-vehicle can be obtained by a positioning device built into the self-vehicle, and the positioning device includes GPS, etc.

[0161] A camera is provided on the vehicle, which is at least used to collect images in the driving direction of the vehicle. In this way, the obstacle information of the obstacle vehicle can be obtained by analyzing "the image collected by the camera provided on the vehicle and used to collect images in the driving direction of the vehicle".

[0162] The participant information of the traffic participants in the area where the ego vehicle is located may be obtained by analyzing “the images captured by the camera provided on the ego vehicle for capturing images in various directions of the ego vehicle”.

[0163] Alternatively, a radar is provided in the ego vehicle, and the participant information of the traffic participants in the area where the ego vehicle is located can be obtained by analyzing data collected by the radar provided in the ego vehicle.

[0164] The road topology information of the road can be obtained by the vehicle by retrieving map data from a third party and analyzing the map data.

[0165] In step S203, the driving state of the obstacle vehicle is acquired based on the vehicle information of the own vehicle, the obstacle information of the obstacle vehicle, the participant information of the traffic participants in the area where the own vehicle is located, and the road topology information of the road.

[0166] In one embodiment of the present application, a state recognition model may be used to process vehicle information of the ego vehicle, obstacle information of the obstacle vehicle, participant information of traffic participants in the area where the ego vehicle is located, and road topology information of the road to obtain the driving state of the obstacle vehicle. The specific process includes:

[0167] 2031. Extract features from the vehicle information of the ego vehicle to obtain vehicle features of the ego vehicle. Extract features from the obstacle information of the obstacle vehicle and the participant information of the traffic participant to obtain mixed features for the obstacle vehicle and the traffic participant, and extract features from the road topology information of the road to obtain road features of the road.

[0168] In the present application, the feature extraction network in the state recognition model can be used to perform feature extraction on the vehicle information of the own vehicle to obtain the vehicle characteristics of the own vehicle, feature extraction on the obstacle information of the obstacle vehicle and the participant information of the traffic participants to obtain mixed features for the obstacle vehicle and the traffic participants, and feature extraction on the road topology information of the road to obtain the road characteristics of the road.

[0169] In one embodiment, the feature extraction network includes: a convolutional neural network, which may include a self-vehicle convolution layer, a mixed convolution layer, and a road convolution layer.

[0170] The ego vehicle convolution layer can be used to extract features of the vehicle information of the ego vehicle to obtain the vehicle features of the ego vehicle.

[0171] A hybrid convolutional layer is used to extract features of obstacle information of obstacle vehicles and participant information of traffic participants to obtain hybrid features for obstacle vehicles and traffic participants.

[0172] The road convolution layer is used to extract the road topology information of the road and obtain the road features of the road.

[0173] Vehicle features, mixed features, and road features can all be vectors or matrices of the same dimension.

[0174] 2032. Obtain interactive features under the combined effects of vehicle features, mixed features, and road features.

[0175] In this application, the interactive features of vehicle features, mixed features and road features can be obtained by means of the feature fusion network in the state recognition model. The feature fusion network includes at least a vehicle attention layer, a road attention layer, a mixed attention layer and a splicing layer.

[0176] Specifically, this step can be implemented through the following process, including:

[0177] 11) Obtain road attention features of vehicle features and mixed features for road features, obtain vehicle attention features of road features and mixed features for vehicle features, and obtain mixed attention features of road features and vehicle features for mixed features.

[0178] For example, the road attention features of vehicle features and hybrid features on road features can be obtained with the help of the road attention layer in the feature fusion network. The vehicle attention features of road features and hybrid features on vehicle features can be obtained with the help of the vehicle attention layer in the feature fusion network. The hybrid attention features of road features and hybrid features on hybrid features can be obtained with the help of the hybrid attention layer in the feature fusion network.

[0179] 12) Concatenate the road attention features, vehicle attention features and mixed attention features to obtain interactive features.

[0180] Then, the road attention features, vehicle attention features, and mixed attention features can be spliced ​​together with the help of the splicing layer in the feature fusion network to obtain interactive features.

[0181] 2033. Identify the driving status of the obstacle vehicle based on the interaction features.

[0182] The state recognition network in the state recognition model can be used to recognize the driving state of the obstacle vehicle according to the interaction characteristics.

[0183] Furthermore, in another embodiment of the present application, after obtaining the driving state of the obstacle vehicle in the driving direction of the self-vehicle on the road, the driving strategy of the self-vehicle can be decided according to the driving state of the obstacle vehicle. For example, when the state of the obstacle vehicle is not a parked state, the self-vehicle is paused, and the self-vehicle is controlled to drive based on the original path after waiting for the obstacle vehicle to leave.

[0184] Alternatively, when the obstacle vehicle is in a stopped state, a path for bypassing the obstacle vehicle is planned, and the self-vehicle is automatically controlled to travel based on the planned path. Alternatively, the planned path is displayed so that the driver of the self-vehicle manually controls the self-vehicle to travel based on the planned path.

[0185] Furthermore, in another embodiment of the present application, driving prompt information on the road can be generated according to the driving state of the obstacle vehicle, and the driving prompt information is sent to traffic participants in the area where the vehicle is located, so that the traffic participants can control driving according to the driving prompt information.

[0186] For example, the ego vehicle belongs to a convoy, which also includes other vehicles. The other vehicles are located on the same road as the ego vehicle, or the other vehicles are located in lanes adjacent to the lane where the ego vehicle is located or other closer lanes.

[0187] Driving prompt information can be sent to other vehicles in the fleet, so that other vehicles in the fleet can perform driving control according to the driving prompt information.

[0188] In the present application, when the ego vehicle is traveling on a road, it is identified whether there is an obstacle vehicle in the driving direction of the ego vehicle on the road. When there is an obstacle vehicle in the driving direction of the ego vehicle on the road, vehicle information of the ego vehicle, obstacle information of the obstacle vehicle, participant information of traffic participants in the area where the ego vehicle is located, and road topology information of the road are obtained. Based on the vehicle information of the ego vehicle, obstacle information of the obstacle vehicle, participant information of traffic participants in the area where the ego vehicle is located, and road topology information of the road, the driving state of the obstacle vehicle is obtained.

[0189] Among them, the state recognition model in the present application can be an end-to-end model. In this way, the state recognition model can be obtained through supervised training of large-scale data, thereby improving the generalization ability of the state recognition model, and further improving the accuracy of identifying the driving state of the obstacle vehicle based on the state recognition model.

[0190] Furthermore, when identifying the driving state of an obstacle vehicle based on a state recognition model, more data is used. For example, in addition to using "vehicle information of the ego vehicle traveling on the road" and "obstacle information of obstacle vehicles in the driving direction of the ego vehicle on the road", environmental information is also used, for example, including "participant information of traffic participants in the area where the ego vehicle is located" and "road topology information of the road". This allows the state recognition model to learn more information related to the ego vehicle and obstacles. More information can be used to assist in identifying the driving state of the obstacle vehicle, thereby improving the accuracy of identifying the driving state of the obstacle vehicle based on the state recognition model.

[0191] For example, based on the state recognition model, the road attention features of "vehicle characteristics of the ego vehicle traveling on the road" and "mixed characteristics for obstacle vehicles in the driving direction of the ego vehicle on the road and traffic participants in the area where the ego vehicle is located" versus "road characteristics of the road" can be obtained, the vehicle attention features of "road characteristics of the road" and "mixed characteristics for obstacle vehicles in the driving direction of the ego vehicle on the road and traffic participants in the area where the ego vehicle is located" versus "vehicle characteristics of the ego vehicle traveling on the road", and the mixed attention features of "road characteristics of the road" and "vehicle characteristics of the ego vehicle traveling on the road" versus "mixed characteristics for obstacle vehicles in the driving direction of the ego vehicle on the road and traffic participants in the area where the ego vehicle is located". In order to obtain the influence relationship and correlation relationship among "vehicle characteristics of the ego vehicle traveling on the road", "mixed characteristics of obstacle vehicles in the driving direction of the ego vehicle on the road and traffic participants in the area where the ego vehicle is located", and "road characteristics of the road", the influence relationship and correlation relationship among "vehicle characteristics of the ego vehicle traveling on the road", "mixed characteristics of obstacle vehicles in the driving direction of the ego vehicle on the road and traffic participants in the area where the ego vehicle is located", and "road characteristics of the road" can be used to assist in identifying the driving state of the obstacle vehicle, thereby improving the accuracy of identifying the driving state of the obstacle vehicle based on the state recognition model.

[0192] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by the present application.

[0193] Reference Figure 5 , shows a device for training a state recognition model of the present application, the device comprising:

[0194] The first acquisition module 11 is used to acquire at least one sample data set, wherein the sample data set includes: vehicle information of a sample self-vehicle traveling on a sample road, obstacle information of a sample obstacle vehicle in a driving direction of the sample self-vehicle on the sample road, participant information of a sample traffic participant in an area where the sample self-vehicle is located, road topology information of the sample road, and annotated driving status of the sample obstacle vehicle;

[0195] A creation module 12 is used to create a network structure of a state recognition model;

[0196] The using module 13 is used to train the network parameters in the network structure of the state recognition model using at least one sample data set until the network parameters converge, thereby obtaining the state recognition model.

[0197] In an optional implementation, the network structure includes at least a feature extraction network, a feature fusion network and a state recognition network;

[0198] The feature extraction network is used to extract features from the vehicle information of the sample ego vehicle to obtain the vehicle features of the sample ego vehicle, to extract features from the obstacle information of the sample obstacle vehicle and the participant information of the sample traffic participant to obtain mixed features for the sample obstacle vehicle and the sample traffic participant, and to extract features from the road topology information of the sample road to obtain the road features of the sample road;

[0199] The feature fusion network is used to obtain the interactive features of vehicle features, mixed features and road features;

[0200] The state recognition network is used to identify the driving state of the sample obstacle vehicle based on the interaction features.

[0201] In an optional implementation, the feature fusion network includes at least a vehicle attention layer, a road attention layer, a hybrid attention layer, and a splicing layer;

[0202] The road attention layer is used to obtain the road attention features of vehicle features and mixed features to road features, the vehicle attention layer is used to obtain the vehicle attention features of road features and mixed features to vehicle features, the mixed attention layer is used to obtain the mixed attention features of road features and vehicle features to mixed features, and the splicing layer is used to splice the road attention features, vehicle attention features and mixed attention features to obtain interactive features.

[0203] In the present application, at least one sample data set is obtained, and the sample data set includes: vehicle information of a sample self-vehicle traveling on a sample road, obstacle information of a sample obstacle vehicle in the driving direction of the sample self-vehicle on the sample road, participant information of a sample traffic participant in the area where the sample self-vehicle is located, road topology information of the sample road, and annotated driving status of the sample obstacle vehicle. A network structure of a state recognition model is created. The network parameters in the network structure of the state recognition model are trained using at least one sample data set until the network parameters converge, thereby obtaining a state recognition model.

[0204] Among them, the state recognition model in the present application can be an end-to-end model, which can support supervised training of large-scale data, thereby improving the generalization ability of the trained state recognition model, and further improving the accuracy of identifying the driving state of obstacle vehicles based on the state recognition model.

[0205] In addition, when training the state recognition model, more data is used. For example, in addition to using "vehicle information of the sample self-vehicle traveling on the sample road" and "obstacle information of the sample obstacle vehicle in the driving direction of the sample self-vehicle on the sample road", environmental information is also used, for example, including "participant information of the sample traffic participants in the area where the sample self-vehicle is located" and "road topology information of the sample road". In this way, the state recognition model can learn more information related to the self-vehicle and the obstacle. More information can be used to assist in identifying the driving state of the obstacle vehicle, thereby improving the accuracy of identifying the driving state of the obstacle vehicle based on the state recognition model.

[0206] For example, based on the state recognition model, the road attention features of "vehicle characteristics of the sample self-vehicle traveling on the sample road" and "mixed characteristics of sample obstacle vehicles in the traveling direction of the sample self-vehicle on the sample road and sample traffic participants in the area where the sample self-vehicle is located" versus "road characteristics of the sample road" can be learned, the vehicle attention features of "road characteristics of the sample road" and "mixed characteristics of sample obstacle vehicles in the traveling direction of the sample self-vehicle on the sample road and sample traffic participants in the area where the sample self-vehicle is located" versus "vehicle characteristics of the sample self-vehicle traveling on the sample road", and the mixed attention features of "road characteristics of the sample road" and "vehicle characteristics of the sample self-vehicle traveling on the sample road" versus "mixed characteristics of sample obstacle vehicles in the traveling direction of the sample self-vehicle on the sample road and sample traffic participants in the area where the sample self-vehicle is located". In order to obtain the influence relationship and correlation relationship among "vehicle characteristics of the sample self-vehicle traveling on the sample road", "mixed characteristics of sample obstacle vehicles in the driving direction of the sample self-vehicle on the sample road and sample traffic participants in the area where the sample self-vehicle is located", and "road characteristics of the sample road", the influence relationship and correlation relationship among "vehicle characteristics of the sample self-vehicle traveling on the sample road", "mixed characteristics of sample obstacle vehicles in the driving direction of the sample self-vehicle on the sample road and sample traffic participants in the area where the sample self-vehicle is located", and "road characteristics of the sample road", etc. can be used to assist in identifying the driving state of the obstacle vehicle, thereby improving the accuracy of identifying the driving state of the obstacle vehicle based on the state recognition model.

[0207] Reference Figure 6 , shows a device for obtaining the driving state of an obstacle vehicle of the present application, which is applied to a self-vehicle, and the device includes:

[0208] The identification module 21 is used to identify whether there is an obstacle vehicle in the driving direction of the vehicle on the road when the vehicle is driving on the road;

[0209] A second acquisition module 22 is used to acquire vehicle information of the own vehicle, obstacle information of the obstacle vehicle, participant information of traffic participants in the area where the own vehicle is located, and road topology information of the road when the obstacle vehicle exists;

[0210] The third acquisition module 23 is used to acquire the driving state of the obstacle vehicle according to the vehicle information of the own vehicle, the obstacle information of the obstacle vehicle, the participant information of the traffic participants in the area where the own vehicle is located, and the road topology information of the road.

[0211] In an optional implementation, the second acquisition module includes:

[0212] An extraction unit is used to extract features from the vehicle information of the own vehicle to obtain the vehicle features of the own vehicle; extract features from the obstacle information of the obstacle vehicle and the participant information of the traffic participant to obtain mixed features for the obstacle vehicle and the traffic participant; and extract features from the road topology information of the road to obtain the road features of the road;

[0213] An acquisition unit, used for acquiring interactive features under the joint action of vehicle features, mixed features and road features;

[0214] The recognition unit is used to recognize the driving state of the obstacle vehicle according to the interaction characteristics.

[0215] In an optional implementation, the acquiring unit includes:

[0216] An acquisition subunit, used to acquire road attention features of vehicle features and mixed features to road features, acquire vehicle attention features of road features and mixed features to vehicle features, and acquire mixed attention features of road features and vehicle features to mixed features;

[0217] The splicing subunit is used to splice the road attention features, vehicle attention features and mixed attention features to obtain interactive features.

[0218] In an optional implementation, the device further includes:

[0219] The first driving module is used to stop the driving of the self-vehicle when the state of the obstacle vehicle is not the parking state, and wait for the obstacle vehicle to drive away before controlling the self-vehicle to drive based on the original path;

[0220] or,

[0221] The second driving module is used to plan a path for bypassing the obstacle vehicle when the obstacle vehicle is in a parked state, and automatically control the self-vehicle to drive based on the planned path; or, the display module is used to display the planned path so that the driver of the self-vehicle can manually control the self-vehicle to drive based on the planned path.

[0222] In an optional implementation, the device further includes:

[0223] A generating module, used for generating driving prompt information on the road according to the driving state of the obstacle vehicle;

[0224] The sending module is used to send driving prompt information to the traffic participants so that the traffic participants can control their driving according to the driving prompt information.

[0225] In the present application, when the ego vehicle is traveling on a road, it is identified whether there is an obstacle vehicle in the driving direction of the ego vehicle on the road. When there is an obstacle vehicle in the driving direction of the ego vehicle on the road, vehicle information of the ego vehicle, obstacle information of the obstacle vehicle, participant information of traffic participants in the area where the ego vehicle is located, and road topology information of the road are obtained. Based on the vehicle information of the ego vehicle, obstacle information of the obstacle vehicle, participant information of traffic participants in the area where the ego vehicle is located, and road topology information of the road, the driving state of the obstacle vehicle is obtained.

[0226] Among them, the state recognition model in the present application can be an end-to-end model. In this way, the state recognition model can be obtained through supervised training of large-scale data, thereby improving the generalization ability of the state recognition model, and further improving the accuracy of identifying the driving state of the obstacle vehicle based on the state recognition model.

[0227] Furthermore, when identifying the driving state of an obstacle vehicle based on a state recognition model, more data is used. For example, in addition to using "vehicle information of the ego vehicle traveling on the road" and "obstacle information of obstacle vehicles in the driving direction of the ego vehicle on the road", environmental information is also used, for example, including "participant information of traffic participants in the area where the ego vehicle is located" and "road topology information of the road". This allows the state recognition model to learn more information related to the ego vehicle and obstacles. More information can be used to assist in identifying the driving state of the obstacle vehicle, thereby improving the accuracy of identifying the driving state of the obstacle vehicle based on the state recognition model.

[0228] For example, based on the state recognition model, the road attention features of "vehicle characteristics of the ego vehicle traveling on the road" and "mixed characteristics for obstacle vehicles in the driving direction of the ego vehicle on the road and traffic participants in the area where the ego vehicle is located" versus "road characteristics of the road" can be obtained, the vehicle attention features of "road characteristics of the road" and "mixed characteristics for obstacle vehicles in the driving direction of the ego vehicle on the road and traffic participants in the area where the ego vehicle is located" versus "vehicle characteristics of the ego vehicle traveling on the road", and the mixed attention features of "road characteristics of the road" and "vehicle characteristics of the ego vehicle traveling on the road" versus "mixed characteristics for obstacle vehicles in the driving direction of the ego vehicle on the road and traffic participants in the area where the ego vehicle is located". In order to obtain the influence relationship and correlation relationship among "vehicle characteristics of the ego vehicle traveling on the road", "mixed characteristics of obstacle vehicles in the driving direction of the ego vehicle on the road and traffic participants in the area where the ego vehicle is located", and "road characteristics of the road", the influence relationship and correlation relationship among "vehicle characteristics of the ego vehicle traveling on the road", "mixed characteristics of obstacle vehicles in the driving direction of the ego vehicle on the road and traffic participants in the area where the ego vehicle is located", and "road characteristics of the road" can be used to assist in identifying the driving state of the obstacle vehicle, thereby improving the accuracy of identifying the driving state of the obstacle vehicle based on the state recognition model.

[0229] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0230] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, the electronic device executes one or more methods in the above embodiments. In the present application embodiment, the electronic device includes a server, a gateway, a sub-device, etc., and the sub-device is an Internet of Things device or other device.

[0231] The embodiments of the present disclosure may be implemented as an apparatus configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the apparatus may include electronic devices such as servers (clusters), terminal devices such as IoT devices, and the like.

[0232] Figure 7 An exemplary device 1300 that can be used to implement various embodiments of the present application is schematically shown.

[0233] For one embodiment, Figure 7An exemplary apparatus 1300 is shown having one or more processors 1302, a control module (chip set) 1304 coupled to at least one of the (one or more) processors 1302, a memory 1306 coupled to the control module 1304, a non-volatile memory (NVM) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1304.

[0234] The processor 1302 may include one or more single-core or multi-core processors, and the processor 1302 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 1300 can be used as a server device such as a gateway in the embodiments of the present application.

[0235] In some embodiments, the device 1300 may include one or more computer-readable media (e.g., memory 1306 or NVM / storage device 1308) having instructions 1314 and one or more processors 1302 configured to execute the instructions 1314 in combination with the one or more computer-readable media to implement a module to perform the actions of the present disclosure.

[0236] For one embodiment, the control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1302 and / or any suitable device or component in communication with the control module 1304 .

[0237] The control module 1304 may include a memory controller module to provide an interface to the memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0238] The memory 1306 may be used, for example, to load and store data and / or instructions 1314 for the device 1300. For one embodiment, the memory 1306 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 1306 may include a double data rate quad synchronous dynamic random access memory (DDR4 SDRAM).

[0239] For one embodiment, control module 1304 may include one or more input / output controllers to provide an interface to NVM / storage device 1308 and input / output device(s) 1310 .

[0240] For example, NVM / storage 1308 may be used to store data and / or instructions 1314. NVM / storage 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0241] NVM / storage device 1308 may include storage resources that are physically part of the device on which apparatus 1300 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 1308 may be accessed via input / output device(s) 1310 over a network.

[0242] (One or more) input / output devices 1310 may provide an interface for the apparatus 1300 to communicate with any other appropriate device, and the input / output device 1310 may include a communication component, a phonetic component, a sensor component, etc. The network interface 1312 may provide an interface for the apparatus 1300 to communicate through one or more networks, and the apparatus 1300 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0243] For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers of the control module 1304 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304 to form a system-on-chip (SoC).

[0244] In various embodiments, the device 1300 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the device 1300 may have more or fewer components and / or a different architecture. For example, in some embodiments, the device 1300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0245] An embodiment of the present application provides an electronic device, comprising: one or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, enable the electronic device to perform one or more methods in the present application.

[0246] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0247] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0248] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of the processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable information processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable information processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0249] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable information processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0250] These computer program instructions can also be loaded onto a computer or other programmable information processing terminal device, so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0251] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0252] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0253] The above is a detailed introduction to a method and device for training a state recognition model and a method and device for obtaining the driving state of an obstacle vehicle provided in the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for training a state recognition model, characterized in that: The method comprises: Acquire at least one sample data set, the sample data set comprising: vehicle information of a sample self-vehicle traveling on a sample road, obstacle information of a sample obstacle vehicle in a traveling direction of the sample self-vehicle on the sample road, participant information of a sample traffic participant in an area where the sample self-vehicle is located, road topology information of the sample road, and annotated driving status of the sample obstacle vehicle; Create a network structure for the state recognition model; Using at least one sample data set to train network parameters in a network structure of a state recognition model until the network parameters converge, thereby obtaining a state recognition model; The network structure at least includes a feature extraction network, a feature fusion network and a state recognition network; The feature extraction network is used to extract features from the vehicle information of the sample ego vehicle to obtain the vehicle features of the sample ego vehicle, to extract features from the obstacle information of the sample obstacle vehicle and the participant information of the sample traffic participant to obtain mixed features for the sample obstacle vehicle and the sample traffic participant, and to extract features from the road topology information of the sample road to obtain the road features of the sample road; The feature fusion network is used to obtain the interactive features of vehicle features, mixed features and road features; The state recognition network is used to identify the driving state of the sample obstacle vehicle based on the interaction features.

2. The method according to claim 1, characterized in that The feature fusion network includes at least a vehicle attention layer, a road attention layer, a mixed attention layer, and a splicing layer; The road attention layer is used to obtain the road attention features of vehicle features and mixed features to road features, the vehicle attention layer is used to obtain the vehicle attention features of road features and mixed features to vehicle features, the mixed attention layer is used to obtain the mixed attention features of road features and vehicle features to mixed features, and the splicing layer is used to splice the road attention features, vehicle attention features and mixed attention features to obtain interactive features.

3. A method for obtaining the driving state of an obstacle vehicle, characterized in that: Applied to a self-vehicle, the method comprises: When the vehicle is traveling on a road, identifying whether there is an obstacle vehicle on the road in the driving direction of the vehicle; In the case where the obstacle vehicle exists, obtaining vehicle information of the own vehicle, obstacle information of the obstacle vehicle, participant information of traffic participants in the area where the own vehicle is located, and road topology information of the road; Acquire the driving state of the obstacle vehicle according to the vehicle information of the own vehicle, the obstacle information of the obstacle vehicle, the participant information of the traffic participants in the area where the own vehicle is located, and the road topology information of the road; The obtaining of the driving state of the obstacle vehicle according to the vehicle information of the own vehicle, the obstacle information of the obstacle vehicle, the participant information of the traffic participants in the area where the own vehicle is located, and the road topology information of the road includes: Feature extraction is performed on the vehicle information of the ego vehicle to obtain the vehicle features of the ego vehicle; feature extraction is performed on the obstacle information of the obstacle vehicle and the participant information of the traffic participant to obtain mixed features for the obstacle vehicle and the traffic participant; and feature extraction is performed on the road topology information of the road to obtain the road features of the road; Obtain interactive features of vehicle features, mixed features, and road features; Identify the driving status of the obstacle vehicle based on the interactive features; The obtaining of interactive features under the combined effects of vehicle features, mixed features and road features includes: With the help of the feature fusion network in the state recognition model, the interactive features of vehicle features, mixed features and road features are obtained.

4. The method according to claim 3, characterized in that The obtaining of interactive features under the combined effects of vehicle features, mixed features and road features includes: Obtaining road attention features of vehicle features and mixed features for road features, obtaining vehicle attention features of road features and mixed features for vehicle features, and obtaining mixed attention features of road features and vehicle features for mixed features; The road attention features, vehicle attention features and mixed attention features are concatenated to obtain interactive features.

5. The method according to claim 3, characterized in that: The method further comprises: When the state of the obstacle vehicle is not a parking state, the self-vehicle is stopped, and the self-vehicle is controlled to drive along the original path after the obstacle vehicle leaves; or, When the obstacle vehicle is in a parked state, a path is planned for bypassing the obstacle vehicle, and the self-vehicle is automatically controlled to travel based on the planned path; or, the planned path is displayed so that the driver of the self-vehicle manually controls the self-vehicle to travel based on the planned path.

6. The method according to claim 3, characterized in that The method further comprises: Generating driving prompt information on the road according to the driving state of the obstacle vehicle; The driving prompt information is sent to the traffic participant so that the traffic participant can control driving according to the driving prompt information.

7. A device for training a state recognition model, characterized in that: The device comprises: A first acquisition module is used to acquire at least one sample data set, wherein the sample data set includes: vehicle information of a sample self-vehicle traveling on a sample road, obstacle information of a sample obstacle vehicle in a traveling direction of the sample self-vehicle on the sample road, participant information of a sample traffic participant in an area where the sample self-vehicle is located, road topology information of the sample road, and annotated driving status of the sample obstacle vehicle; Create a module for creating a network structure of a state recognition model; A using module is used to train the network parameters in the network structure of the state recognition model using at least one sample data set until the network parameters converge, thereby obtaining the state recognition model; The network structure at least includes a feature extraction network, a feature fusion network and a state recognition network; The feature extraction network is used to extract features from the vehicle information of the sample ego vehicle to obtain the vehicle features of the sample ego vehicle, to extract features from the obstacle information of the sample obstacle vehicle and the participant information of the sample traffic participant to obtain mixed features for the sample obstacle vehicle and the sample traffic participant, and to extract features from the road topology information of the sample road to obtain the road features of the sample road; The feature fusion network is used to obtain the interactive features of vehicle features, mixed features and road features; The state recognition network is used to identify the driving state of the sample obstacle vehicle based on the interaction features.

8. A device for obtaining the driving state of an obstacle vehicle, characterized in that: Applied to a vehicle, the device comprises: An identification module, used for identifying whether there is an obstacle vehicle in the driving direction of the vehicle on the road when the vehicle is driving on the road; A second acquisition module is used to acquire vehicle information of the own vehicle, obstacle information of the obstacle vehicle, participant information of traffic participants in the area where the own vehicle is located, and road topology information of the road when the obstacle vehicle exists; a third acquisition module, configured to acquire the driving state of the obstacle vehicle according to the vehicle information of the own vehicle, the obstacle information of the obstacle vehicle, the participant information of the traffic participants in the area where the own vehicle is located, and the road topology information of the road; The second acquisition module includes: An extraction unit is used to extract features from the vehicle information of the own vehicle to obtain the vehicle features of the own vehicle; extract features from the obstacle information of the obstacle vehicle and the participant information of the traffic participant to obtain mixed features for the obstacle vehicle and the traffic participant; and extract features from the road topology information of the road to obtain the road features of the road; An acquisition unit, used for acquiring interactive features under the combined effects of vehicle features, mixed features and road features; An identification unit, used to identify the driving state of the obstacle vehicle according to the interaction characteristics; The acquisition unit is specifically used for: acquiring the interactive features under the joint action of vehicle features, mixed features and road features by means of the feature fusion network in the state recognition model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images