Vehicle scene warning decision method, device, computer equipment and storage medium

Abstract

The present application relates to a vehicle scene warning decision method, device, computer equipment and storage medium. The method includes: obtaining a vehicle condition data sequence and a scene image in the current driving scene; inputting the scene image into a scene feature extraction network of a trained scene decision model to extract the spatial features of the scene image; inputting the vehicle condition data sequence into a vehicle condition prediction network of a trained scene decision model to obtain a predicted vehicle motion sequence; fusing the scene features and the vehicle motion sequence, and obtaining the warning information of the vehicle in the current driving scene through the decision maker of the scene decision model. This method can jointly predict the warning information of the vehicle in the current driving scene based on the spatial features of the current scene and the time-related vehicle motion features, has a certain degree of foresight, and can improve the accuracy of the warning.

Description

Technical Field

[0001] The present application relates to the field of automobile intelligent technology, and in particular to a vehicle scene warning decision method, device, computer equipment and storage medium. Background Art

[0002] At present, more and more vehicles are equipped with vehicle warning systems. In the face of complex vehicle operation scenarios, the system can also calmly predict and identify information such as vehicle status, vehicle surroundings, vehicle trajectory, collision accidents, etc. However, in extremely complex traffic scenarios or situations with high driving risks, the current vehicle warning system does not have real-time information and there is a certain delay in the transmission of information, resulting in the vehicle warning system not having "preemptive" nature. Summary of the invention

[0003] Based on this, it is necessary to provide a vehicle scene warning decision method, device, computer equipment and storage medium to improve the accuracy of warning in response to the above technical problems.

[0004] In a first aspect, the present application provides a vehicle scene warning decision method, the method comprising:

[0005] Acquire a vehicle condition data sequence and a scene image of a preceding driving scene;

[0006] The scene image is input into the scene feature extraction network of the trained scene decision model to extract the spatial features of the scene image;

[0007] Input the vehicle condition data sequence into the vehicle condition prediction network of the trained scenario decision model to obtain the predicted vehicle motion sequence;

[0008] The scene features and vehicle motion sequences are integrated, and the warning information of the vehicle in the current driving scene is obtained through the decision maker of the scene decision model.

[0009] In one embodiment, a scene image is input into a scene feature extraction network of a trained scene decision model to extract spatial features of the scene image, including:

[0010] The scene image is input into the scene feature extraction network of the trained scene decision model, and the scene features of the scene image are extracted through the feature extraction structure of the scene feature extraction network;

[0011] The scene image and scene features are fused as the input of the attention structure of the scene feature extraction network, and the attention weight is output through the attention structure;

[0012] According to the scene features and attention weights, the spatial features of the scene image are obtained.

[0013] In one embodiment, the vehicle condition data sequence is input into the vehicle condition prediction network of the trained scenario decision model to obtain the predicted vehicle motion sequence, including:

[0014] The vehicle condition data sequence is input into the vehicle condition prediction network of the trained scenario decision model. The encoding structure of the vehicle condition prediction network extracts the high-level information of the vehicle condition data sequence and generates a feature vector.

[0015] The feature vector is input into the decoding structure of the vehicle condition prediction network. The decoding structure decodes the encoded information with the feature vector as the initial state to obtain the predicted vehicle motion sequence.

[0016] In one embodiment, the scene features and the vehicle motion sequence are integrated, and the warning information of the vehicle in the current driving scene is obtained through the decision maker of the scene decision model, including:

[0017] The scene features and vehicle motion sequence are integrated to obtain the prediction result of whether there are safety hazards in the current driving scene through the decision maker of the scene decision model;

[0018] When the prediction result is that there is a safety hazard in the current driving scenario, a warning message is displayed on the vehicle's human-computer interaction interface, and the warning information includes the type of safety hazard and / or the method for dealing with the safety hazard.

[0019] In one embodiment, the method further comprises:

[0020] Acquire training samples, each training sample including a scene training sample, a first spatial feature of the labeled scene training sample, a vehicle condition training sample, a first vehicle motion sequence of the labeled vehicle condition training sample, and first warning information corresponding to the labeled scene training sample and the vehicle condition training sample;

[0021] Input the scene training sample into the scene feature extraction network to obtain the predicted second spatial feature, and adjust the parameters of the scene feature extraction network according to the difference between the first spatial feature and the second spatial feature;

[0022] Inputting the vehicle condition training sample into the vehicle condition prediction network to obtain a predicted second vehicle motion sequence, and adjusting the parameters of the vehicle condition prediction network according to the difference between the second vehicle motion sequence and the first vehicle motion sequence;

[0023] The second spatial feature and the second vehicle motion sequence are used as training inputs of the decision maker, the second warning information output by the decision maker is obtained, and the parameters of the decision maker are adjusted according to the difference between the first warning information and the second warning information, thereby completing one training;

[0024] The training process is iterated multiple times. When the loss function meets the target value, the training is stopped to obtain the trained scene decision model.

[0025] In one of the embodiments, the scene training samples are multiple frames of images within a sampling period;

[0026] The vehicle condition training samples are the vehicle condition data within the sampling period.

[0027] In a second aspect, the present application also provides a vehicle scene warning decision device, the device comprising: a collection module, a spatial feature extraction module, a vehicle motion sequence acquisition module and a decision module;

[0028] An acquisition module is used to obtain a vehicle condition data sequence and a scene image in a current driving scene;

[0029] A spatial feature extraction module is used to input a scene image into a scene feature extraction network of a trained scene decision model to extract spatial features of the scene image;

[0030] The vehicle motion sequence acquisition module is used to input the vehicle condition data sequence into the vehicle condition prediction network of the trained scenario decision model to obtain the predicted vehicle motion sequence;

[0031] The decision module is used to integrate scene features and vehicle motion sequences, and obtain warning information of the vehicle in the current driving scene through the decision maker of the scene decision model.

[0032] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0033] Acquire a vehicle condition data sequence and a scene image of a preceding driving scene;

[0034] The scene image is input into the scene feature extraction network of the trained scene decision model to extract the spatial features of the scene image;

[0035] Input the vehicle condition data sequence into the vehicle condition prediction network of the trained scenario decision model to obtain the predicted vehicle motion sequence;

[0036] The scene features and vehicle motion sequences are integrated, and the warning information of the vehicle in the current driving scene is obtained through the decision maker of the scene decision model.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0038] Acquire a vehicle condition data sequence and a scene image of a preceding driving scene;

[0039] The scene image is input into the scene feature extraction network of the trained scene decision model to extract the spatial features of the scene image;

[0040] Input the vehicle condition data sequence into the vehicle condition prediction network of the trained scenario decision model to obtain the predicted vehicle motion sequence;

[0041] The scene features and vehicle motion sequences are integrated, and the warning information of the vehicle in the current driving scene is obtained through the decision maker of the scene decision model.

[0042] In a fifth aspect, the present application further provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the following steps are implemented:

[0043] Acquire a vehicle condition data sequence and a scene image of a preceding driving scene;

[0044] The scene image is input into the scene feature extraction network of the trained scene decision model to extract the spatial features of the scene image;

[0045] Input the vehicle condition data sequence into the vehicle condition prediction network of the trained scenario decision model to obtain the predicted vehicle motion sequence;

[0046] The scene features and vehicle motion sequences are integrated, and the warning information of the vehicle in the current driving scene is obtained through the decision maker of the scene decision model.

[0047] The above-mentioned vehicle scene warning decision method, device, computer equipment and storage medium extract the spatial features of the scene image by setting a scene feature extraction network, and set a vehicle condition prediction network to obtain the predicted vehicle motion sequence to characterize the vehicle motion features related to time. The decision maker obtains the warning information of the vehicle in the current driving scene based on the fusion of the spatial features of the scene image and the predicted vehicle motion sequence. Since the warning information is predicted based on the spatial features of the current scene and the vehicle motion features related to time, it has a certain degree of foresight and can improve the accuracy of the warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 An application environment diagram of a vehicle scene warning decision method in an embodiment;

[0049] Figure 2 A schematic diagram of a process flow of a vehicle scene warning decision method in one embodiment;

[0050] Figure 3 A schematic diagram of the structure of a scenario decision model in an embodiment;

[0051] Figure 4A schematic diagram of a flow chart of extracting spatial features of a scene image in a vehicle scene warning decision method in another embodiment;

[0052] Figure 5 A schematic diagram of the structure of a scene feature extraction network in another embodiment;

[0053] Figure 6 A schematic diagram of a flow chart for obtaining a predicted vehicle motion sequence in another embodiment;

[0054] Figure 7 A schematic diagram of the structure of a vehicle condition prediction network in another embodiment;

[0055] Figure 8 A schematic diagram of a process of obtaining warning information of a vehicle in a current driving scenario in another embodiment;

[0056] Fig. 9 A schematic diagram of a flow chart of a scenario decision model training method in another embodiment;

[0057] Fig.10 A schematic diagram of a scene decision model training process in another embodiment;

[0058] Fig.11 It is a structural diagram of a vehicle scene warning decision device in another embodiment;

[0059] Fig.12 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] The vehicle scene warning decision method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 obtains the vehicle condition data sequence and the scene image in the current driving scene, and transmits the vehicle condition data sequence and the scene image of the current driving scene to the server 104 through the network. The server 104 inputs the scene image into the scene feature extraction network of the trained scene decision model to extract the spatial features of the scene image; inputs the vehicle condition data sequence into the vehicle condition prediction network of the trained scene decision model to obtain the predicted vehicle motion sequence; fuses the scene feature and the vehicle motion sequence, and obtains the warning information of the vehicle in the current driving scene through the decision maker of the scene decision model; transmits the warning information to the terminal 102 through the network, and the human-computer interaction interface of the terminal 102 displays the warning information. The data storage system can store the scene decision model required by the server 104, the vehicle condition data sequence and the scene image in the current driving scene. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. Among them, the terminal 102 can be an intelligent vehicle-mounted device. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0062] In one embodiment, Figure 2 As shown, a vehicle scene warning decision method is provided, and the method is applied to Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0063] Step 202: Acquire a vehicle condition data sequence and a scene image of a current driving scene.

[0064] The scene image of the current driving scene may be collected by using a camera installed on the vehicle or a camera at an intersection. The scene image of the current driving scene includes n frames of video sequence images.

[0065] The vehicle condition data sequence of the vehicle can be collected through the intelligent vehicle terminal in the vehicle networking system. The vehicle condition data sequence of the vehicle may include heading angle, speed, acceleration, etc. The collection period of the vehicle condition data sequence should be included in the content of each frame of the vehicle intersection driving image, that is, the collection frequency of the driving image is f, then the vehicle condition data of the vehicle in the image within the 1 / f period should be collected.

[0066] Specifically, the terminal 102 collects the scene image of the current driving scene through the camera on the vehicle or the camera at the intersection, collects the vehicle condition data sequence of the vehicle through the intelligent vehicle terminal, and sends the collected scene image and vehicle condition data sequence to the server 104.

[0067] Step 204: input the scene image into the scene feature extraction network of the trained scene decision model to extract the spatial features of the scene image.

[0068] Among them, the scene feature extraction network of the scene decision model is used to extract the spatial features of the scene image.

[0069] Specifically, the server 104 pre-trains the scene decision model, receives the scene image transmitted by the terminal 102, and inputs the scene image into the scene feature extraction network of the trained scene decision model to extract the spatial features of the scene image.

[0070] Step 206 , input the vehicle condition data sequence into the vehicle condition prediction network of the trained scenario decision model to obtain a predicted vehicle motion sequence.

[0071] Among them, the vehicle condition prediction network is used to output the predicted vehicle motion sequence to achieve the "advancement" of decision-making information.

[0072] Specifically, the server 104 pre-trains the scenario decision model, receives the vehicle condition data sequence transmitted by the terminal 102, and inputs the vehicle condition data sequence into the vehicle condition prediction network of the trained scenario decision model to obtain a predicted vehicle motion sequence.

[0073] Step 208, the scene features and the vehicle motion sequence are integrated, and the warning information of the vehicle in the current driving scene is obtained through the decision maker of the scene decision model.

[0074] Among them, Figure 3 As shown, the scene decision model includes a scene feature extraction network, a vehicle condition prediction network and a decision maker; the scene feature extraction network is used to extract the spatial features of the scene image, and the vehicle condition prediction network is used to output the predicted vehicle motion sequence. The spatial features of the scene image and the predicted vehicle motion sequence are concatenated to obtain thickness-increased features, and then the final decision image is obtained through the decoding network. The final decision image is input into the decision maker to obtain the warning information of the vehicle in the current driving scenario.

[0075] Specifically, the server 104 inputs the scene image and the vehicle motion sequence into the scene feature extraction network and the vehicle condition prediction network respectively, obtains the spatial features of the scene image and the vehicle motion sequence, fuses the spatial features of the scene image and the vehicle motion sequence, and obtains the features with increased thickness through the splicing operation, and then obtains the final decision image through the decoding network, and inputs the final decision image into the decision maker to obtain the warning information of the vehicle in the current driving scenario.

[0076] In the above vehicle scene warning decision method, a scene feature extraction network is set to extract the spatial features of the scene image, and a vehicle condition prediction network is set to obtain a predicted vehicle motion sequence to characterize the time-related vehicle motion features. The decision maker obtains the warning information of the vehicle in the current driving scene based on the fusion of the spatial features of the scene image and the predicted vehicle motion sequence. Since the warning information is predicted based on the spatial features of the current scene and the time-related vehicle motion features, it has a certain degree of foresight and can improve the accuracy of the warning.

[0077] In one embodiment, Figure 4 As shown, the scene image is input into the scene feature extraction network of the trained scene decision model to extract the spatial features of the scene image, including:

[0078] Step 402: Input the scene image into the scene feature extraction network of the trained scene decision model, and extract the scene features of the scene image through the feature extraction structure of the scene feature extraction network.

[0079] Among them, the feature extraction structure of the scene feature extraction network can be a residual neural network.

[0080] Specifically, Figure 5 As shown, the server 104 adjusts the scene image to M×M×3, and inputs the adjusted scene image into the residual neural network to obtain scene features of the scene image.

[0081] Step 404: fuse the scene image and the scene features as input of the attention structure of the scene feature extraction network, and output the attention weight through the attention structure.

[0082] Among them, in extremely complex traffic scenes or situations with high driving risks, many useless features or noise will affect the extraction of effective features, resulting in certain difficulties in the vehicle warning decision system from scene to decision. Therefore, in order to solve this problem, this embodiment sets an attention structure in the scene feature extraction network, which can obtain more representative effective information, reduce the impact of invalid information on effective features, and achieve the purpose of fine feature extraction.

[0083] In one embodiment, the scene feature extraction network can be a multi-layer network structure with a residual neural network as the main body, and its structure includes a convolution layer, a pooling layer, an activation layer and an attention structure, and the attention structure includes a convolution kernel and an activation function.

[0084] Specifically, Figure 5As shown, the attention structure adds the current input scene image and the scene features of the scene image at the pixel level through a splicing operation. The added features are used as the input of the attention structure, and attention weights expressing the importance of different information are generated through training.

[0085] Step 406, obtaining the spatial features of the scene image according to the scene features and the attention weights.

[0086] Specifically, Figure 5 As shown in FIG, the scene feature extraction network multiplies the attention weights by the scene features at the pixel level to obtain a refined output, which is the spatial feature of the scene image.

[0087] In this embodiment, by setting an attention structure in the scene feature extraction network, more representative effective information can be obtained, reducing the impact of invalid information on effective features, thereby achieving the purpose of fine feature extraction.

[0088] In one embodiment, Figure 6 As shown, the vehicle condition data sequence is input into the vehicle condition prediction network of the trained scenario decision model to obtain the predicted vehicle motion sequence, including:

[0089] Step 602: input the vehicle condition data sequence into the vehicle condition prediction network of the trained scenario decision model. The encoding structure of the vehicle condition prediction network extracts high-level information of the vehicle condition data sequence to generate a feature vector.

[0090] The vehicle condition data sequence is a T×B matrix Q, where T represents the vehicle condition data sampling points within a period T, and B is the number of vehicle condition data, including vehicle speed, steering angle, positioning, etc. The feature vector generated by the vehicle condition prediction network can be used as a feature vector to predict the vehicle condition motion state of the next frame.

[0091] Specifically, Figure 7 As shown, the server 104 inputs the matrix Q into the vehicle condition prediction network, extracts high-level information from the vehicle condition data sequence through the encoding structure of the vehicle condition prediction network, generates a feature vector, and uses the feature vector as a feature vector for predicting the vehicle condition motion state of the next frame.

[0092] In one embodiment, the vehicle condition prediction network may be a multi-layer network structure with a long short-term memory network as the main body, which includes a convolution layer and an activation layer. The encoding structure of the vehicle condition prediction network is a long short-term memory network encoding structure.

[0093] Step 604, input the feature vector into the decoding structure of the vehicle condition prediction network, and the decoding structure decodes the encoded information with the feature vector as the initial state to obtain the predicted vehicle motion sequence.

[0094] Among them, the decoding structure of the vehicle condition prediction network is a long short-term memory network decoding structure.

[0095] In this embodiment, the predicted vehicle motion sequence is obtained through the encoding structure and decoding structure of the vehicle condition prediction network, which provides auxiliary decision-making information for the decision maker, improves the accuracy of early warning decisions, and greatly improves traffic safety in complex traffic scenarios.

[0096] In one embodiment, Figure 8 As shown, the scene features and vehicle motion sequence are integrated, and the warning information of the vehicle in the current driving scene is obtained through the decision maker of the scene decision model, including:

[0097] Step 802, the scene features and the vehicle motion sequence are integrated, and a prediction result of whether there is a safety hazard in the current driving scene is obtained through a decision maker of the scene decision model.

[0098] The decision maker takes the output of the scene feature extraction network and the vehicle condition prediction network as its input to obtain the optimal warning decision in the current scene. The training label of the decision maker is the optimal warning decision that should be given in the current last frame of the traffic scene combined with the vehicle condition data sequence at this time.

[0099] Specifically, the server 104 fuses the scene features and the vehicle motion sequence through a splicing operation to obtain features with increased thickness, and then obtains a final decision image through a decoding network, and inputs the decision image into the decision maker to obtain a decision result on whether there is a safety hazard in the current driving scene. If the decision result analyzed by the server 104 will cause a traffic accident, the predicted decision result should be displayed on the human-computer interaction interface to remind the driver to avoid the accident.

[0100] Step 804: When the prediction result indicates that there is a safety hazard in the current driving scene, a warning message is displayed on the human-machine interaction interface of the vehicle. The warning message includes the type of safety hazard and / or a method for dealing with the safety hazard.

[0101] In this embodiment, since the scene decision model adds a vehicle condition prediction network, the decision maker obtains the optimal warning decision for the vehicle in the current driving scene based on the fusion of the spatial features of the scene image and the predicted vehicle motion sequence. Since the optimal warning information is predicted based on the spatial features of the current scene and the time-related vehicle motion features, it has a certain degree of foresight and can improve the accuracy of the warning.

[0102] In one embodiment, Fig. 9 As shown, the vehicle scene warning decision method also includes a model training process, which specifically includes:

[0103] Step 902, obtaining training samples, each training sample includes a scene training sample, a first spatial feature of the labeled scene training sample, a vehicle condition training sample, a first vehicle motion sequence of the labeled vehicle condition training sample, and first warning information corresponding to the labeled scene training sample and the vehicle condition training sample.

[0104] Among them, the scene training samples include multiple frames of images within a sampling period; the first spatial feature of the annotated scene training samples is the spatial feature corresponding to the last frame of the image, and the first spatial feature is used to identify the scene training samples; the vehicle condition training samples are the vehicle condition data within the next sampling period; the first vehicle motion sequence of the annotated vehicle condition training samples is the predicted vehicle motion sequence, and the first vehicle motion sequence is used to identify the vehicle condition training samples; the first warning information corresponding to the annotated scene training samples and the vehicle condition training samples is the optimal decision data corresponding to the input decision image.

[0105] Step 904: input the scene training sample into the scene feature extraction network to obtain the predicted second spatial feature, and adjust the parameters of the scene feature extraction network according to the difference between the first spatial feature and the second spatial feature.

[0106] The difference between the first spatial feature and the second spatial feature is obtained according to an error function, wherein the error function is a mean square error, and the mathematical expression of the mean square error is: Where L represents the mean square error; y′ n Represents the input of the model; y n Represents the output of the model. In this embodiment, the input of the scene feature extraction network is the first spatial feature; the output of the scene feature extraction network is the second spatial feature. The parameters of the scene feature extraction network are represented by the weight and bias of each convolution kernel in the convolution layer of the scene feature extraction network.

[0107] Specifically, Fig.10 As shown, the server 104 uses the first spatial feature of the annotated scene training sample as the label of the annotated scene training sample, uses the scene training sample as the training input of the scene feature extraction network, obtains the predicted second spatial feature, performs error processing on the first spatial feature and the second spatial feature, obtains the error value between the two, and determines whether the error value meets the training requirements. If so, the training is stopped; otherwise, the parameters of the scene feature extraction network are updated through the gradient update method, and the next iteration is executed; through multiple iterative training, the parameters of the scene feature extraction network are updated until the loss function converges, the training of the scene feature extraction network is completed, and a trained scene feature extraction network is obtained.

[0108] Step 906: input the vehicle condition training sample into the vehicle condition prediction network, obtain the predicted second vehicle motion sequence, and adjust the parameters of the vehicle condition prediction network according to the difference between the second vehicle motion sequence and the first vehicle motion sequence.

[0109] The difference between the second vehicle motion sequence and the first vehicle motion sequence is obtained according to the error function. The error function is the same as the error function used in the scene feature extraction network training, and will not be repeated here. In this embodiment, the input of the vehicle condition prediction network is the first spatial feature; the output of the vehicle condition prediction network is the second spatial feature. The parameters of the vehicle condition prediction network are characterized by the weight and bias of each convolution kernel in the convolution layer of the vehicle condition prediction network.

[0110] Specifically, Fig.10 As shown, the training process of the vehicle condition prediction network is similar to the training process of the scene feature extraction network, which will not be repeated here.

[0111] Step 908, using the second spatial feature and the second vehicle motion sequence as training inputs of the decision maker, obtaining the second warning information output by the decision maker, adjusting the parameters of the decision maker according to the difference between the first warning information and the second warning information, and completing one training.

[0112] Among them, the difference between the first warning information and the second warning information is obtained according to the error function. The error function is the same as the error function used in the scene feature extraction network training, which will not be repeated here. In this embodiment, the input of the decision maker is the first warning information, and the first warning information is the feature after the second spatial feature and the second vehicle motion sequence are fused; the output of the decision maker is the second warning information, and the second warning information is the optimal decision corresponding to the input scene image.

[0113] Specifically, Fig.10 As shown in FIG. 1 , the training process of the decision maker is similar to the training process of the scene feature extraction network and will not be repeated here.

[0114] Step 910, iterate the training process multiple times, and when the loss function meets the target value, stop the training to obtain the trained scene decision model.

[0115] Specifically, the training process is iterated multiple times until the loss functions of the scene feature extraction network, the vehicle condition prediction network and the decision maker all meet the target values. At this point, the training is completed and a trained scene decision model can be obtained.

[0116] In this embodiment, the scene feature extraction network, the vehicle condition prediction network and the decision maker form a residual network structure. During training, since the scene decision warning model is a residual network structure, the network training efficiency and feature extraction accuracy are improved.

[0117] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0118] Based on the same inventive concept, the embodiment of the present application also provides a vehicle scene warning decision device for implementing the vehicle scene warning decision method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more vehicle scene warning decision device embodiments provided below can refer to the limitations of the vehicle scene warning decision method above, and will not be repeated here.

[0119] In one embodiment, Fig.11 As shown, a vehicle scene warning decision device is provided, comprising: a collection module, a spatial feature extraction module, a vehicle motion sequence acquisition module and a decision module, wherein:

[0120] The acquisition module is used to obtain the vehicle condition data sequence and the scene image in the current driving scene.

[0121] The spatial feature extraction module is used to input the scene image into the scene feature extraction network of the trained scene decision model to extract the spatial features of the scene image.

[0122] The vehicle motion sequence acquisition module is used to input the vehicle condition data sequence into the vehicle condition prediction network of the trained scenario decision model to obtain the predicted vehicle motion sequence.

[0123] The decision module is used to integrate scene features and vehicle motion sequences, and obtain warning information of the vehicle in the current driving scene through the decision maker of the scene decision model.

[0124] In one embodiment, a vehicle scene warning decision device further includes: a feature extraction module, a fusion module and a weighting module;

[0125] The feature extraction module is used to input the scene image into the scene feature extraction network of the trained scene decision model, and extract the scene features of the scene image through the feature extraction structure of the scene feature extraction network.

[0126] The fusion module is used to fuse the scene image and the scene features as the input of the attention structure of the scene feature extraction network, and output the attention weight through the attention structure.

[0127] The weighting module is used to obtain the spatial features of the scene image according to the scene features and attention weights.

[0128] In one embodiment, a vehicle scene warning decision device further includes: an encoding module and a decoding module;

[0129] The encoding module is used to input the vehicle condition data sequence into the vehicle condition prediction network of the trained scenario decision model. The encoding structure of the vehicle condition prediction network extracts the high-level information of the vehicle condition data sequence and generates a feature vector.

[0130] The decoding module is used to input the feature vector into the decoding structure of the vehicle condition prediction network. The decoding structure decodes the encoded information with the feature vector as the initial state to obtain the predicted vehicle motion sequence.

[0131] In one embodiment, a vehicle scene warning decision device further includes: a prediction module and a display module;

[0132] The prediction module is used to fuse scene features and vehicle motion sequences, and obtain the prediction result of whether there are safety hazards in the current driving scene through the decision maker of the scene decision model.

[0133] The display module is used to display warning information on the human-computer interaction interface of the vehicle when the prediction result shows that there are safety hazards in the current driving scene. The warning information includes the type of safety hazard and / or the treatment method for the safety hazard.

[0134] In one embodiment, a vehicle scene warning decision device further includes: a training sample acquisition module, a scene feature extraction network training module, a vehicle condition prediction network training module, a decision maker training module and an iteration module;

[0135] The training sample acquisition module is used to acquire training samples, each training sample includes a scene training sample, a first spatial feature of the labeled scene training sample, a vehicle condition training sample, a first vehicle motion sequence of the labeled vehicle condition training sample, and a first warning information corresponding to the labeled scene training sample and the vehicle condition training sample.

[0136] The scene feature extraction network training module is used to input the scene training samples into the scene feature extraction network, obtain the predicted second spatial features, and adjust the parameters of the scene feature extraction network according to the difference between the first spatial features and the second spatial features.

[0137] The vehicle condition prediction network training module is used to input the vehicle condition training samples into the vehicle condition prediction network, obtain the predicted second vehicle motion sequence, and adjust the parameters of the vehicle condition prediction network according to the difference between the second vehicle motion sequence and the first vehicle motion sequence.

[0138] The decision maker training module is used to use the second spatial feature and the second vehicle motion sequence as training inputs of the decision maker, obtain the second warning information output by the decision maker, adjust the parameters of the decision maker according to the difference between the first warning information and the second warning information, and complete a training.

[0139] The iteration module is used to iterate the training process multiple times. When the loss function meets the target value, the training is stopped to obtain the trained scene decision model.

[0140] Each module in the above-mentioned vehicle scene warning decision device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0141] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.12 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a vehicle scene warning decision method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0142] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0144] Acquire a vehicle condition data sequence and a scene image of a preceding driving scene;

[0145] The scene image is input into the scene feature extraction network of the trained scene decision model to extract the spatial features of the scene image;

[0146] Input the vehicle condition data sequence into the vehicle condition prediction network of the trained scenario decision model to obtain the predicted vehicle motion sequence;

[0147] The scene features and vehicle motion sequences are integrated, and the warning information of the vehicle in the current driving scene is obtained through the decision maker of the scene decision model.

[0148] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0149] The scene image is input into the scene feature extraction network of the trained scene decision model to extract the spatial features of the scene image, including:

[0150] The scene image is input into the scene feature extraction network of the trained scene decision model, and the scene features of the scene image are extracted through the feature extraction structure of the scene feature extraction network;

[0151] The scene image and scene features are fused as the input of the attention structure of the scene feature extraction network, and the attention weight is output through the attention structure;

[0152] According to the scene features and attention weights, the spatial features of the scene image are obtained.

[0153] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0154] The vehicle condition data sequence is input into the vehicle condition prediction network of the trained scenario decision model. The encoding structure of the vehicle condition prediction network extracts the high-level information of the vehicle condition data sequence and generates a feature vector.

[0155] The feature vector is input into the decoding structure of the vehicle condition prediction network. The decoding structure decodes the encoded information with the feature vector as the initial state to obtain the predicted vehicle motion sequence.

[0156] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0157] The scene features and vehicle motion sequence are integrated to obtain the prediction result of whether there are safety hazards in the current driving scene through the decision maker of the scene decision model;

[0158] When the prediction result is that there is a safety hazard in the current driving scenario, a warning message is displayed on the vehicle's human-computer interaction interface, and the warning information includes the type of safety hazard and / or the method for dealing with the safety hazard.

[0159] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0160] Acquire training samples, each training sample including a scene training sample, a first spatial feature of the labeled scene training sample, a vehicle condition training sample, a first vehicle motion sequence of the labeled vehicle condition training sample, and first warning information corresponding to the labeled scene training sample and the vehicle condition training sample;

[0161] Input the scene training sample into the scene feature extraction network to obtain the predicted second spatial feature, and adjust the parameters of the scene feature extraction network according to the difference between the first spatial feature and the second spatial feature;

[0162] Inputting the vehicle condition training sample into the vehicle condition prediction network to obtain a predicted second vehicle motion sequence, and adjusting the parameters of the vehicle condition prediction network according to the difference between the second vehicle motion sequence and the first vehicle motion sequence;

[0163] The second spatial feature and the second vehicle motion sequence are used as training inputs of the decision maker, the second warning information output by the decision maker is obtained, and the parameters of the decision maker are adjusted according to the difference between the first warning information and the second warning information, thereby completing one training;

[0164] The training process is iterated multiple times. When the loss function meets the target value, the training is stopped to obtain the trained scene decision model.

[0165] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0166] The scene training samples are multiple frames of images within the sampling period;

[0167] The vehicle condition training samples are the vehicle condition data within the sampling period.

[0168] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0169] Acquire a vehicle condition data sequence and a scene image of a preceding driving scene;

[0170] The scene image is input into the scene feature extraction network of the trained scene decision model to extract the spatial features of the scene image;

[0171] Input the vehicle condition data sequence into the vehicle condition prediction network of the trained scenario decision model to obtain the predicted vehicle motion sequence;

[0172] The scene features and vehicle motion sequences are integrated, and the warning information of the vehicle in the current driving scene is obtained through the decision maker of the scene decision model.

[0173] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0174] The scene image is input into the scene feature extraction network of the trained scene decision model to extract the spatial features of the scene image, including:

[0175] The scene image is input into the scene feature extraction network of the trained scene decision model, and the scene features of the scene image are extracted through the feature extraction structure of the scene feature extraction network;

[0176] The scene image and scene features are fused as the input of the attention structure of the scene feature extraction network, and the attention weight is output through the attention structure;

[0177] According to the scene features and attention weights, the spatial features of the scene image are obtained.

[0178] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0179] The vehicle condition data sequence is input into the vehicle condition prediction network of the trained scenario decision model. The encoding structure of the vehicle condition prediction network extracts the high-level information of the vehicle condition data sequence and generates a feature vector.

[0180] The feature vector is input into the decoding structure of the vehicle condition prediction network. The decoding structure decodes the encoded information with the feature vector as the initial state to obtain the predicted vehicle motion sequence.

[0181] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0182] The scene features and vehicle motion sequence are integrated to obtain the prediction result of whether there are safety hazards in the current driving scene through the decision maker of the scene decision model;

[0183] When the prediction result is that there is a safety hazard in the current driving scenario, a warning message is displayed on the vehicle's human-computer interaction interface, and the warning information includes the type of safety hazard and / or the method for dealing with the safety hazard.

[0184] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0185] Acquire training samples, each training sample including a scene training sample, a first spatial feature of the labeled scene training sample, a vehicle condition training sample, a first vehicle motion sequence of the labeled vehicle condition training sample, and first warning information corresponding to the labeled scene training sample and the vehicle condition training sample;

[0186] Input the scene training sample into the scene feature extraction network to obtain the predicted second spatial feature, and adjust the parameters of the scene feature extraction network according to the difference between the first spatial feature and the second spatial feature;

[0187] Inputting the vehicle condition training sample into the vehicle condition prediction network to obtain a predicted second vehicle motion sequence, and adjusting the parameters of the vehicle condition prediction network according to the difference between the second vehicle motion sequence and the first vehicle motion sequence;

[0188] The second spatial feature and the second vehicle motion sequence are used as training inputs of the decision maker, the second warning information output by the decision maker is obtained, and the parameters of the decision maker are adjusted according to the difference between the first warning information and the second warning information, thereby completing one training;

[0189] The training process is iterated multiple times. When the loss function meets the target value, the training is stopped to obtain the trained scene decision model.

[0190] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0191] The scene training samples are multiple frames of images within the sampling period;

[0192] The vehicle condition training samples are the vehicle condition data within the sampling period.

[0193] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0194] Acquire a vehicle condition data sequence and a scene image of a preceding driving scene;

[0195] The scene image is input into the scene feature extraction network of the trained scene decision model to extract the spatial features of the scene image;

[0196] Input the vehicle condition data sequence into the vehicle condition prediction network of the trained scenario decision model to obtain the predicted vehicle motion sequence;

[0197] The scene features and vehicle motion sequences are integrated, and the warning information of the vehicle in the current driving scene is obtained through the decision maker of the scene decision model.

[0198] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0199] The scene image is input into the scene feature extraction network of the trained scene decision model to extract the spatial features of the scene image, including:

[0200] The scene image is input into the scene feature extraction network of the trained scene decision model, and the scene features of the scene image are extracted through the feature extraction structure of the scene feature extraction network;

[0201] The scene image and scene features are fused as the input of the attention structure of the scene feature extraction network, and the attention weight is output through the attention structure;

[0202] According to the scene features and attention weights, the spatial features of the scene image are obtained.

[0203] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0204] The vehicle condition data sequence is input into the vehicle condition prediction network of the trained scenario decision model. The encoding structure of the vehicle condition prediction network extracts the high-level information of the vehicle condition data sequence and generates a feature vector.

[0205] The feature vector is input into the decoding structure of the vehicle condition prediction network. The decoding structure decodes the encoded information with the feature vector as the initial state to obtain the predicted vehicle motion sequence.

[0206] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0207] The scene features and vehicle motion sequence are integrated to obtain the prediction result of whether there are safety hazards in the current driving scene through the decision maker of the scene decision model;

[0208] When the prediction result is that there is a safety hazard in the current driving scenario, a warning message is displayed on the vehicle's human-computer interaction interface, and the warning information includes the type of safety hazard and / or the method for dealing with the safety hazard.

[0209] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0210] Acquire training samples, each training sample including a scene training sample, a first spatial feature of the labeled scene training sample, a vehicle condition training sample, a first vehicle motion sequence of the labeled vehicle condition training sample, and first warning information corresponding to the labeled scene training sample and the vehicle condition training sample;

[0211] Input the scene training sample into the scene feature extraction network to obtain the predicted second spatial feature, and adjust the parameters of the scene feature extraction network according to the difference between the first spatial feature and the second spatial feature;

[0212] Inputting the vehicle condition training sample into the vehicle condition prediction network to obtain a predicted second vehicle motion sequence, and adjusting the parameters of the vehicle condition prediction network according to the difference between the second vehicle motion sequence and the first vehicle motion sequence;

[0213] The second spatial feature and the second vehicle motion sequence are used as training inputs of the decision maker, the second warning information output by the decision maker is obtained, and the parameters of the decision maker are adjusted according to the difference between the first warning information and the second warning information, thereby completing one training;

[0214] The training process is iterated multiple times. When the loss function meets the target value, the training is stopped to obtain the trained scene decision model.

[0215] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0216] The scene training samples are multiple frames of images within the sampling period;

[0217] The vehicle condition training samples are the vehicle condition data within the sampling period.

[0218] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0219] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0220] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0221] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A vehicle scene warning decision method, characterized in that: The method comprises: Acquire a vehicle condition data sequence and a scene image of a current driving scene; Inputting the scene image into a scene feature extraction network of a trained scene decision model to extract spatial features of the scene image; Inputting the vehicle condition data sequence into a vehicle condition prediction network of a trained scenario decision model to obtain a predicted vehicle motion sequence; The scene features and the vehicle motion sequence are integrated to obtain warning information of the vehicle in the current driving scene through the decision maker of the scene decision model; The method further comprises: Acquire training samples, each training sample comprising a scene training sample, a first spatial feature of the scene training sample annotated, a vehicle condition training sample, a first vehicle motion sequence of the vehicle condition training sample annotated, and first warning information corresponding to the scene training sample and the vehicle condition training sample annotated; Inputting the scene training sample into the scene feature extraction network to obtain a predicted second spatial feature, and adjusting a parameter of the scene feature extraction network according to a difference between the first spatial feature and the second spatial feature; Inputting the vehicle condition training sample into the vehicle condition prediction network to obtain a predicted second vehicle motion sequence, and adjusting the parameters of the vehicle condition prediction network according to the difference between the second vehicle motion sequence and the first vehicle motion sequence; Using the second spatial feature and the second vehicle motion sequence as training inputs of a decision maker, obtaining second warning information output by the decision maker, adjusting parameters of the decision maker according to a difference between the first warning information and the second warning information, and completing one training; The training process is iterated multiple times. When the loss function meets the target value, the training is stopped to obtain the trained scene decision model.

2. The method according to claim 1, characterized in that Inputting the scene image into a scene feature extraction network of a trained scene decision model to extract spatial features of the scene image includes: Inputting the scene image into a scene feature extraction network of a trained scene decision model, and extracting scene features of the scene image through a feature extraction structure of the scene feature extraction network; fusing the scene image and the scene feature as input of an attention structure of a scene feature extraction network, and outputting an attention weight through the attention structure; The spatial features of the scene image are obtained according to the scene features and the attention weights.

3. The method according to claim 1, characterized in that The step of inputting the vehicle condition data sequence into a vehicle condition prediction network of a trained scenario decision model to obtain a predicted vehicle motion sequence comprises: Inputting the vehicle condition data sequence into a vehicle condition prediction network of a trained scenario decision model, wherein the encoding structure of the vehicle condition prediction network extracts high-level information of the vehicle condition data sequence to generate a feature vector; The feature vector is input into a decoding structure of the vehicle condition prediction network, and the decoding structure decodes the encoded information with the feature vector as an initial state to obtain a predicted vehicle motion sequence.

4. The method according to claim 1, characterized in that The step of fusing the scene features with the vehicle motion sequence and obtaining warning information of the vehicle in the current driving scene through the decision maker of the scene decision model includes: The scene features and the vehicle motion sequence are integrated to obtain a prediction result of whether there is a safety hazard in the current driving scene through a decision maker of the scene decision model; When the prediction result is that there is a safety hazard in the current driving scene, a warning message is displayed on the human-computer interaction interface of the vehicle, and the warning message includes the type of safety hazard and / or the treatment method for the safety hazard.

5. The method according to claim 1, characterized in that The scene training samples are multiple frames of images within a sampling period; The vehicle condition training samples are vehicle condition data within a sampling period.

6. A vehicle scene warning decision device, characterized in that: The device comprises: An acquisition module is used to obtain a vehicle condition data sequence and a scene image in a current driving scene; A spatial feature extraction module, used for inputting the scene image into a scene feature extraction network of a trained scene decision model to extract spatial features of the scene image; A vehicle motion sequence acquisition module, used for inputting the vehicle condition data sequence into the vehicle condition prediction network of the trained scenario decision model to obtain a predicted vehicle motion sequence; A decision module, used to fuse the scene features with the vehicle motion sequence, and obtain warning information of the vehicle in the current driving scene through the decision maker of the scene decision model; The device also includes a training sample acquisition module, a scene feature extraction network training module, a vehicle condition prediction network training module, a decision maker training module and an iteration module; The training sample acquisition module is used to acquire training samples, each training sample including a scene training sample, a first spatial feature of the scene training sample annotated, a vehicle condition training sample, a first vehicle motion sequence of the vehicle condition training sample annotated, and first warning information corresponding to the scene training sample and the vehicle condition training sample annotated; The scene feature extraction network training module is used to input the scene training sample into the scene feature extraction network, obtain the predicted second spatial feature, and adjust the parameters of the scene feature extraction network according to the difference between the first spatial feature and the second spatial feature; The vehicle condition prediction network training module is used to input the vehicle condition training sample into the vehicle condition prediction network, obtain the predicted second vehicle motion sequence, and adjust the parameters of the vehicle condition prediction network according to the difference between the second vehicle motion sequence and the first vehicle motion sequence; The decision maker training module is used to use the second spatial feature and the second vehicle motion sequence as training inputs of the decision maker, obtain the second warning information output by the decision maker, adjust the parameters of the decision maker according to the difference between the first warning information and the second warning information, and complete one training; The iteration module is used to iterate multiple training processes. When the loss function meets the target value, the training is stopped to obtain the trained scene decision model.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images