Vehicle driving state intelligent detection and optimization method based on deep learning

Abstract

The invention relates to the technical field of automatic driving, and discloses a vehicle driving state intelligent detection and optimization method based on deep learning, and the method comprises the steps: collecting image data outside an intelligent vehicle in real time, and carrying out the image processing; the image processing comprises the step of carrying out road obstacle anomaly detection and recognition based on a semantic segmentation technology of a deep separable convolutional network to obtain a recognition result of an unknown obstacle; acquiring driving data of the intelligent vehicle in real time and processing the data to obtain a driving data processing result; the data processing comprises the step of carrying out early warning judgment on driving data including the vehicle speed, the steering angle and the acceleration; and inputting an unknown obstacle recognition result obtained by image processing and a driving data processing result obtained by data processing into an adaptive driving decision model constructed based on a deep learning algorithm for prediction, and adaptively adjusting an intelligent vehicle driving optimal strategy according to a prediction result. According to the invention, the detection capability of the intelligent vehicle on unknown obstacles and the driving safety are obviously improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a method for intelligent detection and optimization of vehicle driving status based on deep learning. Background Art

[0002] With the rapid development of intelligent driving technology, intelligent vehicles have been able to achieve autonomous driving in a variety of scenarios. However, in practical applications, intelligent vehicles still face many challenges, especially when dealing with emergencies and unknown obstacles. These challenges pose a serious threat to the driving safety of intelligent vehicles. Therefore, it is particularly important to develop an intelligent method that can effectively detect and deal with these situations.

[0003] In the field of intelligent driving, traditional obstacle detection methods mainly rely on the recognition of known objects, such as vehicles and pedestrians. Such methods improve recognition accuracy by training a large number of known object samples. However, when faced with unknown objects or emergencies, the performance of these methods often drops significantly, and may even lead to misidentification or missed identification, thereby increasing driving risks.

[0004] In addition, smart cars also need to process a variety of sensor data in real time during driving, such as image data collected by cameras and driving data collected by on-board radars. These data are crucial for the decision-making and control systems of smart cars. However, how to effectively integrate and utilize these data to improve the driving safety and stability of smart cars is still a hot topic in current research.

[0005] Therefore, an intelligent technical solution is urgently needed to solve the above technical problems. Summary of the invention

[0006] One purpose of the present application is to provide a method for intelligent detection and optimization of vehicle driving status based on deep learning, which is used to at least solve the technical problem that traditional obstacle detection methods in the field of autonomous driving at this stage have misidentification or missed identification, thereby increasing driving risks.

[0007] To achieve the above objectives, some embodiments of the present application provide the following technical solutions:

[0008] A method for intelligent detection and optimization of vehicle driving status based on deep learning, comprising:

[0009] Collecting the image data outside the smart car in real time and performing image processing; the image processing includes detecting and identifying abnormal road obstacles based on the semantic segmentation technology of the deep separable convolutional network to obtain the recognition results of unknown obstacles;

[0010] Real-time collection of driving data of the smart car and data processing to obtain driving data processing results; the data processing includes early warning judgment of driving data including vehicle speed, steering angle, and acceleration;

[0011] The recognition results of the unknown obstacles obtained by image processing and the driving data processing results obtained by data processing are input into an adaptive driving decision model constructed based on a deep learning algorithm for prediction, and the optimal driving strategy of the smart car is adaptively adjusted according to the prediction results.

[0012] Furthermore, the real-time collection of image data outside the smart car and image processing; the image processing includes detecting and identifying abnormal road obstacles based on semantic segmentation technology of a deep separable convolutional network, and obtaining identification results of unknown obstacles, including:

[0013] Perform feature extraction on the image collected outside the vehicle to obtain image features;

[0014] The unknown objectness score is calculated according to the image features combined with the objectness score and the unknown score, and the abnormality detection capability of the algorithm for unknown objects is enhanced by calculating the unknown objectness score.

[0015] Furthermore, the step of extracting features from the image collected outside the vehicle to obtain image features includes:

[0016] A convolution operation is used to perform preliminary image feature extraction on the original image collected outside the vehicle to obtain a first image feature;

[0017] Batch normalization, used to perform batch normalization processing on the first image features output after the convolution operation;

[0018] The deep convolution is used to extract the refined features of the first image after batch normalization processing to obtain the second image features.

[0019] Random inactivation is used to randomly inactivate the second image features after deep convolution, and finally obtain the key image features.

[0020] Furthermore, the formula of the convolution operation is:

[0021]

[0022] In the formula, Input(i+m,j+n,c in ) is the input tensor at position (i+m,j+n) and c in The value of the channel; W(m,n,c in ,c out ) is the weight of the corresponding position of the convolution kernel; b(c out ) is the bias term, applicable to the output channel cout ; m represents the index of the convolution kernel in the vertical direction, ranging from 1 to k h , where k h is the height of the convolution kernel; n represents the horizontal index of the convolution kernel, ranging from 1 to k l , where k l is the width of the convolution kernel; i represents the different height positions of the input image; j represents the different width positions of the input image; the size of the convolution kernel W is k h ×k l ×C in ×C out , C in Indicates the number of input channels, C out is the number of output channels.

[0023] Furthermore, the formula for batch normalization is:

[0024]

[0025] In the formula, x ic =Conv(Input,W)+b is the first image feature obtained after the convolution operation; μ c is the mean, and is the variance, and Among them, ∈ is a preset value; γ and β are learnable parameters.

[0026] Furthermore, the formula of the depth convolution is:

[0027]

[0028] In the formula, x k is the output of batch normalization, that is, x k =BatchNorm(x i ), K is the total number of input channels of the depthwise convolution, W k is the convolution kernel corresponding to the kth channel.

[0029] Furthermore, the formula for random deactivation is:

[0030] ImageFeature=DepthwiseConv(x k )·Mask

[0031]

[0032] Among them, Mask is a binary random matrix of the same shape as the input, M i is the i-th element of the Mask matrix, each element M iThe probability p is set to 0, 0 indicates inactivation, and the probability 1-p is set to 1, 1 indicates retention; p is the probability of inactivation.

[0033] Furthermore, the step of calculating the unknown objectness score according to the image feature in combination with the objectness score and the unknown score comprises:

[0034] Calculate the object score Object_Score:

[0035] Object_Score(i,j)=f(ImageFeature(i,j))

[0036]

[0037] Where (i, j) is the pixel position, Object_Score(i, j) represents the objectness score at the pixel position (i, j) in the image, and ImageFeature(i, j) represents the key image feature at the pixel position (i, j) in the image;

[0038] Calculate the unknown score Unknown_Score:

[0039] Unknown_Score(i,j)=f(ImageFeature(i,j))-Threshold

[0040] Among them, Unknown_Score(i,j) represents the unknown score at the pixel position (i,j) in the image, and Threshold is the threshold used to distinguish between normal and abnormal;

[0041] According to the object score and the unknown score, the unknown object score Unknown_Object_Score is calculated:

[0042] Unknown_Object_Score(i,j)=Object_Score(i,j)+Unknown_Score(i,j)

[0043] Among them, Unknown_Object_Score(i,j) represents the unknown object score at the pixel position (i,j) in the image.

[0044] Furthermore, the real-time collection of driving data of the smart car and data processing are performed to obtain driving data processing results; the data processing includes early warning judgment of driving data including vehicle speed, steering angle, and acceleration, including:

[0045] Early warning judgment of the speed of the smart car is carried out. When the actual speed v 实际 >v预謷 , where v 预謷 The speed warning is to determine whether the smart car meets the warning condition of excessive speed, and then trigger the warning, that is, output the speed warning instruction;

[0046] The steering angle of the smart car is warned and judged, including: sampling and calculating the absolute value of the difference between the steering angles θ(t1) and θ(t2) of the smart car at time t1 and time t2 respectively; when Δθ>θ appears multiple times within a period of time 阈值 , that is, it is considered that the steering angle of the smart car fluctuates left and right, and the steering angle warning condition is met, then the warning is triggered, that is, the steering angle warning instruction is output;

[0047] Early warning judgment is made on the acceleration of the smart car. When |a|>a 预警 , where a is the acceleration of the smart car, a 预警 To warn of acceleration, that is, if the smart car is considered to meet the warning condition of excessive acceleration, a warning is triggered, that is, an acceleration warning instruction is output.

[0048] Furthermore, the unknown obstacle recognition result obtained by image processing and the driving data processing result obtained by data processing are input into an adaptive driving decision model constructed based on a deep learning algorithm for prediction, and the optimal driving strategy of the smart car is adaptively adjusted according to the prediction result, including:

[0049] Build an adaptive driving decision model based on deep neural network (DNN);

[0050] The processing result of the image collected outside the vehicle, i.e., the unknown object score Unknown_Object_Score, and the processing result of the intelligent vehicle driving data are spliced ​​into a vector and input into the trained adaptive driving decision model for prediction. The optimal driving strategy of the intelligent vehicle is adaptively adjusted according to the prediction result. The optimal strategy includes deceleration, parking, and route change.

[0051] The processing result of the intelligent vehicle driving data includes a speed warning instruction, a steering angle warning instruction, and an acceleration warning instruction;

[0052] The adaptive driving decision model adopts a structure combining a convolutional neural network (CNN) and a fully connected network (FCN) in a deep neural network (DNN), wherein the convolutional neural network (CNN) is used for feature extraction and the fully connected network (FCN) is used for decision output; its network structure includes an input layer, a feature extraction layer, a fully connected layer and an output layer.

[0053] Compared with the related art, the embodiment of the present application provides an intelligent detection and optimization method for automobile driving state based on deep learning. By introducing a deep separable convolutional network for semantic segmentation, and combining the object score and the unknown score to calculate the unknown object score, the present invention significantly improves the detection capability of unknown obstacles, which helps the smart car to drive more safely in a complex and changeable road environment and reduce traffic accidents caused by unknown obstacles. By making early warning judgments on data such as the speed, steering angle and acceleration of the smart car, potential driving risks can be discovered in time, providing more comprehensive safety protection for the smart car. The constructed adaptive driving decision model based on deep neural network (DNN) can make predictions based on the image processing results and the driving data processing results, and adaptively adjust the driving strategy of the smart car. This adaptability enables the smart car to better cope with various emergencies and improve the safety and stability of driving. In summary, the present invention significantly improves the detection capability and driving safety of the smart car for unknown obstacles, and provides strong technical support for the development of smart cars. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0055] Figure 1 A flowchart of a method for intelligent detection and optimization of vehicle driving status based on deep learning provided in an embodiment of the present application;

[0056] Figure 2 A flowchart of the specific algorithm of the method for intelligent detection and optimization of vehicle driving status based on deep learning provided in an embodiment of the present application;

[0057] Figure 3 An exemplary structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0059] Those skilled in the art can understand that the terms "first", "second" and the like in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor represent the necessary logical order between them. It should also be understood that in the embodiments of the present disclosure, "multiple" can refer to two or more, and "at least one" can refer to one, two or more. It should also be understood that for any component, data or structure mentioned in the embodiments of the present disclosure, in the absence of explicit limitation or contrary revelation given in the context, it can generally be understood as one or more. In addition, the term "and / or" in the present disclosure is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are an "or" relationship. It should also be understood that the description of each embodiment in the present disclosure emphasizes the differences between the embodiments, and the same or similar parts can refer to each other. For the sake of brevity, they will not be repeated one by one.

[0060] At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. The techniques, methods and devices known to ordinary technicians in the relevant fields may not be discussed in detail, but where appropriate, the techniques, methods and devices should be considered part of the specification. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] First embodiment

[0063] In order to increase the safety of autonomous driving of smart cars, the present invention proposes a method for intelligent detection and optimization of car driving status based on deep learning. Figure 1The flowchart of a method for intelligent detection and optimization of vehicle driving status based on deep learning provided in the embodiment of the present application is as follows. The method is divided into two parts according to the different processed data: one is the processing of image data collected outside the vehicle, and the other is the processing of intelligent vehicle driving data. The specific algorithm step flowchart is as follows Figure 2 As shown. Figure 1 and 2 As shown, a method for intelligent detection and optimization of vehicle driving status based on deep learning includes:

[0064] S110: real-time collection of image data outside the smart car and image processing; image processing includes semantic segmentation technology based on deep separable convolutional network to detect and identify abnormal road obstacles and obtain recognition results of unknown obstacles;

[0065] This step S110, through the road obstacle detection based on semantic segmentation, focuses on the recognition of unknown object obstacles on the road, that is, the anomaly detection of the image data collected outside the car, and uses the customized object score (the semantic prediction probability of each pixel is used as the object score) and the unknown score (the pixel-level anomaly score output from the panoramic segmentation head to represent the possibility of unknown objects) to improve the detection performance. Each pixel of the image needs to be assigned a semantic label. For example, in an image, a pixel may be labeled as "car" and another pixel may be labeled as "pedestrian".

[0066] Specifically, step S110 includes:

[0067] S111: Extract features of the image collected outside the vehicle to obtain image features, including:

[0068] S1111: Extract image features ImageFeature through deep separable convolution, the formula is: ImageFeature

[0069] =DropConnect(BatchNorm(DepthwiseConv(BatchNorm(Conv(Input,W)+b)))

[0070] Further, S1111: extracting image features through depthwise separable convolution, including:

[0071] (1) Convolution operation (Conv), which is used to perform preliminary image feature extraction on the original image collected outside the vehicle to obtain the first image feature;

[0072] Furthermore, the formula for the convolution operation (Conv) is:

[0073]

[0074] In the formula, Input(i+m,j+n,c in ) is the input tensor at position (i+m,j+n) and c in The value of the channel; W(m,n,c in ,c out ) is the weight of the corresponding position of the convolution kernel; b(c out ) is the bias term, applicable to the output channel c out ; m represents the index of the convolution kernel in the vertical direction, ranging from 1 to k h , where k h is the height of the convolution kernel; n represents the horizontal index of the convolution kernel, ranging from 1 to k l , where k l is the width of the convolution kernel; i represents the different height positions of the input image; j represents the different width positions of the input image; the size of the convolution kernel W is k h ×k l ×C in ×C out , C in Indicates the number of input channels, C out is the number of output channels.

[0075] The input of the above convolution operation is a three-dimensional vector Input, whose dimensions are H×L×C in , where H represents the height and L represents the width; the size of the convolution kernel W is k h ×k l ×C in ×C out .

[0076] (2) batch normalization (BatchNorm), used to perform batch normalization processing on the first image features output after the convolution operation;

[0077] Furthermore, the formula for batch normalization (BatchNorm) is:

[0078]

[0079] In the formula, x ic =Conv(Input,W)+b is the first image feature obtained after the convolution operation; μ c is the mean, and is the variance, and Among them, ∈ is a preset value, which is a very small number, usually 1e -5 ; γ and β are learnable parameters.

[0080] (3) Deep convolution (Depthwi seConv), which is used to refine the first image features after batch normalization processing to obtain the second image features.

[0081] Depthwise Conv is different from convolution operation (Conv). To reduce computational complexity, each input channel of depthwise convolution has its own independent convolution kernel, instead of applying the convolution kernel to all channels together like convolution operation.

[0082] Furthermore, the formula for depth convolution is:

[0083]

[0084] In the formula, x k is the output of batch normalization, that is, x k =BatchNorm(x i ), K is the total number of input channels of the depthwise convolution, W k is the convolution kernel corresponding to the kth channel.

[0085] (4) DropConnect is used to randomly inactivate the second image features after deep convolution and finally obtain the key image features.

[0086] The formula for random dropout (DropConnect) is:

[0087] ImageFeature=DepthwiseConv(x k )·Mask

[0088]

[0089] Among them, Mask is a binary random matrix of the same shape as the input, M i is the i-th element of the Mask matrix, each element M i The probability p is set to 0, 0 indicates inactivation, and the probability 1-p is set to 1, 1 indicates retention; p is the probability of inactivation, usually 0.2.

[0090] Through the above series of processing, the key features in the external vehicle image can be efficiently extracted, providing strong support for subsequent image recognition, analysis and other tasks.

[0091] S1112: Calculating the unknown object score according to the image feature combined with the object score and the unknown score includes:

[0092] (1) Calculate the object score Object_Score:

[0093] Object_Score(i,j)=f(ImageFeature(i,j))

[0094]

[0095] Where (i, j) is the pixel position, Object_Score(i, j) represents the objectness score at the pixel position (i, j) in the image, and ImageFeature(i, j) represents the key image feature at the pixel position (i, j) in the image;

[0096] (2) Calculate the unknown score Unknown_Score:

[0097] Unknown_Score(i,j)=f(ImageFeature(i,j))-Threshold

[0098] Among them, Unknown_Score(i,j) represents the unknown score at the pixel position (i,j) in the image, and Threshold is the threshold used to distinguish between normal and abnormal;

[0099] (3) Calculate the unknown object score Unknown_Object_Score based on the object score and the unknown score:

[0100] Unknown_Object_Score(i,j)=Object_Score(i,j)+Unknown_Score(i,j)

[0101] Among them, Unknown_Object_Score(i,j) represents the unknown object score at the pixel position (i,j) in the image.

[0102] The unknown objectness score is calculated based on the image feature ImageFeature extracted in step S111. The unknown objectness score is calculated by combining the objectness score and the unknown score, and the algorithm's abnormality detection capability for unknown objects is enhanced by calculating the unknown objectness score.

[0103] In addition, by performing performance testing on the S1111 algorithm for extracting image features through deep separable convolution, the evaluation criteria for the comparative experiment on processing image data collected outside the vehicle include panoramic quality, resolution quality, and recognition quality. It can be concluded that the algorithm for extracting image features through deep separable convolution proposed in the present invention performs better in the overall segmentation task, is more accurate in identifying objects, and has higher recognition quality.

[0104] S112: Calculate the unknown objectness score according to the image features combined with the objectness score and the unknown score, and enhance the algorithm's abnormality detection capability for unknown objects by calculating the unknown objectness score.

[0105] This step S110 significantly improves the detection capability of unknown obstacles by introducing a deep separable convolutional network for semantic segmentation and combining the objectness score and the unknown score to calculate the unknown objectness score, which helps smart cars to drive more safely in complex and changeable road environments and reduce traffic accidents caused by unknown obstacles.

[0106] S120: collecting driving data of the smart car in real time and performing data processing to obtain driving data processing results; the data processing includes making early warning judgments on driving data including vehicle speed, steering angle, and acceleration;

[0107] Specifically, step S120 includes:

[0108] (1) Early warning of the speed of the smart car. When the actual speed v 实际 >v 预謷 , where v 预謷 The speed warning is to determine whether the smart car meets the warning condition of excessive speed, and then trigger the warning, that is, output the speed warning instruction;

[0109] (2) Making early warning judgments on the steering angle of the smart car, including: sampling and calculating the absolute values ​​of the steering angles θ(t1) and θ(t2) of the smart car at time t1 and time t2 respectively; when Δθ>θ appears multiple times within a period of time 阈值 , that is, it is considered that the steering angle of the smart car fluctuates left and right, and the steering angle warning condition is met, then the warning is triggered, that is, the steering angle warning instruction is output;

[0110] (3) Early warning judgment of the acceleration of the smart car, when |a|>a 预警 , where a is the acceleration of the smart car, a 预警 To warn of acceleration, that is, if the smart car is considered to meet the warning condition of excessive acceleration, a warning is triggered, that is, an acceleration warning instruction is output.

[0111] In this step, the driving data of the smart car, including speed, steering angle, acceleration, etc., are collected and processed in real time. By making early warning judgments on these data, the present invention can timely discover potential driving risks and provide more comprehensive safety protection for the smart car.

[0112] S130: Input the recognition result of the unknown obstacle obtained by image processing and the driving data processing result obtained by data processing into an adaptive driving decision model constructed based on a deep learning algorithm for prediction, and adaptively adjust the optimal driving strategy of the smart car according to the prediction result.

[0113] Specifically, step S130 includes:

[0114] Build an adaptive driving decision model based on deep neural network (DNN);

[0115] Among them, the adaptive driving decision model adopts a structure that combines the convolutional neural network (CNN) and the fully connected network (FCN) in the deep neural network (DNN). The convolutional neural network (CNN) is used for feature extraction and the fully connected network (FCN) is used for decision output; its network structure includes input layer, feature extraction layer, fully connected layer and output layer.

[0116] The network structure is as follows:

[0117] (1) Input layer:

[0118] Input 1: Unknown_Object_Score, shape: [1] (single value);

[0119] Input 2: the result of intelligent vehicle driving data processing, including speed warning command, steering angle warning command, acceleration warning command, in the shape of [3] (three Boolean values ​​or numerical values);

[0120] Concatenates two inputs into a vector with a shape of [4].

[0121] (2) Feature extraction layer:

[0122] A 1D convolutional layer (Conv1 D) is used to extract features from the concatenated vectors. The convolution kernel size is 1 and the number of output channels is 16. Of course, the number of channels can also be other values.

[0123] This is followed by a batch normalization layer (BatchNorm) and a ReLU activation function.

[0124] (3) Fully connected layer:

[0125] Use two fully connected layers (Dense), the first fully connected layer has, for example, 32 neurons, and the second fully connected layer has 3 neurons, corresponding to the three outputs of slowing down, stopping, and changing the route.

[0126] Each fully connected layer is followed by a batch normalization layer and a ReLU activation function (except the output layer).

[0127] (4) Output layer:

[0128] The output is normalized using the Softmax activation function to obtain the probability distribution of each decision (slow down, stop, change route).

[0129] The training process of the adaptive driving decision model based on deep neural network (DNN) includes the following steps:

[0130] (1) Data preparation:

[0131] Collect data of smart cars driving on real roads, including external images, driving data (speed, steering angle, acceleration) and corresponding driving decisions (slowing down, stopping, changing routes).

[0132] Preprocess the image outside the vehicle to obtain the Unknown_Object_Score.

[0133] The driving data is processed to obtain speed warning instructions, steering angle warning instructions, and acceleration warning instructions.

[0134] (2) Data preprocessing:

[0135] Concatenate the Unknown_Object_Score and the driving data processing results into an input vector.

[0136] Encode driving decisions as one-hot vectors as labels.

[0137] (3) Model training:

[0138] Use the cross entropy loss function as the optimization objective, select the Adam optimizer for parameter update, and set the appropriate learning rate, batch size, and number of training rounds for model training.

[0139] (4) Model evaluation:

[0140] Use indicators such as accuracy, recall, and F1 score to evaluate model performance; perform cross-validation on the validation set to ensure the generalization ability of the model.

[0141] After the training is completed, the model is exported to a format that can be run on the smart car, and the exported model is integrated into the driving decision-making system of the smart car. The real-time external images and driving data are collected and input into the model for decision-making. The driving strategy of the smart car is adjusted according to the output of the model (slowing down, stopping, changing route).

[0142] Through this step S130, an adaptive driving decision model based on a deep neural network (DNN) is constructed, and the model can make predictions based on the image processing results and the driving data processing results, and adaptively adjust the driving strategy of the smart car.

[0143] Furthermore, in practical applications, the processing result of the image collected outside the vehicle obtained in step S110, i.e., the unknown object score Unknown_Object_Score, and the processing result of the smart car driving data obtained in step S120, i.e., the speed warning instruction, the steering angle warning instruction, and the acceleration warning instruction, are spliced ​​into a vector and input into the trained adaptive driving decision model for prediction. The optimal driving strategy of the smart car is adaptively adjusted according to the prediction results. The optimal strategy includes deceleration, parking, and route change.

[0144] In addition, in the present invention, the method also includes real-time monitoring of the output of the adaptive driving decision model to ensure the correctness and safety of the decision; regularly collecting new driving data for model updating and retraining to improve the performance and adaptability of the model.

[0145] According to the above embodiment of the present invention, the above technical solution introduces a deep separable convolutional network for semantic segmentation, and combines the object score and the unknown score to calculate the unknown object score. The present invention significantly improves the detection capability of unknown obstacles, helps the smart car to drive more safely in a complex and changeable road environment, and reduces traffic accidents caused by unknown obstacles. By making early warning judgments on data such as the speed, steering angle and acceleration of the smart car, potential driving risks can be discovered in time, providing more comprehensive safety protection for the smart car. The constructed adaptive driving decision model based on the deep neural network (DNN) can make predictions based on the image processing results and the driving data processing results, and adaptively adjust the driving strategy of the smart car. This adaptability enables the smart car to better cope with various emergencies and improve the safety and stability of driving. In summary, the present invention significantly improves the detection capability and driving safety of the smart car for unknown obstacles, and provides strong technical support for the development of smart cars.

[0146] Second embodiment

[0147] Figure 3 This is a schematic diagram of an exemplary structure of an electronic device provided in the third embodiment of the present application. Figure 3 As shown, an electronic device, which may be various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device may also be various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices.

[0148] The electronic device includes: one or more processors; and a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor executes the steps of the method provided in any one or more of the above embodiments. Figure 3An exemplary structural diagram of the electronic device is disclosed. Figure 3 As shown, the electronic device includes: one or more processors 1101, memory 1102, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Among them, the components shown in this article, their connections and relationships, and their functions are only used as examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0149] The electronic device may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected via a bus or other means. Figure 2 The example of connecting through bus is taken in the following.

[0150] The input device 1103 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1104 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0151] To provide interaction with a user, the electronic device may be a computer. The computer has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0152] In the embodiments of the present application, a computer program / instruction is stored on a computer-readable medium, and when the computer program / instruction is executed by a processor, the steps of the method provided by any one or more of the above embodiments are implemented. The computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more computer-readable instructions.

[0153] The memory 1102 can be used as a non-transient computer-readable storage medium, which can be used to store non-transient software programs, non-transient computer executable programs and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1102, so as to implement the program instructions / modules corresponding to the method provided by any one or more embodiments in the embodiments of the present application.

[0154] The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1102 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1102 may optionally include a memory remotely arranged relative to the processor 1101, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0155] It should be noted that the computer-readable medium described in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0156] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0157] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0158] In the above-described embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. For example, it can be implemented by using an application specific integrated circuit (ASIC), a general-purpose computer or any other similar hardware device. In certain embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented by hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0159] The computer program product provided in the embodiment of the present application includes one or more computer programs / instructions, and when the computer program / instructions are executed by the processor, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive so lid state di sk (SSD)), etc.

[0160] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0161] The scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim may also be implemented by one unit or device through software or hardware. The words "first", "second", etc. are only used to distinguish the description, and do not indicate any particular order, nor can they be understood as indicating or implying relative importance.

[0162] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily mention changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims, and the above embodiments should be regarded as exemplary and non-restrictive.

Claims

1. A method for intelligent detection and optimization of vehicle driving status based on deep learning, characterized in that: include: Collecting the image data outside the smart car in real time and performing image processing; the image processing includes detecting and identifying abnormal road obstacles based on the semantic segmentation technology of the deep separable convolutional network to obtain the recognition results of unknown obstacles; Real-time collection of driving data of the smart car and data processing to obtain driving data processing results; the data processing includes early warning judgment of driving data including vehicle speed, steering angle, and acceleration; The recognition results of the unknown obstacles obtained by image processing and the driving data processing results obtained by data processing are input into an adaptive driving decision model constructed based on a deep learning algorithm for prediction, and the optimal driving strategy of the smart car is adaptively adjusted according to the prediction results.

2. The method according to claim 1, characterized in that in, The real-time collection of image data outside the smart car and image processing; the image processing includes detecting and identifying abnormal road obstacles based on the semantic segmentation technology of the deep separable convolutional network, and obtaining the recognition result of unknown obstacles, including: Perform feature extraction on the image collected outside the vehicle to obtain image features; The unknown objectness score is calculated according to the image features combined with the objectness score and the unknown score, and the abnormality detection capability of the algorithm for unknown objects is enhanced by calculating the unknown objectness score.

3. The method according to claim 2, characterized in that in, The feature extraction of the image collected outside the vehicle to obtain the image features includes: A convolution operation is used to perform preliminary image feature extraction on the original image collected outside the vehicle to obtain a first image feature; Batch normalization, used to perform batch normalization processing on the first image features output after the convolution operation; The deep convolution is used to extract the refined features of the first image after batch normalization processing to obtain the second image features. Random inactivation is used to randomly inactivate the second image features after deep convolution, and finally obtain the key image features.

4. The method according to claim 3, characterized in that in, The formula for the convolution operation is: In the formula, Input(i+m,j+n,c in ) is the input tensor at position (i+m,j+n) and c in The value of the channel; W(m,n,c in ,c out ) is the weight of the corresponding position of the convolution kernel; b(c out ) is the bias term, applicable to the output channel c out ; m represents the index of the convolution kernel in the vertical direction, ranging from 1 to k h , where k h is the height of the convolution kernel; n represents the horizontal index of the convolution kernel, ranging from 1 to k l , where k l is the width of the convolution kernel; i represents the different height positions of the input image; j represents the different width positions of the input image; the size of the convolution kernel W is k h ×k l ×C in ×C out , C in Indicates the number of input channels, C out is the number of output channels.

5. The method according to claim 4, characterized in that in, The formula for batch normalization is: In the formula, x ic =Conv(Input,W)+b is the first image feature obtained after the convolution operation; μ c is the mean, and is the variance, and Among them, ∈ is a preset value; γ and β are learnable parameters.

6. The method according to claim 5, characterized in that in, The formula for the depthwise convolution is: In the formula, x k is the output of batch normalization, that is, x k =BatchNorm(x i ), K is the total number of input channels of the depthwise convolution, W k is the convolution kernel corresponding to the kth channel.

7. The method according to claim 6, wherein: The formula for random deactivation is: ImageFeature=DepthwiseConv(x k )·Mask Among them, Mask is a binary random matrix of the same shape as the input, M i is the i-th element of the Mask matrix, each element M i The probability p is set to 0, 0 indicates inactivation, and the probability 1-p is set to 1, 1 indicates retention; p is the probability of inactivation.

8. The method according to claim 7, characterized in that in, The calculating the unknown objectness score according to the image feature combined with the objectness score and the unknown score comprises: Calculate the object score Object_Score: Object_Score(i,j)=f(ImageFeature(i,j)) Where (i, j) is the pixel position, Object_Score(i, j) represents the objectness score at the pixel position (i, j) in the image, and ImageFeature(i, j) represents the key image feature at the pixel position (i, j) in the image; Calculate the unknown score Unknown_Score: Unknown_Score(i,j)=f(ImageFeature(i,j))-Threshold Among them, Unknown_Score(i,j) represents the unknown score at the pixel position (i,j) in the image, and Threshold is the threshold used to distinguish between normal and abnormal; According to the object score and the unknown score, the unknown object score Unknown_Object_Score is calculated: Unknown_Object_Score(i,j)=Object_Score(i,j)+Unknown_Score(i,j) Among them, Unknown_Object_Score(i,j) represents the unknown object score at the pixel position (i,j) in the image.

9. The method according to claim 8, characterized in that in, The real-time collection of driving data of the smart car and data processing are performed to obtain driving data processing results; the data processing includes early warning judgment of driving data including vehicle speed, steering angle, and acceleration, including: Early warning judgment of the speed of the smart car is carried out. When the actual speed v 实际 >v 预謷 , where v 预謷 The speed warning is to determine whether the smart car meets the warning condition of excessive speed, and then trigger the warning, that is, output the speed warning instruction; The steering angle of the smart car is warned and judged, including: sampling and calculating the absolute value of the difference between the steering angles θ(t1) and θ(t2) of the smart car at time t1 and time t2 respectively; when Δθ>θ appears multiple times within a period of time 阈值 , that is, it is considered that the steering angle of the smart car fluctuates left and right, and the steering angle warning condition is met, then the warning is triggered, that is, the steering angle warning instruction is output; Early warning judgment is made on the acceleration of the smart car. When |a|>a 预警 , where a is the acceleration of the smart car, a 预警 To warn of acceleration, that is, if the smart car is considered to meet the warning condition of excessive acceleration, a warning is triggered, that is, an acceleration warning instruction is output.

10. The method according to claim 9, characterized in that in, The method of inputting the unknown obstacle recognition result obtained by image processing and the driving data processing result obtained by data processing into an adaptive driving decision model constructed based on a deep learning algorithm for prediction, and adaptively adjusting the optimal driving strategy of the smart car according to the prediction result, includes: Build an adaptive driving decision model based on deep neural network (DNN); The processing result of the image collected outside the vehicle, i.e., the unknown object score Unknown_Object_Score, and the processing result of the intelligent vehicle driving data are spliced ​​into a vector and input into the trained adaptive driving decision model for prediction. The optimal driving strategy of the intelligent vehicle is adaptively adjusted according to the prediction result. The optimal strategy includes deceleration, parking, and route change. The processing result of the intelligent vehicle driving data includes a speed warning instruction, a steering angle warning instruction, and an acceleration warning instruction; The adaptive driving decision model adopts a structure combining a convolutional neural network (CNN) and a fully connected network (FCN) in a deep neural network (DNN), wherein the convolutional neural network (CNN) is used for feature extraction and the fully connected network (FCN) is used for decision output; its network structure includes an input layer, a feature extraction layer, a fully connected layer and an output layer.

Images