Intelligent driving warning system and method based on road condition recognition

Abstract

This application discloses an intelligent driving warning system and method based on road condition recognition. First, the road monitoring image is passed through a lane line target detection network to obtain a lane line region of interest, and the road monitoring image is passed through a vehicle target detection network to obtain a vehicle region of interest. Then, the lane line region of interest is passed through a first convolutional neural network model using a spatial attention mechanism to obtain a lane line feature map, and the vehicle region of interest is passed through a second convolutional neural network model using a spatial attention mechanism to obtain a vehicle feature map. Then, the lane line feature map and the vehicle feature map are fused to obtain a classification feature map. Finally, the classification feature map is passed through a classifier to obtain a classification result for indicating whether a light warning prompt is generated for surrounding traffic participants. In this way, road safety can be improved while improving traffic efficiency.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and more specifically, to an intelligent driving warning system and method based on road condition recognition. Background Art

[0002] A self-driving car (also known as an unmanned vehicle, computer-driven vehicle, or wheeled mobile robot) is an intelligent vehicle that uses a computer system to achieve unmanned operation. Current self-driving technology lacks the ability to proactively alert surrounding traffic participants.

[0003] Therefore, an optimized intelligent driving warning solution based on road condition recognition is expected. Summary of the Invention

[0004] In order to solve the above technical problems, the present application provides an intelligent driving warning system and method based on road condition recognition. It first passes the road monitoring image through the lane line target detection network to obtain the lane line area of ​​interest, and passes the road monitoring image through the vehicle target detection network to obtain the vehicle area of ​​interest. Then, the lane line area of ​​interest is passed through the first convolutional neural network model using the spatial attention mechanism to obtain a lane line feature map, and the vehicle area of ​​interest is passed through the second convolutional neural network model using the spatial attention mechanism to obtain a vehicle feature map. Then, the lane line feature map and the vehicle feature map are fused to obtain a classification feature map. Finally, the classification feature map is passed through a classifier to obtain a classification result for determining whether to generate a light warning prompt for surrounding traffic participants. In this way, road safety can be improved while improving traffic efficiency.

[0005] According to one aspect of the present application, an intelligent driving warning system based on road condition recognition is provided, which includes:

[0006] A monitoring module, configured to obtain road monitoring images collected by a camera deployed on a vehicle;

[0007] A lane detection module is used to pass the road monitoring image through a lane target detection network to obtain a lane region of interest;

[0008] a vehicle detection module, configured to pass the road monitoring image through a vehicle target detection network to obtain a vehicle region of interest;

[0009] A lane feature extraction module is configured to extract the lane region of interest through a first convolutional neural network model using a spatial attention mechanism to obtain a lane feature map;

[0010] A vehicle feature extraction module, configured to extract the vehicle region of interest through a second convolutional neural network model using a spatial attention mechanism to obtain a vehicle feature map;

[0011] an interaction module, configured to fuse the lane feature map and the vehicle feature map to obtain a classification feature map; and

[0012] The warning result generation module is used to pass the classification feature map through the classifier to obtain the classification result.

[0013] Determine whether to generate a light warning prompt for surrounding traffic participants based on the classification result.

[0014] In the above-mentioned intelligent driving warning system based on road condition recognition, the lane line target detection network and the vehicle target detection network are target detection networks based on anchor windows.

[0015] In the above-mentioned intelligent driving warning system based on road condition recognition, the anchor window-based target detection network is Fast R-CNN, Faster R-CNN or RetinaNet.

[0016] In the above-mentioned intelligent driving warning system based on road condition recognition, the lane feature extraction module includes:

[0017] A lane line convolutional coding unit, configured to perform deep convolutional coding on the lane line region of interest using the convolutional coding portion of the first convolutional neural network model using the spatial attention mechanism to obtain a lane line high-dimensional feature map;

[0018] a lane line spatial attention unit, configured to input the lane line high-dimensional feature map into the spatial attention part of the first convolutional neural network model using the spatial attention mechanism to obtain a lane line spatial attention map; and

[0019] The lane line spatial attention applying unit is used to respectively calculate the lane line high-dimensional feature map and the lane line spatial attention map by multiplying the position points to obtain the lane line feature map.

[0020] In the above-mentioned intelligent driving warning system based on road condition recognition, the vehicle feature extraction module includes:

[0021] A vehicle convolutional coding unit, configured to perform deep convolutional coding on the vehicle region of interest using the convolutional coding portion of the second convolutional neural network model using the spatial attention mechanism to obtain a high-dimensional feature map of the vehicle;

[0022] a vehicle spatial attention unit, configured to input the vehicle high-dimensional feature map into the spatial attention part of the second convolutional neural network model using the spatial attention mechanism to obtain a vehicle spatial attention map; and

[0023] A vehicle spatial attention applying unit is used to respectively calculate the point-by-point multiplication of the vehicle high-dimensional feature map and the vehicle spatial attention map to obtain the vehicle feature map.

[0024] In the above-mentioned intelligent driving warning system based on road condition recognition, the interaction module is further used to: fuse the lane feature map and the vehicle feature map using the following formula to obtain the classification feature map;

[0025] Wherein, the formula is:

[0026]

[0027] Among them, F i Represents the lane feature map, F j represents the vehicle feature map, F c represents the classification feature map, α and β represent weight hyperparameters, exp(·) represents the exponential operation of the feature map, and the exponential operation on the feature map represents the natural exponential function value raised to the power of the eigenvalues ​​at each position in the feature map.

[0028] In the above-mentioned intelligent driving warning system based on road condition recognition, the warning result generation module is further used to:

[0029] Expanding the classification feature map into a classification feature vector according to row vectors or column vectors;

[0030] Performing full-connection encoding on the classification feature vector using multiple fully-connected layers of the classifier to obtain an encoded classification feature vector; and

[0031] The encoded classification feature vector is passed through the Softmax classification function of the classifier to obtain the classification result.

[0032] According to another aspect of the present application, an intelligent driving warning method based on road condition recognition is provided, which includes:

[0033] Acquiring road monitoring images collected by cameras deployed on vehicles;

[0034] Passing the road monitoring image through a lane line target detection network to obtain a lane line region of interest;

[0035] Passing the road monitoring image through a vehicle target detection network to obtain a vehicle region of interest;

[0036] The lane line region of interest is passed through a first convolutional neural network model using a spatial attention mechanism to obtain a lane line feature map;

[0037] The vehicle region of interest is passed through a second convolutional neural network model using a spatial attention mechanism to obtain a vehicle feature map;

[0038] fusing the lane feature map and the vehicle feature map to obtain a classification feature map; and

[0039] Pass the classification feature map through the classifier to obtain the classification result,

[0040] Determine whether to generate a light warning prompt for surrounding traffic participants based on the classification result.

[0041] In the above-mentioned intelligent driving warning method based on road condition recognition, the lane line target detection network and the vehicle target detection network are target detection networks based on anchor windows.

[0042] In the above-mentioned intelligent driving warning method based on road condition recognition, the anchor window-based target detection network is Fast R-CNN, Faster R-CNN or RetinaNet.

[0043] Compared with the existing technology, the intelligent driving warning system and method based on road condition recognition provided by this application has the following beneficial effects:

[0044] 1. First, a road monitoring image is passed through a lane line target detection network to obtain a lane line region of interest. The road monitoring image is passed through a vehicle target detection network to obtain a vehicle region of interest. Next, the lane line region of interest is passed through a first convolutional neural network model using a spatial attention mechanism to obtain a lane line feature map. The vehicle region of interest is passed through a second convolutional neural network model using a spatial attention mechanism to obtain a vehicle feature map. The lane line feature map and the vehicle feature map are then fused to obtain a classification feature map. Finally, the classification feature map is passed through a classifier to obtain a classification result used to determine whether to generate a light warning prompt for surrounding traffic participants. In this way, road safety can be improved while improving traffic efficiency.

[0045] 2. The lane ROI is trained on a first convolutional neural network model using a spatial attention mechanism to generate a lane feature map. The vehicle ROI is trained on a second convolutional neural network model using a spatial attention mechanism to generate a vehicle feature map. The lane feature map and vehicle feature map are then fused to generate a classification feature map. This classification feature map acquisition method enables more accurate feature extraction, thereby ensuring the reliability of warnings.

[0046] 3. By integrating the spatial attention mechanism into the convolutional neural network model to obtain the first and second convolutional neural network models, the problem of low feature recognition accuracy caused by the different amounts of information contained in the pixel values ​​at various locations in the lane line region of interest and the vehicle region of interest is solved.

[0047] 4. Perform domain-adaptive class graph topological fusion on the lane feature map and the vehicle feature map, so that the fused classification feature map not only has pixel-level feature sparsity and non-network properties, but also has relatively good feature density. This maximizes the alignment and aggregation of the distribution of different classes of objects in the lane feature map and the vehicle feature map, thereby improving the classification accuracy of the classification feature map. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0049] Figure 1 This is a diagram of an application scenario of an intelligent driving warning system based on road condition recognition according to an embodiment of the present application.

[0050] Figure 2 Schematic diagram of a block diagram of an intelligent driving warning system based on road condition recognition according to an embodiment of the present application.

[0051] Figure 3 Schematic diagram of a block diagram of the lane feature extraction module in the intelligent driving warning system based on road condition recognition according to an embodiment of the present application.

[0052] Figure 4 Schematic diagram of a block diagram of the vehicle feature extraction module in the intelligent driving warning system based on road condition recognition according to an embodiment of the present application.

[0053] Figure 5 This is a flowchart of an intelligent driving warning method based on road condition recognition according to an embodiment of the present application.

[0054] Figure 6 Schematic diagram of the system architecture of the intelligent driving warning method based on road condition recognition according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described herein.

[0056] Scenario Overview

[0057] As mentioned above, in the current autonomous driving technology, there is a lack of a function to actively remind traffic participants around the main vehicle. Therefore, in the technical solution of this application, a proactive reminder solution for traffic participants around the main vehicle is constructed through machine vision and artificial intelligence-based image processing technology, so as to proactively identify the encroachment of surrounding traffic participants on the main vehicle's driving path and proactively send signals, for example, through actuators such as lights and horns to remind surrounding traffic participants. This can effectively remind surrounding traffic participants to reasonably avoid and improve road safety; at the same time, it can effectively remind surrounding traffic participants to reasonably avoid and improve the driving efficiency of the main vehicle; and it can reduce the burden on the main vehicle driver, so that the main vehicle driver can have a more relaxed driving experience.

[0058] Specifically, in the technical solution of the present application, a road monitoring image captured by a camera deployed at the front, side, or rear of the vehicle is first obtained. It should be understood that in existing vehicles, cameras are deployed at multiple locations on the vehicle, such as the front, side, or rear of the vehicle. Therefore, the cameras deployed at the front, side, or rear of the vehicle can be used to capture the road conditions on the vehicle's driving path. That is, the road monitoring image is captured by the camera deployed at the front of the vehicle. The road monitoring image can be a road monitoring image captured by the front camera of the vehicle in front of the vehicle, or a road monitoring image captured by the side camera of the vehicle to the side of the vehicle, or a road monitoring image captured by the rear camera of the vehicle behind the vehicle, etc.

[0059] Next, the road surveillance image is passed through a lane line target detection network to obtain a lane line region of interest; at the same time, the road surveillance image is passed through a vehicle target detection network to obtain a vehicle region of interest. That is, the target detection network is used to extract the lane line region of interest and the vehicle region of interest from the road surveillance image, respectively. This is essentially performing image semantic segmentation on the road surveillance image to extract the regions of interest of different objects in the road surveillance image. In a specific technical solution of the present application, the lane line target detection network and the vehicle target detection network can be anchor window-based target detection networks, such as Fast R-CNN, Faster R-CNN, or RetinaNet.

[0060] After performing image semantic recognition to obtain the lane line region of interest and the vehicle region of interest, the lane line region of interest is passed through a first convolutional neural network model using a spatial attention mechanism to obtain a lane line feature map, while the vehicle region of interest is passed through a second convolutional neural network model using a spatial attention mechanism to obtain a vehicle feature map. That is, a convolutional neural network model with excellent performance in the image feature extraction neighborhood is used to extract high-dimensional local implicit features of the lane line region of interest and the vehicle region of interest. Those skilled in the art should know that a convolutional neural network model uses a convolution kernel with learnable parameters to perceive local features of an image. In particular, in the embodiment of the present application, considering that the amount of information contained in the pixel values ​​at each position in the lane line region of interest and the vehicle region of interest is different, in order to fully utilize the information difference of the image pixels in the spatial dimension, the spatial attention mechanism is integrated into the convolutional neural network model to obtain the first convolutional neural network model and the second convolutional neural network model.

[0061] Furthermore, the lane feature map and the vehicle feature map are fused to obtain a classification feature map. The classification feature map is used to represent the interaction between the lane feature map and the vehicle feature map in the high-dimensional feature space, that is, the high-dimensional implicit interaction pattern characteristics between the vehicle and the road. At the data level, the classification feature map can be obtained by calculating the position-weighted sum between the lane feature map and the vehicle feature map. Then, the classification feature map is passed through a classifier to obtain a classification result, and it is determined whether to generate a light warning prompt for surrounding traffic participants based on the classification result. That is, the classifier is used to determine the class probability label to which the classification feature map belongs, and the class probability label includes generating a light warning prompt for surrounding traffic participants (first label) and not generating a light warning prompt for surrounding traffic participants (second label).

[0062] However, in the process of obtaining a classification feature map by fusing the lane feature map and the vehicle feature map according to position points, there is a semantic mismatch between the feature distribution of the lane feature map and the feature distribution of the vehicle feature map, which will affect the expression certainty and accuracy of the interaction features between the lane feature map and the vehicle feature map, and thus affect the accuracy of the classification result of the classification feature map through the classifier.

[0063] Therefore, preferably, in the technical solution of the present application, domain adaptive class graph topology fusion is performed on the lane feature map and the vehicle feature map:

[0064]

[0065] In this way, the characteristic values ​​of each position in the feature distribution of the lane feature map and the vehicle feature map are used as nodes, and the information interpretation of the spatial distance between the corresponding positions between the feature distributions is used as the edge to achieve robust information interaction and propagation between different feature domain maps along the predetermined direction of the feature distribution, so that the fused classification feature map not only has pixel-level feature sparsity and non-network properties, but also has relatively good feature density, thereby maximizing the alignment and aggregation of the distributions of different types of targets in the lane feature map and the vehicle feature map, so as to improve the classification accuracy of the classification feature map.

[0066] Based on this, the present application provides an intelligent driving warning system based on road condition recognition, which includes: a monitoring module for acquiring a road monitoring image collected by a camera deployed at the front or rear of a vehicle; a lane line detection module for passing the road monitoring image through a lane line target detection network to obtain a lane line region of interest; a vehicle detection module for passing the road monitoring image through a vehicle target detection network to obtain a vehicle region of interest; a lane line feature extraction module for passing the lane line region of interest through a first convolutional neural network model using a spatial attention mechanism to obtain a lane line feature map; a vehicle feature extraction module for passing the vehicle region of interest through a second convolutional neural network model using a spatial attention mechanism to obtain a vehicle feature map; an interaction module for fusing the lane line feature map and the vehicle feature map to obtain a classification feature map; and a warning result generation module for passing the classification feature map through a classifier to obtain a classification result, and the classification result is used to indicate whether a light warning prompt is generated for surrounding traffic participants.

[0067] Figure 1 FIG is an application scenario diagram of an intelligent driving warning system based on road condition recognition according to an embodiment of the present application. Figure 1 As shown, in this application scenario, first, the camera deployed at the front of the vehicle (for example, Figure 1 C) collected road monitoring images (e.g., Figure 1 Then, the road monitoring image is input into a server that is equipped with an intelligent driving warning algorithm based on road condition recognition (for example, Figure 1 S) as shown in , wherein the server is capable of using the intelligent driving warning algorithm based on road condition recognition to process the road monitoring image to obtain a classification result for determining whether to generate a light warning prompt for surrounding traffic participants.

[0068] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0069] Exemplary Systems

[0070] Figure 2 FIG is a block diagram of an intelligent driving warning system based on road condition recognition according to an embodiment of the present application. Figure 2 As shown, according to an embodiment of the present application, an intelligent driving warning system 100 based on road condition recognition includes: a monitoring module 110, which is used to obtain a road monitoring image collected by a camera deployed at the front of a vehicle; a lane line detection module 120, which is used to pass the road monitoring image through a lane line target detection network to obtain a lane line region of interest; a vehicle detection module 130, which is used to pass the road monitoring image through a vehicle target detection network to obtain a vehicle region of interest; a lane line feature extraction module 140, which is used to pass the lane line region of interest through a first convolutional neural network model using a spatial attention mechanism to obtain a lane line feature map; a vehicle feature extraction module 150, which is used to pass the vehicle region of interest through a second convolutional neural network model using a spatial attention mechanism to obtain a vehicle feature map; an interaction module 160, which is used to fuse the lane line feature map and the vehicle feature map to obtain a classification feature map; and a warning result generation module 170, which is used to pass the classification feature map through a classifier to obtain a classification result, and determine whether to generate a light warning prompt for surrounding traffic participants based on the classification result.

[0071] More specifically, in this embodiment of the present application, the monitoring module 110 is configured to acquire road monitoring images captured by a camera deployed at the front of the vehicle. It will be appreciated that existing vehicles often have cameras deployed at multiple locations. Therefore, the camera deployed at the front of the vehicle can be used to capture road conditions along the vehicle's travel path, i.e., the road monitoring images are captured by the camera deployed at the front of the vehicle. Current autonomous driving technology lacks the ability to proactively alert traffic participants around the vehicle.

[0072] In the technical solution of the present application, an active reminder scheme for traffic participants around the main vehicle is constructed through machine vision and artificial intelligence-based image processing technology, so as to actively identify the encroachment of surrounding traffic participants on the main vehicle's driving path, and actively send signals, for example, through actuators such as lights and horns to remind surrounding traffic participants. This can effectively remind surrounding traffic participants to make reasonable avoidance and improve road safety; at the same time, it can effectively remind surrounding traffic participants to make reasonable avoidance and improve the driving efficiency of the main vehicle; and it can reduce the burden on the main vehicle driver, so that the main vehicle driver can have a more relaxed driving experience.

[0073] More specifically, in the embodiment of the present application, the lane line detection module 120 is used to pass the road monitoring image through a lane line target detection network to obtain a lane line region of interest.

[0074] More specifically, in the embodiment of the present application, the vehicle detection module 130 is configured to pass the road monitoring image through a vehicle target detection network to obtain a vehicle region of interest.

[0075] That is, the target detection network is used to extract the lane line region of interest and the vehicle region of interest from the road monitoring image respectively. This is essentially performing image semantic segmentation on the road monitoring image to extract the regions of interest of different objects in the road monitoring image.

[0076] Accordingly, in a specific example, the lane line target detection network and the vehicle target detection network are anchor window-based target detection networks, and the anchor window-based target detection network can be Fast R-CNN, Faster R-CNN or RetinaNet.

[0077] After performing image semantic recognition to obtain the lane line region of interest and the vehicle region of interest, the lane line region of interest is passed through a first convolutional neural network model using a spatial attention mechanism to obtain a lane line feature map, and the vehicle region of interest is passed through a second convolutional neural network model using a spatial attention mechanism to obtain a vehicle feature map.

[0078] More specifically, in this embodiment of the present application, the lane feature extraction module 140 is configured to extract the lane ROI using a first convolutional neural network model using a spatial attention mechanism to obtain a lane feature map. A convolutional neural network model with excellent performance in image feature extraction is used to extract high-dimensional local implicit features of the lane ROI. Those skilled in the art will appreciate that a convolutional neural network model uses a convolution kernel with learnable parameters to perceive local features in an image.

[0079] In particular, in an embodiment of the present application, taking into account the different amounts of information contained in the pixel values ​​at various positions in the lane line area of ​​interest, in order to fully utilize the information differences of image pixels in the spatial dimension, the spatial attention mechanism is integrated into the convolutional neural network model to obtain the first convolutional neural network model.

[0080] Accordingly, in a specific example, Figure 3As shown, the lane line feature extraction module 140 includes: a lane line convolutional coding unit 141, which is used to perform deep convolution coding on the lane line region of interest using the convolutional coding part of the first convolutional neural network model using the spatial attention mechanism to obtain a lane line high-dimensional feature map; a lane line spatial attention unit 142, which is used to input the lane line high-dimensional feature map into the spatial attention part of the first convolutional neural network model using the spatial attention mechanism to obtain a lane line spatial attention map; and a lane line spatial attention application unit 143, which is used to respectively calculate the position point multiplication of the lane line high-dimensional feature map and the lane line spatial attention map to obtain the lane line feature map.

[0081] More specifically, in the embodiment of the present application, the vehicle feature extraction module 150 is used to obtain a vehicle feature map by using a second convolutional neural network model with a spatial attention mechanism for the vehicle region of interest. A convolutional neural network model with excellent performance in the image feature extraction neighborhood is used to extract the high-dimensional image local implicit features of the vehicle region of interest. Those skilled in the art should know that the convolutional neural network model uses a convolution kernel with learnable parameters to perceive local features of the image. In particular, in the embodiment of the present application, considering that the amount of information contained in the pixel values ​​at each position in the vehicle region of interest is different, in order to make full use of the information difference of the image pixels in the spatial dimension, the spatial attention mechanism is integrated into the convolutional neural network model to obtain the second convolutional neural network model.

[0082] Accordingly, in a specific example, Figure 4 As shown, the vehicle feature extraction module 150 includes: a vehicle convolutional coding unit 151, which is used to perform deep convolution coding on the vehicle region of interest using the convolutional coding part of the second convolutional neural network model using the spatial attention mechanism to obtain a vehicle high-dimensional feature map; a vehicle spatial attention unit 152, which is used to input the vehicle high-dimensional feature map into the spatial attention part of the second convolutional neural network model using the spatial attention mechanism to obtain a vehicle spatial attention map; and a vehicle spatial attention applying unit 153, which is used to respectively calculate the position point multiplication of the vehicle high-dimensional feature map and the vehicle spatial attention map to obtain the vehicle feature map.

[0083] More specifically, in this embodiment of the present application, the interaction module 160 is configured to fuse the lane feature map and the vehicle feature map to obtain a classification feature map. The classification feature map is used to represent the interaction between the lane feature map and the vehicle feature map in a high-dimensional feature space, i.e., the high-dimensional implicit interaction pattern characteristics between the vehicle and the road. At the data level, the classification feature map can be obtained by calculating the position-weighted sum of the lane feature map and the vehicle feature map.

[0084] However, in the process of obtaining a classification feature map by fusing the lane feature map and the vehicle feature map at each location point, there is a semantic mismatch between the feature sets of different categories in the feature distribution of the lane feature map and the feature distribution of the vehicle feature map. This will affect the expression certainty and accuracy of the interactive features between the lane feature map and the vehicle feature map, and thus affect the accuracy of the classification result of the classification feature map by the classifier. Therefore, preferably, in the technical solution of the present application, the lane feature map and the vehicle feature map are subjected to domain-adaptive class graph topological fusion.

[0085] Accordingly, in a specific example, the interaction module 160 is further configured to: fuse the lane feature map and the vehicle feature map using the following formula to obtain the classification feature map; wherein the formula is:

[0086]

[0087] Among them, F i Represents the lane feature map, F j represents the vehicle feature map, F c represents the classification feature map, α and β represent weight hyperparameters, exp(·) represents the exponential operation of the feature map, and the exponential operation on the feature map represents the natural exponential function value raised to the power of the eigenvalues ​​at each position in the feature map.

[0088] In this way, the characteristic values ​​of each position in the feature distribution of the lane feature map and the vehicle feature map are used as nodes, and the information interpretation of the spatial distance between the corresponding positions between the feature distributions is used as the edge to achieve robust information interaction and propagation between different feature domain maps along the predetermined direction of the feature distribution, so that the fused classification feature map not only has pixel-level feature sparsity and non-network properties, but also has relatively good feature density, thereby maximizing the alignment and aggregation of the distributions of different types of targets in the lane feature map and the vehicle feature map, so as to improve the classification accuracy of the classification feature map.

[0089] More specifically, in this embodiment of the present application, the warning result generation module 170 is configured to pass the classification feature graph through a classifier to obtain a classification result, wherein the classification result is used to indicate whether a light warning prompt for surrounding traffic participants is generated. In other words, the classifier is used to determine a class probability label to which the classification feature graph belongs, wherein the class probability label includes a first label indicating that a light warning prompt for surrounding traffic participants is generated, and a second label indicating that a light warning prompt for surrounding traffic participants is not generated.

[0090] Accordingly, in a specific example, the warning result generation module 170 is further used to: expand the classification feature map into a classification feature vector according to a row vector or a column vector; use multiple fully connected layers of the classifier to fully connect encode the classification feature vector to obtain an encoded classification feature vector; and pass the encoded classification feature vector through the Softmax classification function of the classifier to obtain the classification result.

[0091] In summary, the intelligent driving warning system 100 based on road condition recognition according to the embodiment of the present application is explained, which first passes the road monitoring image through the lane line target detection network to obtain the lane line region of interest, passes the road monitoring image through the vehicle target detection network to obtain the vehicle region of interest, then passes the lane line region of interest through the first convolutional neural network model using the spatial attention mechanism to obtain a lane line feature map, passes the vehicle region of interest through the second convolutional neural network model using the spatial attention mechanism to obtain a vehicle feature map, then, fuses the lane line feature map and the vehicle feature map to obtain a classification feature map, and finally, passes the classification feature map through a classifier to obtain a classification result for indicating whether a light warning prompt is generated for surrounding traffic participants. In this way, road safety can be improved while improving traffic efficiency.

[0092] As described above, the intelligent driving warning system 100 based on road condition recognition according to the embodiment of the present application can be implemented in various terminal devices, such as a server having an intelligent driving warning algorithm based on road condition recognition. In one example, the intelligent driving warning system 100 based on road condition recognition can be integrated into the terminal device as a software module and / or a hardware module. For example, the intelligent driving warning system 100 based on road condition recognition can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the intelligent driving warning system 100 based on road condition recognition can also be one of the many hardware modules of the terminal device.

[0093] Alternatively, in another example, the intelligent driving warning system 100 based on road condition recognition and the terminal device may also be separate devices, and the intelligent driving warning system 100 based on road condition recognition may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0094] Exemplary Methods

[0095] Figure 5 Flowchart of the intelligent driving warning method based on road condition recognition according to the embodiment of the present application. Figure 5As shown, according to the intelligent driving warning method based on road condition recognition according to an embodiment of the present application, it includes: S110, obtaining a road monitoring image collected by a camera deployed at the front of a vehicle; S120, passing the road monitoring image through a lane line target detection network to obtain a lane line region of interest; S130, passing the road monitoring image through a vehicle target detection network to obtain a vehicle region of interest; S140, passing the lane line region of interest through a first convolutional neural network model using a spatial attention mechanism to obtain a lane line feature map; S150, passing the vehicle region of interest through a second convolutional neural network model using a spatial attention mechanism to obtain a vehicle feature map; S160, fusing the lane line feature map and the vehicle feature map to obtain a classification feature map; and, S170, passing the classification feature map through a classifier to obtain a classification result, and determining whether to generate a light warning prompt for surrounding traffic participants based on the classification result.

[0096] Figure 6 Schematic diagram of the system architecture of the intelligent driving warning method based on road condition recognition according to an embodiment of the present application. Figure 6 As shown, in the system architecture of the intelligent driving warning method based on road condition recognition, first, a road monitoring image collected by a camera deployed at the front of a vehicle is obtained; then, the road monitoring image is passed through a lane line target detection network to obtain a lane line region of interest; then, the road monitoring image is passed through a vehicle target detection network to obtain a vehicle region of interest; then, the lane line region of interest is passed through a first convolutional neural network model using a spatial attention mechanism to obtain a lane line feature map; then, the vehicle region of interest is passed through a second convolutional neural network model using a spatial attention mechanism to obtain a vehicle feature map; then, the lane line feature map and the vehicle feature map are fused to obtain a classification feature map; finally, the classification feature map is passed through a classifier to obtain a classification result, and the classification result is used to determine whether to generate a light warning prompt for surrounding traffic participants.

[0097] In a specific example, in the above-mentioned intelligent driving warning method based on road condition recognition, the lane line target detection network and the vehicle target detection network are target detection networks based on anchor windows.

[0098] In a specific example, in the above-mentioned intelligent driving warning method based on road condition recognition, the anchor window-based target detection network is Fast R-CNN, Faster R-CNN or RetinaNet.

[0099] In a specific example, in the above-mentioned intelligent driving warning method based on road condition recognition, the lane line region of interest is passed through the first convolutional neural network model using the spatial attention mechanism to obtain a lane line feature map, including: using the convolution coding part of the first convolutional neural network model using the spatial attention mechanism to perform deep convolution encoding on the lane line region of interest to obtain a lane line high-dimensional feature map; inputting the lane line high-dimensional feature map into the spatial attention part of the first convolutional neural network model using the spatial attention mechanism to obtain a lane line spatial attention map; and, respectively calculating the position point multiplication of the lane line high-dimensional feature map and the lane line spatial attention map to obtain the lane line feature map.

[0100] In a specific example, in the above-mentioned intelligent driving warning method based on road condition recognition, the vehicle region of interest is passed through a second convolutional neural network model using a spatial attention mechanism to obtain a vehicle feature map, including: using the convolution coding part of the second convolutional neural network model using the spatial attention mechanism to perform deep convolution coding on the vehicle region of interest to obtain a vehicle high-dimensional feature map; inputting the vehicle high-dimensional feature map into the spatial attention part of the second convolutional neural network model using the spatial attention mechanism to obtain a vehicle spatial attention map; and, respectively calculating the position point multiplication of the vehicle high-dimensional feature map and the vehicle spatial attention map to obtain the vehicle feature map.

[0101] In a specific example, in the above-mentioned intelligent driving warning method based on road condition recognition, the fusion of the lane line feature map and the vehicle feature map to obtain a classification feature map further includes: fusing the lane line feature map and the vehicle feature map to obtain the classification feature map using the following formula; wherein the formula is:

[0102]

[0103] Among them, F i Represents the lane feature map, F j represents the vehicle feature map, F c represents the classification feature map, α and β represent weight hyperparameters, exp(·) represents the exponential operation of the feature map, and the exponential operation on the feature map represents the natural exponential function value raised to the power of the eigenvalues ​​at each position in the feature map.

[0104] In a specific example, in the above-mentioned intelligent driving warning method based on road condition recognition, the classification feature map is passed through a classifier to obtain a classification result, and whether to generate a light warning prompt for surrounding traffic participants is determined based on the classification result, including: expanding the classification feature map into a classification feature vector according to a row vector or a column vector; using multiple fully connected layers of the classifier to fully connect encode the classification feature vector to obtain an encoded classification feature vector; and passing the encoded classification feature vector through the Softmax classification function of the classifier to obtain the classification result.

[0105] Here, those skilled in the art will appreciate that the specific operations of each step in the above-mentioned intelligent driving warning method based on road condition recognition have been described in the above reference. Figures 1 to 4 It has been introduced in detail in the description of the intelligent driving warning system based on road condition recognition, and therefore, its repeated description will be omitted.

[0106] The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this application are merely illustrative and not restrictive, and it should not be assumed that these advantages, strengths, and effects are required of each embodiment of this application. In addition, the specific details disclosed above are merely illustrative and facilitating understanding, and are not restrictive. The above details do not limit this application to necessarily being implemented using the above specific details.

[0107] The block diagrams of the devices, devices, equipment, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0108] It should also be noted that in the apparatus, device, and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0109] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0110] The above description has been provided for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. An intelligent driving warning system based on road condition recognition, characterized in that: include: A monitoring module, configured to obtain road monitoring images collected by a camera deployed on a vehicle; A lane detection module is used to pass the road monitoring image through a lane target detection network to obtain a lane region of interest; a vehicle detection module, configured to pass the road monitoring image through a vehicle target detection network to obtain a vehicle region of interest; A lane feature extraction module is configured to extract the lane region of interest through a first convolutional neural network model using a spatial attention mechanism to obtain a lane feature map; A vehicle feature extraction module, configured to extract the vehicle region of interest through a second convolutional neural network model using a spatial attention mechanism to obtain a vehicle feature map; An interaction module, configured to fuse the lane feature map and the vehicle feature map to obtain a classification feature map; as well as The warning result generation module is used to pass the classification feature map through the classifier to obtain the classification result. determining whether to generate a light warning prompt for surrounding traffic participants based on the classification result; The interaction module is further configured to: fuse the lane feature map and the vehicle feature map using the following formula to obtain the classification feature map; Wherein, the formula is: in, represents the lane feature map, represents the vehicle feature map, represents the classification feature map, and represents the weight hyperparameter, Represents the exponential operation of the feature map. The exponential operation on the feature map represents the natural exponential function value raised to the power of the eigenvalues ​​at each position in the feature map.

2. The intelligent driving warning system based on road condition recognition according to claim 1 is characterized in that: The lane line target detection network and the vehicle target detection network are target detection networks based on anchor windows.

3. The intelligent driving warning system based on road condition recognition according to claim 2 is characterized in that: The anchor window-based object detection network is Fast R-CNN, Faster R-CNN or RetinaNet.

4. The intelligent driving warning system based on road condition recognition according to claim 3 is characterized in that: The lane feature extraction module includes: A lane line convolutional coding unit, configured to perform deep convolutional coding on the lane line region of interest using the convolutional coding portion of the first convolutional neural network model using the spatial attention mechanism to obtain a lane line high-dimensional feature map; a lane line spatial attention unit, configured to input the lane line high-dimensional feature map into the spatial attention part of the first convolutional neural network model using the spatial attention mechanism to obtain a lane line spatial attention map; and The lane line spatial attention applying unit is used to respectively calculate the lane line high-dimensional feature map and the lane line spatial attention map by multiplying the position points to obtain the lane line feature map.

5. The intelligent driving warning system based on road condition recognition according to claim 4 is characterized in that: The vehicle feature extraction module includes: A vehicle convolutional coding unit, configured to perform deep convolutional coding on the vehicle region of interest using the convolutional coding portion of the second convolutional neural network model using the spatial attention mechanism to obtain a high-dimensional feature map of the vehicle; a vehicle spatial attention unit, configured to input the vehicle high-dimensional feature map into the spatial attention part of the second convolutional neural network model using the spatial attention mechanism to obtain a vehicle spatial attention map; and A vehicle spatial attention applying unit is used to respectively calculate the point-by-point multiplication of the vehicle high-dimensional feature map and the vehicle spatial attention map to obtain the vehicle feature map.

6. The intelligent driving warning system based on road condition recognition according to claim 1 is characterized in that: The early warning result generation module is further used to: Expanding the classification feature map into a classification feature vector according to row vectors or column vectors; Performing full-connection encoding on the classification feature vector using multiple fully-connected layers of the classifier to obtain an encoded classification feature vector; as well as The encoded classification feature vector is passed through the Softmax classification function of the classifier to obtain the classification result.

7. An intelligent driving warning method based on road condition recognition, characterized in that: include: Acquiring road monitoring images collected by cameras deployed on vehicles; Passing the road monitoring image through a lane line target detection network to obtain a lane line region of interest; Passing the road monitoring image through a vehicle target detection network to obtain a vehicle region of interest; The lane line region of interest is passed through a first convolutional neural network model using a spatial attention mechanism to obtain a lane line feature map; The vehicle region of interest is passed through a second convolutional neural network model using a spatial attention mechanism to obtain a vehicle feature map; fusing the lane feature map and the vehicle feature map to obtain a classification feature map; as well as Passing the classification feature graph through a classifier to obtain a classification result, and determining whether to generate a light warning prompt for surrounding traffic participants based on the classification result; The lane feature map and the vehicle feature map are fused using the following formula to obtain the classification feature map: Wherein, the formula is: in, represents the lane feature map, represents the vehicle feature map, represents the classification feature map, and represents the weight hyperparameter, Represents the exponential operation of the feature map. The exponential operation on the feature map represents the natural exponential function value raised to the power of the eigenvalues ​​at each position in the feature map.

8. The intelligent driving warning method based on road condition recognition according to claim 7 is characterized in that: The lane line target detection network and the vehicle target detection network are target detection networks based on anchor windows.

9. The intelligent driving warning method based on road condition recognition according to claim 8 is characterized in that: The anchor window-based object detection network is Fast R-CNN, Faster R-CNN or RetinaNet.

Images