Road safety risk identification and early warning system and method based on image semantic segmentation

Abstract

The present invention relates to the technical field of road safety early warning technology, and specifically to a road safety risk identification and early warning system and method based on image semantic segmentation. The system comprises: an image acquisition module for acquiring a real-time panoramic image of the driving vehicle's surroundings; a feature extraction module for extracting multi-scale color and texture features and multi-scale spatial features from the real-time panoramic image; a cross-modal relationship enhancement module for constructing a cross-modal feature relationship matrix and dynamically fusing multi-scale features based on the cross-modal feature relationship matrix to obtain multi-modal fusion features; a semantic segmentation module for performing a hierarchical analysis of the multi-level detection problem of road safety from the perspective of task decomposition and detecting multi-modal fusion features; and a safety early warning module for providing graded early warnings for road safety. The present invention can simultaneously provide high-precision recognition of multi-scale and multi-category objects on the road, thereby improving the accuracy of detecting road anomalies during driving.

Description

Technical Field

[0001] The present invention relates to the field of road safety warning technology, and more particularly to a road safety risk identification and warning system and method based on image semantic segmentation. Background Art

[0002] With the continued growth of motor vehicles and the development of intelligent transportation systems and autonomous driving technologies, people's demand for road safety is growing. Predicting and warning road anomalies can effectively reduce traffic accidents and improve driving safety. In complex traffic environments, timely identification and warning of potential road anomalies can greatly avoid potential dangers.

[0003] Environmental perception is the first step in realizing intelligent driving. Intelligent vehicles obtain information about their surroundings through various sensors such as cameras, millimeter-wave radars, ultrasonic radars, and lidars. The detection and recognition of obstacle and road structured data is an important research topic in environmental perception, which can be achieved through semantic segmentation of structured roads based on deep learning.

[0004] Existing semantic segmentation methods for road images only perform well for large-scale obstacle classifications, often overlooking smaller obstacles (such as road nails and cracks). Furthermore, existing methods generally focus on the spatial information of the data and fail to implement adaptive feature enhancement for different regions. This leads to confusion in the segmentation results for objects with highly similar spatial information and results in low segmentation accuracy. Furthermore, existing methods often directly output the type of safety hazard associated with the current road condition, failing to conduct multi-level, multi-task road safety assessments. Consequently, existing deep learning models are unable to comprehensively consider multi-scale road surface objects and cannot distinguish obstacles whose spatial or color textures are easily confused. This makes them unable to effectively address road anomalies caused by road construction, traffic accidents, or other emergencies, and their detection accuracy urgently needs to be improved. Summary of the Invention

[0005] In view of this, the present invention provides a road safety risk identification and warning system and method based on image semantic segmentation, which can take into account high-precision identification of multi-scale and multi-category objects on the road, and improve the detection accuracy of road anomalies during driving.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a road safety risk identification and warning system based on image semantic segmentation, comprising: an image acquisition module, a feature extraction module, a cross-modal relationship enhancement module, a semantic segmentation module, and a safety warning module;

[0008] The image acquisition module is used to acquire a real-time panoramic image of a preset range around the driving vehicle;

[0009] The feature extraction module is used to extract multi-scale color texture features and multi-scale spatial features in the real-time panoramic image;

[0010] The cross-modal relationship enhancement module is used to calculate the cosine similarity between color texture features and spatial features at multiple scales, construct a cross-modal feature relationship matrix, and dynamically fuse multi-scale features according to the cross-modal feature relationship matrix to obtain multi-modal fusion features;

[0011] The semantic segmentation module performs a hierarchical analysis on the multi-level detection problem of road safety from the perspective of task decomposition, detects the multimodal fusion features, and obtains road safety detection results;

[0012] The safety warning module is used to provide graded warnings on road safety based on road safety detection results.

[0013] Furthermore, the feature extraction module includes a first feature extraction branch and a second feature extraction branch, wherein the first feature extraction branch captures semantic information, context information, and local detail information of the real-time panoramic image at different scales as multi-scale spatial features through dense convolution and residual connection;

[0014] The second feature extraction branch captures color information and texture information of the real-time panoramic image at different scales as multi-scale color and texture features by combining a color histogram and a gray-level co-occurrence matrix.

[0015] Furthermore, the first feature extraction branch includes a residual network unit, a dense convolution block, a multi-scale spatial fusion unit and an attention guidance unit;

[0016] The residual network unit is used to downsample the real-time panoramic image and extract intermediate feature maps at different scales;

[0017] The dense convolution block is composed of a plurality of depth-wise separable convolutional layers and residual connections with the same dilation rate, and is used to extract and transform the real-time panoramic image to obtain low-level features of the real-time panoramic image;

[0018] The multi-scale spatial fusion unit is used to perform matrix element addition on the features output by the residual network unit and the features output by the dense convolution block in the same dimension;

[0019] The attention guidance unit is used to perform multi-angle screening and aggregation on the features output by the multi-scale spatial fusion unit from the channel level, pixel level and global level to obtain the final extracted multi-scale spatial features.

[0020] Furthermore, the second feature extraction branch includes a scale division unit, a color extraction unit, a texture extraction unit and a multi-scale color and texture fusion unit;

[0021] The scale division unit is used to divide the real-time panoramic image into images of different scales;

[0022] The color extraction unit is used to convert images of different scales from RGB space to HSV space, calculate the color distribution of the image, and form the color features of the image;

[0023] The texture extraction unit is used to convert images of different scales into grayscale images, calculate the local pixel relationship value of the grayscale image, and calculate the histogram of the grayscale relationship of the grayscale image to obtain the LBP texture feature of the image;

[0024] The multi-scale color and texture fusion unit is used to fuse the color features and LBP texture features of each scale image to obtain multi-scale color and texture features.

[0025] Furthermore, the cross-modal relationship enhancement module includes a relationship matrix construction unit and a dynamic fusion unit;

[0026] The relationship matrix construction unit is used to respectively splice the spatial features and color and texture features of the real-time panoramic image at multiple scales to obtain a spatial feature splicing result and a color and texture feature splicing result, and then construct a cross-modal feature relationship matrix based on the cosine similarity between the spatial feature splicing result and the color and texture feature splicing result;

[0027] The dynamic fusion unit is used to identify high-similarity regions, medium-similarity regions, and low-similarity regions based on the cross-modal feature relationship matrix; if the cosine similarity between the spatial feature stitching result and the color and texture feature stitching result of a certain region in the real-time panoramic image is between 0.8 and 1.0, the current region is considered to be a high-similarity region. For the high-similarity region, the recognition of color-sensitive targets is suitable, and the color and texture feature stitching result is used as the dominant factor;

[0028] If the cosine similarity between the spatial feature stitching result and the color and texture feature stitching result of a certain area in the real-time panoramic image is between 0.3 and 0.8, the current area is regarded as a medium similarity area, and for the medium similarity area, the stitching results of the two modal features are stitched or weighted fused;

[0029] If the cosine similarity between the spatial feature stitching result and the color and texture feature stitching result of a certain area in the real-time panoramic image is less than 0.3, the current area is regarded as a low-similarity area. For the low-similarity area, it is suitable for the recognition of geometric feature targets, with the spatial feature stitching result as the dominant factor.

[0030] Furthermore, the expression of the cross-modal feature relationship matrix is:

[0031] ;

[0032] ;

[0033] in, , represents the cross-modal feature relationship matrix, whose dimension is ; ; represents the inner product, represents the L2 norm; Represents the i-th row vector of the cross-modal feature relationship matrix; Represents the j-th row vector of the cross-modal feature relationship matrix; Represents the result of spatial feature splicing; Indicates the result of color and texture feature splicing; Indicates the dimension of the feature concatenation result, represents the length of the tensor, Represents the tensor width, and the tensor length is equivalent to the number of sampling points, that is, Each point has a length of The vector describes the features.

[0034] Furthermore, the semantic segmentation module divides the multi-level detection problem of road safety into three subtasks, each of which corresponds to a prediction network;

[0035] The prediction network corresponding to the first subtask identifies and segments the object categories contained in the road in the panoramic image around the vehicle;

[0036] The prediction network corresponding to the second subtask identifies whether the identified object category will pose a safety hazard to road safety;

[0037] The prediction network corresponding to the third subtask detects the level of road safety hazards.

[0038] Furthermore, the loss functions of the three prediction networks are expressed as:

[0039] ;

[0040] ;

[0041] ;

[0042] in, Indicates the total number of pixels in the panoramic image, 、 、 Represents the labels of the three subtasks, 、 、 Represent the parameters of the three prediction networks, Represents the i-th pixel in the input multimodal fusion feature; 、 、 Represent the prediction results of the three prediction networks respectively;

[0043] The losses of the three prediction networks are weighted and summed as the final loss of the semantic segmentation module:

[0044] ;

[0045] in, 、 、 Represent the weights of the three losses respectively.

[0046] Furthermore, the object categories identified by the prediction network corresponding to the first subtask include potholes, road water, road reflections, overpass reflections, low-altitude floating objects, traffic signs, road cracks, small sharp objects on the road, dynamic obstacles, static obstacles, accident areas and construction areas; the prediction network corresponding to the second subtask analyzes the location of the objects identified by the first prediction network to predict whether there are safety hazards; the prediction network corresponding to the third subtask comprehensively analyzes the safety hazard judgment results of each object category output by the second prediction network, and outputs the safety hazard level, including normal passage, mild safety hazard, moderate safety hazard and severe safety hazard.

[0047] In a second aspect, the present invention provides a road safety risk identification and warning method based on image semantic segmentation, which is applicable to the system described above and includes the following steps:

[0048] Obtain real-time panoramic images of a preset range around the driving vehicle;

[0049] Extracting multi-scale color texture features and multi-scale spatial features from the real-time panoramic image;

[0050] Calculating the cosine similarity between color texture features and spatial features at multiple scales, constructing a cross-modal feature relationship matrix, and dynamically fusing multi-scale features based on the cross-modal feature relationship matrix to obtain multi-modal fusion features;

[0051] Conduct a hierarchical analysis of the multi-level road safety detection problem from the perspective of task decomposition, detect the multimodal fusion features, and obtain road safety detection results;

[0052] Based on the results of road safety inspections, graded warnings are issued for road safety.

[0053] It can be seen from the above technical solutions that compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. This invention fully utilizes the spatial and color-texture features of road images and constructs a cross-modal feature relationship matrix. It considers the spatial and color-texture relationships of various regions at different scales, enabling the model to distinguish objects that are easily confused by space or color texture, and adaptively fuses features from different regions to enhance the dominance of spatial or color-texture features, ensuring accurate recognition of road obstacles.

[0055] 2. By extracting multi-scale features from the panoramic image around the vehicle, the present invention can take into account multi-scale obstacles or road defects in the positioning area without ignoring small-scale objects, making regional object recognition more complete and ensuring more accurate recognition of small-scale obstacles or sharp objects on the road, thereby ensuring driving safety.

[0056] 3. By performing multi-task detection on road safety issues, the present invention effectively reduces the deviation caused by a single network for multi-task detection, simplifies the learning objectives, and ensures prediction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0058] Figure 1 A schematic diagram of the structure of the road safety risk identification and early warning system based on image semantic segmentation provided by the present invention;

[0059] Figure 2 A schematic structural diagram of the first feature extraction branch provided by the present invention;

[0060] Figure 3 A schematic structural diagram of the second feature extraction branch provided by the present invention;

[0061] Figure 4 This is a schematic diagram of the structure of the cross-modal relationship enhancement module provided by the present invention;

[0062] Figure 5 A schematic diagram of the structure of the semantic segmentation module provided by the present invention;

[0063] Figure 6 This is a flowchart of the road safety risk identification and early warning method based on image semantic segmentation provided by the present invention. DETAILED DESCRIPTION

[0064] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0065] like Figure 1 As shown, an embodiment of the present invention discloses a road safety risk identification and warning system based on image semantic segmentation, comprising: an image acquisition module, a feature extraction module, a cross-modal relationship enhancement module, a semantic segmentation module, and a safety warning module;

[0066] The image acquisition module is used to acquire a real-time panoramic image of a preset range around the driving vehicle;

[0067] The feature extraction module is used to extract multi-scale color texture features and multi-scale spatial features from real-time panoramic images;

[0068] The cross-modal relationship enhancement module is used to calculate the cosine similarity between color texture features and spatial features at multiple scales, construct a cross-modal feature relationship matrix, and dynamically fuse multi-scale features based on the cross-modal feature relationship matrix to obtain multi-modal fusion features;

[0069] The semantic segmentation module conducts a hierarchical analysis of the multi-level road safety detection problem from the perspective of task decomposition, detects multimodal fusion features, and obtains road safety detection results; the semantic segmentation module contains three prediction networks: prediction network 1, prediction network 2, and prediction network 3. Among them, prediction network 1 is used to identify the object categories contained in the road, prediction network 2 is used to determine whether each object category contained in the road will cause safety hazards, and prediction network 3 is used to output the safety hazard level.

[0070] The safety warning module is used to provide graded warnings on road safety based on the results of road safety inspections.

[0071] The following further describes the implementation details of the present invention.

[0072] 1) The image acquisition module is used to obtain a panoramic image within a preset range around the current driving vehicle, including road image data and surrounding environment image data.

[0073] 2) The feature extraction module includes a first feature extraction branch and a second feature extraction branch. The first feature extraction branch captures semantic information, contextual information, and local detail information at different scales of the real-time panoramic image through dense convolution and residual connection as multi-scale spatial features;

[0074] The second feature extraction branch captures the color and texture information of the real-time panoramic image at different scales by combining the color histogram and the gray-level co-occurrence matrix as multi-scale color and texture features.

[0075] 2.1) If Figure 2 As shown, the first feature extraction branch includes a residual network unit, a dense convolution block, a multi-scale spatial fusion unit and an attention guidance unit.

[0076] The residual network unit is used to downsample the panoramic image and extract intermediate feature maps at different scales. Specifically, four convolutions with a step size of 1 are used to downsample the panoramic image to extract the multi-scale features of the panoramic image, and four intermediate feature maps r1~r4 are obtained.

[0077] The dense convolution block performs preliminary feature extraction and transformation on the panoramic image to capture low-level features of the panoramic image, thereby obtaining intermediate features and encoding them to obtain low-level features d1 to d4 of the panoramic image. The specific structure of the dense convolution block consists of four depthwise separable convolutional layers with the same dilation rate and residual connections, aiming to expand the receptive field with a small number of parameters.

[0078] The multi-scale spatial fusion unit is used to add the matrix elements of the features r1~r4 output by the residual network unit and the features d1~d4 output by the dense convolution block under the same dimension.

[0079] The dense stream convolution branch focuses on contextual information and detailed features, and the residual network unit provides high-level semantic features. By linking these two branches, the performance and accuracy of the model in image processing tasks can be improved, and the expressiveness and performance of the model can be improved by fusing the features of different branches.

[0080] The attention guidance unit progressively fuses the multi-scale features output by the multi-scale spatial fusion unit at adjacent scales. Specifically, it can fuse features at three scales. The number of scales to be fused can be selected based on the task and model requirements. The three-scale feature fusion strategy aims to balance low-level detail information, mid-level semantic information, and high-level global information. This fusion strategy can balance detail, context, and global perspective to meet the needs of different tasks.

[0081] Specifically, low-level features have a smaller receptive field and can capture local details of the image. Therefore, depth-wise separable convolution can be used for downsampling to reduce the image size and expand the receptive field of each output pixel to capture broader contextual information.

[0082] High-level features contain a wealth of semantic information and can be subjected to dimensionality reduction to reduce the number of feature channels, thereby reducing the number of parameters and computational overhead. Appropriate convolution operations can fuse feature channels with similar semantics, enhancing the semantic expressiveness of high-level features. This helps the model better understand the objects, scenes, and semantic information in the image.

[0083] After the dimensionality reduction operation, deconvolution or transposed convolution is performed to restore the size of the feature map to its original size.

[0084] Finally, the low-level features with spatial information and the high-level features with semantic information are fused through cascade operations and convolution operations to obtain multi-scale features. .

[0085] The attention guidance unit is used to perform multi-angle screening and aggregation on the multi-scale features output by the multi-scale fusion unit from the channel level, pixel level and global level to obtain an enhanced representation of the multi-scale features as the final extracted multi-scale spatial features.

[0086] Specifically, the convolution block-based attention mechanism, the pixel-by-pixel attention mechanism, and the global attention mechanism can be used to filter and aggregate the multi-scale features respectively. Each attention mechanism extracts information at different levels in the multi-scale features, and then the features extracted by each attention mechanism are aggregated to obtain the final multi-scale spatial features, which are finally expressed as:

[0087] ;

[0088] in, represents the Sigmoid activation function, represents the output of the attention mechanism based on the convolutional block, Representation and Input Point-by-point attention features of the same shape, i.e., the output of the pixel-by-pixel attention mechanism, represents the global attention output matrix.

[0089] 2.2) If Figure 3 As shown, the second feature extraction branch includes a scale division unit, a color extraction unit, a texture extraction unit and a multi-scale color and texture fusion unit;

[0090] The scale division unit is used to divide the real-time panoramic image into images of different scales;

[0091] The color extraction unit is used to convert images of different scales from RGB space to HSV space, calculate the color distribution of the image, and form the color features of the image;

[0092] The texture extraction unit is used to convert images of different scales into grayscale images, calculate the local pixel relationship value of the grayscale image, and calculate the histogram of the grayscale relationship of the grayscale image to obtain the LBP texture features of the image; specifically, the grayscale values of P adjacent pixels with a radius of R are g0, g1, ..., g P-1 , and the gray value g of the current pixel in turn c Compare, if g c If the pixel is large, it is set to 1, otherwise it is set to 0.

[0093] The multi-scale color and texture fusion unit is used to fuse the color features and LBP texture features of each scale image to obtain the multi-scale color and texture features. .

[0094] 3) If Figure 4 As shown, the cross-modal relationship enhancement module includes a relationship matrix construction unit and a dynamic fusion unit;

[0095] The relationship matrix construction unit is used to stitch the spatial features and color and texture features of the real-time panoramic image at multiple scales, respectively, to obtain spatial feature stitching results and color and texture feature stitching results, and then construct a cross-modal feature relationship matrix based on the cosine similarity between the spatial feature stitching results and the color and texture feature stitching results;

[0096] The dynamic fusion unit is used to identify high-similarity regions, medium-similarity regions, and low-similarity regions based on the cross-modal feature relationship matrix. For high-similarity regions, the color and texture feature splicing results are dominant. For medium-similarity regions, the splicing results of the two modal features are spliced or weighted fused. For low-similarity regions, the spatial feature splicing results are dominant.

[0097] Specifically, the expression of the cross-modal feature relationship matrix is:

[0098] ;

[0099] ;

[0100] in, , represents the cross-modal feature relationship matrix, whose dimension is ; ; represents the inner product, represents the L2 norm; Represents the i-th row vector of the cross-modal feature relationship matrix; Represents the j-th row vector of the cross-modal feature relationship matrix; Represents the result of spatial feature splicing; Indicates the result of color and texture feature splicing; Indicates the dimension of the feature concatenation result, represents the length of the tensor, Represents the tensor width, and the tensor length is equivalent to the number of sampling points, that is, Each point has a length of The vector describes the features.

[0101] Since the features of the cross-modal feature relationship matrix are calculated to emphasize spatial information and color and texture information respectively, the segmentation result of the present invention takes into account the relationship between cross-modal points, so that the model can more accurately identify small-scale and easily confused objects.

[0102] If the cosine similarity between the spatial feature stitching results and the color and texture feature stitching results of a certain area in the real-time panoramic image is between 0.8 and 1.0, the current area is regarded as a high-similarity area and is suitable for the recognition of color-sensitive targets; if the cosine similarity between the spatial feature stitching results and the color and texture feature stitching results of a certain area in the real-time panoramic image is between 0.3 and 0.8, the current area is regarded as a medium-similarity area; if the cosine similarity between the spatial feature stitching results and the color and texture feature stitching results of a certain area in the real-time panoramic image is less than 0.3, the current area is regarded as a low-similarity area and is suitable for the recognition of geometric feature targets.

[0103] For example, for the reflective road surface and the reflection of the overpass, the calculation result shows that the area belongs to a low-similarity area. At this time, the spatial features are dominant and input into the subsequent modules for prediction to avoid identifying the reflection in the reflective road surface as a real object and avoid identifying the reflection of the overpass as a crack.

[0104] For partially obscured pedestrians, low-altitude drones, inflatable advertising balloons, etc., the calculated result is that the area is a high-similarity area. At this time, color and texture features are used as the main input to the subsequent modules for prediction to avoid blurring the boundary contours of pedestrians, avoiding misidentification of drones as falling objects, and avoiding misidentification of inflatable advertising balloons as pedestrians, so as to avoid category confusion and reduce the false alarm rate.

[0105] For similar areas, a weighted fusion of spatial and color / texture features is performed to dynamically balance color and shape information. For example, for nighttime traffic signs, the weight of color / texture features can be increased to avoid color distortion. For waterlogged areas, the weight of spatial features can be increased to avoid incorrectly expanding road areas.

[0106] For tiny sharp objects or obstacles on the road, such as nails and stones, the spatial features and color texture features are spliced to achieve accurate identification of sharp objects or stones, avoiding vehicle blowouts or collisions due to misidentification.

[0107] In addition, the overall image features can be adjusted according to the specific driving scenario. For night scenes, spatial features are enhanced to avoid reflection interference. For rainy and foggy weather, color and texture features are enhanced to compensate for insufficient visibility.

[0108] 4) If Figure 5 As shown in Figure 2, the semantic segmentation module divides the multi-level detection problem of road safety into three subtasks, each of which corresponds to a prediction network;

[0109] Among them, the first subtask corresponds to the prediction network 1, which identifies and segments the object categories contained in the road in the panoramic image around the vehicle;

[0110] The second subtask corresponds to prediction network 2, which determines whether the identified object category will pose a safety hazard to road safety;

[0111] The third subtask corresponds to prediction network 3, which detects the level of road safety hazards.

[0112] The first subtask is "object category recognition and segmentation", which is a multi-classification task. The prediction network identifies the various object categories in the panoramic image of the current vehicle's surrounding environment and outputs categories 1-n. The identified object categories include one or more of potholes, road water, road reflections, overpass reflections, low-altitude floating objects, traffic signs, road cracks, small sharp objects on the road, dynamic obstacles, static obstacles, accident areas and construction areas.

[0113] The second subtask, "Whether a safety hazard exists," involves multiple output heads, each corresponding to a safety hazard category. This is a binary classification task. Based on the object category output by the first prediction network, a "safety hazard presence" prediction is made, outputting whether a safety hazard exists for categories 1 through n. The second prediction network analyzes the location of objects identified by the first prediction network and predicts whether each category of object poses a safety hazard.

[0114] The prediction network corresponding to the third subtask comprehensively analyzes the safety hazard judgment results of each object category output by the second prediction network and outputs the safety hazard level. It is a four-category problem, and the output safety hazard level includes normal passage, mild safety hazard, moderate safety hazard and severe safety hazard.

[0115] The present invention trains three prediction networks for the three subtasks respectively. Each prediction network only focuses on its own small part of the task, which can effectively reduce the deviation caused by the multi-task coupling part, simplify the learning objectives of the model, and enable the model to better learn the key points of the task.

[0116] Specifically, the three prediction networks share the underlying features at the shallower layers, and learn their own parameters at the deeper layers. Their loss functions are expressed as:

[0117] ;

[0118] ;

[0119] ;

[0120] in, Indicates the total number of pixels in the panoramic image, 、 、 Represents the labels of the three subtasks, 、 、 Represent the parameters of the three prediction networks, Represents the i-th pixel in the input multimodal fusion feature; 、 、 Represent the prediction results of the three prediction networks respectively;

[0121] The losses of the three prediction networks are weighted and summed as the final loss of the semantic segmentation module:

[0122] ;

[0123] in, 、 、 Represents the weights of the three losses. The weights of the three losses are set according to the difficulty of the tasks, or according to the importance of the tasks, or using attention to let the model learn a set of weights.

[0124] 5) The warning module can broadcast warning information in voice form to remind the driver to drive safely, and synchronize the warning information to the user's handheld terminal or vehicle terminal.

[0125] like Figure 6 As shown, an embodiment of the present invention further provides a road safety risk identification and warning method based on image semantic segmentation, which is applicable to the above system and includes the following steps:

[0126] S1. Acquire a real-time panoramic image of a preset range around the driving vehicle;

[0127] S2, extracting multi-scale color texture features and multi-scale spatial features from real-time panoramic images;

[0128] S3. Calculate the cosine similarity between color texture features and spatial features at multiple scales, construct a cross-modal feature relationship matrix, and dynamically fuse multi-scale features based on the cross-modal feature relationship matrix to obtain multi-modal fusion features;

[0129] S4. Conduct a hierarchical analysis of the multi-level road safety detection problem from the perspective of task decomposition, detect multimodal fusion features, and obtain road safety detection results;

[0130] S5. Provide graded warnings for road safety based on the road safety inspection results.

[0131] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0132] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A road safety risk identification and warning system based on image semantic segmentation, characterized in that: include: Image acquisition module, feature extraction module, cross-modal relationship enhancement module, semantic segmentation module and safety warning module; The image acquisition module is used to acquire a real-time panoramic image of a preset range around the driving vehicle; The feature extraction module is used to extract multi-scale color texture features and multi-scale spatial features in the real-time panoramic image; The cross-modal relationship enhancement module is used to calculate the cosine similarity between color texture features and spatial features at multiple scales, construct a cross-modal feature relationship matrix, and dynamically fuse multi-scale features according to the cross-modal feature relationship matrix to obtain multi-modal fusion features; the cross-modal relationship enhancement module includes a relationship matrix construction unit and a dynamic fusion unit; the dynamic fusion unit is used to identify high similarity areas, medium similarity areas, and low similarity areas according to the cross-modal feature relationship matrix; If the cosine similarity between the spatial feature stitching result and the color and texture feature stitching result of a certain area in the real-time panoramic image is between 0.8 and 1.0, the current area is considered a high-similarity area. For a high-similarity area, it is suitable for identifying color-sensitive targets, with the color and texture feature stitching result as the dominant factor. If the cosine similarity between the spatial feature stitching result and the color and texture feature stitching result of a certain area in the real-time panoramic image is between 0.3 and 0.8, the current area is regarded as a medium similarity area, and for the medium similarity area, the stitching results of the two modal features are stitched or weighted fused; If the cosine similarity between the spatial feature stitching result and the color and texture feature stitching result of a certain area in the real-time panoramic image is less than 0.3, the current area is regarded as a low-similarity area. For the low-similarity area, it is suitable for the recognition of geometric feature targets, with the spatial feature stitching result as the main factor; The semantic segmentation module performs a hierarchical analysis on the multi-level detection problem of road safety from the perspective of task decomposition, detects the multimodal fusion features, and obtains road safety detection results; The semantic segmentation module divides the multi-level detection problem of road safety into three subtasks, each corresponding to a prediction network; The prediction network corresponding to the first subtask identifies and segments the object categories contained in the road in the panoramic image around the vehicle; The prediction network corresponding to the second subtask identifies whether the identified object category will pose a safety hazard to road safety; The prediction network corresponding to the third subtask detects the level of road safety hazards; The safety warning module is used to provide graded warnings on road safety based on road safety detection results.

2. The road safety risk identification and warning system based on image semantic segmentation according to claim 1 is characterized in that: The feature extraction module includes a first feature extraction branch and a second feature extraction branch, wherein the first feature extraction branch captures semantic information, context information, and local detail information of the real-time panoramic image at different scales as multi-scale spatial features through dense convolution and residual connection; The second feature extraction branch captures color information and texture information of the real-time panoramic image at different scales as multi-scale color and texture features by combining a color histogram and a gray-level co-occurrence matrix.

3. The road safety risk identification and warning system based on image semantic segmentation according to claim 2 is characterized in that: The first feature extraction branch includes a residual network unit, a dense convolution block, a multi-scale spatial fusion unit and an attention guidance unit; The residual network unit is used to downsample the real-time panoramic image and extract intermediate feature maps at different scales; The dense convolution block is composed of a plurality of depth-wise separable convolutional layers and residual connections with the same dilation rate, and is used to extract and transform the real-time panoramic image to obtain low-level features of the real-time panoramic image; The multi-scale spatial fusion unit is used to perform matrix element addition on the features output by the residual network unit and the features output by the dense convolution block in the same dimension; The attention guidance unit is used to perform multi-angle screening and aggregation on the features output by the multi-scale spatial fusion unit from the channel level, pixel level and global level to obtain the final extracted multi-scale spatial features.

4. The road safety risk identification and warning system based on image semantic segmentation according to claim 2 is characterized in that: The second feature extraction branch includes a scale division unit, a color extraction unit, a texture extraction unit and a multi-scale color and texture fusion unit; The scale division unit is used to divide the real-time panoramic image into images of different scales; The color extraction unit is used to convert images of different scales from RGB space to HSV space, calculate the color distribution of the image, and form the color features of the image; The texture extraction unit is used to convert images of different scales into grayscale images, calculate the local pixel relationship value of the grayscale image, and calculate the histogram of the grayscale relationship of the grayscale image to obtain the LBP texture feature of the image; The multi-scale color and texture fusion unit is used to fuse the color features and LBP texture features of each scale image to obtain multi-scale color and texture features.

5. The road safety risk identification and warning system based on image semantic segmentation according to claim 1 is characterized in that: The relationship matrix construction unit is used to respectively splice the spatial features and color and texture features of the real-time panoramic image at multiple scales to obtain spatial feature splicing results and color and texture feature splicing results, and then construct a cross-modal feature relationship matrix based on the cosine similarity between the spatial feature splicing results and the color and texture feature splicing results.

6. The road safety risk identification and warning system based on image semantic segmentation according to claim 5 is characterized in that: The expression of the cross-modal feature relationship matrix is: in, represents the cross-modal feature relationship matrix, whose dimension is 2N1×2N1; i, j = 1, 2, ..., N1; <·> represents the inner product, ||·||2 represents the L2 norm; F M [i,·] represents the i-th row vector of the cross-modal feature relationship matrix; F M [j,·] represents the j-th row vector of the cross-modal feature relationship matrix; F represents the spatial feature concatenation result; Represents the result of color and texture feature splicing; N1×D represents the dimension of the feature splicing result, N1 represents the tensor length, D represents the tensor width, and the tensor length is equivalent to the number of sampling points, that is, each of the N1 points has a vector description feature of length D.

7. The road safety risk identification and warning system based on image semantic segmentation according to claim 1 is characterized in that: The loss functions of the three prediction networks are expressed as: Where N represents the total number of pixels in the panoramic image, t i 、y i 、z i Represent the labels of the three subtasks, θ1, θ2, θ3 represent the parameters of the three prediction networks, Represents the i-th pixel in the input multimodal fusion feature; Represent the prediction results of the three prediction networks respectively; The losses of the three prediction networks are weighted and summed as the final loss of the semantic segmentation module: L=α1L1+α2L2+a3L3; Among them, α1, α2, and α3 represent the weights of the three losses respectively.

8. The road safety risk identification and warning system based on image semantic segmentation according to claim 1 is characterized in that: The object categories identified by the prediction network corresponding to the first subtask include at least potholes, road water, road reflections, overpass reflections, low-altitude floating objects, traffic signs, road cracks, small sharp objects on the road, dynamic obstacles, static obstacles, accident areas and construction areas; the prediction network corresponding to the second subtask analyzes the location of the objects identified by the first prediction network to predict whether there are safety hazards; the prediction network corresponding to the third subtask comprehensively analyzes the safety hazard judgment results of each object category output by the second prediction network, and outputs the safety hazard level, including normal passage, mild safety hazard, moderate safety hazard and severe safety hazard.

9. A road safety risk identification and early warning method based on image semantic segmentation, characterized in that: The system according to any one of claims 1 to 8 comprises the following steps: Obtain real-time panoramic images of a preset range around the driving vehicle; Extracting multi-scale color texture features and multi-scale spatial features from the real-time panoramic image; Calculating the cosine similarity between color texture features and spatial features at multiple scales, constructing a cross-modal feature relationship matrix, and dynamically fusing multi-scale features based on the cross-modal feature relationship matrix to obtain multi-modal fusion features; Conduct a hierarchical analysis of the multi-level road safety detection problem from the perspective of task decomposition, detect the multimodal fusion features, and obtain road safety detection results; Based on the results of road safety inspections, graded warnings are issued for road safety.

Images