Ground depression identification method and system based on intelligent auxiliary driving
By adopting the combination technology of graph neural network and adversarial network in the intelligent assisted driving system, accurate identification and early warning of ground depressions is achieved, and the problems of inaccurate identification and high cost in the prior art are solved, and driving safety is improved.
Patent Information
- Application Number
- CN202510061861.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-15
- Publication Date
- 2025-05-16
AI Technical Summary
Existing cars do not have the ability to identify and avoid ground depressions, pits, steps and cliffs, and high-precision ground detection equipment is costly and cannot meet the needs of low-cost and accurate identification.
The graph neural network based on the fusion of attention mechanism and multi-scale feature is used to intelligently recognize the ground depressions of images, and the learning representation of the graph neural network is enhanced by the adversarial network to achieve accurate ground depression area and depth recognition.
It improves the accuracy and safety of the intelligent assisted driving system in identifying ground depressions, reduces equipment costs, and realizes low-cost accurate identification of intelligent ground depressions in automobiles.
Smart Images
Figure CN120014595A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent driving technology, and in particular to a ground depression recognition method and system based on intelligent assisted driving. Background Art
[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.
[0003] As cars become more intelligent, their application scenarios are becoming more and more extensive. However, current assisted driving can only identify pedestrians, vehicles, obstacles and other objects with a certain volume, and cannot identify and avoid ground depressions, pits, steps, cliffs, etc.
[0004] At present, the inventors have found that the existing automobile detection of ground conditions is limited to real-time detection using radar or infrared. However, due to the complex road conditions and irregular concave surfaces of deep pits, direct detection will result in large errors. In addition, in order to achieve accurate ground detection, higher requirements are placed on the accuracy of detection equipment. Ordinary radars and cameras cannot meet the detection needs, and the installation of high-precision detection equipment will also incur high costs. Therefore, how to achieve low-cost intelligent ground depression recognition for automobiles has become a problem that needs to be solved urgently in the prior art. Summary of the invention
[0005] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a ground depression recognition method and system based on intelligent assisted driving, which uses a graph neural network based on attention mechanism and multi-scale feature fusion to intelligently recognize ground depressions in images, and uses an adversarial network to enhance the learning representation of the graph neural network to obtain accurate ground depression area and depth recognition results, thereby improving the safety of intelligent assisted driving.
[0006] In order to achieve the above object, the present invention is implemented through the following technical solutions:
[0007] A first aspect of the present invention provides a method for identifying ground depressions based on intelligent assisted driving, comprising the following steps:
[0008] Acquire the sunken image to be detected around the vehicle, and pre-process the sunken image to be detected;
[0009] The concave recognition detection model is used to perform concave recognition on the concave image to be detected, and the concave area and depth are obtained. The concave recognition detection model extracts multi-scale features from the concave image to be detected and fuses them, uses the attention mechanism to learn the association between features, and conducts adversarial training on the concave recognition detection model to enhance its learning ability.
[0010] The area and depth of the depression are compared with the preset threshold to determine whether to issue a warning to the driver.
[0011] Furthermore, the radar is used to emit electromagnetic waves to detect the road surface ahead, and when a sudden increase in data is detected, the camera obtains a depth image of the road surface ahead as a depression image to be detected.
[0012] Furthermore, the preprocessing of the concave image to be detected includes normalization and denoising operations.
[0013] Furthermore, the concave recognition detection model uses convolutional neural networks of different scales to extract the concave images to be detected, and uses a hierarchical feature fusion module to fuse the extracted multi-scale features, uses a graph neural network to perform concave recognition on the fused features, and uses a bilinear attention mechanism to learn the correlation between features.
[0014] Furthermore, the adversarial network includes a view generator and a view discriminator. The view generator is used to generate a variety of concave images, which are input into the graph neural network for training. According to the learning situation of the graph neural network, the view discriminator is used to identify the authenticity of the concave images.
[0015] Furthermore, the specific process of giving early warning to the driver includes:
[0016] The area and depth of the depression are compared with the preset thresholds. If the area and depth of the depression exceed the preset thresholds, the driver is given an early warning.
[0017] After issuing a warning reminder for one cycle, if the driver does not take any action, the system determines whether the active control system is turned on. If the active control is not turned on, it continues to send reminders to the driver. If the active control system is turned on, it determines whether the conditions for changing lanes are met. If the conditions for changing lanes are met, it changes to the right lane. If the conditions for changing lanes are not met, it gradually decelerates to a stop within a cycle, and turns on the hazard lights to alert vehicles coming from behind.
[0018] Furthermore, the warning methods include steering wheel vibration reminder and voice reminder.
[0019] A second aspect of the present invention provides a ground depression recognition system based on intelligent assisted driving, comprising:
[0020] An image acquisition module is configured to acquire an image of a to-be-detected sunken cavity around the vehicle and pre-process the image of the to-be-detected sunken cavity;
[0021] The image recognition module is configured to use a concave recognition detection model to perform concave recognition on a concave image to be detected, and obtain a concave area and depth, wherein the concave recognition detection model extracts multi-scale features from the concave image to be detected and fuses them, uses an attention mechanism to learn the association between features, and performs adversarial training on the concave recognition detection model to enhance its learning ability;
[0022] The warning module is configured to compare the depression area and depth with a preset threshold value to determine whether to issue a warning to the driver.
[0023] The third aspect of the present invention provides a medium having a program stored thereon, which, when executed by a processor, implements the steps in the method for identifying ground depressions based on intelligent assisted driving as described in the first aspect of the present invention.
[0024] The fourth aspect of the present invention provides a device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the ground depression identification method based on intelligent assisted driving as described in the first aspect of the present invention are implemented.
[0025] One or more of the above technical solutions have the following beneficial effects:
[0026] The present invention discloses a ground depression recognition method and system based on intelligent assisted driving, and constructs an intelligent assisted driving system that can recognize ground depressions. When serious ground depressions, cliff steps, etc. appear on the route ahead or behind the vehicle, the vehicle assisted driving system will give an early warning and perform emergency braking when necessary to ensure the safety of the vehicle and passengers.
[0027] The present invention achieves more accurate recognition of depression conditions by using a graph neural network in combination with an attention mechanism and a feature fusion module to perform association learning on images, and adds adversarial network terminology to train the graph neural network, greatly improving the accuracy of image recognition.
[0028] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0030] Figure 1 This is a flow chart of a method for identifying ground depressions based on intelligent assisted driving in Embodiment 1 of the present invention;
[0031] Figure 2This is a schematic diagram of radar detection in Embodiment 1 of the present invention. DETAILED DESCRIPTION
[0032] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0033] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "include" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or their combinations;
[0034] Embodiment 1:
[0035] Embodiment 1 of the present invention provides a method for identifying ground depressions based on intelligent assisted driving, such as Figure 1 As shown, the following steps are included:
[0036] Step 1: Obtain the image of the sunken area to be detected around the vehicle and preprocess the image of the sunken area to be detected.
[0037] Step 1.1: Use the radar to emit electromagnetic waves to detect the road ahead. When a sudden increase in data is detected, the camera obtains a depth image of the road ahead as a depression image to be detected.
[0038] This embodiment collects information about the car's surroundings through a radar and a depth camera installed on the car body. The radar emits electromagnetic waves and calculates the distance by measuring the time difference from the emission to the reception of the electromagnetic waves. Figure 2 As shown in the figure, when a pothole suddenly appears on the road surface, the distance of the electromagnetic wave feedback emitted in the direction of the pothole will suddenly increase. The millimeter wave radar emits high-frequency electromagnetic waves, which propagate at the speed of light and are reflected back when encountering an object in front and received by the radar. By measuring the time difference between the transmitted and received signals, the distance of the object can be calculated. When the car is driving on a flat road, the speed v of the radar wave propagation is a constant value, and the time difference Δt from the radar transmitting wave to the wave reflected from the front / rear ground is also a constant value. When a pothole / cliff appears on the front / rear road surface, the distance of the wave propagation increases sharply, and the time difference Δt will suddenly increase. At this time, the depth camera is combined with the front / rear road surface images to assist in the rapid measurement and simulation of the front / rear road surface conditions. According to the simulation results, the safety risk level is evaluated, and corresponding warnings or active braking actions are made.
[0039] The depth image of the road ahead obtained by the camera refers to the front of the camera, not the front of the vehicle. The road ahead or behind the vehicle can be obtained through the settings in this embodiment. The depressions in this embodiment include steps, deep pits, ground collapses, cross sections, etc.
[0040] Since ordinary cameras cannot directly obtain depth information, this embodiment uses a depth camera, or uses an ordinary camera and a depth camera in combination, or uses stereo vision technology to extract depth information from images taken by two or more cameras. The depth image provides the distance from each pixel to the camera, that is, the Z coordinate.
[0041] Once the depth image is obtained, the depth information can be combined with the RGB image to convert the pixel coordinates into 3D points in the camera coordinate system through the camera's intrinsic parameters. This process involves combining the depth value with the x and y coordinates of each pixel in the RGB image to calculate the X, Y, and Z coordinates of each point in the camera coordinate system.
[0042] Specifically, in order to extract point cloud data from the image, the points in the camera coordinate system are converted to points in the world coordinate system according to the camera's intrinsic parameters (focal length, principal point coordinates, etc.) and extrinsic parameters (camera position and rotation relative to the world coordinate system). The image points are converted to a point cloud in the world coordinate system according to the camera's intrinsic parameters. Code can be written to convert the depth image into point cloud data. The generated point cloud data is further processed using the depression recognition detection model in this embodiment to identify and calculate the area and depth of the ground depression. This may include steps such as filtering, segmentation, and feature extraction of the point cloud.
[0043] It should be noted that performing coordinate transformation on the depth image according to the internal and external parameters and distortion coefficients of the camera to obtain point cloud data is a conventional method in the art, and the detailed steps will not be repeated here.
[0044] Step 1.2: Preprocess the image to be detected including normalization and denoising operations to improve the effect of subsequent network processing.
[0045] In a specific implementation, first, an image of a depression to be detected with depth point cloud data of the ground is obtained. These data contain the three-dimensional coordinates (x, y, z) of each point on the ground. The original point cloud data is processed, including noise reduction, filtering, etc., to improve data quality.
[0046] The extracted point cloud data is converted into a graph structure, where each point is a node and the connection between nodes is represented as an edge. Graph neural networks can be used to learn embedded representations of nodes that can capture local and global contextual information of nodes.
[0047] Step 2: Use the concave recognition detection model to perform concave recognition on the concave image to be detected to obtain the concave area and depth.
[0048] Among them, the concave recognition detection model extracts multi-scale features from the concave image to be detected and fuses them, uses the attention mechanism to learn the association between features, and performs adversarial training on the concave recognition detection model to enhance its learning ability.
[0049] In a specific implementation, the concave recognition detection model includes a convolutional neural network and a graph neural network.
[0050] Step 2.1: The concave recognition detection model uses convolutional neural networks of different scales to extract the concave images to be detected. The convolutional neural networks can be networks of different depths and widths, such as ResNet, MobileNet, etc., to capture feature information at different levels.
[0051] Step 2.2: Use the hierarchical feature fusion module to fuse the extracted multi-scale features.
[0052] The hierarchical feature fusion module designs a parallel hierarchical structure of local and global feature blocks to effectively extract local features and global representations at different semantic scales. It can adaptively fuse local features from different levels, global representations, and fused semantic information from previous levels according to input features. In order to avoid the problems of gradient vanishing, explosion, and network degradation, a residual Inverted MLP (IRMLP) is also set up to effectively capture the global and local feature information of each level. In addition, a shortcut connection module is designed to adaptively fuse the semantic information of features at different levels and scales, enhance feature transfer, and avoid information loss. This embodiment integrates features of different scales through a feature fusion module to obtain more comprehensive road surface information.
[0053] Step 2.3: Use graph neural network to perform concave recognition on the fused features, and use bilinear attention mechanism to learn the correlation between features.
[0054] In a specific implementation, the extracted features are input into a graph neural network (GNN) encoder, where each node represents a road surface area, and the node features include features extracted from CNN and features enhanced by a bilinear attention mechanism. The graph neural network is used to process graph structure data, and the information of neighboring nodes is aggregated through a message passing mechanism to learn the representation of the nodes. Afterwards, the graph neural network outputs an image that identifies the concave area, and based on the point cloud data of the concave area, the minimum bounding box algorithm is used to calculate the area of the concave area, and the height difference between the center point of the concave area and the surrounding non-concave area points is calculated to estimate the depth of the concave area.
[0055] The bilinear attention mechanism is a method for learning the correlation between features. It enhances feature representation by calculating the bilinear interactions between features. In the concave recognition detection model, the bilinear attention mechanism can help the model capture the complex relationship between different features, thereby improving the accuracy of concave recognition. Here is how to use the bilinear attention mechanism to learn the correlation between features:
[0056] This embodiment designs a bilinear attention layer in the graph neural network, which receives any two feature maps as input and calculates the bilinear interaction between them, and then obtains attention weights through a softmax function. These weights are then used to weight the feature maps to highlight important features and suppress unimportant features.
[0057] Step 2.4: Use the adversarial network to perform adversarial training on the graph neural network to enhance its learning ability.
[0058] In this embodiment, the adversarial network includes a view generator and a view discriminator. The view generator is used to generate a variety of concave images, which are input into the graph neural network for training. According to the learning situation of the graph neural network, the view discriminator is used to identify the authenticity of the concave images.
[0059] During the adversarial training process, the view generator generates enhanced views, the view discriminator distinguishes between real views and generated views, and the graph neural network learns the representation of the graph. Through this adversarial process, the model is able to learn more robust feature representations. The node parameters of the graph neural network and the view discriminator are the same.
[0060] Step 3: Determine whether to issue a warning to the driver based on the comparison of the depression area and depth with the preset threshold.
[0061] Step 3.1: Compare the concave area and depth with the preset threshold
[0062] In a specific implementation, when the width of the depression is less than half the wheel width, it is assessed as low risk; when it is greater than half the wheel width but less than the wheel width, it is assessed as medium risk; and when it is greater than the wheel width, it is assessed as high risk.
[0063] Thresholds are set based on the criteria for high and medium risks. If the pit is judged to be small in area and shallow in depth, and will not cause safety problems or adverse experiences, the warning is cancelled and the vehicle continues to drive. If the pit is judged to be large in area and height difference in the front / rear, and there is a safety risk or it may cause severe turbulence, a reminder is issued to the driver (steering wheel vibration + voice reminder).
[0064] Step 3.2: If the depression area and depth exceed the preset threshold, a warning reminder is given to the driver.
[0065] Among them, the warning methods include steering wheel vibration reminder and voice reminder.
[0066] Step 3.3: After a warning reminder is issued for one cycle, if the driver does not take any action, determine whether the active control system is turned on. If the active control is not turned on, continue to send reminders to the driver. If the active control system is turned on, determine whether the conditions for changing lanes are met. If the conditions for changing lanes are met, change lanes to the right lane. If the conditions for changing lanes are not met, gradually decelerate to a stop within one cycle, and turn on the double flash lights to warn the rear vehicles. In this embodiment, one cycle is set to 3 seconds.
[0067] Embodiment 2:
[0068] Embodiment 2 of the present invention provides a ground depression recognition system based on intelligent assisted driving, including:
[0069] An image acquisition module is configured to acquire an image of a to-be-detected sunken cavity around the vehicle and pre-process the image of the to-be-detected sunken cavity;
[0070] The image recognition module is configured to use a concave recognition detection model to perform concave recognition on a concave image to be detected, and obtain a concave area and depth, wherein the concave recognition detection model extracts multi-scale features from the concave image to be detected and fuses them, uses an attention mechanism to learn the association between features, and performs adversarial training on the concave recognition detection model to enhance its learning ability;
[0071] The warning module is configured to compare the depression area and depth with a preset threshold value to determine whether to issue a warning to the driver.
[0072] Embodiment three:
[0073] Embodiment 3 of the present invention provides a medium on which a program is stored. When the program is executed by a processor, the steps in the ground depression recognition method based on intelligent assisted driving as described in Embodiment 1 of the present invention are implemented.
[0074] Embodiment 4:
[0075] Embodiment 4 of the present invention provides a device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the ground depression identification method based on intelligent assisted driving as described in Embodiment 1 of the present invention are implemented.
[0076] The steps involved in the above embodiments 2, 3 and 4 correspond to the method embodiment 1. For the specific implementation methods, please refer to the relevant description part of embodiment 1.
[0077] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.
[0078] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.
Claims
1. A method for identifying ground depressions based on intelligent assisted driving, characterized in that: The following steps are involved: Acquire the sunken image to be detected around the vehicle, and pre-process the sunken image to be detected; The concave recognition detection model is used to perform concave recognition on the concave image to be detected, and the concave area and depth are obtained. The concave recognition detection model extracts multi-scale features from the concave image to be detected and fuses them, uses the attention mechanism to learn the association between features, and conducts adversarial training on the concave recognition detection model to enhance its learning ability. The area and depth of the depression are compared with the preset threshold to determine whether to issue a warning to the driver.
2. The method for identifying ground depressions based on intelligent assisted driving according to claim 1, characterized in that: The radar emits electromagnetic waves to detect the road ahead. When a sudden increase in data is detected, the camera obtains a depth image of the road ahead as the depression image to be detected.
3. The method for identifying ground depressions based on intelligent assisted driving according to claim 1, characterized in that: The preprocessing of the concave image to be detected includes normalization and denoising operations.
4. The method for identifying ground depressions based on intelligent assisted driving according to claim 1, characterized in that: The concave recognition detection model uses convolutional neural networks of different scales to extract the concave images to be detected, and uses the hierarchical feature fusion module to fuse the extracted multi-scale features, uses the graph neural network to perform concave recognition on the fused features, and uses the bilinear attention mechanism to learn the correlation between features.
5. The method for identifying ground depressions based on intelligent assisted driving according to claim 4, characterized in that: The adversarial network includes a view generator and a view discriminator. The view generator is used to generate a variety of concave images and input them into the graph neural network for training. According to the learning situation of the graph neural network, the view discriminator is used to identify the authenticity of the concave images.
6. The method for identifying ground depressions based on intelligent assisted driving according to claim 1, characterized in that: The specific process of warning the driver includes: The area and depth of the depression are compared with the preset threshold value. If the area and depth of the depression exceed the preset threshold value, the driver is given an early warning. After issuing a warning reminder for one cycle, if the driver does not take any action, the system determines whether the active control system is turned on. If the active control is not turned on, it continues to send reminders to the driver. If the active control system is turned on, it determines whether the conditions for changing lanes are met. If the conditions for changing lanes are met, it changes to the right lane. If the conditions for changing lanes are not met, it gradually decelerates to a stop within a cycle, and turns on the hazard lights to alert vehicles coming from behind.
7. The method for identifying ground depressions based on intelligent assisted driving according to claim 6, characterized in that: Warning methods include steering wheel vibration reminder and voice reminder.
8. A ground depression recognition system based on intelligent assisted driving, characterized in that: include: An image acquisition module is configured to acquire an image of a to-be-detected sunken cavity around the vehicle and pre-process the image of the to-be-detected sunken cavity; The image recognition module is configured to use a concave recognition detection model to perform concave recognition on a concave image to be detected, and obtain a concave area and depth, wherein the concave recognition detection model extracts multi-scale features from the concave image to be detected and fuses them, uses an attention mechanism to learn the association between features, and performs adversarial training on the concave recognition detection model to enhance its learning ability; The warning module is configured to compare the depression area and depth with a preset threshold value to determine whether to issue a warning to the driver.
9. A computer-readable storage medium, characterized in that: A plurality of instructions are stored therein, and the instructions are suitable for being loaded by a processor of a terminal device and executing the ground depression recognition method based on intelligent assisted driving as described in any one of claims 1 to 7.
10. A terminal device, characterized in that: It includes a processor and a computer-readable storage medium, the processor is used to implement each instruction; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded by the processor and executing the ground depression recognition method based on intelligent assisted driving as described in any one of claims 1-7.