Intelligent obstacle avoidance method and device based on deep learning, equipment and storage medium

Abstract

The invention relates to the field of intelligent obstacle avoidance, and particularly discloses an intelligent obstacle avoidance method and device based on deep learning, equipment and a storage medium, and the method comprises the steps: firstly obtaining a panoramic image, collected by a camera, of the surrounding environment of an unmanned vehicle and oscillogram, collected by a laser radar, of unmanned vehicle echo signals at a plurality of preset time points; and then feature extraction and correlation analysis are performed on the two images by using a deep learning technology, and finally a decoding value is obtained through a decoder to obtain the obstacle avoidance angle of the unmanned vehicle at the next moment, so that the environment perception and obstacle avoidance capabilities of the unmanned vehicle are improved, and safer and more efficient automatic driving experience is realized.

Description

Technical Field

[0001] The present application relates to the field of intelligent obstacle avoidance, and more specifically, to an intelligent obstacle avoidance method and apparatus, device, and storage medium based on deep learning. Background Art

[0002] Obstacle avoidance is a technology or system designed to help machines, vehicles, or other equipment avoid collisions or conflicts with obstacles. This technology usually uses sensors, cameras, and other devices to sense the surrounding environment and identify the location and shape of obstacles through data processing and decision-making algorithms. Based on this information, the system can take corresponding measures, such as adjusting direction, slowing down, or stopping, to avoid collisions with obstacles. Obstacle avoidance technology is widely used in autonomous vehicles, drones, robots, and other fields to improve safety and reliability while reducing the risk of accidents and losses.

[0003] However, current obstacle avoidance systems all have the disadvantage of being difficult to accurately and autonomously avoid obstacles.

[0004] Therefore, an intelligent obstacle avoidance method, apparatus, device, and storage medium based on deep learning is desired. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed. The embodiments of this application provide an intelligent obstacle avoidance method and device based on deep learning, equipment, and storage medium, which first obtains a panoramic image of the surrounding environment of the unmanned vehicle collected by a camera and a waveform diagram of the unmanned vehicle echo signal collected by a laser radar at multiple predetermined time points, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally obtains a decoded value through a decoder to obtain the obstacle avoidance angle of the unmanned vehicle at the next moment, thereby improving the environmental perception and obstacle avoidance capabilities of the unmanned vehicle, thereby achieving a safer and more efficient autonomous driving experience.

[0006] According to one aspect of the present application, there is provided an intelligent obstacle avoidance method based on deep learning, which includes:

[0007] Obtaining a panoramic image of the surrounding environment of the unmanned vehicle collected by a camera and a waveform diagram of the unmanned vehicle echo signal collected by a laser radar at multiple predetermined time points;

[0008] Extracting a panoramic surrounding environment associated feature vector and an echo signal feature vector from the panoramic image of the surrounding environment of the unmanned vehicle collected by the camera and the waveform diagram of the echo signal of the unmanned vehicle at multiple predetermined time points collected by the laser radar;

[0009] Performing systematic deviation correction based on decomposition domain mapping on the panoramic surrounding environment association feature vector and the echo signal feature vector to obtain an obstacle avoidance decoding feature vector;

[0010] The obstacle avoidance decoding feature vector is passed through a decoder to obtain a decoded value, and the decoded value is used to represent the obstacle avoidance angle of the unmanned vehicle at the next moment.

[0011] According to another aspect of the present application, there is provided an intelligent obstacle avoidance device based on deep learning, comprising:

[0012] The obstacle avoidance data image acquisition module is used to obtain a panoramic image of the environment around the unmanned vehicle collected by the camera and a waveform diagram of the unmanned vehicle echo signal at multiple predetermined time points collected by the laser radar;

[0013] The obstacle avoidance data image feature extraction module is used to extract the panoramic surrounding environment associated feature vector and the echo signal feature vector from the panoramic image of the unmanned vehicle surrounding environment collected by the camera and the waveform diagram of the unmanned vehicle echo signal at multiple predetermined time points collected by the laser radar;

[0014] An obstacle avoidance angle association construction module is used to perform systematic deviation correction on the panoramic surrounding environment association feature vector and the echo signal feature vector based on decomposition domain mapping to obtain an obstacle avoidance decoding feature vector;

[0015] The obstacle avoidance angle result generation module is used to pass the obstacle avoidance decoding feature vector through a decoder to obtain a decoded value, and the decoded value is used to represent the obstacle avoidance angle of the unmanned vehicle at the next moment.

[0016] According to another aspect of the present application, a device is provided, comprising: a processor; and a memory, in which computer program instructions are stored, and when the computer program instructions are executed by the processor, the processor executes the intelligent obstacle avoidance method based on deep learning as described above.

[0017] According to another aspect of the present application, a storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the intelligent obstacle avoidance method based on deep learning as described above.

[0018] Compared with the prior art, the present application provides an intelligent obstacle avoidance method and device based on deep learning, equipment, and storage medium, which first obtains a panoramic image of the unmanned vehicle's surrounding environment captured by a camera and a waveform diagram of the unmanned vehicle's echo signal at multiple predetermined time points captured by a lidar, and then uses deep learning technology to perform feature extraction and correlation analysis on the two. Finally, a decoded value is obtained through a decoder to obtain the obstacle avoidance angle of the unmanned vehicle at the next moment, thereby improving the unmanned vehicle's environmental perception and obstacle avoidance capabilities, thereby achieving a safer and more efficient autonomous driving experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0020] Figure 1 Flow chart of an intelligent obstacle avoidance method based on deep learning according to an embodiment of the present application.

[0021] Figure 2 The present invention is a flowchart for extracting surrounding environment features from a panoramic image of the surrounding environment of the unmanned vehicle captured by a camera in an intelligent obstacle avoidance method based on deep learning according to an embodiment of the present application.

[0022] Figure 3 The present invention is a flowchart for extracting panoramic features from a panoramic image of the surrounding environment of the unmanned vehicle captured by a camera in an intelligent obstacle avoidance method based on deep learning according to an embodiment of the present application.

[0023] Figure 4 is a block diagram of an intelligent obstacle avoidance system based on deep learning according to an embodiment of the present application.

[0024] Figure 5 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0026] Figure 1 FIG. 1 is a flow chart of an intelligent obstacle avoidance method based on deep learning according to an embodiment of the present application. Figure 1As shown, according to the deep learning-based intelligent obstacle avoidance method of the embodiment of the present application, it includes: S110, obtaining a panoramic image of the unmanned vehicle's surrounding environment captured by a camera and a waveform diagram of the unmanned vehicle's echo signal at multiple predetermined time points captured by a lidar; S120, extracting a panoramic surrounding environment-related feature vector and an echo signal feature vector from the panoramic image of the unmanned vehicle's surrounding environment captured by the camera and the waveform diagram of the unmanned vehicle's echo signal at multiple predetermined time points captured by the lidar; S130, performing a systematic deviation correction on the panoramic surrounding environment-related feature vector and the echo signal feature vector based on decomposition domain mapping to obtain an obstacle avoidance decoding feature vector; S140, passing the obstacle avoidance decoding feature vector through a decoder to obtain a decoded value, and the decoded value is used to represent the obstacle avoidance angle of the unmanned vehicle at the next moment.

[0027] In the above-mentioned intelligent obstacle avoidance method based on deep learning, the step S110 obtains a panoramic image of the surrounding environment of the unmanned vehicle collected by the camera and a waveform of the echo signal of the unmanned vehicle collected by the laser radar at multiple predetermined time points. It should be understood that the purpose of obtaining the panoramic image of the surrounding environment of the unmanned vehicle collected by the camera and the waveform of the echo signal of the unmanned vehicle collected by the laser radar at multiple predetermined time points is to obtain rich environmental information to support the implementation of the intelligent obstacle avoidance method. Specifically, the two sensors each have different advantages and capabilities, and combining them can provide more comprehensive and accurate environmental perception. The panoramic image collected by the camera can provide rich visual information, including roads, obstacles, signs and traffic participants. By analyzing and processing the image, the features of the surrounding environment, such as road boundaries, lane lines, traffic signs, etc., can be extracted. This information is very important for judging the road state, detecting obstacles and planning obstacle avoidance paths. The echo signal collected by the laser radar can provide information such as distance, direction and reflection intensity. Since the laser beam emitted by the laser radar can accurately measure the distance and position relationship with the obstacle, the laser radar is very effective in detecting and locating obstacles. By analyzing the waveform of the echo signal, the shape, size, and location information of the obstacle can be extracted. The combined use of cameras and lidar can make up for their respective limitations. Cameras can provide richer visual information, but may be affected in low light, bad weather, or occlusion. Lidar can provide accurate distance measurement, but may not be sensitive to some detailed information. By fusing the data of the two, a more comprehensive and reliable environmental perception can be obtained, providing more accurate input for intelligent obstacle avoidance methods.

[0028] In the above-mentioned intelligent obstacle avoidance method based on deep learning, the step S120 extracts the panoramic surrounding environment associated feature vector and the echo signal feature vector from the panoramic image of the unmanned vehicle surrounding environment collected by the camera and the waveform of the unmanned vehicle echo signal collected by the laser radar at multiple predetermined time points. It should be understood that the purpose of extracting the panoramic surrounding environment associated feature vector and the echo signal feature vector from the panoramic image of the unmanned vehicle surrounding environment collected by the camera and the waveform of the unmanned vehicle echo signal collected by the laser radar at multiple predetermined time points is to comprehensively utilize the information of vision and laser radar, so as to obtain more comprehensive and accurate environmental perception and obstacle detection information. By extracting the panoramic surrounding environment associated feature vector, the visual features such as the position and shape of the object in the image can be captured to help the unmanned vehicle understand the surrounding environment. Extracting the echo signal feature vector can obtain the distance, reflection intensity and other information of the object measured by the laser radar, providing more accurate object detection and distance measurement. Combining these two feature vectors, richer and more accurate environmental perception features can be obtained, providing more reliable input for tasks such as navigation and obstacle avoidance of the unmanned vehicle.

[0029] In a specific embodiment of the present application, the step S120 includes: extracting surrounding environment features from the panoramic image of the surrounding environment of the unmanned vehicle collected by the camera to obtain a surrounding environment associated feature vector; extracting panoramic features from the panoramic image of the surrounding environment of the unmanned vehicle collected by the camera to obtain a panoramic associated feature vector; associating the surrounding environment associated feature vector and the panoramic associated feature vector to obtain the panoramic surrounding environment associated feature vector; extracting features from the waveform graph of the unmanned vehicle echo signal at multiple predetermined time points collected by the laser radar to obtain the echo signal feature vector. It should be understood that the panoramic image contains rich environmental information, such as road structure, obstacles, traffic signs, etc. By extracting features from the image, these visual information can be converted into a set of numerical feature vectors. Common feature extraction methods include edge detection, corner detection, color histogram, texture features, etc. These methods can extract features related to the environment from the image, such as road boundaries, lane lines, the shape and color of traffic signs, etc. By extracting environmental features in the image as feature vectors, visual information can be converted into numerical representations that can be processed by machine learning models. In this way, the model can predict the obstacle avoidance behavior of the unmanned vehicle by learning the relationship between these feature vectors and the obstacle avoidance angle.

[0030] Furthermore, in order to capture the overall environmental information in the panoramic image to provide a more comprehensive environmental perception, in the technical solution of the present application, a panoramic feature extraction is performed on the panoramic image of the environment around the unmanned vehicle captured by the camera to obtain a panoramic associated feature vector. The panoramic image provides a wide field of view of the environment around the unmanned vehicle, including information in multiple directions such as the front, rear, left and right. By performing panoramic feature extraction on the panoramic image, the environmental features in various directions can be comprehensively considered to obtain a more comprehensive environmental associated feature vector. This can be achieved through simple vector splicing or more complex feature fusion methods. Through panoramic feature extraction, the overall characteristics of the environment around the unmanned vehicle can be better characterized, providing a more comprehensive and accurate environmental perception.

[0031] Furthermore, associating the surrounding environment associated feature vector and the panoramic associated feature vector can make comprehensive use of the information obtained by different sensors or perspectives to obtain a more comprehensive and accurate panoramic surrounding environment associated feature vector. The feature vector provided by each sensor may only reflect the local environmental information observed by it, but cannot fully describe the entire environment. Therefore, associating the feature vectors obtained by different sensors can make up for their respective limitations and provide a more comprehensive environmental perception. For example, a camera can provide rich visual information, but in some cases there may be blind spots in the field of view or be affected by factors such as lighting. LiDAR can provide accurate distance measurement, but may not be sensitive to some detailed information. By associating the feature vectors obtained by the camera and the LiDAR, visual information and distance information can be comprehensively utilized to obtain a more comprehensive and accurate environment associated feature vector. Associating feature vectors can be achieved through feature fusion, feature weighting or other related methods.

[0032] In particular, the laser radar obtains the distance information of the surrounding environment by emitting a laser beam and measuring the echo signal returned by it. The waveform of the echo signal can reflect the interaction between the laser beam and the surrounding objects, such as reflection intensity, reflection time, etc. By extracting the features of the waveform of the echo signal, information related to the environment can be extracted from it. Specifically, the feature extraction of the echo signal can analyze the waveform of the echo signal, such as calculating the peak value, valley value, slope of the waveform, etc. The statistical features of the echo signal can also be calculated, such as the mean, variance, maximum value, minimum value, etc. The frequency domain features, such as spectrum distribution, frequency components, etc., can also be extracted by Fourier transforming or wavelet transforming the echo signal. By extracting the features of the echo signal, the distance information can be converted into a feature vector.

[0033] Figure 2 The flowchart is a method for extracting surrounding environment features from a panoramic image of the surrounding environment of the unmanned vehicle captured by a camera in an intelligent obstacle avoidance method based on deep learning according to an embodiment of the present application. Figure 2As shown, in a specific embodiment of the present application, the panoramic image of the surrounding environment of the unmanned vehicle collected by the camera is subjected to surrounding environment feature extraction to obtain a surrounding environment associated feature vector, including: S210, denoising the panoramic image of the surrounding environment of the unmanned vehicle collected by the camera to obtain a denoised surrounding environment panoramic image; S220, passing the denoised surrounding environment panoramic image through a surrounding environment feature extractor including a deep-shallow feature fusion module to obtain a surrounding environment feature map; S230, pooling the surrounding environment feature map to obtain the surrounding environment associated feature vector. It should be understood that the purpose of denoising the panoramic image of the surrounding environment of the unmanned vehicle collected by the camera is to remove noise and interference in the image to improve the quality and clarity of the image, so as to more accurately perceive the environment and make decisions. Since the image collected by the camera may be affected by many factors, such as the noise of the camera itself, changes in lighting conditions, motion blur, etc. These factors may cause noise, color distortion, blur, etc. in the image, reducing the quality and usability of the image. Therefore, by performing denoising on the collected panoramic image to remove noise and interference in the image, the image is made clearer and more accurate. The noise reduction algorithm can select appropriate methods according to the characteristics of the image and the type of noise, such as mean filtering, median filtering, Gaussian filtering, etc. These algorithms can remove isolated noise points or reduce the noise in the image by smoothing the image pixels. The denoised image can also improve the adaptability of the unmanned vehicle to environmental changes and improve the safety and stability of driving.

[0034] Furthermore, the denoised panoramic image of the surrounding environment is passed through a surrounding environment feature extractor including a deep-shallow feature fusion module to obtain a surrounding environment feature map with better representation capabilities. Such a feature map can better reflect the important information of the surrounding environment of the unmanned vehicle. The deep-shallow feature fusion module is a technology that combines a deep learning model with a shallow feature extraction method. The deep learning model can learn the high-level feature representation of the image through a multi-layer neural network and has a strong representation ability, so the deep learning model can capture more abstract and semantic features. The shallow feature extraction method can extract low-level features of the image, such as edges, textures, etc., through traditional image processing and feature extraction algorithms, so it can provide more detailed and local feature information. By fusing the deep learning model and the shallow feature extraction method, their respective advantages can be fully utilized to obtain a more comprehensive and richer feature representation, which contains rich semantic information and detail information. Specifically, a shallow feature matrix is ​​extracted from the i-th layer of the surrounding environment feature extractor including the deep-shallow feature fusion module, the i-th layer being the first to sixth layers of the surrounding environment feature extractor including the deep-shallow feature fusion module; a deep feature matrix is ​​extracted from the j-th layer of the surrounding environment feature extractor including the deep-shallow feature fusion module, the ratio between the j-th layer and the i-th layer being greater than or equal to 5; and the deep-shallow feature fusion module is used to fuse the shallow feature matrix and the deep feature matrix to obtain the surrounding environment feature map.

[0035] Furthermore, the pooling operation compresses the features of a local area of ​​the feature map into a representative value by statistically aggregating the local features. Pooling the surrounding environment feature map can aggregate local features and extract key information in the image, such as edges, textures, shapes, etc. This can reduce redundant features and retain important features, thereby improving the robustness of the model.

[0036] Figure 3 The flowchart is a method for extracting panoramic features from the panoramic image of the surrounding environment of the unmanned vehicle captured by the camera in the intelligent obstacle avoidance method based on deep learning according to an embodiment of the present application. Figure 3As shown, in a specific embodiment of the present application, the panoramic image of the surrounding environment of the unmanned vehicle collected by the camera is subjected to panoramic feature extraction to obtain a panoramic associated feature vector, including: S310, the panoramic image of the surrounding environment of the unmanned vehicle collected by the camera is passed through a panoramic image feature extractor based on a spatial attention mechanism to obtain a panoramic feature map; S320, the panoramic feature map is passed through a panoramic feature extractor based on a convolutional neural network to obtain the panoramic associated feature vector. It should be understood that by passing the panoramic image of the surrounding environment of the unmanned vehicle collected by the camera through a panoramic image feature extractor based on a spatial attention mechanism, a more discriminative and important panoramic feature map can be obtained. In the field of image processing and computer vision, the attention mechanism is an important method to imitate the human visual system. It can enable the system to focus on important areas or features when processing images and ignore unimportant information. The spatial attention mechanism is a common attention mechanism that emphasizes or suppresses different parts of the image by assigning different weights to different spatial positions of the image. In the panoramic image feature extractor, the method based on the spatial attention mechanism can help the system automatically learn and pay attention to important areas and features in the surrounding environment of the unmanned vehicle. By applying a spatial attention mechanism to the panoramic image, the extractor can pay more attention to areas with key information, such as roads, obstacles or traffic signs. This can improve the accuracy and robustness of feature extraction, allowing the unmanned vehicle to better perceive and understand the surrounding environment. Specifically, each layer of the panoramic image feature extractor based on the spatial attention mechanism performs convolution processing, pooling processing and non-linear activation processing on the input data in the forward pass of the layer to output the panoramic feature map by the last layer of the panoramic image feature extractor based on the spatial attention mechanism, wherein the input of the panoramic image feature extractor based on the spatial attention mechanism is the panoramic image of the surrounding environment of the unmanned vehicle captured by the camera.

[0037] Furthermore, the panoramic feature map can be passed through a panoramic feature extractor based on a convolutional neural network, which can combine the global information and local information in the image, and extract a panoramic associated feature vector with more semantics and representation capabilities. In the panoramic image feature extractor, the convolutional neural network can help capture the spatial relationship, shape, texture and other features in the panoramic image, and perform high-level representation and encoding of these features. Specifically, each layer of the panoramic feature extractor based on the convolutional neural network is used to perform the following operations on the input data in the forward pass of the layer: convolution processing is performed on the input data based on the convolution kernel to generate a convolution feature map; global mean pooling processing is performed on the convolution feature map based on the feature matrix to generate a pooled feature map; and nonlinear activation is performed on the feature values ​​at each position in the pooled feature map to generate an activated feature augmentation map; wherein the output of the last layer of the panoramic feature extractor based on the convolutional neural network is the panoramic associated feature vector, the input from the second layer to the last layer of the panoramic feature extractor based on the convolutional neural network is the output of the previous layer, and the input of the panoramic feature extractor based on the convolutional neural network is the panoramic feature map.

[0038] In a specific embodiment of the present application, the waveform graph of the unmanned vehicle echo signal collected by the laser radar at multiple predetermined time points is feature extracted to obtain the echo signal feature vector, including: passing the waveform graph of the unmanned vehicle echo signal collected by the laser radar at multiple predetermined time points through an echo signal feature extractor based on a convolutional neural network to obtain multiple echo signal feature matrices; arranging the multiple echo signal feature matrices along the channel dimension to obtain an echo signal feature graph; passing the echo signal feature graph through an echo signal dimensionality reduction model based on a convolutional neural network to obtain the echo signal feature vector. It should be understood that passing the waveform graph of the unmanned vehicle echo signal collected by the laser radar at multiple predetermined time points through an echo signal feature extractor based on a convolutional neural network can extract multiple echo signal feature matrices, thereby better representing the spatial characteristics and structural information of the laser radar data. Laser radar is a commonly used sensor for obtaining distance and reflection intensity information in the surrounding environment. In unmanned vehicles, laser radars can provide high-precision maps and obstacle detection data. The echo signal is the signal returned after the laser beam emitted by the lidar interacts with the surrounding objects. By processing and analyzing the echo signal, the position, shape and motion information of the object can be obtained. Through the echo signal feature extractor based on convolutional neural network, the echo signal waveforms collected by the lidar at multiple predetermined time points can be converted into multiple echo signal feature matrices. These feature matrices can better represent the spatial characteristics and structural information of the lidar data.

[0039] Furthermore, arranging multiple echo signal feature matrices along the channel dimension can unify them into a feature map. In this way, the echo signal features at different time points can be aligned so that their corresponding positions in the same feature map are the same, so as to integrate the echo signal features at different time points, making it easier to process the feature map using the convolution operation in the convolutional neural network.

[0040] Furthermore, the echo signal feature map can be passed through the echo signal dimensionality reduction model based on the convolutional neural network to obtain the echo signal feature vector. It should be understood that the echo signal feature map is a two-dimensional data structure, in which the channel at each position corresponds to the echo signal features at different time points. Through the convolutional neural network, the spatial features in the echo signal feature map can be effectively extracted. The convolution layer can automatically learn and extract the local patterns and structures in the image, thereby capturing the spatial correlation of the echo signal. Through the echo signal dimensionality reduction model based on the convolutional neural network, spatial features can be extracted from the echo signal feature map, the dimension can be reduced, and a better feature representation can be learned, thereby obtaining a more compact and more representative echo signal feature vector.

[0041] In a specific embodiment of the present application, the step S130 performs a systematic deviation correction based on the decomposition domain mapping on the panoramic surrounding environment associated feature vector and the echo signal feature vector to obtain an obstacle avoidance decoding feature vector. It should be understood that the use of the panoramic surrounding environment associated feature vector or the echo signal feature vector alone may be limited by certain situations or interfered by noise. By fusing these two feature vectors, the limitations of a single feature can be reduced and the adaptability to different scenes and complex situations can be improved. The fused feature vector can better reflect the obstacle avoidance classification characteristics of the environment around the unmanned vehicle. In addition, the fusion of the panoramic surrounding environment associated feature vector and the echo signal feature vector can provide more information, thereby enhancing the ability of obstacle avoidance classification. The fused feature vector can contain more meaningful features, so that the classification model can better distinguish different obstacle avoidance situations and improve the accuracy and robustness of obstacle avoidance decisions.

[0042] In particular, considering that the image data collected by the camera and the echo signal collected by the lidar essentially represent two different types of perceptual information. The camera image provides rich visual details such as color, texture and shape, which are suitable for tasks such as identifying objects and understanding scene semantics; while the echo signal of the lidar mainly captures distance and reflection intensity information, and is good at accurate distance measurement and three-dimensional space modeling. After being processed by their respective feature extractors, although the feature representations of their respective fields (i.e., the panoramic surrounding environment associated feature vector and the echo signal feature vector) can be obtained, these features still retain the characteristics of the original sensor data. Specifically, the panoramic surrounding environment associated feature vector focuses more on visual pattern recognition and scene understanding, which emphasizes the position, size and relationship between visible objects in the environment. In contrast, the echo signal feature vector reflects more of the geometric structure and spatial layout of the physical world, which is particularly important for obstacle detection and distance estimation. Since these two feature vectors focus on information at the visual level and the geometric level respectively, even if they are both used to describe the environment around the unmanned vehicle, they will be different in numerical representation and carry different types of data distribution and statistical properties. Therefore, when trying to directly transfer the knowledge of one domain (for example, a model trained only for image data) to another domain (i.e., a new environment that integrates image and lidar data), it is difficult for the model to directly and effectively transfer its learned knowledge. Based on this, in the technical solution of the present application, a systematic deviation correction based on decomposition domain mapping is performed on the panoramic surrounding environment association feature vector and the echo signal feature vector to obtain an obstacle avoidance decoding feature vector.

[0043] Specifically, in an embodiment of the present application, a systematic deviation correction based on decomposition domain mapping is performed on the panoramic surrounding environment associated feature vector and the echo signal feature vector to obtain an obstacle avoidance decoding feature vector, including: mapping the panoramic surrounding environment associated feature vector and the echo signal feature vector to an eigendecomposition space to obtain a panoramic surrounding environment associated eigenfeature vector and an echo signal eigenfeature vector; performing inter-domain feature calibration on the panoramic surrounding environment associated eigenfeature vector and the echo signal eigenfeature vector to obtain a panoramic surrounding environment associated modulated eigenfeature vector and an echo signal modulated eigenfeature vector; constructing a panoramic surrounding environment-echo signal fine-grained semantic information field between the panoramic surrounding environment associated modulated eigenfeature vector and the echo signal modulated eigenfeature vector; and fusing the panoramic surrounding environment associated modulated eigenfeature vector and the echo signal modulated eigenfeature vector based on the panoramic surrounding environment-echo signal fine-grained semantic information field to obtain the obstacle avoidance decoding feature vector.

[0044] More specifically, in an embodiment of the present application, a panoramic surrounding environment-echo signal fine-grained semantic information field is constructed between the panoramic surrounding environment associated modulated intrinsic feature vector and the echo signal modulated intrinsic feature vector, including: multiplying the transposed vectors of the panoramic surrounding environment associated modulated intrinsic feature vector and the echo signal modulated intrinsic feature vector, and then dividing by the scale of the echo signal modulated intrinsic feature vector to obtain a panoramic surrounding environment-echo signal fine-grained interaction matrix; and performing convolution encoding on the panoramic surrounding environment-echo signal fine-grained matrix to obtain the panoramic surrounding environment-echo signal fine-grained semantic information field.

[0045] More specifically, in an embodiment of the present application, the panoramic surrounding environment associated modulated intrinsic feature vector and the echo signal modulated intrinsic feature vector are fused based on the panoramic surrounding environment-echo signal fine-grained semantic information field to obtain an obstacle avoidance decoding feature vector, including: multiplying the panoramic surrounding environment-echo signal fine-grained semantic information field by the panoramic surrounding environment associated modulated intrinsic feature vector to obtain the panoramic surrounding environment associated feature vector after information field mapping; multiplying the panoramic surrounding environment-echo signal fine-grained semantic information field by the echo signal modulated intrinsic feature vector to obtain the echo signal feature vector after information field mapping; weighted fusion of the panoramic surrounding environment associated feature vector after information field mapping and the echo signal feature vector after information field mapping to obtain the obstacle avoidance decoding feature vector.

[0046] In the embodiment of the present application, specifically, the panoramic surrounding environment associated feature vector and the echo signal feature vector are subjected to systematic deviation correction based on decomposition domain mapping to obtain an obstacle avoidance decoding feature vector, including: processing the panoramic surrounding environment associated feature vector and the echo signal feature vector according to the following formula to obtain an obstacle avoidance decoding feature vector; wherein the formula is:

[0047]

[0048] v c =αv kt +βv st

[0049] Among them, V 1 represents the panoramic surrounding environment associated feature vector, V 2 represents the echo signal eigenvector, PCA(·) represents principal component analysis, U 1 represents the sequence of panorama surroundings associated eigendecomposition vectors, Λ 1 represents the panoramic surrounding environment correlation diagonal matrix, λ 11 and λ 1mdenote the eigenvalues ​​of the first and mth positions of the panoramic surrounding environment association diagonal matrix, T denotes the transpose of the vector, concat{·;·} denotes vector concatenation, and v 11 、v 12 、v 1m They represent the first, second and mth eigenvectors of the sequence of panoramic surrounding environment associated eigendecomposition vectors, V k represents the panoramic surrounding environment associated eigenvector, U 2 represents the sequence of eigendecomposition vectors of the echo signal, Λ 2 represents the panoramic surrounding environment correlation diagonal matrix, λ 21 and λ 2m They represent the eigenvalues ​​of the first and mth positions of the diagonal matrix of the echo signal, respectively. 21 、v 22 、v 2m Respectively represent the first, second and mth eigenvectors of the sequence of eigendecomposition vectors of the echo signal, V s represents the eigenvector of the echo signal, v i represents the ith eigenvalue of the panoramic surrounding environment associated eigenvector, v j represents the jth eigenvalue of the echo signal’s intrinsic eigenvector, represents the square of the vector norm, exp represents the natural exponential function, V' k represents the eigenvector after panoramic surrounding environment correlation modulation, V' s represents the intrinsic eigenvector of the echo signal after modulation, represents matrix multiplication, S represents the scale of the vector, M x Represents the panoramic surrounding environment-echo signal fine-grained interaction matrix, Conv 3×3 (M x ) indicates that M x A convolutional encoding operation based on a convolution kernel of 3×3 is performed, Ω represents the panoramic surrounding environment-echo signal fine-grained semantic information field, v kt represents the associated feature vector of the panoramic surrounding environment after information field mapping, v st represents the feature vector of the echo signal after information field mapping, α and β represent weighted hyperparameters, and v c Represents the obstacle avoidance decoding feature vector.

[0050] That is, in the technical solution of the present application, the panoramic surrounding environment associated feature vector and the echo signal feature vector are subjected to systematic deviation correction based on decomposition domain mapping. The process first maps the panoramic surrounding environment associated feature vector and the echo signal feature vector to the intrinsic decomposition space to obtain the panoramic surrounding environment associated intrinsic feature vector and the echo signal intrinsic feature vector. Those of ordinary skill in the art should know that the process of mapping to the intrinsic decomposition space is a combination of dimensionality reduction and feature extraction. In a specific example, the eigenvalue decomposition technique of principal component analysis (PCA) is used to find the intrinsic decomposition space, and the intrinsic decomposition space mapping is performed. The panoramic surrounding environment associated intrinsic feature vector and the echo signal intrinsic feature vector after information field mapping not only represent the projection of the original feature vector in the new coordinate system, but also reflect the main direction of change of the data, which helps to avoid the "dimensionality disaster", that is, the problem that the sample density drops sharply as the number of features increases.

[0051] Next, after mapping the feature vector to the intrinsic decomposition space, it is necessary to perform inter-domain feature calibration on the panoramic surrounding environment associated intrinsic feature vector and the echo signal intrinsic feature vector to adjust the source domain feature vector to better match the data distribution of the target domain. This operation aims to capture and compensate for the systematic deviation between the two domains to reduce the hidden disturbance of the class expression of the local feature distribution of the panoramic surrounding environment associated intrinsic feature vector and the echo signal intrinsic feature vector to the overall class expression. After modulation, the panoramic surrounding environment associated modulated intrinsic feature vector and the echo signal modulated intrinsic feature vector can retain the source domain characteristics while being more in line with the actual needs of the target domain, enhancing the model's understanding and application of the target domain data.

[0052] After completing the scale modulation of the feature vector, a panoramic surrounding environment-echo signal fine-grained semantic information field is constructed between the panoramic surrounding environment associated modulated intrinsic feature vector and the echo signal modulated intrinsic feature vector. The panoramic surrounding environment-echo signal fine-grained semantic information field is used to describe a more detailed relationship between feature vectors. The introduction of the panoramic surrounding environment-echo signal fine-grained semantic information field enables the model to go beyond simple point-to-point comparison and consider more complex patterns and contextual information, thereby enhancing the model's sensitivity to subtle differences to improve the quality of feature representation. Finally, the panoramic surrounding environment associated modulated intrinsic feature vector and the echo signal modulated intrinsic feature vector are fused based on the panoramic surrounding environment-echo signal fine-grained semantic information field to obtain an obstacle avoidance decoding feature vector.

[0053] In a specific embodiment of the present application, in step S140, the obstacle avoidance decoding feature vector is passed through a decoder to obtain a decoded value, and the decoded value is used to represent the obstacle avoidance angle of the unmanned vehicle at the next moment. It should be understood that the obstacle avoidance decoding feature vector may be a low-dimensional representation obtained through a dimensionality reduction or encoding process. By decoding the feature vector through a decoder, it can be reconstructed into the original high-dimensional feature representation. The purpose of this is to retain the information of the original features, and a certain degree of feature enhancement and correction can be performed during the decoding process. The decoded value can be used to represent the obstacle avoidance angle of the unmanned vehicle at the next moment. The output of the decoder may be a continuous value or a discrete value, representing the obstacle avoidance angle that the unmanned vehicle should take. By processing and mapping the decoded value, specific obstacle avoidance angle information can be obtained to help the unmanned vehicle make corresponding control decisions.

[0054] In summary, the embodiment of the present application first obtains a panoramic image of the unmanned vehicle's surrounding environment captured by a camera and a waveform diagram of the unmanned vehicle's echo signal at multiple predetermined time points captured by a lidar, and then uses deep learning technology to perform feature extraction and correlation analysis on the two. Finally, a decoder is used to obtain a decoding value to obtain the unmanned vehicle's obstacle avoidance angle at the next moment, thereby improving the unmanned vehicle's environmental perception and obstacle avoidance capabilities, thereby achieving a safer and more efficient autonomous driving experience.

[0055] Figure 4 FIG. 1 is a block diagram of an intelligent obstacle avoidance system based on deep learning according to an embodiment of the present application. Figure 4 As shown, according to the deep learning-based intelligent obstacle avoidance system 100 of the embodiment of the present application, it includes: an obstacle avoidance data image acquisition module 110, which is used to obtain a panoramic image of the unmanned vehicle's surrounding environment captured by a camera and a waveform diagram of the unmanned vehicle's echo signal at multiple predetermined time points captured by a laser radar; an obstacle avoidance data image feature extraction module 120, which is used to extract a panoramic surrounding environment associated feature vector and an echo signal feature vector from the panoramic image of the unmanned vehicle's surrounding environment captured by the camera and the waveform diagram of the unmanned vehicle's echo signal at multiple predetermined time points captured by the laser radar; an obstacle avoidance angle result generation module 130, which is used to obtain the unmanned vehicle's obstacle avoidance angle at the next moment based on the panoramic surrounding environment associated feature vector and the echo signal feature vector.

[0056] As described above, the intelligent obstacle avoidance system 100 based on deep learning according to the embodiment of the present application can be implemented in various terminal devices, such as a server deployed with an intelligent obstacle avoidance control algorithm based on deep learning. In one example, the intelligent obstacle avoidance system 100 based on deep learning can be integrated into the terminal device as a software module and / or a hardware module. For example, the intelligent obstacle avoidance system 100 based on deep learning can be a software module in the operating system of the terminal device, or it can be an application developed for the terminal device; of course, the intelligent obstacle avoidance system 100 based on deep learning can also be one of the many hardware modules of the terminal device.

[0057] Alternatively, in another example, the deep learning-based intelligent obstacle avoidance system 100 and the terminal device may also be separate devices, and the deep learning-based intelligent obstacle avoidance system 100 may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0058] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned intelligent obstacle avoidance system based on deep learning have been referred to above. Figures 1 to 3 It has been introduced in detail in the description of the deep learning-based intelligent obstacle avoidance method, and therefore, its repeated description will be omitted.

[0059] In addition, an embodiment of the present invention further provides another device, including: one or more processors, a memory, a communication component and a communication bus, wherein the processor, the memory and the communication component communicate with each other via the communication bus;

[0060] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute an operation corresponding to the data processing method of any of the above embodiments of the present invention.

[0061] Figure 5 is a block diagram of an electronic device according to an embodiment of the present application. Figure 5 , which shows a schematic diagram of the structure of an electronic device suitable for implementing a terminal device or a server in an embodiment of the present application. Figure 5As shown, the electronic device includes one or more processors, a communication unit, etc. The one or more processors are, for example: one or more central processing units (CPUs), and / or one or more graphics processing units (GPUs), etc. The processor can perform various appropriate actions and processes according to executable instructions stored in a read-only memory (ROM) or executable instructions loaded from a storage part to a random access memory (RAM). The communication unit may include but is not limited to a network card, and the network card may include but is not limited to an IB (Infiniband) network card. The processor may communicate with a read-only memory and / or a random access memory to execute executable instructions, and is connected to the communication unit through a bus and communicates with other target devices through the communication unit, thereby completing the operation corresponding to any method provided in the embodiments of the present application, for example, the primary shard manager shards the features in the feature queue; for any shard, the primary shard manager stores the any shard in the corresponding shard feature library, and records the mapping relationship between the any shard and the corresponding shard feature library in a mapping table; in response to a failure of the primary shard manager, a shard manager is reselected from at least one slave shard manager as the new primary shard manager; the new primary shard manager obtains shard information of the shard currently being written from all shard feature libraries, and recreates the mapping relationship of the shard currently being written based on the shard information of the shard currently being written in all shard feature libraries.

[0062] In addition, various programs and data required for the operation of the device can also be stored in the RAM. The CPU, ROM and RAM are connected to each other via a bus. In the case of RAM, the ROM is an optional module. The RAM stores executable instructions, or writes executable instructions into the ROM at runtime, and the executable instructions enable the processor to perform operations corresponding to any of the above methods of the present invention. The input / output (I / O) interface is also connected to the bus. The communication unit can be integrated or can be configured to have multiple submodules (e.g., multiple IB network cards) and be linked on the bus.

[0063] The following components are connected to the I / O interface: an input section including a keyboard, a mouse, etc.; an output section including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section including a hard disk, etc.; and a communication section including a network interface card such as a LAN card, a modem, etc. The communication section performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that a computer program read therefrom is installed into the storage section as needed.

[0064] If necessary, Figure 5The architecture shown is only an optional implementation method. In the specific practice, the above can be modified according to actual needs. Figure 5 The number and type of components can be selected, deleted, added or replaced; different functional components can be set up by separate settings or integrated settings, for example, the GPU and CPU can be set up separately or the GPU can be integrated on the CPU, the communication unit can be set up separately or integrated on the CPU or GPU, etc. These replaceable implementations all fall within the protection scope disclosed in the present invention.

[0065] In addition, an embodiment of the present invention further provides a computer program, including computer-readable code. When the computer-readable code runs on a device, a processor in the device executes instructions for implementing each step in the data processing method of any of the above embodiments of the present invention.

[0066] In addition, an embodiment of the present invention further provides a computer-readable storage medium for storing computer-readable instructions, which, when executed, implement the operations of each step in the data processing method of any of the above embodiments of the present invention.

[0067] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0068] The method and apparatus of the present invention may be implemented in many ways. For example, the method and apparatus of the present invention may be implemented by software, hardware, firmware, or any combination of software, hardware, firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present invention are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present invention may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present invention. Thus, the present invention also covers a recording medium storing a program for executing the method according to the present invention.

[0069] The description of the present invention is given for the purpose of illustration and description, and is not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for particular uses.

Claims

1. An intelligent obstacle avoidance method based on deep learning, characterized in that: include: Obtaining a panoramic image of the surrounding environment of the unmanned vehicle collected by a camera and a waveform diagram of the unmanned vehicle echo signal collected by a laser radar at multiple predetermined time points; Extracting a panoramic surrounding environment associated feature vector and an echo signal feature vector from the panoramic image of the surrounding environment of the unmanned vehicle collected by the camera and the waveform diagram of the echo signal of the unmanned vehicle at multiple predetermined time points collected by the laser radar; Performing systematic deviation correction based on decomposition domain mapping on the panoramic surrounding environment association feature vector and the echo signal feature vector to obtain an obstacle avoidance decoding feature vector; The obstacle avoidance decoding feature vector is passed through a decoder to obtain a decoded value, and the decoded value is used to represent the obstacle avoidance angle of the unmanned vehicle at the next moment.

2. The intelligent obstacle avoidance method based on deep learning according to claim 1, characterized in that: Extracting a panoramic surrounding environment associated feature vector and an echo signal feature vector from the panoramic image of the surrounding environment of the unmanned vehicle collected by the camera and the waveform diagram of the echo signal of the unmanned vehicle at multiple predetermined time points collected by the laser radar, including: Extracting surrounding environment features from the panoramic image of the surrounding environment of the unmanned vehicle captured by the camera to obtain a surrounding environment associated feature vector; Performing panoramic feature extraction on the panoramic image of the surrounding environment of the unmanned vehicle collected by the camera to obtain a panoramic correlation feature vector; Associating the surrounding environment associated feature vector with the panoramic associated feature vector to obtain the panoramic surrounding environment associated feature vector; Feature extraction is performed on the waveform diagram of the unmanned vehicle echo signal collected by the laser radar at multiple predetermined time points to obtain the echo signal feature vector.

3. The intelligent obstacle avoidance method based on deep learning according to claim 2, characterized in that: Extracting surrounding environment features from the panoramic image of the surrounding environment of the unmanned vehicle captured by the camera to obtain a surrounding environment associated feature vector includes: Denoising the panoramic image of the surrounding environment of the unmanned vehicle collected by the camera to obtain a denoised panoramic image of the surrounding environment; Passing the de-noised surrounding environment panoramic image through a surrounding environment feature extractor including a deep-shallow feature fusion module to obtain a surrounding environment feature map; The surrounding environment feature map is pooled to obtain the surrounding environment associated feature vector.

4. The intelligent obstacle avoidance method based on deep learning according to claim 3 is characterized in that: Performing panoramic feature extraction on the panoramic image of the surrounding environment of the unmanned vehicle collected by the camera to obtain a panoramic correlation feature vector includes: The panoramic image of the surrounding environment of the unmanned vehicle collected by the camera is passed through a panoramic image feature extractor based on a spatial attention mechanism to obtain a panoramic feature map; The panoramic feature map is passed through a panoramic feature extractor based on a convolutional neural network to obtain the panoramic association feature vector.

5. The intelligent obstacle avoidance method based on deep learning according to claim 4, characterized in that: Extracting features from waveforms of the unmanned vehicle echo signals at a plurality of predetermined time points collected by the laser radar to obtain the echo signal feature vector includes: The waveform diagrams of the unmanned vehicle echo signals collected by the laser radar at multiple predetermined time points are passed through an echo signal feature extractor based on a convolutional neural network to obtain multiple echo signal feature matrices; Arranging the plurality of echo signal feature matrices along a channel dimension to obtain an echo signal feature graph; The echo signal feature graph is passed through an echo signal dimension reduction model based on a convolutional neural network to obtain the echo signal feature vector.

6. The intelligent obstacle avoidance method based on deep learning according to claim 5, characterized in that: Performing systematic deviation correction based on decomposition domain mapping on the panoramic surrounding environment association feature vector and the echo signal feature vector to obtain an obstacle avoidance decoding feature vector, including: Mapping the panoramic surrounding environment associated feature vector and the echo signal feature vector to an eigendecomposition space to obtain a panoramic surrounding environment associated eigenfeature vector and an echo signal eigenfeature vector; Performing inter-domain feature calibration on the panoramic surrounding environment associated intrinsic feature vector and the echo signal intrinsic feature vector to obtain a panoramic surrounding environment associated modulated intrinsic feature vector and an echo signal modulated intrinsic feature vector; Constructing a panoramic surrounding environment-echo signal fine-grained semantic information field between the panoramic surrounding environment associated modulated intrinsic feature vector and the echo signal modulated intrinsic feature vector; The panoramic surrounding environment associated modulated eigenvalue vector and the echo signal modulated eigenvalue vector are fused based on the panoramic surrounding environment-echo signal fine-grained semantic information field to obtain the obstacle avoidance decoding feature vector.

7. The intelligent obstacle avoidance method based on deep learning according to claim 6, characterized in that: Constructing a panoramic surrounding environment-echo signal fine-grained semantic information field between the panoramic surrounding environment associated modulated intrinsic feature vector and the echo signal modulated intrinsic feature vector, including: After multiplying the panoramic surrounding environment associated modulated intrinsic feature vector and the transposed vector of the echo signal modulated intrinsic feature vector, the resultant multiplication is divided by the scale of the echo signal modulated intrinsic feature vector to obtain a panoramic surrounding environment-echo signal fine-grained interaction matrix; The panoramic surroundings-echo signal fine-grained matrix is ​​convolutionally encoded to obtain the panoramic surroundings-echo signal fine-grained semantic information field.

8. The intelligent obstacle avoidance method based on deep learning according to claim 7, characterized in that: Based on the panoramic surrounding environment-echo signal fine-grained semantic information field, the panoramic surrounding environment associated modulated intrinsic feature vector and the echo signal modulated intrinsic feature vector are fused to obtain an obstacle avoidance decoding feature vector, including: Multiplying the panoramic surrounding environment-echo signal fine-grained semantic information field by the panoramic surrounding environment associated modulated intrinsic feature vector to obtain a panoramic surrounding environment associated feature vector after information field mapping; Multiplying the panoramic surrounding environment-echo signal fine-grained semantic information field by the modulated intrinsic feature vector of the echo signal to obtain an echo signal feature vector after information field mapping; The panoramic surrounding environment associated feature vector after the information field mapping and the echo signal feature vector after the information field mapping are weightedly fused to obtain the obstacle avoidance decoding feature vector.

9. An intelligent obstacle avoidance device based on deep learning, characterized in that: include: The obstacle avoidance data image acquisition module is used to obtain a panoramic image of the environment around the unmanned vehicle collected by the camera and a waveform diagram of the unmanned vehicle echo signal at multiple predetermined time points collected by the laser radar; The obstacle avoidance data image feature extraction module is used to extract the panoramic surrounding environment associated feature vector and the echo signal feature vector from the panoramic image of the unmanned vehicle surrounding environment collected by the camera and the waveform diagram of the unmanned vehicle echo signal at multiple predetermined time points collected by the laser radar; An obstacle avoidance angle association construction module is used to perform systematic deviation correction on the panoramic surrounding environment association feature vector and the echo signal feature vector based on decomposition domain mapping to obtain an obstacle avoidance decoding feature vector; The obstacle avoidance angle result generation module is used to pass the obstacle avoidance decoding feature vector through a decoder to obtain a decoded value, and the decoded value is used to represent the obstacle avoidance angle of the unmanned vehicle at the next moment.

10. A device and a storage medium, characterized in that: include: at least one processor of the device; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the intelligent obstacle avoidance method based on deep learning as described in any one of claims 1 to 8; The storage medium stores a plurality of instructions; the plurality of instructions are used for the processor to load and execute the deep learning-based intelligent obstacle avoidance method as described in any one of claims 1 to 8.

Images