Visual fusion perception method and system based on millimeter wave radar

Abstract

The invention relates to the technical field of visual fusion perception, and particularly discloses a visual fusion perception method and system based on a millimeter wave radar, and the method comprises the steps: obtaining an obstacle image collected by a camera, carrying out the foreground and background division of the obstacle image collected by the camera through employing a computer vision technology based on deep learning, and carrying out the recognition of the foreground and background. According to the method, foreground features and background features of an obstacle image are extracted respectively, and a possible occlusion relationship in a traffic scene is fully understood by performing joint perception analysis on the foreground features and the background features, so that the obstacle type can be identified more accurately, and the obstacle identification capability in a complex environment can be effectively enhanced; therefore, the safety of obstacle avoidance is improved.

Description

Technical Field

[0001] The present application relates to the field of visual fusion perception technology, and more specifically, to a visual fusion perception method and system based on millimeter wave radar. Background Art

[0002] Millimeter-wave radar is a radar that works in the millimeter-wave band. Since the wavelength of millimeter waves is between microwaves and centimeter waves, it can distinguish and identify very small targets, and can identify multiple targets at the same time. It has imaging capabilities, small size, good maneuverability and concealment, and has now been widely used in intelligent driving, intelligent navigation, intelligent medical care and other fields. And with the development of machine vision technology, intelligent perception technology that integrates millimeter-wave radar and machine vision has also developed rapidly.

[0003] For example, the invention patent with publication number CN111060904A proposes a blind spot monitoring method based on millimeter wave and visual fusion perception, which improves the detection and recognition capabilities of objects in the driver's blind spot by building a multimodal information fusion model and combining the perception advantages of millimeter wave radar and camera. However, in actual traffic scenes, obstacles may be partially blocked by other objects, and the above blind spot monitoring method does not take this complex situation into account, which limits the recognition effect of its visual sensor.

[0004] Therefore, an optimized visual fusion perception method is desired. Summary of the invention

[0005] The present application provides a visual fusion perception method and system based on millimeter-wave radar, which can fully understand the possible occlusion relationships in traffic scenes, thereby more accurately identifying obstacle types.

[0006] In a first aspect, a visual fusion perception method based on millimeter wave radar is provided, comprising: calibrating the millimeter wave radar and the visual sensor respectively, and then performing joint calibration of the two sensors and external parameter calibration; effectively determining the target based on the millimeter wave radar; effectively identifying the obstacle based on the machine vision sensor; building a fusion model of the millimeter wave radar and the machine vision; taking different alarm methods and information prompt methods according to different categories of obstacles, wherein the effective identification of obstacles based on the machine vision sensor comprises: obtaining an obstacle image captured by a camera; extracting the background features and foreground features of the obstacle image respectively to obtain an obstacle background feature map and an obstacle foreground feature map; performing autocorrelation feature enhancement on the obstacle background feature map and the obstacle foreground feature map respectively to obtain an enhanced obstacle background feature map and an enhanced obstacle foreground feature map; inputting the enhanced obstacle background feature map and the enhanced obstacle foreground feature map into a foreground-background joint perception module to obtain a foreground-background joint perception feature map; and determining the type label of the obstacle based on the foreground-background joint perception feature map.

[0007] In a possible implementation, respectively extracting background features and foreground features of the obstacle image to obtain an obstacle background feature map and an obstacle foreground feature map includes: inputting the obstacle image into a foreground-background segmentation module to obtain an obstacle background partial image and an obstacle foreground partial image; and respectively performing image feature extraction on the obstacle background partial image and the obstacle foreground partial image to obtain the obstacle background feature map and the obstacle foreground feature map.

[0008] In a possible implementation, image features are extracted from the obstacle background image and the obstacle foreground image respectively to obtain the obstacle background feature map and the obstacle foreground feature map, including: inputting the obstacle background image into a background feature extractor based on a first atrous convolutional neural network model to obtain the obstacle background feature map; and inputting the obstacle foreground image into a foreground feature extractor based on a second atrous convolutional neural network model to obtain the obstacle foreground feature map.

[0009] In a possible implementation, the obstacle background feature map and the obstacle foreground feature map are respectively subjected to autocorrelation feature enhancement to obtain an enhanced obstacle background feature map and an enhanced obstacle foreground feature map, including: inputting the obstacle background feature map and the obstacle foreground feature map into a feature space structure consistency self-attention cross-channel enhancement module to obtain the enhanced obstacle background feature map and the enhanced obstacle foreground feature map.

[0010] In a possible implementation, the obstacle background feature map and the obstacle foreground feature map are respectively input into a feature space structure consistency self-attention cross-channel enhancement module to obtain the enhanced obstacle background feature map and the enhanced obstacle foreground feature map, including: performing layer normalization on the obstacle background feature map to obtain a normalized obstacle background feature map; performing point convolution processing on the normalized obstacle background feature map to obtain an obstacle background channel context association representation feature map; performing convolution encoding on the obstacle background channel context association representation feature map to obtain an obstacle background space context association representation feature map; performing channel-space global interactive attention fusion on the obstacle background channel context association representation feature map and the obstacle background space context association representation feature map to obtain the enhanced obstacle background feature map.

[0011] In a possible implementation, the obstacle background channel context association representation feature map and the obstacle background space context association representation feature map are subjected to channel-space global interactive attention fusion to obtain the enhanced obstacle background feature map, including: copying the obstacle background space context association representation feature map to obtain a backup obstacle background space context association representation feature map; reshaping the obstacle background channel context association representation feature map, the obstacle background space context association representation feature map and the backup obstacle background space context association representation feature map to obtain an obstacle background channel context association representation feature matrix, an obstacle background space context association representation feature matrix, and a backup obstacle background space context association representation feature map. matrix and a backup obstacle background spatial context association representation feature matrix; calculate the cross-channel cross covariance matrix between the obstacle background channel context association representation feature matrix and the obstacle background spatial context association representation feature matrix; use the Softmax function to activate the cross-channel cross covariance matrix to obtain the obstacle background feature global interactive attention matrix; calculate the product between the backup obstacle background spatial context association representation feature matrix and the obstacle background feature global interactive attention matrix to obtain the attention enhanced obstacle background feature representation matrix; perform feature reshaping on the attention enhanced obstacle background feature representation matrix to obtain the enhanced obstacle background feature map.

[0012] In a possible implementation, the enhanced obstacle background feature map and the enhanced obstacle foreground feature map are input into a foreground-background joint perception module to obtain a foreground-background joint perception feature map, including: reshaping the enhanced obstacle background feature map and the enhanced obstacle foreground feature map to obtain an enhanced obstacle background feature vector and an enhanced obstacle foreground feature vector; associatively encoding the enhanced obstacle background feature vector and the enhanced obstacle foreground feature vector to obtain a foreground-background dependency matrix; linearly interpolating the foreground-background dependency matrix to obtain a dimensionally adjusted foreground-background dependency matrix; matrix multiplying the dimensionally adjusted foreground-background dependency matrix with the enhanced obstacle foreground feature vector to obtain a foreground-background joint perception feature vector; and reshaping the foreground-background joint perception feature vector to obtain the foreground-background joint perception feature map.

[0013] In a possible implementation, determining the type label of the obstacle based on the foreground-background joint perception feature map includes: inputting the foreground-background joint perception feature map into a classifier-based obstacle identifier to obtain a recognition result, wherein the recognition result is used to represent the type label of the obstacle.

[0014] In a second aspect, a visual fusion perception system based on millimeter-wave radar is provided, including: an obstacle image acquisition module, used to acquire an obstacle image collected by a camera; a foreground and background feature extraction module, used to extract background features and foreground features of the obstacle image respectively to obtain an obstacle background feature map and an obstacle foreground feature map; an autocorrelation feature enhancement module, used to perform autocorrelation feature enhancement on the obstacle background feature map and the obstacle foreground feature map respectively to obtain an enhanced obstacle background feature map and an enhanced obstacle foreground feature map; a foreground and background joint perception module, used to input the enhanced obstacle background feature map and the enhanced obstacle foreground feature map into a foreground-background joint perception module to obtain a foreground-background joint perception feature map; and an obstacle type determination module, used to determine a type label of the obstacle based on the foreground-background joint perception feature map.

[0015] In a possible implementation, the feature extraction module includes: a foreground-background division unit, which is used to input the obstacle image into a foreground-background division module to obtain an obstacle background partial image and an obstacle foreground partial image; and an image feature extraction unit, which is used to perform image feature extraction on the obstacle background partial image and the obstacle foreground partial image respectively to obtain the obstacle background feature map and the obstacle foreground feature map.

[0016] In a third aspect, a chip is provided, which includes an input / output interface, at least one processor, at least one memory and a bus, wherein the at least one memory is used to store instructions, and the at least one processor is used to call instructions in the at least one memory to execute the method in the first aspect.

[0017] According to a fourth aspect, a computer-readable medium is provided for storing a computer program, wherein the computer program includes instructions for executing the method in the first aspect.

[0018] In a fifth aspect, a computer program product comprising instructions is provided. When a computer runs the instructions of the computer program product, the computer executes the method in the above-mentioned second aspect.

[0019] The present application has at least the following technical effects: Compared with the prior art, the present application provides a visual fusion perception method and system based on millimeter-wave radar, which obtains an obstacle image captured by a camera, and uses deep learning-based computer vision technology to divide the obstacle image captured by the camera into foreground and background, and extracts the foreground features and background features of the obstacle image respectively. By performing joint perception analysis on its foreground features and background features, it fully understands the possible occlusion relationships in traffic scenes, thereby more accurately identifying obstacle types, which can effectively enhance the ability to recognize obstacles in complex environments, thereby improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a schematic flowchart of the effective recognition of obstacles based on machine vision sensors in the visual fusion perception method based on millimeter-wave radar in an embodiment of the present application.

[0021] Figure 2 This is a schematic diagram of data flow for effective identification of obstacles based on machine vision sensors in the visual fusion perception method based on millimeter wave radar in an embodiment of the present application.

[0022] Figure 3 This is a schematic flowchart of respectively extracting the background features and foreground features of the obstacle image to obtain an obstacle background feature map and an obstacle foreground feature map in the visual fusion perception method based on millimeter wave radar in an embodiment of the present application.

[0023] Figure 4 In the visual fusion perception method based on millimeter-wave radar in an embodiment of the present application, the obstacle background feature map and the obstacle foreground feature map are respectively input into the feature space structure consistency self-attention cross-channel enhancement module to obtain a schematic flow chart of the enhanced obstacle background feature map and the enhanced obstacle foreground feature map.

[0024] Figure 5In the visual fusion perception method based on millimeter-wave radar in an embodiment of the present application, a schematic flowchart of performing channel-space global interactive attention fusion on the obstacle background channel context association representation feature map and the obstacle background spatial context association representation feature map to obtain the enhanced obstacle background feature map.

[0025] Figure 6 In the visual fusion perception method based on millimeter-wave radar in an embodiment of the present application, the enhanced obstacle background feature map and the enhanced obstacle foreground feature map are input into the foreground-background joint perception module to obtain a schematic flowchart of the foreground-background joint perception feature map.

[0026] Figure 7 This is a schematic block diagram of a visual fusion perception system based on millimeter-wave radar according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present application.

[0028] Considering that in actual traffic scenes, obstacles may be partially blocked by other objects, and the above blind spot monitoring method does not take this complex situation into account, which limits the recognition effect of its visual sensor. In response to the above technical problems, the technical concept of this application is to use computer vision technology based on deep learning to divide the obstacle image captured by the camera into foreground and background, extract the foreground features and background features of the obstacle image respectively, and perform joint perception analysis on its foreground features and background features to fully understand the possible occlusion relationship in the traffic scene, so as to more accurately identify the obstacle type, which can effectively enhance the ability to recognize obstacles in complex environments, thereby improving driving safety.

[0029] Based on this, the present application provides a visual fusion perception method based on millimeter wave radar, including: calibrating the millimeter wave radar and visual sensor separately, and then performing joint calibration of the two sensors and external parameter calibration; effectively determining the target based on the millimeter wave radar; effectively identifying obstacles based on the machine vision sensor; building a fusion model based on millimeter wave radar and machine vision; and taking different alarm methods and information prompt methods according to different types of obstacles. In the technical solution of the present application, Figure 1 This is a schematic flowchart of the effective recognition of obstacles based on machine vision sensors in the visual fusion perception method based on millimeter-wave radar in an embodiment of the present application. Figure 2This is a schematic diagram of data flow for effective obstacle recognition based on machine vision sensors in the visual fusion perception method based on millimeter wave radar in an embodiment of the present application. Figure 1 and Figure 2 As shown, effective recognition of obstacles based on machine vision sensors includes: S110, acquiring an obstacle image captured by a camera; S120, respectively extracting background features and foreground features of the obstacle image to obtain an obstacle background feature map and an obstacle foreground feature map; S130, respectively performing autocorrelation feature enhancement on the obstacle background feature map and the obstacle foreground feature map to obtain an enhanced obstacle background feature map and an enhanced obstacle foreground feature map; S140, inputting the enhanced obstacle background feature map and the enhanced obstacle foreground feature map into a foreground-background joint perception module to obtain a foreground-background joint perception feature map; S150, determining the type label of the obstacle based on the foreground-background joint perception feature map.

[0030] In the above-mentioned visual fusion perception method based on millimeter-wave radar, the S110 obtains the obstacle image captured by the camera. Specifically, the camera, as a visual sensor, can provide rich visual information such as color, texture, shape, etc. This information is crucial to understanding the appearance characteristics of obstacles and helps to more accurately identify and classify obstacles. By obtaining the obstacle image, the image can be further divided by computer vision techniques such as deep learning to identify obstacles in the image, even if they may be partially obscured by other objects. By analyzing the images captured by the camera, the system can more accurately identify the type of obstacle. This is very important for improving driving safety and realizing intelligent transportation system functions such as collision avoidance.

[0031] In the above-mentioned visual fusion perception method based on millimeter wave radar, the S120 extracts the background features and foreground features of the obstacle image respectively to obtain an obstacle background feature map and an obstacle foreground feature map. Specifically, in a traffic scene, an obstacle may be partially or completely blocked by other objects. By extracting the foreground and background features respectively, the details of the obstacle and the environmental context information can be captured more accurately. The foreground features usually contain direct information about the obstacle, while the background features provide information about the environment in which the obstacle is located.

[0032] Optionally, in one embodiment of the present application, Figure 3 This is a schematic flow chart of extracting the background features and foreground features of the obstacle image to obtain an obstacle background feature map and an obstacle foreground feature map in the visual fusion perception method based on the millimeter wave radar of the embodiment of the present application. Figure 3As shown, respectively extracting background features and foreground features of the obstacle image to obtain an obstacle background feature map and an obstacle foreground feature map, including: S121, inputting the obstacle image into a foreground-background segmentation module to obtain an obstacle background partial image and an obstacle foreground partial image; S122, respectively performing image feature extraction on the obstacle background partial image and the obstacle foreground partial image to obtain the obstacle background feature map and the obstacle foreground feature map.

[0033] In the above-mentioned visual fusion perception method based on millimeter wave radar, the S121 inputs the obstacle image into the foreground-background division module to obtain an obstacle background part image and an obstacle foreground part image. Specifically, by dividing the obstacle image into a foreground part and a background part, the edge information of the obstacle can be highlighted, which helps to identify the loss of boundary information caused by occlusion, thereby better analyzing possible occlusion relationships and providing more environmental context information for understanding the relative position and shape of the obstacle. At the same time, by clearly distinguishing the foreground and the background, the interference of another part of the information can also be removed in the subsequent feature extraction process, thereby improving the discrimination of the feature representation.

[0034] In the above-mentioned visual fusion perception method based on millimeter wave radar, the S122 performs image feature extraction on the obstacle background partial image and the obstacle foreground partial image respectively to obtain the obstacle background feature map and the obstacle foreground feature map. Specifically, in order to effectively capture the feature representation of the obstacle background partial image and the obstacle foreground partial image, the present application adopts a first hole convolutional neural network model and a second hole convolutional neural network model to perform image feature extraction on the obstacle background partial image and the obstacle foreground partial image respectively, so as to utilize the hole convolution characteristics to effectively capture the long-distance context information in the image, while maintaining a high feature resolution, improving the extraction accuracy of background features and foreground features, and enhancing the feature expression ability, thereby generating an obstacle background feature map and an obstacle foreground feature map.

[0035] Optionally, in one embodiment of the present application, image features are extracted from the obstacle background partial image and the obstacle foreground partial image respectively to obtain the obstacle background feature map and the obstacle foreground feature map, including: inputting the obstacle background partial image into a background feature extractor based on a first hole convolutional neural network model to obtain the obstacle background feature map; inputting the obstacle foreground partial image into a foreground feature extractor based on a second hole convolutional neural network model to obtain the obstacle foreground feature map.

[0036] In the above-mentioned visual fusion perception method based on millimeter-wave radar, the S130 performs autocorrelation feature enhancement on the obstacle background feature map and the obstacle foreground feature map to obtain an enhanced obstacle background feature map and an enhanced obstacle foreground feature map. Specifically, in order to further enhance the expressiveness of background features and foreground features, the present application introduces a feature space structure consistency self-attention cross-channel enhancement module to process the obstacle background feature map and the obstacle foreground feature map respectively. Among them, the feature space structure consistency self-attention cross-channel enhancement module can utilize the channel interaction of the feature map to capture its channel context association information, and at the same time retain the spatial structure of the feature map, so as to take into account the robustness and richness of the feature expression. Specifically, the input feature map is firstly subjected to layer normalization to eliminate the scale differences between layers and ensure the stability of feature processing. Then, the channel context association and spatial structure information of the feature map are captured through multi-layer convolution operations, and the channel context association information is used as a query, and the spatial structure information is used as a key and a value. Based on the self-attention mechanism, the feature cross-channel global association interaction is performed. While retaining the spatial structure of the feature map, the channel association information between the feature structures is fused in units of the feature spatial structure to enhance the expressive power of the feature map, thereby obtaining an enhanced obstacle background feature map and an enhanced obstacle foreground feature map. That is, the obstacle background feature map and the obstacle foreground feature map are respectively enhanced by self-correlation features to obtain an enhanced obstacle background feature map and an enhanced obstacle foreground feature map, including: inputting the obstacle background feature map and the obstacle foreground feature map into a feature spatial structure consistency self-attention cross-channel enhancement module to obtain the enhanced obstacle background feature map and the enhanced obstacle foreground feature map.

[0037] Optionally, in one embodiment of the present application, Figure 4 This is a schematic flow chart of the method for visual fusion perception based on millimeter wave radar in an embodiment of the present application, in which the obstacle background feature map and the obstacle foreground feature map are respectively input into the feature space structure consistency self-attention cross-channel enhancement module to obtain the enhanced obstacle background feature map and the enhanced obstacle foreground feature map. Figure 4As shown, the obstacle background feature map and the obstacle foreground feature map are respectively input into the feature space structure consistency self-attention cross-channel enhancement module to obtain the enhanced obstacle background feature map and the enhanced obstacle foreground feature map, including: S131, layer normalization of the obstacle background feature map to obtain a normalized obstacle background feature map; S132, point convolution processing of the normalized obstacle background feature map to obtain an obstacle background channel context association representation feature map; S133, convolution encoding of the obstacle background channel context association representation feature map to obtain an obstacle background space context association representation feature map; S134, channel-space global interactive attention fusion of the obstacle background channel context association representation feature map and the obstacle background space context association representation feature map to obtain the enhanced obstacle background feature map.

[0038] Optionally, in one embodiment of the present application, Figure 5 This is a schematic flow chart of performing channel-space global interactive attention fusion on the obstacle background channel context association representation feature map and the obstacle background space context association representation feature map in the visual fusion perception method based on millimeter wave radar in an embodiment of the present application to obtain the enhanced obstacle background feature map. Figure 5 As shown, the obstacle background channel context association representation feature map and the obstacle background space context association representation feature map are subjected to channel-space global interactive attention fusion to obtain the enhanced obstacle background feature map, including: S1341, copying the obstacle background space context association representation feature map to obtain a backup obstacle background space context association representation feature map; S1342, reshaping the obstacle background channel context association representation feature map, the obstacle background space context association representation feature map and the backup obstacle background space context association representation feature map to obtain an obstacle background channel context association representation feature matrix, an obstacle background space context association representation feature matrix and a backup obstacle background space context association representation feature map. S1343, calculating the cross-channel cross covariance matrix between the obstacle background channel context association representation feature matrix and the obstacle background spatial context association representation feature matrix; S1344, using the Softmax function to activate the cross-channel cross covariance matrix to obtain the obstacle background feature global interactive attention matrix; S1345, calculating the product between the backup obstacle background spatial context association representation feature matrix and the obstacle background feature global interactive attention matrix to obtain the attention enhanced obstacle background feature representation matrix; S1346, reshaping the attention enhanced obstacle background feature representation matrix to obtain the enhanced obstacle background feature map.

[0039] More specifically, in one embodiment of the present application, the obstacle background feature map and the obstacle foreground feature map are respectively subjected to autocorrelation feature enhancement to obtain an enhanced obstacle background feature map and an enhanced obstacle foreground feature map, and the formulaic representation is as follows: the obstacle background feature map is processed using the following autocorrelation attention enhancement formula to obtain the enhanced obstacle background feature map, wherein the autocorrelation attention enhancement formula is:

[0040] F ln =Layer Normalization(F i )

[0041] F Q =Conv 1×1 (F ln )

[0042] F K =Conv 3×3 (Conv 1×1 (F ln ))

[0043] F V =Copy(F K )

[0044] M Q =Reshape(F Q )

[0045] M K =Reshape(F K )

[0046] M V =Reshape(F V )

[0047]

[0048] Among them, F i represents the obstacle background feature map, Layer Normalization (·) represents the layer normalization operation, F ln Represents the normalized obstacle background feature map, Conv 1×1 represents point convolution, F Q Represents the obstacle background channel context association representation feature map, Conv 3×3 represents the convolution processing based on 3×3 convolution kernel, F K represents the obstacle background spatial context association representation feature map, Copy(·) represents the copy operation, F V represents the backup obstacle background space context association representation feature map, reshape(·) represents the feature shape reshaping, MQ 、M K and M V They represent the obstacle background channel context association representation feature matrix, the obstacle background space context association representation feature matrix and the backup obstacle background space context association representation feature matrix, M a represents the cross-channel cross covariance matrix, θ is the scaling factor, softmax represents the normalized exponential function, Represents matrix multiplication operation, F a Here, specifically, the obstacle foreground feature map is enhanced by performing autocorrelation feature enhancement on the obstacle foreground feature map to obtain the enhanced obstacle foreground feature map, and the above formula is also referred to.

[0049] In the above-mentioned visual fusion perception method based on millimeter wave radar, the S140 inputs the enhanced obstacle background feature map and the enhanced obstacle foreground feature map into the foreground-background joint perception module to obtain the foreground-background joint perception feature map. Specifically, the enhanced obstacle background feature map provides global information about the traffic environment, road layout, etc., while the enhanced obstacle foreground feature map focuses more on describing the specific detail features of the obstacle. In order to achieve the deep fusion of background features and foreground features in the feature space to enhance the expression of obstacle features under occlusion, the present application further introduces a foreground-background joint perception module to jointly encode the enhanced obstacle background feature map and the enhanced obstacle foreground feature map. Specifically, the foreground-background joint perception module adaptively adjusts the foreground features based on the dependency relationship between the background features and the foreground features to enhance the contextual relevance between the foreground features and the background features, and generates the foreground-background joint perception feature map. That is, the obstacle foreground features are supplemented and corrected by using the background information to assist in identifying the edges and details of the obstacles, so as to more accurately understand the shape and relative position of the obstacles.

[0050] Optionally, in one embodiment of the present application, Figure 6In the visual fusion perception method based on millimeter-wave radar in an embodiment of the present application, the enhanced obstacle background feature map and the enhanced obstacle foreground feature map are input into the foreground-background joint perception module to obtain a schematic flowchart of the foreground-background joint perception feature map. Inputting the enhanced obstacle background feature map and the enhanced obstacle foreground feature map into the foreground-background joint perception module to obtain the foreground-background joint perception feature map, including: S141, reshaping the enhanced obstacle background feature map and the enhanced obstacle foreground feature map to obtain an enhanced obstacle background feature vector and an enhanced obstacle foreground feature vector; S142, associatively encoding the enhanced obstacle background feature vector and the enhanced obstacle foreground feature vector to obtain a foreground-background dependency matrix; S143, linearly interpolating the foreground-background dependency matrix to obtain a dimensionally adjusted foreground-background dependency matrix; S144, matrix multiplying the dimensionally adjusted foreground-background dependency matrix with the enhanced obstacle foreground feature vector to obtain a foreground-background joint perception feature vector; S145, reshaping the foreground-background joint perception feature vector to obtain the foreground-background joint perception feature map.

[0051] In the above-mentioned visual fusion perception method based on millimeter wave radar, the S150 determines the type label of the obstacle based on the foreground and background joint perception feature map. Specifically, the foreground and background joint perception feature map is obtained by in-depth analysis and fusion of the foreground features and background features of the obstacle. It integrates the local details of the obstacle and the global information of the surrounding environment, and provides rich contextual data for obstacle identification. Optionally, in one embodiment of the present application, the type label of the obstacle is determined based on the foreground and background joint perception feature map, including: inputting the foreground and background joint perception feature map into a classifier-based obstacle identifier to obtain a recognition result, and the recognition result is used to represent the type label of the obstacle. Specifically, the classifier-based obstacle identifier can effectively learn the obstacle feature representation in the foreground and background joint perception feature map through training, and map it to a predefined obstacle category space based on the learned feature representation, thereby achieving accurate classification of the obstacle.

[0052] In a preferred embodiment, the foreground and background joint perception feature map is passed through a classifier-based obstacle identifier to obtain a recognition result, which includes: calculating the feature mean of the foreground and background joint perception feature map, and dividing the feature mean by the difference between the maximum eigenvalue and the minimum eigenvalue of the foreground and background joint perception feature map to obtain a foreground and background joint perception distribution representation value; subtracting the foreground background joint perception distribution representation value and dividing it by the foreground background joint perception distribution representation value to obtain a foreground background joint perception distribution modulation value; activating the foreground and background joint perception feature map through a probabilistic function to obtain a probabilistic foreground background joint perception feature map; performing point subtraction between the probabilistic foreground background joint perception feature map and the foreground background joint perception distribution modulation value, taking the absolute value and calculating the base 2 of the foreground background joint perception feature map. The negative of the numerical value is used to obtain a probabilistic foreground background joint perceptual distribution modulation information feature map; after dividing the foreground background joint perceptual distribution representation value by one minus the difference of each eigenvalue of the probabilistic foreground background joint perceptual feature map, all eigenvalues ​​of the probabilistic foreground background joint perceptual feature map are summed, and divided by the scale of the foreground background joint perceptual feature map to obtain a probabilistic foreground background joint perceptual distribution modulation bias value; the probabilistic foreground background joint perceptual distribution modulation information feature map and the probabilistic foreground background joint perceptual distribution modulation bias value are multiplied by a weight as a hyperparameter to obtain an optimized foreground background joint perceptual feature map; and the optimized foreground background joint perceptual feature map is input into the classifier-based obstacle identifier to obtain the recognition result.

[0053] The optimized foreground-background joint perception feature map is expressed as:

[0054]

[0055] And among them:

[0056]

[0057] in, represents the feature mean of the foreground-background joint perception feature map, f max and f min represents the maximum eigenvalue and the minimum eigenvalue of the foreground-background joint perception feature map respectively, p represents the foreground-background joint perception distribution representation value, F represents the probabilistic foreground-background joint perception feature map obtained after the foreground-background joint perception feature map is activated by the probabilistic function, and f represents the probabilistic foreground-background joint perception feature map obtained by the probabilistic function. irepresents the i-th eigenvalue of the probabilistic foreground and background joint perceptual feature map, ε is the weight as a hyperparameter, S is the scale of the foreground and background joint perceptual feature map, that is, the width of the feature matrix of the foreground and background joint perceptual feature map is multiplied by the height and then multiplied by the number of channels of the foreground and background joint perceptual feature map, and F′ represents the optimized foreground and background joint perceptual feature map.

[0058] Specifically, considering that the enhanced obstacle background feature map and the enhanced obstacle foreground feature map respectively express the image semantic features of the background part and the foreground part in the obstacle image after self-attention and cross-channel enhancement by feature space structure consistency, when the enhanced obstacle background feature map and the enhanced obstacle foreground feature map are input into the foreground-background joint perception module, the obtained foreground-background joint perception feature map will also have insufficient foreground-background joint perception coverage due to the difference in attention interleaving weights in the image space dimension and the channel dimension, thereby causing outlier class inference mapping deviation, affecting the accuracy of the recognition result obtained by the foreground-background joint perception feature map through the classifier-based obstacle identifier.

[0059] Therefore, in the preferred example described above, the eigenvalue-based probability information distribution planning of the foreground and background joint perception feature map is performed by using the Bernoulli probability modulation distribution of the foreground and background joint perception feature map relative to the eigenvalue distribution, and the probability reverse mapping of the probability features of the foreground and background joint perception feature map as a whole is used as an extended coverage of the set mapping space of the foreground and background joint perception feature map to independently understand the interaction path between the intuitive probability information distribution and the abstract probability space mapping of the foreground and background joint perception feature map, so as to improve the accuracy of the recognition result obtained by the optimized foreground and background joint perception representation vector through the classifier-based obstacle identifier by avoiding the counterfactual reasoning mapping of the outlier feature distribution of the foreground and background joint perception feature map to the class regression probability.

[0060] In summary, the visual fusion perception method based on millimeter wave radar according to the embodiment of the present application is explained. In the step of effectively identifying obstacles based on machine vision sensors, the obstacle image captured by the camera is obtained, and the obstacle image captured by the camera is divided into foreground and background using computer vision technology based on deep learning, and the foreground features and background features of the obstacle image are extracted respectively. By performing joint perception analysis on the foreground features and background features, the possible occlusion relationship in the traffic scene is fully understood, so as to more accurately identify the obstacle type. In this way, the ability to identify obstacles in complex environments can be effectively enhanced, thereby improving driving safety.

[0061] Figure 7 Schematic block diagram of a visual fusion perception system based on millimeter wave radar according to an embodiment of the present application. Figure 7 As shown, according to an embodiment of the present application, a visual fusion perception system 200 based on a millimeter-wave radar includes: an obstacle image acquisition module 210, which is used to acquire an obstacle image collected by a camera; a foreground and background feature extraction module 220, which is used to extract background features and foreground features of the obstacle image respectively to obtain an obstacle background feature map and an obstacle foreground feature map; an autocorrelation feature enhancement module 230, which is used to perform autocorrelation feature enhancement on the obstacle background feature map and the obstacle foreground feature map respectively to obtain an enhanced obstacle background feature map and an enhanced obstacle foreground feature map; a foreground and background joint perception module 240, which is used to input the enhanced obstacle background feature map and the enhanced obstacle foreground feature map into a foreground-background joint perception module to obtain a foreground-background joint perception feature map; and an obstacle type determination module 250, which is used to determine the type label of the obstacle based on the foreground-background joint perception feature map.

[0062] Optionally, in one embodiment of the present application, the feature extraction module includes: a foreground-background division unit, used to input the obstacle image into a foreground-background division module to obtain an obstacle background partial image and an obstacle foreground partial image; an image feature extraction unit, used to perform image feature extraction on the obstacle background partial image and the obstacle foreground partial image, respectively, to obtain the obstacle background feature map and the obstacle foreground feature map.

[0063] The specific operations of each step in the above-mentioned visual fusion perception system based on millimeter-wave radar have been referred to above. Figures 1 to 6 It has been introduced in detail in the description of the visual fusion perception method based on millimeter-wave radar, and therefore, its repeated description will be omitted.

[0064] An embodiment of the present invention further provides a chip system, which includes at least one processor. When program instructions are executed in the at least one processor, the method provided in the embodiment of the present application is implemented.

[0065] An embodiment of the present invention further provides a computer storage medium on which a computer program is stored. When the computer program is executed by a computer, the computer executes the method of the above method embodiment.

[0066] An embodiment of the present invention further provides a computer program product comprising instructions, which, when executed by a computer, enables the computer to execute the method of the above method embodiment.

[0067] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0068] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0069] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0070] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0071] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

Claims

1. A visual fusion perception method based on millimeter wave radar, comprising: The millimeter wave radar and visual sensor are calibrated separately, and then the two sensors are calibrated jointly and the external parameters are calibrated; Effective determination of targets based on millimeter-wave radar; effective identification of obstacles based on machine vision sensors; construction of a fusion model of millimeter-wave radar and machine vision; different alarm methods and information prompt methods are adopted according to different types of obstacles. The characteristics are that effective identification of obstacles based on machine vision sensors include: Obtaining obstacle images captured by a camera; Extracting background features and foreground features of the obstacle image respectively to obtain an obstacle background feature map and an obstacle foreground feature map; Performing autocorrelation feature enhancement on the obstacle background feature map and the obstacle foreground feature map respectively to obtain an enhanced obstacle background feature map and an enhanced obstacle foreground feature map; Inputting the enhanced obstacle background feature map and the enhanced obstacle foreground feature map into a foreground-background joint perception module to obtain a foreground-background joint perception feature map; Based on the foreground-background joint perception feature map, a type label of the obstacle is determined.

2. The visual fusion perception method based on millimeter wave radar according to claim 1 is characterized in that: Extracting background features and foreground features of the obstacle image respectively to obtain an obstacle background feature map and an obstacle foreground feature map, including: Inputting the obstacle image into a foreground-background segmentation module to obtain an obstacle background portion image and an obstacle foreground portion image; Image features are extracted from the obstacle background portion image and the obstacle foreground portion image respectively to obtain the obstacle background feature map and the obstacle foreground feature map.

3. The visual fusion perception method based on millimeter wave radar according to claim 2 is characterized in that: The image features of the obstacle background portion image and the obstacle foreground portion image are respectively extracted to obtain the obstacle background feature map and the obstacle foreground feature map, including: Inputting the obstacle background partial image into a background feature extractor based on a first hole convolutional neural network model to obtain the obstacle background feature map; The obstacle foreground partial image is input into a foreground feature extractor based on a second hole convolutional neural network model to obtain the obstacle foreground feature map.

4. The visual fusion perception method based on millimeter wave radar according to claim 3 is characterized in that: The obstacle background feature map and the obstacle foreground feature map are respectively subjected to autocorrelation feature enhancement to obtain an enhanced obstacle background feature map and an enhanced obstacle foreground feature map, including: The obstacle background feature map and the obstacle foreground feature map are respectively input into a feature space structure consistency self-attention cross-channel enhancement module to obtain the enhanced obstacle background feature map and the enhanced obstacle foreground feature map.

5. The visual fusion perception method based on millimeter wave radar according to claim 4 is characterized in that: Inputting the obstacle background feature map and the obstacle foreground feature map into a feature space structure consistency self-attention cross-channel enhancement module to obtain the enhanced obstacle background feature map and the enhanced obstacle foreground feature map, including: Performing layer normalization on the obstacle background feature map to obtain a normalized obstacle background feature map; Performing point convolution processing on the normalized obstacle background feature map to obtain an obstacle background channel context association representation feature map; Performing convolution encoding on the obstacle background channel context association representation feature map to obtain an obstacle background space context association representation feature map; The obstacle background channel context association representation feature map and the obstacle background space context association representation feature map are subjected to channel-space global interactive attention fusion to obtain the enhanced obstacle background feature map.

6. The visual fusion perception method based on millimeter wave radar according to claim 5 is characterized in that: Performing channel-space global interactive attention fusion on the obstacle background channel context association representation feature map and the obstacle background space context association representation feature map to obtain the enhanced obstacle background feature map, including: Copying the obstacle background spatial context association representation feature map to obtain a backup obstacle background spatial context association representation feature map; Reshaping the obstacle background channel context association representation feature map, the obstacle background space context association representation feature map and the backup obstacle background space context association representation feature map to obtain an obstacle background channel context association representation feature matrix, an obstacle background space context association representation feature matrix and a backup obstacle background space context association representation feature matrix; Calculating a cross-channel cross covariance matrix between the obstacle background channel context association representation feature matrix and the obstacle background spatial context association representation feature matrix; Use the Softmax function to activate the cross-channel cross covariance matrix to obtain the obstacle background feature global interaction attention matrix; Calculating the product of the backup obstacle background spatial context association representation feature matrix and the obstacle background feature global interactive attention matrix to obtain an attention-enhanced obstacle background feature representation matrix; The attention-enhanced obstacle background feature representation matrix is ​​reshaped to obtain the enhanced obstacle background feature map.

7. The visual fusion perception method based on millimeter wave radar according to claim 6 is characterized in that: Inputting the enhanced obstacle background feature map and the enhanced obstacle foreground feature map into a foreground-background joint perception module to obtain a foreground-background joint perception feature map, including: Reshaping the enhanced obstacle background feature map and the enhanced obstacle foreground feature map to obtain an enhanced obstacle background feature vector and an enhanced obstacle foreground feature vector; Performing associative coding on the enhanced obstacle background feature vector and the enhanced obstacle foreground feature vector to obtain a foreground-background dependency matrix; Performing linear interpolation on the foreground-background dependency matrix to obtain a dimensionally adjusted foreground-background dependency matrix; Performing matrix multiplication of the dimensionally adjusted foreground-background dependency matrix and the enhanced obstacle foreground feature vector to obtain a foreground-background joint perception feature vector; The foreground-background joint perception feature vector is reshaped to obtain the foreground-background joint perception feature map.

8. The visual fusion perception method based on millimeter wave radar according to claim 7 is characterized in that: Determining the type label of the obstacle based on the foreground-background joint perception feature map includes: The foreground-background joint perception feature map is input into a classifier-based obstacle identifier to obtain a recognition result, and the recognition result is used to represent a type label of the obstacle.

9. The visual fusion perception system based on millimeter wave radar is characterized by: include: The obstacle image acquisition module is used to acquire the obstacle image collected by the camera; A foreground and background feature extraction module, used to extract the background features and foreground features of the obstacle image to obtain an obstacle background feature map and an obstacle foreground feature map; An autocorrelation feature enhancement module, used to perform autocorrelation feature enhancement on the obstacle background feature map and the obstacle foreground feature map respectively to obtain an enhanced obstacle background feature map and an enhanced obstacle foreground feature map; A foreground-background joint perception module, used for inputting the enhanced obstacle background feature map and the enhanced obstacle foreground feature map into a foreground-background joint perception module to obtain a foreground-background joint perception feature map; The obstacle type determination module is used to determine the type label of the obstacle based on the foreground and background joint perception feature map.

10. The visual fusion perception method based on millimeter wave radar according to claim 9 is characterized in that: The feature extraction module comprises: A foreground-background segmentation unit, used for inputting the obstacle image into a foreground-background segmentation module to obtain an obstacle background portion image and an obstacle foreground portion image; The image feature extraction unit is used to extract image features from the obstacle background portion image and the obstacle foreground portion image respectively to obtain the obstacle background feature map and the obstacle foreground feature map.

Images