Moving object sensing method and system fusing millimeter wave radar and camera

Abstract

The invention discloses a moving object sensing method and system fusing a millimeter wave radar and a camera, and relates to the technical field of intelligent sensing. Target object region extraction and background region separation are carried out on image data by adopting a computer vision technology based on deep learning; according to the method, the target object region is subjected to deep feature extraction and semantic enhancement based on the semantic features of the background region, so that the category of the target object is intelligently identified, the accuracy and robustness of target classification in a complex scene can be effectively improved, and the requirement of moving object sensing accuracy is better met.

Description

Technical Field

[0001] The present application relates to the field of intelligent sensing technology, and more specifically, to a moving object sensing method and system integrating millimeter wave radar and camera. Background Art

[0002] The working principle of millimeter wave radar is based on the transmitter generating millimeter wave signals and sending them to the target area. The antenna receives the signal returned by the target and transmits it to the receiver. The receiver receives and processes the echo signal and analyzes the target position, speed and other information. Millimeter wave radar can detect the target motion state by using the Doppler effect. It can sense people and objects in a non-contact manner and provide fine-grained perception capabilities. It is applied to various human sensing such as human positioning, gesture recognition and important monitoring, such as human presence detection, position tracking, fall detection, sleep monitoring and gesture recognition. The camera can obtain target images or videos and is currently an important information acquisition device.

[0003] In the process of using millimeter-wave radar to perceive moving objects, although there are many advantages, there are also problems such as signal interference and limited transmission distance. The operation of the camera will be affected by light, orientation, etc.; in general, as two mainstream sensors, millimeter-wave radar and camera are lacking in functionality, accuracy and reliability when used alone. However, the performance of the two is highly complementary. By fusing the sensor data of millimeter-wave radar and camera, the accuracy and reliability of moving object perception can be significantly improved. However, the existing millimeter-wave radar and camera fusion algorithms generally stay at the "target layer" fusion or "decision layer" fusion, and cannot make good use of the complementary characteristics of millimeter-wave radar and camera data, resulting in insufficient information utilization, and thus the accuracy and stability are not enough to meet actual needs.

[0004] In this regard, the invention patent with publication number CN116794650A proposes a target detection method and device that fuses millimeter-wave radar and camera data. It uses millimeter-wave radar and camera to collect point cloud data and image data of target objects in the same field of view, uses computer vision algorithms to detect targets on image data, outputs the pixel area range and category information of each target object, and projects the point cloud data to the pixel position in the image space. It assigns the position and speed of the target object according to the radar data points, and outputs the complete information of the target in combination with the target category. In this way, the complementary characteristics of millimeter-wave radar and camera data can be fully utilized to improve the accuracy and stability of target perception.

[0005] However, in the above scheme, there are certain limitations in using the target detection device to perform target detection and classification. For example, the traditional target detection device is not ideal for target detection and classification in complex scenes, especially when the target object is partially blocked or the background is complicated, missed detection or false detection may occur. In addition, the detection process usually relies on preset category information, and it may not be able to accurately identify new categories of targets that have not been seen.

[0006] Therefore, an optimized moving object perception method that integrates millimeter-wave radar and camera is needed to solve the above technical problems. Summary of the invention

[0007] In order to solve the above technical problems, this application is proposed. The embodiment of this application provides a moving object perception method and system that integrates millimeter wave radar and camera, which uses computer vision technology based on deep learning to extract target object areas and separate background areas from image data, and performs deep feature extraction and semantic enhancement on target object areas based on background area semantic features, so as to intelligently identify the category of target objects, which can effectively improve the accuracy and robustness of target classification in complex scenes, thereby improving the demand for accurate perception of moving objects.

[0008] According to one aspect of the present application, a method for sensing moving objects by fusing millimeter-wave radar and camera is provided, which includes: S1, installing the millimeter-wave radar and camera at fixed relative positions, so that the fields of view of the millimeter-wave radar and camera overlap in order to effectively implement data fusion; S2, acquiring millimeter-wave radar point cloud data in real time, each frame of millimeter-wave radar point cloud data being a collection of time and data points; S3, acquiring camera data in real time, each frame of camera data being time and a two-dimensional image; S4, pairing the millimeter-wave radar data and camera data according to time; S5, for each pair of paired millimeter-wave radar data and camera data: using a computer vision-based algorithm to perform target detection on the image, and outputting the pixel area range of each target and the category information of the target ; According to the relative position of the millimeter-wave radar and the camera and the internal and external parameter information of the camera, project each millimeter-wave radar data point to the pixel position in the image space; if the pixel position corresponding to the millimeter-wave radar data point is located in the pixel area range of a certain target, it is considered that the data point is generated by the target, and each image-recognized target corresponds to a number of millimeter-wave radar generated data points; S6, filter the millimeter-wave radar data points corresponding to it according to the category information of the target; S7, assign the position and speed of the target according to the filtered millimeter-wave radar data points, and output the complete information of the target in combination with the category of the target, characterized in that the image is detected by using a computer vision-based algorithm and the category information of the target is output, including:

[0009] Extracting the target object region and the background region of the image respectively to obtain the target object region of interest image and the target object background portion;

[0010] Extracting image features from the target object region of interest image to obtain a target object image feature map;

[0011] Extracting background semantic features from the background part of the target object to obtain a background semantic description feature vector;

[0012] Based on the background semantic description feature vector, the target object image feature map is subjected to feature association enhancement processing to obtain a background semantic association enhanced target object image feature map;

[0013] The target object image feature map is enhanced based on the background semantic association, and the category information of the target is output.

[0014] According to another aspect of the present application, a moving object perception system integrating a millimeter wave radar and a camera is provided, which includes:

[0015] An area of ​​interest and background area extraction module, used to extract the target object area and the background area of ​​the image respectively to obtain an image of the area of ​​interest of the target object and a background part of the target object;

[0016] A target object image feature extraction module, used for performing image feature extraction on the target object region of interest image to obtain a target object image feature map;

[0017] A background semantic feature extraction module, used for extracting background semantic features from the background part of the target object to obtain a background semantic description feature vector;

[0018] A feature association enhancement module is used to perform feature association enhancement processing on the target object image feature map based on the background semantic description feature vector to obtain a background semantic association enhanced target object image feature map;

[0019] The category information generation module is used to enhance the target object image feature map based on the background semantic association and output the category information of the target.

[0020] This application has at least the following technical effects:

[0021] Compared with the prior art, the present application provides a method and system for perceiving moving objects that integrate millimeter-wave radar and camera. It uses computer vision technology based on deep learning to extract target object areas and separate background areas from image data, and performs deep feature extraction and semantic enhancement on target object areas based on semantic features of background areas, so as to intelligently identify the categories of target objects. This can effectively improve the accuracy and robustness of target classification in complex scenes, thereby better meeting the needs of improving the accuracy of moving object perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0023] Figure 1 It is a flow chart of a moving object perception method integrating millimeter-wave radar and camera according to an embodiment of the present application;

[0024] Figure 2 A schematic diagram of data flow of a moving object perception method integrating millimeter-wave radar and camera according to an embodiment of the present application;

[0025] Figure 3 It is a flowchart of sub-step S5 of the moving object perception method integrating millimeter-wave radar and camera according to an embodiment of the present application;

[0026] Figure 4 It is a block diagram of a moving object perception system integrating millimeter-wave radar and camera according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0028] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0029] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0030] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0031] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0032] It should be noted that in this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where the device is located and with the authorization of the owner of the corresponding device.

[0033] Traditional target detection devices are not ideal for target detection and classification in complex scenes, especially when the target object is partially blocked or the background is complex, which may cause missed detection or false detection. In addition, the detection process usually relies on preset category information, and may not accurately identify new categories of targets that have not been seen. Therefore, an optimized moving object perception method that integrates millimeter-wave radar and camera is expected.

[0034] In the technical solution of the present application, a moving object perception method integrating millimeter-wave radar and camera is proposed. Figure 1 This is a flow chart of a moving object perception method integrating millimeter-wave radar and camera according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of the moving object perception method integrating millimeter wave radar and camera according to an embodiment of the present application. Figure 1 and Figure 2As shown, according to the embodiment of the present application, the method for sensing moving objects by fusing millimeter-wave radar and camera includes the following steps: S1, installing the millimeter-wave radar and camera at fixed relative positions. To effectively implement data fusion, the fields of view of the millimeter-wave radar and camera overlap; S2, acquiring millimeter-wave radar point cloud data in real time, where each frame of millimeter-wave radar point cloud data is a collection of time and data points; S3, acquiring camera data in real time, where each frame of camera data is time and a two-dimensional image; S4, pairing the millimeter-wave radar data and camera data according to time; S5, for each pair of paired millimeter-wave radar data and camera data: using a computer vision-based algorithm to perform target detection on the image, and outputting Output the pixel area range of each target and the category information of the target; project each millimeter-wave radar data point to the pixel position in the image space according to the relative position of the millimeter-wave radar and the camera and the internal and external parameter information of the camera; if the pixel position corresponding to the millimeter-wave radar data point is within the pixel area range of a certain target, it is considered that the data point is generated by the target, and each image-recognized target corresponds to several data points generated by the millimeter-wave radar; S6, filter the corresponding millimeter-wave radar data points according to the target category information; S7, assign the position and speed of the target according to the filtered millimeter-wave radar data points, and output the complete information of the target in combination with the target category. In particular, in a specific example of the present application, such as Figure 3 As shown, the S5 includes: S51, respectively extracting the target object area and the background area of ​​the image to obtain the target object area of ​​interest image and the target object background part; S52, performing image feature extraction on the target object area of ​​interest image to obtain a target object image feature map; S53, performing background semantic feature extraction on the target object background part to obtain a background semantic description feature vector; S54, based on the background semantic description feature vector, performing feature association enhancement processing on the target object image feature map to obtain a background semantic association enhanced target object image feature map; S55, based on the background semantic association enhanced target object image feature map, outputting the category information of the target.

[0035] In particular, the S51 extracts the target object region and the background region of the image respectively to obtain the target object region of interest image and the target object background part. Based on this, in the technical solution of the present application, first, in order to extract the target object region in the image, a target detection network based on the Yolo model is used to process the image, and the efficient positioning capability of the Yolo model is utilized to effectively locate the target object region in the image by dividing the input image into grids, and predicting the bounding box, confidence and category probability of the target in each grid, thereby obtaining the target object region of interest image. Next, in order to further combine the contextual environment information of the target object to improve the accuracy of target classification, in the technical solution of the present application, an image mask technology is used to further process the image, a mask is generated according to the target object region of interest image, the target object region of interest in the image is removed, and the background information is retained to obtain the target object background part, thereby achieving effective separation of the target object region and the background region.

[0036] In particular, the S52 extracts image features from the target object region of interest image to obtain a target object image feature map. In a specific example of the present application, the target object region of interest image is input into an object feature extractor based on a first dilated convolutional neural network model to obtain the target object image feature map. Since in a traditional convolutional neural network, as the number of network layers increases, the spatial resolution of the feature gradually decreases, resulting in the loss of some detail information. To this end, the present application uses a first dilated convolutional neural network model as an object feature extractor to process the target object region of interest image, so as to utilize the special network structure of the dilated convolutional neural network, through a large receptive field and sparse connection characteristics, to fully mine the spatial context information of the target object, while retaining the detail features of the target object, and generating a target object image feature map. It is worth mentioning that the dilated convolutional neural network is a convolutional neural network (CNN) that uses a dilated convolution operation to expand the receptive field while maintaining spatial resolution. The dilated convolution operation is similar to the standard convolution operation, but it introduces a "hole" in the convolution kernel. A hole refers to the spacing between elements in a convolution kernel. The receptive field refers to the range of influence of a single neuron in a neural network on a specific area in the input image. In CNN, the receptive field is determined by the size and stride of the convolution kernel. Dilated convolution expands the receptive field by introducing holes in the convolution kernel without increasing the size of the convolution kernel or reducing the stride.

[0037] In particular, the S53 performs background semantic feature extraction on the background part of the target object to obtain a background semantic description feature vector. In a specific example of the present application, the background part of the target object is input into a background semantic feature extractor based on a second hole convolutional neural network model to obtain the background semantic description feature vector. Considering that when classifying a target, background area information can provide additional context clues about the target object, help understand the role, function or status of the target object in the scene, thereby improving the accuracy of classification. Therefore, in the technical solution of the present application, a background semantic feature extractor based on a second hole convolutional neural network model is further used to perform semantic extraction on the background part of the target object to extract the deep semantic features of the background part and obtain a background semantic description feature vector, thereby providing richer context information for subsequent target classification. It is worth mentioning that in order to fully explore the deep semantic information of the background part, the second hole convolutional neural network model has a deeper network structure and a larger receptive field than the first hole convolutional neural network model.

[0038] In particular, the S54, based on the background semantic description feature vector, performs feature association enhancement processing on the target object image feature map to obtain a background semantic association enhanced target object image feature map. In a specific example of the present application, the target object image feature map and the background semantic description feature vector are input into a cross-modal joint encoder based on an imitation transformer structure to obtain the background semantic association enhanced target object image feature map. In order to effectively associate background semantic information with target object features, thereby enhancing the representation ability of the target object image feature map, the present application further uses a cross-modal joint encoder based on an imitation transformer structure to process the target object image feature map and the background semantic description feature vector, by imitating the self-attention mechanism of the Transformer structure, and combining the gating unit to achieve effective fusion of different modal features. Specifically, the encoder first performs point convolution encoding and spatial convolution encoding on the target object image feature map to extract its deep spatial features and enhance its spatial expression ability, and constructs a query feature matrix and a key feature matrix based on this, and calculates the product of the query feature matrix and the key feature matrix transpose to reveal its internal correlation and construct the local semantic autocorrelation library matrix of the target object image. Then, the attention gating unit based on the GELU function is used to dynamically adjust the autocorrelation weight to achieve adaptive selection and enhancement of features and generate a weight matrix. Then, the weight matrix is ​​multiplied by the background semantic description feature vector to obtain a cross-modal interactive fusion conditional weight vector, thereby achieving conditional fusion between modalities and enhancing the cross-modal association between background features and target object features. Finally, the cross-modal interactive fusion conditional weight vector is used to weight the target object image feature map channel by channel to integrate background semantic information and target object features, so that it better reflects the semantic information and spatial features of the target object in complex scenes, thereby generating a background semantic association enhanced target object image feature map.

[0039] In an embodiment of the present application, the target object image feature map and the background semantic description feature vector are input into a cross-modal joint encoder based on an imitation transformer structure to obtain the background semantic association enhanced target object image feature map, including: performing point convolution encoding and spatial convolution encoding on the target object image feature map to obtain a target object image spatial deepening feature map; copying the target object image spatial deepening feature map to obtain a backup target object image spatial deepening feature map; performing feature shape reshaping on the backup target object image spatial deepening feature map and the target object image spatial deepening feature map respectively to obtain a backup target object image spatial deepening feature map. The feature matrix and the spatial deepening feature matrix of the target object image are used as the imitation query feature matrix and the imitation key feature matrix; the product between the imitation query feature matrix and the transposed matrix of the imitation key feature matrix is ​​calculated to obtain the local semantic autocorrelation library matrix of the target object image; the local semantic autocorrelation library matrix of the target object image is input into the attention gating unit based on the GELU function to obtain the local semantic autocorrelation library weight matrix of the target object image; the background semantic description feature vector is used as the value feature vector, and the product between the local semantic autocorrelation library weight matrix of the target object image and the value feature vector is calculated to obtain the cross-modal interaction fusion condition weight vector.

[0040] In summary, in the above embodiment, the target object image feature map and the background semantic description feature vector are input into a cross-modal joint encoder based on an imitation transformer structure to obtain the background semantic association enhanced target object image feature map, including: processing the background semantic description feature vector and the target object image feature map with the following cross-modal joint encoding formula to obtain the background semantic association enhanced target object image feature map, wherein the cross-modal joint encoding formula is:

[0041]

[0042]

[0043]

[0044]

[0045]

[0046] in, represents the target object image feature map, Representation layer normalization operation represents point convolution, Represents a convolution operation based on a 3×3 convolution kernel, represents the spatial deepening feature map of the target object image, Indicates a copy operation. represents the backup target object image space deepening feature map, represents the feature shape reshaping, represents the transpose of a matrix, represents the matrix multiplication operation, represents the Gaussian error linear unit, Represents the weight matrix of the local semantic autocorrelation library of the target object image, represents the background semantic description feature vector, represents the cross-modal interaction fusion condition weight vector, Represents the background semantic association enhanced target object image feature map.

[0047] In particular, the S55 outputs the category information of the target based on the background semantic association enhanced target object image feature map. In a specific example of the present application, the background semantic association enhanced target object image feature map is input into a target recognition module based on a classifier to obtain a recognition result, and the recognition result is the category information of the target. Specifically, the background semantic association enhanced target object image feature map is first expanded into a classification feature vector based on a row vector or a column vector; then the classification feature vector is fully connected encoded using multiple fully connected layers of the classifier to obtain an encoded classification feature vector; and then the encoded classification feature vector is passed through the Softmax classification function of the classifier to obtain the recognition result.

[0048] In a preferred example, it is considered that the target object image feature map and the background semantic description feature vector represent the image semantic features of the target object region of interest image and the image semantic features of the target object background part, respectively. Accordingly, when the target object image feature map and the background semantic description feature vector are input into a cross-modal joint encoder based on an emulated transformer structure, the gated cross-modal joint encoder based on an emulated transformer structure uses an emulated transformer structure to implement semantic gated cross-modal joint encoding between the target object image features and the background semantic features. However, since the target object image feature map and the background semantic description feature vector have large differences in source domain image content and the feature encoding dimensions do not correspond, the background semantic association strengthens the lack of cross-domain joint feature instance determination of the target object image feature map, thereby affecting the accuracy of the recognition result obtained by the classifier-based target recognition module.

[0049] Preferably, inputting the background semantic association enhanced target object image feature map into a classifier-based target recognition module to obtain a recognition result comprises:

[0050] Expanding the background semantic association enhanced target object image feature map into a background semantic association enhanced target object image feature vector;

[0051] The distance between each pair of eigenvalues ​​of the background semantic association enhanced target object image feature vector is calculated, such as the L2 distance, and the square root of the distance is taken to obtain the background semantic association enhanced target object image distance representation matrix, that is:

[0052]

[0053] in, represents the background semantic association enhanced target object image feature vector, and The first character representing the background semantic association enhancement target object image feature vector and Eigenvalues, The first character representing the background semantic association enhancement target object image feature vector and The distance between eigenvalues, Representing the background semantic association strengthens the target object image distance representation matrix The eigenvalue of the position;

[0054] The background semantic association enhancement target object image autocorrelation matrix of the background semantic association enhancement target object image feature vector is obtained, that is, ,in, represents the background semantic association enhanced target object image feature vector, represents the transpose of a vector, represents matrix multiplication, Represents the background semantic association to strengthen the target object image self-association matrix;

[0055] The background semantic association enhanced target object image feature vector is matrix-multiplied with the background semantic association enhanced target object image distance representation matrix to obtain the background semantic association enhanced target object image primary mapping vector, that is, ,in, represents the background semantic association enhanced target object image feature vector, Represents the background semantic association to enhance the target object image distance representation matrix, represents matrix multiplication, Represents the background semantic association to strengthen the first-level mapping vector of the target object image;

[0056] The background semantic association enhanced target object image primary mapping vector is matrix-multiplied with the matrix product of the background semantic association enhanced target object image distance representation matrix and the background semantic association enhanced target object image self-correlation matrix to obtain the background semantic association enhanced target object image multi-level mapping vector ,in, Represents the background semantic association to enhance the first-level mapping vector of the target object image, Represents the background semantic association to enhance the target object image distance representation matrix, represents matrix multiplication, Represents the background semantic association to enhance the target object image self-association matrix, Represents a multi-level mapping vector of the target object image enhanced by background semantic association;

[0057] The background semantic association enhanced target object image multi-level mapping vector is interpolated with the background semantic association enhanced target object image associated eigenvector (if the eigenvalue is insufficient, interpolation or zero padding) composed of the eigenvalue of the background semantic association enhanced target object image autocorrelation matrix to obtain an optimized background semantic association enhanced target object image feature vector;

[0058] The optimized background semantic association enhanced target object image feature vector is input into a classifier-based target recognition module to obtain a recognition result.

[0059] Therefore, by using the linear target mapping representation of the similarity distance representation matrix of the background semantic association enhanced target object image feature vector after the background semantic association enhanced target object image feature map is expanded, the self-correlation complete similarity instantiation of the background semantic association enhanced target object image feature map is instantiated by a secondary target mapping representation based on a multi-level distribution hierarchy, and the association mismatch negative influence factor is compensated by the association fusion kernel bias to improve the degree of instance determination of the feature value of the background semantic association enhanced target object image feature map under similarity constraints, that is, the degree of significance of the feature value as an instance for classification regression determination, thereby improving the accuracy of the recognition result obtained by the background semantic association enhanced target object image feature map through the classifier-based target recognition module.

[0060] In summary, the moving object perception method of the fusion millimeter wave radar and camera according to the embodiment of the present application is explained, which uses computer vision technology based on deep learning to extract the target object area and separate the background area of ​​the image data, and extracts deep features and enhances the semantics of the target object area based on the semantic features of the background area, so as to intelligently identify the category of the target object. In this way, the accuracy and robustness of target classification in complex scenes can be effectively improved, thereby better improving the demand for the accuracy of moving object perception.

[0061] Furthermore, a moving object perception system integrating millimeter-wave radar and camera is also provided.

[0062] Figure 4FIG. 1 is a block diagram of a moving object perception system integrating a millimeter-wave radar and a camera according to an embodiment of the present application. Figure 4 As shown, according to the embodiment of the present application, the moving object perception system 300 integrating the millimeter-wave radar and the camera includes: an interest region and background region extraction module 310, which is used to extract the target object region and the background region of the image respectively to obtain a target object interest region image and a target object background part; a target object image feature extraction module 320, which is used to perform image feature extraction on the target object interest region image to obtain a target object image feature map; a background semantic feature extraction module 330, which is used to perform background semantic feature extraction on the target object background part to obtain a background semantic description feature vector; a feature association enhancement module 340, which is used to perform feature association enhancement processing on the target object image feature map based on the background semantic description feature vector to obtain a background semantic association enhanced target object image feature map; and a category information generation module 350, which is used to output the category information of the target based on the background semantic association enhanced target object image feature map.

[0063] As described above, the moving object perception system 300 that integrates the millimeter-wave radar and the camera according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with a moving object perception algorithm that integrates the millimeter-wave radar and the camera. In one possible implementation, the moving object perception system 300 that integrates the millimeter-wave radar and the camera according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the moving object perception system 300 that integrates the millimeter-wave radar and the camera can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the moving object perception system 300 that integrates the millimeter-wave radar and the camera can also be one of the many hardware modules of the wireless terminal.

[0064] Alternatively, in another example, the motion object perception system 300 integrating the millimeter-wave radar and the camera and the wireless terminal may be separate devices, and the motion object perception system 300 integrating the millimeter-wave radar and the camera may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0065] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A moving object perception method integrating millimeter-wave radar and camera, including: S1, the millimeter wave radar and the camera are installed in fixed relative positions. To effectively implement data fusion, the fields of view of the millimeter wave radar and the camera overlap; S2, real-time acquisition of millimeter-wave radar point cloud data, each frame of millimeter-wave radar point cloud data is a collection of time and data points; S3, real-time acquisition of camera data, each frame of camera data is time and a two-dimensional image; S4, pairing the millimeter wave radar data and the camera data according to time; S5, for each pair of paired millimeter-wave radar data and camera data: use a computer vision-based algorithm to detect targets in the image, and output the pixel area range and target category information of each target; project each millimeter-wave radar data point to the pixel position in the image space according to the relative position of the millimeter-wave radar and the camera and the internal and external parameter information of the camera; if the pixel position corresponding to the millimeter-wave radar data point is located in the pixel area range of a certain target, it is considered that the data point is generated by the target, and each image-recognized target corresponds to a number of millimeter-wave radar generated data points; S6, filtering the millimeter-wave radar data points corresponding to the target according to the category information of the target; S7, assigning values ​​to the position and speed of the target according to the filtered millimeter-wave radar data points, and outputting complete information of the target in combination with the category of the target, characterized in that the image is detected by using a computer vision-based algorithm, and the category information of the target is output, including: Extracting the target object region and the background region of the image respectively to obtain the target object region of interest image and the target object background portion; Extracting image features from the target object region of interest image to obtain a target object image feature map; Extracting background semantic features from the background part of the target object to obtain a background semantic description feature vector; Based on the background semantic description feature vector, the target object image feature map is subjected to feature association enhancement processing to obtain a background semantic association enhanced target object image feature map; The target object image feature map is enhanced based on the background semantic association, and the category information of the target is output.

2. The moving object perception method integrating millimeter wave radar and camera according to claim 1 is characterized in that: Extracting the target object region and the background region of the image respectively to obtain the target object region of interest image and the target object background portion, including: Performing target detection on the image to obtain an image of a region of interest of the target object; Based on the target object region of interest image, a mask is applied to the image to obtain the target object background portion.

3. The moving object perception method integrating millimeter wave radar and camera according to claim 2 is characterized in that: Performing target detection on the image to obtain an image of a region of interest of the target object includes: The image is input into a target detection network based on the Yolo model to obtain an image of the target object's region of interest.

4. The moving object perception method integrating millimeter wave radar and camera according to claim 3 is characterized in that: Extracting image features from the target object region of interest image to obtain a target object image feature map includes: The target object region of interest image is input into an object feature extractor based on a first hole convolutional neural network model to obtain the target object image feature map.

5. The moving object perception method integrating millimeter wave radar and camera according to claim 4 is characterized in that: Extracting background semantic features from the background part of the target object to obtain a background semantic description feature vector includes: The background part of the target object is input into a background semantic feature extractor based on a second hole convolutional neural network model to obtain the background semantic description feature vector.

6. The moving object perception method integrating millimeter wave radar and camera according to claim 5 is characterized in that: Based on the background semantic description feature vector, the target object image feature map is subjected to feature association enhancement processing to obtain a background semantic association enhanced target object image feature map, including: The target object image feature map and the background semantic description feature vector are input into a cross-modal joint encoder based on an imitation transformer structure to obtain the background semantic association enhanced target object image feature map.

7. The moving object perception method integrating millimeter wave radar and camera according to claim 6 is characterized in that: Inputting the target object image feature map and the background semantic description feature vector into a cross-modal joint encoder based on a transformer-like structure to obtain the background semantic association enhanced target object image feature map, including: Performing point convolution coding and spatial convolution coding on the target object image feature map to obtain a target object image spatial deepening feature map; Copying the target object image spatial deepening feature map to obtain a backup target object image spatial deepening feature map; Reshaping the backup target object image space deepening feature map and the target object image space deepening feature map to obtain a backup target object image space deepening feature matrix and a target object image space deepening feature matrix as a simulated query feature matrix and a simulated key feature matrix respectively; Calculating the product of the pseudo-query feature matrix and the transposed matrix of the pseudo-key feature matrix to obtain a local semantic autocorrelation library matrix of the target object image; Inputting the target object image local semantic autocorrelation library matrix into the attention gating unit based on the GELU function to obtain the target object image local semantic autocorrelation library weight matrix; The background semantic description feature vector is used as a value feature vector, and the product between the target object image local semantic autocorrelation library weight matrix and the value feature vector is calculated to obtain a cross-modal interaction fusion condition weight vector.

8. The moving object perception method integrating millimeter wave radar and camera according to claim 7 is characterized in that: Based on the background semantic association enhanced target object image feature map, outputting the category information of the target, including: The background semantic association enhanced target object image feature map is input into a classifier-based target recognition module to obtain a recognition result, where the recognition result is the category information of the target.

9. A moving object perception system integrating millimeter wave radar and camera, characterized in that: include: An area of ​​interest and background area extraction module is used to extract the target object area and the background area of ​​the image respectively to obtain an image of the area of ​​interest of the target object and a background part of the target object; A target object image feature extraction module, used for performing image feature extraction on the target object region of interest image to obtain a target object image feature map; A background semantic feature extraction module, used for extracting background semantic features from the background part of the target object to obtain a background semantic description feature vector; A feature association enhancement module is used to perform feature association enhancement processing on the target object image feature map based on the background semantic description feature vector to obtain a background semantic association enhanced target object image feature map; The category information generation module is used to enhance the target object image feature map based on the background semantic association and output the category information of the target.

Images