A target detection method, a target detection model training method and a device

Abstract

The embodiment of the present application provides a target detection method, a training method and a device for a target detection model, and relates to the field of deep learning technology. The target detection method includes: obtaining a first image to be detected, RAD domain data to be detected, and image features of a second image to be detected at a previous acquisition moment of a scene to be detected; extracting features of the first image to be detected to obtain features of the image to be detected; extracting features of the RAD domain data to be detected to obtain RAD features to be processed; generating a speed map to be processed based on the features of the image to be detected and the image features of the second image to be detected; fusing the features of the image to be detected and the elements at the same pixel position in the speed map to be processed to obtain features of the fused image to be processed; splicing the features of the fused image to be processed and the RAD features to be processed to obtain joint features to be detected; performing target detection based on the joint features to be detected to obtain detection results. This solution can improve the accuracy of target detection.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a target detection method, a target detection model training method and a device. Background Art

[0002] RAD The RAD domain data is three-dimensional data obtained by processing the original echo data of the radar. The three dimensions of the RAD domain data are distance, azimuth and Doppler velocity, which represent the corresponding relationship between the signal strength of the echo received by the radar and the distance, azimuth and Doppler velocity.

[0003] Currently, RAD domain data can be fused with image data for target detection. However, there are large differences between the two modalities of data, RAD domain data and image data, and it is difficult to fuse RAD domain data with image data well, which leads to poor results in target detection by fusing RAD domain data with image data.

[0004] Therefore, how to better fuse RAD domain data with image data to improve the accuracy of target detection is an urgent problem to be solved. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a target detection method, a target detection model training method and a device to improve the accuracy of target detection. The specific technical solution is as follows:

[0006] The present application embodiment first provides a target detection method, the method comprising:

[0007] Acquire the first image to be detected and RAD domain data to be detected of the scene to be detected collected at the current collection time, and the image features of the second image to be detected of the scene to be detected collected at the previous collection time;

[0008] Performing feature extraction on the first image to be detected to obtain features of the image to be detected; and performing feature extraction on the RAD domain data to be detected to obtain RAD features to be processed;

[0009] Generate a speed map to be processed based on the image features to be detected and the image features of the second image to be detected; wherein the speed value at each pixel position in the speed map to be processed represents: the radial speed at the real position represented by the element at the same pixel position in the image features to be detected;

[0010] Fusing the image feature to be detected and the elements at the same pixel position in the speed map to be processed to obtain a fused image feature to be processed;

[0011] splicing the fused image feature to be processed and the RAD feature to be processed to obtain a joint feature to be detected;

[0012] Target detection is performed based on the joint features to be detected to obtain a detection result.

[0013] Optionally, performing feature extraction on the first image to be detected to obtain features of the image to be detected includes:

[0014] Extracting features of the first image to be detected by an image feature extraction network in a pre-trained target detection model to obtain features of the image to be detected; wherein the target detection model further includes: a RAD feature extraction network, a speed map output layer, a first encoding layer and a detection network;

[0015] Performing feature extraction on the RAD domain data to be detected to obtain RAD features to be processed includes:

[0016] The RAD feature extraction network is used to extract features of the RAD domain data to be detected, so as to obtain RAD features to be processed; based on the features of the image to be detected and the image features of the second image to be detected, a speed map to be processed is generated, including: inputting the features of the image to be detected and the image features of the second image to be detected into the speed map output layer, so as to obtain a speed map to be processed;

[0017] The step of fusing the image feature to be detected and the elements at the same pixel position in the speed map to be processed to obtain the fused image feature to be processed includes: using the first coding layer to fuse the image feature to be detected and the elements at the same pixel position in the speed map to be processed to obtain the fused image feature to be processed;

[0018] Performing target detection based on the joint features to be detected to obtain a detection result includes: performing target detection on the joint features to be detected using the detection network to obtain a detection result.

[0019] Optionally, before using the first coding layer to fuse the elements at the same pixel position in the image feature to be detected and the speed map to be processed to obtain the fused image feature to be processed, the method also includes: obtaining a depth map to be processed corresponding to the image feature to be detected; wherein the depth value at each pixel position in the depth map to be processed represents: the depth at the real position represented by the element at the same pixel position in the image feature to be detected; for each pixel position in the image feature to be detected, based on the coordinate system conversion relationship between the image acquisition device and the radar that collects the RAD domain data to be detected, and the depth value at the same pixel position in the depth map to be processed, determine the polar coordinate value of the pixel position in the radar coordinate system of the radar;

[0020] Using the first coding layer to fuse the image feature to be detected and the elements at the same pixel position in the speed map to be processed to obtain the fused image feature to be processed, including: using the first coding layer to fuse the image feature to be detected and the elements at the same pixel position in the speed map to be processed, and the polar coordinate values ​​of the same pixel position in the radar coordinate system to obtain the fused image feature to be processed.

[0021] Optionally, the first coding layer is used to fuse the elements at the same pixel position in the image feature to be detected and the speed map to be processed, as well as the polar coordinate values ​​of the same pixel position in the radar coordinate system to obtain the fused image feature to be processed, including: splicing the elements at the same pixel position in the image feature to be detected and the speed map to be processed, as well as the polar coordinate values ​​of the same pixel position in the radar coordinate system to obtain the spliced ​​image feature to be processed; inputting the spliced ​​image feature to be processed into the first coding layer to obtain a first initial fused image feature; wherein the first initial fused image feature has the same number of channels as the image feature to be detected; and fusing the first initial fused image feature with the image feature to be detected to obtain the fused image feature to be processed.

[0022] Optionally, the RAD feature extraction network includes: a RAD feature extraction layer and a second encoding layer;

[0023] The RAD feature extraction network is used to extract features of the RAD domain data to be detected to obtain RAD features to be processed, including: extracting features of the RAD domain data to be detected through the RAD feature extraction layer to obtain a first initial RAD feature; for each coordinate position in the first initial RAD feature, the second coding layer is used to fuse the element at the coordinate position in the first initial RAD feature, the distance, azimuth and Doppler velocity represented by the coordinate position to obtain the RAD feature to be processed.

[0024] Optionally, for each coordinate position in the first initial RAD feature, the second coding layer is used to fuse the element at the coordinate position in the first initial RAD feature, the distance, azimuth and Doppler velocity represented by the coordinate position to obtain the RAD feature to be processed, including: for each coordinate position in the first initial RAD feature, the element at the coordinate position is spliced ​​with the distance, azimuth and Doppler velocity represented by the coordinate position to obtain the spliced ​​RAD feature to be processed; the spliced ​​RAD feature to be processed is input into the second coding layer to obtain a first fused RAD feature; wherein the first fused RAD feature has the same number of channels as the first initial RAD feature; the first fused RAD feature is fused with the first initial RAD feature to obtain the RAD feature to be processed.

[0025] Optionally, the RAD feature extraction layer includes: a first feature extraction unit, a second feature extraction unit and a third feature extraction unit;

[0026] The RAD feature extraction layer is used to extract features from the RAD domain data to be detected to obtain a first initial RAD feature, including: projecting the RAD domain data to be detected along the Doppler velocity dimension, the azimuth dimension and the distance dimension to obtain a RA image to be processed, a RD image to be processed and an AD image to be processed; extracting features from the RA image to be processed, the RD image to be processed and the AD image to be processed by the first feature extraction unit, the second feature extraction unit and the third feature extraction unit to obtain a first initial RA feature, a first initial RD feature and a first initial AD feature; expanding the first initial RA feature in the Doppler velocity dimension, the first initial RD feature and the azimuth dimension, and the first initial AD feature in the distance dimension according to a uniform size to obtain a RA feature to be used, a RD feature to be used and an AD feature to be used; splicing the RA feature to be used, the RD feature to be used and the AD feature to be used along the channel dimension to obtain a first initial RAD feature.

[0027] Optionally, the image acquisition device for acquiring the first image to be detected and the radar for acquiring the RAD domain data to be detected are both arranged on a preset carrier; the detection network comprises: an initialization layer, a decoding layer and a detection head;

[0028] The detection network is used to perform target detection on the joint features to be detected to obtain detection results, including: for each detection anchor point set within the preset perception range of the preset carrier, the distance, azimuth, preset Doppler speed of the detection anchor point are fused with the absolute speed parameter corresponding to the detection anchor point through the initialization layer to obtain a query vector corresponding to the detection anchor point; the absolute speed parameter corresponding to each detection anchor point is determined based on the absolute speed of the detection anchor point; each query vector is fused with the joint features to be detected based on the attention mechanism through the decoding layer to obtain an updated query vector; each updated query vector is input into the detection head to obtain the detection result of each detection anchor point.

[0029] Optionally, the absolute speed parameter corresponding to each detection anchor point is: ; where exp represents the natural exponential function; v ego is the moving speed of the preset carrier at the current collection moment; is a preset constant; a j is the position of the detection anchor point; d k The preset Doppler velocity for this detection anchor point.

[0030] Optionally, the detection result of each detection anchor point includes at least one of: the confidence of the detection anchor point, the type at the detection anchor point, the position at the detection anchor point, the orientation at the detection anchor point, and the speed at the detection anchor point.

[0031] Optionally, the target detection model further includes: a depth map output layer;

[0032] Obtaining a depth map to be processed corresponding to the image feature to be detected, including: inputting the image feature to be detected into the depth map output layer to obtain a depth map to be processed; wherein the depth map output layer is: trained based on the image feature of a first sample image and a real depth map corresponding to the first sample image; the depth value at each pixel position in the real depth map represents: the depth at the real position represented by the element at the same pixel position in the image feature of the first sample image.

[0033] Optionally, the speed map output layer is: obtained based on image features of a first sample image, image features of a second sample image, and training of a real speed map; the first sample image and the second sample image are acquired at two adjacent acquisition moments for the same scene; the speed value at each pixel position in the real speed map represents: image features of the later acquired images in the first sample image and the second sample image, and radial speed at the real position represented by elements at the same pixel position.

[0034] The embodiment of the present application also provides a method for training a target detection model, wherein the target detection model includes: an image feature extraction network, a RAD feature extraction network, a speed map output layer, a first encoding layer, and a detection network; the method includes:

[0035] Acquire a first sample image and sample RAD domain data of a sample scene at a first acquisition moment, and image features of a second sample image of the sample scene acquired at an acquisition moment previous to the first acquisition moment;

[0036] Performing feature extraction on the first sample image through the image feature extraction network to obtain sample image features; and performing feature extraction on the sample RAD domain data through the RAD feature extraction network to obtain sample RAD features;

[0037] Input the sample image feature and the image feature of the second sample image into the velocity map output layer to obtain a sample velocity map; wherein the velocity value at each pixel position in the sample velocity map represents: the radial velocity at the real position represented by the element at the same pixel position in the sample image feature;

[0038] Using the first coding layer, the sample image feature and the elements at the same pixel position in the sample velocity map are fused to obtain a sample fused image feature;

[0039] Concatenating the sample fusion image feature with the sample RAD feature to obtain a sample joint feature;

[0040] Using the detection network to perform target detection on the sample joint features to obtain a prediction result of the target detection;

[0041] The loss function is calculated based on the true value of the prediction result and the pre-calibrated detection result to adjust the model parameters of the target detection model until the preset convergence condition is reached to obtain a trained target detection model.

[0042] Optionally, before using the first coding layer to fuse the sample image feature and the elements at the same pixel position in the sample velocity map to obtain the sample fused image feature, the method further includes: obtaining a sample depth map corresponding to the sample image feature; wherein the depth value at each pixel position in the sample depth map represents: the depth at the real position represented by the element at the same pixel position in the sample image feature; for each pixel position in the sample image feature, based on the coordinate system conversion relationship between the image acquisition device and the radar that acquires the sample RAD domain data, and the depth value at the same pixel position in the sample depth map, determining the polar coordinate value of the pixel position in the radar coordinate system of the radar;

[0043] The sample image feature and the elements at the same pixel position in the sample velocity map are fused by using the first coding layer to obtain the sample fused image feature, including: the sample image feature and the elements at the same pixel position in the sample velocity map, and the polar coordinate values ​​of the same pixel position in the radar coordinate system are fused by using the first coding layer to obtain the sample fused image feature.

[0044] Optionally, the first coding layer is used to fuse the sample image features and the elements at the same pixel position in the sample velocity map, as well as the polar coordinate values ​​of the same pixel position in the radar coordinate system to obtain sample fused image features, including: splicing the sample image features and the elements at the same pixel position in the sample velocity map, as well as the polar coordinate values ​​of the same pixel position in the radar coordinate system to obtain sample spliced ​​image features; inputting the sample spliced ​​image features into the first coding layer to obtain a second initial fused image feature; wherein the second initial fused image feature has the same number of channels as the sample image feature; and fusing the second initial fused image feature with the sample image feature to obtain a sample fused image feature.

[0045] Optionally, the RAD feature extraction network includes: a RAD feature extraction layer and a second encoding layer;

[0046] The sample RAD domain data is subjected to feature extraction through the RAD feature extraction network to obtain sample RAD features, including: extracting features from the sample RAD domain data through the RAD feature extraction layer to obtain a second initial RAD feature; for each coordinate position in the second initial RAD feature, using the second coding layer to fuse the element at the coordinate position in the second initial RAD feature, the distance, azimuth and Doppler velocity represented by the coordinate position to obtain a sample RAD feature.

[0047] Optionally, for each coordinate position in the second initial RAD feature, the second coding layer is used to fuse the element at the coordinate position in the second initial RAD feature, the distance, azimuth and Doppler velocity represented by the coordinate position to obtain a sample RAD feature, including: for each coordinate position in the second initial RAD feature, splicing the element at the coordinate position with the distance, azimuth and Doppler velocity represented by the coordinate position to obtain a sample spliced ​​RAD feature; inputting the sample spliced ​​RAD feature into the second coding layer to obtain a second fused RAD feature; wherein the second fused RAD feature has the same number of channels as the second initial RAD feature; fusing the second fused RAD feature with the second initial RAD feature to obtain a sample RAD feature.

[0048] Optionally, the RAD feature extraction layer includes: a first feature extraction unit, a second feature extraction unit and a third feature extraction unit;

[0049] The RAD feature extraction layer is used to extract features from the sample RAD domain data to obtain a second initial RAD feature, including: projecting the sample RAD domain data along the Doppler velocity dimension, the azimuth dimension and the distance dimension to obtain a sample RA image, a sample RD image and a sample AD image; extracting features from the sample RA image, the sample RD image and the sample AD image through the first feature extraction unit, the second feature extraction unit and the third feature extraction unit to obtain a second initial RA feature, a second initial RD feature and a second initial AD feature; expanding the second initial RA feature in the Doppler velocity dimension, the second initial RD feature in the azimuth dimension and the second initial AD feature in the distance dimension according to a uniform size to obtain a sample RA feature, a sample RD feature and a sample AD feature; and splicing the sample RA feature, the sample RD feature and the sample AD feature along the channel dimension to obtain a second initial RAD feature.

[0050] Optionally, the image acquisition device for acquiring the first sample image and the radar for acquiring the sample RAD domain data are both arranged on the sample object; the detection network comprises: an initialization layer, a decoding layer and a detection head;

[0051] The detection network is used to perform target detection on the sample joint features to obtain a prediction result of the target detection, including: for each sample anchor point set within the preset perception range of the sample object, the distance, azimuth, preset Doppler speed of the sample anchor point are fused with the absolute speed parameter corresponding to the sample anchor point through the initialization layer to obtain a query vector corresponding to the sample anchor point; the absolute speed parameter corresponding to each sample anchor point is determined based on the absolute speed of the sample anchor point; each query vector is fused with the sample joint features based on the attention mechanism through the decoding layer to obtain an updated query vector; and each updated query vector is input into the detection head to obtain a prediction result for each sample anchor point.

[0052] The present application also provides a target detection device, the device comprising:

[0053] The detection data acquisition module is used to acquire the first detection image and RAD domain data of the detection scene collected at the current acquisition time, and the image features of the second detection image of the detection scene collected at the previous acquisition time;

[0054] A feature extraction module to be detected, used for performing feature extraction on the first image to be detected to obtain the image features to be detected; and performing feature extraction on the RAD domain data to be detected to obtain the RAD features to be processed;

[0055] A speed map output module to be processed is used to generate a speed map to be processed based on the image features to be detected and the image features of the second image to be detected; wherein the speed value at each pixel position in the speed map to be processed represents: the radial speed at the real position represented by the element at the same pixel position in the image features to be detected;

[0056] A feature fusion module to be processed, used for fusing the image feature to be detected and the elements at the same pixel position in the speed map to be processed to obtain a fused image feature to be processed;

[0057] A feature splicing module to be processed, used for splicing the fused image feature to be processed and the RAD feature to be processed to obtain a joint feature to be detected;

[0058] The first target detection module is used to perform target detection based on the joint features to be detected to obtain a detection result.

[0059] The embodiment of the present application also provides a training device for a target detection model, wherein the target detection model includes: an image feature extraction network, a RAD feature extraction network, a speed map output layer, a first encoding layer, and a detection network; the device includes:

[0060] A sample data acquisition module, used to acquire a first sample image and sample RAD domain data of a sample scene at a first acquisition moment, and image features of a second sample image of the sample scene acquired at an acquisition moment before the first acquisition moment;

[0061] A sample feature extraction module, configured to extract features from the first sample image through the image feature extraction network to obtain sample image features; and to extract features from the sample RAD domain data through the RAD feature extraction network to obtain RAD features of the sample to be processed;

[0062] A sample speed map output module, used for inputting the sample image features and the image features of the second sample image into the speed map output layer to obtain a sample speed map;

[0063] A sample feature fusion module, used to fuse the sample image feature and the elements at the same pixel position in the sample velocity map using the first coding layer to obtain a sample fused image feature;

[0064] A sample feature splicing module, used for splicing the sample fusion image feature and the sample RAD feature to obtain a sample joint feature;

[0065] A second detection module, used to perform target detection on the sample joint features using the detection network to obtain a prediction result of the target detection;

[0066] The model parameter adjustment module is used to calculate the loss function based on the true value of the prediction result and the pre-calibrated detection result, so as to adjust the model parameters of the target detection model until the preset convergence condition is reached to obtain a trained target detection model.

[0067] The present application also provides an electronic device, including:

[0068] Memory, used to store computer programs;

[0069] The processor is used to implement any of the above-mentioned target detection methods or target detection model training methods when executing the program stored in the memory.

[0070] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it implements any of the above-mentioned target detection methods or target detection model training methods.

[0071] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the target detection methods or target detection model training methods described above.

[0072] Beneficial effects of the embodiments of the present application:

[0073] The target detection method provided by the embodiment of the present application first obtains the first image to be detected and the RAD domain data to be detected of the scene to be detected collected at the current acquisition moment, and performs feature extraction on each of them to obtain the image features to be detected and the RAD features to be processed; then, based on the image features to be detected of the first image to be detected and the image features of the second image to be detected at the previous acquisition moment, a speed map to be processed is obtained, and the speed map to be processed can represent the radial velocity at the real position represented by the element at each pixel position in the image features to be detected, and then the image features to be detected and the elements at the same pixel position in the speed map to be processed are fused to obtain the fused image features to be processed, which can introduce speed information for the image features to be detected, and because the RAD domain data also contains speed information, therefore, this scheme splices the obtained fused image features to be processed with the RAD features to be processed of the RAD domain data to be detected, and can implicitly establish the connection between the two cross-modal features of image features and RAD features, that is, it can better fuse the image data with the RAD domain data, thereby improving the accuracy of target detection.

[0074] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0076] Figure 1 A schematic diagram of a target detection method according to an embodiment of the present invention;

[0077] Figure 2 A schematic diagram of the process of image feature processing provided in an embodiment of the present application;

[0078] Figure 3 A schematic diagram of the RAD feature processing flow provided in the embodiment of the present application;

[0079] Figure 4 Another schematic diagram of the process of the target detection method provided in the embodiment of the present application;

[0080] Figure 5 Another schematic diagram of a process of the target detection method provided in the embodiment of the present application;

[0081] Figure 6 A flowchart of a method for training a target detection model provided in an embodiment of the present application;

[0082] Figure 7 A schematic diagram of the structure of a target detection device provided in an embodiment of the present application;

[0083] Figure 8 A schematic diagram of the structure of a training device for a target detection model provided in an embodiment of the present application;

[0084] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0085] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0086] At present, the image data collected by image acquisition equipment can accurately reflect the color, texture and other information of objects in the scene, while the radar data collected by radar can more accurately reflect the position, movement speed and other information of objects. Therefore, image data and radar data have complementary characteristics. Fusion of radar data and image data can effectively make up for the shortcomings of single sensors in measurement accuracy and environmental adaptability, improve the accuracy of target detection, and enhance the detection capability of the system. The technology of fusing radar data and image data to improve the accuracy of target detection has great application value and potential in the fields of autonomous driving, intelligent transportation, etc.

[0087] At present, most radar data and image data fusion solutions are usually based on the fusion of point cloud data processed by radar constant false alarm rate (CFAR) and image data, but this will result in the loss of a large amount of original information collected by the radar, making target detection inaccurate.

[0088] RAD domain data is a form of representation of radar underlying three-dimensional data, namely three-dimensional tensor, which retains richer original features than point cloud data. Based on radar RAD domain underlying data and image fusion technology, it can effectively alleviate the information loss caused by CFAR processing, improve the detection capability of low signal-to-noise ratio (SNR) targets, and enhance robustness. However, RAD domain data is a dense three-dimensional data block, which is quite different from image data in terms of data dimension, resolution, spatial coordinate system, etc., which makes cross-modal fusion of RAD domain data and image data, especially in complex multi-target scenarios, very difficult.

[0089] In order to solve the above problems and better integrate RAD domain data with image data to improve the accuracy of target detection, the present application provides a target detection method, a target detection model training method and a device. The target detection method includes:

[0090] Acquire the image features of the first image to be detected and the RAD domain data to be detected of the scene to be detected collected at the current collection time, and the second image to be detected of the scene to be detected collected at the previous collection time;

[0091] Performing feature extraction on the first image to be detected to obtain features of the image to be detected; and performing feature extraction on the RAD domain data to be detected to obtain RAD features to be processed;

[0092] Generate a speed map to be processed based on the image features to be detected and the image features of the second image to be detected; wherein the speed value at each pixel position in the speed map to be processed represents: the radial speed at the real position represented by the element at the same pixel position in the image features to be detected;

[0093] The elements at the same pixel position of the image feature to be detected and the speed map to be processed are fused to obtain the fused image feature to be processed;

[0094] The fused image features to be processed and the RAD features to be processed are spliced ​​to obtain the joint features to be detected;

[0095] Target detection is performed based on the joint features to be detected to obtain detection results.

[0096] In this embodiment, the first image to be detected and the RAD domain data to be detected of the scene to be detected collected at the current acquisition moment are first obtained, and feature extraction is performed respectively to obtain the image features to be detected and the RAD features to be processed; then, based on the image features to be detected of the first image to be detected and the image features of the second image to be detected at the previous acquisition moment, a speed map to be processed is obtained, and the speed map to be processed can represent the radial velocity at the real position represented by the element at each pixel position in the image features to be detected, and then the image features to be detected and the elements at the same pixel position in the speed map to be processed are fused to obtain the fused image features to be processed, which can introduce speed information into the image features to be detected, and because the RAD domain data also contains speed information, the obtained fused image features to be processed are then spliced ​​with the RAD features to be processed, which can implicitly establish the connection between the two cross-modal features of image features and RAD features, so as to better fuse the image data with the RAD domain data, thereby improving the accuracy of target detection.

[0097] The target detection method provided by the embodiment of the present application is introduced below in conjunction with the accompanying drawings. Figure 1As shown, the target detection method may include the following steps:

[0098] S101, acquiring image features of a first image to be detected and RAD domain data to be detected of a scene to be detected collected at a current collection moment, and a second image to be detected of the scene to be detected collected at a previous collection moment;

[0099] The first image to be detected can be acquired by the image acquisition device at the current acquisition time for the scene to be detected, and the second image to be detected can be acquired by the image acquisition device at the previous acquisition time for the scene to be detected. In one implementation, the first image to be detected and the second image to be detected can be acquired by the image acquisition device periodically detecting the scene to be detected. For example, the first image to be detected and the second image to be detected can be two adjacent video frames in the video acquired by the image acquisition device.

[0100] The RAD domain data to be detected is a three-dimensional tensor. Each element in the RAD domain data to be detected represents the signal strength of the echo corresponding to a specific distance, azimuth and Doppler speed. In other words, each coordinate position in the RAD domain data to be detected represents the corresponding distance, azimuth and Doppler speed. The RAD domain data to be detected can be collected by the radar for the scene to be detected. Specifically, it can be obtained by performing fast Fourier transform on the original echo data collected by the radar in the distance dimension, azimuth dimension and Doppler speed dimension respectively.

[0101] S102, extracting features from the first image to be detected to obtain features of the image to be detected; and extracting features from the RAD domain data to be detected to obtain RAD features to be processed;

[0102] In this step, the features of the first image to be detected can be extracted by a preset convolutional neural network to obtain the features of the image to be detected, wherein the convolutional neural network can be ResNet (deep residual network) etc. Exemplarily, in the specific implementation process, the first image to be detected with a width of W and a height of H can be input into a ResNet-101 network with the terminal fully connected layer removed to obtain features of W / 32×H / 32×2048 dimensions, and then the obtained features are upsampled by three layers of deconvolution operation to obtain the features of the image to be detected. , where N w =W / 4,N h =H / 4,N C = 384. The above-mentioned image features to be detected are only exemplary introductions, and this embodiment does not specifically limit the dimensions of the image features to be detected.

[0103] In addition, each step of this embodiment can be executed periodically, that is, target detection is performed periodically on the scene to be detected. Therefore, the image features of the second image to be detected acquired at the last acquisition moment in this embodiment can also be obtained through the above processing.

[0104] For the RAD domain data to be detected, another preset convolutional neural network can be used to extract features from the RAD domain data to be detected. In one implementation, a 3D convolution kernel can be used to extract features from the RAD domain data to be detected to obtain the RAD features to be processed.

[0105] S103, generating a speed map to be processed based on the features of the image to be detected and the image features of the second image to be detected;

[0106] The velocity value at each pixel position in the velocity map to be processed represents: the radial velocity at the real position represented by the element at the same pixel position in the image feature to be detected;

[0107] In this embodiment, the image feature to be detected and the image feature of the second image to be detected have the same size. In one implementation, the image feature to be detected and the image feature of the second image to be detected can be spliced ​​along the channel dimension, and then the spliced ​​image features are input into a preset radial velocity estimation model to obtain a velocity map to be processed. The velocity map to be processed has the same width and height as the image feature to be detected.

[0108] It can be understood that each pixel position in the speed map to be processed actually represents an object at the corresponding position in the scene to be detected, and the element at the pixel position represents the radial velocity at the real position, that is, the velocity of the object, and the component velocity in the direction of the line connecting the object and the image acquisition device.

[0109] The radial velocity estimation model can be trained based on the image features of the first sample image and the image features of the second sample image, and the real velocity map; wherein the first sample image and the second sample image are acquired at two adjacent acquisition moments for the same scene; the velocity value at each pixel position in the real velocity map represents: the image features of the later acquired image in the first sample image and the second sample image, and the radial velocity at the real position represented by the elements at the same pixel position. Specifically, the image features of the first sample image and the image features of the second sample image can be input into the radial velocity estimation model of the initial structure to obtain the prediction map output by the radial velocity estimation model, and then the model loss is calculated based on the prediction map and the corresponding real velocity map, and then the model parameters of the radial velocity estimation model are adjusted based on the model loss to obtain the trained radial velocity estimation model.

[0110] S104, fusing the image feature to be detected and the elements at the same pixel position in the speed map to be processed to obtain the fused image feature to be processed;

[0111] In this step, the speed map to be processed and the image features to be detected can be spliced ​​along the channel dimension, and then the spliced ​​features are input into a preset coding layer for encoding to obtain the fused image features to be processed. The coding layer can be a multi-layer perceptron (MLP).

[0112] Exemplarily, the image features to be detected are , the processing speed graph is ; N w 、N h 、N C Respectively represent the width, height and number of channels of the feature, so when fusion is performed, the image features to be detected can be The feature vector at each pixel position in , the feature vector is concatenated with the speed value corresponding to the same pixel position in the speed map to be processed, and the concatenated feature vector is recorded as ; represents the channel splicing operation, and then the spliced ​​feature vector is input into the preset multi-layer perceptron for encoding to obtain the encoded feature vector, which is recorded as ; Where MLP represents a multi-layer perceptron, and vel represents the corresponding speed value. Finally, the fused image features to be processed can be composed of the encoded feature vectors.

[0113] It can be seen that the features of the fused image to be processed obtained through the processing of this embodiment are dynamically calculated values, which are calculated in real time with the image features and speed map of the currently acquired image. It can be combined with the timing information of the image to fully utilize the information of the image features, so as to improve the accuracy of subsequent target detection.

[0114] S105, concatenating the fused image features to be processed and the RAD features to be processed to obtain joint features to be detected;

[0115] Exemplarily, the feature of the fused image to be processed is ; The RAD features to be processed are ; When stitching, the features of the fused image to be processed can be Expand to feature vector ; The RAD features to be processed Expand to feature vector ; Then, the obtained feature vectors can be concatenated to obtain the joint feature to be detected .

[0116] S106, performing target detection based on the joint features to be detected to obtain a detection result.

[0117] After obtaining the joint features to be detected, in one implementation, the joint features to be detected can be input into a preset detection head to obtain a detection result. In a specific implementation process, the detection head can be a FFN (Prediction feed-forward networks) network.

[0118] It can be understood that the purpose of target detection technology is to determine whether there are objects of interest in the scene, and the properties of the objects of interest, such as type, position, direction, and other information. For example, in the field of unmanned driving, the objects of interest to be detected by target detection may be obstacles such as vehicles and buildings. Therefore, in this embodiment, the detection results may include at least one of: the type of the object of interest present in the scene to be detected, the position of the object of interest, the direction of the object of interest, the speed of the object of interest, etc. The specific detection results required can be set on the detection head according to the needs.

[0119] In this embodiment, the first image to be detected and the RAD domain data to be detected of the scene to be detected acquired at the current acquisition moment are first obtained, and feature extraction is performed respectively to obtain the image features to be detected and the RAD features to be processed; then, based on the image features to be detected of the first image to be detected and the image features of the second image to be detected at the previous acquisition moment, a speed map to be processed is obtained, and the speed map to be processed can represent the radial velocity at the real position represented by the element at each pixel position in the image features to be detected, and then the image features to be detected and the elements at the same pixel position in the speed map to be processed are fused to obtain the fused image features to be processed, which can introduce speed information into the image features to be detected, and because the RAD domain data also contains speed information, the obtained fused image features to be processed are then spliced ​​with the RAD features to be processed of the RAD domain data to be detected, and the connection between the two cross-modal features, image features and RAD features, can be implicitly established by introducing speed information, laying the foundation for feature interaction in the subsequent decoding process, thereby promoting better fusion of the two.

[0120] Moreover, the features of the fused image to be processed are dynamically calculated values, which are calculated in real time with the image features and speed map of the currently acquired image. The image's temporal information can be combined to fully utilize the information of the image features, and further utilize the potential information to enhance the perception ability. Therefore, this solution can improve the accuracy of target detection. This solution also has good versatility and can be applied to scenes such as traffic and driving where target detection is required.

[0121] In one embodiment of the present application, the above-mentioned feature extraction, feature fusion and target detection based on the joint features to be detected can be performed by a pre-trained target detection model, that is:

[0122] In the above steps, extracting features from the first image to be detected to obtain features of the image to be detected may include:

[0123] The first image to be detected is subjected to feature extraction through an image feature extraction network in a pre-trained target detection model to obtain features of the image to be detected; wherein the target detection model further includes: a RAD feature extraction network, a speed map output layer, a first encoding layer and a detection network.

[0124] In the above steps, feature extraction is performed on the RAD domain data to be detected to obtain the RAD features to be processed, which may include:

[0125] The RAD feature extraction network is used to extract features from the RAD domain data to be detected, and the RAD features to be processed are obtained.

[0126] In the above steps, generating the speed map to be processed based on the features of the image to be detected and the image features of the second image to be detected may include:

[0127] The image features to be detected and the image features of the second image to be detected are input into the above-mentioned speed map output layer to obtain the speed map to be processed. The speed map output layer can be: based on the image features of the first sample image and the image features of the second sample image, and the real speed map training; the first sample image and the second sample image are acquired at two adjacent acquisition moments for the same scene; the speed value at each pixel position in the real speed map represents: the image features of the later acquired image in the first sample image and the second sample image, and the radial speed at the real position represented by the elements at the same pixel position.

[0128] That is to say, the speed map output layer can be pre-supervised trained before the target detection model is trained, so as to improve the accuracy of the speed map output by the speed map output layer, and also accelerate the convergence of the target detection model when training the entire target detection model. The speed map output layer can be a multi-layer perceptron. In the specific implementation process, the image features to be detected and the image features of the second image to be detected can be spliced ​​along the channel dimension, and then the spliced ​​image features are input into the speed map output layer to obtain the speed map to be processed.

[0129] In the above steps, the elements at the same pixel position of the image feature to be detected and the speed map to be processed are fused to obtain the fused image feature to be processed, which may include:

[0130] The first coding layer is used to fuse the elements at the same pixel position of the image feature to be detected and the speed map to be processed to obtain the fused image feature to be processed.

[0131] In the above steps, target detection is performed based on the joint features to be detected to obtain the detection results, which may include:

[0132] The detection network is used to perform target detection on the joint features to be detected to obtain the detection results.

[0133] In this embodiment, the entire target detection process can be completed by a pre-trained target detection model, and the target detection model can be obtained by end-to-end training to improve the accuracy of target detection. The specific training process will be introduced in the following embodiment.

[0134] In one embodiment of the present application, before using the first coding layer to fuse the image features to be detected and the elements at the same pixel position in the speed map to be processed to obtain the fused image features to be processed, the target detection method further includes:

[0135] Step A1, obtaining a depth map to be processed corresponding to the image feature to be detected; wherein the depth value at each pixel position in the depth map to be processed represents: the depth at the real position represented by the element at the same pixel position in the image feature to be detected;

[0136] It can be understood that each pixel position in the depth map actually represents an object at a position in the scene to be detected, and the element at the pixel position represents the depth at the real position, that is, the distance between the object and the image acquisition device.

[0137] In one implementation, the depth map to be processed can be obtained by detecting the scene to be detected by an infrared sensor, a binocular camera, etc. In another implementation, the target detection model also includes: a depth map output layer; in this case, the depth map to be processed corresponding to the image feature to be detected can also be obtained by the following steps:

[0138] The image features to be detected are input into the depth map output layer to obtain the depth map to be processed; wherein the depth map output layer is: trained based on the image features of the first sample image and the real depth map corresponding to the first sample image; the depth value at each pixel position in the real depth map represents: the depth at the real position represented by the element at the same pixel position in the image features of the first sample image.

[0139] In this implementation, the depth map output layer can also be a multi-layer perceptron. The depth map output layer can be pre-supervised trained before the target detection model is trained, so as to improve the accuracy of the depth map output by the depth map output layer and accelerate the model convergence speed when training the entire target detection model. Specifically, the image features of the first sample image can be input into the depth map output layer of the initial structure to obtain the predicted image output by the depth map output layer. After that, the model loss is calculated based on the predicted image and the corresponding true depth map, and then the model parameters of the depth map output layer are adjusted based on the model loss to obtain the trained depth map output layer.

[0140] Step A2, for each pixel position in the image feature to be detected, based on the coordinate system conversion relationship between the image acquisition device and the radar that acquires the RAD domain data to be detected, and the depth value at the same pixel position in the depth map to be processed, determine the polar coordinate value of the pixel position in the radar coordinate system of the radar;

[0141] Among them, the conversion relationship between the image acquisition device and the radar coordinate system can be expressed by the following formula:

[0142] ; Wherein, (u, v) represents any pixel position in the image feature to be detected; (x, y, z) is the Cartesian coordinate value of the pixel position in the radar coordinate system; depth is the depth value at the same pixel position in the depth map to be processed; K is the intrinsic parameter matrix of the image acquisition device, R and t represent the rotation matrix and translation matrix from the image coordinate system of the image acquisition device to the radar coordinate system; T represents matrix transpose.

[0143] Through this formula, the Cartesian coordinate value of any pixel coordinate in the image feature to be detected can be converted to the radar coordinate system. Then, the polar coordinate value of the pixel position in the radar coordinate system of the radar can be obtained according to the conversion relationship between the Cartesian coordinate value and the polar coordinate value. The conversion relationship can be expressed by the formula as follows:

[0144] ;in, Represents polar coordinate values.

[0145] In this embodiment, the features of the fused image to be processed can be obtained through the following steps:

[0146] The first coding layer is used to fuse the elements at the same pixel position in the image feature to be detected and the speed map to be processed, as well as the polar coordinate value of the same pixel position in the radar coordinate system, to obtain the fused image feature to be processed.

[0147] That is to say, this embodiment can fuse each element in the image feature to be detected with the corresponding radial velocity and polar coordinate value to obtain the fused image feature to be processed.

[0148] In one implementation, this step may include:

[0149] Step B1, splicing the elements at the same pixel position in the image feature to be detected and the speed map to be processed, as well as the polar coordinate values ​​of the same pixel position in the radar coordinate system, to obtain the spliced ​​image feature to be processed;

[0150] Exemplarily, the image features to be detected are , the image features to be detected can be Each eigenvector in It is concatenated with the corresponding speed value and the corresponding depth value to obtain the concatenated feature vector, which can be recorded as ; The features of the spliced ​​image to be processed are The concatenated feature vectors.

[0151] Step B2, inputting the to-be-processed spliced ​​image feature into the first coding layer to obtain a first initial fused image feature; wherein the first initial fused image feature has the same number of channels as the to-be-detected image feature;

[0152] The first coding layer can be a multi-layer perceptron. In the specific implementation process, each feature vector of the spliced ​​image feature to be processed can be processed by the first coding layer. Encode so that the number of elements contained in each encoded feature vector is the same as the number of channels of the image feature to be detected. The first initial fused image feature obtained also contains The encoded feature vector.

[0153] Step B3, fusing the first initial fused image feature with the image feature to be detected to obtain the fused image feature to be processed.

[0154] In this step, the first initial fused image feature and the image feature to be detected can have the same width, height and number of channels. In this way, the values ​​of the first initial fused image feature and the image feature to be detected at the same position can be added respectively to achieve feature fusion and obtain the fused image feature to be processed.

[0155] By fusing the first initial fused image features obtained through a series of processing with the initially obtained features of the image to be detected, the image information can be more fully utilized, thereby further improving the accuracy of target detection.

[0156] In a specific implementation of this embodiment, the process of processing the image features can be as follows: Figure 2 As shown:

[0157] First, the features of the image to be detected collected at the current acquisition moment and the image features of the second image to be detected collected at the previous acquisition moment are spliced, and the spliced ​​features are input into the speed map output layer to obtain a speed map; the features of the image to be detected are input into the depth map output layer to obtain a depth map.

[0158] Afterwards, the image feature to be detected is concatenated with the corresponding depth value in the depth map and the corresponding speed value in the speed map, and is input into the first coding layer for coding calculation to obtain the first initial fused image feature. Finally, the first initial fused image feature is added to the elements at the corresponding positions of the image feature to be detected to obtain the fused image feature to be processed.

[0159] In this embodiment, by fusing the elements at the same pixel position in the image feature to be detected and the velocity map to be processed, as well as the polar coordinate values ​​of the same pixel position in the radar coordinate system, the distance, azimuth and radial velocity information are introduced into the image feature, which can better establish a connection with the distance, azimuth and Doppler velocity information contained in the RAD domain data, so as to further improve the fusion effect of the RAD domain data and the image data, and can also better decouple the error of the distance-azimuth information in the image information, thereby further improving the accuracy of target detection.

[0160] In one embodiment of the present application, the RAD feature extraction network may include: a RAD feature extraction layer and a second encoding layer;

[0161] The above-mentioned RAD feature extraction network is used to extract features of the RAD domain data to be detected, and obtains the RAD features to be processed, which may include the following steps:

[0162] Step C1, extracting features from the RAD domain data to be detected through a RAD feature extraction layer to obtain a first initial RAD feature;

[0163] In one implementation, the RAD feature extraction layer may be a 3D convolution kernel, that is, the 3D convolution kernel is used to perform feature extraction on the RAD domain data to be detected to obtain the RAD features to be processed.

[0164] In one implementation, the RAD feature extraction layer may include: a first feature extraction unit, a second feature extraction unit and a third feature extraction unit; Figure 3 As shown, the above-mentioned feature extraction of the RAD domain data to be detected by the RAD feature extraction layer to obtain the first initial RAD feature may include the following steps:

[0165] Step C11, projecting the RAD domain data to be detected along the Doppler velocity dimension, the azimuth dimension and the distance dimension respectively to obtain a RA image to be processed, a RD image to be processed and an AD image to be processed;

[0166] This step actually compresses the RAD domain data to be detected along three dimensions respectively to obtain three sets of two-dimensional data. Specifically, the RAD domain data to be detected is a three-dimensional tensor, and its three coordinate axes represent the Doppler velocity dimension, azimuth dimension and distance dimension respectively. Therefore, three two-dimensional images can be obtained by projecting along the Doppler velocity dimension, azimuth dimension and distance dimension respectively. Since each coordinate position in the RAD domain data to be detected represents the corresponding distance, azimuth and Doppler velocity, and each coordinate position records the signal strength of the echo received by the radar, therefore, the value at any coordinate position in the RA image to be processed, the RD image to be processed and the AD image to be processed can be a statistical value such as the sum or average value of the signal strength of the RAD domain data to be detected projected to the coordinate position.

[0167] Step C12, performing feature extraction on the RA image to be processed, the RD image to be processed, and the AD image to be processed respectively by the first feature extraction unit, the second feature extraction unit, and the third feature extraction unit to obtain a first initial RA feature, a first initial RD feature, and a first initial AD feature;

[0168] In this embodiment, the first feature extraction unit, the second feature extraction unit and the third feature extraction unit can all be convolutional neural networks.

[0169] Step C13, according to a uniform size, respectively expanding the first initial RA feature in the Doppler velocity dimension, expanding the first initial RD feature in the azimuth dimension, and expanding the first initial AD feature in the distance dimension, to obtain a to-be-used RA feature, a to-be-used RD feature, and a to-be-used AD feature;

[0170] Since the data is reduced in dimension in the above feature extraction step, the features obtained can be copied in multiple copies in this step to achieve expansion and dimensionality increase. In addition, in order to facilitate the subsequent splicing of the features, the features can be expanded according to a uniform size, that is, the sizes of the expanded dimensions are the same.

[0171] For example, for a size The RAD domain data to be detected, the obtained first initial RA feature, the first initial RD feature and the first initial AD feature can be respectively: , and ; where N r Represents the size of the feature in the distance dimension; N a Indicates the size of the feature in the azimuthal dimension; N d represents the size of the feature in the Doppler velocity dimension; N C The number of channels representing features; N r =N R / 2,N a =N R / 2,N d =N R / 2,N C = 384. The above-mentioned features are merely exemplary, and the dimensions of the features are not specifically limited in this embodiment.

[0172] In order to make the feature size uniform after expansion, , which can be copied N d The RA features to be used can be obtained ;against , you can copy N a , and obtain the RD features to be used ;against , you can copy N r Get the AD features to be used .

[0173] Step C14, concatenating the RA feature to be used, the RD feature to be used, and the AD feature to be used along the channel dimension to obtain a first initial RAD feature.

[0174] For example, the expanded RA feature , expanded RD features , Expanded AD features By splicing along the channel dimension, we can get , as the first initial RAD feature.

[0175] In this embodiment, the RAD domain data to be detected is reduced in dimension into three two-dimensional images: a RA image to be processed, an RD image to be processed, and an AD image to be processed, and then features are extracted from these three two-dimensional images respectively, and finally expanded and spliced ​​to obtain RAD features. Compared with the method of directly using a 3D convolution kernel to extract features from three-dimensional data, the amount of calculation can be greatly reduced, thereby improving the efficiency of target detection in this embodiment.

[0176] Step C2, for each coordinate position in the first initial RAD feature, use the second coding layer to fuse the element at the coordinate position in the first initial RAD feature, the distance, azimuth and Doppler velocity represented by the coordinate position to obtain the RAD feature to be processed.

[0177] It can be understood that the RAD domain data to be detected is a three-dimensional tensor, in which each coordinate position can represent the corresponding distance, azimuth and Doppler speed. After the RAD domain data to be detected is feature extracted to obtain the first initial RAD feature, each coordinate position of the first initial RAD feature can also represent the distance, azimuth and Doppler speed. In order to make more full use of the information of the RAD domain data, for each coordinate position in the first initial RAD feature, the element at the coordinate position in the first initial RAD feature, the distance, azimuth and Doppler speed represented by the coordinate position can be fused, thereby further improving the accuracy of target detection.

[0178] In one implementation, this step may adopt a similar splicing and encoding method as steps B1-B3 above, so as to better establish a cross-modal connection between the RAD features and the image features. This step may specifically include:

[0179] Step C21, for each coordinate position in the first initial RAD feature, splicing the element at the coordinate position with the distance, azimuth and Doppler velocity represented by the coordinate position to obtain a spliced ​​RAD feature to be processed;

[0180] The RAD features to be processed in this step can be recorded as ;in, represents the eigenvector composed of the elements at the coordinate position; r, a, and d are the distance, azimuth, and Doppler velocity represented by the coordinate position, respectively.

[0181] Step C22, inputting the to-be-processed spliced ​​RAD feature into the second coding layer to obtain a first fused RAD feature; wherein the first fused RAD feature has the same number of channels as the to-be-detected image feature;

[0182] The second encoding layer can also be a multi-layer perceptron. The first fused RAD feature in this step can be recorded as In this step, the first fused RAD feature and the RAD feature to be detected may have the same size and number of channels, so that the values ​​of the first fused RAD feature and the RAD feature to be detected at the same position may be added respectively to achieve feature fusion and obtain the first fused RAD feature.

[0183] Step C23, fusing the first fused RAD feature with the first initial RAD feature to obtain the RAD feature to be processed.

[0184] In this step, by fusing the first fused RAD feature obtained after a series of processing with the first initial RAD feature obtained initially, the information of the RAD domain data can be more fully utilized, thereby further improving the accuracy of target detection.

[0185] In this embodiment, the RAD domain data to be detected is firstly feature extracted through the RAD feature extraction layer to obtain the first initial RAD feature, and then the elements at each coordinate position in the first initial RAD feature are fused with the corresponding distance, azimuth and Doppler velocity to obtain the RAD feature to be processed. This can make fuller use of the information of the RAD domain data and can also better establish the connection with the image features, thereby further improving the accuracy of target detection.

[0186] In one embodiment of the present application, the image acquisition device for acquiring the first image to be detected and the radar for acquiring the RAD domain data to be detected are both arranged on the same preset carrier; the preset carrier can be an object such as a mobile robot or a car that needs to perform target detection on the surrounding scene, or a stationary object such as a pole or a wall. The detection network can include: an initialization layer, a decoding layer, and a detection head;

[0187] like Figure 4 As shown, the above-mentioned use of the detection network to perform target detection on the joint features to be detected to obtain the detection result may include the following steps:

[0188] Step D1, for each detection anchor point set within the preset sensing range of the preset carrier, the distance, azimuth, preset Doppler speed of the detection anchor point and the absolute speed parameter corresponding to the detection anchor point are merged through the initialization layer to obtain a query vector corresponding to the detection anchor point; the absolute speed parameter corresponding to each detection anchor point is determined based on the absolute speed of the detection anchor point;

[0189] The preset sensing range and the detection anchor points within the preset sensing range can be set according to experience and needs. Each detection anchor point has a corresponding preset distance, direction and Doppler speed. The set of detection anchor points can be recorded as ; Among them, r i , a j , d k Represents the preset distance, azimuth and Doppler speed respectively, N qr , N qa ; N qd Respectively represent the number of values ​​of the detection anchor point in the distance, azimuth and Doppler velocity dimensions. For example, the preset sensing range in the distance dimension is 0~100 meters, then a value can be taken every 10 meters in this dimension, and 11 values ​​are obtained; the preset sensing range in the azimuth dimension is 0~120°, then a value can be taken every 10° in this dimension, and 13 values ​​are obtained; the preset sensing range in the Doppler velocity dimension is -40 meters per second to +40 meters per second, then a value can be taken every 10 meters per second in this dimension, and 9 values ​​are obtained, so a total of 11×13×9 detection anchor points can be obtained.

[0190] In this embodiment, the absolute speed information can be introduced into the query vector corresponding to the detection anchor point. The absolute speed refers to the speed in the world coordinate system. It can be understood that the Doppler speed d of the detection anchor point k That is, the relative speed of the detection anchor point relative to the preset carrier. Therefore, the absolute speed of the detection anchor point can be calculated by the following formula:

[0191] ; where v ego To preset the moving speed of the carrier at the current collection moment.

[0192] In one implementation, the absolute speed parameter corresponding to each detection anchor point may be: ; Where exp represents the natural exponential function; is a preset constant, for example, in the specific implementation process Can be taken .

[0193] The initialization layer can also be a multi-layer perceptron. After fusing the distance, azimuth, preset Doppler velocity of the detection anchor point with the absolute velocity parameter corresponding to the detection anchor point, the distance, azimuth, preset Doppler velocity and absolute velocity parameter of the detection anchor point can be concatenated first, and then the concatenated vector is output to the initialization layer to obtain the query vector corresponding to the detection anchor point. The process can be expressed by the following formula:

[0194] ; Among them, q ijk That is, the query vector corresponding to the detection anchor point.

[0195] Step D2, fusing each query vector with the joint feature to be detected through the decoding layer based on the attention mechanism to obtain an updated query vector;

[0196] Due to the sensor characteristics of the radar, the stationary background clutter will be mixed in the echo signal received by the radar, so there will be a certain amount of noise in the RAD domain data. In this embodiment, the absolute speed information is introduced into the query vector, so that when each query vector is subjected to multimodal feature fusion interaction with the joint feature to be detected through the decoding layer based on the attention mechanism, the weights calculated by the decoding layer through the attention mechanism can implicitly separate the dynamic and static targets in the feature information, that is, the score of the static target information is suppressed through the weight, and the score of the dynamic target information is improved, so that the interference problem of the stationary background clutter can be reduced, and the signal-to-noise ratio of the updated query vector can be improved, thereby further improving the accuracy of target detection.

[0197] Specifically, the decoding layer may be a decoder having multiple layers of cascaded transfomers, wherein the first The layer decoder can be expressed as:

[0198] ; K and V are obtained by performing two different linear changes on the joint features to be detected; D is the normalization constant; softmax represents the normalized exponential function.

[0199] Step D3: input each updated query vector into the detection head to obtain the detection result of each detection anchor point.

[0200] After obtaining the updated query vector corresponding to each detection anchor point, each query vector can be input into the detection head of the target detection model to obtain the detection result for the detection anchor point. The detection result of each detection anchor point includes: at least one of the confidence of the detection anchor point, the type at the detection anchor point, the position at the detection anchor point, the orientation at the detection anchor point, and the speed at the detection anchor point. Among them, if the confidence of a detection anchor point is less than a preset threshold, it is considered that the detection result of the detection anchor point is inaccurate, and the detection result of the detection anchor point can be discarded. The type at the detection anchor point, the position at the detection anchor point, the orientation at the detection anchor point, and the speed at the detection anchor point can respectively represent the type of the object of interest, the position of the object of interest, the orientation of the object of interest, or the speed of the object of interest at the detection anchor point.

[0201] In this embodiment, by fusing the query vector of each detection anchor point with the joint feature to be detected and then using the detection head for detection, the detection head can pay more attention to whether there is an object of interest near the detection anchor point, as well as information such as the type of object of interest near the detection anchor point, the position of the object of interest, the direction of the object of interest or the speed of the object of interest, thereby achieving more refined detection.

[0202] In addition, this embodiment also introduces absolute speed information into the query vector, so that when each query vector is fused with the joint feature to be detected through the decoding layer based on the attention mechanism, the interference problem of static background clutter can be reduced and the signal-to-noise ratio of the updated query vector can be improved. Therefore, the accuracy of target detection can be further improved through this embodiment.

[0203] For ease of understanding, the following Figure 5 The processing flow of the embodiment of the present application is briefly introduced.

[0204] like Figure 5As shown, first, feature extraction is performed on the first image to be detected and the RAD domain data to be detected collected at the current acquisition moment to obtain the image feature to be detected and the first initial RAD feature. After that, image feature encoding is performed by combining the image feature to be detected and the image feature collected at the previous acquisition moment (i.e., the image feature of the second image to be detected), including: generating a speed map to be processed based on the image feature to be detected and the image feature of the second image to be detected; fusing the elements at the same pixel position in the image feature to be detected and the speed map to be processed, as well as the polar coordinate value of the same pixel position in the radar coordinate system, to obtain the fused image feature to be processed.

[0205] In addition, the first initial RAD feature needs to be RAD feature encoded, that is, the RAD domain data to be detected is subjected to feature extraction through the RAD feature extraction layer to obtain the first initial RAD feature; for each coordinate position in the first initial RAD feature, the second encoding layer is used to fuse the element at the coordinate position in the first initial RAD feature, the distance, azimuth and Doppler velocity represented by the coordinate position to obtain the RAD feature to be processed.

[0206] After obtaining the fused image features to be processed and the RAD features to be processed, the fused image features to be processed and the RAD features to be processed can be expanded and concatenated to obtain the joint features to be detected. Finally, based on the attention mechanism, multimodal feature fusion interaction is performed on the joint features to be detected and the preset query vector to obtain an updated query vector; the updated query vector is input into the detection head for target detection to obtain the detection result.

[0207] It can be seen that through this scheme, speed information is introduced into the image features to be detected, and then the obtained fused image features to be processed are spliced ​​with the RAD features to be processed of the RAD domain data to be detected, which can better establish the continuity between the image features and the RAD features, so as to better fuse the image data with the RAD domain data, thereby improving the accuracy of target detection.

[0208] The embodiment of the present application also provides a method for training a target detection model, wherein the target detection model includes: an image feature extraction network, a RAD feature extraction network, a speed map output layer, a first encoding layer, and a detection network; Figure 6 As shown, the training method of the target detection model includes:

[0209] S601, acquiring a first sample image and sample RAD domain data of a sample scene at a first acquisition moment, and image features of a second sample image of the sample scene acquired at an acquisition moment before the first acquisition moment;

[0210] S602, extracting features from the first sample image through an image feature extraction network to obtain sample image features; and extracting features from the sample RAD domain data through a RAD feature extraction network to obtain sample RAD features;

[0211] S603, inputting the sample image features and the image features of the second sample image into the speed map output layer to obtain a sample speed map;

[0212] The velocity value at each pixel position in the sample velocity map represents: the radial velocity at the real position represented by the element at the same pixel position in the sample image feature;

[0213] S604, using the first coding layer to fuse the sample image features and the elements at the same pixel position in the sample velocity map to obtain sample fused image features;

[0214] S605, concatenating the sample fusion image feature and the sample RAD feature to obtain the sample joint feature;

[0215] S606, using the detection network to perform target detection on the sample joint features to obtain a prediction result of the target detection;

[0216] S607, calculating the loss function based on the prediction result and the true value of the pre-calibrated detection result to adjust the model parameters of the target detection model until the preset convergence condition is reached to obtain a trained target detection model.

[0217] Optionally, before using the first coding layer to fuse the elements at the same pixel position in the sample image feature and the sample velocity map to obtain the sample fused image feature, the method also includes: obtaining a sample depth map corresponding to the sample image feature; wherein the depth value at each pixel position in the sample depth map represents: the depth at the real position represented by the element at the same pixel position in the sample image feature; for each pixel position in the sample image feature, based on the coordinate system conversion relationship between the image acquisition device and the radar that collects the sample RAD domain data, and the depth value at the same pixel position in the sample depth map, determine the polar coordinate value of the pixel position in the radar coordinate system of the radar;

[0218] The sample image features and the elements at the same pixel position in the sample velocity map are fused by using the first coding layer to obtain the sample fused image features, including: the sample image features and the elements at the same pixel position in the sample velocity map and the polar coordinate values ​​of the same pixel position in the radar coordinate system are fused by using the first coding layer to obtain the sample fused image features.

[0219] Optionally, the first coding layer is used to fuse the sample image features and the elements at the same pixel position in the sample velocity map, as well as the polar coordinate values ​​of the same pixel position in the radar coordinate system to obtain the sample fused image features, including: splicing the sample image features and the elements at the same pixel position in the sample velocity map, as well as the polar coordinate values ​​of the same pixel position in the radar coordinate system to obtain the sample spliced ​​image features; inputting the sample spliced ​​image features into the first coding layer to obtain the second initial fused image features; wherein the second initial fused image features have the same number of channels as the sample image features; and fusing the second initial fused image features with the sample image features to obtain the sample fused image features.

[0220] Optionally, the RAD feature extraction network includes: a RAD feature extraction layer and a second encoding layer;

[0221] The sample RAD domain data is subjected to feature extraction through a RAD feature extraction network to obtain a sample RAD feature, including: extracting features from the sample RAD domain data through a RAD feature extraction layer to obtain a second initial RAD feature; for each coordinate position in the second initial RAD feature, using a second coding layer to fuse the element at the coordinate position in the second initial RAD feature, the distance, azimuth and Doppler velocity represented by the coordinate position to obtain a sample RAD feature.

[0222] Optionally, for each coordinate position in the second initial RAD feature, the second coding layer is used to fuse the element at the coordinate position in the second initial RAD feature, the distance, azimuth and Doppler velocity represented by the coordinate position to obtain a sample RAD feature, including: for each coordinate position in the second initial RAD feature, splicing the element at the coordinate position with the distance, azimuth and Doppler velocity represented by the coordinate position to obtain a sample spliced ​​RAD feature; inputting the sample spliced ​​RAD feature into the second coding layer to obtain a second fused RAD feature; wherein the second fused RAD feature has the same number of channels as the second initial RAD feature; fusing the second fused RAD feature with the second initial RAD feature to obtain a sample RAD feature.

[0223] Optionally, the RAD feature extraction layer includes: a first feature extraction unit, a second feature extraction unit and a third feature extraction unit;

[0224] The sample RAD domain data is subjected to feature extraction through the RAD feature extraction layer to obtain a second initial RAD feature, including: projecting the sample RAD domain data along the Doppler velocity dimension, the azimuth dimension and the distance dimension to obtain a sample RA image, a sample RD image and a sample AD image; extracting features from the sample RA image, the sample RD image and the sample AD image through the first feature extraction unit, the second feature extraction unit and the third feature extraction unit to obtain a second initial RA feature, a second initial RD feature and a second initial AD feature; expanding the second initial RA feature in the Doppler velocity dimension, the second initial RD feature in the azimuth dimension and the second initial AD feature in the distance dimension according to a uniform size to obtain a sample RA feature, a sample RD feature and a sample AD feature; splicing the sample RA feature, the sample RD feature and the sample AD feature along the channel dimension to obtain a second initial RAD feature.

[0225] Optionally, an image acquisition device for acquiring the first sample image and a radar for acquiring sample RAD domain data are both arranged on the sample object; the detection network comprises: an initialization layer, a decoding layer and a detection head;

[0226] The detection network is used to perform target detection on the sample joint features to obtain a prediction result of the target detection, including: for each sample anchor point set within the preset perception range of the sample object, the distance, azimuth, preset Doppler speed of the sample anchor point are fused with the absolute speed parameter corresponding to the sample anchor point through the initialization layer to obtain a query vector corresponding to the sample anchor point; the absolute speed parameter corresponding to each sample anchor point is determined based on the absolute speed of the sample anchor point; each query vector is fused with the sample joint features based on the attention mechanism through the decoding layer to obtain an updated query vector; and each updated query vector is input into the detection head to obtain a prediction result for each sample anchor point.

[0227] Optionally, the absolute speed parameter corresponding to each sample anchor point is: ; Where exp represents the natural exponential function; v1 ego is the moving speed of the sample object at the current acquisition moment; is a preset constant; a 1j is the orientation of the sample anchor point; d1 k The preset Doppler velocity for the sample anchor point.

[0228] Optionally, the prediction result of each sample anchor point includes: at least one of: the confidence of the sample anchor point, the type at the sample anchor point, the position at the sample anchor point, the orientation at the sample anchor point, and the speed at the sample anchor point.

[0229] In this embodiment, a first sample image and sample RAD domain data of a sample scene collected at a first collection moment are first obtained, and feature extraction is performed respectively to obtain sample image features and sample RAD features; then, a sample velocity map is obtained based on the sample image features of the first sample image and the image features of the second sample image at the previous collection moment. The sample velocity map can represent the radial velocity at the real position represented by the element at each pixel position in the sample image features, and then the sample image features and the elements at the same pixel position in the sample velocity map are fused to obtain sample fused image features, which can introduce velocity information into the sample image features. Since the RAD domain data also contains velocity information, the obtained sample fused image features are then spliced ​​with the sample RAD features, so that the connection between the two cross-modal features, image features and RAD features, can be implicitly established, so as to better fuse the image data with the RAD domain data to obtain sample joint features. Based on this, the target detection model is trained, and the accuracy of target detection performed by the trained target detection model can be improved.

[0230] Based on the same inventive concept, the present application also provides a target detection device, such as Figure 7 The target detection device comprises:

[0231] The to-be-detected data acquisition module 701 is used to acquire the first to-be-detected image and the to-be-detected RAD domain data of the to-be-detected scene acquired at the current acquisition moment, and the image features of the second to-be-detected image of the to-be-detected scene acquired at the previous acquisition moment;

[0232] The feature extraction module 702 is used to extract features from the first image to be detected to obtain features of the image to be detected; and to extract features from the RAD domain data to be detected to obtain RAD features to be processed;

[0233] The speed map output module 703 is used to generate a speed map to be processed based on the image feature to be detected and the image feature of the second image to be detected; wherein the speed value at each pixel position in the speed map to be processed represents: the radial speed at the real position represented by the element at the same pixel position in the image feature to be detected;

[0234] The feature fusion module 704 is used to fuse the image feature to be detected and the elements at the same pixel position in the speed map to be processed to obtain the fused image feature to be processed;

[0235] The feature splicing module 705 is used to splice the fused image feature to be processed and the RAD feature to be processed to obtain a joint feature to be detected;

[0236] The first target detection module 706 is used to perform target detection based on the joint features to be detected to obtain a detection result.

[0237] Optionally, the feature extraction module 702 to be detected is specifically used to: extract features of the first image to be detected through an image feature extraction network in a pre-trained target detection model to obtain features of the image to be detected; wherein the target detection model also includes: a RAD feature extraction network, a speed map output layer, a first encoding layer and a detection network; and extract features of the RAD domain data to be detected through the RAD feature extraction network to obtain RAD features to be processed;

[0238] The speed map output module 703 is specifically used to: input the image features to be detected and the image features of the second image to be detected into the speed map output layer to obtain the speed map to be processed;

[0239] The feature fusion module 704 is specifically used to: use the first coding layer to fuse the image feature to be detected and the elements at the same pixel position in the speed map to be processed to obtain the fused image feature to be processed;

[0240] The first target detection module 706 is specifically used to: use the detection network to perform target detection on the joint features to be detected to obtain a detection result.

[0241] Optionally, the target detection device further includes:

[0242] A depth map generation module to be processed is used for obtaining a depth map to be processed corresponding to the image feature to be detected before the feature fusion module to be processed 704 fuses the image feature to be detected and the elements at the same pixel position in the speed map to be processed using the first coding layer to obtain the fused image feature to be processed; wherein the depth value at each pixel position in the depth map to be processed represents: the depth at the real position represented by the element at the same pixel position in the image feature to be detected;

[0243] A first polar coordinate value determination module is used to determine, for each pixel position in the image feature to be detected, the polar coordinate value of the pixel position in the radar coordinate system of the radar based on the coordinate system conversion relationship between the image acquisition device and the radar that acquires the RAD domain data to be detected, and the depth value at the same pixel position in the depth map to be processed;

[0244] The feature fusion module 704 is specifically used to: use the first coding layer to fuse the image feature to be detected and the elements at the same pixel position in the speed map to be processed, as well as the polar coordinate values ​​of the same pixel position in the radar coordinate system, to obtain the fused image feature to be processed.

[0245] Optionally, the feature fusion module to be processed 704 includes:

[0246] A first feature stitching submodule is used to stitch the elements at the same pixel position in the image feature to be detected and the speed map to be processed, as well as the polar coordinate values ​​of the same pixel position in the radar coordinate system, to obtain a stitched image feature to be processed;

[0247] A first initial fusion submodule, used for inputting the to-be-processed spliced ​​image feature into the first coding layer to obtain a first initial fused image feature; wherein the first initial fused image feature has the same number of channels as the to-be-detected image feature;

[0248] The first feature fusion submodule is used to fuse the first initial fused image feature with the image feature to be detected to obtain the fused image feature to be processed.

[0249] Optionally, the RAD feature extraction network includes: a RAD feature extraction layer and a second encoding layer;

[0250] The feature extraction module 702 includes:

[0251] A first initial RAD feature extraction module, used for performing feature extraction on the RAD domain data to be detected through the RAD feature extraction layer to obtain a first initial RAD feature;

[0252] The first RAD fusion submodule is used to fuse the elements at the coordinate position in the first initial RAD feature, the distance, azimuth and Doppler velocity represented by the coordinate position, and obtain the RAD feature to be processed by using the second coding layer.

[0253] Optionally, the first RAD fusion submodule includes:

[0254] A first RAD feature stitching subunit is used to stitch, for each coordinate position in the first initial RAD feature, the element at the coordinate position with the distance, azimuth and Doppler velocity represented by the coordinate position to obtain a stitched RAD feature to be processed;

[0255] A first initial RAD feature fusion subunit, used for inputting the to-be-processed spliced ​​RAD feature into the second coding layer to obtain a first fused RAD feature; wherein the first fused RAD feature has the same number of channels as the first initial RAD feature;

[0256] The first initial RAD feature fusion subunit is used to fuse the first fused RAD feature with the first initial RAD feature to obtain the RAD feature to be processed.

[0257] Optionally, the RAD feature extraction layer includes: a first feature extraction unit, a second feature extraction unit and a third feature extraction unit;

[0258] The first initial RAD feature extraction module includes:

[0259] A first projection submodule is used to project the RAD domain data to be detected along the Doppler velocity dimension, the azimuth dimension and the distance dimension respectively to obtain a RA image to be processed, a RD image to be processed and an AD image to be processed;

[0260] A first RAD feature extraction submodule is used to extract features of the RA image to be processed, the RD image to be processed, and the AD image to be processed through the first feature extraction unit, the second feature extraction unit, and the third feature extraction unit, respectively, to obtain a first initial RA feature, a first initial RD feature, and a first initial AD feature;

[0261] A first expansion submodule is used to expand the first initial RA feature in a Doppler velocity dimension, expand the first initial RD feature and the azimuth dimension, and expand the first initial AD feature in a distance dimension according to a uniform size to obtain a to-be-used RA feature, a to-be-used RD feature, and a to-be-used AD feature;

[0262] The first RAD feature concatenation submodule is used to concatenate the RA feature to be used, the RD feature to be used, and the AD feature to be used along the channel dimension to obtain a first initial RAD feature.

[0263] Optionally, the image acquisition device for acquiring the first image to be detected and the radar for acquiring the RAD domain data to be detected are both arranged on a preset carrier; the detection network comprises: an initialization layer, a decoding layer and a detection head;

[0264] The first target detection module 706 includes:

[0265] A first query vector determination submodule is used to fuse the distance, azimuth, preset Doppler speed and absolute speed parameter corresponding to each detection anchor point set within the preset perception range of the preset carrier through the initialization layer to obtain a query vector corresponding to the detection anchor point; the absolute speed parameter corresponding to each detection anchor point is determined based on the absolute speed of the detection anchor point;

[0266] A first query vector updating submodule, configured to fuse each query vector with the joint feature to be detected through the decoding layer based on an attention mechanism to obtain an updated query vector;

[0267] The first detection submodule is used to input each updated query vector into the detection head to obtain a detection result of each detection anchor point.

[0268] Optionally, the absolute speed parameter corresponding to each detection anchor point is: ; where exp represents the natural exponential function; v ego is the moving speed of the preset carrier at the current collection moment; is a preset constant; a j is the position of the detection anchor point; d k The preset Doppler velocity for this detection anchor point.

[0269] Optionally, the detection result includes: at least one of: a confidence level of the object of interest, a type of the object of interest, a position of the object of interest, a direction of the object of interest, and a speed of the object of interest.

[0270] Optionally, the target detection model further includes: a depth map output layer;

[0271] The module for generating a depth map to be processed is specifically used to: input the image features to be detected into the depth map output layer to obtain a depth map to be processed; wherein the depth map output layer is: trained based on the image features of the first sample image and the real depth map corresponding to the first sample image; the depth value at each pixel position in the real depth map represents: the depth at the real position represented by the element at the same pixel position in the image features of the first sample image.

[0272] Optionally, the speed map output layer is: obtained based on image features of a first sample image, image features of a second sample image, and training of a real speed map; the first sample image and the second sample image are acquired at two adjacent acquisition moments for the same scene; the speed value at each pixel position in the real speed map represents: image features of the later acquired images in the first sample image and the second sample image, and radial speed at the real position represented by elements at the same pixel position.

[0273] The embodiment of the present application also provides a training device for a target detection model, wherein the target detection model includes: an image feature extraction network, a RAD feature extraction network, a speed map output layer, a first encoding layer, and a detection network; Figure 8 As shown, the device comprises:

[0274] The sample data acquisition module 801 is used to acquire a first sample image and sample RAD domain data of a sample scene at a first acquisition moment, and image features of a second sample image of the sample scene acquired at an acquisition moment before the first acquisition moment;

[0275] The sample feature extraction module 802 is used to extract features from the first sample image through the image feature extraction network to obtain sample image features; and to extract features from the sample RAD domain data through the RAD feature extraction network to obtain RAD features of the sample to be processed;

[0276] A sample speed map output module 803 is used to input the sample image features and the image features of the second sample image into the speed map output layer to obtain a sample speed map;

[0277] A sample feature fusion module 804 is used to fuse the sample image feature and the elements at the same pixel position in the sample velocity map using the first coding layer to obtain a sample fused image feature;

[0278] A sample feature splicing module 805 is used to splice the sample fusion image feature and the sample RAD feature to obtain a sample joint feature;

[0279] A second detection module 806 is used to perform target detection on the sample joint features using the detection network to obtain a prediction result of the target detection;

[0280] The model parameter adjustment module 807 is used to calculate the loss function based on the true value of the prediction result and the pre-calibrated detection result, so as to adjust the model parameters of the target detection model until the preset convergence condition is reached to obtain a trained target detection model.

[0281] Optionally, the training device of the target detection model further includes:

[0282] A sample depth map generation module is used to obtain a sample depth map corresponding to the sample image feature before the sample feature fusion module 804 fuses the sample image feature and the elements at the same pixel position in the sample speed map using the first coding layer to obtain the sample fused image feature; wherein the depth value at each pixel position in the sample depth map represents: the depth at the real position represented by the element at the same pixel position in the sample image feature;

[0283] A second polar coordinate value determination module is used to determine, for each pixel position in the sample image feature, a polar coordinate value of the pixel position in the radar coordinate system of the radar based on a coordinate system conversion relationship between the image acquisition device and the radar that acquires the sample RAD domain data, and a depth value at the same pixel position in the sample depth map;

[0284] The sample feature fusion module 804 is specifically used to: use the first coding layer to fuse the sample image features and the elements at the same pixel position in the sample velocity map, as well as the polar coordinate values ​​of the same pixel position in the radar coordinate system, to obtain sample fused image features.

[0285] Optionally, the sample feature fusion module 804 includes:

[0286] A second feature stitching submodule is used to stitch the sample image feature and the elements at the same pixel position in the sample velocity map, as well as the polar coordinate value of the same pixel position in the radar coordinate system, to obtain a sample stitching image feature;

[0287] A second initial fusion submodule, used for inputting the sample spliced ​​image feature into the first coding layer to obtain a second initial fused image feature; wherein the second initial fused image feature has the same number of channels as the sample image feature;

[0288] The second feature fusion submodule is used to fuse the second initial fused image feature with the sample image feature to obtain a sample fused image feature.

[0289] Optionally, the RAD feature extraction network includes: a RAD feature extraction layer and a second encoding layer;

[0290] The sample feature extraction module 802 includes:

[0291] A second initial RAD feature extraction submodule, used for performing feature extraction on the sample RAD domain data through the RAD feature extraction layer to obtain a second initial RAD feature;

[0292] The second RAD fusion submodule is used to fuse the elements at the coordinate position in the second initial RAD feature, the distance, azimuth and Doppler velocity represented by the coordinate position in the second initial RAD feature using the second coding layer for each coordinate position in the second initial RAD feature to obtain a sample RAD feature.

[0293] Optionally, the second RAD fusion submodule includes:

[0294] A second RAD feature stitching subunit is used to stitch, for each coordinate position in the second initial RAD feature, the element at the coordinate position with the distance, azimuth and Doppler velocity represented by the coordinate position to obtain a sample stitching RAD feature;

[0295] A second initial RAD feature fusion subunit is used to input the sample splicing RAD feature into the second encoding layer to obtain a second fused RAD feature; wherein the second fused RAD feature has the same number of channels as the second initial RAD feature;

[0296] The second initial RAD feature fusion subunit is used to fuse the second fused RAD feature with the second initial RAD feature to obtain a sample RAD feature.

[0297] Optionally, the RAD feature extraction layer includes: a first feature extraction unit, a second feature extraction unit and a third feature extraction unit;

[0298] The second initial RAD feature extraction module includes:

[0299] A second projection submodule is used to project the sample RAD domain data along the Doppler velocity dimension, the azimuth dimension and the distance dimension respectively to obtain a sample RA image, a sample RD image and a sample AD image;

[0300] A second RAD feature extraction submodule is used to perform feature extraction on the sample RA image, the sample RD image and the sample AD image through the first feature extraction unit, the second feature extraction unit and the third feature extraction unit, respectively, to obtain a second initial RA feature, a second initial RD feature and a second initial AD feature;

[0301] A second expansion submodule is used to expand the second initial RA feature in a Doppler velocity dimension, expand the second initial RD feature in an azimuth dimension, and expand the second initial AD feature in a distance dimension according to a uniform size to obtain a sample RA feature, a sample RD feature, and a sample AD feature;

[0302] The second RAD feature concatenation submodule is used to concatenate the sample RA feature, the sample RD feature and the sample AD feature along the channel dimension to obtain a second initial RAD feature.

[0303] Optionally, the image acquisition device for acquiring the first sample image and the radar for acquiring the sample RAD domain data are both arranged on the sample object; the detection network comprises: an initialization layer, a decoding layer and a detection head;

[0304] The second target detection module 806 includes:

[0305] A second query vector determination submodule is used for fusing the distance, azimuth, preset Doppler velocity and absolute velocity parameter corresponding to each sample anchor point set within the preset perception range of the sample object through the initialization layer to obtain a query vector corresponding to the sample anchor point; the absolute velocity parameter corresponding to each sample anchor point is determined based on the absolute velocity of the sample anchor point;

[0306] A second query vector updating submodule is used to fuse each query vector with the sample joint feature based on the attention mechanism through the decoding layer to obtain an updated query vector;

[0307] The second detection submodule is used to input each updated query vector into the detection head to obtain a prediction result for each sample anchor point.

[0308] Optionally, the absolute speed parameter corresponding to each sample anchor point is: ; Where exp represents the natural exponential function; v1 ego is the moving speed of the sample object at the current acquisition moment; is a preset constant; a 1j is the orientation of the sample anchor point; d1 k The preset Doppler velocity for the sample anchor point.

[0309] Optionally, the prediction result of each sample anchor point includes: at least one of: the confidence of the sample anchor point, the type at the sample anchor point, the position at the sample anchor point, the orientation at the sample anchor point, and the speed at the sample anchor point.

[0310] The present application also provides an electronic device, such as Fig. 9 As shown, including:

[0311] Memory 901, used for storing computer programs;

[0312] The processor 902 is used to implement any of the above-mentioned target detection methods or target detection model training methods when executing the program stored in the memory 901.

[0313] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 902, the communication interface, and the memory 901 communicate with each other via the communication bus.

[0314] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0315] The communication interface is used for communication between the above electronic device and other devices.

[0316] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0317] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0318] In another embodiment provided in the present application, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned target detection methods or target detection model training methods are implemented.

[0319] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any target detection method or target detection model training method in the above-mentioned embodiments.

[0320] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0321] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0322] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0323] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A target detection method, characterized in that: The method comprises: Acquire the image features of the first image to be detected and the RAD domain data to be detected of the scene to be detected collected at the current collection moment, and the second image to be detected of the scene to be detected collected at the previous collection moment; Performing feature extraction on the first image to be detected to obtain features of the image to be detected; and performing feature extraction on the RAD domain data to be detected to obtain RAD features to be processed; Generate a speed map to be processed based on the image features to be detected and the image features of the second image to be detected; wherein the speed value at each pixel position in the speed map to be processed represents: the elements at the same pixel position in the image features to be detected represent the radial speed of the object at the real position in the scene to be detected; The elements at the same pixel position in the image feature to be detected and the speed map to be processed, and the polar coordinate value of the same pixel position in the radar coordinate system of the radar that collects the RAD domain data to be detected are fused to obtain the fused image feature to be processed; wherein the polar coordinate value of each pixel position in the radar coordinate system is determined based on the coordinate system conversion relationship between the image acquisition device that collects the first image to be detected and the second image to be detected and the radar, and the depth of the real position represented by the pixel position; splicing the fused image feature to be processed and the RAD feature to be processed to obtain a joint feature to be detected; Target detection is performed based on the joint features to be detected to obtain a detection result.

2. The method according to claim 1, characterized in that Extracting features of the first image to be detected to obtain features of the image to be detected includes: Extracting features of the first image to be detected by an image feature extraction network in a pre-trained target detection model to obtain features of the image to be detected; wherein the target detection model further includes: a RAD feature extraction network, a speed map output layer, a first encoding layer and a detection network; Performing feature extraction on the RAD domain data to be detected to obtain RAD features to be processed includes: Performing feature extraction on the RAD domain data to be detected through the RAD feature extraction network to obtain RAD features to be processed; Generating a speed map to be processed based on the image features to be detected and the image features of the second image to be detected includes: Inputting the image features to be detected and the image features of the second image to be detected into the speed map output layer to obtain a speed map to be processed; The elements at the same pixel position in the image feature to be detected and the speed map to be processed, and the polar coordinate values ​​of the same pixel position in the radar coordinate system of the radar that collects the RAD domain data to be detected are fused to obtain the fused image feature to be processed, including: Using the first coding layer, the image feature to be detected and the element at the same pixel position in the speed map to be processed, and the polar coordinate value of the same pixel position in the radar coordinate system of the radar that collects the RAD domain data to be detected are fused to obtain the fused image feature to be processed; Performing target detection based on the joint features to be detected to obtain detection results includes: The detection network is used to perform target detection on the joint features to be detected to obtain a detection result.

3. The method according to claim 2, characterized in that Before obtaining the fused image feature to be processed by using the first coding layer to fuse the elements at the same pixel position in the image feature to be detected and the speed map to be processed, and the polar coordinate values ​​of the same pixel position in the radar coordinate system of the radar that collects the RAD domain data to be detected, the method further includes: Obtain a depth map to be processed corresponding to the image feature to be detected; wherein the depth value at each pixel position in the depth map to be processed represents: the depth at the real position represented by the element at the same pixel position in the image feature to be detected; For each pixel position in the image feature to be detected, the polar coordinate value of the pixel position in the radar coordinate system is determined based on the coordinate system conversion relationship between the image acquisition device and the radar, and the depth value at the same pixel position in the depth map to be processed.

4. The method according to claim 3, characterized in that Using the first coding layer, the elements at the same pixel position in the image feature to be detected and the speed map to be processed, and the polar coordinate values ​​of the same pixel position in the radar coordinate system of the radar that collects the RAD domain data to be detected are fused to obtain the fused image feature to be processed, including: splicing the elements at the same pixel position in the image feature to be detected and the speed map to be processed, as well as the polar coordinate values ​​of the same pixel position in the radar coordinate system, to obtain a spliced ​​image feature to be processed; Inputting the to-be-processed spliced ​​image feature into the first coding layer to obtain a first initial fused image feature; wherein the first initial fused image feature has the same number of channels as the to-be-detected image feature; The first initial fused image feature is fused with the image feature to be detected to obtain a fused image feature to be processed.

5. The method according to any one of claims 2 to 4, characterized in that: The RAD feature extraction network includes: a RAD feature extraction layer and a second encoding layer; The RAD feature extraction network is used to extract features from the RAD domain data to be detected, so as to obtain RAD features to be processed, including: Performing feature extraction on the RAD domain data to be detected through the RAD feature extraction layer to obtain a first initial RAD feature; For each coordinate position in the first initial RAD feature, the second coding layer is used to fuse the element at the coordinate position in the first initial RAD feature, the distance, azimuth and Doppler velocity represented by the coordinate position to obtain the RAD feature to be processed.

6. The method according to claim 5, characterized in that For each coordinate position in the first initial RAD feature, the second coding layer is used to fuse the element at the coordinate position in the first initial RAD feature, the distance, the azimuth, and the Doppler velocity represented by the coordinate position to obtain the RAD feature to be processed, including: For each coordinate position in the first initial RAD feature, splicing the element at the coordinate position with the distance, azimuth, and Doppler velocity represented by the coordinate position to obtain a spliced ​​RAD feature to be processed; Inputting the to-be-processed spliced ​​RAD feature into the second encoding layer to obtain a first fused RAD feature; wherein the first fused RAD feature has the same number of channels as the first initial RAD feature; The first fused RAD feature is fused with the first initial RAD feature to obtain a RAD feature to be processed.

7. The method according to claim 5, characterized in that The RAD feature extraction layer includes: a first feature extraction unit, a second feature extraction unit and a third feature extraction unit; The RAD feature extraction layer extracts features from the RAD domain data to be detected to obtain a first initial RAD feature, including: Projecting the RAD domain data to be detected along the Doppler velocity dimension, the azimuth dimension and the distance dimension respectively to obtain a RA image to be processed, a RD image to be processed and an AD image to be processed; By using the first feature extraction unit, the second feature extraction unit and the third feature extraction unit, feature extraction is performed on the RA image to be processed, the RD image to be processed and the AD image to be processed respectively to obtain a first initial RA feature, a first initial RD feature and a first initial AD feature; According to a unified size, the first initial RA feature is expanded in a Doppler velocity dimension, the first initial RD feature and the azimuth dimension are expanded, and the first initial AD feature is expanded in a distance dimension to obtain a to-be-used RA feature, a to-be-used RD feature, and a to-be-used AD feature; The RA feature to be used, the RD feature to be used, and the AD feature to be used are concatenated along the channel dimension to obtain a first initial RAD feature.

8. The method according to claim 2, characterized in that: The image acquisition device for acquiring the first image to be detected and the radar for acquiring the RAD domain data to be detected are both arranged on a preset carrier; the detection network comprises: an initialization layer, a decoding layer and a detection head; Utilizing the detection network to perform target detection on the joint features to be detected, and obtaining a detection result, including: For each detection anchor point set within the preset perception range of the preset carrier, the distance, azimuth, preset Doppler speed of the detection anchor point and the absolute speed parameter corresponding to the detection anchor point are fused through the initialization layer to obtain a query vector corresponding to the detection anchor point; the absolute speed parameter corresponding to each detection anchor point is determined based on the absolute speed of the detection anchor point; By fusing each query vector with the joint feature to be detected through the decoding layer based on the attention mechanism, an updated query vector is obtained; The updated query vectors are input into the detection head to obtain the detection result of each detection anchor point.

9. The method according to claim 8, characterized in that The absolute speed parameter corresponding to each detection anchor point is: ; where exp represents the natural exponential function; v ego is the moving speed of the preset carrier at the current collection moment; is a preset constant; a j is the position of the detection anchor point; d k The preset Doppler velocity for this detection anchor point.

10. The method according to claim 8, characterized in that The detection result of each detection anchor point includes at least one of: the confidence of the detection anchor point, the type at the detection anchor point, the position at the detection anchor point, the orientation at the detection anchor point, and the speed at the detection anchor point.

11. The method according to claim 3, characterized in that The target detection model also includes: a depth map output layer; Obtaining a depth map to be processed corresponding to the image feature to be detected, including: The image features to be detected are input into the depth map output layer to obtain a depth map to be processed; wherein the depth map output layer is trained based on the image features of the first sample image and the real depth map corresponding to the first sample image; the depth value at each pixel position in the real depth map represents the depth at the real position represented by the element at the same pixel position in the image features of the first sample image.

12. The method according to claim 2, characterized in that: The speed map output layer is: obtained based on the image features of the first sample image and the image features of the second sample image, and the real speed map training; the first sample image and the second sample image are acquired at two adjacent acquisition moments for the same scene; the speed value at each pixel position in the real speed map represents: the image features of the later acquired images in the first sample image and the second sample image, and the radial speed at the real position represented by the elements at the same pixel position.

13. A method for training a target detection model, characterized in that: The target detection model includes: an image feature extraction network, a RAD feature extraction network, a speed map output layer, a first encoding layer and a detection network; the method includes: Acquire a first sample image and sample RAD domain data of a sample scene at a first acquisition moment, and image features of a second sample image of the sample scene acquired at an acquisition moment previous to the first acquisition moment; Performing feature extraction on the first sample image through the image feature extraction network to obtain sample image features; and performing feature extraction on the sample RAD domain data through the RAD feature extraction network to obtain sample RAD features; Input the sample image features and the image features of the second sample image into the velocity map output layer to obtain a sample velocity map; wherein the velocity value at each pixel position in the sample velocity map indicates that the elements at the same pixel position in the sample image features represent the radial velocity of the object at the real position in the sample scene; Using the first coding layer, the sample image feature and the elements at the same pixel position in the sample velocity map, and the polar coordinate value of the same pixel position in the radar coordinate system of the radar that collects the sample RAD domain data are fused to obtain a sample fused image feature; wherein the polar coordinate value of each pixel position in the radar coordinate system is determined based on the coordinate system conversion relationship between the image acquisition device that collects the first sample image and the second sample image and the radar, and the depth of the real position represented by the pixel position; Concatenating the sample fusion image feature with the sample RAD feature to obtain a sample joint feature; Using the detection network to perform target detection on the sample joint features to obtain a prediction result of the target detection; The loss function is calculated based on the true value of the prediction result and the pre-calibrated detection result to adjust the model parameters of the target detection model until the preset convergence condition is reached to obtain a trained target detection model.

14. The method according to claim 13, characterized in that Before obtaining the sample fused image feature by using the first coding layer to fuse the sample image feature and the element at the same pixel position in the sample velocity map, and the polar coordinate value of the same pixel position in the radar coordinate system of the radar that collects the sample RAD domain data, the method further includes: Obtain a sample depth map corresponding to the sample image feature; wherein the depth value at each pixel position in the sample depth map represents: the depth at the real position represented by the element at the same pixel position in the sample image feature; For each pixel position in the sample image feature, based on the coordinate system conversion relationship between the image acquisition device and the radar, and the depth value at the same pixel position in the sample depth map, the polar coordinate value of the pixel position in the radar coordinate system is determined.

15. The method according to claim 14, characterized in that The sample image feature and the element at the same pixel position in the sample velocity map and the polar coordinate value of the same pixel position in the radar coordinate system of the radar that collects the sample RAD domain data are fused by using the first coding layer to obtain a sample fused image feature, including: splicing the sample image feature and the elements at the same pixel position in the sample velocity map, and the polar coordinate value of the same pixel position in the radar coordinate system to obtain a sample spliced ​​image feature; Inputting the sample spliced ​​image feature into the first coding layer to obtain a second initial fused image feature; wherein the second initial fused image feature has the same number of channels as the sample image feature; The second initial fused image feature is fused with the sample image feature to obtain a sample fused image feature.

16. The method according to any one of claims 13 to 15, characterized in that: The RAD feature extraction network includes: a RAD feature extraction layer and a second encoding layer; The sample RAD domain data is subjected to feature extraction by the RAD feature extraction network to obtain sample RAD features, including: Performing feature extraction on the sample RAD domain data through the RAD feature extraction layer to obtain a second initial RAD feature; For each coordinate position in the second initial RAD feature, the second coding layer is used to fuse the element at the coordinate position in the second initial RAD feature, the distance, azimuth and Doppler velocity represented by the coordinate position to obtain a sample RAD feature.

17. The method according to claim 16, characterized in that For each coordinate position in the second initial RAD feature, the second coding layer is used to fuse the element at the coordinate position in the second initial RAD feature, the distance, the azimuth, and the Doppler velocity represented by the coordinate position to obtain a sample RAD feature, including: For each coordinate position in the second initial RAD feature, stitching the element at the coordinate position with the distance, azimuth, and Doppler velocity represented by the coordinate position to obtain a sample stitching RAD feature; Inputting the sample concatenated RAD feature into the second encoding layer to obtain a second fused RAD feature; wherein the second fused RAD feature has the same number of channels as the second initial RAD feature; The second fused RAD feature is fused with the second initial RAD feature to obtain a sample RAD feature.

18. The method according to claim 16, characterized in that The RAD feature extraction layer includes: a first feature extraction unit, a second feature extraction unit and a third feature extraction unit; The RAD feature extraction layer extracts features from the sample RAD domain data to obtain a second initial RAD feature, including: Projecting the sample RAD domain data along the Doppler velocity dimension, azimuth dimension and distance dimension respectively to obtain a sample RA image, a sample RD image and a sample AD image; By using the first feature extraction unit, the second feature extraction unit and the third feature extraction unit, feature extraction is performed on the sample RA image, the sample RD image and the sample AD image respectively to obtain a second initial RA feature, a second initial RD feature and a second initial AD feature; According to a uniform size, the second initial RA feature is expanded in the Doppler velocity dimension, the second initial RD feature is expanded in the azimuth dimension, and the second initial AD feature is expanded in the distance dimension to obtain a sample RA feature, a sample RD feature, and a sample AD feature; The sample RA feature, the sample RD feature and the sample AD feature are concatenated along the channel dimension to obtain a second initial RAD feature.

19. The method according to claim 13, characterized in that The image acquisition device for acquiring the first sample image and the radar for acquiring the sample RAD domain data are both arranged on the sample object; the detection network comprises: an initialization layer, a decoding layer and a detection head; The detection network is used to perform target detection on the sample joint features to obtain a prediction result of the target detection, including: For each sample anchor point set within the preset perception range of the sample object, the distance, azimuth, preset Doppler velocity of the sample anchor point and the absolute velocity parameter corresponding to the sample anchor point are fused through the initialization layer to obtain a query vector corresponding to the sample anchor point; the absolute velocity parameter corresponding to each sample anchor point is determined based on the absolute velocity of the sample anchor point; By fusing each query vector with the sample joint feature based on the attention mechanism through the decoding layer, an updated query vector is obtained; The updated query vectors are input into the detection head to obtain the prediction result for each sample anchor point.

20. A target detection device, characterized in that: The device comprises: The detection data acquisition module is used to acquire the first detection image and RAD domain data of the detection scene collected at the current acquisition time, and the image features of the second detection image of the detection scene collected at the previous acquisition time; A feature extraction module to be detected, used for performing feature extraction on the first image to be detected to obtain the image features to be detected; and performing feature extraction on the RAD domain data to be detected to obtain the RAD features to be processed; A speed map output module to be processed is used to generate a speed map to be processed based on the image features to be detected and the image features of the second image to be detected; wherein the speed value at each pixel position in the speed map to be processed represents: the element at the same pixel position in the image features to be detected represents the radial speed of the object at the real position in the scene to be detected; The feature fusion module to be processed is used to fuse the elements at the same pixel position in the image feature to be detected and the speed map to be processed, and the polar coordinate value of the same pixel position in the radar coordinate system of the radar that collects the RAD domain data to be detected, so as to obtain the fused image feature to be processed; wherein the polar coordinate value of each pixel position in the radar coordinate system is determined based on the coordinate system conversion relationship between the image acquisition device that collects the first image to be detected and the second image to be detected and the radar, and the depth of the real position represented by the pixel position; A feature splicing module to be processed, used for splicing the fused image feature to be processed and the RAD feature to be processed to obtain a joint feature to be detected; The first target detection module is used to perform target detection based on the joint features to be detected to obtain a detection result.

21. A training device for a target detection model, characterized in that: The target detection model includes: an image feature extraction network, a RAD feature extraction network, a speed map output layer, a first encoding layer and a detection network; the device includes: A sample data acquisition module, used to acquire a first sample image and sample RAD domain data of a sample scene at a first acquisition moment, and image features of a second sample image of the sample scene acquired at an acquisition moment before the first acquisition moment; A sample feature extraction module, configured to extract features from the first sample image through the image feature extraction network to obtain sample image features; and to extract features from the sample RAD domain data through the RAD feature extraction network to obtain RAD features of the sample to be processed; A sample velocity map output module is used to input the sample image features and the image features of the second sample image into the velocity map output layer to obtain a sample velocity map; wherein the velocity value at each pixel position in the sample velocity map indicates that the elements at the same pixel position in the sample image features represent the radial velocity of the object at the real position in the sample scene; A sample feature fusion module, used to fuse the sample image feature and the elements at the same pixel position in the sample velocity map, and the polar coordinate value of the same pixel position in the radar coordinate system of the radar that collects the sample RAD domain data, using the first coding layer, to obtain a sample fused image feature; wherein the polar coordinate value of each pixel position in the radar coordinate system is determined based on the coordinate system conversion relationship between the image acquisition device that collects the first sample image and the second sample image and the radar, and the depth of the real position represented by the pixel position; A sample feature splicing module, used for splicing the sample fusion image feature and the sample RAD feature to obtain a sample joint feature; A second detection module, used to perform target detection on the sample joint features using the detection network to obtain a prediction result of the target detection; The model parameter adjustment module is used to calculate the loss function based on the true value of the prediction result and the pre-calibrated detection result, so as to adjust the model parameters of the target detection model until the preset convergence condition is reached to obtain a trained target detection model.

22. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-12, or any of the methods described in claims 13-19, when executing a program stored in a memory.

23. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements any method described in claims 1-12, or any method described in claims 13-19.

24. A computer program product comprising instructions, characterized in that When the computer program product runs on a computer, the computer implements the method described in any one of claims 1 to 12, or any one of claims 13 to 19.

Images