Target detection method and device
By generating radar stream BEV pseudo-image and camera stream BEV pseudo-image and performing feature fusion methods, the problem of poor fusion effect of 4D millimeter wave and front camera in the prior art is solved, and more accurate object detection is achieved.
Patent Information
- Application Number
- CN202411629830.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-14
- Publication Date
- 2025-02-28
AI Technical Summary
The existing forward-view object detection methods for 4D mmWave and front-camera fusion have the problems of lack of targeted optimization for 4D mmWave and inaccurate image depth estimation, which makes it difficult to achieve better front-fusing effects.
By acquiring the point cloud data of the 4D millimeter wave radar of the same frame and the camera image data, a radar stream BEV pseudo-image and camera stream BEV pseudo-image are generated, and feature fusion is performed to obtain the fusion feature and target detection is performed based on the fusion feature.
Improves the accuracy of camera flow depth prediction, improves the accuracy of feature fusion, and achieves more accurate target detection results.
Smart Images
Figure CN119540696A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of detection technology, and in particular, relates to a target detection method and a device thereof. Background Art
[0002] With the continuous development of autonomous driving technology, its application is becoming more and more extensive. Forward-looking target detection of autonomous driving vehicles is the basis of autonomous driving technology. The detection results can be used for decision-making planning, and its accuracy affects the output results of autonomous driving. Forward-looking target detection based on four-dimensional (4D) millimeter waves has attracted widespread attention due to the advantages of 4D millimeter-wave radar such as strong anti-interference ability and long detection distance. The forward-looking target detection method based on 4D millimeter wave and camera forward fusion overcomes the shortcomings of sparse 4D millimeter wave point cloud and lack of target contour information to a certain extent. However, most of the current forward fusion solutions imitate laser radar and image forward fusion solutions. On the one hand, they lack targeted optimization for 4D millimeter waves, and on the other hand, they do not consider the shortcomings of inaccurate image depth estimation, and it is often difficult to achieve a good forward fusion effect.
[0003] It can be seen that it is necessary to provide a high-precision 4D millimeter wave and image front fusion target detection method. Summary of the invention
[0004] The embodiments of the present application provide a target detection method and a device thereof to ensure the accuracy of target detection.
[0005] In a first aspect, an embodiment of the present application provides a target detection method, comprising:
[0006] Get the point cloud data of the 4D millimeter wave radar and the image data of the camera in the same frame.
[0007] Generate radar flow BEV pseudo image based on point cloud data,
[0008] Generate a camera stream BEV pseudo image based on the radar stream BEV pseudo image and image data,
[0009] The radar stream BEV pseudo image and the camera stream BEV pseudo image are feature fused to obtain the fused feature.
[0010] Target detection is performed based on the fused features to obtain the target detection result.
[0011] In a second aspect, an embodiment of the present application provides a target detection device, including:
[0012] The acquisition module is used to obtain the point cloud data of the 4D millimeter wave radar and the image data of the camera in the same frame.
[0013] The first generation module is used to generate a radar flow BEV pseudo image based on the point cloud data.
[0014] The second generation module is used to generate a camera stream BEV pseudo image according to the radar stream BEV pseudo image and the image data,
[0015] The fusion module is used to fuse the features of the radar stream BEV pseudo image and the camera stream BEV pseudo image to obtain the fusion feature.
[0016] The detection module is used to perform target detection based on the fusion features and obtain the target detection result.
[0017] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising:
[0018] A processor and a memory storing programs or instructions;
[0019] The above method is implemented when the processor executes the program or instruction.
[0020] In a fourth aspect, an embodiment of the present application provides a machine-readable storage medium, on which a program or instruction is stored, and the program or instruction implements the above method when executed by a processor.
[0021] In a fifth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the above method.
[0022] The target detection method and device of the embodiment of the present application can obtain the point cloud data of the 4D millimeter-wave radar and the image data of the camera in the same frame, generate a radar stream BEV pseudo image according to the point cloud data, generate a camera stream BEV pseudo image according to the radar stream BEV pseudo image and the image data, fuse the features of the radar stream BEV pseudo image and the camera stream BEV pseudo image to obtain fused features, perform target detection based on the fused features, and obtain target detection results.
[0023] In this way, a radar stream BEV pseudo image can be generated based on the point cloud data, and the radar stream BEV pseudo image of the same frame can be used to assist in the generation of the camera stream BEV pseudo image, thereby improving the depth prediction accuracy of the camera stream, thereby improving the accuracy of feature fusion and achieving more accurate target detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0025] Figure 1is a flowchart of a target detection method provided by an embodiment of the present application;
[0026] Figure 2 is a schematic diagram of feature expansion in the target detection method provided in an embodiment of the present application;
[0027] Figure 3a It is a schematic diagram of an attention mechanism feature extraction in the target detection method provided in an embodiment of the present application;
[0028] Figure 3b It is a schematic diagram of another attention mechanism feature extraction in the target detection method provided in an embodiment of the present application;
[0029] Figure 3c It is a schematic diagram of another attention mechanism feature extraction in the target detection method provided in the embodiment of the present application;
[0030] Figure 4 It is a schematic diagram of camera stream BEV pseudo image generation in the target detection method provided in an embodiment of the present application;
[0031] Figure 5 is a schematic diagram of feature fusion in the target detection method provided in an embodiment of the present application;
[0032] Figure 6 is a schematic structural diagram of a target detection device provided by another embodiment of the present application;
[0033] Figure 7 It is a structural schematic diagram of an electronic device provided in yet another embodiment of the present application. DETAILED DESCRIPTION
[0034] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.
[0035] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0036] In order to solve the problems in the prior art, the present application provides a target detection method and a device thereof. The target detection method provided by the present application is first introduced below.
[0037] Figure 1 FIG. 1 is a flow chart of a target detection method provided by an embodiment of the present application. Figure 1 As shown, the target detection method may include the following steps:
[0038] Step 101, obtaining point cloud data of the 4D millimeter wave radar and image data of the camera in the same frame.
[0039] In step 101, data from the 4D millimeter wave radar and the camera may be collected. Considering the difference in collection frequency between the two, the point cloud data of the 4D millimeter wave radar and the image data of the camera that are closest in time may be obtained as the same frame data according to the timestamp.
[0040] Step 102: Generate a radar stream Bird's Eye View (BEV) pseudo image based on the point cloud data.
[0041] In step 102, feature extraction may be performed on the point cloud data, and then the data may be converted into a BEV coordinate system according to its position information to generate a radar flow BEV pseudo image.
[0042] For example, the point cloud data can be preprocessed and Pillarized to obtain the point cloud information of non-empty Pillars, extract the radar cross-section features and velocity features of the point cloud, and use the attention mechanism to fully extract its features to obtain the radar flow BEV pseudo-image. It can be understood that Pillarization refers to dividing the point cloud data into a grid of fixed size on the XY plane, and each grid unit extends in the Z-axis direction to form a pillar (Pillar). Due to the uneven distribution of point clouds in space, the number of point clouds in different Pillars may be different, and there may be a part of Pillars that may not have point clouds, which can be called empty Pillars, while non-empty Pillars can be Pillars containing point clouds.
[0043] Step 103 : Generate a camera stream BEV pseudo image according to the radar stream BEV pseudo image and the image data.
[0044] In step 103, the radar stream BEV pseudo image auxiliary image data may be used to perform view conversion, and after extracting features, the data may be converted to a BEV coordinate system to generate a camera stream BEV pseudo image.
[0045] For example, the depth of the pixel can be predicted based on the image data, and then the view is transformed through the intrinsic and extrinsic parameters of the camera to obtain the camera stream features in the voxelized world coordinate system. The radar stream BEV pseudo image is used to assist the camera stream features in feature extraction, and the extracted features are converted to the BEV coordinate system to generate the camera stream BEV pseudo image.
[0046] Step 104 , feature fusion is performed on the radar stream BEV pseudo image and the camera stream BEV pseudo image to obtain fused features.
[0047] In step 104, a feature fusion algorithm may be used to fuse the BEV pseudo image of the radar stream and the BEV pseudo image of the camera stream to obtain fused features. The feature fusion algorithm may be an addition, concatenation, attention mechanism, or a Transformer model of an encoder-decoder architecture based on a self-attention mechanism, etc., which is not specifically limited here.
[0048] Step 105: Perform target detection based on the fused features to obtain a target detection result.
[0049] In step 105, the fused features can be input into the detection head for target detection to obtain target detection results, wherein the detection head can include target category, three-dimensional position in world coordinates, size, direction angle and speed, etc., so as to obtain prediction results such as target category, position, size, direction, speed, etc.
[0050] In an embodiment of the present application, the target detection method can obtain the point cloud data of the 4D millimeter-wave radar and the image data of the camera in the same frame, generate a radar stream BEV pseudo image according to the point cloud data, generate a camera stream BEV pseudo image according to the radar stream BEV pseudo image and the image data, perform feature fusion on the radar stream BEV pseudo image and the camera stream BEV pseudo image to obtain fused features, perform target detection based on the fused features, and obtain a target detection result.
[0051] In this way, a radar stream BEV pseudo image can be generated based on the point cloud data, and the radar stream BEV pseudo image of the same frame can be used to assist in the generation of the camera stream BEV pseudo image, thereby improving the depth prediction accuracy of the camera stream, thereby improving the accuracy of feature fusion and achieving more accurate target detection.
[0052] In some embodiments, the above step 102 may include the following steps:
[0053] The point cloud data is processed by Pillar to obtain the point cloud information of non-empty Pillar, which includes the radar cross-section characteristics and velocity characteristics of the point cloud.
[0054] Perform feature expansion on the radar cross section feature and the velocity feature to obtain the first feature;
[0055] Extract the first feature to obtain the second feature.
[0056] Use the attention mechanism to extract the second feature and obtain the third feature. The attention mechanism includes feature level attention and channel level attention.
[0057] Based on the third feature, a radar flow BEV pseudo image is generated.
[0058] In this embodiment, the point cloud data can be divided into grids according to the x-coordinate and y-coordinate of the world coordinate system where the point cloud data is located. It can be considered that all point clouds falling into the grid are in the same pillar, that is, these point clouds constitute this Pillar.
[0059] Each point cloud in Pillar can be represented by a D = 11-dimensional vector, which are (x, y, z, r, v obj ,x p ,y p ,z p ,x f ,y f ,z f ).
[0060] Among them, x, y, z are the real three-dimensional coordinate information of each point cloud. r is the radar cross section of the point cloud. obj is the absolute motion speed of the point cloud, v x ,v y is the relative velocity of the point cloud in the x and y directions. p ,y p ,z p are the offsets of each point cloud relative to the Pillar centroid coordinates in the coordinate system. The Pillar centroid can be defined as Where N is the number of non-zero points in the Pillar, (x p ,y p ,z p )=(xx pc ,yy pc ,zz pc ). f ,y f ,z f The offset of each point cloud relative to the Pillar center coordinate in this coordinate system.
[0061] In order to ensure data consistency and facilitate subsequent calculations, a hyperparameter N can be set to limit the number of point clouds in each Pillar. It is understandable that if the point cloud data in the Pillar exceeds N, it will be randomly sampled to N. If the point cloud data in the Pillar is less than N, the missing part will be filled by zero. The value of N can be set according to actual needs, for example, N can be set to 16.
[0062] The point cloud data may include P non-empty Pillars, and the point cloud information of the non-empty Pillars after the point cloud data is Pillarized can be represented by a (D, P, N)-dimensional tensor.
[0063] As mentioned above, the point cloud information can include the radar cross-section feature r and velocity feature v of the point cloud. obj Based on this, the radar cross section feature and the velocity feature can be expanded to obtain the expanded first feature. The first feature can include at least one of the maximum value of the radar cross section, the average value of the velocity, and the degree of point cloud discreteness, which is not specifically limited here.
[0064] In some embodiments, the first feature may include a maximum value of the radar cross section, an average value of the velocity, and a degree of point cloud dispersion. Figure 2 As shown, r_max is the maximum value of the radar cross section r of each point cloud in the Pillar, and vr_average is the absolute motion speed v of each point cloud in the Pillar. obj The average value of kde is the degree of dispersion of each point cloud in Pillar relative to other point clouds.
[0065] In some embodiments, the expression of the discrete degree of the point cloud may be:
[0066]
[0067] Among them, kde is the degree of point cloud discretization, R is half of the Pillar size, dx, dy, and dz are the sizes of the Pillar in the x, y, and z dimensions respectively. The value range of dx, dy, and dz can be set according to actual conditions and is not specifically limited here.
[0068] The expanded first feature can be input into the feature extraction layer to extract features, and the point cloud features can be learned by the feature extraction layer, so that the first feature (D+3, P, N) can be converted into the second feature (C, P, N).
[0069] The second feature can be further learned using the attention mechanism of feature level attention and channel level attention, and (C R ,P,N) dimension's third feature F out It is understandable that the second feature can be generated by feature level attention and channel level attention respectively to generate corresponding features, and then the two features can be merged with the second feature to obtain the third feature. Alternatively, the corresponding features can be generated by one of them first and then merged with the second feature, and then the merged features can be generated by the other one and then merged with the second feature again to obtain the third feature.
[0070] The third feature F out Take the maximum value and convert it to the BEV coordinate system based on the Pillar position code to generate The radar flow BEV pseudo image is shown in Figure 1, where H and W are the number of grids on the x-axis and y-axis in the BEV coordinate system, respectively.
[0071] In this way, feature expansion can be performed based on the radar scattering cross section and velocity characteristics specific to the 4D millimeter-wave radar. Dimensionality enhancement and further feature extraction can improve the target detection accuracy of the 4D millimeter-wave radar, that is, improve the accuracy of the radar stream BEV pseudo-image, thereby further improving the target detection accuracy of the subsequent fusion mode.
[0072] In some embodiments, the above-mentioned use of the attention mechanism to extract the second feature to obtain the third feature may include the following steps:
[0073] The second feature is extracted through channel-level attention to obtain the first sub-feature.
[0074] The second feature is extracted by feature level attention to obtain the second sub-feature.
[0075] The first sub-feature, the second sub-feature and the second feature are combined to obtain the third feature.
[0076] In this embodiment, if Figure 3a As shown, the second feature 301 of (C, P, N) dimension can be subjected to feature extraction 304 through channel level attention 302 to generate the first sub-feature of (C, P, 1) dimension, and the second feature 301 of (C, P, N) dimension can be subjected to feature extraction 304 through feature level attention 303 to generate the second sub-feature of (1, P, N) dimension, and then the first sub-feature, the second sub-feature and the second feature 301 can be merged, and features can be extracted by point-by-point multiplication to generate (C R ,P,N) dimension's third feature F out 305.
[0077] In some embodiments, the above-mentioned use of the attention mechanism to extract the second feature to obtain the third feature may include the following steps:
[0078] The second feature is extracted through channel-level attention to obtain the first sub-feature.
[0079] The first sub-feature is combined with the second feature to obtain the third sub-feature.
[0080] The third sub-feature is extracted by feature level attention to obtain the fourth sub-feature.
[0081] The fourth sub-feature is combined with the second feature to obtain the third feature.
[0082] In this embodiment, if Figure 3b As shown, the second feature 301 of the (C, P, N) dimension can be first subjected to feature extraction 304 by channel level attention 302 to generate a first sub-feature of the (C, P, 1) dimension, and the first sub-feature is merged with the second feature 301 to generate a third sub-feature of the (C, P, N) dimension, and then subjected to feature level attention 303 to feature extraction 304 to obtain a fourth sub-feature, and then the fourth sub-feature is merged with the second feature 301 to generate (C R ,P,N) dimension's third feature F out 305.
[0083] In some embodiments, the above-mentioned use of the attention mechanism to extract the second feature to obtain the third feature may include the following steps:
[0084] The second feature is extracted by feature level attention to obtain the second sub-feature.
[0085] The fifth sub-feature is obtained by combining the second sub-feature with the second feature.
[0086] The fifth sub-feature is extracted through channel-level attention to obtain the sixth sub-feature.
[0087] The sixth sub-feature is combined with the second feature to obtain the third feature.
[0088] In this embodiment, if Figure 3c As shown, the second feature 301 of the (C, P, N) dimension can be first subjected to feature level attention 303 to perform feature extraction 304 to generate a second sub-feature of the (1, P, N) dimension, and the second sub-feature is merged with the second feature 301 to generate a fifth sub-feature of the (C, P, N) dimension, and then subjected to channel level attention 302 to perform feature extraction 304 to obtain a sixth sub-feature, and then the sixth sub-feature is merged with the second feature 301 to generate (C R ,P,N) dimension's third feature F out 305.
[0089] In some embodiments, the above step 103 may include the following steps:
[0090] Perform view conversion on image data.
[0091] According to the radar stream BEV pseudo image, feature extraction is performed on the image data after view conversion processing to obtain the fourth feature,
[0092] Based on the fourth feature, a camera stream BEV pseudo image is generated.
[0093] In this embodiment, if Figure 4 As shown, the semantic features of the camera can be extracted, and then a view cone 401 is created to perform depth prediction on the pixels in the image data. The view conversion 402 is performed using the camera's internal and external parameters. The converted camera stream features can fall into a voxelized world coordinate system with a dimension of (C c ,N v ,H,W,D v ). Among them, C c is the number of features in the camera stream, N v is the maximum number of pixels falling on each voxel, H and W are the number of grids on the x-axis and y-axis in the world coordinate system, respectively, and are consistent with the BEV coordinate system, and D v The number of grids in the z direction in the world coordinate system.
[0094] Based on (C c ,N v ,H,W,D v )-dimensional camera stream features can upgrade the radar stream BEV pseudo image 403 from the (C, H, W) dimension to the (C, H, W, D v ) dimension to assist the camera stream in feature extraction 404 and generate (C c ,H,W,D v) dimension. c ,H,W,D v ) dimension by flattening the fourth characteristic (C I ,H,W) dimensions of camera stream BEV pseudo image 405.
[0095] In this way, using the radar stream BEV pseudo image of the 4D millimeter-wave radar to assist the camera stream BEV pseudo image generation helps the camera stream depth prediction and improves the camera's target detection accuracy, thereby further improving the target detection accuracy of the fusion modality.
[0096] In some embodiments, the above step 104 may include the following steps:
[0097] Using the attention mechanism, the features of the radar stream BEV pseudo image and the camera stream BEV pseudo image are fused to obtain the fused features.
[0098] In this embodiment, if Figure 5 As shown, the attention mechanism can be used to fuse the features of the radar stream BEV pseudo image and the camera stream BEV pseudo image.
[0099] For example, the radar stream BEV pseudo image 501 as Q, camera stream BEV pseudo image 502 is input as K and V to the attention model and executed in the attention model Where σ represents the sigmoid operation, generating The intersection feature F M 503 as V, the radar stream BEV pseudo image F R 501 is input as K and Q to the attention model and executed in the attention model Generate fusion features
[0100]
[0101] Based on the target detection method provided in the above embodiment, the present application also provides an embodiment of a target detection device.
[0102] Figure 6 A schematic structural diagram of a target detection device provided in another embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.
[0103] Reference Figure 6 , the target detection device 600 includes:
[0104] The acquisition module 601 is used to acquire the point cloud data of the 4D millimeter wave radar and the image data of the camera in the same frame.
[0105] The first generating module 602 is used to generate a radar flow BEV pseudo image according to the point cloud data.
[0106] The second generating module 603 is used to generate a camera stream BEV pseudo image according to the radar stream BEV pseudo image and the image data,
[0107] The fusion module 604 is used to fuse the radar stream BEV pseudo image and the camera stream BEV pseudo image to obtain a fusion feature.
[0108] The detection module 605 is used to perform target detection based on the fused features to obtain a target detection result.
[0109] In some embodiments, the first generation module 602 may include:
[0110] The processing unit is used to perform Pillar processing on the point cloud data to obtain point cloud information with non-empty Pillars. The point cloud information includes radar cross-section characteristics and velocity characteristics of the point cloud.
[0111] An expansion unit, used for expanding the radar cross section feature and the velocity feature to obtain a first feature;
[0112] The first extraction unit is used to extract the first feature to obtain the second feature.
[0113] The second extraction unit is used to extract the second feature by using an attention mechanism to obtain a third feature, where the attention mechanism includes feature level attention and channel level attention.
[0114] A generating unit is used to generate a radar flow BEV pseudo image based on the third feature.
[0115] In some embodiments, the first feature includes a maximum value of a radar cross section, an average value of a velocity, and a degree of dispersion of a point cloud.
[0116] In some embodiments, the expression of the discrete degree of the point cloud may be:
[0117]
[0118] Among them, kde is the degree of discretization of the point cloud, R is half of the Pillar size, and dx, dy, and dz are the sizes of the Pillar in the x, y, and z dimensions, respectively.
[0119] In some embodiments, the second extraction unit may also be used to:
[0120] The second feature is extracted through channel-level attention to obtain the first sub-feature.
[0121] The second feature is extracted by feature level attention to obtain the second sub-feature.
[0122] The first sub-feature, the second sub-feature and the second feature are combined to obtain the third feature.
[0123] In some embodiments, the second extraction unit may also be used to:
[0124] The second feature is extracted through channel-level attention to obtain the first sub-feature.
[0125] The first sub-feature is combined with the second feature to obtain the third sub-feature.
[0126] The third sub-feature is extracted by feature level attention to obtain the fourth sub-feature.
[0127] The fourth sub-feature is combined with the second feature to obtain the third feature.
[0128] In some embodiments, the second extraction unit may also be used to:
[0129] The second feature is extracted by feature level attention to obtain the second sub-feature.
[0130] The fifth sub-feature is obtained by combining the second sub-feature with the second feature.
[0131] The fifth sub-feature is extracted through channel-level attention to obtain the sixth sub-feature.
[0132] The sixth sub-feature is combined with the second feature to obtain the third feature.
[0133] In some embodiments, the second generating module 603 may also be used to:
[0134] Perform view conversion on image data.
[0135] According to the radar stream BEV pseudo image, feature extraction is performed on the image data after view conversion processing to obtain the fourth feature,
[0136] Based on the fourth feature, a camera stream BEV pseudo image is generated.
[0137] In some embodiments, the fusion module 604 may also be used to:
[0138] Using the attention mechanism, the features of the radar stream BEV pseudo image and the camera stream BEV pseudo image are fused to obtain the fused features.
[0139] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application, and are devices corresponding to the above-mentioned target detection method. All implementation methods in the above-mentioned method embodiment are applicable to the embodiments of the device. Its specific functions and the technical effects brought about can be found in the method embodiment part, which will not be repeated here.
[0140] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0141] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in yet another embodiment of the present application is shown.
[0142] The device may include a processor 701 and a memory 702 storing programs or instructions.
[0143] When the processor 701 executes the program, the steps in any of the above method embodiments are implemented.
[0144] Exemplarily, the program may be divided into one or more modules / units, one or more modules / units are stored in the memory 702 and executed by the processor 701 to complete the present application. One or more modules / units may be a series of program instruction segments capable of completing a specific function, and the instruction segments are used to describe the execution process of the program in the device.
[0145] Specifically, the processor 701 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0146] The memory 702 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 702 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 702 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 702 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 702 is a non-volatile solid-state memory.
[0147] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, typically, the memory includes one or more tangible (non-transitory) machine-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.
[0148] The processor 701 implements any one of the methods in the above embodiments by reading and executing the program or instruction stored in the memory 702 .
[0149] In one example, the electronic device may further include a communication interface 703 and a bus 704. The processor 701, the memory 702, and the communication interface 703 are connected via the bus 704 and communicate with each other.
[0150] The communication interface 703 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.
[0151] Bus 704 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 704 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.
[0152] In addition, in combination with the method in the above embodiment, the embodiment of the present application may provide a machine-readable storage medium for implementation. The machine-readable storage medium stores a program or instruction; when the program or instruction is executed by a processor, any one of the methods in the above embodiment is implemented. The machine-readable storage medium can be read by a machine such as a computer.
[0153] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0154] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0155] An embodiment of the present application provides a computer program product, which is stored in a machine-readable storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0156] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.
[0157] The functional modules shown in the above block diagram can be implemented as hardware, software, firmware or their combination. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), suitable firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable media" can include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. Code segments can be downloaded via computer grids such as the Internet, intranets, etc.
[0158] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.
[0159] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by a computer program or instruction. These programs or instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.
[0160] The above are only specific implementation methods of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.
Claims
1. A target detection method, characterized in that: The method comprises: Get the point cloud data of the 4D millimeter wave radar and the image data of the camera in the same frame. Generate a radar flow BEV pseudo image based on the point cloud data, generating a camera stream BEV pseudo image according to the radar stream BEV pseudo image and the image data, The radar stream BEV pseudo image and the camera stream BEV pseudo image are subjected to feature fusion to obtain fusion features. Target detection is performed based on the fusion features to obtain a target detection result.
2. The method according to claim 1, characterized in that The step of generating a radar flow BEV pseudo image according to the point cloud data comprises: Pillaring the point cloud data to obtain non-empty Pillar point cloud information, wherein the point cloud information includes radar cross-section characteristics and velocity characteristics of the point cloud. Performing feature expansion on the radar cross section feature and the velocity feature to obtain a first feature, Extract the first feature to obtain the second feature. Use an attention mechanism to extract the second feature to obtain a third feature, wherein the attention mechanism includes feature level attention and channel level attention. Based on the third feature, a radar flow BEV pseudo image is generated.
3. The method according to claim 2, characterized in that The first features include the maximum value of the radar cross section, the average value of the velocity and the degree of point cloud dispersion.
4. The method according to claim 3, characterized in that The expression of the discrete degree of the point cloud is: Among them, kde is the degree of discretization of the point cloud, R is half of the Pillar size, and dx, dy, and dz are the sizes of the Pillar in the x, y, and z dimensions, respectively.
5. The method according to claim 2, characterized in that: The using the attention mechanism to extract the second feature to obtain the third feature includes: The second feature is extracted by channel level attention to obtain the first sub-feature. The second feature is subjected to feature level attention to extract features, and a second sub-feature is obtained. The first sub-feature, the second sub-feature and the second feature are combined to obtain a third feature.
6. The method according to claim 2, characterized in that The using the attention mechanism to extract the second feature to obtain the third feature includes: The second feature is extracted by channel level attention to obtain the first sub-feature. The first sub-feature is combined with the second feature to obtain a third sub-feature, The third sub-feature is subjected to feature level attention to extract features, thereby obtaining a fourth sub-feature. The fourth sub-feature is combined with the second feature to obtain a third feature.
7. The method according to claim 2, characterized in that The using the attention mechanism to extract the second feature to obtain the third feature includes: The second feature is subjected to feature level attention to extract features, and a second sub-feature is obtained. The second sub-feature is combined with the second feature to obtain a fifth sub-feature, The fifth sub-feature is subjected to feature extraction through channel-level attention to obtain the sixth sub-feature. The sixth sub-feature is combined with the second feature to obtain a third feature.
8. The method according to claim 1, characterized in that The step of generating a camera stream BEV pseudo image according to the radar stream BEV pseudo image and the image data comprises: performing view conversion processing on the image data, According to the radar stream BEV pseudo image, feature extraction is performed on the image data after view conversion processing to obtain a fourth feature, Based on the fourth feature, a camera stream BEV pseudo image is generated.
9. The method according to claim 1, characterized in that: The step of fusing the radar stream BEV pseudo image and the camera stream BEV pseudo image to obtain a fused feature includes: The radar stream BEV pseudo image and the camera stream BEV pseudo image are feature fused using an attention mechanism to obtain a fused feature.
10. A target detection device, characterized in that: The device comprises: The acquisition module is used to obtain the point cloud data of the 4D millimeter wave radar and the image data of the camera in the same frame. A first generating module is used to generate a radar flow BEV pseudo image according to the point cloud data, A second generating module is used to generate a camera stream BEV pseudo image according to the radar stream BEV pseudo image and the image data, A fusion module is used to fuse the radar stream BEV pseudo image and the camera stream BEV pseudo image to obtain a fusion feature. The detection module is used to perform target detection based on the fusion features to obtain a target detection result.