Radar and camera sensor combined detection method

Abstract

A radar and camera sensor combined detection method, comprising: obtaining a plurality of image perception data groups and a plurality of raw radar perception data groups of a target object, the target object comprising a moving target object and a stationary target object (S11); according to a first data processing method, processing the plurality of raw radar perception data groups to obtain a plurality of radar sparse tensor information groups (S12); and forming a stereoscopic multi-dimensional depth perception data combination and outputting same, the stereoscopic multi-dimensional depth perception data combination being formed in a multi-dimensional matrix structure, and the stereoscopic multi-dimensional depth perception data combination comprising a plurality of radar sparse tensor information groups assigned to the multi-dimensional matrix structure and a corresponding plurality of image perception data groups (S13). The method solves the problems of how to effectively identify static objects, occluded objects, etc. and process a large amount of radar raw data.

Description

Combined detection method of radar and camera sensors Technical Field

[0001] The present invention relates to the field of sensor detection, and in particular to a method for combined detection of radar and camera sensors. Background Art

[0002] In the field of target recognition and environmental perception, radar (including: millimeter-wave radar (sensor); terahertz sensors and lidar can also be used to generate multi-dimensional target perception information output, and are also included in the definition of "radar" in our invention) and cameras are both widely used sensors. The two types of sensors have their own characteristics and dimensions of environmental perception, so they are often combined together to form a sensor combination to jointly complete the tasks of environmental and target perception.

[0003] Radar can provide information such as target detection distance, relative movement speed (Doppler information), material (radar RCS data information), and the angle of the target's space. Information such as target detection distance and relative movement speed are the weaknesses of camera system detection. Therefore, the fusion system of radar and camera is a very good combination of target and environment perception. The system can achieve a "1+1>2" fusion processing output effect.

[0004] However, a current disadvantage of radar (particularly millimeter-wave radar) is that the spatial angular positioning accuracy and resolution of targets produced by processed sensor data are significantly lower than those of cameras (especially high-resolution cameras). This is an area where radar systems need to be improved. Some radar sampling algorithms rely heavily on Doppler information to detect objects, making it impossible to detect static objects. The industry is also beginning to develop 4D millimeter-wave imaging radars. 4D millimeter-wave radars are not only an improved version of millimeter-wave radars, but also introduce many important research topics.

[0005] 4D millimeter-wave radar generates far more raw data than traditional radar, posing challenges to signal processing and data generation. Effectively distinguishing and identifying clutter and noise signals is even more challenging. The sparseness and noise of 4D millimeter-wave radar point clouds generated by existing radar signal processing are even greater than those of lidar point clouds. Therefore, the perception, positioning, and mapping algorithms need to be redefined and redesigned to enhance and re-interpret the perception characteristics of 4D millimeter-wave radar.

[0006] A major reason for the sparse point clouds of current 4D radar products is the significant information loss caused by signal filters (for example, the popular CFAR (Constant False Alarm Rate) detection algorithm). To address this issue, a growing number of alternative algorithms, such as those based on machine learning, are being proposed to replace CFAR in data processing. After performing the FFT on the two dimensions of range and velocity, the radar system bypasses CFAR signal filtering and directly determines the target's spatial angular position (3D stereoscopic dimensions: including the horizontal azimuth and elevation angles of the target's spatial position, obtained through data analysis of the received reflected signals, such as through Fast Fourier Transform (FFT) analysis of their phase dimension). At each sampling point in three-dimensional space, the radar obtains inferred power measurements encompassing the distance, Doppler, azimuth, and elevation dimensions of that sampling point. This information is also referred to in the industry as 4D radar tensors (4DRT). Furthermore, the radar system can adjust the order in which it processes the 4D tensor signals—the distance, Doppler (velocity), azimuth, and elevation dimensions of the sampling point. Some algorithms process the distance dimension first, then perform azimuth and elevation positioning, and then extract Doppler (velocity) information. Various algorithms exist. Based on this 4D tensor, the radar system then uses further algorithms (which may include machine learning and neural networks) to extract and identify target and environment features. This method can retain effective perception information to the greatest extent possible. However, since there is no data filtering and extraction (compared to the steps of processing with the CFAR algorithm), the amount of data output by the system and that needs to be processed by subsequent modules is very large. This places high demands on the system's transmission bandwidth, memory cache, and the computing power of the subsequent system for further target and environment recognition processing. The system cost will also be very high.

[0007] The most challenging aspect lies in the radar algorithm itself. Among the massive amount of 4D tensor sampling information acquired by the radar system, how can one distinguish which information is the effective reflection information of the target from the radar signal (this is what the system hopes to obtain), which information is background noise (noise signal, which needs to be filtered out), which information is signal crosstalk caused by other radar systems (which needs to be identified and filtered out), and which information is multipath crosstalk of the radar signal... How to effectively distinguish them and maximize the identification of effective target reflection information is the direction in which the industry is currently looking forward to technological breakthroughs.

[0008] Therefore, how to effectively identify static objects, obstructing objects, etc., and process large amounts of radar raw data has become a technical focus that technical personnel in this field urgently need to solve.

[0009] Summary of the Invention

[0010] The present invention provides a method for combined detection of radar and camera sensors to solve the problem of how to effectively identify static objects, obstructing objects, etc., and process a large amount of radar raw data.

[0011] According to a first aspect of the present invention, a method for combined detection of radar and camera sensors is provided, comprising:

[0012] Acquire multiple image perception data groups and multiple raw radar perception data groups of target objects; wherein the target objects include moving target objects and stationary target objects; each group of the raw radar perception data groups includes a combination of power measurement information data corresponding to radar 4D tensors in four dimensions: range, Doppler, horizontal azimuth, and pitch angle, sensed by the radar sensor at each sampling point in the three-dimensional space within the radar sensor's detection domain; wherein the power measurement information data includes: data on signal reflection energy, or data on radar cross section or signal-to-noise ratio information corresponding to the signal reflection energy; the multiple image perception data groups represent data acquired by the image sensor;

[0013] Processing the plurality of raw radar perception data sets according to a first data processing method to obtain a plurality of radar sparse tensor information sets; wherein the first data processing method is used to remove random noise data from the raw radar perception data sets, as well as random noise generated by target object detection and radar signal crosstalk information data; the random noise data represents random discrete data obtained from detection points within a detection space;

[0014] forming a stereoscopic multi-dimensional depth perception data combination and outputting the stereoscopic multi-dimensional depth perception data combination; the stereoscopic multi-dimensional depth perception data combination is formed in a multi-dimensional matrix structure; the stereoscopic multi-dimensional depth perception data combination includes a plurality of the radar sparse tensor information groups assigned to the multi-dimensional matrix structure and a corresponding plurality of the image perception data groups;

[0015] The multidimensional matrix structure includes a plurality of matrix layers arranged in sequence along a first direction, and the matrix layers include at least one image pixel matrix layer and at least one radar matrix layer; the radar matrix layer includes a plurality of array-arranged radar matrix elements; the radar matrix elements represent elements that correspond one-to-one to the array-arranged pixel elements in the image pixel matrix layer along the first direction; each of the pixel elements corresponds to the image perception data group; the radar matrix elements in the radar matrix layer are all assigned corresponding radar sparse tensor information groups.

[0016] Optionally, forming a stereoscopic multi-dimensional depth perception data combination specifically includes:

[0017] forming the radar matrix layer and the image pixel matrix layer, and determining a first mapping relationship; wherein the first mapping relationship represents a correspondence between the radar matrix elements and the arrayed pixel elements in the image pixel matrix layer;

[0018] According to the first mapping relationship, the radar matrix elements are mapped one-to-one with the pixel elements arranged in the array in the image pixel matrix layer along the first direction;

[0019] Assigning a plurality of radar sparse tensor information groups to corresponding radar matrix elements in the radar matrix layer;

[0020] In the image pixel matrix layer, each pixel element corresponds to the image perception data group; and the radar matrix layer includes a plurality of radar matrix elements arranged in an array.

[0021] Optionally, forming the radar matrix layer specifically includes:

[0022] Cutting a two-dimensional plane perpendicular to the central axis of the radar sensing field of view and projecting it into the detection field of the radar sensor, wherein the two-dimensional plane includes a plurality of mapping units divided according to a preset spatial angle resolution;

[0023] Projecting each detection point within the radar perception field of view represented by the radar sparse tensor information group onto the mapping unit on the two-dimensional plane according to its corresponding direction of arrival;

[0024] Each of the mapping units is assigned or marked with the radar sparse tensor information group in the corresponding direction of arrival;

[0025] Optionally, forming the first mapping relationship includes:

[0026] Determining a second coordinate based on the first coordinate; the second coordinate represents the coordinate of the first coordinate after coordinate transformation in an image coordinate system; the first coordinate represents the coordinates of the plurality of radar sparse tensor information groups in a coordinate system in a detection domain of the radar sensor; the image coordinate system represents the coordinate system in which an image acquisition region of the image sensor is located;

[0027] Aligning the detection field of the radar sensor with the image acquisition area of ​​the image sensor; wherein the central axis of the detection field of the radar sensor is aligned with the optical axis of the image acquisition sensor, and the detection vertex of the detection field of the radar sensor coincides with the center of the entrance pupil of the image acquisition sensor;

[0028] Projecting the detection area of ​​the radar sensor onto a two-dimensional plane where the imaging surface of the image acquisition sensor is located according to the optical axis of the image acquisition sensor and the center of the entrance pupil to obtain a projection area corresponding to the radar sensor;

[0029] The first mapping relationship is determined according to a positional relationship between the projection area and an imaging area of ​​the image acquisition sensor; the imaging area of ​​the image acquisition sensor is located on a two-dimensional plane where an imaging surface of the image acquisition sensor is located.

[0030] Optionally, in each of the image pixel matrix layers, the image pixel data group includes: any one of: a brightness data group, a grayscale data group, a red light data group, a green light data group, a blue light data group, a Y data group, a U data group, a V data group or an optical flow data group.

[0031] Optionally, the radar sensor is at least one of the following: microwave radar, ultrasonic radar, laser radar, infrared sensor, and terahertz imaging sensor.

[0032] Optionally, the radar sparse tensor information group includes a radar 4D sparse tensor information group or a radar 3D sparse tensor information group; wherein, the radar 4D sparse tensor information group represents the detection and combination of radar 4D tensor power measurement information of four spatial dimensions corresponding to Doppler, distance, horizontal azimuth and pitch angle of the power measurement information received by the radar corresponding to the specified spatial position; wherein, the radar 3D sparse tensor information group is the detection and combination of radar 3D tensor power measurement information of three three-dimensional spatial dimensions corresponding to Doppler, distance and azimuth of the power measurement information received by the radar corresponding to the specified spatial position.

[0033] According to the second aspect of the present invention, a method for combined detection of radar and camera sensors for machine learning is provided, wherein the stereoscopic multi-dimensional depth perception data combination obtained by the combined detection method of radar and camera sensors described in any one of the first aspects of the present invention is used for machine learning through the collection and sample labeling of system data samples.

[0034] According to a third aspect of the present invention, there is provided a device for combined detection of radar and camera sensors, comprising:

[0035] Data acquisition module; used to acquire several image perception data groups and several original radar perception data groups of the target object;

[0036] An image processing module; configured to process the plurality of raw radar perception data groups according to a first data processing method to obtain corresponding radar 4D sparse tensors or radar 3D tensor information groups; and form a stereoscopic multi-dimensional depth perception data combination;

[0037] In which, the stereoscopic multi-dimensional depth perception data combination is formed in a multi-dimensional matrix structure; the stereoscopic multi-dimensional depth perception data combination includes a number of the radar 4D tensor information assigned to the multi-dimensional matrix structure, or the radar 3D tensor information and a corresponding number of the image perception data groups.

[0038] According to a fourth aspect of the present invention, there is provided an electronic device, comprising a processor and a memory, wherein the memory is configured to store code;

[0039] The processor is used to execute the code in the memory to implement the method for combined detection of radar and camera sensors as described in any one of the first aspects of the present invention.

[0040] According to a fifth aspect of the present invention, there is provided a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for combined detection of radar and camera sensors as described in any one of the first aspects of the present invention.

[0041] The radar and camera sensor combined detection method provided by the present invention, by collecting a raw radar perception data set, can then detect a comprehensive range of moving and stationary targets through a first data processing method and a combination of three-dimensional multi-dimensional depth perception data. It also eliminates invalid information, reducing the computing power burden. Furthermore, the multipath reflection detection information of the target retained after eliminating the invalid information can also achieve functions that traditional automotive sensor detection systems cannot. This combined information contains rich perception information from the system's multi-dimensional perception, which can be used by the subsequent cognitive system (target determination system) to recognize and identify the target and environment. This system can maximize the advantages of radar target perception and can mine more information, increase data dimensions, and achieve more accurate target detection.

[0042] Furthermore, the technical solution provided by this invention can select either a radar 4D sparse tensor information group or a radar 3D sparse tensor information group, depending on the application scenario, to form a stereoscopic, multi-dimensional depth perception data combination and output. The radar 4D sparse tensor information group is a combination of Doppler, range, azimuth, and elevation angles, as well as power measurement information, processed based on radar signal processing principles for a specified spatial (3D) location. This reduction in one dimension simplifies data processing and output, enabling high-speed transmission, processing, and storage of limited system detection data, thus simplifying system processing.

[0043] Furthermore, a method for combined detection of radar and camera sensors for machine learning is also provided, and the stereoscopic multi-dimensional depth perception data combination obtained by the method for combined detection of radar and camera sensors described in any one of the aforementioned first embodiments of the present invention is used for machine learning through the collection and sample labeling of system data samples; therefore, it is very direct and efficient, and the product (the radar and camera fusion product itself) can be the data collector, and the collected data itself has complete visual scene interpretation information, so sample labeling can be done directly and conveniently, and there is no need to add an additional camera for sample collection and use an additional system to specifically associate video information with radar data information, which saves costs, and the comprehensive and complete radar sparse tensor information group increases the accuracy of machine learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] FIG1 is a schematic flow chart of a method for combined detection using a radar and camera sensor according to an embodiment of the present invention;

[0046] FIG2 is a schematic diagram of detection points within a radar sensing field of view formed by a radar sensing space provided by an embodiment of the present invention;

[0047] FIG3 is a schematic diagram of the corresponding relationship between the radar perception field of view and the target object;

[0048] FIG4 is a schematic diagram of a first application scenario of a method for combined detection using radar and camera sensors provided by an embodiment of the present invention;

[0049] FIG5 is a schematic diagram of a second application scenario of the method for combined detection using radar and camera sensors provided by one embodiment of the present invention;

[0050] FIG6 is a schematic diagram of a third application scenario of the method for combined detection using radar and camera sensors provided by an embodiment of the present invention;

[0051] FIG7 is a schematic structural diagram of a device for combined detection of radar and camera sensors provided by one embodiment of the present invention;

[0052] FIG8 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0054] The terms "first," "second," "third," "fourth," and the like (if any) in the description and claims of the present invention and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0055] Traditional automotive sensing systems have the following defects:

[0056] On the one hand, the detection range of traditional signal filters is relatively scattered, and the information of the target objects finally obtained is incomplete; for example, static objects, occluded objects, etc. will be eliminated by traditional filters and therefore cannot be detected.

[0057] On the other hand, the amount of raw radar data is huge, so an effective way to remove it is needed:

[0058] Specifically, the number of 4D radar tensor data blocks in range and azimuth is related to the sampling frequency of the radar signal; the number of horizontal azimuth and elevation angles is related to the number of antenna channels; and the amount of detection data from the 4D tensor of a high-resolution radar can be very large, posing great challenges to data transmission, storage, and processing in terms of both technology and system cost.

[0059] In addition, in machine learning: In our current machine learning sampling, data sampling and sample labeling for radar (including lidar) is very difficult if the radar data sampling samples are not directly labeled with visual information. Therefore, when sampling and labeling radar (including lidar) data, a camera is usually added to the sample collection system to synchronously record the scene. The radar data sampling samples are then labeled based on the scene interpretation of the camera-recorded video playback. This increases the cost of collecting radar system samples (requiring a sample collection system specifically for data collection and an additional camera for recording the scene) and sample labeling (labeling the radar data sampling samples based on the scene interpretation of the camera-recorded video playback).

[0060] In view of this, the inventors of the present application propose: first, to combine the perception information of the image perception dimension and the radar perception dimension into a hyper-dimensionally coupled data combination; the hyper-dimensionally coupled data combination includes: valid data emitted by the detected target, as well as a large amount of data reflected by non-detected targets; wherein coupling refers to the correlation information of the mutual dependence, mutual influence, and mutual restriction between the various data obtained by the radar and camera combined detection system for target and environment perception, which is "associated and stored in a unified data corresponding format"; the "hyper-dimension" of hyper-dimensional coupling refers to "image perception dimension plus radar perception dimension"; the "addition" here is not a simple mathematical addition of the number of image perception dimensions and the number of radar perception dimensions, but coupling here refers to the product combination of the physical perception dimensions between them;

[0061] Secondly, by processing the radar's 4D tensor data, filtering out invalid information, retaining the valid information contained therein and combining it with the image for output, the subsequent system can perform target and environment recognition very efficiently.

[0062] Thirdly, radar "imaging" is performed on the radar 4D sparse tensor to combine it into millimeter-wave "radar pixels": radar "imaging" is not just imaging from a sparse "radar point cloud", but rather generating a full set of "radar pixels", i.e., for each effectively resolvable signal arrival azimuth angle received by the radar, a "radar pixel" is generated at the radar receiving end. Each "radar pixel" has one or more radar 4D tensor data in its corresponding signal arrival direction; one or more radar target detection tensor information includes information on moving and stationary targets; the signal arrival direction divides the radar's perception domain into corresponding horizontal azimuth and elevation azimuth angles based on the radar's spatial angular resolution, and the number of radar horizontal azimuth and elevation azimuth angles are mapped into a two-dimensional matrix.

[0063] Then, the "radar pixels" are connected to the image pixels, and the data correspondence between them is mapped according to their aperture detection correlation within the common perception range, thus forming the "multi-dimensional" perception of a complete fusion perception system.

[0064] Specifically, the present invention processes radar 4D tensor data and filters out invalid information through the following technical solutions:

[0065] We combine the camera image with the 4D radar tensor. The 4D radar tensor refers to the power measurement information obtained by the radar at the sampling point in three-dimensional space, including the distance, Doppler, horizontal azimuth and pitch angle dimensions at that sampling point.

[0066] Combine the image data with the radar's 4D tensor information to generate multi-dimensional data, and then generate it through algorithm processing based on the image and radar 4D tensor information.

[0067] We can also directly process radar 4D tensor information and use machine algorithms to extract effective information through data and training.

[0068] The clustering algorithm can be used to extract effective information from the radar 4D tensor information and filter out random noise (random noise signals).

[0069] The methods for obtaining the signal's azimuth of arrival include: for radar, the time-domain spectrum of the received signal represents the signal's energy distribution at various frequencies; the spatial spectrum represents the signal's energy distribution in various spatial directions. Therefore, by obtaining the signal's spatial spectrum, the direction of arrival (DOA) of the signal is determined, and the spatial angle information of the signal's incidence (including the target's horizontal and vertical spatial angle positioning information) is calculated. Therefore, spatial spectrum estimation is also called DOA estimation.

[0070] Therefore, the technical solution provided by this application can not only detect a full range of moving and stationary targets, but also eliminate invalid information, reducing the computing power burden. Moreover, the multipath reflection detection information of the target retained after eliminating the invalid information can also realize functions that traditional automotive sensor detection systems cannot achieve. In this way, the combined information will contain rich perception information of the system's multi-dimensional perception to provide the subsequent cognitive system (target determination system) with cognitive identification of the target and the environment. The system can maximize the use and play of the advantages of radar target perception, can mine more information, have more data dimensions, and more accurate target detection.

[0071] In machine learning, comprehensive and complete radar data is more conducive to machine learning.

[0072] The following is an introduction to some of the professional titles mentioned in this invention:

[0073] The four dimensions of 4D radar are range, azimuth, altitude, and Doppler velocity; the set of 4D radar tensors (4DRT) consists of power measurements along the Doppler, range, azimuth, and elevation dimensions. The power measurement information can also be expressed as signal-to-noise ratio (SNR) or radar cross section (RCS).

[0074] The difference between a 4D radar tensor and a radar point cloud is that a radar point cloud contains information about target reflection points, while a 4D radar tensor is a combination of Doppler, range, azimuth, elevation, and power measurements, processed based on radar signal processing principles at a specified spatial (3D) location. This information may include target signal reflections, radar multipath reflections, radar signal crosstalk (interference signals from the current radar or other radars), and random noise. Radar point cloud information can be considered a subset of the 4D radar tensor.

[0075] The output of an FMCW radar is a radar tensor (RT), a dense tensor populated with non-zero power measurements along all axes. RT provides rich information about the environment; a 4D radar tensor (4DRT) consists of power measurements along the Doppler, range, azimuth, and elevation dimensions.

[0076] Based on clustering algorithms, datasets are divided into different classes or clusters based on one or more specific metrics (such as target distance or speed). This ensures that the metrics of data objects within the same class or cluster are as similar as possible, while the differences between data objects in different classes are as large as possible. In other words, after clustering, data of the same class is clustered together as much as possible, while data of different classes are separated as much as possible. In radar detection scenarios, based on the spatial continuity (area and volume in 3D physical space) of detected targets (such as pedestrians, vehicles, and traffic signs), the power measurement information of the same detected target in the range, Doppler, horizontal azimuth, and pitch angle dimensions detected by the radar is inevitably similar and correlated. We use this physical property to filter random noise and extract the reflection information of the detected target.

[0077] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0078] Referring to FIG. 1 to FIG. 8 , according to one embodiment of the present invention, a method for combined detection of radar and camera sensors is provided. The flowchart of the method is shown in FIG. 1 . The method specifically includes:

[0079] S11: Acquire several image perception data groups and several raw radar perception data groups of target objects; wherein the target objects include moving target objects and stationary target objects; each group of the raw radar perception data groups includes a combination of power measurement information data corresponding to radar 4D tensors in four dimensions of range, Doppler, horizontal azimuth, and pitch angle perceived by the radar sensor at each sampling point in the three-dimensional space where the radar sensor's detection domain is located, wherein the power measurement information data includes: data on signal reflection energy, or data on radar cross section or signal-to-noise ratio information corresponding to the signal reflection energy; the several image perception data groups represent data acquired by the image sensor;

[0080] S12: Processing the plurality of raw radar perception data sets according to a first data processing method to obtain a plurality of radar sparse tensor information sets; wherein the first data processing method is used to remove random noise data from the raw radar perception data sets, as well as random noise generated by target object detection and radar signal crosstalk information data; the random noise data represents random discrete data obtained from detection within a detection space;

[0081] In a specific embodiment, the random discrete data detected and obtained in the detection space refers to: random discrete data detected and obtained;

[0082] S13: Forming a stereoscopic multi-dimensional depth perception data combination and outputting the stereoscopic multi-dimensional depth perception data combination; the stereoscopic multi-dimensional depth perception data combination is formed in a multi-dimensional matrix structure; the stereoscopic multi-dimensional depth perception data combination includes a plurality of the radar sparse tensor information groups assigned to the multi-dimensional matrix structure and a corresponding plurality of the image perception data groups; wherein the stereoscopic multi-dimensional depth perception data combination represents that each frame of data includes at least one frame of single-frame pixel image data captured by the image sensor, and radar sparse tensor information groups of all receiving channels collected by the radar sensor within the same time period.

[0083] The multidimensional matrix structure includes a plurality of matrix layers arranged in sequence along a first direction, and the matrix layers include at least one image pixel matrix layer and at least one radar matrix layer; the radar matrix layer includes a plurality of array-arranged radar matrix elements; the radar matrix elements represent elements that correspond one-to-one to the array-arranged pixel elements in the image pixel matrix layer along the first direction; each of the pixel elements corresponds to the image perception data group; the radar matrix elements in the radar matrix layer are all assigned corresponding radar sparse tensor information groups.

[0084] The first data processing method includes: clustering algorithm analysis, machine learning and neural network processing, etc.

[0085] In one embodiment, a clustering algorithm is used to analyze and process the plurality of raw radar perception data groups to obtain a plurality of radar sparse tensor information groups. The radar-based DOA algorithm extracts and filters the signals (target signal reflection energy or signal-to-noise ratio (SNR) information) received by the radar from each resolvable and detectable signal direction of arrival (i.e., the spatial angular resolution of the radar), and detects valid signals (those with a relatively large signal-to-noise ratio, obvious target characteristics - such as motion characteristics, and obvious clustering characteristics of signals with obvious correlation with surrounding signals, etc.).

[0086] In other embodiments, machine learning and neural network processing are used to process the plurality of raw radar perception data groups to obtain a plurality of radar sparse tensor information groups. Specifically, the function of the first data processing method is described using a cluster analysis algorithm as an example:

[0087] Our detection targets are mostly spatial continua. We can ignore targets that are too small. The effective reflection signal of the detection target must have the characteristics of correlated distribution. The target object refers to the image acquisition area of ​​the image sensor or all physical objects within the detection field of the radar sensor. The detection target refers to several concrete detection target elements within the target object, such as the road surface, a vehicle, a street lamp, etc. The effective reflection signal includes: the Doppler, distance, azimuth and pitch angle of the received power measurement information after processing in three-dimensional space, and the combination of power measurement information.

[0088] For example, the motion speed (Doppler) and the energy of the detected signal of the same target in adjacent original radar perception data groups are the same or similar. In this way, we can use data clustering algorithm analysis to filter out effective signals and filter out random noise information. The retained original radar perception data group is saved as a radar sparse tensor information group. The detection data of the original radar perception data group will be effectively reduced a lot, which is conducive to the subsequent data transmission, storage and processing.

[0089] The random noise data refers to random discrete data obtained from the detection, which can be understood as data in which the movement speed and the energy of the detection signal are randomly distributed and the values ​​vary greatly. Therefore, it is considered to be random discrete data obtained from the detection of non-target reflection points.

[0090] The technical solution provided by the present invention, by collecting a set of raw radar perception data, can then detect a comprehensive range of moving and stationary targets through a two-step process: a first data processing method and a combination of three-dimensional multi-dimensional depth perception data. This method also eliminates invalid information, reducing the computing burden. Furthermore, the multipath reflection detection information of the target retained after invalid information elimination can also achieve functions that are unattainable by traditional automotive sensor detection systems. This combined information contains rich multi-dimensional perception information from the system, enabling subsequent cognitive systems (target determination systems) to identify targets and the environment. This system can maximize the advantages of radar target perception, extract more information, increase data dimensions, and achieve more accurate target detection.

[0091] In one embodiment, each frame of data in the stereoscopic, multi-dimensional depth perception data combination also includes one or more of: metadata, a timestamp of the event perception, information about the triggered event, and descriptive information about the perception results of the preceding and following frames. In another embodiment, multiple raw radar perception data sets can be combined and output with the stereoscopic, multi-dimensional depth perception data combination. This allows for efficient and lossless data combination without loss of valid detection information, enabling high-speed transmission and storage of detection data and enabling subsequent processing to analyze and mine all perception dimensions of the acquired data.

[0092] In one embodiment, in step S13, forming a stereoscopic multi-dimensional depth perception data combination specifically includes steps S131-S133:

[0093] S131: forming the radar matrix layer and the image pixel matrix layer, and determining a first mapping relationship; wherein the first mapping relationship represents a correspondence between the radar matrix elements and the arrayed pixel elements in the image pixel matrix layer;

[0094] Wherein, each frame data in the stereoscopic multi-dimensional depth perception data combination further includes: the first mapping data; wherein, the first mapping data refers to data representing the first mapping relationship.

[0095] S132: According to the first mapping relationship, one-to-one correspondence is made between the radar matrix elements and the pixel elements arranged in the array in the image pixel matrix layer along the first direction;

[0096] S133: assigning a plurality of radar sparse tensor information groups to corresponding radar matrix elements in the radar matrix layer;

[0097] In the image pixel matrix layer, each pixel element corresponds to the image perception data group; and the radar matrix layer includes a plurality of radar matrix elements arranged in an array.

[0098] In one embodiment, in step S131, forming the radar matrix layer specifically includes steps S1311-S1312:

[0099] S1311: Cutting a two-dimensional plane perpendicular to the central axis of the radar sensing field of view and projecting it into the detection field of the radar sensor;

[0100] S1312: Projecting each detection point within the radar perception field of view represented by the radar sparse tensor onto the mapping unit on the two-dimensional plane according to its corresponding direction of arrival; wherein each mapping unit is assigned or labeled with a radar sparse tensor information group corresponding to the direction of arrival;

[0101] In one embodiment, each of the mapping units includes at least one detection point within the radar sensing field of view.

[0102] In other embodiments, in a few cases, a preset characteristic value can also be used to mark that the mapping unit has no (no need) detection points within the radar perception field of view; the preset characteristic value can be set arbitrarily to indicate that the mapping unit has no (no need) detection points within the radar perception field of view.

[0103] Specifically, as shown in Figure 2, according to the preset radar spatial angle resolution, the radar sensing space is divided into a number of radar sensing units from the horizontal azimuth angle and the pitch angle, where each radar sensing unit corresponds to a corresponding detection point within the radar sensing field of view;

[0104] As shown in Figure 3, a two-dimensional plane is projected perpendicular to the central axis of the radar's perception field of view (FOV) into the radar perception space shown in Figure 2. Each detection point within the radar perception field of view (FOV) of the radar perception space is projected onto a small area (block) on this two-dimensional plane based on its direction of arrival. This is the mapping unit (block) of the radar perception field of view. The number of radar horizontal azimuth angles and elevation azimuth angles are mapped onto a two-dimensional matrix on the two-dimensional plane. Each mapping unit of the radar perception field of view is defined as a "radar pixel," and the combined area of ​​all "radar pixels" is the radar's perception field of view.

[0105] The direction of arrival corresponding to each of the radar sparse tensor information groups divided according to the spatial angle resolution includes:

[0106] Dividing the detection domain of the radar sensor of the radar sparse tensor information group into a corresponding number of horizontal azimuth angles and a corresponding number of elevation azimuth angles according to the spatial angle resolution of the radar, each horizontal azimuth angle and elevation azimuth angle corresponding to a detection point within the radar sensing field of view;

[0107] Each of the mapping units of the radar perception field of view is assigned a radar sparse tensor information group corresponding to the direction of arrival, and a plurality of the mapping units of the radar perception field of view form the radar matrix layer.

[0108] Before step S133: assigning a plurality of radar sparse tensor information groups to the corresponding radar matrix elements in the radar matrix layer, the method further includes:

[0109] At each "radar pixel," one or more corresponding radar sparse tensor information groups (range, azimuth, altitude, and Doppler velocity information; other low-level features such as signal reflection energy or the corresponding radar cross section (RCS) or signal-to-noise ratio (SNR) information) are extracted. The simplest and most effective combination of radar 4D sparse tensor data can be the range of the detected target + the signal reflection energy or the corresponding signal-to-noise ratio (SNR) information. Of course, if some signal arrival directions do not receive valid reflected signals (energy), such as when the radar is facing the open sky, we can also mark these signal arrival directions as having no valid targets.

[0110] In step S133, based on the mapping relationship between the radar sensor's detection field and the two-dimensional object surface corresponding to the camera imaging surface, one or more corresponding radar sparse tensor data groups for each "radar pixel" from the radar are associated with the corresponding camera image pixels with the mapping relationship. A stereoscopic multi-dimensional depth perception data combination is established with the camera pixels as the mapping basis. The 4D tensor data of the target detected by the radar is then associated with each pixel of the camera imaging based on the mapping relationship. The data detected by the multiple sensors are then synchronously output.

[0111] In one embodiment, in step S132, forming the first mapping relationship includes steps S1321-S1324:

[0112] S1321: Determine a second coordinate based on the first coordinate; the second coordinate represents the coordinate of the first coordinate after coordinate transformation in the image coordinate system; the first coordinate represents the coordinates of the plurality of radar sparse tensor information groups in the coordinate system of the detection domain of the radar sensor; the image coordinate system represents the coordinate system of the image acquisition area of ​​the image sensor;

[0113] S1322: Aligning the detection field of the radar sensor with the image acquisition area of ​​the image sensor; wherein the central axis of the detection field of the radar sensor is aligned with the optical axis of the image acquisition sensor, and the detection vertex of the detection field of the radar sensor coincides with the center of the entrance pupil of the image acquisition sensor;

[0114] S1323: Projecting the detection area of ​​the radar sensor onto a two-dimensional plane where the imaging surface of the image acquisition sensor is located according to the optical axis of the image acquisition sensor and the center of the entrance pupil to obtain a projection area corresponding to the radar sensor;

[0115] S1324: Determine the first mapping relationship based on the positional relationship between the projection area and the imaging area of ​​the image acquisition sensor; the imaging area of ​​the image acquisition sensor is located on the two-dimensional plane where the imaging surface of the image acquisition sensor is located.

[0116] Corresponding radar data to image data: 2.1 Before corresponding radar data to image data, the process also includes: filtering out some direction of arrival data;

[0117] (Of course, in some cases, some signal arrival direction information can be directly filtered out (non-continuous radar pixel combination), which is also an acceptable combination. For example, if the system confirms that the information in these wave directions is all noise or non-target detection targets, we can directly filter out the signals in these corresponding arrival directions and no longer connect them to the corresponding pixels of the camera; this combination can reduce data transmission and storage.

[0118] In one embodiment, in each of the image pixel matrix layers, the image pixel data group includes: any one of: a brightness data group, a grayscale data group, a red light data group, a green light data group, a blue light data group, a Y data group, a U data group, a V data group or an optical flow data group.

[0119] In one embodiment, the radar sensor is at least one of the following: a microwave radar, an ultrasonic radar, a laser radar, an infrared sensor, and a terahertz imaging sensor. The image sensor is a camera, etc.

[0120] In one embodiment, the radar 4D sparse tensor information group represents the detection and combination of radar 4D tensor power measurement information of four spatial dimensions corresponding to Doppler, distance, horizontal azimuth and pitch angle of the radar corresponding to the specified spatial position for the received power measurement information; wherein, the radar 3D sparse tensor information group is the detection and combination of radar 3D tensor power measurement information of three three-dimensional spatial dimensions corresponding to Doppler, distance and azimuth of the radar corresponding to the specified spatial position for the received power measurement information.

[0121] In the technical solution provided by the present invention, according to the application scenario, a radar 4D sparse tensor information group or a radar 3D sparse tensor information group is selected to form a three-dimensional multi-dimensional depth perception data combination and output; wherein, the radar 4D sparse tensor information group is a combination of Doppler, distance, azimuth and pitch angles and power measurement information after processing the received power measurement information corresponding to a specified spatial (3D) position based on the radar signal processing principle.

[0122] In some application scenarios, if the system does not prioritize extracting the height information of the target object, we can detect and combine the power measurement information corresponding to Doppler, distance, and horizontal azimuth angles for the received power measurement information corresponding to the specified spatial (3D) position based on the radar signal processing principle (which can also be regarded as processing the radar 3D tensor information). Then, the power measurement information corresponding to Doppler, distance, and horizontal azimuth angles for the received power measurement information corresponding to the specified spatial (3D) position is combined and output accordingly with the camera pixel data; in this way, the data processing and output of one less dimension will be simplified to meet the high-speed transmission, processing, and storage of limited system detection data, which can meet the simplified processing of the system. The radar signal processing principle is an existing mature technology, and the present invention will not elaborate on it here. In addition, the radar signal's Doppler, distance, and horizontal azimuth power measurement information is first detected, which is very consistent with the BEV perception application in the autonomous driving system, and is a good and efficient application combination docking. The concept of BEV perception is to convert the camera image from a perspective view to a bird's eye view (BEV). The BEV view contains the most important distance and azimuth information for autonomous driving (altitude information is implicit in the BEV characteristics). We use the Doppler, distance, and horizontal azimuth information of the target detected by the radar to assist the BEV perception application in the autonomous driving system, which can make the system perception more accurate and efficient.

[0123] Secondly, according to one embodiment of the present invention, a method for combined detection of radar and camera sensors for machine learning is also provided. The stereoscopic multi-dimensional depth perception data combination obtained by the combined detection method of radar and camera sensors described in any one of the aforementioned first embodiments of the present invention is used for machine learning through the collection and sample labeling of system data samples.

[0124] The technical solution provided by the present invention provides these data with both complete image information and radar perception information of corresponding pixels. This combined data is a very good sample collection and annotation mode for machine learning and neural network training.

[0125] We combined several image perception data groups with radar sparse tensor information groups to establish a three-dimensional multi-dimensional depth perception data combination based on the pixels of the image sensor as the mapping basis. It is used for system data sample collection and sample labeling, which is very direct and efficient. The product (the radar and camera fusion product itself) can be a data collector. The collected data itself has complete visual scene interpretation information, so sample labeling can be done directly and conveniently. There is no need to add an additional camera for sample collection and use an additional system to specifically associate video information with radar data information, which saves costs. The comprehensive and complete radar sparse tensor information group increases the accuracy of machine learning.

[0126] In addition, according to an embodiment of the present invention, as shown in FIG7 , a device 20 for combined detection of radar and camera sensors is also provided, comprising:

[0127] Data acquisition module 21; used to acquire several image perception data sets and several original radar perception data sets of the target object;

[0128] An image processing module 22 is configured to process the plurality of raw radar perception data groups according to a first data processing method to obtain a radar 4D sparse tensor or a radar 3D tensor information group; and form a stereoscopic multi-dimensional depth perception data combination;

[0129] An image processing module; specifically, it can be understood as being configured to process the plurality of raw radar perception data groups based on their semantic classification according to a first data processing method to obtain corresponding radar 4D sparse tensors or radar 3D tensor information groups; and to form a stereoscopic multi-dimensional depth perception data combination;

[0130] In which, the stereoscopic multi-dimensional depth perception data combination is formed in a multi-dimensional matrix structure; the stereoscopic multi-dimensional depth perception data combination includes a number of the radar 4D tensor information assigned to the multi-dimensional matrix structure, or the radar 3D tensor information and a corresponding number of the image perception data groups.

[0131] The following are some examples of application scenarios of the technical solution provided by the present invention:

[0132] In the first embodiment, as shown in FIG4 , the technical solution provided by the present invention allows the system to see "invisible dangerous information". Specifically, as shown in FIG4 , the original radar data group corresponding to the preceding vehicle and the original radar data group of the large truck can be detected. After the three-dimensional multi-dimensional depth perception data combination is formed in a multi-dimensional matrix structure, a large truck traveling on the road ahead can be seen. However, whether there are other vehicles in front of the truck here is not visible in the image information.

[0133] For intelligent driving systems (autonomous driving) on ​​highways, the system needs to be able to detect whether there are other vehicles in front of the car. If there is another car in front of the car in front, the sudden braking of the "front car" may cause the car in front to suddenly decelerate, causing the car behind to fail to brake in time and cause a collision. We combine the 4D radar tensor within the radar detection field of view with the image to synthesize "multi-dimensional" information. The radar will detect the existence of the "previous vehicle" based on the multipath reflection of the radar signal in the gap between the bottom of the preceding vehicle and the road (in the bottom area of ​​the preceding vehicle (multipath reflection point); specifically, for example: there are multiple reflection points within a range of distance ~75M and movement speed ~100KM / hour, so we can infer the detection result: there is another vehicle in front of this truck (not detected in the image, obscured by the truck), moving at a speed of nearly 100KM / hour, and a distance of nearly 75M. The system then performs multipath reflection distance conversion processing based on the image information and derives that the relative distance of the preceding vehicle is 68 meters) (Of course, the detection and detection results here are for illustration only, and the actual detection values ​​need to be inferred based on the same principle). For this application scenario, we can use our system to sample data for machine learning. We sample and label data for various scenarios (with and without the preceding vehicle, various road conditions, various weather conditions, and other combinations). As the number of valid samples increases, the system iterates through training, effectively improving target detection accuracy. Similarly, directly labeling radar data samples without integrating visual information is difficult.

[0134] As shown in Figure 5, in the second embodiment, the 4D radar tensors selected from all directions within the radar detection field of view are combined, so that the radar "imaging" is not just based on a sparse "radar point cloud". Our solution is to generate a full set of "radar pixels" (each pixel has radar target detection tensor information, including information on moving targets and stationary targets) to connect with the image pixels, thereby forming a complete multi-dimensional perception. In this way, the radar in our system's perception of the environment and targets is not just a sparse point cloud perception of targets (vehicles, pedestrians, traffic signs, etc.). The perception information of our radar system includes the complete radar effective detection information of the 4D radar tensor combination detected in all resolvable directions within the radar perception field of view within all horizontal azimuth and elevation angles according to the angular resolution range of the radar system antenna design. We can also detect stationary targets in the environment, such as the detection of manhole covers on the road surface shown in the figure, the detection of road surface undulation, the detection of curbs and steps, and so on.

[0135] In this way, we can overcome the weaknesses and challenges of current independent radar system target detection: the sparsity and relative discreteness of detected targets, and the uncertainty of detecting relatively stationary targets (current common radar systems filter out the signals of relatively stationary targets, resulting in the inability to detect relatively stationary targets). This allows us to achieve hyper-dimensional coupling of the system, give full play to the target perception advantages of radar sensors, and enable the system to accurately perceive targets and the environment.

[0136] In the third embodiment, as shown in Figure 7, radar's angular resolution is currently much lower than that of cameras. Many current camera products have reached 8 million pixels or even over 10 million pixels, with very high angular resolution between pixels. However, radar (particularly millimeter-wave radar), due to the limited number of effective antennas, currently has an angular resolution generally higher than 0.1 degrees (or even 1 degree). However, radar has good resolution of target distance and relative velocity at each wave arrival angle, and millimeter-wave radar also has the ability to detect multiple targets (multiple targets at different distances in front and behind) in the same longitudinal dimension.

[0137] Referring to the image in the upper right corner of Figure 6, we take an example detection scenario. In the same sensing area, the camera image is 390x360 image pixels, while the corresponding millimeter-wave radar has only 9x9 "radar pixels." Therefore, in one "radar pixel," the radar simultaneously senses radar reflection signals from a car parked on the road 15 meters away (with a relative motion speed of 0 ms / s), a pedestrian walking 20 meters away (with a relative motion speed of 1.5 ms / s), and a stationary roadside building 50 meters away (with a relative motion speed of 0 ms / s). A radar pixel contains a radar 4D sparse tensor group.

[0138] The radar's perception output will be: target 1 (with a relatively high RCS value, 15 meters away, and a relative movement speed of 0 ms / s per second), target 2 (with a relatively low RCS value, 20 meters away, and a relative movement speed of 1.5 ms / s per second), and target 3 (with a relatively high RCS value, 50 meters away, and a relative movement speed of 0 ms / s per second).

[0139] In such scenarios, if image information is not involved and radar signal processing is used alone, it is very challenging for the radar to distinguish these three signals from the basic radar signal categories: "effective target detection information," "target multipath reflection detection information," "radar signal crosstalk information" (interference signals from the radar or other radars), and "random noise information." In many cases, targets with low signal reflection intensity (low RCS values) and stationary targets are treated as noise and filtered out.

[0140] However, since we have image information, we can determine based on the image information that there are vehicles, pedestrians and distant buildings in the direction of the radar wave. Corresponding to the three target perception information sensed by the radar, the system can accurately detect and determine the information of these targets.

[0141] As shown in FIG8 , according to another embodiment of the present invention, there is further provided an electronic device 30 , comprising a processor 31 and a memory 32 , wherein the memory 31 is configured to store code;

[0142] The processor 31 is configured to execute the code in the memory 32 to implement the method for combined detection of radar and camera sensors as described in any one of the first embodiments of the present invention.

[0143] According to another embodiment of the present invention, a storage medium is provided, on which a computer program is stored. When the program is executed by the processor 31, the method for combined detection of radar and camera sensors as described in any one of the aforementioned first embodiments of the present invention is implemented.

[0144] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for combined detection of radar and camera sensors, characterized in that: include: Acquire several image perception data groups and several original radar perception data groups of target objects; wherein the target objects include moving target objects and stationary target objects; each group of the original radar perception data groups includes a combination of power measurement information data corresponding to radar 4D tensors in four dimensions of distance, Doppler, horizontal azimuth and pitch angle perceived by the radar sensor at each sampling point in the three-dimensional space where the detection domain of the radar sensor is located, wherein the power measurement information data includes: data of signal reflection energy, or data of radar cross section or signal-to-noise ratio information corresponding to the signal reflection energy; the several image perception data groups represent the data acquired by the image sensor; According to a first data processing method, the plurality of original radar perception data groups are processed to obtain a plurality of radar sparse tensor information groups; wherein the first data processing method is used to remove random noise data in the original radar perception data group; and data of random noise and crosstalk information of radar signals generated by the target object detection; the random noise data represents random discrete data obtained by detection in the detection space; Forming a stereoscopic multi-dimensional depth perception data combination and outputting the stereoscopic multi-dimensional depth perception data combination; the stereoscopic multi-dimensional depth perception data combination is formed in a multi-dimensional matrix structure; the stereoscopic multi-dimensional depth perception data combination includes a plurality of the radar sparse tensor information groups assigned to the multi-dimensional matrix structure and a corresponding plurality of the image perception data groups; The multidimensional matrix structure includes a plurality of matrix layers arranged in sequence along a first direction, and the matrix layers include at least one image pixel matrix layer and at least one radar matrix layer; the radar matrix layer includes a plurality of array-arranged radar matrix elements; the radar matrix elements represent elements corresponding one to one to the array-arranged pixel elements in the image pixel matrix layer along the first direction; each of the pixel elements includes the image perception data group; the radar matrix elements in the radar matrix layer are all assigned corresponding radar sparse tensor information groups.

2. The method for combined detection of radar and camera sensors according to claim 1, characterized in that: The formation of a stereoscopic multi-dimensional depth perception data combination specifically includes: Forming the radar matrix layer and the image pixel matrix layer, and determining a first mapping relationship; wherein the first mapping relationship represents a correspondence between the radar matrix elements and the array-arranged pixel elements in the image pixel matrix layer; According to the first mapping relationship, the radar matrix elements are matched one-to-one with the pixel elements arranged in the array in the image pixel matrix layer along the first direction; Assigning a number of radar sparse tensor information groups to corresponding radar matrix elements in the radar matrix layer; Among them, in the image pixel matrix layer, each of the pixel elements corresponds to the image perception data group; and in the radar matrix layer, a plurality of radar matrix elements arranged in an array are included.

3. The method for combined detection of radar and camera sensors according to claim 2, characterized in that: Forming the radar matrix layer specifically includes: Cutting into a two-dimensional plane at a direction perpendicular to the central axis of the radar sensing field of view and projecting it into the detection field of the radar sensor, wherein the two-dimensional plane includes a plurality of mapping units divided according to a preset spatial angle resolution; Projecting each detection point within the radar perception field of view represented by the radar sparse tensor information group to the mapping unit on the two-dimensional plane according to its corresponding direction of arrival; Each of the mapping units is assigned or marked with the radar sparse tensor information group in the corresponding direction of arrival.

4. The method for combined detection of radar and camera sensors according to claim 3, characterized in that: Forming the first mapping relationship includes: Determine a second coordinate according to the first coordinate; the second coordinate represents the coordinate of the first coordinate in the image coordinate system after the coordinate transformation; the first coordinate represents the coordinate of the plurality of radar sparse tensor information groups in the coordinate system where the detection domain of the radar sensor is located; the image coordinate system represents the coordinate system where the image acquisition area of ​​the image sensor is located; Aligning the detection field of the radar sensor with the image acquisition area of ​​the image sensor; wherein the central axis of the detection field of the radar sensor is aligned with the optical axis of the image acquisition sensor, and: the detection vertex of the detection field of the radar sensor coincides with the entrance pupil center of the image acquisition sensor; According to the optical axis of the image acquisition sensor and the entrance pupil center, the detection field of the radar sensor is projected onto a two-dimensional plane where the imaging surface of the image acquisition sensor is located, so as to obtain a projection area corresponding to the radar sensor; The first mapping relationship is determined according to the positional relationship between the projection area and the imaging area of ​​the image acquisition sensor; the imaging area of ​​the image acquisition sensor is within the image acquisition sensor. The imaging surface of the device is located in the two-dimensional plane.

5. The method for combined detection of radar and camera sensors according to claim 1, characterized in that: In each of the image pixel matrix layers, the image pixel data group includes any one of: a brightness data group, a grayscale data group, a red light data group, a green light data group, a blue light data group, a Y data group, a U data group, a V data group or an optical flow data group.

6. The method for combined detection of radar and camera sensors according to claim 1, characterized in that: The radar sensor is at least one of the following: a microwave radar, an ultrasonic radar, a laser radar, an infrared sensor, and a terahertz imaging sensor.

7. The method for combined detection of radar and camera sensors according to claim 1, characterized in that: The radar sparse tensor information group includes a radar 4D sparse tensor information group or a radar 3D sparse tensor information group; wherein, the radar 4D sparse tensor information group represents the detection and combination of radar 4D tensor power measurement information of four spatial dimensions corresponding to Doppler, distance, horizontal azimuth and pitch angle for the received power measurement information corresponding to the specified spatial position; wherein, the radar 3D sparse tensor information group is the detection and combination of radar 3D tensor power measurement information of three stereoscopic spatial dimensions corresponding to Doppler, distance and azimuth for the received power measurement information corresponding to the specified spatial position.

8. A method for combined detection of radar and camera sensors for machine learning, characterized in that: The stereoscopic multi-dimensional depth perception data combination obtained by the method of combined detection of radar and camera sensors as described in any one of claims 1 to 7 is used for machine learning through the collection and sample labeling of system data samples.

9. A device for combined detection of radar and camera sensors, characterized in that: include: Data acquisition module; Used to obtain a plurality of image perception data groups and a plurality of original radar perception data groups of the target object; Image processing module; Used to process the plurality of original radar perception data groups according to the first data processing method to obtain a radar 4D sparse tensor or a radar 3D tensor information group; and form a stereoscopic multi-dimensional depth perception data combination; Among them, the stereoscopic multi-dimensional depth perception data combination is formed in a multi-dimensional matrix structure; the stereoscopic multi-dimensional depth perception data combination includes a number of the radar 4D tensor information assigned to the multi-dimensional matrix structure, or the radar 3D tensor information and a corresponding number of the image perception data groups.

10. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory is used to store codes; The processor is used to execute the code in the memory to implement the method for combined detection of radar and camera sensors as described in any one of claims 1 to 7.

11. A storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for combined detection of radar and camera sensors as claimed in any one of claims 1 to 7.

Images