A sensor data processing method and apparatus

Abstract

The application discloses a sensor data processing method and device, and relates to the technical field of radar technology. The method comprises the following steps: acquiring an image photographed by a camera at the same timestamp in the same direction as radar point cloud data, performing depth estimation on the image to obtain a depth estimation matrix, and labeling a target in the image through a label frame; determining the position of the target in the depth estimation matrix according to the depth estimation matrix and the pixel points of the label frame in the image, and then determining the depth information of the target in the image according to the position of the target in the depth estimation matrix, taking the mean value of the depth information to obtain the predicted position of the target in the image; and determining the value of the point cloud cluster containing the target from the center of the radar point cloud data according to the lateral distance and the predicted position of the target in the image. The method complements the advantages of various sensors, and improves the labeling efficiency and robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of radar, in particular to a sensor data processing method and device. BACKGROUND

[0002] With the development of technology, millimeter wave radars are widely used in various industries. It is a common means in existing positioning and tracking technologies to detect the distance, speed, direction, and attitude of target objects through millimeter wave radars or image data.

[0003] However, due to the data characteristics of millimeter wave radars, labeling is difficult. Therefore, how to fully fuse and register various types of sensor data to achieve target labeling in radar point clouds and reduce the workload of manual radar point cloud labeling is a problem to be solved. SUMMARY

[0004] In a first aspect, the present application provides a sensor data processing method, which comprises: acquiring radar point cloud data, acquiring an image taken by a camera in the same direction as the radar point cloud data at the same timestamp, performing depth estimation on the image to obtain a depth estimation matrix, and labeling a target in the image by a labeling box; determining the position of the target in the depth estimation matrix according to the pixel points of the depth estimation matrix and the labeling box in the image, and then determining the depth information of the target in the image according to the position of the target in the depth estimation matrix, taking the average of the depth information to obtain the predicted position of the target in the image; determining the lateral distance between the radar corresponding to the radar point cloud data and the camera, and determining the value of the point cloud cluster containing the target from the center of the radar point cloud data according to the lateral distance and the predicted position of the target in the image, wherein the point cloud cluster containing the target includes the radar point cloud data.

[0005] In any of the above technical solutions, further, the depth estimation matrix obtained by performing depth estimation on the image comprises:

[0006] The depth estimation matrix is obtained by performing depth estimation on the image by an FCRN depth estimation method.

[0007] In any of the above technical solutions, further, the method further comprises:

[0008] The value of the point cloud cluster containing the target from the center of the radar point cloud data is added to a preset error to obtain the value of the calibrated point cloud cluster containing the target from the center of the radar point cloud data.

[0009] In any of the above technical solutions, further, the method further comprises:

[0010] Convert at least one direction of radar point cloud data to the same coordinate system, splice the radar point cloud data converted to the same coordinate system together to obtain the radar point cloud data, combine the radar point cloud data and the image to obtain the motion information of the target, label the radar point cloud data according to the motion information, and determine the point cloud cluster containing the target;

[0011] Determine the preset error according to the point cloud cluster containing the target and the predicted position of the target in the image.

[0012] In any of the above technical solutions, further, the combining the radar point cloud data and the image to obtain the motion information of the target comprises:

[0013] Save the radar point cloud data as a gif format file;

[0014] Obtain the motion information of the target according to the gif format file and the image.

[0015] In any of the above technical solutions, further, the obtaining the motion information of the target according to the gif format file and the image comprises:

[0016] Obtain the motion direction and the motion distance of the target according to the gif format file and the image, and further determine the motion information of the target according to the motion direction and the motion distance of the target.

[0017] In any of the above technical solutions, further, the determining the position of the target in the depth estimation matrix according to the depth estimation matrix and the pixel point of the bounding box in the image comprises:

[0018] Determine the conversion relationship between the pixels in the image and the pixels in the depth estimation matrix according to the size of the image and the size of the depth estimation matrix;

[0019] Determine the position of the target in the depth estimation matrix according to the conversion relationship and the pixel point of the bounding box in the image.

[0020] In a second aspect, the present application also provides a sensor data processing apparatus, comprising: a first processing module configured to acquire radar point cloud data, acquire an image taken by a camera in the same direction as the radar point cloud data at the same timestamp, perform depth estimation on the image to obtain a depth estimation matrix, and label a target in the image by a label box; a second processing module configured to determine a position of the target in the depth estimation matrix according to a pixel point of the label box in the image and the depth estimation matrix, and further determine depth information of the target in the image according to the position of the target in the depth estimation matrix, and obtain a predicted position of the target in the image by averaging the depth information; and a third processing module configured to determine a lateral distance between a radar corresponding to the radar point cloud data and the camera, and determine a value of a point cloud cluster containing the target from a center of the radar point cloud data according to the lateral distance and the predicted position of the target in the image, wherein the point cloud cluster containing the target comprises the radar point cloud data.

[0021] In a third aspect, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the sensor data processing method according to any one of the above aspects when executing the program.

[0022] In a fourth aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the steps of the sensor data processing method according to any one of the above aspects.

[0023] In a fifth aspect, the present application also provides a computer program product comprising a computer program, wherein the computer program is executable on a processor to implement the steps of the sensor data processing method according to any one of the above aspects.

[0024] The present application has the following advantages: the technical solution in the present application determines a value of a point cloud cluster containing a target from a center of radar point cloud data according to a predicted position of the target in an image and a lateral distance between a radar corresponding to the radar point cloud data and the camera, i.e. completes fusion and registration of multiple types of sensors, thereby improving the efficiency and robustness of labeling. BRIEF DESCRIPTION OF DRAWINGS

[0025] The above and / or additional aspects of the present application will become apparent and more readily appreciated from the following description of embodiments, taken in conjunction with the accompanying drawings, in which:

[0026] Figure 1 is a schematic flowchart of a sensor data processing method according to an embodiment of the present application;

[0027] Figure 2 is a schematic diagram of a positional relationship between a camera and a radar;

[0028] Figure 3 is a schematic diagram of a left camera labeled result and a radar pedestrian target extracted according to a depth estimation result;

[0029] Figure 4 is a structural schematic diagram of some embodiments of the sensor data processing apparatus provided by the present application;

[0030] Figure 5 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0031] In order to enable a more clear understanding of the above-mentioned purposes, features and advantages of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0032] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and therefore, the scope of protection of the present application is not limited by the specific embodiments disclosed below.

[0033] As shown in Figure 1 The present embodiment provides a sensor data processing method, which comprises:

[0034] Step 1, acquiring radar point cloud data, acquiring an image taken by a camera in the same direction as the radar point cloud data at the same timestamp, performing depth estimation on the image to obtain a depth estimation matrix, and labeling a target in the image by a label frame.

[0035] In some optional implementations, the depth estimation matrix can be obtained by performing depth estimation on the image by a FCRN depth estimation method.

[0036] As an example, three-way cameras can be installed according to the positions of carbon plates, and the distances between the actual three-way cameras are measured. The image data obtained by the three-way cameras are respectively subjected to depth estimation, and the depth estimation application method is a FCRN method. The FCRN method is one of the most commonly used model depth prediction methods in iOS. The model uses a single image to predict the depth level of a scene based on CNN (ResNet-50), and uses a pre-trained model to utilize a residual network.

[0037] Wherein, the surround-view radar point cloud data (i.e. radar point cloud data of multiple azimuths and angles) can be acquired, the surround-view radar point cloud data of multiple azimuths and angles is spliced together to obtain the radar point cloud data recorded in step 1, and then images captured by multiple cameras in the same direction and angle as the surround-view radar point cloud data are acquired, the images are respectively subjected to depth estimation to obtain respective depth estimation matrices, and the target in the images is respectively labeled by a label frame.

[0038] In step 2, the position of the target in the depth estimation matrix is determined according to the pixel points of the label frame in the image and the depth estimation matrix, and then the depth information of the target in the image is determined according to the position of the target in the depth estimation matrix, and the depth information is averaged to obtain the predicted position of the target in the image.

[0039] In some optional implementations, the conversion relationship between the pixels in the image and the pixels in the depth estimation matrix can be determined according to the size of the image and the size of the depth estimation matrix, and then the position of the target in the depth estimation matrix is determined according to the conversion relationship and the pixel points of the label frame in the image.

[0040] As an example, the label format of interest can be set in the labeling tool, for example, three labels of selected labelers, potted plants and walls. For camera image data, the label frame position can be obtained by visually observing the original image and manually labeling

[0041] x bounding_box_min ,x bounding_box_max ,y bounding_box_min ,y bounding_box_max ,

[0042] Wherein x bounding_box_min ,x bounding_box_max is the minimum and maximum horizontal coordinate of the label frame, and y bounding_box_min ,y bounding_box_max is the minimum and maximum vertical coordinate of the label frame.

[0043] As an example, the position of the target in the depth estimation matrix can be obtained according to the conversion relationship between the pixel points of the camera label frame and the depth estimation matrix to obtain the depth of the target of interest. For example, the size of the image is 1024*1280, and the size of the depth estimation matrix is 128*160. According to 1024 / 128=1280 / 160=8, the pixel position of the label frame can be obtained according to the conversion relationship, i.e. x bounding_box =8×x prediction_matrix and y bounding_box =8×y prediction_matrix , wherein x bounding_box ,y bounding_box represents the pixel position of the label frame, x prediciton_matrix ,y prediciton_matrixCoordinates representing the estimated matrix. According to the pixel position conversion relationship, x bounding_box_min bounding_box_max bounding_box_min bounding_box_max Then, the pixel position x prediction_matrix_min prediction_matrix_max prediction_matrix_min prediction_matrix_max According to the pixel position, the average value of the depth information result in the annotation box range is obtained to obtain the predicted position d pre .

[0044] Step 3, determining the lateral distance between the radar corresponding to the radar point cloud data and the camera, and determining the value of the point cloud cluster containing the target from the center of the radar point cloud data according to the lateral distance and the predicted position of the target in the image, wherein the point cloud cluster containing the target includes the radar point cloud data.

[0045] In some optional implementations, the value of the point cloud cluster containing the target from the center of the radar point cloud data can be added to a preset error to obtain the value of the calibrated point cloud cluster containing the target from the center of the radar point cloud data.

[0046] In some optional implementations, the preset error can be determined by the following method:

[0047] Convert the radar point cloud data in at least one direction to the same coordinate system, splice the radar point cloud data converted to the same coordinate system together to obtain the radar point cloud data, combine the radar point cloud data and the image to obtain the motion information of the target, label the radar point cloud data according to the motion information, and determine the point cloud cluster containing the target; and determine the preset error according to the point cloud cluster containing the target and the predicted position of the target in the image.

[0048] Wherein, the radar point cloud data can be saved as a gif format file, and the motion information of the target can be obtained according to the gif format file and the image.

[0049] Wherein, the motion direction and motion distance of the target can be obtained according to the gif format file and the image, and then the motion information of the target can be determined according to the motion direction and motion distance of the target.

[0050] ​​​​​​In some embodiments, the distance between the point cloud cluster containing the target and the center of the radar point cloud data can be determined based on the lateral distance between the cameras at multiple azimuths and angles and the surrounding radar at the corresponding azimuths and angles, and based on the lateral distance and the predicted position of the target in the image. That is, the corresponding radar point cloud cluster containing the target can be determined by using the relevant data of the cameras at the corresponding azimuths and angles (images taken by the cameras, lateral distance between the radar and the cameras, etc.), so that labeled radar data at multiple azimuths and angles can be obtained.

[0051] As an example, see reference Figure 2 ( Figure 2 (a) is a diagram showing the sensor installation location; Figure 2 (b) shows the planar positional relationship of the millimeter-wave radar; Figure 2 (c) For radar camera data, the multi-directional radar point cloud data obtained by the radar can be stitched together first, with the geometric center of the radar plane as the origin of the coordinate system. Figure 2 Establish a coordinate system based on the coordinate system direction in (c), and then according to Figure 2 (b) The parameters marked in the diagram are used to obtain the coordinates of the center point of the multi-directional radar in the coordinate system. The ordinate of the radar-detected points in the positive y-axis direction is increased by 0.39, and the x-coordinate of the radar-detected points in the positive x-axis direction is negative, with the ordinate increased by 0.25. Then the x and y coordinates are swapped. Similarly, the ordinate of the radar-detected points in the negative x-axis direction is increased by 0.25 and then negative, and the x and y coordinates are swapped again. This transfers the panoramic point cloud to the same coordinate system. The multi-directional radar detection point data transformed to the same coordinate system are then stitched together into a data matrix. Each frame of the radar point cloud image is saved, and the stitched point cloud image is used as the original radar annotation data (i.e., the radar point cloud data obtained in step 1). Since radar point cloud images do not provide very clear object information and can only determine obstacle information, observing a single frame image cannot obtain target information. However, during the dynamic display of the image, the movement of the same target can be seen. Therefore, continuously displaying each frame of data allows for the observation of more intuitive target movement information. The radar point cloud image is displayed in Matlab at 0.05-second intervals and saved as a .gif file. By observing the movement of the point cloud clusters in the .gif file and combining this with the target movement information observed by cameras in the same direction at the same time stamp, the radar point cloud image can be manually annotated. In the .gif file, it can be observed that the movement direction of the point cloud cluster on the left is the same as the movement direction and distance of the pedestrian target in the left-looking camera. Therefore, this point cloud cluster in the radar image can be considered the pedestrian target.

[0052] As an example, for radar point cloud data, the radar point cloud can first be clustered using the DBSCAN algorithm. Then, based on the approximate coordinate range of the bounding boxes in the radar image, the approximate location of the target can be obtained. Point cloud clusters representing the target can be extracted, and the radial distances within the point cloud clusters can be averaged to obtain the estimated location r of the target in the radar point cloud. pre Then, based on the estimated position r of the target in the radar point cloud... pre The preset error is determined by the predicted position of the target in the image.

[0053] As an example, we can first determine based on Figure 2 (a) records the lateral distances between the sensor installation locations of each radar and camera. For the unlabeled radar point cloud image, d is obtained through the above embodiment. pre According to the estimated position d pre Then switch to the corresponding position in the surround-view radar point cloud.

[0054] If it's a forward-facing camera in front of the car, then in d represents the estimated range of distance from the center of the radar point cloud. front-radar A fixed value representing the lateral horizontal distance between the center of the forward-looking camera and the radar point cloud image is selected from the surround-view radar image. Coordinates, along with the target's direction of motion and distance traveled, are extracted and estimated simultaneously, where y radar The vertical coordinate represents the vertical coordinate in the surround-view radar, and the position difference of the same target is obtained by comparing the detection results of the radar and the camera for the same target with known true position.

[0055] If it's a rear-view camera, then Where d back-radar The horizontal distance between the center of the left-view camera and the radar point cloud image is represented by the selected image in the surround-view radar map. Coordinates, where y radar The vertical coordinate represents the surround-view radar.

[0056] If it's a left-facing camera, then Where d left-radar The horizontal distance between the center of the left-view camera and the radar point cloud image is represented by the selected image in the surround-view radar map. Coordinates, where x radar The horizontal axis represents the surround-view radar.

[0057] If the camera is viewed from the right, then Where d right-radar The horizontal distance between the center of the right-view camera and the radar point cloud image is represented by the selected image in the surround-view radar map. coordinate.

[0058] When a target appears simultaneously in two adjacent cameras, including the front-view camera and the left-view camera, the front-view camera and the right-view camera, the rear-view camera and the left-view camera, and the rear-view camera and the right-view camera, the front-view camera and the rear-view camera provide the vertical coordinates in the radar point cloud, and the left-view camera and the right-view camera provide the horizontal coordinates in the radar point cloud. This allows us to obtain the target's specific coordinate parameters in the radar point cloud.

[0059] As an example, three millimeter-wave radars with a field of view (FOV) of 120° are used indoors, according to... Figure 2 (b) is installed correctly, where the lateral distance d between the forward-looking radar and the forward-looking camera is... front-radar The distance d between the left-looking radar and the left-looking camera, and between the right-looking radar and the right-looking camera is 20.2 cm. left-radar and d right-radar The value is 22 cm, and the value of dif_pre is 20-25 cm.

[0060] like Figure 3 As shown ( Figure 3 (a) A left-view camera image with the target labeled; Figure 3 (b) is the radar point cloud extracted based on the depth estimation results; Figure 3 (c) Pedestrian targets in radar images obtained by combining the target's direction of motion. Figure 3 (a) The bounding box for the person is x: 468-670, y: 229-716, which translates to x: 59-83, y: 29-89 in the depth matrix. Since the bounding box includes distant scenes, it is relatively reduced to x: 65-77, y: 55-63. The depth estimate at this point is 3.7143m. Therefore, the radar-estimated location is... It is 3.93m, x radar It should be between -4.0143 and -3.543. Within this range, you can circle... Figure 3 (c) Point cloud targets to the left of the zero point, with coordinates of -3.7593, such as... Figure 3 As shown in (b), according to Figure 3 (b) and the point cloud motion states available in the .gif file, marked as follows: Figure 3 (c) shows the pedestrian target.

[0061] The above embodiments apply to the registration method of multi-directional and multi-sensor surround-view data. The target position is given by the depth estimation of a monocular camera, and the target is marked in the radar point cloud by establishing a coordinate system and mapping it to the surround-view radar point cloud map, thereby reducing the workload of manual radar point cloud marking.

[0062] As an example, camera and radar point cloud data can be imported into an online annotation tool for online annotation.

[0063] The steps in the present application can be adjusted, combined and deleted in sequence according to actual needs.

[0064] The technical solution in the present application obtains the overall radar point cloud after splicing the surround-view radar data, places the data of the radar and the camera in the same coordinate system for complete data coarse registration through the position relationship between the external measurement radar plane and the camera plane; the depth of the camera image is obtained through monocular depth estimation, the target is labeled, then the corresponding matrix points of the target are corresponded in the depth matrix to obtain the depth position of the target, and finally the position of the target is mapped in the radar point cloud image. The present application performs registration through depth estimation of the camera combined with point cloud clustering of the millimeter wave radar, thereby performing target extraction and registration of the radar and the camera in the surround-view system, and can perform semi-automatic joint labeling of the target in the surround-view millimeter wave radar and the camera.

[0065] The present application can relate to a system for implementing the above method, comprising: a depth estimation unit, a target labeling unit and a surround-view target joint labeling unit, wherein: the depth estimation unit performs distance estimation of all objects in the image according to image data to obtain distance information of all targets in the image, the target labeling unit performs target labeling according to the original camera image, surround-view point cloud information and radar prior information to obtain the position of the target of interest and the size of the labeling box, and the joint labeling unit performs corresponding of the labeling results in the surround-view multi-sensor according to the depth estimation matrix and the position relationship between each sensor to obtain the joint labeling results of the camera and the radar.

[0066] Please refer to Figure 4 , Figure 4 is a structural schematic diagram of some embodiments of the sensor data processing apparatus provided according to the present application, as an implementation of the method shown in each of the above figures, the present application also provides some embodiments of a sensor data processing apparatus, these apparatus embodiments correspond to the implementation of some method embodiments shown in Figure 1 , and the apparatus can be applied in various electronic devices.

[0067] As Figure 4As shown, the sensor data processing apparatus of some embodiments includes a first processing module 401, a second processing module 402, and a third processing module 403. The first processing module is configured to obtain radar point cloud data, obtain an image captured by a camera in the same direction as the radar point cloud data at the same timestamp, perform depth estimation on the image to obtain a depth estimation matrix, and label a target in the image by a bounding box. The second processing module is configured to determine the position of the target in the depth estimation matrix according to the depth estimation matrix and the pixel points of the bounding box in the image, and further determine the depth information of the target in the image according to the position of the target in the depth estimation matrix. The third processing module is configured to determine the lateral distance between the radar corresponding to the radar point cloud data and the camera, and determine the value of the distance between the point cloud cluster containing the target and the center of the radar point cloud data according to the lateral distance and the predicted position of the target in the image, wherein the point cloud cluster containing the target includes the radar point cloud data.

[0068] In an optional implementation of some embodiments, the first processing module is configured to perform depth estimation on the image by an FCRN depth estimation method to obtain a depth estimation matrix.

[0069] In an optional implementation of some embodiments, the apparatus further includes a fourth processing module configured to add the value of the distance between the point cloud cluster containing the target and the center of the radar point cloud data to a preset error to obtain the value of the distance between the calibrated point cloud cluster containing the target and the center of the radar point cloud data.

[0070] In an optional implementation of some embodiments, the apparatus further includes a fifth processing module configured to convert the radar point cloud data in at least one direction to the same coordinate system, splice the radar point cloud data converted to the same coordinate system together to obtain the radar point cloud data, combine the radar point cloud data and the image to obtain the motion information of the target, label the radar point cloud data according to the motion information to determine the point cloud cluster containing the target, and determine the preset error according to the point cloud cluster containing the target and the predicted position of the target in the image.

[0071] In an optional implementation of some embodiments, the fifth processing module is configured to save the radar point cloud data as a gif format file, and obtain the motion information of the target according to the gif format file and the image.

[0072] In an optional implementation of some embodiments, the fifth processing module is configured to obtain the motion direction and motion distance of the target according to the gif format file and the image, and further determine the motion information of the target according to the motion direction and motion distance of the target.

[0073] In some optional implementations of embodiments, the second processing module is configured to: determine the transformation relationship between pixels in the image and pixels in the depth estimation matrix based on the size of the image and the size of the depth estimation matrix; and determine the position of the target in the depth estimation matrix based on the transformation relationship and the pixels of the bounding box in the image.

[0074] It is understandable that the modules described in this device are consistent with the reference. Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device and the modules and units contained therein, and will not be repeated here.

[0075] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logic instructions in the memory 530 to execute a sensor data processing method, which includes: acquiring radar point cloud data; acquiring an image taken by a camera in the same direction as the radar point cloud data at the same time stamp; performing depth estimation on the image to obtain a depth estimation matrix; and labeling the target in the image with a bounding box; determining the position of the target in the depth estimation matrix based on the depth estimation matrix and the pixels of the bounding box in the image; further determining the depth information of the target in the image based on the position of the target in the depth estimation matrix; and taking the average value of the depth information to obtain the predicted position of the target in the image; determining the lateral distance between the radar corresponding to the radar point cloud data and the camera; and determining the value of the distance between the point cloud cluster containing the target and the center of the radar point cloud data based on the lateral distance and the predicted position of the target in the image, wherein the point cloud cluster containing the target includes the radar point cloud data.

[0076] In addition, the logic instructions in the memory 530 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or parts of the present application that essentially contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the above-mentioned method of various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0077] In another aspect, the present application also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions that, when executed by a computer, enable the computer to perform a sensor data processing method provided by each of the methods, the method comprising: obtaining radar point cloud data, obtaining an image captured by a camera in the same direction as the radar point cloud data at the same timestamp, performing depth estimation on the image to obtain a depth estimation matrix, and labeling a target in the image by a label box; determining the position of the target in the depth estimation matrix according to the pixel points of the label box in the image and the depth estimation matrix, and further determining the depth information of the target in the image according to the position of the target in the depth estimation matrix, and taking the average of the depth information to obtain the predicted position of the target in the image; determining the lateral distance between the radar corresponding to the radar point cloud data and the camera, and determining the value of the point cloud cluster containing the target from the center of the radar point cloud data according to the lateral distance and the predicted position of the target in the image, wherein the point cloud cluster containing the target includes the radar point cloud data.

[0078] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the sensor data processing method provided above, which comprises: obtaining radar point cloud data, obtaining an image captured by a camera in the same direction as the radar point cloud data at the same timestamp, performing depth estimation on the image to obtain a depth estimation matrix, and labeling a target in the image by a label box; determining the position of the target in the depth estimation matrix according to the pixel points of the label box in the image and the depth estimation matrix, and then determining the depth information of the target in the image according to the position of the target in the depth estimation matrix, and taking the average of the depth information to obtain the predicted position of the target in the image; determining the lateral distance between the radar corresponding to the radar point cloud data and the camera, and determining the value of the point cloud cluster containing the target from the center of the radar point cloud data according to the lateral distance and the predicted position of the target in the image, wherein the point cloud cluster containing the target includes the radar point cloud data.

[0079] The device embodiments described above are merely illustrative, wherein the units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0080] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and necessary general hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the above-mentioned method of each embodiment or some parts of the embodiment.

[0081] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A sensor data processing method, characterized by, The method comprises: acquiring radar point cloud data, acquiring an image at the same timestamp as the radar point cloud data in the same direction of the camera, performing depth estimation on the image to obtain a depth estimation matrix, and labeling a target in the image through a label box; determining the position of the target in the depth estimation matrix according to the depth estimation matrix and the pixel points of the label box in the image, and then determining the depth information of the target in the image according to the position of the target in the depth estimation matrix, and taking the average of the depth information to obtain the predicted position of the target in the image; determining the lateral distance between the radar corresponding to the radar point cloud data and the camera, and determining the value of the point cloud cluster containing the target from the center of the radar point cloud data according to the lateral distance and the predicted position of the target in the image, wherein the point cloud cluster containing the target includes the radar point cloud data.

2. The sensor data processing method of claim 1, wherein, The depth estimation matrix obtained by performing depth estimation on the image comprises: performing depth estimation on the image through an FCRN depth estimation method to obtain a depth estimation matrix.

3. The sensor data processing method of claim 1, wherein, The method further comprises: adding the value of the point cloud cluster containing the target from the center of the radar point cloud data to a preset error to obtain the value of the calibrated point cloud cluster containing the target from the center of the radar point cloud data.

4. The sensor data processing method of claim 3, wherein, The method further comprises: converting the radar point cloud data in at least one direction to the same coordinate system, splicing the radar point cloud data converted to the same coordinate system together to obtain the radar point cloud data, combining the radar point cloud data and the image to obtain the motion information of the target, labeling the radar point cloud data according to the motion information, and determining the point cloud cluster containing the target; determining the preset error according to the point cloud cluster containing the target and the predicted position of the target in the image.

5. The sensor data processing method of claim 4, wherein, Combining the radar point cloud data and the image to obtain the motion information of the target comprises: saving the radar point cloud data as a gif format file; obtaining the motion information of the target according to the gif format file and the image.

6. The sensor data processing method of claim 5, wherein, Obtaining the motion information of the target according to the gif format file and the image comprises: obtaining the motion direction and motion distance of the target according to the gif format file and the image, and then determining the motion information of the target according to the motion direction and motion distance of the target.

7. The sensor data processing method of claim 1, wherein, Determining the position of the target in the depth estimation matrix according to the depth estimation matrix and the pixel points of the label box in the image comprises: determining the conversion relationship between the pixels in the image and the pixels in the depth estimation matrix according to the size of the image and the size of the depth estimation matrix; determining the position of the target in the depth estimation matrix according to the conversion relationship and the pixel points of the label box in the image.

8. A sensor data processing apparatus, characterized by It comprises: a first processing module for acquiring radar point cloud data, acquiring an image at the same timestamp as the radar point cloud data in the same direction of the camera, performing depth estimation on the image to obtain a depth estimation matrix, and labeling a target in the image through a label box; The second processing module is configured to determine the position of the target in the depth estimation matrix according to the depth estimation matrix and the pixel points of the bounding box in the image, and further determine the depth information of the target in the image according to the position of the target in the depth estimation matrix, and obtain the predicted position of the target in the image by averaging the depth information. The third processing module is configured to determine the lateral distance between the radar corresponding to the radar point cloud data and the camera, and determine the value of the point cloud cluster containing the target from the center of the radar point cloud data according to the lateral distance and the predicted position of the target in the image, wherein the point cloud cluster containing the target includes the radar point cloud data.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the sensor data processing method according to any one of claims 1 to 7 when executing the program. 10.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the steps of the sensor data processing method according to any one of claims 1 to 7 when executed by the processor.

Images