Blind Spot Data Processing Method, Device, Computer Equipment, and Storage Medium

Abstract

The present application relates to a blind area data processing method, apparatus, computer device, and storage medium. The method includes: obtaining point cloud data and image data at the synchronization time, registering the point cloud data and the image data in the overlapping area between the area covered by the point cloud data and the area covered by the image data, determining the global mapping relationship between the point cloud data and the image data, filling virtual point cloud data in the detection blind area of the lidar according to the global mapping relationship, and mapping the virtual point cloud data into the image data by using the global mapping relationship, so as to determine the depth information of the detection target based on the global mapping relationship, the virtual point cloud data in the detection blind area of the lidar, and the pixel points of the detection target in the detection blind area of the lidar. In this method, based on the combined mapping of the lidar detection area and the camera detection area, virtual point cloud data in the lidar detection blind area is obtained, reducing the cost of supplementing the lidar blind area.

Description

Technical Field

[0001] This application relates to the technical field of radar, and particularly to a method and apparatus for blind area data processing, a computer device, and a storage medium. Background Art

[0002] With the development of technology, lidar has been widely used in various industries. Detecting parameters such as the distance, speed, azimuth, and attitude of target objects through lidar has become a common means in existing positioning and tracking technologies. Due to the data characteristics of lidar and installation reasons, there is a blind area during the detection process of lidar, which will cause the loss of data information.

[0003] The prior art sets multiple lidars in the target area so that one lidar can cover the blind area of another lidar to achieve the purpose of eliminating the blind area.

[0004] However, the method of eliminating the blind area by mutual coverage of multiple lidars in the prior art has a high cost. Summary of the Invention

[0005] Based on this, in view of the above technical problems, it is necessary to provide a method and apparatus for blind area data processing, a computer device, and a storage medium.

[0006] In a first aspect, a method for blind area data processing is provided, and the method includes:

[0007] Obtain point cloud data and image data at the synchronization time;

[0008] Register the point cloud data and the image data in the overlapping area to determine the global mapping relationship between the point cloud data and the image data; the overlapping area is the overlapping area between the area covered by the point cloud data and the area covered by the image data; the global area is used to represent all the covered areas of the image data, and the covered area of the image data includes the detection blind area of the lidar;

[0009] Fill virtual point cloud data in the detection blind area of the lidar according to the global mapping relationship, and map the virtual point cloud data into the image data by using the global mapping relationship;

[0010] Based on the global mapping relationship, the virtual point cloud data in the detection blind area of the lidar, and the pixel points of the detection target in the detection blind area of the lidar in the image data, determine the depth information of the detection target.

[0011] In one of the embodiments, the registering the point cloud data and the image data in the overlapping area to determine the global mapping relationship between the point cloud data and the image data includes:

[0012] Obtain the internal parameter matrix and the external parameter matrix;

[0013] Establish a global mapping relationship between the coordinates of the point cloud data and the coordinates of the image data in the pixel coordinate system according to the internal parameter matrix and the external parameter matrix.

[0014] In one embodiment, the above method further includes:

[0015] According to the external parameter matrix, convert the coordinates of the point cloud data in the actual world coordinate system into the coordinates of the point cloud data in the camera coordinate system;

[0016] According to the internal parameter matrix, convert the coordinates of the point cloud data in the camera coordinate system into the coordinates of the point cloud data in the pixel coordinate system.

[0017] In one embodiment, the above-mentioned filling of virtual point cloud data in the detection blind area of the lidar and mapping the virtual point cloud data into the image data according to the global mapping relationship includes:

[0018] Taking the center of the lidar as the origin, construct a virtual point cloud with a preset radius step and angle interval;

[0019] Densify the virtual point cloud to obtain the densified virtual point cloud;

[0020] According to the global mapping relationship, map the densified virtual point cloud into the image data to obtain the virtual point cloud data of the detection blind area of the lidar in the image data.

[0021] In one embodiment, the above-mentioned determining the depth information of the detection target based on the global mapping relationship, the virtual point cloud data of the detection blind area of the lidar, and the pixel points of the detection target in the detection blind area of the lidar in the image data includes:

[0022] Based on the image data, determine the target pixel points corresponding to the detection target in the detection blind area of the lidar, and fit two fitting circles that tightly sandwich the target pixel points in the pixel coordinate system;

[0023] Determine the two intersection points of the perpendicular line passing through the target pixel point and the fitting circle;

[0024] According to the global mapping relationship, determine the two virtual point clouds closest to the two intersection points from the virtual point cloud data of the detection blind area of the lidar;

[0025] According to the distances from the two virtual point clouds to the target pixel point and the depth information of the two virtual point clouds in the actual world coordinate system, obtain the depth information of the target pixel point in the actual world coordinate system, and use the depth information as the depth information of the detection target.

[0026] In one embodiment, the above-mentioned obtaining the point cloud data and the image data at the synchronous time includes:

[0027] Obtain the first sampling moment of the point cloud data and the second sampling moment of the image data;

[0028] Calculate the time difference between the first sampling moment and the second sampling moment;

[0029] If the time difference is less than or equal to a preset time deviation threshold, determine that the point cloud data and the image data are data collected at the same synchronized time;

[0030] If the time difference is greater than the preset time deviation threshold, perform a correction operation.

[0031] In one embodiment, the correction operation is the first correction operation or the second correction operation is performed. The first correction operation is to obtain the third sampling moment of the next frame of image data according to a preset frame rate step, and re - execute the step of calculating the time difference between the first sampling moment and the third sampling moment; the second correction operation is to obtain the third sampling moment of the next frame of point cloud data according to a preset frame rate step, and re - execute the step of calculating the time difference between the second sampling moment and the third sampling moment.

[0032] In one embodiment, the above - mentioned obtaining the first sampling moment of the point cloud data and the second sampling moment of the image data includes:

[0033] Determine the first sampling moment of the point cloud data according to the candidate sampling moment on the preset time axis and the first sampling moment deviation; the candidate sampling moment is the sampling moment corresponding to the preset time axis when the camera or radar collects data; the first sampling moment deviation is the time deviation between the radar time axis and the preset time axis;

[0034] Determine the second sampling moment of the image data according to the candidate sampling moment on the preset time axis and the second sampling moment deviation; the second sampling moment deviation is the time deviation between the camera time axis and the preset time axis.

[0035] In one embodiment, the above - mentioned point cloud data is the data obtained by filtering the original point cloud data according to the image range corresponding to the image data.

[0036] In a second aspect, a blind - area data processing device is provided. The device includes:

[0037] A first acquisition module, configured to acquire point cloud data and image data under time synchronization;

[0038] A registration module, configured to register the point cloud data and the image data in the overlapping area, and determine the global mapping relationship between the point cloud data and the image data; the overlapping area is the overlapping area between the area covered by the point cloud data and the area covered by the image data; the global area is used to represent all the covered areas of the image data, and the covered area of the image data includes the detection blind area of the lidar;

[0039] A filling module, configured to fill virtual point cloud data in a detection blind area of a lidar according to a global mapping relationship, and map the virtual point cloud data into image data by using the global mapping relationship;

[0040] A determination module, configured to determine depth information of a detection target based on the global mapping relationship, the virtual point cloud data of the detection blind area of the lidar, and the pixel points of the detection target in the detection blind area of the lidar in the image data.

[0041] In a third aspect, a computer device is provided, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the blind area data processing method described in any one of the first aspects above is implemented.

[0042] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the blind area data processing method described in any one of the first aspects above is implemented.

[0043] For the above blind area data processing method, device, computer device and storage medium, the server obtains point cloud data and image data at the synchronization time, registers the point cloud data and the image data in the overlapping area between the area covered by the point cloud data and the area covered by the image data, determines the global mapping relationship between the point cloud data and the image data, fills virtual point cloud data in the detection blind area of the lidar according to the global mapping relationship, and maps the virtual point cloud data into the image data by using the global mapping relationship, so as to determine the depth information of the detection target based on the global mapping relationship, the virtual point cloud data of the detection blind area of the lidar, and the pixel points of the detection target in the detection blind area of the lidar in the image data. In this method, the server constructs a global mapping relationship of the area covered by the image data including the detection blind area of the lidar according to the point cloud data and the image data in the overlapping area, and supplements the virtual point cloud data of the detection blind area of the lidar based on the global mapping relationship, so that when a detection target appears in the image data, the depth information of the detection target in the detection blind area of the lidar can be determined according to the global mapping relationship and the virtual point cloud data of the blind area, and the situation of target loss caused by the detection target being in the detection blind area of the lidar will not occur. Moreover, this method is based on the combined mapping of the lidar detection area and the camera detection area to obtain the virtual point cloud data of the lidar detection blind area, and does not require multiple lidars to supplement the blind area, reducing the cost of blind area supplementation of the lidar. Description of the Drawings

[0044] Figure 1 It is an application environment diagram of the blind area data processing method in an embodiment;

[0045] Figure 2Schematic flowchart of the blind area data processing method in an embodiment;

[0046] Figure 3 Schematic flowchart of the blind area data processing method in an embodiment;

[0047] Figure 3a Schematic diagram of the distribution of the image plane point cloud in the overlapping area in an embodiment;

[0048] Figure 4 Schematic flowchart of the blind area data processing method in an embodiment;

[0049] Figure 4a Schematic diagram of the distribution of the image plane point cloud in the overlapping area after densification processing in an embodiment;

[0050] Figure 4b Schematic diagram of the mapping between the image plane point cloud and the image data in the overlapping area after densification processing in an embodiment;

[0051] Figure 5 Schematic flowchart of the blind area data processing method in an embodiment;

[0052] Figure 5a Schematic diagram of the original point cloud data distribution in an embodiment;

[0053] Figure 5b Schematic diagram of the point cloud data distribution in an embodiment;

[0054] Figure 5c Schematic diagram of the distribution of the image plane point cloud after filling the blind area with virtual point cloud in an embodiment;

[0055] Figure 6 Schematic flowchart of the blind area data processing method in an embodiment;

[0056] Figure 6a Schematic diagram of the fitting circle model in an embodiment;

[0057] Figure 7 Schematic flowchart of the blind area data processing method in an embodiment;

[0058] Figure 8 Schematic flowchart of the blind area data processing method in an embodiment;

[0059] Figure 9 Schematic flowchart of the blind area data processing method in an embodiment;

[0060] Figure 10 Structure block diagram of the blind area data processing device in an embodiment;

[0061] Figure 11It is a structural block diagram of a blind area data processing device in an embodiment;

[0062] Figure 12 It is an internal structure diagram of a computer device in an embodiment. Specific implementation manners

[0063] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] The blind area data processing method provided by the present application can be applied to an application environment as Figure 1 shown. Among them, the server 101 communicates with the radar 102 and the image acquisition device 103 through a network. Among them, the server 101 can be an independent server or a server cluster composed of multiple servers; the radar 102 is any kind of lidar; the image acquisition device 103 is any kind of high-definition image acquisition device, for example, a high-definition camera. It should be noted that when installing the image acquisition device 103 and the radar 102, the image acquisition device is installed in the detection blind area of the radar to achieve the purpose of acquiring image data in the radar blind area.

[0065] Next, the technical solution of the present application and how the technical solution of the present application solves the above technical problems will be specifically described through embodiments in combination with the accompanying drawings. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. It should be noted that for the blind area data processing method provided by the embodiments of the present application, the execution subject is the server 101, or it can also be a blind area data processing device, and this blind area data processing device can become part or all of the server 101 through software, hardware or a combination of software and hardware. In the following method embodiments, the execution subject is the server 101 as an example for description. Figures 2 - 9 In one embodiment, as

[0066] shown, a blind area data processing method is provided, which involves the server obtaining point cloud data and image data at the synchronous time, filling virtual point cloud data in the detection blind area of the lidar based on the global mapping relationship between the point cloud data and the image data, and thus determining the depth information of the detection target according to the global mapping relationship, the virtual point cloud data in the detection blind area of the lidar, and the pixel points of the detection target in the image data in the detection blind area of the lidar. The process includes the following steps: Figure 2

[0067]

[0067] S201. Obtain point cloud data and image data at time synchronization.

[0068] Among them, the point cloud data is the data after processing the original point cloud data. For example, the processing of the original point cloud data includes removing non-ground points, removing points outside the image range, removing miscellaneous points, etc. The point cloud data is the data collected by the lidar; the image data is the data collected by the image acquisition device. It should be noted that the detection area of the image acquisition device covers the detection blind area of the lidar.

[0069] Optionally, in one embodiment, the point cloud data is the data obtained by filtering the original point cloud data according to the image range corresponding to the image data.

[0070] Among them, the original point cloud data refers to all the point cloud data within the detection range collected by the lidar. In this embodiment, the point cloud data obtained by the server refers to the point cloud data that is filtered to remove the point cloud outside the detection range of the image acquisition device and retained according to the detection range of the image acquisition device. Optionally, when the server filters the original point cloud data, it can also delete and filter the point cloud data corresponding to the points that are not received successfully, non-ground points, noise points, and miscellaneous points.

[0071] In this embodiment, to obtain the corresponding point cloud data and image data, the server needs to obtain the two types of data at the same synchronization time for filling the point cloud in the radar blind area. Optionally, the server can adjust the sampling frequencies of the image acquisition device and the radar so that they collect data in the same phase. On the premise of the same phase, the server obtains the point cloud data and image data at the corresponding moment for supplementing the point cloud in the radar blind area. For example, the server can obtain the point cloud data and image data at the same moment in the same phase; or, the server can obtain the point cloud data and image data at a certain moment within the preset time deviation range. This embodiment does not make a limitation on this.

[0072] S202. Register the point cloud data and the image data in the overlapping area to determine the global mapping relationship between the point cloud data and the image data; the overlapping area is the overlapping area between the area covered by the point cloud data and the area covered by the image data; the global area is used to represent all the covered areas of the image data, and the covered area of the image data includes the detection blind area of the lidar.

[0073] Among them, the mapping relationship refers to the corresponding relationship between the point cloud points in the point cloud data and the pixel points in the image data. In this embodiment, the global area refers to the area covered by the image data, which includes the area covered by the point cloud data and the detection blind area of the lidar.

[0074] In this embodiment, the server can determine the registration method of the camera and the lidar according to the acquired image data and point cloud data. For example, the server can analyze and register the point cloud data and image data in the overlapping area to obtain the internal parameter matrix and external parameter matrix corresponding to the camera and the lidar. The internal parameter matrix and external parameter matrix can represent the corresponding relationship between the coordinates of the point cloud data in the pixel coordinate system and the coordinates of the pixel points in the image data. Specifically, according to the internal parameter matrix and external parameter matrix, the three-dimensional coordinates of the point cloud data can be converted into two-dimensional coordinates in the pixel coordinate system, so as to realize the corresponding relationship between the coordinates of the point cloud data in the pixel coordinate system and the coordinates of the pixel points in the image data. This embodiment does not make any limitations on this.

[0075] S203. Fill the detection blind area of the lidar with virtual point cloud data according to the global mapping relationship, and map the virtual point cloud data to the image data by using the global mapping relationship.

[0076] The virtual point cloud data refers to the point cloud data constructed according to the point cloud data collected by the lidar and the global mapping relationship, rather than the real point cloud data collected by the radar.

[0077] In this embodiment, optionally, the server can fill the detection blind area of the lidar with virtual point cloud data according to the global mapping relationship and a preset filling algorithm, so as to map the virtual data filled in the detection blind area of the lidar to the image plane point cloud to obtain the plane point cloud corresponding to the filled virtual point cloud data. For example, the server can fill the virtual point cloud data in the detection blind area of the lidar according to the distribution characteristics and presented shape of the point cloud data detected by the lidar. For example, the detection blind area of the lidar is a fan-shaped area centered on the lidar. According to the distribution characteristics of the point cloud data, it is determined that the center point of the detection blind area is the center of the lidar. Taking this center point as the origin, with the determined radius step and angle interval, the virtual point cloud data of the detection blind area of the lidar is constructed according to the global mapping relationship, and then the constructed virtual point cloud data is mapped to the plane image to obtain the plane point cloud data corresponding to the virtual point cloud data. This embodiment does not make any limitations on this.

[0078] S204. Determine the depth information of the detection target based on the global mapping relationship, the virtual point cloud data in the detection blind area of the lidar, and the pixel points of the detection target in the image data in the detection blind area of the lidar.

[0079] In this embodiment, when the detection target appears in the detection blind area of the lidar, the lidar cannot obtain the depth information of the detection target. Therefore, this embodiment combines image data to implement the detection of the detection target in the detection blind area of the lidar. For example, after the server constructs the global mapping relationship between the image data and the point cloud data, when a detection target appears in the area corresponding to the detection blind area of the lidar in the image data, the server can obtain the target pixel points of the detection target in the image data, and then determine the virtual point cloud data corresponding to the target pixel points according to the global mapping relationship, and then determine the depth information corresponding to the detection target according to the preset prediction algorithm for determining the depth information of the detection target based on the virtual point cloud data corresponding to the target pixel points. Optionally, the server can perform linear fitting according to the distribution of the virtual point cloud data, and determine the depth information corresponding to the target pixel points, that is, the depth information of the detection target, according to the linear model obtained by fitting and the coordinates of the virtual point cloud data. Optionally, the server can also construct a deep learning model according to the characteristics of the virtual point cloud data, use the target pixel points as the input, and output its corresponding three-dimensional coordinates through model prediction as the depth information of the target pixel points. This embodiment does not limit this.

[0080] In the above blind area data processing method, the server obtains the point cloud data and the image data at the same synchronization time, registers the point cloud data and the image data in the overlapping area between the area covered by the point cloud data and the area covered by the image data, determines the global mapping relationship between the point cloud data and the image data, fills the virtual point cloud data in the detection blind area of the lidar according to the global mapping relationship, and maps the virtual point cloud data to the image data by using the global mapping relationship, so as to determine the depth information of the detection target based on the global mapping relationship, the virtual point cloud data in the detection blind area of the lidar, and the pixel points of the detection target in the image data in the detection blind area of the lidar. In this method, the server constructs the global mapping relationship of the area covered by the image data including the detection blind area of the lidar according to the point cloud data and the image data in the overlapping area, and supplements the virtual point cloud data in the detection blind area of the lidar based on the global mapping relationship, so that when a detection target appears through the image data, the depth information of the detection target in the detection blind area of the lidar can be determined according to the global mapping relationship and the virtual point cloud data of the blind area, and the situation of target loss caused by the detection target being in the detection blind area of the lidar will not occur. Moreover, this method obtains the virtual point cloud data of the detection blind area of the lidar based on the combined mapping of the lidar detection area and the camera detection area, and does not need to supplement the blind area through multiple lidars, reducing the cost of blind area supplement of the lidar.

[0081] Since the point cloud data and the image data are in different coordinate systems, when the server establishes the mapping relationship between the two, it needs to determine the transformation matrix of the coordinate system of the point cloud data according to the preset calibration and registration method, so as to establish the mapping relationship between the point cloud data and the image data in the overlapping area. In one embodiment, as Figure 3 shown, the above registration of the point cloud data and the image data in the overlapping area to determine the global mapping relationship between the point cloud data and the image data includes:

[0082] S301. Obtain the internal parameter matrix and the external parameter matrix.

[0083] Among them, the internal parameter matrix and the external parameter matrix can be obtained through calibration.

[0084] In this embodiment, the server can obtain the internal parameter matrix and the external parameter matrix through the following calibration process. Specifically, it includes: by selecting a reference coordinate system, measuring and calculating the 6 degrees of freedom of rotation and translation of the radar and the camera relative to the reference coordinate system respectively. The coordinate system corresponding to the radar refers to the world coordinate system of the radar. Among them, the 6 degrees of freedom of rotation and translation refer to the external parameter matrix, including parameters such as the translation vector and the rotation matrix. The server determines the internal parameter matrix required for the conversion between the camera coordinate system and the pixel coordinate system according to the preset calibration method. The internal parameter matrix includes parameters such as internal parameters, lateral distortion coefficients, and tangential distortion coefficients.

[0085] S302. According to the internal parameter matrix and the external parameter matrix, establish the global mapping relationship between the coordinates of the point cloud data and the coordinates of the image data in the pixel coordinate system.

[0086] Among them, the external parameter matrix is used to convert the coordinates of the point cloud data in the world coordinate system to the coordinates in the camera coordinate system; the internal parameter matrix is used to convert the coordinates of the point cloud data in the camera coordinate system to the coordinates in the pixel coordinate system.

[0087] In this embodiment, the server converts the coordinates of the point cloud data in the world coordinate system to the coordinates in the camera coordinate system according to the external parameter matrix, and converts the coordinates of the point cloud data in the camera coordinate system to the coordinates in the pixel coordinate system according to the internal parameter matrix and the distortion coefficients, so as to obtain the point cloud data in the pixel coordinate system, that is, map the point cloud data to the plane where the image data is located to obtain the point cloud on the image plane. The schematic diagram of the point cloud on the image plane is as Figure 3a shown. In this embodiment, the server determines the registration method according to the image data and the point cloud data, and determines the internal parameter matrix and the external parameter matrix according to the registration method. The internal parameter matrix and the external parameter matrix characterize the corresponding relationship between the image data and the point cloud data. Optionally, this corresponding relationship can be the global mapping relationship between the point cloud data and the image data in all the coverage areas of the camera, that is, including the blind area detected by the radar. Exemplarily, the mapping relationship can be expressed as {M video -M lidar}, this embodiment does not limit this.

[0088] In this embodiment, the server determines the calibration method of the camera and the lidar according to the point cloud data and the image data in the overlapping area of the lidar and the camera image, determines the internal parameter matrix and the external parameter matrix according to the calibration method, and thus determines the mapping relationship between the point cloud data and the image data in the entire domain of the camera image according to the internal and external parameter matrices, so that the server can complete the supplement of the virtual point cloud data in the lidar detection blind area corresponding to the entire domain of the subsequent camera image according to this mapping relationship.

[0089] Optionally, when mapping the coordinates of the point cloud data in the overlapping area of the server to the coordinates of the image data, the coordinate system of the point cloud data can be converted according to the determined transformation matrix. In one embodiment, as Figure 4 shown, this method further includes:

[0090] S401. According to the external parameter matrix, convert the coordinates of the point cloud data in the overlapping area in the actual world coordinate system to the coordinates of the point cloud data in the camera coordinate system.

[0091] Among them, the external parameter matrix includes a rotation matrix and a translation vector.

[0092] In this embodiment, the server can convert the coordinates of the original point cloud data to the corresponding coordinates in the camera coordinate system according to the external parameter matrix. The expression is as follows:

[0093]

[0094] Among them, (X, Y, Z) represents the coordinates of the point cloud data in the world coordinate system; (x, y, z) represents the coordinates of the point cloud data in the camera coordinate system; R represents the rotation matrix; t represents the translation vector.

[0095] S402. According to the internal parameter matrix, convert the coordinates of the point cloud data in the overlapping area in the camera coordinate system to the coordinates of the point cloud data in the pixel coordinate system.

[0096] Among them, the internal parameters include f x , f y , c x , c y . In this embodiment, after converting to obtain the coordinates of the point cloud data in the camera coordinate system, the server can convert the coordinates of the point cloud data in the camera coordinate system to the coordinates in the pixel coordinate system according to the following formula. The process of converting the coordinates of the point cloud data in the camera coordinate system to the coordinates in the pixel coordinate system is as follows:

[0097] x′ = x / z

[0098] y′ = y / z

[0099] r 2 = x' 2 + y' 2

[0100]

[0101]

[0102] u = f x * x″ + c x

[0103] v = f y * y″ + c y

[0104] Where: (u, v) represents the coordinates of the point cloud data in the pixel coordinate system; (x, y, z) represents the coordinates of the point cloud data in the camera coordinate system; f x , f y , c x , c y represents the camera internal parameters; k1, k2, k3, k4, k5, k6 are the radial distortion coefficients; p1, p2 are the tangential distortion coefficients; r is the distance from this point to the origin of the image coordinate system.

[0105] In this embodiment, the server can convert the coordinates of the point cloud data in the world coordinate system into the coordinates in the pixel coordinate system according to the external parameter matrix and the internal parameter matrix, that is, map the point cloud corresponding to the point cloud data to the plane image through the mapping transformation to obtain the plane point cloud, and establish the mapping relationship between the plane point cloud in the camera coverage area and the image data, laying a foundation for realizing the virtual point cloud data supplement in the detection blind area of the lidar.

[0106] In this embodiment, the server densifies the point cloud data in the overlapping area to expand the sample size of the point cloud data. When filling the virtual point cloud data in the non-overlapping area according to the point cloud data in the overlapping area, due to the increase in the sample size, the accuracy and accuracy of the virtual point cloud data filling are improved.

[0107] The server constructs virtual point cloud data in the non-overlapping area according to the known point cloud data. In one embodiment, as Figure 5 shown, filling the virtual point cloud data in the detection blind area of the lidar according to the global mapping relationship and mapping the virtual point cloud data to the image data by using the global mapping relationship includes:

[0108] S501. Taking the center of the lidar as the origin, constructing a virtual point cloud with a preset radius step and angle interval.

[0109] Among them, the preset radius step size and angle interval are determined according to the actual detection area of the lidar.

[0110] In this embodiment, since the ideal detection area of the lidar is 360 degrees, the distribution characteristics of the point cloud data detected by it present a concentric circle distribution characteristic, that is, with the lidar as the origin and different radii, presenting a distribution characteristic in circles. The original point cloud data distribution diagram is as Figure 5a shown. The server takes the center of the lidar as the origin, and constructs virtual point clouds with the determined radius step size and the determined angle interval respectively. Taking the above example as an illustration, the server can determine the radius step size according to the value range of the radius (0, 5m). For example, the radius step size can be taken as 0.2m, and determine the angle interval according to the value range of the angle (45°, 135°). For example, the angle interval can be taken as 5°. Taking the target center of the non-overlapping area as the origin, with a radius step size of 0.2m and an angle interval of 5°, virtual point cloud data sets with radii of 5m, 4.8m, 4.6m and angles of 45°, 50° can be established respectively. This embodiment does not make any limitations on this.

[0111] S502. Perform densification processing on the virtual point cloud to obtain the densified virtual point cloud.

[0112] Among them, densification processing refers to expanding the sparse point cloud data. Optionally, the server can perform densification processing by converting the coordinates of the point cloud data in the actual world coordinate system into coordinates in the polar coordinate system.

[0113] In this embodiment, for example, according to the conversion relationship between the world coordinate system and the polar coordinate system, the server converts the coordinates of the point cloud data into coordinates in the polar coordinate system. Optionally, the server can first obtain the top view point (X, Y, 0) of the three-dimensional point (X, Y, Z) of the virtual point cloud. On the Z = 0 plane, convert the virtual point cloud points (X, Y) into the polar coordinate system and calculate the polar angle of the point cloud data corresponding to each point. Since the lidar has different beam bundles, specifically, the server can add filling points by equally dividing the polar length for two points with adjacent beam bundles and equal polar angles. The schematic diagram of the point cloud data before densification processing is as Figure 3a shown, and the schematic diagram of the point cloud data after densification processing is as Figure 4a shown. Optionally, in order to further increase the data volume of the virtual point cloud, the server can also fit the discrete virtual point clouds on the same radius into a curve by constructing point cloud lines. This embodiment does not make any limitations on this.

[0114] S503. According to the global mapping relationship, map the densified virtual point cloud into the image data to obtain the virtual point cloud data of the detection blind area of the lidar in the image data.

[0115] In this embodiment, after the server constructs the virtual point cloud of the detection blind area of the lidar, it densifies the constructed virtual point cloud to increase the data volume of the virtual point cloud. Then, according to the global mapping relationship, the virtual point cloud of the lidar detection blind area is mapped to the image plane point cloud to obtain the virtual point cloud data of the lidar detection blind area in the image data. The schematic diagram of the point cloud data of the plane image after filling the virtual point cloud data in the lidar detection blind area is shown in Figure 5b as follows.

[0116] In this embodiment, the server constructs the virtual point cloud data of the lidar detection blind area according to the global mapping relationship. When obtaining the depth information of any coordinate in the lidar detection blind area, the virtual point cloud data provides a role in calculation and reference.

[0117] When supplementing the virtual point cloud data in the lidar detection blind area, the depth information of the blind area can be supplemented by constructing a mathematical model. In one embodiment, as shown in Figure 6 the above method for determining the depth information of the detection target based on the global mapping relationship, the virtual point cloud data of the lidar detection blind area, and the pixel points of the detection target in the lidar detection blind area in the image data includes:

[0118] S601. Based on the image data, determine the target pixel points corresponding to the detection target in the lidar detection blind area, and fit two fitting circles that tightly sandwich the target pixel points in the pixel coordinate system.

[0119] Among them, the fitting circle refers to a fitting circle model obtained by linearly fitting the target pixel points corresponding to the virtual point cloud data in the lidar detection blind area. The fitting circle model diagram can be referred to in Figure 6a as follows.

[0120] In this embodiment, when the server detects a detection target in the image data area corresponding to the lidar detection blind area, it determines the target pixel points (x0, y0) corresponding to the detection target. According to the coordinate position of the target pixel points and the parameter information of the fitting circle, the server can determine the two fitting circle models closest to the target pixel points (x0, y0). Optionally, the closest distance can be two fitting circle models that tightly sandwich the target pixel points up and down, which are C1 and C2 respectively.

[0121] S602. Determine the two intersection points of the perpendicular line passing through the target pixel points and the fitting circle.

[0122] In this embodiment, optionally, the server passes a perpendicular line x = x0 perpendicular to the X-axis through the target pixel point (x0, y0). At this time, the perpendicular line x = x0 intersects with multiple fitting circle models, and the intersections with the two fitting circles C1 and C2 are obtained, and the intersection point 1 (x0, y1) corresponding to C1 and the intersection point 2 (x0, y2) corresponding to C2 are obtained.

[0123] S603. Determine two virtual point clouds closest to the two intersections from the virtual point cloud data in the detection blind area of the lidar according to the global mapping relationship.

[0124] In this embodiment, the server determines two virtual point clouds closest to the intersection point 1 (x0, y1) and the intersection point 2 (x0, y2) from the point cloud data in the detection blind area of the lidar according to the global mapping relationship, and obtains the virtual point cloud 1 (X1, Y1, Z1) and the virtual point cloud 2 (X2, Y2, Z2).

[0125] S604. Obtain the depth information of the target pixel point in the actual world coordinate system according to the distances from the two virtual point clouds to the target pixel point and the depth information of the two virtual point clouds in the actual world coordinate system, and use the depth information as the depth information of the detection target.

[0126] In this embodiment, the server performs an equal ratio calculation on the virtual point cloud 1 (X1, Y1, Z1) and the virtual point cloud (X2, Y2, Z2) according to the distance ratio between the target pixel point (x0, y0) and the intersection point 1 (x0, y1) and the intersection point 2 (x0, y2), determines the three-dimensional coordinates corresponding to (x0, y0), and obtains the depth information of the target pixel point in the world coordinate system, that is, determines the depth information of the detection target.

[0127] In this embodiment, when the server detects a detection target in the image area corresponding to the detection blind area of the lidar based on the image data, the depth information of the detection target in the actual world coordinate can be determined according to the pixel points of the detection target in the image data, the global mapping relationship between the image data and the point cloud data, and the virtual point cloud in the detection blind area of the lidar. That is, by using the image data and the mapping relationship between the image data and the point cloud data, the data supplementary detection of the detection target in the detection blind area of the lidar is realized. The supplementary process is simple, and it is not necessary to use multiple radars to cover and supplement the detection blind area, which simplifies the process of supplementing the point cloud data in the detection blind area of the lidar and reduces the cost of supplementing the point cloud in the detection blind area of the lidar.

[0128] The server can directly obtain the sampling moments of the radar and the image acquisition device, but the sampling moments of the two may not be sampling moments under time synchronization. In one embodiment, as Figure 7 shown, the obtaining of the point cloud data and the image data under the synchronous time includes:

[0129] S701. Obtain the first sampling moment of the point cloud data and the second sampling moment of the image data.

[0130] Among them, the first sampling moment refers to the sampling moment corresponding to the radar collecting the current point cloud data; the second sampling moment refers to the sampling moment corresponding to the image acquisition device collecting the current image data.

[0131] In this embodiment, the server can obtain the first sampling moment t1 corresponding to the point cloud data while obtaining the point cloud data. Similarly, the server can obtain the second sampling moment t2 corresponding to the current image data while obtaining the image data; Optionally, the server can also obtain the first sampling moment t1 corresponding to the current point cloud data and the second sampling moment t2 corresponding to the current image data from the radar and the image acquisition device respectively after receiving the point cloud data and the image data. This embodiment does not make a limitation on this.

[0132] S702. Calculate the time difference between the first sampling moment and the second sampling moment.

[0133] In this embodiment, the server calculates the time difference between the first sampling moment t1 and the second sampling moment t2. Optionally, the server can use the absolute value of the calculation result as the value of the time difference, that is, t c =|t1 - t2|. This embodiment does not make a limitation on this.

[0134] S703. If the time difference is less than or equal to the preset time deviation threshold, determine that the point cloud data and the image data are collected at the same synchronized time.

[0135] Among them, the preset time deviation threshold refers to the time deviation threshold determined according to the actual scenario, the radar accuracy, and the accuracy of the image acquisition device.

[0136] In this embodiment, the server determines the time deviation threshold as δ according to the actual on-site situation and the data accuracy of the device. When t c ≤δ, the server determines that the point cloud data and the image data are collected at the same synchronized time.

[0137] S704. If the time difference is greater than the preset time deviation threshold, perform a correction operation.

[0138] Among them, the preset frame rate step refers to the frame rate step determined according to the sampling frequencies of the radar and the image acquisition device.

[0139] In this embodiment, the server determines that t c> δ, the server determines that the point cloud data and the image data are not the data collected at the synchronous time. The server can perform a correction operation to correct the above time synchronization process. Optionally, the correction operation can be a first correction operation or a second correction operation; the first correction operation is to obtain the third sampling moment of the next frame of image data according to a preset frame rate step, and re - execute the step of calculating the time difference between the first sampling moment and the third sampling moment. For example, the server can calculate the time difference between the third sampling moment and the first sampling moment by obtaining the third sampling moment of the next frame of image data, so as to compare this time difference with a preset time deviation threshold to determine whether the third sampling moment is the synchronous moment with the first sampling moment; if the third moment is still not, then continue to obtain the image data of the fourth sampling moment, and so on, until the image data corresponding to the sampling moment that is time - synchronous with the first sampling moment is obtained. Optionally, the second correction operation is to obtain the third sampling moment of the next frame of point cloud data according to a preset frame rate step, and re - execute the step of calculating the time difference between the second sampling moment and the third sampling moment. For example, the server can calculate the time difference between the third sampling moment and the second sampling moment by obtaining the third sampling moment of the next frame of point cloud data, so as to compare this time difference with a preset time deviation threshold to determine whether the third sampling moment is the synchronous moment with the second sampling moment; if the third moment is still not, then continue to obtain the fourth sampling moment of the point cloud data, and so on, until the point cloud data corresponding to the sampling moment that is time - synchronous with the second sampling moment is obtained. This embodiment does not make a limitation on this.

[0140] In this embodiment, the server determines whether the data is synchronous data according to the sampling moments of the point cloud data and the image data, which ensures the correspondence between the point cloud data and the image data to a certain extent.

[0141] If the server cannot directly collect the sampling moments of the radar or the image acquisition device, in one embodiment, as Figure 8 shown, the obtaining of the first sampling moment of the point cloud data and the second sampling moment of the image data includes:

[0142] S801. Determine the first sampling moment of the point cloud data according to the candidate sampling moment on the preset time axis and the first sampling moment deviation; the candidate sampling moment is the sampling moment corresponding to the preset time axis when the camera or radar collects data; the first sampling moment deviation is the time deviation between the radar time axis and the preset time axis.

[0143] Among them, the preset time axis refers to a pre-set reference time axis. For example, this reference time axis can be the time axis of the server. At this time, the candidate sampling moment is the sampling moment of the server itself obtained by the server, that is, the sampling moment based on the server time axis. The first sampling moment deviation refers to the moment deviation between the server time axis and the radar time axis determined according to the actual environment.

[0144] In this embodiment, optionally, set the candidate sampling moment of the server time axis as t, and the first sampling moment deviation as Δt1. The server determines the first sampling moment t1' of the point cloud data according to the candidate sampling moment t and the first sampling moment deviation Δt1, where t1' = |t - Δt1|.

[0145] S802. Determine the second sampling moment of the image data according to the candidate sampling moment on the preset time axis and the second sampling moment deviation; the second sampling moment deviation is the time deviation between the camera time axis and the preset time axis.

[0146] Among them, the second sampling moment deviation refers to the moment deviation between the server time axis and the image acquisition device time axis determined according to the actual environment.

[0147] In this embodiment, optionally, set the candidate sampling moment of the server time axis as t, and the first sampling moment deviation as Δt2. The server determines the first sampling moment t2' of the point cloud data according to the candidate sampling moment t and the first sampling moment deviation Δt2, where t2' = |t - Δt2|.

[0148] In this embodiment, if the server cannot directly obtain the sampling moments of the radar and / or the image acquisition device, it can determine their respective acquisition moments through the candidate sampling moment based on the reference time axis and their respective corresponding moment deviations, so as to determine the point cloud data and the image data under time synchronization and realize the corresponding relationship between the point cloud data and the image data.

[0149] To better illustrate the above method, as Figure 9 shown, this embodiment provides a blind area data processing method, which specifically includes:

[0150] S101. Obtain the first sampling moment of the point cloud data and the second sampling moment of the image data;

[0151] S102. Calculate the time difference between the first sampling moment and the second sampling moment;

[0152] S103. If the time difference is less than or equal to the preset time deviation threshold, determine that the point cloud data and the image data are the data collected under the synchronous time;

[0153] S104. Register the point cloud data and the image data in the overlapping area to obtain the internal parameter matrix and the external parameter matrix;

[0154] S105. Establish a global mapping relationship between the coordinates of the point cloud data and the coordinates of the image data in the pixel coordinate system according to the internal reference matrix and the external reference matrix;

[0155] S106. Construct a virtual point cloud with the center of the lidar as the origin and a preset radius step and angle interval;

[0156] S107. Densify the virtual point cloud to obtain the densified virtual point cloud;

[0157] S108. Map the densified virtual point cloud into the image data according to the global mapping relationship to obtain the virtual point cloud data of the detection blind area of the lidar in the image data;

[0158] S109. Determine the depth information of the detection target based on the global mapping relationship, the virtual point cloud data of the detection blind area of the lidar, and the pixel points of the detection target in the detection blind area of the lidar in the image data.

[0159] In this embodiment, the server supplements the depth information of the point cloud in the radar blind area based on the point cloud data and the image data under time synchronization, and realizes the complete sharing of target information between the non-blind area and the blind area, without causing the loss of the target. Moreover, this method does not require multiple radars to supplement the blind area, reducing the cost of blind area supplement.

[0160] The blind area data processing method provided in the above embodiment has a similar implementation principle and technical effect to the above method embodiment, and will not be elaborated here.

[0161] It should be understood that although Figures 2 - 9 the steps in the flowchart are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, Figures 2 - 9 at least a part of the steps in

[0162] In one embodiment, as Figure 10 shown, a blind area data processing device is provided, including: a first acquisition module 01, an establishment module 02, a filling module 03, and a second acquisition module 04, where:

[0163] The first acquisition module 01 is configured to acquire point cloud data and image data under time synchronization;

[0164] The establishment module 02 is configured to register the point cloud data and the image data in the overlapping area to determine the global mapping relationship between the point cloud data and the image data; the overlapping area is the overlapping area between the area covered by the point cloud data and the area covered by the image data; the global area is used to represent all the covered areas of the image data, and the covered area of the image data includes the detection blind area of the lidar;

[0165] The filling module 03 is configured to fill virtual point cloud data in the detection blind area of the lidar according to the global mapping relationship, and map the virtual point cloud data into the image data by using the global mapping relationship;

[0166] The second acquisition module 04 is configured to determine the depth information of the detection target based on the global mapping relationship, the virtual point cloud data in the detection blind area of the lidar, and the pixel points of the detection target in the detection blind area of the lidar in the image data.

[0167] In one embodiment, the establishment module 02 is specifically configured to acquire the internal parameter matrix and the external parameter matrix; according to the internal parameter matrix and the external parameter matrix, establish the global mapping relationship between the coordinates of the point cloud data and the coordinates of the image data in the pixel coordinate system.

[0168] In one embodiment, as Figure 11 shown, the above-mentioned blind area data processing device further includes a conversion module 05, configured to convert the coordinates of the point cloud data in the actual world coordinate system into the coordinates of the point cloud data in the camera coordinate system according to the external parameter matrix; according to the internal parameter matrix, convert the coordinates of the point cloud data in the camera coordinate system into the coordinates of the point cloud data in the pixel coordinate system.

[0169] In one embodiment, the filling module 03 is specifically configured to construct a virtual point cloud with the center of the lidar as the origin, with a preset radius step and an angular interval; perform densification processing on the virtual point cloud to obtain the densified virtual point cloud; according to the global mapping relationship, map the densified virtual point cloud into the image data to obtain the virtual point cloud data of the detection blind area of the lidar in the image data.

[0170] In one embodiment, the second acquisition module 04 is specifically configured to, based on the image data, determine the target pixel points corresponding to the detection targets in the detection blind area of the lidar, and fit two fitting circles that tightly sandwich the target pixel points in the pixel coordinate system; determine the two intersection points of the perpendicular line passing through the target pixel points and the fitting circles; according to the global mapping relationship, determine the two virtual point clouds closest to the two intersection points from the virtual point cloud data in the detection blind area of the lidar; obtain the depth information of the target pixel points in the actual world coordinate system based on the distances from the two virtual point clouds to the target pixel points and the depth information of the two virtual point clouds in the actual world coordinate system, and use the depth information as the depth information of the detection targets.

[0171] In one embodiment, the first acquisition module 01 is specifically configured to calculate the time difference between the first sampling moment and the second sampling moment; if the time difference is less than or equal to a preset time deviation threshold, determine that the point cloud data and the image data are the data collected at the synchronous time; if the time difference is greater than the preset time deviation threshold, perform a correction operation.

[0172] In one embodiment, the correction operation is a first correction operation or a second correction operation. The first correction operation is to obtain the third sampling moment of the next frame of image data according to a preset frame rate step, and re-execute the step of calculating the time difference between the first sampling moment and the third sampling moment; the second correction operation is to obtain the third sampling moment of the next frame of point cloud data according to a preset frame rate step, and re-execute the step of calculating the time difference between the second sampling moment and the third sampling moment.

[0173] In one embodiment, the first acquisition module 01 is specifically configured to determine the first sampling moment of the point cloud data according to the candidate sampling moment on the preset time axis and the first sampling moment deviation; the candidate sampling moment is the sampling moment corresponding to the preset time axis when the camera or the radar collects data; the first sampling moment deviation is the time deviation between the radar time axis and the preset time axis; determine the second sampling moment of the image data according to the candidate sampling moment on the preset time axis and the second sampling moment deviation; the second sampling moment deviation is the time deviation between the camera time axis and the preset time axis.

[0174] In one embodiment, the point cloud data is the data obtained by filtering the original point cloud data according to the image range corresponding to the image data.

[0175] For the specific limitations of the blind area data processing device, reference may be made to the limitations of the blind area data processing method in the foregoing text, which will not be elaborated herein. Each module in the above-mentioned blind area data processing device can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so as to facilitate the processor to call and execute the operations corresponding to each of the above modules.

[0176] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as Figure 12 shown. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a carrier network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a blind area data processing method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, a touchpad, or a mouse, etc.

[0177] Those skilled in the art can understand that Figure 12 the structure shown in

[0178] is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0178] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the following steps are implemented:

[0179] Obtain point cloud data and image data at the synchronization time;

[0180] Register the point cloud data and the image data in the overlapping area to determine the global mapping relationship between the point cloud data and the image data; the overlapping area is the overlapping area between the area covered by the point cloud data and the area covered by the image data; the global area is used to represent all the covered areas of the image data, and the covered area of the image data includes the detection blind area of the lidar;

[0181] Fill the detection blind area of the lidar with virtual point cloud data according to the global mapping relationship, and map the virtual point cloud data into the image data by using the global mapping relationship;

[0182] Determine the depth information of the detection target based on the global mapping relationship, the virtual point cloud data in the detection blind area of the lidar, and the pixel points of the detection target in the detection blind area of the lidar in the image data.

[0183] The computer device provided in the above embodiment has the same implementation principle and technical effects as the above method embodiment, and will not be elaborated here.

[0184] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0185] Obtain the point cloud data and image data at the synchronization time;

[0186] Register the point cloud data and image data in the overlapping area to determine the global mapping relationship between the point cloud data and the image data; the overlapping area is the overlapping area between the area covered by the point cloud data and the area covered by the image data; the global area is used to represent all covered areas of the image data, and the covered area of the image data includes the detection blind area of the lidar;

[0187] Fill the detection blind area of the lidar with virtual point cloud data according to the global mapping relationship, and map the virtual point cloud data into the image data by using the global mapping relationship;

[0188] Determine the depth information of the detection target based on the global mapping relationship, the virtual point cloud data in the detection blind area of the lidar, and the pixel points of the detection target in the detection blind area of the lidar in the image data.

[0189] The computer-readable storage medium provided in the above embodiment has the same implementation principle and technical effects as the above method embodiment, and will not be elaborated here.

[0190] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0191] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0192] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A method for blind area data processing, characterized in that, The method includes: Obtaining point cloud data and image data at the synchronization time; Registering the point cloud data and the image data in the overlapping area to determine the global mapping relationship between the point cloud data and the image data; the overlapping area is the overlapping area between the area covered by the point cloud data and the area covered by the image data; the global area is used to represent all the covered areas of the image data, and the covered area of the image data includes the detection blind area of the lidar; According to the global mapping relationship, filling virtual point cloud data in the detection blind area of the lidar, and mapping the virtual point cloud data into the image data by using the global mapping relationship; Based on the global mapping relationship, the virtual point cloud data in the detection blind area of the lidar, and the pixel points of the detection target in the detection blind area of the lidar in the image data, determining the depth information of the detection target.

2. The method according to claim 1, characterized in that, The registering the point cloud data and the image data in the overlapping area to determine the global mapping relationship between the point cloud data and the image data includes: Obtaining the internal parameter matrix and the external parameter matrix; According to the internal parameter matrix and the external parameter matrix, establishing the global mapping relationship between the coordinates of the point cloud data and the coordinates of the image data in the pixel coordinate system.

3. The method according to claim 2, characterized in that, The method further includes: According to the external parameter matrix, converting the coordinates of the point cloud data in the actual world coordinate system into the coordinates of the point cloud data in the camera coordinate system; According to the internal parameter matrix, converting the coordinates of the point cloud data in the camera coordinate system into the coordinates of the point cloud data in the pixel coordinate system.

4. The method according to claim 1, characterized in that, The filling virtual point cloud data in the detection blind area of the lidar according to the global mapping relationship, and mapping the virtual point cloud data into the image data by using the global mapping relationship includes: Taking the center of the lidar as the origin, and constructing a virtual point cloud with a preset radius step and angle interval; Performing densification processing on the virtual point cloud to obtain the densified virtual point cloud; According to the global mapping relationship, mapping the densified virtual point cloud into the image data to obtain the virtual point cloud data of the detection blind area of the lidar in the image data.

5. The method according to claim 1, characterized in that, The determining the depth information of the detection target based on the global mapping relationship, the virtual point cloud data in the detection blind area of the lidar, and the pixel points of the detection target in the detection blind area of the lidar in the image data includes: Based on the image data, determining the target pixel points corresponding to the detection target in the detection blind area of the lidar, and fitting two fitting circles that tightly sandwich the target pixel points in the pixel coordinate system; Determining the two intersection points of the perpendicular line passing through the target pixel point and the fitting circle; According to the global mapping relationship, determining the two virtual point clouds closest to the two intersection points from the virtual point cloud data in the detection blind area of the lidar; Based on the distances from the two virtual point clouds to the target pixel point and the depth information of the two virtual point clouds in the actual world coordinate system, obtain the depth information of the target pixel point in the actual world coordinate system, and use the depth information as the depth information of the detection target.

6. The method according to claim 1, characterized in that, The obtaining the point cloud data and the image data at the synchronous time includes: Obtain the first sampling moment of the point cloud data and the second sampling moment of the image data; Calculate the time difference between the first sampling moment and the second sampling moment; If the time difference is less than or equal to a preset time deviation threshold, determine that the point cloud data and the image data are the data collected at the synchronous time; If the time difference is greater than the preset time deviation threshold, perform a correction operation.

7. The method according to claim 6, characterized in that, The correction operation is the first correction operation or the second correction operation is performed. The first correction operation is to obtain the third sampling moment of the next frame of image data according to a preset frame rate step, and re - execute the step of calculating the time difference between the first sampling moment and the third sampling moment; The second correction operation is to obtain the third sampling moment of the next frame of point cloud data according to a preset frame rate step, and re - execute the step of calculating the time difference between the second sampling moment and the third sampling moment.

8. The method according to claim 6, characterized in that, The obtaining the first sampling moment of the point cloud data and the second sampling moment of the image data includes: Determine the first sampling moment of the point cloud data according to the candidate sampling moment on the preset time axis and the first sampling moment deviation; the candidate sampling moment is the sampling moment corresponding to the preset time axis when the camera or radar collects data; the first sampling moment deviation is the time deviation between the radar time axis and the preset time axis; Determine the second sampling moment of the image data according to the candidate sampling moment on the preset time axis and the second sampling moment deviation; the second sampling moment deviation is the time deviation between the camera time axis and the preset time axis.

9. The method according to claim 1, characterized in that, The point cloud data is the data obtained by filtering the original point cloud data according to the image range corresponding to the image data.

10. A blind area data processing device, characterized in that, The device includes: A first obtaining module, configured to obtain point cloud data and image data at time synchronization; A establishing module, configured to register the point cloud data and the image data in the overlapping area, and determine the global mapping relationship between the point cloud data and the image data; the overlapping area is the overlapping area between the area covered by the point cloud data and the area covered by the image data; the global area is used to represent all the covered areas of the image data, and the covered area of the image data includes the detection blind area of the lidar; A filling module, configured to fill virtual point cloud data in the detection blind area of the lidar according to the global mapping relationship, and map the virtual point cloud data into the image data by using the global mapping relationship; A determining module, configured to determine the depth information of the detection target based on the global mapping relationship, the virtual point cloud data in the detection blind area of the lidar, and the pixel points of the detection target in the detection blind area of the lidar in the image data.

11. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium, having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images