A data processing method, apparatus and device, and a storage medium

Abstract

Embodiments of the present application disclose a data processing method and device, electronic equipment and storage medium. The method comprises: if the collection frequency of the laser radar is higher than the collection frequency of the camera, and only the laser radar has collected data at the current time, determining a direction deviation according to the collection direction of the first point cloud collected by the laser radar at the current time and the collection direction of the first image data collected by the camera at the last reference time; determining an image collection blind spot according to the direction deviation and a deviation threshold; and adding color information to the first point cloud according to the first image data and the image collection blind spot. The technical solution of the embodiments of the present application can more accurately add color information to the point cloud collected by the laser radar, thereby providing a guarantee for subsequent accurate construction of a three-dimensional point cloud model.

Description

[0001] This application is a divisional application of the invention patent application filed on June 29, 2022, with application number 202210744784.3 entitled "A data processing method, apparatus, device and storage medium". Technical Field

[0002] The embodiments of the present invention relate to the field of three-dimensional modeling technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology

[0003] 3D modeling is typically the process of fusing data collected by LiDAR and data collected by cameras to generate a 3D image carrying color information.

[0004] However, in 3D modeling, the scanning frequency of LiDAR is usually much higher than the frame rate of camera capture. Currently, for moments when only LiDAR is scanning point clouds and the camera is not capturing images, the point cloud scanned by the LiDAR in this scan is typically mapped to the most recently captured image by the camera to determine the color information of the point cloud in this scan. However, in actual space, there are certain locations where the LiDAR can scan the corresponding point cloud in this scan, but the camera may not have been able to capture the color information corresponding to that location in the previous scan. This leads to inaccurate color information assignment in the 3D modeling result, which urgently needs improvement. Summary of the Invention

[0005] This invention provides a data processing method, apparatus, device, and storage medium that can more accurately add color information to point clouds collected by lidar, thereby ensuring the accurate construction of subsequent 3D point cloud models.

[0006] In a first aspect, embodiments of the present invention provide a data processing method, the method comprising:

[0007] If the acquisition frequency of the LiDAR is higher than that of the camera, and only the LiDAR is acquiring data at the current moment, then the first point cloud acquired by the LiDAR at the current moment is mapped to the first image data acquired at the previous reference moment, to obtain the mapped point of the first point cloud in the first image data; where the previous reference moment is the moment when the LiDAR and the camera last acquired data simultaneously before the current moment; the LiDAR and the camera are installed on the same mobile device;

[0008] Based on the current height information of the mapping point and the recorded height information of the surrounding points, the image acquisition blind spot is determined from the mapping point;

[0009] Color information is added to the first point cloud based on the first image data and the blind spots in the image acquisition.

[0010] Secondly, embodiments of the present invention also provide a data processing apparatus, the apparatus comprising:

[0011] The mapping point acquisition module is used to map the first point cloud acquired by the LiDAR at the current moment to the first image data acquired at the previous reference moment if the acquisition frequency of the LiDAR is higher than that of the camera, and only the LiDAR is acquiring data at the current moment, thereby obtaining the mapping point of the first point cloud in the first image data; wherein, the previous reference moment is the moment when the LiDAR and the camera last acquired data simultaneously before the current moment; the LiDAR and the camera are installed on the same mobile device;

[0012] The image blind spot determination module is used to determine the image acquisition blind spot from the mapping point based on the current height information of the mapping point and the recorded height information of the surrounding points of the mapping point;

[0013] The color information addition module is used to add color information to the first point cloud based on the first image data and the blind spots in the image acquisition.

[0014] Thirdly, embodiments of the present invention also provide an electronic device, comprising:

[0015] One or more processors;

[0016] Memory, used to store one or more programs;

[0017] When one or more programs are executed by one or more processors, the one or more processors implement the data processing method as described in any embodiment of the present invention.

[0018] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described in any embodiment of the present invention.

[0019] The technical solution of this invention addresses the situation where the acquisition frequency of the LiDAR is higher than that of the camera, and only the LiDAR is acquiring data at the current moment. It maps the first point cloud acquired by the LiDAR at the current moment to the first image data acquired at the previous reference moment. Based on the first image data acquired at the previous reference moment, the mapping point of the first point cloud at the current moment is obtained. Based on the current height information of the mapping point and the recorded height information of the surrounding points, image acquisition blind spots are determined from the mapping point. Then, color information is added to the first point cloud according to the first image data and the image acquisition blind spots. This solution addresses the situation where only the LiDAR is scanning the point cloud, and the camera is not acquiring images. It adds a process of determining whether each point cloud acquired by the LiDAR is a blind spot (i.e., an area not captured by the camera) based on height information. Considering the camera blind spots before adding color information to the point cloud improves the accuracy of adding point cloud color information, thus providing a guarantee for the subsequent accurate construction of a 3D point cloud model. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a data processing method provided in Embodiment 1 of the present invention.

[0022] Figure 2 This is a flowchart of a data processing method provided in Embodiment 2 of the present invention.

[0023] Figure 3 This is a flowchart of a data processing method provided in Embodiment 3 of the present invention.

[0024] Figure 4 This is a flowchart of a data processing method provided in Embodiment 4 of the present invention.

[0025] Figure 5 This is a schematic diagram of the structure of a data processing device provided in Embodiment 5 of the present invention.

[0026] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment Six of the present invention. Detailed Implementation

[0027] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0028] Example 1

[0029] Figure 1 This is a flowchart of a data processing method provided in Embodiment 1 of the present invention. This embodiment is applicable to the processing of 3D modeling data, and is particularly suitable for adding color information to point clouds acquired by LiDAR based on image data acquired by a camera during 3D modeling. This method can be executed by a data processing device, which can be implemented in hardware and / or software and can be integrated into an electronic device. Figure 1 As shown, the method includes:

[0030] S110. If the acquisition frequency of the lidar is higher than that of the camera, and only the lidar is acquiring data at the current moment, then the first point cloud acquired by the lidar at the current moment is mapped to the first image data acquired at the previous reference moment to obtain the mapping point of the first point cloud in the first image data.

[0031] The previous reference time is the most recent time when the lidar and camera simultaneously collected data before the current time.

[0032] The lidar and camera are mounted on the same mobile device. This mobile device can be any device capable of moving the camera and lidar. For example, it could be a mobile robot or a drone.

[0033] The first point cloud can be the point cloud collected by the LiDAR scanning the surrounding environment at the current moment. There are usually multiple first point clouds. The first image data can be the image data collected by the camera at the previous reference moment. The mapping point can be the pixel in the first image data corresponding to the first point cloud mapped to the previous reference moment. It should be noted that each point cloud collected by the LiDAR has a corresponding mapping point in the first image data.

[0034] It should be noted that in this embodiment of the invention, the acquisition frequency of the LiDAR is higher than that of the camera, and the current moment is defined as when only the LiDAR has acquired data, while the camera has not acquired any data. In this case, the data acquired by the camera at the previous reference moment can be used as reference data to determine the actual data that the camera should acquire at the current moment, thereby determining whether there is an image acquisition blind spot in the first point cloud acquired by the LiDAR at the current moment. For example, assuming the LiDAR data acquisition moments are t1, t2, t3, t4, and t5, and the camera data acquisition moments are t1 and t5, if the current moment is any of t2-t4, then the previous reference moment can be t1.

[0035] Specifically, since the first point cloud data is data in the radar coordinate system and the first image data is data in the image coordinate system, the process of mapping the first point cloud collected by the lidar at the current moment to the first image data collected at the previous reference moment to obtain the mapping point of the first point cloud in the first image data can be as follows: transform each first point cloud in the radar coordinate system from the current moment to the previous reference moment, find the point cloud corresponding to each first point cloud in the second point cloud data collected at the previous reference moment, that is, the associated point cloud, and then transform the associated point cloud from the radar coordinate system to the camera coordinate system, and then transform it from the camera coordinate system to the image coordinate system. At this time, the position point of each point cloud in the image coordinate system is the mapping point of the first point cloud in the first image data, thus realizing the process of mapping the first point cloud to the first image data corresponding to the previous reference moment and finding the mapping point.

[0036] S120. Based on the current height information of the mapping point and the recorded height information of the surrounding points, determine the image acquisition blind spot from the mapping point.

[0037] The current altitude information can be the altitude of the first point cloud corresponding to the mapped point. This current altitude information can be the altitude of the first point cloud in the radar coordinate system or the altitude in the camera coordinate system. The recorded altitude information can be the most recently recorded altitude information. It should be noted that the recording time of the recorded altitude information of each surrounding point can be the same or different.

[0038] Surrounding points can be any points adjacent to the mapping point, centered on it. For example, the eight points above, below, to the left, to the right, to the upper left, to the lower left, to the upper right, and to the lower right of the mapping point can be considered as surrounding points of the mapping point.

[0039] Image acquisition blind spots can be locations that the lidar can scan at the current moment, but the camera did not capture at the previous reference moment.

[0040] Optionally, in actual space, the height information of adjacent points usually does not differ significantly. Therefore, in this embodiment, the process of determining image acquisition blind spots from mapping points based on the current height information of the mapping point and the recorded height information of surrounding points can be achieved by determining whether the mapping point is a blind spot based on the height difference between the current height information of the mapping point and the recorded height information of each surrounding point. For example, the height difference between the current height information of the mapping point and the recorded height information of each surrounding point can be determined. If the height difference does not exceed a preset height difference, such as 0.5m, then the mapping point is determined not to be an image acquisition blind spot; if the height difference is greater than the preset height difference, such as 0.5m, then the mapping point is determined to be an image acquisition blind spot.

[0041] Based on the above technical solution, according to the current height information of the mapping point and the recorded height information of the surrounding points, the preferred implementation method for determining the image acquisition blind spot from the mapping point may include the following:

[0042] S1201. Based on the recorded height information and initial information of the mapping points, determine the candidate points from the mapping points.

[0043] The initial information judgment condition can be a condition for determining whether the point cloud position corresponding to the mapped point is the initial value. Candidate points can be selected from the mapped points to further determine whether they are blind spots in image acquisition.

[0044] Specifically, in this embodiment, a memory space can be pre-allocated to record the height coordinates of each pixel in the image. Each pixel is initialized with a small initial value, such as -9999. For any first point cloud, if it maps to a location in the image and the recorded height information at that location is the initial value, then the mapped point of the first point cloud is not a candidate point. In this case, the recorded height information of the mapped point in the allocated memory space needs to be updated based on the height information of the first point cloud. If it maps to a location in the image and the recorded height information at that location is not the initial value, meaning that the recorded height information at that location has been updated before, then the mapped point of the first point cloud is a candidate point.

[0045] S1202. Based on the current height information of the candidate points and the recorded height information of the surrounding points of the candidate points, determine the image acquisition blind spots from the candidate points.

[0046] Specifically, it can be determined whether each candidate point is a blind spot for image acquisition based on the height difference between the current height information of the candidate point and the recorded height information of each surrounding point.

[0047] This solution sets initial information judgment conditions to preliminarily screen candidate points. For each candidate point initially selected, it combines its current height information with the recorded height information of its surrounding points to determine whether each candidate point is an image acquisition blind spot. It does not require a height difference judgment for each mapping point. While ensuring the accuracy of image acquisition blind spot determination, it greatly improves the efficiency of image acquisition blind spot determination.

[0048] Furthermore, the process of determining image acquisition blind spots from candidate points based on the current height information of the candidate points and the recorded height information of the surrounding points preferably includes: for each candidate point, determining the height difference between the candidate point and each surrounding point based on the current height information of the candidate point and the recorded height information of the surrounding points. If the number of surrounding points with a height difference greater than a height threshold is greater than a quantity threshold, then the candidate point is determined as an image acquisition blind spot.

[0049] The height threshold is the upper limit of the height difference. The number threshold is the upper limit of the number of surrounding points whose height difference is greater than the height threshold. The height threshold and the number threshold can be set and adjusted based on experience.

[0050] Specifically, for each identified candidate point, the height difference between the candidate point and its surrounding points is calculated by subtracting the candidate point's current height information from the recorded height data of the surrounding points. Then, it is determined whether the number of surrounding points with a height difference greater than a height threshold is greater than a quantity threshold. If it is greater, the candidate point is identified as an image acquisition blind spot.

[0051] This solution introduces both height and quantity thresholds to determine blind spots in image acquisition, thereby improving the accuracy of blind spot determination.

[0052] S130. Based on the first image data and the image acquisition blind spot, add color information to the first point cloud.

[0053] Specifically, in this embodiment, the point cloud corresponding to the image acquisition blind spot in the first point cloud can be obtained based on the determined image acquisition blind spot, and then the point cloud corresponding to the non-image acquisition blind spot in the first point cloud can be determined; then, for the point cloud corresponding to the non-image acquisition blind spot, the color information corresponding to the point cloud in the first image data can be used as the color information of the first point cloud.

[0054] The technical solution of this invention addresses the situation where the LiDAR's acquisition frequency is higher than the camera's acquisition frequency, and only the LiDAR is acquiring data at the current moment. It maps the first point cloud acquired by the LiDAR at the current moment to the first image data acquired at the previous reference moment. Based on the first image data acquired at the previous reference moment, the mapping point of the first point cloud at the current moment is obtained. Based on the current height information of the mapping point and the recorded height information of surrounding points, image acquisition blind spots are determined from the mapping points. Then, color information is added to the first point cloud based on the first image data and the image acquisition blind spots. This solution addresses the situation where only the LiDAR is scanning the point cloud, and the camera is not acquiring images. It adds a process of determining whether each point cloud acquired by the LiDAR is a blind spot (i.e., an area not captured by the camera) based on height information. Considering the camera blind spots before adding color information to the point cloud improves the accuracy of adding point cloud color information, thus ensuring the accurate construction of the subsequent 3D point cloud model.

[0055] Example 2

[0056] Figure 2 This is a flowchart of a data processing method provided in Embodiment 2 of the present invention. This embodiment further optimizes the above technical solution by determining candidate points as image acquisition blind spots if the number of surrounding points with a height difference greater than a height threshold exceeds a quantity threshold. Figure 2 As shown, the method includes:

[0057] S210. If the acquisition frequency of the lidar is higher than that of the camera, and only the lidar is acquiring data at the current moment, then the first point cloud acquired by the lidar at the current moment is mapped to the first image data acquired at the previous reference moment to obtain the mapping point of the first point cloud in the first image data.

[0058] The previous reference time is the most recent time when the lidar and camera simultaneously collected data before the current time; the lidar and camera are installed on the same mobile device.

[0059] S220. Based on the recorded height information and initial information of the mapping points, determine the candidate points from the mapping points.

[0060] S230. For each candidate point, determine the height difference between the candidate point and each surrounding point based on the current height information of the candidate point and the recorded height information of the surrounding points.

[0061] S240. When the lidar is a multi-echo lidar, if the number of surrounding points with a height difference greater than the height threshold is greater than the quantity threshold, then the direction deviation is determined based on the acquisition direction of the first point cloud corresponding to the candidate point and the acquisition direction of the first image data.

[0062] The acquisition direction of the first point cloud can be the emission direction of the laser beam of each first point cloud acquired by the lidar, and the acquisition direction of the first image data can be the direction corresponding to the lens when the camera acquires the first image data.

[0063] Specifically, in this embodiment, when the lidar is a multi-echo lidar, if the number of surrounding points with a height difference greater than the height threshold is greater than the quantity threshold, it indicates that the mapped point may be a blind spot for image acquisition. In this case, the deviation between the acquisition direction of the first point cloud and the acquisition direction of the first image data can be calculated as the direction deviation.

[0064] S250. If the directional deviation is greater than the deviation threshold, the candidate point is determined as a blind spot in image acquisition.

[0065] Among them, the deviation threshold is the maximum value of the directional deviation, and the deviation threshold can be set and adjusted based on experience.

[0066] If the direction of the first point cloud acquisition deviates from the direction of the first image data acquisition by more than the deviation threshold, then the candidate point is regarded as the image acquisition blind spot.

[0067] Optionally, when the lidar is a single-echo lidar, this embodiment can determine the directional deviation based on the acquisition direction of the first point cloud and the acquisition direction of the first image data after acquiring the first point cloud. If the directional deviation is greater than the deviation threshold, the first point cloud is directly determined as an image acquisition blind spot; if the directional deviation is less than or equal to the deviation threshold, then the operations S210-S230 are executed; if the number of surrounding points with a height difference greater than the height threshold is greater than the quantity threshold, then the candidate points corresponding to each surrounding point are taken as image acquisition blind spots, and the operations S240-S250 are not executed.

[0068] S260. Based on the first image data and the image acquisition blind spot, add color information to the first point cloud.

[0069] The technical solution of this invention addresses the situation where the acquisition frequency of the lidar is higher than that of the camera, and only the lidar is currently acquiring data. It obtains the mapped points of the first point cloud in the first image data by mapping the first point cloud acquired by the lidar at the current moment to the first image data acquired at the previous reference moment. Based on the recorded height information and initial information judgment conditions of the mapped points, candidate points are determined from the mapped points. For each candidate point, the height difference between the candidate point and each surrounding point is determined based on the current height information of the candidate point and the recorded height information of the surrounding points. If the lidar is a multi-echo lidar, and the number of surrounding points with height differences greater than a height threshold is greater than a quantity threshold, a directional deviation is determined based on the acquisition direction of the first point cloud corresponding to the candidate point and the acquisition direction of the first image data. If the directional deviation is greater than a deviation threshold, the candidate point is determined as an image acquisition blind spot. Color information is added to the first point cloud based on the first image data and the image acquisition blind spot. This solution introduces the acquisition direction of the point cloud and the acquisition direction of the image when determining image acquisition blind spots, further improving the accuracy of image acquisition blind spot determination.

[0070] Example 3

[0071] Figure 3 This is a flowchart of a data processing method provided in Embodiment 3 of the present invention. Based on the above technical solution, the first point cloud acquired by the lidar at the current moment is mapped to the first image data acquired at the previous reference moment, and the mapping points of the first point cloud in the first image data are further optimized, such as... Figure 3 As shown, the method includes:

[0072] S310. If the acquisition frequency of the lidar is higher than that of the camera, and only the lidar is acquiring data at the current moment, then the pose transformation matrix from the previous reference moment to the current moment is determined based on the attitude information recorded by the odometer at the current moment and the attitude information at the previous reference moment.

[0073] Odometry can be an effective sensor that provides real-time pose information for mobile devices, enabling them to acquire relative positioning. Pose information includes location and attitude information. The odometry pose information can be located in a global coordinate system, such as a geodetic coordinate system or a world coordinate system.

[0074] In this embodiment, the attitude change matrix corresponding to the previous reference time can be determined based on the attitude information recorded by the odometer at the current time and the attitude information corresponding to the previous reference time, according to the following formula (1).

[0075] T = T t *(T t1 ) -1 (1)

[0076] Where T is the attitude change matrix from the previous reference time to the current time; T t The attitude information at the current moment; T t1 This refers to the attitude information corresponding to the previous reference time.

[0077] Optionally, in this embodiment, there are many ways to determine the attitude information recorded in the odometer. One possible method is as follows:

[0078] The attitude information at the current moment and the attitude information at the previous reference moment, collected by the combined inertial navigation system installed on the mobile device, are used as the attitude information at the current moment and the attitude information at the previous reference moment recorded by the odometer.

[0079] In this embodiment, the integrated inertial navigation system (INS) can be a sensor that records the pose information of the mobile device. Specifically, this embodiment can directly record the pose information collected by the integrated INS at the moment when the lidar scans and collects data into the odometer, thereby obtaining pose information including the current moment and the previous reference moment.

[0080] Another possible approach is to use inter-frame matching with the radar to determine the attitude information of the current moment and the attitude information of the previous reference moment recorded by the odometer. Specifically, this can be achieved by using point cloud data scanned from adjacent frames of the lidar, employing an inter-frame matching algorithm to calculate the position and attitude information corresponding to each acquisition moment, and then recording it in the odometer.

[0081] This scheme preferably uses a combined inertial navigation system to determine the attitude information recorded by the odometer. The advantage of this setting is that it can further ensure the accuracy and convenience of attitude information determination.

[0082] S320. Based on the pose transformation matrix, determine the associated point cloud in the second point cloud collected at the previous reference time, which is the first point cloud collected by the lidar at the current moment.

[0083] The second point cloud can be the point cloud collected by the lidar at the previous reference time. The second point cloud and the first point cloud are point cloud data collected by the lidar at different times.

[0084] The associated point cloud can be the corresponding point cloud in the second point cloud that is mapped from the current time to the previous reference time in the first point cloud.

[0085] The pose transformation matrix can include a rotation matrix and a translation matrix. The rotation matrix can be a 3x3 matrix, representing the rotational change when the pose is changed from one moment to another. The translation matrix can be a 3x1 matrix, representing the positional change when the position is translated from one moment to another.

[0086] Specifically, in this embodiment, the correspondence between the first point cloud at the current time and the associated point cloud at the second point cloud at the previous reference time can be determined based on the translation matrix and rotation matrix in the pose transformation matrix, according to the following formula (2).

[0087] P1=P*R+t (2)

[0088] Where P1 is the associated point cloud in the second point cloud at the previous reference time; P is the first point cloud at the current time; R is the rotation matrix corresponding to mapping the point cloud from the current time to the previous reference time; and t is the translation matrix corresponding to mapping the point cloud from the current time to the previous reference time.

[0089] S330. Transform the associated point cloud from the radar coordinate system to the image coordinate system of the first image data acquired at the previous reference time, and obtain the mapping points of the first point cloud in the first image data.

[0090] Optionally, transforming the associated point cloud from the radar coordinate system to the image coordinate system of the first image data acquired at the previous reference time can be done by transforming the associated point cloud from the radar coordinate system to the camera coordinate system based on the coordinate system transformation matrix between the radar coordinate system and the camera coordinate system, and then transferring the camera coordinate system to the corresponding pixel in the first image data in the image coordinate system based on the pinhole camera model.

[0091] A coordinate system transformation matrix can include a coordinate system rotation matrix and a coordinate system translation matrix. The matrix structure and function of the coordinate system rotation matrix and the coordinate system translation matrix are similar to those of the pose transformation matrix described above, and will not be elaborated upon here.

[0092] In this embodiment, the associated point cloud can be transformed from the radar coordinate system to the camera coordinate system based on the coordinate system rotation matrix and coordinate system translation matrix in the coordinate system transformation matrix, according to the following formula (3).

[0093] P C =P L *R LC +t LC (3)

[0094] Among them, P C This represents the associated point cloud in the camera coordinate system at the previous reference time; P L This represents the associated point cloud in the radar coordinate system at the previous reference time; R LC The coordinate system rotation matrix for transforming the radar coordinate system to the camera coordinate system; t LC This is the coordinate system translation matrix for transforming the radar coordinate system to the camera coordinate system.

[0095] Specifically, the pixel points in the first image data that are transferred from the camera coordinate system to the image coordinate system based on the pinhole camera model can be obtained by using the pinhole imaging principle of the pinhole camera model to project the associated point cloud in the camera coordinate system onto the pixel points in the image coordinate system, thereby obtaining the mapping points of the associated point cloud in the image coordinate system.

[0096] S340. Based on the current height information of the mapping point and the recorded height information of the surrounding points, determine the image acquisition blind spot from the mapping point.

[0097] S350: Add color information to the first point cloud based on the first image data and the image acquisition blind spot.

[0098] The technical solution of this invention determines the pose transformation matrix from the previous reference time to the current time by using the attitude information recorded by the odometer at the current moment and the attitude information at the previous reference time. Based on the pose transformation matrix, the associated point cloud in the second point cloud acquired by the lidar at the current moment is determined. The associated point cloud is transformed from the lidar coordinate system to the image coordinate system of the first image data acquired at the previous reference time to obtain the mapping point of the first point cloud in the first image data. According to the current height information of the mapping point and the recorded height information of the surrounding points of the mapping point, the image acquisition blind spot is determined from the mapping point. Color information is added to the first point cloud according to the first image data and the image acquisition blind spot. This embodiment provides a preferred implementation method for transforming the first point cloud data from the current moment to the image coordinate system to obtain the associated point cloud of the first point cloud in the second point cloud, providing conditions for image acquisition blind spot screening.

[0099] Example 4

[0100] Figure 4 The flowchart below illustrates a data processing method according to Embodiment 4 of the present invention. Based on the above technical solution, further optimizations have been made. Specifically, a preferred implementation method is given after adding color information to the first point cloud, such as... Figure 4 As shown, the data processing method includes:

[0101] S410. If the acquisition frequency of the lidar is higher than that of the camera, and only the lidar is acquiring data at the current moment, then the first point cloud acquired by the lidar at the current moment is mapped to the first image data acquired at the previous reference moment to obtain the mapping point of the first point cloud in the first image data.

[0102] The previous reference time is the most recent time when the lidar and camera simultaneously collected data before the current time; the lidar and camera are installed on the same mobile device.

[0103] S420. Based on the current height information of the mapping point and the recorded height information of the surrounding points, determine the image acquisition blind spot from the mapping point.

[0104] S430. Based on the first image data and the image acquisition blind spot, add color information to the first point cloud.

[0105] S440: Update the 3D point cloud model based on the pose information, color information, and odometry information of the first point cloud.

[0106] Specifically, based on the pose information of the first point cloud at the current moment and the odometry information of the LiDAR at the current moment, the pose information of each first point cloud is converted into the global coordinate system to obtain the position information of each first point cloud in the 3D model. Then, based on the color information added to each first point cloud by S430, the color information is assigned to the corresponding position of the first point cloud in the 3D point cloud model, thereby updating the 3D point cloud model. That is, every time the LiDAR collects point cloud data, the above operation needs to be performed to update the 3D point cloud model.

[0107] It should be noted that, for the image acquisition blind spots in the first point cloud, since the S430 does not add color information for them, when updating the 3D point cloud model in this step, it is not necessary to assign color information to the corresponding positions of the image acquisition blind spots.

[0108] S450. Based on the blind spots in the image acquisition, determine the blind spot point cloud in the first point cloud;

[0109] Among them, the blind spot point cloud can be a point cloud that is selected from the first point cloud and does not have corresponding image data in the image data collected by the camera at the previous reference time.

[0110] Specifically, based on the image acquisition blind spots, the blind spot point cloud in the first point cloud can be determined by selecting the image acquisition blind spots from the mapping points and using their corresponding first point clouds as the blind spot point cloud.

[0111] S460. If a first point cloud color complementation event is detected, then based on the updated 3D point cloud model, determine the color-complementable point clouds in the blind area point clouds of the first point cloud.

[0112] Among them, complementary color point cloud can be a point cloud in the point cloud data of the updated 3D point cloud model without added color information.

[0113] The first point cloud color supplementation event is an event that requires filling in color information for the blind point cloud in the first point cloud. Specifically, the first color supplementation event may include: if the next time step after the current time step is not a reference time step, detecting that the operation of filling in color information for the blind point cloud in the third point cloud at the next time step has been completed; if the next time step after the current time step is a reference time step, detecting that the operation of adding color information for the fourth point cloud at a non-reference time step after the next time step has been completed.

[0114] The reference time is the moment when the lidar and camera simultaneously collect data.

[0115] Non-reference times are those when the lidar collects data but the camera does not.

[0116] The third point cloud can be the point cloud collected by the lidar at the next moment after the current moment, when the next moment after the current moment is not the reference moment.

[0117] The fourth point cloud can be the point cloud collected by the lidar at each time between the next reference time and the next reference time, with the next time after the current time as the reference time.

[0118] Specifically, if the next moment after the current moment is not the reference moment, and it is detected that the operation of determining the complementary color point cloud and filling the color information of the determined complementary color point cloud has been performed on the third point cloud, it means that the first point cloud complementary color event has been detected.

[0119] If the next moment after the current moment is taken as the base moment, if the fourth point cloud is detected to have completed the operation of adding color information, it means that the first point cloud's color complement event has been detected.

[0120] For example, assume the times for lidar data acquisition are t1, t2, t3, t4, t5, t6, t7, t8, t9, t 10 The camera data was collected at times t1, t5, and t6. 10 If the current time is t2 or t3, then the next time t3 or t4 is a non-baseline time. If it is detected that the operation of determining the complementary color point cloud and filling in color information for the determined complementary color point cloud has been completed on the point cloud at time t3 or t4 (i.e., the third point cloud), it indicates that the first point cloud color completion event has been detected. If the current time is t4, then the next time t5 is the baseline time. If it is detected that the operation of adding color information has been completed on the point cloud at times t6-t9 (i.e., the fourth point cloud), it indicates that the first point cloud color completion event has been detected.

[0121] It should be noted that this embodiment takes into account the characteristics of radar-acquired data when determining the first point cloud color complementation event. The data is output based on the sequential order of time and the distance of the echoes, in ascending order of time and ascending order of distance. Based on this characteristic, the timing for point cloud color complementation is determined, thereby ensuring the accuracy of the complementation results.

[0122] Optionally, in this embodiment, after identifying the blind point cloud in the first point cloud, the complementary point cloud is not immediately determined. Instead, it is determined only after a complementary color event of the first point cloud is detected, based on the most recently updated 3D point cloud model between the detection of the complementary color event and the determination of the complementary point cloud in the blind point cloud of the first point cloud. Specifically, the determination process can be based on the point cloud data of the updated 3D point cloud model to determine whether color information has been added to the blind point cloud in the first point cloud. If no color information has been added, the blind point cloud is considered a complementary point cloud.

[0123] S470. Based on the image data acquired at the next reference time, fill in the color information of the color-complementary point cloud.

[0124] The next reference time can be the time when the lidar and camera simultaneously collected data most recently since the current time.

[0125] Specifically, the process of filling in color information for the complementary color point cloud based on the image data acquired at the next reference time can be similar to the process in S410-S430. For example, the complementary color point cloud is mapped onto the second image data acquired at the next reference time. Based on the second image data acquired at the next reference time, the mapping points of the current complementary color point cloud at the next reference time (i.e., the mapping points of the complementary color point cloud) are obtained. Based on the current height information of the mapping points of the complementary color point cloud and the recorded height information of the surrounding points, image acquisition blind spots in the complementary color point cloud are determined from the mapping points of the complementary color point cloud. Then, based on the second image data and the image acquisition blind spots in the complementary color point cloud, color information is added to the non-image acquisition blind spots in the complementary color point cloud.

[0126] The technical solution of this invention, after adding color information to the first point cloud, updates the 3D point cloud model based on the pose information, color information, and current odometer information of the first point cloud. Taking into account image acquisition blind spots, color information is copied only for non-image acquisition blind spots, ensuring the accuracy of the 3D point cloud model. Furthermore, this solution further processes the blind spot point cloud by identifying the blind spot corresponding to the image acquisition blind spot, filters out complementary color point clouds, and fills in the complementary color point clouds with color information based on the image data acquired at the next reference time and the updated 3D point cloud model. This further processing of the image acquisition blind spot, using the image data acquired at the next reference time, ensures the accuracy of color assignment for complementary color point clouds in the blind spot point cloud, thereby guaranteeing the accuracy of the 3D point cloud model.

[0127] Example 5

[0128] Figure 5 This is a schematic diagram of a data processing device provided in Embodiment 5 of the present invention. This embodiment is applicable to the processing of 3D modeling data, especially to the process of adding color information to point clouds collected by LiDAR based on image data acquired by a camera during 3D modeling. This device can implement the data processing method of the above embodiments of the present invention. The data processing device can be implemented in hardware and / or software, and can be integrated into an electronic device. Figure 5 As shown, the device includes:

[0129] The mapping point acquisition module 510 is used to map the first point cloud acquired by the LiDAR at the current moment to the first image data acquired at the previous reference moment if the acquisition frequency of the LiDAR is higher than that of the camera, and only the LiDAR is acquiring data at the current moment, thereby obtaining the mapping point of the first point cloud in the first image data; wherein, the previous reference moment is the moment when the LiDAR and the camera last acquired data simultaneously before the current moment; the LiDAR and the camera are installed on the same mobile device;

[0130] The image blind spot determination module 520 is used to determine the image acquisition blind spot from the mapping point based on the current height information of the mapping point and the recorded height information of the surrounding points of the mapping point;

[0131] The color information adding module 530 is used to add color information to the first point cloud based on the first image data and the image acquisition blind spot.

[0132] The technical solution of this invention addresses the issue of the difficulty in identifying image acquisition blind spots when the acquisition frequency of the lidar is higher than that of the camera, and only the lidar is acquiring data at the current moment. It maps the first point cloud acquired by the lidar at the current moment to the first image data acquired at the previous reference moment. Based on the first image data acquired at the previous reference moment, the mapping point of the first point cloud at the current moment is obtained. Based on the current height information of the mapping point and the recorded height information of surrounding points, image acquisition blind spots are identified from the mapping points. Then, color information is added to the first point cloud according to the first image data and the image acquisition blind spots. This solves the problem of difficulty in identifying image acquisition blind spots, enables blind spot filtering of mapping points, and completes the color addition of the first point cloud to the point cloud corresponding to points that are not image acquisition blind spots.

[0133] Optionally, the image blind spot determination module 520 may include:

[0134] The candidate point determination unit is used to determine candidate points from the mapping points based on the recorded height information and initial information of the mapping points and the judgment conditions.

[0135] The image blind spot determination unit is used to determine the image acquisition blind spot from the candidate points based on the current height information of the candidate points and the recorded height information of the surrounding points of the candidate points.

[0136] Furthermore, the image blind spot determination unit includes:

[0137] The height difference determination subunit is used to determine the height difference between the candidate point and each surrounding point for each candidate point, based on the current height information of the candidate point and the recorded height information of the surrounding points.

[0138] The blind spot determination subunit is used to determine candidate points as image acquisition blind spots if the number of surrounding points with a height difference greater than a height threshold is greater than a quantity threshold.

[0139] Specifically, the blind spot determination subunit can be used for:

[0140] When the lidar is a multi-echo lidar, if the number of surrounding points with a height difference greater than the height threshold is greater than the quantity threshold, the directional deviation is determined based on the acquisition direction of the first point cloud corresponding to the candidate point and the acquisition direction of the first image data.

[0141] If the directional deviation is greater than the deviation threshold, the candidate point is determined to be a blind spot in image acquisition.

[0142] Optionally, the color information adding module 530 can be used for:

[0143] The pose matrix determination unit is used to determine the pose transformation matrix from the previous reference time to the current time based on the pose information of the current time recorded by the odometer and the pose information of the previous reference time.

[0144] The associated point cloud determination unit is used to determine the associated point cloud in the second point cloud acquired at the previous reference time from the first point cloud acquired by the lidar at the current moment, based on the pose transformation matrix.

[0145] The mapping point determination unit is used to transform the associated point cloud from the radar coordinate system to the image coordinate system of the first image data acquired at the previous reference time, so as to obtain the mapping point of the first point cloud in the first image data.

[0146] Accordingly, before determining the pose transformation matrix from the previous reference time to the current time based on the attitude information recorded by the odometer at the current time and the previous reference time, the attitude matrix determination unit also includes:

[0147] The attitude information of the current time and the previous reference time collected by the combined inertial navigation system installed on the mobile device is used as the attitude information of the current time and the previous reference time recorded by the odometer.

[0148] Accordingly, after adding color information to the first point cloud, the data processing device also includes:

[0149] The point cloud data update module is used to update the 3D point cloud model based on the pose information, color information and current odometry information of the first point cloud.

[0150] Accordingly, after adding color information to the first point cloud, the data processing device also includes:

[0151] The blind spot point cloud determination module is used to determine the blind spot point cloud in the first point cloud based on the blind spots acquired in the image.

[0152] The complementary color point cloud determination module is used to determine the complementary color point cloud in the blind area point cloud of the first point cloud based on the updated 3D point cloud model if a first point cloud complementary color event is detected.

[0153] The color information filling module is used to fill in color information for the colorable point cloud based on the image data acquired at the next reference time.

[0154] The first point cloud color complement event includes any of the following events:

[0155] If the next time step is not the baseline time step, it is detected that the operation of filling in color information has been completed for the blind point cloud in the third point cloud at the next time step.

[0156] With the next moment after the current moment as the reference moment, it was detected that the operation of adding color information has been completed on the fourth point cloud at a non-reference moment after the next moment.

[0157] The data processing apparatus provided in the embodiments of the present invention can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0158] Example 6

[0159] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment Six of the present invention. Figure 6 A block diagram is shown that is suitable for implementing embodiments of the present invention. Figure 6 The device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0160] like Figure 6 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: one or more processors or processing units 610, system memory 620, and bus 630 connecting different system components (including system memory 620 and processing unit 610).

[0161] Bus 630 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0162] Electronic device 600 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 600, including volatile and non-volatile media, removable and non-removable media.

[0163] System memory 620 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 621 and / or cache memory (cache 622). Electronic device 600 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 623 may be used to read and write non-removable, non-volatile magnetic media (… Figure 6 Not shown; usually referred to as a "hard drive"). Although Figure 6Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 630 via one or more data media interfaces. System memory 620 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0164] A program / utility 625 having a set (at least one) of program modules 624 may be stored, for example, in system memory 620. Such program modules 624 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 624 typically perform the functions and / or methods described in the embodiments of this invention.

[0165] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, display 710, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 640. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 650. As shown, network adapter 650 communicates with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0166] The processing unit 610 executes various functional applications and data processing by running programs stored in the system memory 620, such as implementing the data processing method provided in the embodiments of the present invention.

[0167] Example 7

[0168] Embodiment 7 of the present invention also provides a computer-readable storage medium storing a computer program (or computer-executable instructions) thereon, which is used by a processor to perform the data processing method provided in the embodiments of the present invention.

[0169] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0170] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0171] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0172] Computer program code for performing the operations of embodiments of the present invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0173] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the embodiments of the present invention have been described in detail above, the embodiments of the present invention are not limited to the above embodiments. Many other equivalent embodiments may be included without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data processing method, characterized in that, The lidar is a single-echo lidar, including the following: If the acquisition frequency of the lidar is higher than that of the camera, and only the lidar is acquiring data at the current moment, then the directional deviation is determined based on the acquisition direction of the first point cloud acquired by the lidar at the current moment and the acquisition direction of the first image data acquired by the camera at the previous reference moment. The previous reference time is the most recent time when the lidar and the camera simultaneously acquired data before the current time; the lidar and the camera are installed on the same mobile device; Based on the directional deviation and the deviation threshold, the blind spot in image acquisition is determined; Based on the first image data and the image acquisition blind spot, add color information to the first point cloud; The phrase "determining image acquisition blind spots based on directional deviation and deviation threshold" includes the following: If the directional deviation is greater than the deviation threshold, the first point cloud is determined to be a blind spot in image acquisition; If the directional deviation is less than or equal to the deviation threshold, the first point cloud collected by the lidar at the current moment is mapped to the first image data collected at the previous reference moment to obtain the mapping point of the first point cloud in the first image data. Based on the current height information of the mapping point and the recorded height information of the surrounding points of the mapping point, image acquisition blind spots are determined from the mapping point; The phrase "adding color information to the first point cloud based on the first image data and the image acquisition blind spot" includes the following: Based on the identified image acquisition blind spots, the point cloud corresponding to the image acquisition blind spots in the first point cloud is obtained, and the point cloud corresponding to the non-image acquisition blind spots in the first point cloud is determined. For point clouds corresponding to non-image acquisition blind spots, the color information corresponding to the point cloud in the first image data is used as the color information of the first point cloud.

2. The data processing method according to claim 1, characterized in that, The step of "determining image acquisition blind spots from the mapping point based on the current height information of the mapping point and the recorded height information of the surrounding points of the mapping point" includes: Candidate points are determined from the mapping points based on the recorded height information and initial information of the mapping points and the judgment conditions. Based on the current height information of the candidate points and the recorded height information of the surrounding points of the candidate points, image acquisition blind spots are determined from the candidate points.

3. The data processing method according to claim 2, characterized in that, The step of "determining image acquisition blind spots from the candidate points based on the current height information of the candidate points and the recorded height information of the surrounding points of the candidate points" includes: For each candidate point, the height difference between the candidate point and each surrounding point is determined based on the current height information of the candidate point and the recorded height information of the surrounding points of the candidate point. If the number of surrounding points with a height difference greater than the height threshold is greater than the quantity threshold, then the candidate point is determined to be an image acquisition blind spot.

4. The data processing method according to any one of claims 1 to 3, characterized in that, The step of mapping the first point cloud acquired by the lidar at the current moment to the first image data acquired at the previous reference moment, to obtain the mapping point of the first point cloud in the first image data, includes: Based on the attitude information at the current moment recorded by the odometer and the attitude information at the previous reference moment, determine the pose transformation matrix from the previous reference moment to the current moment. Based on the pose transformation matrix, determine the associated point cloud in the second point cloud collected at the previous reference time for the first point cloud collected by the lidar at the current moment. The associated point cloud is transformed from the radar coordinate system to the image coordinate system of the first image data acquired at the previous reference time, so as to obtain the mapping points of the first point cloud in the first image data.

5. The data processing method according to claim 4, characterized in that, Before determining the pose transformation matrix from the previous reference time to the current time based on the attitude information recorded by the odometer at the current time and at the previous reference time, the method further includes: using the attitude information of the current time and at the previous reference time collected by the combined inertial navigation system installed on the mobile device as the attitude information of the current time and at the previous reference time recorded by the odometer.

6. The method according to claim 4, characterized in that, After adding color information to the first point cloud, the method further includes: updating the 3D point cloud model based on the pose information, color information, and odometry information of the first point cloud at the current moment.

7. The method according to claim 6, characterized in that, After adding color information to the first point cloud, the process also includes: Based on the blind spots acquired from the image, determine the blind spot point cloud in the first point cloud; If the first point cloud color complementation event is detected, then based on the updated 3D point cloud model, determine the color-complementable point clouds in the blind point clouds of the first point cloud; Based on the image data acquired at the next reference time, color information is added to the color-complementary point cloud; wherein, the first point cloud color-complementary event includes any of the following events: If the next time step is not the baseline time step, it is detected that the operation of filling in color information has been completed for the blind point cloud in the third point cloud at the next time step. With the next moment after the current moment as the reference moment, it was detected that the operation of adding color information has been completed on the fourth point cloud at a non-reference moment after the next moment.

8. A data processing apparatus, characterized in that, include: The directional deviation acquisition module is used to determine the directional deviation if the lidar is a single-echo lidar, the lidar's acquisition frequency is higher than the camera's acquisition frequency, and only the lidar is currently acquiring data. This is done based on the acquisition direction of the first point cloud acquired by the lidar at the current moment and the acquisition direction of the first image data acquired by the camera at the previous reference moment. The previous reference moment is the most recent time before the current moment when the lidar and camera simultaneously acquired data. The lidar and the camera are installed on the same mobile device. The image blind spot determination module determines image acquisition blind spots based on directional deviation and a deviation threshold. The process of "determining image acquisition blind spots based on directional deviation and a deviation threshold" includes the following: if the directional deviation is greater than the deviation threshold, the first point cloud is determined as an image acquisition blind spot; if the directional deviation is less than or equal to the deviation threshold, the first point cloud acquired by the lidar at the current moment is mapped to the first image data acquired at the previous reference moment to obtain the mapping point of the first point cloud in the first image data; based on the current height information of the mapping point and the recorded height information of the surrounding points of the mapping point, the image acquisition blind spot is determined from the mapping point. The color information adding module is used to add color information to the first point cloud based on the first image data and the image acquisition blind spot. The "adding color information to the first point cloud based on the first image data and the image acquisition blind spot" includes the following: based on the determined image acquisition blind spot, obtaining the point cloud corresponding to the image acquisition blind spot in the first point cloud, and determining the point cloud corresponding to the non-image acquisition blind spot in the first point cloud; for the point cloud corresponding to the non-image acquisition blind spot, using the color information corresponding to the point cloud in the first image data as the color information of the first point cloud.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the data processing method as described in any one of claims 1 to 7.

Images