A data space synchronization method and device, a terminal device and a storage medium

Abstract

The application relates to the technical field of data processing, and discloses a data space synchronization method and device, a terminal device and a storage medium. In the method, a sample image and a sample point cloud that have completed space synchronization processing are used as a training set to train a neural network as a space synchronization model. After a target image and a target point cloud that have completed target detection are obtained, the target image and the target point cloud can be input into the space synchronization model for space synchronization processing, so that corresponding space synchronization results are obtained. The space synchronization results can specifically include each target detection frame obtained after the target image and the target point cloud complete the space synchronization processing. The method realizes space synchronization of point cloud data and image data through a neural network.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data spatial synchronization method, apparatus, terminal device, and storage medium. Background Technology

[0002] In roadside sensing devices, different sensors such as lidar and cameras need to spatially synchronize the data they collect, that is, to unify point cloud data and image data into a single coordinate system. Therefore, how to achieve spatial synchronization of point cloud data and image data from different sensors is a problem that needs to be studied by those skilled in the art. Summary of the Invention

[0003] In view of this, embodiments of this application provide a data spatial synchronization method, apparatus, terminal device, and storage medium, which can realize spatial synchronization of point cloud data and image data.

[0004] A first aspect of this application provides a data space synchronization method, including:

[0005] Acquire the target image and target point cloud to be synchronized; wherein, both the target image and the target point cloud have undergone target detection processing;

[0006] The target image and the target point cloud are input into a trained spatial synchronization model for processing to obtain the spatial synchronization result of the target image and the target point cloud; wherein, the spatial synchronization model is a neural network model trained with sample images and sample point clouds as the training set, the sample images and the sample point clouds are sample data that have completed spatial synchronization processing, and the spatial synchronization result includes each target detection box obtained after the target image and the target point cloud have completed spatial synchronization processing.

[0007] In this embodiment, a neural network is trained as a spatial synchronization model using sample images and point clouds that have undergone spatial synchronization processing as a training set. After acquiring the target image and point cloud to be synchronized that have undergone target detection, the target image and point cloud can be input into the spatial synchronization model for spatial synchronization processing to obtain the corresponding spatial synchronization result. This spatial synchronization result specifically includes the target detection boxes obtained after the target image and point cloud have undergone spatial synchronization processing. Clearly, this embodiment achieves spatial synchronization of point cloud data and image data through a neural network.

[0008] In one implementation of this application, the sample image and the sample point cloud can be obtained in the following ways:

[0009] Acquire the original image and the original point cloud at the same time and in the same scene; wherein, both the original image and the original point cloud have undergone target detection processing;

[0010] Based on the parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud, a one-to-one correspondence is constructed between each target detection box in the original image and each target detection box in the original point cloud.

[0011] After establishing the one-to-one correspondence, the original image is determined as the sample image, and the original point cloud is determined as the sample point cloud.

[0012] Furthermore, the parameters of each target detection box in the original image include the coordinates of the top-left corner and the bottom-right corner of the detection box, and the parameters of each target detection box in the original point cloud include the coordinates of the center point of the detection box; the step of constructing a one-to-one correspondence between each target detection box in the original image and each target detection box in the original point cloud based on the parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud may include:

[0013] The parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud are transformed to the same coordinate system;

[0014] Based on the coordinates of the top left and bottom right corners of each target detection box in the original image, the coordinates of the center point of each target detection box in the original image are calculated respectively.

[0015] For any target detection box in the original image, calculate the distance between the center point coordinates of the arbitrary target detection box and the center point coordinates of each target detection box in the original point cloud, and associate the target detection boxes in the original point cloud whose distance is less than a first threshold with the arbitrary target detection box.

[0016] Furthermore, the parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud also include the target type of the detection box; the step of mapping the target detection boxes in each target detection box of the original point cloud whose distance is less than the first threshold to the arbitrary target detection box may include:

[0017] The target detection boxes in each target detection box of the original point cloud whose distance is less than the first threshold and whose corresponding target type is the same as the target type corresponding to the arbitrary target detection box are matched with the arbitrary target detection box.

[0018] Furthermore, the parameters of each target detection box in the original image include the coordinates of the top-left corner and the bottom-right corner of the detection box, and the parameters of each target detection box in the original point cloud include the coordinates of the center point, length, width, and height of the detection box; the step of constructing a one-to-one correspondence between each target detection box in the original image and each target detection box in the original point cloud based on the parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud includes:

[0019] The parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud are transformed to the same coordinate system;

[0020] Based on the coordinates of the top left and bottom right corners of each target detection box in the original image, the position regions of each target detection box in the original image are calculated respectively.

[0021] Based on the center point coordinates, length, width, and height of each target detection box in the original point cloud, the position region of each target detection box in the original point cloud is calculated.

[0022] For any target detection box in the original image, the overlap between the position region of the arbitrary target detection box and the position regions of each target detection box in the original point cloud is calculated, and the target detection boxes in each target detection box of the original point cloud with an overlap greater than a second threshold are matched with the arbitrary target detection box.

[0023] In one implementation of this application, the spatial synchronization model can be trained in the following way:

[0024] The spatial synchronization model is trained by using the coordinates of the target points of each target detection box in the sample image as the training data of the training set, and using the coordinates of the center points of each target detection box in the sample point cloud as the ground truth of the training set; wherein, the target point coordinates are the coordinates of any point in the detection box.

[0025] Furthermore, before using the target point coordinates of each target detection box in the sample image as training data for the training set, the following may also be included:

[0026] The sample image is divided into multiple image regions;

[0027] For each image region, the target point coordinates of each target detection box in that image region are determined based on the position of that image region in the sample image.

[0028] Furthermore, the step of inputting the target image and the target point cloud into a trained spatial synchronization model for processing to obtain the spatial synchronization result of the target image and the target point cloud may include:

[0029] Each target detection box of the target image and each target detection box of the target point cloud are input into the spatial synchronization model for processing to obtain a one-to-one correspondence between each target detection box of the target image and each target detection box of the target point cloud.

[0030] Based on the one-to-one correspondence between each target detection box in the target image and each target detection box in the target point cloud, the target detection boxes obtained after the target image and the target point cloud have undergone spatial synchronization processing are determined.

[0031] Furthermore, determining the target detection boxes obtained after spatial synchronization processing of the target image and the target point cloud based on the one-to-one correspondence between the target detection boxes of the target image and the target point cloud may include:

[0032] For any target detection box in the target image, if a target detection box corresponding to the arbitrary target detection box exists in the target point cloud, then a target detection box obtained after spatial synchronization processing of the target image and the target point cloud is determined based on the arbitrary target detection box and the target detection box corresponding to the arbitrary target detection box; if no target detection box corresponding to the arbitrary target detection box exists in the target point cloud, then the arbitrary target detection box is determined as a target detection box obtained after spatial synchronization processing of the target image and the target point cloud.

[0033] A second aspect of this application provides a data space synchronization device, comprising:

[0034] The data acquisition module is used to acquire the target image and target point cloud to be synchronized; wherein, both the target image and the target point cloud have completed target detection processing;

[0035] The data synchronization module is used to input the target image and the target point cloud into a trained spatial synchronization model for processing, and obtain the spatial synchronization result of the target image and the target point cloud; wherein, the spatial synchronization model is a neural network model trained with sample images and sample point clouds as a training set, the sample images and sample point clouds are sample data that have completed spatial synchronization processing, and the spatial synchronization result includes each target detection box obtained after the target image and target point cloud have completed spatial synchronization processing.

[0036] A third aspect of this application provides a terminal device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the data space synchronization method provided in the first aspect of this application.

[0037] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data space synchronization method provided in the first aspect of this application.

[0038] The fifth aspect of this application provides a computer program product that, when run on a terminal device, causes the terminal device to execute the data space synchronization method provided in the first aspect of this application.

[0039] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0040] Figure 1 This is a flowchart of a data space synchronization method provided in an embodiment of this application;

[0041] Figure 2 This is a schematic diagram of one stage of the data space synchronization method provided in the embodiments of this application;

[0042] Figure 3 This is a schematic diagram illustrating the spatial synchronization processing of sample images and sample point clouds provided in the embodiments of this application;

[0043] Figure 4 This is a schematic diagram illustrating the operation of spatially synchronizing the target image and target point cloud provided in the embodiments of this application;

[0044] Figure 5 This is a schematic diagram of the structure of a data space synchronization device provided in an embodiment of this application;

[0045] Figure 6 This is a schematic diagram of a terminal device provided in an embodiment of this application. Detailed Implementation

[0046] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail. Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0047] In the field of environmental monitoring, spatial synchronization of multi-sensor data is crucial. This application proposes a data spatial synchronization method that uses a neural network to achieve spatial synchronization of data from different sensors, such as point cloud data and image data. For more specific technical details regarding the implementation of this application, please refer to the method embodiments described below.

[0048] It should be understood that the execution subject of the various method embodiments of this application can be various types of terminal devices or servers, such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), large-screen TVs, etc. The embodiments of this application do not impose any restrictions on the specific type of terminal device and server.

[0049] Please see Figure 1 This application illustrates a data space synchronization method provided in an embodiment, comprising:

[0050] 101. Obtain the target image and target point cloud to be synchronized;

[0051] First, acquire the target image and target point cloud to be synchronized. Generally, data from different sensors needs to maintain spatiotemporal consistency during use; therefore, the target image and target point cloud acquired here are typically data from the same time and scene. For example, at a highway tollbooth, both LiDAR and cameras are usually installed simultaneously to detect point cloud data and image data from the same scene, respectively. Therefore, the image captured by the camera at the same time can be acquired as the target image, and the point cloud detected by the LiDAR as the target point cloud.

[0052] In addition, target detection processing needs to be completed before spatial synchronization of the target image and target point cloud. Specifically, existing target detection methods for images can be used to process the target image to obtain the target detection boxes in the target image; existing target detection methods for point clouds can be used to process the target point cloud to obtain the target detection boxes in the target point cloud.

[0053] 102. Input the target image and the target point cloud into the trained spatial synchronization model for processing to obtain the spatial synchronization result of the target image and the target point cloud.

[0054] After obtaining the target image and target point cloud, they can be input into a trained spatial synchronization model for processing to obtain the corresponding spatial synchronization results. These results include the target detection boxes obtained after spatial synchronization processing of the target image and target point cloud, and these boxes are represented in the same coordinate system. The spatial synchronization model is a neural network model trained using sample images and sample point clouds as the training set, which are pre-processed sample data that have undergone spatial synchronization.

[0055] In one implementation of this application, the sample image and the sample point cloud can be obtained in the following ways:

[0056] (1) Obtain the original image and the original point cloud at the same time and in the same scene; wherein the original image and the original point cloud have both been processed for target detection.

[0057] (2) Based on the parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud, construct a one-to-one correspondence between each target detection box in the original image and each target detection box in the original point cloud.

[0058] (3) After constructing the one-to-one correspondence, the original image is determined as the sample image, and the original point cloud is determined as the sample point cloud.

[0059] Image data and point cloud data from the same scene at the same time are acquired. Here, we use the original image and the original point cloud as examples, respectively. Both the original image and the original point cloud have undergone object detection processing, yielding the object detection boxes contained within them. For the object detection boxes in the image data, the parameters can generally be represented as [x_min, y_min, x_max, y_max, label, id, score], which correspond to the lower left corner coordinates (x_min, y_min), upper right corner coordinates (x_max, y_max), object type (label, such as pedestrian, bicycle, car, and bus), object number (id), and corresponding type score (score) of the detection box, respectively. For object detection boxes in point cloud data, the parameters can generally be represented as [x, y, z, w, l, h, theta, label, speed, id, score], corresponding to the 3D center coordinates (x, y, z), length (l), width (w), height (h), heading angle (theta), target type (label, e.g., pedestrian, bicycle, car, and bus), target speed (speed), target number (id), and corresponding type score (score), respectively. Spatial synchronization processing between the original image and the original point cloud is required, that is, to map their object detection boxes one-to-one. In practice, a one-to-one correspondence can be constructed between the object detection boxes in the original image and the original point cloud based on their parameters. For example, the distance between object detection boxes can be determined based on their position coordinates (center coordinates or corner coordinates, etc.), thus mapping closely spaced object detection boxes to the same target. After establishing a one-to-one correspondence between the object detection boxes in the original image and the original point cloud, the original image and the original point cloud have essentially completed spatial synchronization processing. Therefore, the original image can be identified as the sample image, and the original point cloud as the sample point cloud. It should be understood that the original image here usually contains a large number of images, and the corresponding original point cloud also contains a large number of points, thus ensuring the total amount of sample data in the training set.

[0060] Specifically, the parameters of each target detection box in the original image include the coordinates of the top-left corner and the bottom-right corner of the detection box, and the parameters of each target detection box in the original point cloud include the coordinates of the center point of the detection box; the step of constructing a one-to-one correspondence between each target detection box in the original image and each target detection box in the original point cloud based on the parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud may include:

[0061] (1) Convert the parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud to the same coordinate system;

[0062] (2) Based on the coordinates of the upper left corner and the lower right corner of each target detection box in the original image, calculate the coordinates of the center point of each target detection box in the original image.

[0063] (3) For any target detection box in the original image, calculate the distance between the center point coordinates of the arbitrary target detection box and the center point coordinates of each target detection box in the original point cloud, and associate the target detection boxes in each target detection box of the original point cloud with the arbitrary target detection box.

[0064] Typically, the original image is two-dimensional data, while the original point cloud is three-dimensional data. During spatial synchronization, it's necessary to transform the target detection boxes in both the original image and the original point cloud to the same coordinate system. This can be done in two ways: firstly, by using existing methods for converting from a three-dimensional to a two-dimensional coordinate system, the data in the original point cloud (mainly containing the coordinates of each target detection box) can be transformed to the two-dimensional camera coordinate system corresponding to the original image; secondly, by using existing methods for converting from a two-dimensional to a three-dimensional coordinate system, the data in the original image (mainly containing the coordinates of each target detection box) can be transformed to the three-dimensional LiDAR coordinate system corresponding to the original point cloud. Then, since the parameters of each target detection box in the original image include the coordinates of its top-left and bottom-right corners (or, alternatively, its bottom-left and top-right corners), the center point coordinates of each target detection box can be obtained by calculating the midpoint of the line connecting the top-left and bottom-right corners. Next, since the target detection boxes in the original image and the original point cloud are represented in the same coordinate system, for a target detection box X in the original image, the distance between the center coordinates of X and the center coordinates of each target detection box in the original point cloud can be calculated. Then, the target detection boxes in the original point cloud whose distances are less than a set threshold are associated with X, indicating that the target detection box and X correspond to the same target. Obviously, for each target detection box in the original image, the same method as for target detection box X can be used to find a corresponding target detection box from the target detection boxes in the original point cloud, thereby establishing a one-to-one correspondence between the target detection boxes in the original image and the target detection boxes in the original point cloud.

[0065] Furthermore, the parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud also include the target type of the detection box; the step of associating the target detection boxes in the original point cloud whose distance is less than the first threshold with the arbitrary target detection box may include:

[0066] The target detection boxes in each target detection box of the original point cloud whose distance is less than the first threshold and whose corresponding target type is the same as the target type corresponding to the arbitrary target detection box are matched with the arbitrary target detection box.

[0067] To further improve the accuracy of spatial synchronization between the original image and the original point cloud, in addition to considering the distance between object detection boxes, an additional check can be added to determine whether the target types are the same. For example, in the example of object detection box X above, suppose that the object detection box Y in each object detection box of the original point cloud is less than a set threshold distance from the center point coordinate of X. Here, X and Y will not be directly treated as the same target and matched. Instead, it will first check whether the target types corresponding to X and Y are the same (e.g., whether the target types are both cars). Only if they are the same will X and Y be treated as the same target and matched.

[0068] Specifically, the parameters of each target detection box in the original image include the coordinates of the top-left corner and the bottom-right corner of the detection box, and the parameters of each target detection box in the original point cloud include the coordinates of the center point, length, width, and height of the detection box; the step of constructing a one-to-one correspondence between each target detection box in the original image and each target detection box in the original point cloud based on the parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud may include:

[0069] (1) Convert the parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud to the same coordinate system;

[0070] (2) Based on the coordinates of the upper left corner and the lower right corner of each target detection box in the original image, calculate the position region of each target detection box in the original image respectively;

[0071] (3) Based on the center point coordinates, length, width and height of each target detection box in the original point cloud, calculate the position region of each target detection box in the original point cloud respectively;

[0072] (4) For any target detection box in the original image, calculate the overlap between the position region of the arbitrary target detection box and the position region of each target detection box in the original point cloud, and associate the target detection boxes in each target detection box of the original point cloud with the arbitrary target detection box.

[0073] The previous section described spatial synchronization between the original point cloud and the original image by detecting the distance between the center point coordinates. This section proposes another implementation method: achieving synchronization by detecting overlap. In practice, the target detection boxes in both the original image and the original point cloud must first be transformed to the same coordinate system. Then, based on the top-left and bottom-right corner coordinates (or alternatively, the bottom-left and top-right corner coordinates) of each target detection box in the original image, the positional region of each target detection box can be calculated. A target detection box is typically a rectangle; knowing the coordinates of its two opposite corners allows for the calculation of its rectangular positional region. Similarly, based on the center point coordinates, length, width, and height of each target detection box in the original point cloud, its positional region can be calculated. Knowing the center point coordinates and dimensions of a rectangular box also allows for the calculation of its rectangular positional region. Next, since the target detection boxes in the original image and the original point cloud are represented in the same coordinate system, for a target detection box X in the original image, the overlap (Iou) between the location region of X and the location regions of each target detection box in the original point cloud can be calculated. Then, the target detection boxes in the original point cloud whose overlap is greater than a set threshold are associated with X, indicating that the target detection box and X correspond to the same target. Obviously, for each target detection box in the original image, the same method as for target detection box X can be used to find a corresponding target detection box from the target detection boxes in the original point cloud, thereby establishing a one-to-one correspondence between the target detection boxes in the original image and the target detection boxes in the original point cloud.

[0074] In one implementation of this application, the spatial synchronization model can be trained in the following way:

[0075] The spatial synchronization model is trained by using the coordinates of the target points of each target detection box in the sample image as the training data of the training set, and using the coordinates of the center points of each target detection box in the sample point cloud as the ground truth of the training set; wherein, the target point coordinates are the coordinates of any point in the detection box.

[0076] After obtaining the sample images and sample point clouds as described above, an arbitrary point (e.g., the center point, one of the four corner points, or other points within the target detection box) is selected in each target detection box of the sample image as the target point. The coordinates of the target points in each target detection box of the sample image are then used as the training data for the training set, and the coordinates of the center points of each target detection box in the sample point cloud are used as the ground truth values ​​for the training set. This process trains the spatial synchronization model. If the data is transformed to a two-dimensional coordinate system, the center point coordinates of each target detection box in the sample point cloud can take two-dimensional components; for example, only (x, y) can be used as the ground truth values ​​for the training set.

[0077] Furthermore, before using the target point coordinates of each target detection box in the sample image as training data for the training set, the following may also be included:

[0078] (1) Divide the sample image into multiple image regions;

[0079] (2) For each image region, determine the target point coordinates of each target detection box in the image region according to the position of the image region in the sample image.

[0080] When selecting target points from the object detection bounding boxes in a sample image, the sample image can first be divided into multiple image regions. For example, it can be divided into three image regions (top, middle, and bottom), or into a nine-square grid, etc. Then, for each image region, the coordinates of the target points selected for each object detection bounding box in that region can be determined based on its position in the sample image. For example, if image region A is located in the center of the sample image, the center coordinates of the object detection bounding box in image region A can be selected as the target point coordinates; if image region B is located in the lower left corner of the sample image, the lower left corner coordinates of the object detection bounding box in image region B can be selected as the target point coordinates, and so on. Research shows that the true target in the object detection bounding box located in the center of the image is generally close to the center of the bounding box, while the true target in the object detection bounding box located at the edge of the image is generally close to the edge of the bounding box. Therefore, by dividing the image into regions to select target points, the accuracy of locating true targets in the object detection bounding boxes can be improved to a certain extent, thereby improving the accuracy of subsequent distance detection between bounding boxes and enhancing the correctness of spatial synchronization operations.

[0081] During the model training phase, training can be performed using methods such as fully connected networks based on the aforementioned training data and ground truth values. The fully connected layers of the neural network can employ linear transformations, the loss function can be the mean squared error loss function, and the optimizer can be the gradient descent method. Through continuous optimization training of the neural network model, the loss value converges, ultimately yielding the spatial synchronization model described above. It should be noted that this application does not limit the type, structure, or training method of the neural network. The specific training process and principles of this neural network can be found in existing technologies and will not be elaborated upon here.

[0082] In one implementation of this application, the step of inputting the target image and the target point cloud into a trained spatial synchronization model for processing to obtain the spatial synchronization result of the target image and the target point cloud may include:

[0083] (1) Input each target detection box of the target image and each target detection box of the target point cloud into the spatial synchronization model for processing to obtain a one-to-one correspondence between each target detection box of the target image and each target detection box of the target point cloud.

[0084] (2) Based on the one-to-one correspondence between each target detection box in the target image and each target detection box in the target point cloud, determine each target detection box obtained after the target image and the target point cloud have completed spatial synchronization processing.

[0085] During the verification phase, the target image and target point cloud can be input into the spatial synchronization model for processing. This model transforms the target detection boxes in both the target image and the target point cloud to the same coordinate system. Then, target detection boxes are matched using detection distance, establishing a one-to-one correspondence between them. Based on this correspondence, the target detection boxes obtained after spatial synchronization processing of the target image and point cloud can be determined, thus yielding the corresponding spatial synchronization result. The spatial synchronization result here is the result of fusion. For example, suppose the target image has target detection boxes C1, C2, and C3, and the target point cloud has target detection boxes L1, L2, and L3, where C1 and L1 correspond, C2 and L2 correspond, and C3 and L3 correspond. Then, during the operation, C1 and L1 will be fused into a target detection box M1, C2 and L2 will be fused into a target detection box M2, and C3 and L3 will be fused into a target detection box M3. The final spatial synchronization result is M1, M2, and M3 represented in the same coordinate system. That is, M1, M2, and M3 are the target detection boxes obtained after the target image and target point cloud have completed spatial synchronization processing.

[0086] Furthermore, determining the target detection boxes obtained after spatial synchronization processing of the target image and the target point cloud based on the one-to-one correspondence between the target detection boxes of the target image and the target point cloud may include:

[0087] For any target detection box in the target image, if a target detection box corresponding to the arbitrary target detection box exists in the target point cloud, then a target detection box obtained after spatial synchronization processing of the target image and the target point cloud is determined based on the arbitrary target detection box and the target detection box corresponding to the arbitrary target detection box; if no target detection box corresponding to the arbitrary target detection box exists in the target point cloud, then the arbitrary target detection box is determined as a target detection box obtained after spatial synchronization processing of the target image and the target point cloud.

[0088] During the training phase of the spatial synchronization model, the coordinates of the target points of each target detection box in the sample image are used as the training data of the training set, and the coordinates of the center points of each target detection box in the sample point cloud are used as the ground truth of the training set. This allows the trained model to achieve spatial synchronization of data over a wider detection range. Generally, LiDAR has a smaller detection range than cameras. Therefore, if the target image and target point cloud correspond to a nearby region (within the LiDAR's detection range), the detected target detection boxes in both the target image and target point cloud are identical in number. However, if the target image and target point cloud correspond to a distant region (outside the LiDAR's detection range), only the target image contains target detection boxes, while the target point cloud will not contain any (in fact, no LiDAR data for that region was collected). For any target detection box X in the target image, if a corresponding target detection box Y exists in the target point cloud, it indicates the presence of corresponding LiDAR data. In this case, target detection boxes X and Y can be fused in the same coordinate system to obtain a single target detection box Z after spatial synchronization processing of the target image and target point cloud. Conversely, if there is no target detection box X in the target point cloud, it means that there is no corresponding LiDAR data. In this case, the target detection box X can be directly regarded as a target detection box obtained after the target image and the target point cloud have completed spatial synchronization processing.

[0089] In this embodiment, a neural network is trained as a spatial synchronization model using sample images and point clouds that have undergone spatial synchronization processing as a training set. After acquiring the target image and point cloud to be synchronized that have undergone target detection, the target image and point cloud can be input into the spatial synchronization model for spatial synchronization processing to obtain the corresponding spatial synchronization result. This spatial synchronization result specifically includes the target detection boxes obtained after the target image and point cloud have undergone spatial synchronization processing. Clearly, this embodiment achieves spatial synchronization of point cloud data and image data through a neural network.

[0090] To facilitate understanding of the data space synchronization method provided in the embodiments of this application, a practical application scenario is listed below.

[0091] like Figure 2 The diagram shown illustrates one stage of the data space synchronization method provided in this application embodiment. Figure 2 The process consists of three phases: First, the data acquisition phase, which involves acquiring sample images and sample point clouds. Next, the model training phase begins. In this phase, the sample image data serves as the training data for the training set, and the sample point cloud data serves as the ground truth for the training set. A neural network is trained to obtain the spatial synchronization model. Finally, the validation phase begins. In this phase, the target image and target point cloud to be synchronized are input into the spatial synchronization model for processing, yielding the corresponding spatial synchronization results.

[0092] During the data acquisition phase, spatial synchronization processing of sample images and sample point clouds is required. For example... Figure 3 The diagram shown illustrates the spatial synchronization processing of sample images and sample point clouds. Figure 3 In the sample point cloud acquired by the LiDAR, there are target detection boxes L1, L2, and L3, while the sample image acquired by the camera contains target detection boxes C1, C2, and C3. Using the distance matching or overlap matching methods described above, a one-to-one correspondence can be found between the target detection boxes in the sample point cloud and the target detection boxes in the sample image, such as... Figure 3 As shown, L1 corresponds to C1, L2 corresponds to C2, and L3 corresponds to C3.

[0093] During the verification phase, a schematic diagram illustrating the spatial synchronization processing of the target image and target point cloud is shown below. Figure 4 As shown. In Figure 4In this process, the target image acquired by the camera and the target point cloud acquired by the lidar are input into the spatial synchronization model. This spatial synchronization model can transform the target detection boxes C4, C5 and C6 of the target image to the same three-dimensional coordinate system as the target detection boxes L4, L5 and L6 of the target point cloud. Then, it finds the correspondence between C4 and L4, C5 and L5, and C6 and L6, thereby outputting the final spatial synchronization result (M4 obtained by fusing C4 and L4, M5 obtained by fusing C5 and L5, and M6 obtained by fusing C6 and L6).

[0094] In summary, the embodiments of this application achieve spatial synchronization of point cloud data (three-dimensional spatial data) and image data (two-dimensional spatial data) through a neural network. Furthermore, by using image data as the training data and point cloud data as the ground truth in the training set when training the neural network model, a wider range of data spatial synchronization can be achieved.

[0095] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0096] The above mainly describes a data space synchronization method. The following will describe a data space synchronization device 5.

[0097] Please see Figure 5 One embodiment of a data space synchronization device in this application includes:

[0098] The data acquisition module 501 is used to acquire the target image and target point cloud to be synchronized; wherein, both the target image and the target point cloud have completed target detection processing;

[0099] The data synchronization module 502 is used to input the target image and the target point cloud into a trained zero-space synchronization model for processing, so as to obtain the spatial synchronization result of the target image and the target point cloud;

[0100] The spatial synchronization model is a neural network model trained using sample images and sample point clouds as training sets. The sample images and sample point clouds are sample data that have undergone spatial synchronization processing. The spatial synchronization result includes the target detection boxes obtained after the target images and target point clouds have undergone spatial synchronization processing.

[0101] 5. In one implementation of the embodiments of this application, the data space synchronization device may further include:

[0102] The sample data acquisition module is used to acquire the original image and the original point cloud at the same time and in the same scene; wherein, both the original image and the original point cloud have completed target detection processing;

[0103] The correspondence construction module is used to construct a one-to-one correspondence between each target detection box in the original image and each target detection box in the original point cloud based on the parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud.

[0104] The sample data determination module is used to determine the original image as the sample image and the original point cloud as the sample point cloud after constructing the one-to-one correspondence.

[0105] Furthermore, the parameters of each target detection box in the original image include the coordinates of the top-left corner and the bottom-right corner of the detection box, and the parameters of each target detection box in the original point cloud include the coordinates of the center point 5 of the detection box; the correspondence construction module may include:

[0106] The first coordinate transformation unit is used to transform the parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud to the same coordinate system.

[0107] The center point coordinate calculation unit is used to calculate the center point coordinates of each target detection box in the original image based on the coordinates of the upper left corner and the lower right corner of each target detection box in the original image.

[0108] The first detection box corresponding unit is used to calculate the distance between the center point coordinates of any target detection box in the original image and the center point coordinates of each target detection box in the original point cloud, and to correspond the target detection boxes in the original point cloud whose distance is less than a first threshold with the arbitrary target detection box.

[0109] Furthermore, the parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud also include the target type of the detection box; the first detection box corresponding unit can be specifically used to: correspond the target detection boxes in each target detection box of the original point cloud whose distance is less than the first threshold and whose corresponding target type is the same as the target type corresponding to the arbitrary target detection box to the arbitrary target detection box.

[0110] Furthermore, the parameters of each target detection box in the original image include the coordinates of the top-left corner and the bottom-right corner of the detection box, and the parameters of each target detection box in the original point cloud include the coordinates of the center point, length, width, and height of the detection box; the correspondence construction module may include:

[0111] The first coordinate transformation unit is used to transform the parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud to the same coordinate system.

[0112] The first location region calculation unit is used to calculate the location region of each target detection box in the original image based on the upper left corner coordinates and lower right corner coordinates of each target detection box in the original image.

[0113] The second location region calculation unit is used to calculate the location region of each target detection box in the original point cloud based on the center point coordinates, length, width and height of each target detection box in the original point cloud.

[0114] The second detection box corresponding unit is used to calculate the overlap between the position region of any target detection box in the original image and the position region of each target detection box in the original point cloud, and to correspond the target detection boxes in the original point cloud whose overlap is greater than a second threshold with the arbitrary target detection box.

[0115] In one implementation of this application, the data space synchronization device may further include:

[0116] The model training module is used to train the spatial synchronization model by using the coordinates of the target points of each target detection box in the sample image as the training data of the training set and the coordinates of the center points of each target detection box in the sample point cloud as the ground truth of the training set; wherein, the target point coordinates are the coordinates of any point in the detection box.

[0117] Furthermore, the data spatial synchronization device may also include:

[0118] An image segmentation module is used to divide the sample image into multiple image regions;

[0119] The target point determination module is used to determine the target point coordinates of each target detection box in each image region based on the position of the image region in the sample image.

[0120] In one implementation of this application, the data synchronization module may include:

[0121] The data spatial synchronization unit is used to input each target detection box of the target image and each target detection box of the target point cloud into the spatial synchronization model for processing, so as to obtain a one-to-one correspondence between each target detection box of the target image and each target detection box of the target point cloud.

[0122] The target detection box determination unit is used to determine each target detection box obtained after the target image and the target point cloud have completed spatial synchronization processing, based on the one-to-one correspondence between each target detection box in the target image and each target detection box in the target point cloud.

[0123] Furthermore, the target detection box determination unit can be specifically used to: for any target detection box in the target image, if there is a target detection box in the target point cloud corresponding to the arbitrary target detection box, then determine a target detection box obtained after spatial synchronization processing of the target image and the target point cloud based on the arbitrary target detection box and the target detection box corresponding to the arbitrary target detection box; if there is no target detection box in the target point cloud corresponding to the arbitrary target detection box, then determine the arbitrary target detection box as a target detection box obtained after spatial synchronization processing of the target image and the target point cloud.

[0124] This application embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements... Figure 1 This represents any data space synchronization method.

[0125] This application also provides a computer program product that, when run on a terminal device, causes the terminal device to perform actions such as... Figure 1 This represents any data space synchronization method.

[0126] Figure 6 This is a schematic diagram of a terminal device provided in an embodiment of this application. For example... Figure 6 As shown, the terminal device 6 in this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, it implements the steps in the embodiments of the various data space synchronization methods described above, for example... Figure 1 Steps 101 to 102 are shown. Alternatively, when the processor 60 executes the computer program 62, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 5 The functions of modules 501 to 502 are shown.

[0127] The computer program 62 can be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 62 in the terminal device 6.

[0128] The processor 60 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0129] The memory 61 can be an internal storage unit of the terminal device 6, such as a hard disk or memory of the terminal device 6. The memory 61 can also be an external storage device of the terminal device 6, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device 6. Furthermore, the memory 61 can include both internal and external storage units of the terminal device 6. The memory 61 is used to store the computer program and other programs and data required by the terminal device. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0130] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0131] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0132] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0133] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0134] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0135] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0136] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0137] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0138] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data spatial synchronization method, characterized in that, include: Acquire the target image and target point cloud to be synchronized; wherein, both the target image and the target point cloud have undergone target detection processing; The target image and the target point cloud are input into a trained spatial synchronization model for processing to obtain the spatial synchronization result of the target image and the target point cloud; wherein, the spatial synchronization model is a neural network model trained with sample images and sample point clouds as the training set, the sample images and sample point clouds are sample data that have completed spatial synchronization processing, and the spatial synchronization result includes each target detection box obtained after the target image and target point cloud have completed spatial synchronization processing; The sample images and the sample point clouds were obtained in the following ways: Acquire the original image and the original point cloud at the same time and in the same scene; wherein, both the original image and the original point cloud have undergone target detection processing; Based on the parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud, a one-to-one correspondence is constructed between each target detection box in the original image and each target detection box in the original point cloud. After establishing the one-to-one correspondence, the original image is determined as the sample image, and the original point cloud is determined as the sample point cloud; The spatial synchronization model is trained in the following manner: The spatial synchronization model is trained by using the coordinates of the target points of each target detection box in the sample image as the training data of the training set, and using the coordinates of the center points of each target detection box in the sample point cloud as the ground truth of the training set; wherein, the target point coordinates are the coordinates of any point in the detection box.

2. The method as described in claim 1, characterized in that, The parameters of each target detection box in the original image include the coordinates of the top-left corner and the bottom-right corner of the detection box, and the parameters of each target detection box in the original point cloud include the coordinates of the center point of the detection box; the step of constructing a one-to-one correspondence between each target detection box in the original image and each target detection box in the original point cloud based on the parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud includes: The parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud are transformed to the same coordinate system; Based on the coordinates of the top left and bottom right corners of each target detection box in the original image, the coordinates of the center point of each target detection box in the original image are calculated respectively. For any target detection box in the original image, calculate the distance between the center point coordinates of the arbitrary target detection box and the center point coordinates of each target detection box in the original point cloud, and associate the target detection boxes in the original point cloud whose distance is less than a first threshold with the arbitrary target detection box.

3. The method as described in claim 2, characterized in that, The parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud also include the target type of the detection box. The step of associating the target detection boxes whose distance is less than a first threshold with any target detection box in each target detection box of the original point cloud includes: The target detection boxes in each target detection box of the original point cloud whose distance is less than the first threshold and whose corresponding target type is the same as the target type corresponding to the arbitrary target detection box are matched with the arbitrary target detection box.

4. The method as described in claim 1, characterized in that, The parameters of each target detection box in the original image include the coordinates of the top-left corner and the bottom-right corner of the detection box. The parameters of each target detection box in the original point cloud include the coordinates of the center point, length, width, and height of the detection box. The step of constructing a one-to-one correspondence between the target detection boxes in the original image and the target detection boxes in the original point cloud based on their parameters includes: The parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud are transformed to the same coordinate system; Based on the coordinates of the top left and bottom right corners of each target detection box in the original image, the position regions of each target detection box in the original image are calculated respectively. Based on the center point coordinates, length, width, and height of each target detection box in the original point cloud, the position region of each target detection box in the original point cloud is calculated. For any target detection box in the original image, the overlap between the position region of the arbitrary target detection box and the position regions of each target detection box in the original point cloud is calculated, and the target detection boxes in each target detection box of the original point cloud with an overlap greater than a second threshold are matched with the arbitrary target detection box.

5. The method as described in claim 1, characterized in that, Before using the target point coordinates of each target detection box in the sample image as training data for the training set, the method further includes: The sample image is divided into multiple image regions; For each image region, the target point coordinates of each target detection box in that image region are determined based on the position of that image region in the sample image.

6. The method as described in claim 1, characterized in that, The step of inputting the target image and the target point cloud into a trained spatial synchronization model for processing to obtain the spatial synchronization result of the target image and the target point cloud includes: Each target detection box of the target image and each target detection box of the target point cloud are input into the spatial synchronization model for processing to obtain a one-to-one correspondence between each target detection box of the target image and each target detection box of the target point cloud. Based on the one-to-one correspondence between each target detection box in the target image and each target detection box in the target point cloud, the target detection boxes obtained after the target image and the target point cloud have undergone spatial synchronization processing are determined.

7. The method as described in claim 6, characterized in that, The step of determining the target detection boxes obtained after spatial synchronization processing of the target image and the target point cloud based on the one-to-one correspondence between the target detection boxes of the target image and the target point cloud includes: For any target detection box in the target image, if a target detection box corresponding to the arbitrary target detection box exists in the target point cloud, then a target detection box obtained after spatial synchronization processing of the target image and the target point cloud is determined based on the arbitrary target detection box and the target detection box corresponding to the arbitrary target detection box; if no target detection box corresponding to the arbitrary target detection box exists in the target point cloud, then the arbitrary target detection box is determined as a target detection box obtained after spatial synchronization processing of the target image and the target point cloud.

8. A data spatial synchronization device, characterized in that, include: The data acquisition module is used to acquire the target image and target point cloud to be synchronized; wherein, both the target image and the target point cloud have completed target detection processing; The data synchronization module is used to input the target image and the target point cloud into a trained spatial synchronization model for processing, and obtain the spatial synchronization result of the target image and the target point cloud; wherein, the spatial synchronization model is a neural network model trained with sample images and sample point clouds as a training set, the sample images and sample point clouds are sample data that have completed spatial synchronization processing, and the spatial synchronization result includes each target detection box obtained after the target image and target point cloud have completed spatial synchronization processing; The sample data acquisition module is used to acquire the original image and the original point cloud at the same time and in the same scene; wherein, both the original image and the original point cloud have completed target detection processing; The correspondence construction module is used to construct a one-to-one correspondence between each target detection box in the original image and each target detection box in the original point cloud based on the parameters of each target detection box in the original image and the parameters of each target detection box in the original point cloud. The sample data determination module is used to determine the original image as the sample image and the original point cloud as the sample point cloud after constructing the one-to-one correspondence. The model training module is used to train the spatial synchronization model by using the coordinates of the target points of each target detection box in the sample image as the training data of the training set and the coordinates of the center points of each target detection box in the sample point cloud as the ground truth of the training set; wherein, the target point coordinates are the coordinates of any point in the detection box.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data space synchronization method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the data space synchronization method as described in any one of claims 1 to 7.

Images