A three-dimensional reconstruction method, device, equipment and medium
By acquiring continuous multi-frame three-dimensional data and images, using the corresponding relationship between pixel information and three-dimensional data in the image to complete the problem of sparse single-frame radar point cloud data, it generates dense and complete shape three-dimensional point cloud data, improving Labeling accuracy and object recognition effect.
Patent Information
- Application Number
- CN202411149107.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-21
AI Technical Summary
In the prior art, the three-dimensional data obtained based on single-frame radar point cloud data is sparse, and the structure and shape information of the object cannot be accurately restored, which affects the labeling accuracy and object recognition effect of subsequent three-dimensional point cloud data.
By acquiring three-dimensional data and images in a continuous multi-frame, the corresponding relationship between pixel information and three-dimensional data in the image is used to complete the three-dimensional data, and the color information of the target object is correlated to generate dense and complete shape three-dimensional point cloud data.
It realizes the acquisition of accurate three-dimensional point cloud data, improves the reflection of object shape and color information, and enhances the density and labeling accuracy of three-dimensional point cloud data.
Smart Images

Figure CN118657896B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of three-dimensional reconstruction, and in particular to a three-dimensional reconstruction method, device, equipment and medium. Background Art
[0002] Target detection, classification, and recognition based on three-dimensional (3D) point cloud data are the main technologies currently used to solve the problem of scene analysis. In related technologies, the single-frame radar point cloud data (i.e., 3D point cloud data) acquired in real time is often sparse and cannot restore the structural shape information of the object, affecting the accuracy of further operations (such as annotation) based on 3D point cloud data in subsequent processes. Summary of the invention
[0003] In order to overcome the problems existing in the related art, the present disclosure provides a three-dimensional reconstruction method, device, equipment and medium.
[0004] According to a first aspect of an embodiment of the present disclosure, a three-dimensional reconstruction method is provided, comprising: acquiring a first three-dimensional data set and a first image set, the first three-dimensional data set comprising a plurality of consecutive frames of first three-dimensional data, the first image set comprising a plurality of consecutive frames of first images, the plurality of frames of first three-dimensional data and the plurality of frames of first images being in one-to-one correspondence, the plurality of frames of first three-dimensional data all comprising three-dimensional data corresponding to a target object, and the plurality of frames of first images all comprising images of the target object; acquiring a second image set according to the first three-dimensional data set and the first image set, the second image set comprising a plurality of consecutive frames of second images with the first three-dimensional data; for each frame of the second image in the second image set, associating the three-dimensional data with each pixel in the second image according to the pixel information of the first three-dimensional data and the second image included in the second image, to obtain a third image set; acquiring target three-dimensional data corresponding to a target time frame according to the three-dimensional data contained in each frame of the third image in the third image set, the target time frame corresponding to a single frame of the first image in the first image set; associating the target three-dimensional data with color information of a corresponding target object, to obtain target three-dimensional data associated with color information, the corresponding target object being a target object included in the single frame of the first image.
[0005] In one embodiment, obtaining a second image set according to the first three-dimensional data set and the first image set includes: for each frame of the first image contained in the first image set, mapping the first three-dimensional data corresponding to the first image in the first three-dimensional data set to the first image according to a first coordinate transformation relationship to obtain the second image, wherein the first coordinate transformation relationship is a correspondence between pixel coordinates in the first image and spatial coordinates in the first three-dimensional data; determining a set of all second images as the second image set; wherein mapping the first three-dimensional data corresponding to the first image in the first three-dimensional data set to the first image according to the first coordinate transformation relationship includes: mapping each three-dimensional coordinate contained in the first three-dimensional data to a corresponding pixel in the corresponding first image according to the first coordinate transformation relationship.
[0006] In one embodiment, for each frame of the second image in the second image set, according to the first three-dimensional data included in the second image and the pixel information of the second image, each pixel in the second image is associated with three-dimensional data to obtain a third image set, including: for each frame of the second image in the second image set, a pixel associated with a three-dimensional coordinate in the second image is determined as a first pixel, and a pixel not associated with a three-dimensional coordinate in the second image is determined as a second pixel, and the three-dimensional coordinate associated with the second pixel is determined according to the correspondence between the pixel information and the three-dimensional coordinate, the three-dimensional coordinate associated with the first pixel, the pixel information of the first pixel and the pixel information of the second pixel, so that the second pixels in the second image are all associated with the corresponding three-dimensional coordinates to obtain a third image, and the pixel information includes the pixel coordinates and color channel values of the pixels; and the set of all third images is determined as the third image set.
[0007] In one embodiment, the target object includes one or more objects contained in each first image frame of the multiple first images; the acquiring of target three-dimensional data corresponding to the target time frame according to the three-dimensional data contained in each third image frame in the third image set includes: for each third image frame in the third image set, determining one or more target objects in the third image according to image semantics, and identifying an image area corresponding to each target object in the third image to obtain a fourth image, and determining all the obtained fourth images as a fourth image set; determining a set of matching image areas corresponding to each target object according to every two adjacent fourth images in the fourth image set, and determining a data conversion relationship, The matching image area set includes matching image areas of multiple consecutive feature matches, and the data conversion relationship includes the three-dimensional coordinate conversion relationship between the three-dimensional data corresponding to each two adjacent matching image areas in the matching image area set; for the matching image area set corresponding to each target object, the three-dimensional data included in each matching image area is spliced frame by frame according to the data conversion relationship, and the spliced three-dimensional data is determined as the first target three-dimensional data of the target object, the first target three-dimensional data is a three-dimensional coordinate set in a reference coordinate system, and the reference coordinate system is a world coordinate system established based on a preset origin; according to the first target three-dimensional data of each target object, the target three-dimensional data corresponding to the target time frame is obtained.
[0008] In one embodiment, the method of determining a set of matching image areas corresponding to each target object and determining a data conversion relationship based on every two adjacent fourth images in the fourth image set includes: determining a set of matching image areas corresponding to each target object based on a pixel feature matching relationship between every two adjacent fourth images in the fourth image set, the matching image area set including matching image areas of multiple consecutive feature matchings; for every two adjacent matching image areas in the matching image area set corresponding to each target object, determining a data conversion relationship based on a correspondence relationship between pixel information and three-dimensional data in the fourth image, and corresponding pixel coordinates and posture information sets of the two adjacent matching image areas in the fourth image, respectively. The posture information set includes posture information of the data acquisition device when acquiring each frame of the first image and the first three-dimensional data.
[0009] In one embodiment, in the process of determining one or more target objects in a third image according to image semantics and identifying an image region corresponding to each target object in the third image to obtain a fourth image, the method further includes: determining global object information of the third image and associating the global object information with the obtained fourth image, wherein the global object information is all target objects contained in the third image; obtaining target three-dimensional data corresponding to the target time frame according to the first target three-dimensional data of each target object includes: determining a fourth image corresponding to the target time frame, determining the fourth image corresponding to the target time frame as a target fourth image, and determining the target objects contained in the target fourth image according to the global object information associated with the target fourth image; The first target three-dimensional data corresponding to the target object contained in the target fourth image is determined as the second target three-dimensional data, and the second target three-dimensional data is mapped to the target coordinate system according to the posture information set and the target data conversion relationship to obtain the target three-dimensional data; wherein the target data conversion relationship is the data conversion relationship corresponding to the target object contained in the target fourth image, the posture information set includes the posture information of the data acquisition device when acquiring each frame of the first image and the first three-dimensional data, the posture information has a corresponding relationship with the coordinate system established based on the data acquisition device, the target coordinate system is the coordinate established based on the data acquisition device when the data acquisition device acquires the target first image, and the target first image is the first image corresponding to the target fourth image.
[0010] In one embodiment, the color information of the corresponding target object corresponds to the color channel value of the pixel in the fourth image; associating the target three-dimensional data with the color information of the corresponding target object includes: associating the target color channel value to each three-dimensional coordinate contained in the target three-dimensional data according to the correspondence between the pixel information and the three-dimensional data in the fourth image, the target data conversion relationship and the posture information set, and the target color channel value is the color channel value contained in the pixel information of the fourth image.
[0011] According to a second aspect of an embodiment of the present disclosure, a three-dimensional reconstruction device is provided, comprising: an acquisition unit, configured to acquire a first three-dimensional data set and a first image set, wherein the first three-dimensional data set includes a plurality of consecutive frames of first three-dimensional data, and the first image set includes a plurality of consecutive frames of first images, wherein the plurality of frames of first three-dimensional data and the plurality of frames of first images have a one-to-one correspondence, wherein the plurality of frames of first three-dimensional data all include three-dimensional data corresponding to a target object, and wherein the plurality of frames of first images all include images including the target object; and a processing unit, configured to acquire a second image set according to the first three-dimensional data set and the first image set, wherein the second image set includes a plurality of consecutive frames of second images with first three-dimensional data, and for each frame of the second image in the second image set, according to pixel information of the first three-dimensional data and the second image included in the second image, associate the three-dimensional data with each pixel in the second image to obtain a third image set, and according to the three-dimensional data included in each frame of the third image in the third image set, acquire target three-dimensional data corresponding to a target time frame, wherein the target time frame corresponds to a single frame of the first image in the first image set, associate the target three-dimensional data with color information of a corresponding target object, and obtain target three-dimensional data associated with color information, wherein the corresponding target object is a target object included in the single frame of the first image.
[0012] In one embodiment, the processing unit obtains the second image set according to the first three-dimensional data set and the first image set in the following manner: for each frame of the first image contained in the first image set, the first three-dimensional data corresponding to the first image in the first three-dimensional data set is mapped to the first image according to a first coordinate transformation relationship to obtain the second image, wherein the first coordinate transformation relationship is the correspondence between the pixel coordinates in the first image and the spatial coordinates in the first three-dimensional data; the set of all second images is determined as the second image set; wherein the processing unit maps the first three-dimensional data corresponding to the first image in the first three-dimensional data set to the first image according to the first coordinate transformation relationship in the following manner: according to the first coordinate transformation relationship, each three-dimensional coordinate contained in the first three-dimensional data is mapped to a corresponding pixel in the corresponding first image.
[0013] In one embodiment, the processing unit uses the following method to associate three-dimensional data with each pixel in the image for each frame of the second image in the second image set according to the first three-dimensional data included in the second image and the pixel information of the second image, so as to obtain a third image: for each frame of the second image in the second image set, determine the pixel associated with the three-dimensional coordinates in the second image as the first pixel, and determine the pixel not associated with the three-dimensional coordinates in the second image as the second pixel, and determine the three-dimensional coordinates associated with the second pixel according to the correspondence between the pixel information and the three-dimensional coordinates, the three-dimensional coordinates associated with the first pixel, the pixel information of the first pixel and the pixel information of the second pixel, so that the second pixels in the second image are all associated with the corresponding three-dimensional coordinates to obtain a third image, and the pixel information includes the pixel coordinates and color channel values of the pixels; and determine the set of all third images as the third image set.
[0014] In one embodiment, the target object includes one or more objects contained in each first image frame of the multiple first image frames; the processing unit obtains target three-dimensional data corresponding to the target time frame according to the three-dimensional data contained in each third image frame in the third image set in the following manner, including: for each third image frame in the third image set, determining one or more target objects in the third image according to image semantics, and identifying the image area corresponding to each target object in the third image to obtain a fourth image, and determining all the obtained fourth images as a fourth image set; determining a set of matching image areas corresponding to each target object according to every two adjacent fourth images in the fourth image set, and determining the data Conversion relationship, the matching image area set includes matching image areas of multiple consecutive feature matches, the data conversion relationship includes the three-dimensional coordinate conversion relationship between the three-dimensional data corresponding to each two adjacent matching image areas in the matching image area set; for the matching image area set corresponding to each target object, the three-dimensional data included in each matching image area are spliced frame by frame according to the data conversion relationship, and the spliced three-dimensional data are determined as the first target three-dimensional data of the target object, the first target three-dimensional data is a three-dimensional coordinate set in a reference coordinate system, and the reference coordinate system is a world coordinate system established based on a preset origin; according to the first target three-dimensional data of each target object, the target three-dimensional data corresponding to the target time frame is obtained.
[0015] In one embodiment, the processing unit determines the set of matching image areas corresponding to each target object and determines the data conversion relationship based on every two adjacent fourth images in the fourth image set in the following manner: determines the set of matching image areas corresponding to each target object based on the pixel feature matching relationship between every two adjacent fourth images in the fourth image set, and the matching image area set includes matching image areas with multiple consecutive feature matches; for every two adjacent matching image areas in the matching image area set corresponding to each target object, determines the data conversion relationship based on the correspondence between pixel information and three-dimensional data in the fourth image, and the corresponding pixel coordinates and posture information sets of the two adjacent matching image areas in the fourth image, respectively. The posture information set includes the posture information of the data acquisition device when acquiring each frame of the first image and the first three-dimensional data.
[0016] In one embodiment, in the process of determining one or more target objects in a third image according to image semantics and identifying an image area corresponding to each target object in the third image to obtain a fourth image, the processing unit is further used to: determine global object information of the third image, and associate the global object information with the obtained fourth image, where the global object information is all target objects contained in the third image; the processing unit obtains target three-dimensional data corresponding to the target time frame according to the first target three-dimensional data of each target object in the following manner: determine a fourth image corresponding to the target time frame, determine the fourth image corresponding to the target time frame as a target fourth image, and determine the target objects contained in the target fourth image according to the global object information associated with the target fourth image. object; determining the first target three-dimensional data corresponding to the target object contained in the target fourth image as the second target three-dimensional data, and mapping the second target three-dimensional data to the target coordinate system according to the posture information set and the target data conversion relationship, so as to obtain the target three-dimensional data; wherein the target data conversion relationship is the data conversion relationship corresponding to the target object contained in the target fourth image, the posture information set includes the posture information of the data acquisition device when acquiring each frame of the first image and the first three-dimensional data, the posture information has a corresponding relationship with the coordinate system established based on the data acquisition device, the target coordinate system is the coordinate established based on the data acquisition device when the data acquisition device acquires the target first image, and the target first image is the first image corresponding to the target fourth image.
[0017] In one embodiment, the color information of the corresponding target object corresponds to the color channel value of the pixel in the fourth image; the processing unit associates the target three-dimensional data with the color information of the corresponding target object in the following manner: according to the correspondence between the pixel information and the three-dimensional data in the fourth image, the target data conversion relationship and the posture information set, the target color channel value is associated with each three-dimensional coordinate contained in the target three-dimensional data, and the target color channel value is the color channel value contained in the pixel information of the fourth image.
[0018] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to: execute the three-dimensional reconstruction method described in the first aspect or any one of the embodiments of the first aspect.
[0019] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the three-dimensional reconstruction method described in the first aspect or any one of the embodiments of the first aspect.
[0020] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: obtaining a first three-dimensional data set including a plurality of consecutive frames of first three-dimensional data, and obtaining a first image set including a plurality of consecutive frames of first images, wherein the plurality of frames of first three-dimensional data and the plurality of frames of first images correspond one to one, the plurality of frames of first three-dimensional data all include three-dimensional data corresponding to the target object, and the plurality of frames of first images all include images including the target object. According to the first three-dimensional data set and the first image set, obtaining a second image set, wherein the second image set includes a plurality of consecutive frames of second images with first three-dimensional data. For each frame of the second image in the second image set, according to the pixel information of the first three-dimensional data and the second image included in the second image, each pixel in the second image is associated with three-dimensional data, thereby obtaining a third image set. According to the three-dimensional data contained in each frame of the third image in the third image set, obtaining target three-dimensional data corresponding to the target time frame. Through the present disclosure, according to the first images and the first three-dimensional data corresponding to each other in a plurality of consecutive frames, obtaining a plurality of consecutive frames of second images including the first three-dimensional data. And according to the pixel information of the second image, each pixel in the second image is associated with three-dimensional data, thereby obtaining a third image set, and then obtaining target three-dimensional data, thereby obtaining a dense and shape-complete three-dimensional point cloud of the object.
[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0023] Figure 1 The figure is a flow chart of a 3D reconstruction method according to an exemplary embodiment.
[0024] Figure 2 The figure is a flowchart of a method for acquiring a second image set according to an exemplary embodiment.
[0025] Figure 3 The figure is a flowchart of a method for acquiring a second image set according to an exemplary embodiment.
[0026] Figure 4 The figure is a flowchart of a method for acquiring a third image set according to an exemplary embodiment.
[0027] Figure 5 The present invention is a flowchart of a method for acquiring three-dimensional data of a target according to an exemplary embodiment.
[0028] Figure 6 The present invention is a flow chart of a method for determining a set of matching image regions and determining a data conversion relationship according to an exemplary embodiment.
[0029] Figure 7 The present invention is a flowchart of a method for acquiring three-dimensional data of a target according to an exemplary embodiment.
[0030] Figure 8 The figure is a block diagram of a 3D reconstruction device according to an exemplary embodiment.
[0031] Fig. 9 It is a block diagram of an electronic device for three-dimensional reconstruction according to an exemplary embodiment. DETAILED DESCRIPTION
[0032] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.
[0033] The three-dimensional reconstruction method provided by the embodiment of the present disclosure is applied to scenes in autonomous driving where a three-dimensional model corresponding to objects in the surrounding environment is constructed based on collected image information and three-dimensional data.
[0034] As artificial intelligence technology is widely used in life and production, three-dimensional (3D) reconstruction (or 3D modeling) technology has become a key component of the visual part of artificial intelligence and an important research field in computer graphics and computer vision. 3D reconstruction technology constructs a 3D model corresponding to the shape of the object by collecting the 3D data of the corresponding object (3D point cloud data, that is, a large number of spatial coordinates that fit the surface of the object). The goal of 3D reconstruction is to capture the 3D shape and appearance of objects and scenes with high quality. The model obtained by 3D reconstruction can better reflect the characteristics of objects in the real world. 3D reconstruction technology can be applied to 3D point cloud annotation technology, that is, after acquiring 3D data and constructing a 3D model, specific information is associated in the constructed 3D model. 3D point cloud annotation technology is a key technology in the field of 3D data processing. 3D point cloud annotation technology involves associating point cloud data in 3D space with specific semantic information, which is used to identify and classify different objects in the scene, such as vehicles, pedestrians, buildings, etc. 3D point cloud annotation technology is widely used in fields such as autonomous driving, robot navigation, and Geographic Information System (GIS). In one example, 3D point cloud annotation technology can create a detailed three-dimensional semantic map by assigning a label to each point in the point cloud. 3D point cloud annotation technology is essential for the training and verification of machine learning and deep learning models. However, the annotation effect of 3D point cloud annotation technology is affected by the density of three-dimensional data (or 3D point cloud data). In related technologies, the 3D point cloud data collected in real time by the three-dimensional data acquisition device in motion is often relatively sparse and cannot accurately restore the structural shape information of the object, affecting the accuracy of semantic segmentation after annotation based on 3D point cloud data in subsequent processes.
[0035] In the related technology, 3D reconstruction technology can be applied to the field of autonomous driving, such as performing 3D point cloud annotation tasks when autonomous driving is performed in a 3D point cloud scene. The 3D model corresponding to the scene near the car is collected and constructed through 3D reconstruction technology, and then the 3D model and other information are obtained and the driving of the car is controlled. In one example, 3D reconstruction technology is applied to 3D point cloud annotation in the scenario of autonomous driving of the car, and the 3D data required for the 3D point cloud annotation task is obtained in the following way: single-frame 3D point cloud data of the nearby scene is obtained by scanning with a laser radar (Light Detection and Ranging, LiDAR), and 3D point cloud annotation is performed based on the obtained single-frame 3D point cloud data. When obtaining 3D point cloud data through a laser radar, the laser radar emits laser pulses and receives reflected light waves, actively scans nearby scenes and objects, measures the distance and speed of objects, and obtains radar point cloud data (i.e., 3D point cloud data) for 3D point cloud annotation. The solution of obtaining three-dimensional point cloud data through LiDAR has the following problems: 1. The amount of point cloud data obtained through a single-frame scan is limited, resulting in sparse 3D point cloud data, which will make it impossible to accurately represent some important scene information in autonomous driving (lane lines, traffic lights, etc.), affecting the accuracy of the labeling task. 2. When only three-dimensional point cloud data is obtained through LiDAR, the obtained data only includes a large number of spatial coordinates, that is, only contains information such as the spatial position and reflectivity of the object, but not the color information of the object, resulting in the inability of the labeler to effectively distinguish the boundaries between different objects during the labeling process. 3. When a single frame is scanned, only the point cloud data of a certain perspective of the object can be obtained. In the obtained single-frame point cloud data, the shape of the object is usually incomplete, so that the labeler needs to use his brain to estimate the shape parameters of the object during the labeling process, which affects the accuracy of the labeling results.
[0036] In view of this, the present disclosure proposes a three-dimensional reconstruction method. A correspondingly arranged three-dimensional data acquisition device (i.e., laser radar) and an image acquisition device (such as a camera) are used to synchronously acquire continuous multi-frame three-dimensional data (i.e., radar point cloud data acquired by the laser radar) and continuous multi-frame images, and a first image set is formed by continuous multi-frame image data, and a first three-dimensional data set is formed by continuous multi-frame three-dimensional data. The continuous multi-frame three-dimensional data is correspondingly associated with the continuous multi-frame image, so that each image contains three-dimensional data. For the image with continuous multi-frames associated with three-dimensional data, the three-dimensional data is supplemented according to the three-dimensional data associated with the image and the pixel information of the pixels in the image, so that each pixel in the image is associated with the corresponding three-dimensional data. According to the three-dimensional data contained in the image after the continuous multi-frame three-dimensional data is supplemented, the target three-dimensional data corresponding to the target time frame is obtained, and the color information of the pixels in the corresponding image is also correspondingly associated with the target three-dimensional data, so as to obtain the target three-dimensional data containing the color information of the target object. Through the present disclosure, dense three-dimensional point cloud data that can accurately reflect the shape and color information of the object is obtained.
[0037] Figure 1 FIG. 1 is a flow chart of a three-dimensional reconstruction method according to an exemplary embodiment. Figure 1 As shown, the method includes steps S101 to S105.
[0038] In step S101, a first three-dimensional data set and a first image set are acquired, wherein the first three-dimensional data set includes a plurality of consecutive frames of first three-dimensional data, and the first image set includes a plurality of consecutive frames of first images, and the plurality of frames of first three-dimensional data and the plurality of frames of first images correspond one to one.
[0039] The multiple frames of first three-dimensional data all include three-dimensional data corresponding to the target object, and the multiple frames of first images all include images of the target object.
[0040] In step S102, a second image set is acquired according to the first three-dimensional data set and the first image set, where the second image set includes a plurality of consecutive frames of second images with the first three-dimensional data.
[0041] In step S103, for each frame of the second image in the second image set, three-dimensional data is associated with each pixel in the second image according to the first three-dimensional data included in the second image and the pixel information of the second image, so as to obtain a third image set.
[0042] In step S104, target three-dimensional data corresponding to a target time frame is acquired according to the three-dimensional data contained in each frame of the third image in the third image set, and the target time frame corresponds to a single frame of the first image in the first image set.
[0043] In step S105, the target three-dimensional data is associated with the color information of the corresponding target object to obtain the target three-dimensional data associated with the color information, and the corresponding target object is the target object included in the single-frame first image.
[0044] In the embodiment of the present disclosure, a data acquisition device is used to synchronously acquire continuous multiple frames of three-dimensional data (first three-dimensional data) and images (first images). The data acquisition device includes a correspondingly set three-dimensional data acquisition device (i.e., a laser radar) and an image acquisition device (such as a camera). There is a corresponding relationship between the three-dimensional data and the image acquired by the two, and the three-dimensional coordinates contained in the three-dimensional data correspond to the pixel coordinates in the image. The data acquisition device in the present disclosure is set on a device with displacement capability (such as a car). Considering the relative positional relationship between the device and the surrounding environment, the target object contained in the continuous multiple frames of three-dimensional data and images may maintain a static posture in the multiple frames of images, or present a moving posture in the multiple frames of images.
[0045] In the disclosed embodiment, after acquiring a plurality of continuous frames of first three-dimensional data and a first image, a first image set is formed by the plurality of continuous frames of first image data, and a first three-dimensional data set is formed by the plurality of continuous frames of first three-dimensional data. According to the correspondence between the first image and the first three-dimensional data, the plurality of continuous frames of three-dimensional data are correspondingly associated with the plurality of continuous frames of images, so that each image contains three-dimensional data, that is, part of the pixels in each first image are associated with the corresponding three-dimensional coordinates in the first three-dimensional data, thereby acquiring a second image containing the first three-dimensional data, and acquiring a second image set formed by the plurality of continuous frames of second images.
[0046] In the disclosed embodiment, after acquiring the second image set, for each frame of the second image set, three-dimensional data is completed based on the first three-dimensional data associated with the second image and the pixel information of the pixels in the second image, so that each pixel in the image is associated with corresponding three-dimensional data (i.e., the three-dimensional coordinates contained in the three-dimensional data), and a third image set is obtained, and then a third image set consisting of a plurality of consecutive third image frames is obtained. Then, based on the three-dimensional data contained in each third image of the third image set, target three-dimensional data corresponding to the target time frame is obtained.
[0047] In an embodiment of the present disclosure, the continuous multiple frames of first three-dimensional data included in the first three-dimensional data set are multi-frame data collected in time series, and the different first three-dimensional data included in the first three-dimensional data set correspond to different time frames. Similarly, the continuous multiple frames of first images included in the first image set are also multi-frame data collected in time series, and the different images included in the first image set correspond to different time frames. The target time frame in the present disclosure is one of the above-mentioned different time frames, corresponding to a group of paired first three-dimensional data and first images. The target time frame can be a specific time frame selected based on user instructions, and the target three-dimensional data is the three-dimensional point cloud data of the target object contained in the first image corresponding to the target time frame.
[0048] Through the present disclosure, a plurality of consecutive frames of second images containing the first three-dimensional data are obtained based on the first images and the first three-dimensional data corresponding to each other. And according to the pixel information of the second image, each pixel of the second image is associated with the three-dimensional data to obtain a third image set, and then the target three-dimensional data containing the color information of the target object is obtained, so as to obtain dense three-dimensional point cloud data that can accurately reflect the shape of the object.
[0049] It can be understood that the difference between the continuous multiple frames of first images, the continuous multiple frames of second images and the continuous multiple frames of third images in the present disclosure mainly lies in the difference in the associated additional information (three-dimensional data), but there is no difference in the inherent pixel information of the images.
[0050] In the embodiment of the present disclosure, the three-dimensional data acquisition device for acquiring the first three-dimensional data and the image acquisition device for acquiring the first image are correspondingly arranged devices, and the relative positional relationship between the two is fixed. Combined with the positional relationship between the three-dimensional data acquisition device and the image acquisition device, as well as the intrinsic parameters of the image acquisition device (such as focal length, principal point coordinates), and the extrinsic parameters of the image acquisition device (rotation matrix and translation vector), the corresponding relationship between the first three-dimensional data and the first image can be obtained, so as to map the first three-dimensional data to the first image. The following embodiment of the present disclosure describes the method for acquiring the second image set.
[0051] Figure 2 FIG. 1 is a flow chart showing a method for acquiring a second image set according to an exemplary embodiment. Figure 2 As shown, the method includes step S201 to step S202.
[0052] In step S201, for each frame of the first image included in the first image set, first three-dimensional data corresponding to the first image in the first three-dimensional data set is mapped to the first image according to a first coordinate transformation relationship to obtain a second image.
[0053] The first coordinate conversion relationship is a corresponding relationship between pixel coordinates in the first image and space coordinates in the first three-dimensional data.
[0054] In step S202, a set of all second images is determined as a second image set.
[0055] In the disclosed embodiment, for each frame of the first image included in the first image set, the first three-dimensional data of the first three-dimensional data corresponding to the first image in the first three-dimensional data set is mapped to the first image, and after the three-dimensional data mapping process is completed for all the first images, all the obtained second images are determined as the second image set. By associating the three-dimensional data in the first image, it is convenient to amplify the three-dimensional data according to the image information in the subsequent process, that is, to densify the three-dimensional data, so as to obtain more three-dimensional data corresponding to the target object and restore the structural details of the target object.
[0056] In the embodiment of the present disclosure, the first three-dimensional data is a three-dimensional point cloud composed of a series of three-dimensional coordinates, and mapping the first three-dimensional data of the first three-dimensional data to the first image is essentially mapping the three-dimensional coordinates contained in the first three-dimensional data to the corresponding pixels of the first image. The following embodiment of the present disclosure further describes the method for obtaining the second image set.
[0057] Figure 3 FIG. 1 is a flow chart showing a method for acquiring a second image set according to an exemplary embodiment. Figure 3 As shown, the method includes step S301 to step S302.
[0058] In step S301, for each frame of the first image included in the first image set, each three-dimensional coordinate included in the first three-dimensional data is mapped to a corresponding pixel in the corresponding first image according to a first coordinate conversion relationship to obtain a second image.
[0059] In step S302, a set of all second images is determined as a second image set.
[0060] In the embodiment of the present disclosure, the coordinate transformation relationship (first coordinate transformation relationship) between the pixel coordinates in the first image and the three-dimensional coordinates in the corresponding first three-dimensional data is determined based on the positional relationship between the three-dimensional data acquisition device and the image acquisition device, as well as the internal parameters of the image acquisition device (such as focal length, principal point coordinates), and the external parameters of the image acquisition device (rotation matrix and translation vector). For each frame of the first image in the first image set, each three-dimensional coordinate contained in the first three-dimensional data is mapped to the corresponding pixel in the corresponding first image according to the first coordinate transformation relationship to obtain a second image. After the associated three-dimensional data of all first images are processed, all the obtained second images are determined as the second image set. It can be understood that the arrangement order of the multiple frames of the second image in the second image is consistent with the arrangement data of the multiple frames of the first image in the first image set, and they are all arranged according to the time sequence when the image is acquired.
[0061] It is understandable that the single frame of three-dimensional data (first three-dimensional data set) collected by the three-dimensional data acquisition device in the present disclosure is relatively sparse, and there is a lack of three-dimensional data for some objects in the scene. Therefore, after the three-dimensional coordinates contained in the first three-dimensional data are mapped to the corresponding pixels of the first image, only some pixels in the obtained second image are associated with the first three-dimensional data (corresponding to the three-dimensional coordinates in the first three-dimensional data). In view of this, the present disclosure completes the three-dimensional data for the pixels in the second image that are not associated with the three-dimensional coordinates, thereby realizing the densification of sparse three-dimensional data. In the following embodiments of the present disclosure, the method for obtaining the third image set is described.
[0062] Figure 4 FIG. 1 is a flow chart showing a method for acquiring a third image set according to an exemplary embodiment. Figure 4 As shown, the method includes step S401 to step S402.
[0063] In step S401, for each frame of the second image in the second image set, the pixel associated with the three-dimensional coordinates in the second image is determined as the first pixel, and the pixel not associated with the three-dimensional coordinates in the second image is determined as the second pixel, and the three-dimensional coordinates associated with the second pixel are determined based on the correspondence between the pixel information and the three-dimensional coordinates, the three-dimensional coordinates associated with the first pixel, the pixel information of the first pixel and the pixel information of the second pixel, so that the second pixels in the second image are all associated with the corresponding three-dimensional coordinates, and a third image is obtained, and the pixel information includes the pixel coordinates and color channel values of the pixels.
[0064] In step S402, a set of all third images is determined as a third image set.
[0065] In an embodiment of the present disclosure, for each frame of the first image in the first image set, after the first three-dimensional data in the first three-dimensional data set and are mapped correspondingly to the first image and the second image set is obtained, considering the sparsity of the three-dimensional data in the first three-dimensional data set, only some pixels in the second image are associated with three-dimensional data (i.e., three-dimensional coordinates), the present disclosure determines, for each frame of the second image in the second image set, the pixel associated with the three-dimensional coordinate as the first pixel, and determines the pixel not associated with the three-dimensional coordinate as the second pixel, and determines the three-dimensional coordinate associated with the second pixel according to the difference in pixel information between the second pixel and the first pixel and the three-dimensional coordinate associated with the first pixel, that is, determines the three-dimensional coordinates corresponding to the pixels in the second image that are not associated with the three-dimensional coordinates, and associates the pixels in the second image that are not associated with the three-dimensional coordinates with the determined three-dimensional coordinates.
[0066] In the embodiment of the present disclosure, the image information includes the pixel coordinates of each pixel in the image and the color channel value of the pixel (i.e., color information). The present disclosure can construct a correspondence between the pixel information (pixel coordinates and color channel value) and the corresponding three-dimensional coordinates for the first pixel associated with the three-dimensional data. Then, for the second pixel not associated with the three-dimensional data, the three-dimensional coordinates corresponding to the second pixel are determined based on the difference between the pixel information (pixel coordinates and color channel value) between the second pixel and the first pixel, and the correspondence between the pixel information (pixel coordinates and color channel value) of the first pixel and the corresponding three-dimensional coordinates.
[0067] In one example, the present disclosure uses the following method to determine the corresponding three-dimensional coordinates for a second pixel in a second image that is not associated with a three-dimensional coordinate: for each pixel in an image (i.e., the second image), if the pixel is not associated with a three-dimensional coordinate, find the target pixel (first pixel) that is known to be associated with a three-dimensional coordinate within a radius r (r is an empirical value) of the pixel (the second pixel), and then calculate the 3D spatial coordinates (i.e., the three-dimensional coordinates) of the pixel based on the distance difference and color difference between the pixel and the target pixel through an interpolation algorithm.
[0068] In the disclosed embodiment, for pixels in the second image that are not associated with three-dimensional coordinates (second pixels), the three-dimensional data corresponding to the second pixels are determined in combination with pixels that are associated with three-dimensional coordinates (first pixels), and the difference in pixel information between the first pixel and the second pixel, and the three-dimensional data is completed, thereby densifying sparse three-dimensional data, so that the target three-dimensional data finally obtained is dense three-dimensional data.
[0069] It can be understood that the target object in the present disclosure includes one or more objects respectively contained in each first image frame in the first image set. Considering that the target object and the data acquisition device (device for acquiring the first three-dimensional data and the first image) may be displaced, the present disclosure needs to determine the target object contained therein for each first image frame in the first image set. In view of this, the present disclosure needs to perform image semantic segmentation on each third image frame in the set of acquired third images (the image content of the third image is consistent with the image content of the corresponding first image), obtain the target object contained in each third image frame, and determine the image area corresponding to the target object in each third image frame, and then determine the three-dimensional data corresponding to each target object according to the image area corresponding to the target object in different third images and according to the feature matching relationship between different third images, and then determine the target three-dimensional data corresponding to the target time frame. The following embodiments of the present disclosure illustrate the method for obtaining target three-dimensional data.
[0070] Figure 5 FIG. 1 is a flow chart showing a method for obtaining three-dimensional data of a target according to an exemplary embodiment. Figure 5 As shown, the method includes steps S501 to S504.
[0071] In step S501, for each third image in the third image set, one or more target objects in the third image are determined according to image semantics, and an image area corresponding to each target object is identified in the third image to obtain a fourth image, and all the obtained fourth images are determined as a fourth image set.
[0072] In step S502, a matching image region set corresponding to each target object is determined according to every two adjacent fourth images in the fourth image set, and a data conversion relationship is determined.
[0073] The matching image region set includes a plurality of consecutive feature-matched matching image regions, and the data conversion relationship includes a three-dimensional coordinate conversion relationship between three-dimensional data corresponding to each two adjacent matching image regions in the matching image region set.
[0074] In step S503, for each set of matching image regions corresponding to the target object, the three-dimensional data included in each matching image region is spliced frame by frame according to the data conversion relationship, and the spliced three-dimensional data is determined as the first target three-dimensional data of the target object.
[0075] The first target three-dimensional data is a set of three-dimensional coordinates in a reference coordinate system, and the reference coordinate system is a world coordinate system established based on a preset origin.
[0076] In step S504, target three-dimensional data corresponding to the target time frame is obtained according to the first target three-dimensional data of each target object.
[0077] In an embodiment of the present disclosure, after acquiring the third image set, for each frame of the third image, the target object in the image is determined based on the image semantics, and the image area corresponding to each target object is identified to obtain a fourth image, that is, image semantic segmentation is performed on each frame of the third image, and the image area corresponding to each target object in the fourth image obtained after segmentation is respectively identified, and after completing the image semantic segmentation for each frame of the third image, a fourth image set is obtained. In one example, the model used in the present disclosure for semantic segmentation of images is the Segment Anything Model (SAM). It can be understood that the present disclosure can use any model with image semantic segmentation capabilities to perform semantic segmentation on the third image.
[0078] It can be understood that the continuous multiple frames of first images in the first image set are all multiple frames of images collected quickly and continuously in a short period of time. Regardless of whether the target object is in a moving state or a non-moving state, some adjacent images in the multiple frames of first images in the first image set will have the same or corresponding technical features corresponding to the same target object, that is, there is a feature matching relationship between adjacent images in the multiple frames of first images. Considering the corresponding relationship between the fourth image set and the first image set, there is a feature matching relationship between adjacent fourth images in the multiple frames of fourth images in the fourth image set, that is, the adjacent fourth images will contain the same target object, and the image areas corresponding to the same target object in different images may change, and the pixels in the adjacent fourth images that have a feature matching relationship correspond to the same target object. Based on this, the present disclosure performs feature matching between the continuous multiple frames of fourth images contained in the fourth image set in pairs according to the time sequence, and for each target object, the image areas corresponding to the target object are determined in the multiple frames of fourth images. Then, for each target object, the image areas corresponding to the target object in the multiple frames of fourth images are determined as the matching image area set of the target object. After obtaining the set of matching image areas corresponding to each target object, the present invention can determine the three-dimensional coordinate conversion relationship between the three-dimensional data corresponding to each two adjacent image areas in the matching image area set according to the pixel coordinates corresponding to each image area in the corresponding fourth image, the correspondence between the pixel information (including pixel coordinates) in the fourth image and the three-dimensional data, and other relevant information.
[0079] In the embodiment of the present disclosure, after obtaining the data conversion relationship corresponding to each matching image area set. For each matching image area set corresponding to each target object, the three-dimensional data included in each matching image area set is spliced frame by frame according to the data conversion relationship, and the spliced three-dimensional data is determined as the first target three-dimensional data of the target object, thereby obtaining the target three-dimensional data of each target object. The first target three-dimensional data in the present disclosure is the three-dimensional point cloud data under the reference coordinate system. The reference coordinate system is a time coordinate system established based on a preset origin in space. It is a fixed coordinate system and will not change due to the displacement of the device. The first target three-dimensional data of all target objects in the present disclosure is the three-dimensional point cloud data corresponding to all target objects in the reference coordinate system. According to the first target three-dimensional data, the target three-dimensional data corresponding to a specific time frame (the first three-dimensional data and the first image corresponding to a specific frame) can be determined.
[0080] It is understandable that the device provided with the data acquisition device in the present disclosure is a device with displacement capability. Considering the possible displacement of the device in the process of acquiring data, the present disclosure needs to synchronize the posture information (pose information) of the acquisition device in the process of acquiring the first image set and the first three-dimensional data set, that is, the posture information of the acquisition device is synchronized at each time the first image and the first three-dimensional data are acquired. When the present disclosure determines the three-dimensional coordinate transformation relationship between the three-dimensional data corresponding to each two adjacent image areas in the matching image area set, it needs to be determined in combination with the posture information when the device acquires the corresponding data. The following embodiments of the present disclosure illustrate the method for determining the matching image area set and the data transformation relationship.
[0081] Figure 6 FIG. 1 is a flow chart of a method for determining a set of matching image regions and determining a data conversion relationship according to an exemplary embodiment. Figure 6 As shown, the method includes step S601 to step S602.
[0082] In step S601, a matching image region set corresponding to each target object is determined based on a pixel feature matching relationship between every two adjacent fourth images in the fourth image set, and the matching image region set includes a plurality of consecutive feature-matched matching image regions.
[0083] In step S602, for every two adjacent matching image areas in the matching image area set corresponding to each target object, the data conversion relationship is determined according to the correspondence between the pixel information and the three-dimensional data in the fourth image, and the pixel coordinates and posture information sets corresponding to the two adjacent matching image areas in the fourth image.
[0084] The posture information set includes posture information of the data acquisition device when acquiring each frame of the first image and the first three-dimensional data.
[0085] In an embodiment of the present disclosure, after acquiring the fourth image set, feature matching is performed between the multiple frames of fourth images included in the fourth image set, and for each target object, the image area corresponding to the target object is determined in the multiple frames of fourth images, and then for each target object, the image areas corresponding to the target object in the multiple frames of fourth images are determined as the matching image area set of the target object. In one example, the present disclosure uses a scale invariant feature transform algorithm (SIFT) to perform feature matching between adjacent fourth images, and in another example, the present disclosure uses a speeded up robust features algorithm (SURF) to perform feature matching between adjacent fourth images.
[0086] In the embodiment of the present disclosure, for the set of matching image regions corresponding to each target object, the pixel coordinates that match between every two adjacent image regions in the set of matching image regions are determined based on the feature matching relationship between every two adjacent image regions. The three-dimensional coordinate conversion relationship between the three-dimensional data corresponding to each two adjacent image regions in the set of matching image regions is determined based on the pixel coordinates corresponding to each image region in the corresponding fourth image, the pixel coordinates that match between every two adjacent image regions, the correspondence between the pixel information (including the pixel coordinates) and the three-dimensional data in the fourth image, and the posture information of the device at different time frames during the data (first three-dimensional data and first image) acquisition process. For the set of matching image regions corresponding to each target object, the present disclosure will obtain a set of data conversion relationships (corresponding to every two adjacent matching image regions).
[0087] It is understandable that, in the process of collecting the first image set and the first three-dimensional data set, if the relative positional relationship between the device provided with the data acquisition device and the target device changes, different first images in the first image set will contain images of the target object at different viewing angles. That is, the continuous multiple frames of fourth images contained in the fourth image set are images of the same target object at different viewing angles, and the three-dimensional data of the corresponding target object contained in different fourth images will also be different, that is, different matching image areas in the present disclosure may be images of the same target object at different viewing angles, that is, the three-dimensional data contained in different matching image areas in the present disclosure will characterize the structural features of the same target object at different viewing angles. For the same target object, by splicing the three-dimensional data contained in different matching image areas corresponding to the target object, the first target data obtained can further restore the overall structural features of the target object.
[0088] In an exemplary embodiment of the present disclosure, the following method is used to obtain the data conversion relationship corresponding to each target object by splicing the three-dimensional data corresponding to each fourth image:
[0089] 1. First use Represents the comprehensive characteristics of the target object:
[0090]
[0091] Where i is the frame number, is the number of segmented objects in the fourth image of each frame, k is the segmentation area number, is the image information of the segmented object (corresponding to the matching image area), (u, v) is the pixel coordinates of the pixel in the matching image area, is the 3D coordinate information (three-dimensional data) corresponding to the segmented object (corresponding to the matching image area), and (x, y, z) is the spatial coordinate contained in the three-dimensional data.
[0092] 2. According to the image matching algorithm, the pixel matching relationship between the i-th frame and the i+1-th frame (i.e., the adjacent fourth image) is obtained, and the corresponding relationship between the pixel coordinates and the three-dimensional coordinates in the fourth image (one-to-one correspondence between the pixel coordinates and the three-dimensional coordinates) is combined to obtain the 3D point cloud matching relationship of the object between the i-th frame and the i+1-th frame. When the number of matching points between adjacent image frames is greater than the threshold (such as 3 pairs), the spatial transformation relationship of the matching three-dimensional data is calculated according to the least squares method and other optimization algorithms. , so that it satisfies:
[0093]
[0094] in, and The pose parameters (device posture information) corresponding to the device when it collects adjacent first images (corresponding to adjacent fourth images). When the matched point cloud corresponds to a static target object, is the unit matrix (characterizing the spatial transformation relationship between the three-dimensional data corresponding to the image area whose image features match those in the adjacent fourth image). When the matched point cloud corresponds to a dynamic target object, It is the spatial displacement information of the object in the previous and next frames. and are the three-dimensional data corresponding to adjacent fourth images respectively.
[0095] 3. Traverse the continuous frames to obtain the transformation relationship between all matching objects (corresponding to the image area whose image features match those in the adjacent fourth image).
[0096] In an exemplary embodiment of the present disclosure, the point cloud data (three-dimensional data) corresponding to the target object in different fourth images are spliced according to the acquired spatial conversion relationship (data conversion relationship) in the following manner to obtain a complete point cloud of the target object in a world coordinate system (reference coordinate system) established based on a preset origin, and at the same time, the global object contained in each frame is recorded (that is, the target object contained in each frame of the fourth image is recorded):
[0097]
[0098] Among them, j is the global number of the object, i is the consecutive frame number, N is the number of consecutive frames, k is the segmentation area number, and s=1…i is the frame number different from i. It is a complete point cloud of the target object in the world coordinate system (reference coordinate system) (including the first target 3D data corresponding to each target object). is the multiplication operator, Represents the corresponding data conversion relationship of the object with global number j, Characterizes the product of the corresponding data transformation relationship of the object with global number j in multiple consecutive frames (1-i). is the three-dimensional data corresponding to the matching image area corresponding to a single target object in a single fourth image, is the three-dimensional data of the image area corresponding to the target object (one or more) in the fourth image, and Pi is the pose parameter (posture information) when the device collects the i-th frame data (including three-dimensional data and image).
[0099] In an embodiment of the present disclosure, in the process of determining one or more target objects in a third image according to image semantics, and identifying the image area corresponding to each target object in the third image to obtain a fourth image, the method further includes: determining global object information of the third image, and associating the global object information with the obtained fourth image, wherein the global object information is all target objects contained in the third image. The present disclosure can determine multiple target objects corresponding to a target time frame according to a fourth image corresponding to a target time frame (a target image frame corresponds to a single-frame first image, and there is a one-to-one correspondence between multiple-frame first images, multiple-frame second images, multiple-frame third images, and multiple-frame fourth images), and further determine target three-dimensional data corresponding to a current time frame according to the target object corresponding to the target time frame and the first target three-dimensional data corresponding to the target object. The following embodiment of the present disclosure describes a method for obtaining target three-dimensional data.
[0100] Figure 7 FIG. 1 is a flow chart showing a method for obtaining three-dimensional data of a target according to an exemplary embodiment. Figure 7 As shown, the method includes step S701 to step S702.
[0101] In step S701, a fourth image corresponding to a target time frame is determined, the fourth image corresponding to the target time frame is determined as a target fourth image, and a target object contained in the target fourth image is determined according to global object information associated with the target fourth image.
[0102] In step S702, the first target three-dimensional data corresponding to the target object contained in the fourth target image is determined as the second target three-dimensional data, and the second target three-dimensional data is mapped to the target coordinate system according to the posture information set and the target data conversion relationship to obtain the target three-dimensional data.
[0103] Among them, the target data conversion relationship is the data conversion relationship corresponding to the target object contained in the target fourth image, the posture information set includes the posture information of the data acquisition device when acquiring each frame of the first image and the first three-dimensional data, the posture information is in a corresponding relationship with the coordinate system established based on the data acquisition device, the target coordinate system is the coordinate established based on the data acquisition device when the data acquisition device acquires the target first image, and the target first image is the first image corresponding to the target fourth image.
[0104] In the embodiment of the present disclosure, the data acquisition device includes a correspondingly arranged three-dimensional data acquisition device (such as a radar) and an image acquisition device (such as a camera), and the data acquisition device is on a device with mobile capabilities. The data acquisition device and the device can be regarded as a whole. Considering that the device will be displaced, the data acquisition device will be displaced accordingly, that is, the posture information of the data acquisition device will be transformed accordingly. The target coordinate system in the present disclosure is a coordinate system established based on the data acquisition device (or a device provided with a data acquisition device). The target coordinate system changes with the displacement of the data acquisition device, and there is a corresponding relationship between the target coordinate system and the posture information of the device. According to the corresponding posture information when the data acquisition device collects each frame of data (three-dimensional data and image), the target three-dimensional data distributed in the target coordinate system when the data acquisition device collects data of a specific frame (corresponding to the target time frame) can be obtained. In the present disclosure, after determining the target time frame, the fourth image corresponding to the target time frame is determined, and the global object information associated with the corresponding fourth image is determined. Based on the global object information, multiple target objects contained in the fourth image corresponding to the target time frame are determined, and the first target three-dimensional data corresponding to the multiple target objects are used as data for obtaining the target three-dimensional data (i.e., the second target three-dimensional data), and then the target three-dimensional data is determined based on the second target three-dimensional data and the posture information (including the posture information corresponding to each frame of data collected by the device). It can be understood that the target time frame is a time frame selected based on the control instruction, which can correspond to any frame of the multiple fourth images, that is, the present disclosure can obtain the target three-dimensional data corresponding to any frame of the fourth image.
[0105] In the embodiment of the present disclosure, after obtaining the complete point cloud (first target three-dimensional data) of each target object in the reference coordinate system, and determining the global object information corresponding to each frame of the fourth image (characterizing the target object contained in the fourth image), the present disclosure can determine the target object contained in the fourth image corresponding to the target time frame, and determine the first target three-dimensional data (i.e., second target three-dimensional data) corresponding to the target object contained in the fourth image. According to the posture data collected by the device provided with the data acquisition device during the data acquisition process, the determined second target three-dimensional data is converted to obtain the target three-dimensional data of the target object in the fourth image corresponding to the target time frame in the target coordinate system, so that the device can obtain the surrounding environment information and object information at the time of collecting specific data (the first three-dimensional data and the first image corresponding to the target time frame) according to the target three-dimensional data.
[0106] In an exemplary embodiment of the present disclosure, the target three-dimensional data is obtained in the following manner:
[0107]
[0108] Where i is the frame number, j is the global number of the object, s=1…i is the frame number different from i, is the three-dimensional data of the target object in the fourth image corresponding to the target time frame in the target coordinate system (i.e., the target three-dimensional data). inv represents the inverse operation, Pi is the pose parameter (posture information) when the device collects the i-th frame data (including three-dimensional data and image), This is the inverse operation of Pi. is the multiplication operator, Represents the corresponding data conversion relationship of the object with global number j, is the product of the corresponding data transformation relationship representing the object with global number j in multiple consecutive frames (1-i), For Perform the inverse operation. It is a complete point cloud of the target object in the world coordinate system (reference coordinate system) (including the first target three-dimensional data corresponding to each target object in the target fourth image).
[0109] In the embodiment of the present disclosure, each three-dimensional coordinate constituting the target three-dimensional data has a pixel associated therewith, based on which, the present disclosure can associate the pixel information of the pixel with each three-dimensional coordinate of the target three-dimensional data, so that the target three-dimensional data contains the color information of the target object. The following embodiment of the present disclosure further illustrates the three-dimensional reconstruction method in the present disclosure.
[0110] In one implementation of the disclosed embodiment, the color information of the corresponding target object corresponds to the color channel value of the pixel in the fourth image; the target three-dimensional data is associated with the color information of the corresponding target object: based on the correspondence between the pixel information and the three-dimensional data in the fourth image, the target data conversion relationship and the posture information set, the target color channel value is associated with each three-dimensional coordinate contained in the target three-dimensional data, and the target color channel value is the color channel value contained in the pixel information of the fourth image.
[0111] In the embodiment of the present disclosure, the three-dimensional coordinates constituting the target three-dimensional data are all three-dimensional coordinates obtained after multiple data conversions of the three-dimensional coordinates associated with the fourth image pixels in the fourth image, that is, the three-dimensional coordinates constituting the target three-dimensional data are all indirectly corresponding to the pixel information of the pixels in the fourth image (including the color channel values of the pixels). Based on this, the present disclosure associates the target color channel values contained in the pixel information of the four images to each three-dimensional coordinate contained in the first three-dimensional data according to the corresponding relationship between the pixel information and the three-dimensional data in the fourth image, the data conversion relationship, and the posture information. Through the present disclosure, the pixel information of the pixel is correspondingly associated with each three-dimensional coordinate of the target three-dimensional data, so that the target three-dimensional data contains the color information of the target object, which is convenient for visual observation and annotation of the target three-dimensional data in the subsequent process.
[0112] In the embodiment of the present disclosure, a correspondingly arranged three-dimensional data acquisition device (such as a radar) and an image acquisition device (such as a camera) are used to synchronously acquire continuous multi-frame three-dimensional data (first three-dimensional data) and continuous multi-frame images (first image), and the continuous multi-frame image data constitutes a first image set, and the continuous multi-frame three-dimensional data constitutes a first three-dimensional data set. The continuous multi-frame three-dimensional data is correspondingly associated with the continuous multi-frame image, so that each image contains three-dimensional data. For the image (second image) with continuous multi-frames associated with three-dimensional data, the three-dimensional data is supplemented according to the three-dimensional data associated in the image and the pixel information of the pixels in the image, so that each pixel in the image is associated with the corresponding three-dimensional data. According to the three-dimensional data contained in the image (third image) after the continuous multi-frame three-dimensional data is supplemented, the target three-dimensional data corresponding to the target time frame is obtained, and the color information of the image pixels is also correspondingly associated with the target three-dimensional data, so as to obtain the target three-dimensional data containing the pixel information of the target object. Through the present disclosure, dense three-dimensional point cloud data (target three-dimensional data) that can accurately reflect the shape and color information of the object is obtained. And through the obtained three-dimensional point cloud data, the scene information is more completely and accurately displayed, which is convenient for the annotation of the point cloud data. The acquired point cloud data contains the color information of the target object, which is convenient for visual observation and point cloud annotation.
[0113] Based on the same concept, the embodiment of the present disclosure also provides a three-dimensional reconstruction device 800 .
[0114] It is understandable that the three-dimensional reconstruction device 800 provided in the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.
[0115] Figure 8 FIG. 8 is a block diagram of a 3D reconstruction device 800 according to an exemplary embodiment. Figure 8 , the device includes an acquisition unit 801 and a processing unit 802.
[0116] The acquisition unit 801 is used to acquire a first three-dimensional data set and a first image set, wherein the first three-dimensional data set includes a plurality of continuous frames of first three-dimensional data, and the first image set includes a plurality of continuous frames of first images. The plurality of frames of first three-dimensional data and the plurality of frames of first images correspond one to one, and the plurality of frames of first three-dimensional data all include three-dimensional data corresponding to a target object, and the plurality of frames of first images all include images including the target object.
[0117] The processing unit 802 is used to obtain a second image set based on the first three-dimensional data set and the first image set, the second image set including a plurality of consecutive frames of second images with the first three-dimensional data, and for each frame of the second image in the second image set, according to the first three-dimensional data and pixel information of the second image included in the second image, associate three-dimensional data with each pixel in the second image to obtain a third image set, and according to the three-dimensional data included in each frame of the third image in the third image set, obtain target three-dimensional data corresponding to a target time frame, the target time frame corresponding to a single frame of the first image in the first image set, associate the target three-dimensional data with color information of a corresponding target object, and obtain target three-dimensional data associated with color information, the corresponding target object being a target object included in the single frame of the first image.
[0118] In one embodiment, the processing unit 802 obtains the second image set according to the first three-dimensional data set and the first image set in the following manner: for each frame of the first image contained in the first image set, the first three-dimensional data corresponding to the first image in the first three-dimensional data set is mapped to the first image according to the first coordinate transformation relationship to obtain the second image, and the first coordinate transformation relationship is the correspondence between the pixel coordinates in the first image and the spatial coordinates in the first three-dimensional data. The set of all second images is determined as the second image set. Among them, the processing unit 802 maps the first three-dimensional data corresponding to the first image in the first three-dimensional data set to the first image according to the first coordinate transformation relationship in the following manner: according to the first coordinate transformation relationship, each three-dimensional coordinate contained in the first three-dimensional data is mapped to the corresponding pixel in the corresponding first image.
[0119] In one embodiment, the processing unit 802 uses the following method to associate three-dimensional data with each pixel in each second image in the second image set according to the first three-dimensional data and the pixel information of the second image included in the second image, so as to obtain a third image: for each second image in the second image set, determine the pixel associated with the three-dimensional coordinate in the second image as the first pixel, determine the pixel not associated with the three-dimensional coordinate in the second image as the second pixel, and determine the three-dimensional coordinate associated with the second pixel according to the correspondence between the pixel information and the three-dimensional coordinate, the three-dimensional coordinate associated with the first pixel, the pixel information of the first pixel and the pixel information of the second pixel, so that the second pixels in the second image are all associated with the corresponding three-dimensional coordinate, so as to obtain a third image, wherein the pixel information includes the pixel coordinate and the color channel value of the pixel. The set of all third images is determined as the third image set.
[0120] In one embodiment, the target object includes one or more objects included in each first image frame of the plurality of first images. The processing unit 802 acquires the target three-dimensional data corresponding to the target time frame according to the three-dimensional data included in each third image frame in the third image set in the following manner, including: for each third image frame in the third image set, determining one or more target objects in the third image according to image semantics, and identifying the image region corresponding to each target object in the third image to obtain a fourth image, and determining all the obtained fourth images as a fourth image set. According to each two adjacent fourth images in the fourth image set, determining a matching image region set corresponding to each target object, and determining a data conversion relationship, the matching image region set includes a matching image region with a plurality of consecutive feature matches, and the data conversion relationship includes a three-dimensional coordinate conversion relationship between the three-dimensional data corresponding to each two adjacent matching image regions in the matching image region set. For each matching image region set corresponding to the target object, splicing the three-dimensional data included in each matching image region frame by frame according to the data conversion relationship, and determining the spliced three-dimensional data as the first target three-dimensional data of the target object, the first target three-dimensional data is a three-dimensional coordinate set in a reference coordinate system, and the reference coordinate system is a world coordinate system established based on a preset origin. According to the first target three-dimensional data of each target object, target three-dimensional data corresponding to the target time frame is obtained.
[0121] In one embodiment, the processing unit 802 determines the set of matching image regions corresponding to each target object and determines the data conversion relationship based on every two adjacent fourth images in the fourth image set in the following manner: the set of matching image regions corresponding to each target object is determined based on the pixel feature matching relationship between every two adjacent fourth images in the fourth image set, and the matching image region set includes matching image regions with multiple consecutive feature matches. For every two adjacent matching image regions in the set of matching image regions corresponding to each target object, the data conversion relationship is determined based on the corresponding relationship between the pixel information in the fourth image and the three-dimensional data, the pixel coordinates and posture information sets corresponding to the two adjacent matching image regions in the fourth image, and the posture information sets include the posture information of the data acquisition device when acquiring each frame of the first image and the first three-dimensional data.
[0122] In one embodiment, in the process of determining one or more target objects in the third image according to image semantics and identifying the image area corresponding to each target object in the third image to obtain the fourth image, the processing unit 802 is also used to: determine the global object information of the third image, and associate the global object information with the obtained fourth image, and the global object information is all the target objects contained in the third image. The processing unit 802 obtains the target three-dimensional data corresponding to the target time frame according to the first target three-dimensional data of each target object in the following manner: determine the fourth image corresponding to the target time frame, determine the fourth image corresponding to the target time frame as the target fourth image, and determine the target object contained in the target fourth image according to the global object information associated with the target fourth image. The first target three-dimensional data corresponding to the target object contained in the target fourth image is determined as the second target three-dimensional data, and the second target three-dimensional data is mapped to the target coordinate system according to the posture information set and the target data conversion relationship to obtain the target three-dimensional data. Among them, the target data conversion relationship is the data conversion relationship corresponding to the target object contained in the target fourth image, the posture information set includes the posture information of the data acquisition device when acquiring each frame of the first image and the first three-dimensional data, the posture information is in a corresponding relationship with the coordinate system established based on the data acquisition device, the target coordinate system is the coordinate established based on the data acquisition device when the data acquisition device acquires the target first image, and the target first image is the first image corresponding to the target fourth image.
[0123] In one embodiment, the color information of the target object corresponds to the color channel value of the pixel in the fourth image. The processing unit 802 associates the target three-dimensional data with the color information of the corresponding target object in the following manner: according to the correspondence between the pixel information and the three-dimensional data in the fourth image, the target data conversion relationship and the posture information set, the target color channel value is associated with each three-dimensional coordinate included in the target three-dimensional data, and the target color channel value is the color channel value included in the pixel information of the fourth image.
[0124] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0125] Fig. 9 It is a block diagram of an electronic device 900 for three-dimensional reconstruction according to an exemplary embodiment.
[0126] like Fig. 9As shown, an embodiment of the present disclosure provides an electronic device 900. The electronic device 900 includes a memory 901, a processor 902, and an input / output interface 903, wherein the input / output interface 903 may also be referred to as an I / O (Input / Output) interface 903. The memory 901 is used to store instructions. The processor 902 is used to call the instructions stored in the memory 901 to execute the speech detection method of the embodiment of the present disclosure. The processor 902 is connected to the memory 901 and the I / O interface 903, respectively, for example, through a bus system and / or other forms of connection mechanisms (not shown). The memory 901 can be used to store programs and data, including the program of the speech detection method involved in the embodiment of the present disclosure, and the processor 902 executes various functional applications and data processing of the electronic device 900 by running the program stored in the memory 901.
[0127] In the embodiment of the present disclosure, the processor 902 can be implemented in at least one hardware form of a digital signal processor (Digital Signal Processing, DSP), a field programmable gate array (Field Programmable Gate Array, FPGA), and a programmable logic array (Programmable Logic Array, PLA). The processor 902 can be a central processing unit (Central Processing Unit, CPU) or one or a combination of other forms of processing units with data processing capabilities and / or instruction execution capabilities.
[0128] The memory 901 in the embodiment of the present disclosure may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD), etc.
[0129] In the embodiment of the present disclosure, the I / O interface 903 may be used to receive input instructions (such as digital or character information, and key signal input related to user settings and function control of the electronic device 900), and may also output various information (such as images or sounds) to the outside. In the embodiment of the present disclosure, the I / O interface 903 may include one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a mouse, a joystick, a trackball, a microphone, a speaker, and a touch panel.
[0130] In some embodiments, the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, perform any of the methods described above.
[0131] In some embodiments, the present disclosure provides a computer program product, which includes a computer program. When the computer program is executed by a processor, any of the methods described above is performed.
[0132] Although operations are described in a particular order in the drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in serial order, or that all shown operations be performed to achieve desired results. In certain circumstances, multitasking and parallel processing may be advantageous.
[0133] The methods and apparatus of the present disclosure can be implemented using standard programming techniques, using rule-based logic or other logic to implement various method steps. It should also be noted that the words "apparatus" and "module" used herein and in the claims are intended to include implementations using one or more lines of software code and / or hardware implementations and / or devices for receiving input.
[0134] Any steps, operations or procedures described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product including a computer-readable medium containing computer program code, which can be executed by a computer processor to perform any or all of the described steps, operations or procedures.
[0135] The foregoing description of the implementation of the present disclosure has been given for the purpose of illustration and description. The foregoing description is not exhaustive nor is it intended to limit the present disclosure to the exact form disclosed, and various variations and modifications may exist in accordance with the above teachings or may be obtained from the practice of the present disclosure. These embodiments are selected and described in order to illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can utilize the present disclosure in various embodiments and various modifications suitable for the specific purpose contemplated.
[0136] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0137] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.
[0138] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.
[0139] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.
[0140] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.
[0141] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following scope of rights.
[0142] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.
Claims
1. A three-dimensional reconstruction method, characterized in that: include: Acquire a first three-dimensional data set and a first image set, wherein the first three-dimensional data set includes a plurality of consecutive frames of first three-dimensional data, and the first image set includes a plurality of consecutive frames of first images, wherein the plurality of frames of first three-dimensional data and the plurality of frames of first images correspond to each other in a one-to-one manner, the plurality of frames of first three-dimensional data all include three-dimensional data corresponding to a target object, and the plurality of frames of first images all include images including the target object; Acquire a second image set according to the first three-dimensional data set and the first image set, wherein the second image set includes a plurality of consecutive frames of second images with the first three-dimensional data; For each frame of the second image in the second image set, according to the first three-dimensional data included in the second image and the pixel information of the second image, each pixel in the second image is associated with three-dimensional data to obtain a third image set; Acquire target three-dimensional data corresponding to a target time frame according to the three-dimensional data contained in each frame of the third image in the third image set, wherein the target time frame corresponds to a single frame of the first image in the first image set; Associating the target three-dimensional data with color information of a corresponding target object to obtain target three-dimensional data associated with color information, wherein the corresponding target object is a target object included in the single-frame first image; Wherein, the target object includes one or more objects contained in each first image frame of the multiple first image frames; The step of acquiring target three-dimensional data corresponding to a target time frame according to the three-dimensional data contained in each frame of the third image in the third image set includes: For each third image in the third image set, determine one or more target objects in the third image according to image semantics, and identify an image region corresponding to each target object in the third image to obtain a fourth image, and determine all the obtained fourth images as a fourth image set; Determine a matching image region set corresponding to each target object according to every two adjacent fourth images in the fourth image set, and determine a data conversion relationship, wherein the matching image region set includes matching image regions of a plurality of consecutive feature matches, and the data conversion relationship includes a three-dimensional coordinate conversion relationship between three-dimensional data corresponding to every two adjacent matching image regions in the matching image region set; For each set of matching image regions corresponding to each target object, three-dimensional data included in each matching image region are spliced frame by frame according to a data conversion relationship, and the spliced three-dimensional data are determined as first target three-dimensional data of the target object, where the first target three-dimensional data is a set of three-dimensional coordinates in a reference coordinate system, where the reference coordinate system is a world coordinate system established based on a preset origin; According to the first target three-dimensional data of each target object, the target three-dimensional data corresponding to the target time frame is obtained.
2. The method according to claim 1, characterized in that: The acquiring a second image set according to the first three-dimensional data set and the first image set includes: For each frame of the first image included in the first image set, first three-dimensional data corresponding to the first image in the first three-dimensional data set is mapped to the first image according to a first coordinate transformation relationship to obtain a second image, wherein the first coordinate transformation relationship is a correspondence relationship between pixel coordinates in the first image and spatial coordinates in the first three-dimensional data; determining a set of all second images as the second image set; The mapping of the first three-dimensional data corresponding to the first image in the first three-dimensional data set to the first image according to the first coordinate transformation relationship includes: According to the first coordinate conversion relationship, each three-dimensional coordinate included in the first three-dimensional data is mapped to a corresponding pixel in the corresponding first image.
3. The method according to claim 2, characterized in that The step of associating three-dimensional data with each pixel in the second image according to the first three-dimensional data and pixel information of the second image included in each frame of the second image in the second image set, so as to obtain a third image set, comprises: For each second image in the second image set, a pixel in the second image associated with a three-dimensional coordinate is determined as a first pixel, a pixel in the second image not associated with a three-dimensional coordinate is determined as a second pixel, and the three-dimensional coordinate associated with the second pixel is determined according to a correspondence between pixel information and the three-dimensional coordinate, the three-dimensional coordinate associated with the first pixel, the pixel information of the first pixel, and the pixel information of the second pixel, so that all second pixels in the second image are associated with the corresponding three-dimensional coordinate, and a third image is obtained, wherein the pixel information includes the pixel coordinates and color channel values of the pixels; A set of all third images is determined as the third image set.
4. The method according to claim 1, characterized in that: The step of determining a matching image region set corresponding to each target object and determining a data conversion relationship based on every two adjacent fourth images in the fourth image set includes: Determine a matching image region set corresponding to each target object according to a pixel feature matching relationship between every two adjacent fourth images in the fourth image set, wherein the matching image region set includes a plurality of consecutive matching image regions with matching features; For every two adjacent matching image areas in the matching image area set corresponding to each target object, a data conversion relationship is determined based on the correspondence between the pixel information and the three-dimensional data in the fourth image, and the pixel coordinates and posture information sets corresponding to the two adjacent matching image areas in the fourth image, respectively. The posture information set includes the posture information of the data acquisition device when acquiring each frame of the first image and the first three-dimensional data.
5. The method according to claim 1, characterized in that In the process of determining one or more target objects in the third image according to image semantics and identifying an image region corresponding to each target object in the third image to obtain the fourth image, the method further includes: Determine global object information of the third image, and associate the global object information with the obtained fourth image, the global object information being all target objects contained in the third image; The step of obtaining target three-dimensional data corresponding to the target time frame according to the first target three-dimensional data of each target object includes: Determine a fourth image corresponding to the target time frame, determine the fourth image corresponding to the target time frame as a target fourth image, and determine a target object included in the target fourth image according to global object information associated with the target fourth image; Determine the first target three-dimensional data corresponding to the target object contained in the fourth target image as the second target three-dimensional data, and map the second target three-dimensional data to the target coordinate system according to the posture information set and the target data conversion relationship to obtain the target three-dimensional data; Among them, the target data conversion relationship is the data conversion relationship corresponding to the target object contained in the target fourth image, the posture information set includes the posture information of the data acquisition device when acquiring each frame of the first image and the first three-dimensional data, and there is a corresponding relationship between the posture information and the coordinate system established based on the data acquisition device. The target coordinate system is the coordinate established based on the data acquisition device when the data acquisition device acquires the target first image, and the target first image is the first image corresponding to the target fourth image.
6. The method according to claim 5, characterized in that The color information of the corresponding target object corresponds to the color channel value of the pixel in the fourth image; The step of associating the target three-dimensional data with the color information of the corresponding target object comprises: According to the correspondence between the pixel information and the three-dimensional data in the fourth image, the target data conversion relationship and the posture information set, the target color channel value is associated with each three-dimensional coordinate contained in the target three-dimensional data, and the target color channel value is the color channel value contained in the pixel information of the fourth image.
7. A three-dimensional reconstruction device, characterized in that: include: an acquisition unit, configured to acquire a first three-dimensional data set and a first image set, wherein the first three-dimensional data set includes a plurality of consecutive frames of first three-dimensional data, and the first image set includes a plurality of consecutive frames of first images, wherein the plurality of frames of first three-dimensional data and the plurality of frames of first images correspond to each other in a one-to-one manner, the plurality of frames of first three-dimensional data all include three-dimensional data corresponding to a target object, and the plurality of frames of first images all include images of the target object; a processing unit, configured to obtain a second image set according to the first three-dimensional data set and the first image set, the second image set including a plurality of consecutive frames of second images with first three-dimensional data, and for each frame of the second image in the second image set, associating three-dimensional data with each pixel in the second image according to the first three-dimensional data and pixel information of the second image included in the second image, so as to obtain a third image set; and according to the three-dimensional data included in each frame of the third image in the third image set, obtaining target three-dimensional data corresponding to a target time frame, the target time frame corresponding to a single frame of the first image in the first image set, associating the target three-dimensional data with color information of a corresponding target object, so as to obtain target three-dimensional data associated with color information, the corresponding target object being a target object included in the single frame of the first image; Wherein, the target object includes one or more objects contained in each first image frame of the multiple first image frames; The processing unit acquires target three-dimensional data corresponding to a target time frame according to the three-dimensional data contained in each frame of the third image in the third image set in the following manner, including: For each third image in the third image set, determine one or more target objects in the third image according to image semantics, and identify an image region corresponding to each target object in the third image to obtain a fourth image, and determine all the obtained fourth images as a fourth image set; Determine a matching image region set corresponding to each target object according to every two adjacent fourth images in the fourth image set, and determine a data conversion relationship, wherein the matching image region set includes matching image regions of a plurality of consecutive feature matches, and the data conversion relationship includes a three-dimensional coordinate conversion relationship between three-dimensional data corresponding to every two adjacent matching image regions in the matching image region set; For each set of matching image regions corresponding to each target object, three-dimensional data included in each matching image region are spliced frame by frame according to a data conversion relationship, and the spliced three-dimensional data are determined as first target three-dimensional data of the target object, where the first target three-dimensional data is a set of three-dimensional coordinates in a reference coordinate system, where the reference coordinate system is a world coordinate system established based on a preset origin; According to the first target three-dimensional data of each target object, the target three-dimensional data corresponding to the target time frame is obtained.
8. The device according to claim 7, characterized in that The processing unit acquires a second image set according to the first three-dimensional data set and the first image set in the following manner: For each frame of the first image included in the first image set, first three-dimensional data corresponding to the first image in the first three-dimensional data set is mapped to the first image according to a first coordinate transformation relationship to obtain a second image, wherein the first coordinate transformation relationship is a correspondence relationship between pixel coordinates in the first image and spatial coordinates in the first three-dimensional data; determining a set of all second images as the second image set; The processing unit maps the first three-dimensional data corresponding to the first image in the first three-dimensional data set to the first image according to the first coordinate transformation relationship in the following manner: According to the first coordinate conversion relationship, each three-dimensional coordinate included in the first three-dimensional data is mapped to a corresponding pixel in the corresponding first image.
9. The device according to claim 8, characterized in that The processing unit adopts the following method to associate three-dimensional data with each pixel in each frame of the second image in the second image set according to the first three-dimensional data and pixel information of the second image included in the second image, so as to obtain a third image: For each second image in the second image set, a pixel in the second image associated with a three-dimensional coordinate is determined as a first pixel, a pixel in the second image not associated with a three-dimensional coordinate is determined as a second pixel, and the three-dimensional coordinate associated with the second pixel is determined according to a correspondence between pixel information and the three-dimensional coordinate, the three-dimensional coordinate associated with the first pixel, the pixel information of the first pixel, and the pixel information of the second pixel, so that all second pixels in the second image are associated with the corresponding three-dimensional coordinate, and a third image is obtained, wherein the pixel information includes the pixel coordinates and color channel values of the pixels; A set of all third images is determined as the third image set.
10. The device according to claim 7, characterized in that The processing unit determines a set of matching image regions corresponding to each target object and determines a data conversion relationship based on every two adjacent fourth images in the fourth image set in the following manner: Determine a matching image region set corresponding to each target object according to a pixel feature matching relationship between every two adjacent fourth images in the fourth image set, wherein the matching image region set includes a plurality of consecutive matching image regions with matching features; For every two adjacent matching image areas in the matching image area set corresponding to each target object, a data conversion relationship is determined based on the correspondence between the pixel information and the three-dimensional data in the fourth image, and the pixel coordinates and posture information sets corresponding to the two adjacent matching image areas in the fourth image, respectively. The posture information set includes the posture information of the data acquisition device when acquiring each frame of the first image and the first three-dimensional data.
11. The device according to claim 7, characterized in that In the process of determining one or more target objects in the third image according to image semantics and identifying an image region corresponding to each target object in the third image to obtain the fourth image, the processing unit is further used to: Determine global object information of the third image, and associate the global object information with the obtained fourth image, the global object information being all target objects contained in the third image; The processing unit obtains the target three-dimensional data corresponding to the target time frame according to the first target three-dimensional data of each target object in the following manner: Determine a fourth image corresponding to the target time frame, determine the fourth image corresponding to the target time frame as a target fourth image, and determine a target object included in the target fourth image according to global object information associated with the target fourth image; Determine the first target three-dimensional data corresponding to the target object contained in the fourth target image as the second target three-dimensional data, and map the second target three-dimensional data to the target coordinate system according to the posture information set and the target data conversion relationship to obtain the target three-dimensional data; Among them, the target data conversion relationship is the data conversion relationship corresponding to the target object contained in the target fourth image, the posture information set includes the posture information of the data acquisition device when acquiring each frame of the first image and the first three-dimensional data, and there is a corresponding relationship between the posture information and the coordinate system established based on the data acquisition device. The target coordinate system is the coordinate established based on the data acquisition device when the data acquisition device acquires the target first image, and the target first image is the first image corresponding to the target fourth image.
12. The device according to claim 11, characterized in that The color information of the corresponding target object corresponds to the color channel value of the pixel in the fourth image; The processing unit associates the target three-dimensional data with the color information of the corresponding target object in the following manner: According to the correspondence between the pixel information and the three-dimensional data in the fourth image, the target data conversion relationship and the posture information set, the target color channel value is associated with each three-dimensional coordinate contained in the target three-dimensional data, and the target color channel value is the color channel value contained in the pixel information of the fourth image.
13. An electronic device, characterized in that: include: processor: a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the three-dimensional reconstruction method according to any one of claims 1 to 6.
14. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor, the processor is enabled to execute the three-dimensional reconstruction method according to any one of claims 1 to 6.
