Instant depth acquisition method, three-dimensional imaging method, device and medium

Abstract

The application provides a real-time depth acquisition method, a three-dimensional imaging method, equipment and a medium, wherein the method comprises acquiring a first point cloud frame of a current time of a scene to be acquired and a first image frame of the current time of the scene to be acquired; the first point cloud frame comprises first point cloud collection and a first depth map; the first image frame comprises a plurality of first target regions; a next time point cloud collection and a next time image frame are acquired, wherein the process of acquiring the next time point cloud collection comprises: based on the plurality of first target regions of the first image frame, the first depth map and a scanning angle resolution set for different target regions, performing region adaptive scanning and global scanning on the scene to be acquired to acquire the next time point cloud collection; and based on the next time point cloud collection, a next time depth map is acquired. The method can acquire the next time point cloud collection and the next time depth map on the basis of the point cloud and the image frame at the current time, and realizes real-time acquisition and processing of depth information.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional imaging, and specifically provides an instant depth acquisition method, a three-dimensional imaging method, a device and a medium. Background Art

[0002] The rapid development of autonomous driving, robotic navigation, augmented reality, and other fields has placed higher demands on three-dimensional environmental perception capabilities. Traditional 3D imaging technologies, such as stereo vision and time-of-flight (ToF), while capable of acquiring 3D information of a scene to a certain extent, still have limitations in terms of measurement range, accuracy, and speed.

[0003] An optical phased array lidar (LIDAR) is a LiDAR system that uses an optical phased array unit as its core scanning component. By adjusting the driving voltage, the phase difference between the waveguides on the optical chip is controlled, thereby controlling the deflection direction of the emitted light beam. Prior art typically involves joint calibration of the camera and LiDAR to obtain the extrinsic parameters from the camera to the optical phased array LiDAR, namely the rotation matrix and translation vector. After obtaining the relative position relationship between the camera and the optical phased array LiDAR, the pixel coordinates in the camera's field of view can be converted to spatial coordinates in the optical phased array LiDAR coordinate system. However, since the three-dimensional coordinates are projected onto the image plane during camera imaging, the lack of depth-related scaling factors makes it impossible to convert two-dimensional pixel coordinates into three-dimensional spatial coordinates when the depth is unknown. Therefore, in target adaptive imaging methods, the key issue is to obtain the scene's real-time depth so that a more accurate LiDAR projection strategy can be adopted for different target areas in the scene.

[0004] Therefore, a new instant depth acquisition solution is needed in this field to solve the above problems. Summary of the Invention

[0005] In order to overcome the above defects, the present invention is proposed to solve the problem of being unable to obtain the real-time depth of the scene.

[0006] In a first aspect, the present invention provides an instant depth acquisition method, comprising: acquiring a first point cloud frame of a scene to be acquired at a current moment, and a first image frame of the scene to be acquired at a current moment; the first point cloud frame comprises a first point cloud acquisition and a first depth map; the first image frame comprises a plurality of first target areas; acquiring a point cloud acquisition at a next moment, and an image frame at a next moment, wherein the process of acquiring the point cloud acquisition at the next moment comprises: performing regional adaptive scanning and global scanning on the scene to be acquired based on a plurality of first target areas, a first depth map, and scanning angle resolutions set for different target areas of the first image frame, to acquire a point cloud acquisition at the next moment; and acquiring a depth map at the next moment based on the point cloud acquisition at the next moment.

[0007] In a technical solution of the above-mentioned instant depth acquisition method, the process of obtaining the first point cloud frame of the scene to be acquired at the current moment and the first image frame of the scene to be acquired at the current moment includes: performing a global scan of the scene to be acquired at the current moment to obtain the first point cloud acquisition at the current moment; obtaining the first depth map at the current moment based on the first point cloud acquisition; performing global image segmentation on the first image frame at the current moment to obtain several first target areas.

[0008] In a technical solution of the above-mentioned instant depth acquisition method, the scene to be acquired is subjected to regional adaptive scanning and global scanning based on several first target areas of the first image frame, the first depth map, and the scanning angle resolutions set for different target areas. The process of obtaining the point cloud acquisition at the next moment includes: based on the resolution set for each first target area, the scene to be acquired is adaptively scanned to obtain a first sub-point cloud acquisition; the resolution set for each first target area is also obtained based on the first depth map; the scene to be acquired is globally scanned to obtain a second sub-point cloud acquisition; and the point cloud acquisition at the next moment is obtained based on the first sub-point cloud acquisition and the second sub-point cloud acquisition.

[0009] In a technical solution of the above-mentioned instant depth acquisition method, the process of performing a global scan on the scene to be acquired to obtain the second sub-point cloud collection at least includes: performing a global uniform scan on the scene at a low resolution to obtain the second sub-point cloud collection.

[0010] In one technical solution of the above-mentioned instant depth acquisition method, the scanning angle resolution set for different target areas is also set based on the target type and size to be acquired by the optical phased array lidar.

[0011] In a second aspect, the present invention provides a three-dimensional imaging method based on an optical phased array laser radar, the method comprising the instant depth acquisition method of any one of the technical solutions of the instant depth acquisition method described above;

[0012] Repeat the above steps to continuously acquire point cloud collection and image frames. If the preset conditions are met, stop the collection.

[0013] In a third aspect, an electronic device includes a processor and a storage device, the storage device is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by the processor to execute the instant depth acquisition method of any technical solution in the technical solutions of the above-mentioned instant depth acquisition method.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium storing a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the instant depth acquisition method of any one of the technical solutions of the instant depth acquisition method described above.

[0015] The one or more technical solutions of the present application have at least one or more of the following beneficial effects:

[0016] In the implementation of the technical solutions of the present application, the present application provides an instant depth acquisition method, comprising: acquiring a first point cloud frame of a current time of a scene to be acquired and a first image frame of the current time of the scene to be acquired; the first point cloud frame comprises first point cloud collection and a first depth map; the first image frame comprises a plurality of first target regions; acquiring next time point cloud collection and a next time image frame, wherein the process of acquiring the next time point cloud collection comprises: based on the plurality of first target regions of the first image frame, the first depth map and the scanning angle resolution set for different target regions, performing region adaptive scanning and global scanning on the scene to be acquired, and acquiring the next time point cloud collection; based on the next time point cloud collection, acquiring a next time depth map. Compared with the prior art, the instant depth acquisition method provided by the present application has the beneficial effects of:

[0017] 1. The method can instantly acquire the next time point cloud collection and the depth map based on the point cloud and the image frame at the current time, realizing real-time acquisition and processing of depth information.

[0018] 2. According to the scanning resolution set for the target region, the region adaptive scanning and the global scanning are combined to scan the scene in a targeted manner. For example, more resources can be concentrated on the main target, reducing the scanning of invalid regions, thereby improving the utilization efficiency of the optical phased array laser radar resources and avoiding the generation of redundant data, to a certain extent, solving the problem of excessive demand for algorithm resource of OPA laser radar distance solving and real-time imaging, and effectively improving the frame rate of the imaging system. The depth information obtained by global scanning can supplement the depth information that may be missing in the target adaptive scanning process, making the imaging process more complete and accurate. In this way, invalid data acquisition and processing caused by missing depth information can be avoided, and the generation of redundant data can be reduced. BRIEF DESCRIPTION OF DRAWINGS

[0019] The disclosure of the present application will become more readily understood by referring to the accompanying drawings. It will be readily understood to those skilled in the art that the drawings are only for the purpose of illustration and are not intended to limit the scope of protection of the present application. In addition, similar numbers in the figures are used to represent similar components, wherein:

[0020] Figure 1 is a main step flow diagram of the instant depth acquisition method according to an embodiment of the present application;

[0021] Figure 2It is a flowchart of the main steps of a method for obtaining the first point cloud frame of the scene to be obtained at the current moment and the first image frame of the scene to be obtained at the current moment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] Some embodiments of the present invention are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0023] Example 1

[0024] See attached Figure 1 , Figure 1 FIG. 1 is a flow chart showing the main steps of a method for acquiring instant depth according to an embodiment of the present invention. Figure 1 As shown, the instant depth acquisition method in the embodiment of the present invention mainly includes the following steps S1 to S3.

[0025] Step S1: Acquire a first point cloud frame of the scene to be acquired at the current moment and a first image frame of the scene to be acquired at the current moment; the first point cloud frame includes a first point cloud acquisition and a first depth map; the first image frame includes a plurality of first target areas;

[0026] In this embodiment, the scene to be captured can refer to any scene requiring point cloud data processing and image data analysis, such as an indoor room, a road traffic scene, an electronics production line, and so on. The current moment refers to the current moment of capture and processing, or it can be any specified time point. The first point cloud frame refers to the point cloud data first captured at the current moment. It represents the distribution of the surface of objects in the scene in three-dimensional space at the current moment, including the three-dimensional coordinate information of the objects in the scene. The first point cloud capture refers to the point cloud data acquired by scanning the scene at the current moment using an optical phased array lidar. The first depth map refers to the distance information image obtained from the image data captured by the camera at the current moment. Its pixel values ​​reflect the distance from each point in the scene to the camera. The first image frame refers to the image frame data first captured at the current moment. Typically, a standard RGB camera or a depth camera is used to capture a two-dimensional image of the scene. These images provide appearance and texture information of the scene, which, together with the point cloud data, is used for depth perception and environmental understanding. The first target area refers to the area of ​​interest in the scene captured at the current moment.

[0027] In one embodiment, Figure 2 As shown, the process of step S1, obtaining the first point cloud frame of the scene to be obtained at the current moment and the first image frame of the scene to be obtained at the current moment, includes:

[0028] Step S11: performing a global scan on the scene to be acquired at the current moment to acquire the first point cloud collection at the current moment;

[0029] In this embodiment, an optical phased array lidar emits a laser beam. After interacting with objects in the scene, the laser beam is received by the lidar and converted into an electrical signal, which is then converted into point cloud data. This point cloud data is organized into a 3D point cloud collection, with each point cloud representing a spatial location and its corresponding feature information. Global scanning ensures the most complete and accurate scene point cloud possible.

[0030] Step S12: acquiring a first depth map at the current moment based on the first point cloud acquisition;

[0031] In this embodiment, the distance between each point cloud and the camera can be calculated by processing and analyzing the first point cloud acquisition. Because optical phased array lidar can quickly acquire point cloud data, more efficient algorithms (such as scene wavefront-based algorithms) can be used to improve calculation speed and accuracy when calculating depth maps. Depth maps can provide more accurate depth information and can obtain the geometric shape of obstacles within a certain range.

[0032] Step S13: performing global image segmentation on the first image frame at the current moment to obtain a plurality of first target areas.

[0033] In this embodiment, global image segmentation is a process of segmenting the entire image into different regions or objects. In this process, a target region can be obtained, which can be a pixel region that only contains the target object.

[0034] Step S2, obtaining a point cloud collection at the next moment and an image frame at the next moment, wherein the process of obtaining the point cloud collection at the next moment includes: performing regional adaptive scanning and global scanning on the scene to be acquired based on a plurality of first target areas of the first image frame, the first depth map, and scanning angle resolutions set for different target areas, to obtain a point cloud collection at the next moment;

[0035] In this embodiment, the image frame at the next moment refers to the point cloud collected a period of time after the current point cloud is collected, that is, the point cloud data at the next time point. The image frame at the next moment refers to the image frame obtained a period of time after the current image is collected, that is, the image data at the next time point. The scanning angle resolution refers to the angular spacing of the laser radar in the horizontal and vertical directions.

[0036] The adaptive scanning process scans each target area individually to meet its specific resolution requirements, rather than performing a uniform scan of the entire scene. Based on the resolution set for each target area, regional adaptive scanning can improve the point cloud density of the corresponding target to achieve higher point cloud precision and accuracy. Global scanning is necessary to prevent the camera from capturing a new target because the previous scan data cannot provide depth information of the new target, thereby being unable to guide the optical phased array lidar to collect the corresponding data. Therefore, a global scan is required to obtain a depth map of the scene to provide the missing depth information during the target adaptive scanning process.

[0037] Alternatively, in an alternative approach, regional adaptive scanning and global sparse scanning can be employed to perform regional adaptive scanning and global scanning of the scene to be acquired based on the plurality of first target regions of the first image frame, the first depth map, and the scanning angle resolutions set for the different target regions. Global sparse scanning acquires depth information by sampling a relatively small number of points within the entire scanning range. This reduces sampling density, data volume, and computational complexity.

[0038] The first image frame provides initial information about the scene to be captured. By analyzing the first image frame, the target region of interest can be identified and, combined with the first depth map, the object's position and shape can be estimated. Setting different scanning angle resolutions allows for flexible adjustment of the density and accuracy of point cloud acquisition. By using data from the previous frame, continuity and consistency are maintained. While the position and shape of an object may change during consecutive scans, data from adjacent frames is generally highly correlated. Therefore, using data from the previous frame as a reference allows for better tracking and analysis of object changes. Secondly, data from the previous frame is already available and closest to the current frame. By using relatively older, yet available, data for adaptive scanning of the target region, reliance on immediate data is reduced, thereby improving real-time performance. Finally, using data from the previous frame reduces computational complexity. Processing camera and lidar frames is typically complex and time-consuming. By utilizing only data from the previous frame, computing resources can be conserved and the algorithm's computational complexity and latency can be reduced.

[0039] In summary, by performing regional adaptive scanning and global scanning on the scene to be acquired based on several first target areas of the first image frame, the first depth map, and the scanning angle resolutions set for different target areas, the point cloud data acquisition process can be optimized, and a more accurate, comprehensive and high-quality point cloud set can be provided at the next moment.

[0040] In one embodiment, based on the first target areas of the first image frame, the first depth map, and the scanning angle resolutions set for different target areas, the scene to be acquired is subjected to regional adaptive scanning and global scanning, and a process for acquiring a point cloud acquisition at the next moment includes:

[0041] Based on the resolution set for each first target area, adaptively scanning the scene to be acquired to obtain a first sub-point cloud collection; the resolution set for each first target area is also obtained based on the first depth map;

[0042] Performing a global scan on the scene to be acquired to obtain a second sub-point cloud collection;

[0043] The next-moment point cloud collection is obtained based on the first sub-point cloud collection and the second sub-point cloud collection.

[0044] In this embodiment, the resolution set in the target area refers to the density of LiDAR sampling points set around the target object to obtain more accurate imaging of the target object. By setting the resolution of different areas, different LiDAR sampling strategies can be adopted for different target objects. Real-time depth information can provide the precise distance from the target object to the LiDAR. When considering the resolution set for each target area, the LiDAR sampling density can be adjusted according to the location and distance of the target object. For distant targets, the resolution can be reduced to reduce processing costs, while for close targets, the resolution can be increased to obtain more detailed information.

[0045] Based on the resolution set for each first target area, the scene to be acquired is adaptively scanned to obtain the first part of the point cloud data. This means that according to the set resolution, each target area of ​​the scene will be scanned as needed to obtain the first part of the point cloud acquisition data, which can help improve acquisition efficiency and reduce the amount of data. Then a global scan is performed on the entire scene to be acquired to obtain the second part of the point cloud data. Compared with adaptive scanning, global scanning covers a wider range and can obtain more comprehensive point cloud data. The point cloud obtained by performing regional adaptive scanning and global sparse scanning on the scene is: point cloud acquisition at the next moment = first sub-point cloud acquisition ∪ second sub-point cloud acquisition.

[0046] In one embodiment, the process of performing a global scan on the scene to be acquired to obtain the second sub-point cloud collection includes at least: performing a global uniform scan on the scene at a low resolution to obtain the second sub-point cloud collection.

[0047] In this embodiment, low resolution can indicate a relatively small number of points in the sub-point cloud, or low accuracy in the positional information of each point. The reason for performing a global, uniform scan of the scene at low resolution to obtain the second sub-point cloud is to enable comprehensive and rapid capture of the scene's overall characteristics. Specifically, employing a low-resolution scanning method can improve the speed and efficiency of data acquisition while reducing the amount of data required for storage and processing, thereby accelerating the acquisition and processing of point cloud data. Furthermore, within the low-resolution range, the entire scene to be acquired can be covered, thereby capturing the scene's global characteristics.

[0048] In one embodiment, the scanning angle resolutions set for different target areas are also acquired based on the target type and size.

[0049] In this embodiment, different types of targets may require different resolutions to acquire valid data. For example, a small, stationary target may require a higher resolution to capture fine features or edges, while a large, moving target may only require a lower resolution to detect its overall shape and position. The size of the target also affects the setting of the scanning angle resolution. For larger targets, a lower resolution can be used, while small targets may require a higher resolution to ensure accurate data acquisition.

[0050] Step S3: Acquire a depth map at the next moment based on the point cloud collection at the next moment.

[0051] In this embodiment, based on the point cloud acquisition data at the next moment obtained in step S2, an instant depth map can be generated by processing the point cloud data, thereby obtaining depth information of objects in the scene.

[0052] Depth information can be used to determine the azimuth and elevation angles of the target area. The azimuth indicates the target's position relative to the sensor in the horizontal plane, while the elevation indicates its position relative to the sensor in the vertical direction. This azimuth and elevation information can be used in adaptive scanning to adjust the lidar scanning method. For example, when the target is directly in front of the sensor, the scanning range can be set wider to obtain more detailed information about the target. When the target is to the side of the sensor, the scanning range can be reduced to improve scanning efficiency. Azimuth and elevation angles can also be used to determine the scanning resolution. For targets farther from the sensor, a larger scanning step size can be used to speed up scanning. For targets closer to the sensor, a smaller scanning step size can be used to improve data accuracy. Therefore, depth information, by providing the azimuth and elevation angles of the target pixel area, influences the range, angle, and resolution of adaptive scanning, thereby achieving more accurate and efficient lidar scanning.

[0053] The process of calculating the azimuth and elevation angles of the target pixel area is as follows: According to formula (1), based on the pose matrix of the camera and the optical phased array lidar and the depth information provided by the first point cloud set, the Cartesian coordinate range of the target area is calculated, and then converted into spherical coordinates to obtain the azimuth and elevation angle ranges, where [X, Y, Z] T Represents the three-dimensional coordinates of the point in the first point cloud, where X, Y, and Z represent the position of the point on the three coordinate axes, R and t represent the rotation matrix and translation matrix of the camera and optical phased array lidar, K represents the camera intrinsic parameter matrix, and Z c Represents the distance scale factor in the camera coordinate system, [u,v,1] T Represents the pixel coordinates of the point in the camera image coordinate system, where u and v represent the horizontal and vertical pixel coordinates of the point on the image plane, respectively.

[0054]

[0055] Example 2

[0056] The present invention also provides a three-dimensional imaging method based on an optical phased array lidar, which includes the method for obtaining real-time depth in any of the above embodiments; repeating the above steps to continuously obtain point cloud acquisition and image frames, and stopping acquisition if a preset condition is met.

[0057] In this embodiment, continuously acquiring point cloud data and image frames means continuously acquiring point cloud data and image information around the target during the LiDAR scanning process. This allows the complete three-dimensional shape and appearance characteristics of the target or scene to be captured. Stopping acquisition when a preset condition is met means setting certain conditions or rules to determine when to end data acquisition, such as reaching a preset number of times or a preset time.

[0058] Example 3

[0059] The present invention also provides an electronic device. In one embodiment of a device according to the present invention, the device includes a processor and a storage device. The storage device can be configured to store a program for executing the instant depth acquisition method of the above-mentioned method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to a program for executing the instant depth acquisition method of the above-mentioned method embodiment. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method section of the embodiment of the present invention. The control device can be a control device device formed by various electronic devices.

[0060] Example 4

[0061] The present invention also provides a computer-readable storage medium. In one embodiment of a computer-readable storage medium according to the present invention, the computer-readable storage medium can be configured to store a program for executing the instant depth acquisition method of the above-mentioned method embodiment, and the program can be loaded and run by a processor to implement the above-mentioned instant depth acquisition method. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present invention is a non-transitory computer-readable storage medium.

[0062] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the original technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A method for real-time depth acquisition, characterized in that: include: Acquire the first point cloud frame of the scene to be acquired at the current moment and the first image frame of the scene to be acquired at the current moment; The first point cloud frame includes a first point cloud acquisition and a first depth map; The first image frame includes a plurality of first target areas; The process of obtaining the first point cloud frame of the scene to be acquired at the current moment and the first image frame of the scene to be acquired at the current moment includes: performing a global scan on the scene to be acquired at the current moment to obtain the first point cloud acquisition at the current moment; obtaining the first depth map at the current moment based on the first point cloud acquisition; performing global image segmentation on the first image frame at the current moment to obtain a plurality of first target areas; wherein the first point cloud frame is the point cloud data acquired for the first time at the current moment, indicating the distribution of the surface of the object in the scene to be acquired in the three-dimensional space at the current moment, that is, the three-dimensional coordinate information of the object in the scene; the first point cloud acquisition is the point cloud data acquired by scanning the scene at the current moment through the optical phased array lidar; the first image frame is the image frame data acquired for the first time at the current moment, that is, the two-dimensional image of the object in the scene to be acquired; Obtaining a point cloud collection at a next moment and an image frame at a next moment, wherein the process of obtaining the point cloud collection at the next moment includes: performing regional adaptive scanning and global scanning on the scene to be acquired based on a plurality of first target areas, a first depth map, and scanning angle resolutions set for different target areas of the first image frame to obtain a point cloud collection at the next moment; specifically, based on the resolution set for each first target area, performing adaptive scanning on the scene to be acquired to obtain a first sub-point cloud collection; the resolution set for each first target area is also obtained based on the first depth map; performing a global scan on the scene to be acquired to obtain a second sub-point cloud collection; and obtaining the point cloud collection at the next moment based on the first sub-point cloud collection and the second sub-point cloud collection; A depth map at the next moment is acquired based on the point cloud collection at the next moment.

2. The method according to claim 1, characterized in that The process of performing a global scan on the scene to be acquired to obtain a second sub-point cloud collection includes at least: The scene is globally scanned uniformly at low resolution to obtain the second sub-point cloud collection.

3. The method according to claim 1, characterized in that The scanning angle resolutions set for different target areas are also acquired by optical phased array laser radar based on the target type and size.

4. A three-dimensional imaging method based on optical phased array laser radar, characterized in that: The method comprises the instantaneous depth acquisition method according to any one of claims 1 to 3; Repeat the above instant depth acquisition method to continuously acquire point cloud acquisition and image frames. If the preset conditions are met, stop the acquisition.

5. An electronic device comprising a processor and a storage device, wherein the storage device is adapted to store a plurality of program codes, wherein: The program code is suitable for being loaded and executed by a processor to perform the instant depth acquisition method according to any one of claims 1 to 3.

6. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and executed by a processor to perform the instant depth acquisition method according to any one of claims 1 to 3.

Images