Instant depth acquisition method, three-dimensional imaging method, equipment and medium

Abstract

The invention provides an instant depth acquisition method, a three-dimensional imaging method, equipment and a medium. The method comprises the following steps: acquiring a current-moment first point cloud frame of a to-be-acquired scene and a current-moment first image frame of the to-be-acquired scene; the first point cloud frame comprises first point cloud acquisition and a first depth map; the first image frame comprises a plurality of first target areas; the process of acquiring the point cloud acquisition at the next moment comprises the following steps: carrying out region adaptive scanning and global scanning on a scene to be acquired based on a plurality of first target regions of a first image frame, a first depth map and scanning angle resolutions set by different target regions; acquiring point cloud acquisition at the next moment; and acquiring a depth map at the next moment based on point cloud acquisition at the next moment. According to the method, on the basis of the point cloud and the image frame at the current moment, the point cloud collection and the depth map at the next moment can be obtained in real time, and real-time obtaining and processing of depth information are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional imaging, and specifically provides an instant depth acquisition method, a three-dimensional imaging method, a device and a medium. Background Art

[0002] With the rapid development of autonomous driving, robot navigation, augmented reality and other fields, higher requirements are placed on the three-dimensional perception of the environment. Although traditional three-dimensional imaging technologies, such as stereo vision and time-of-flight (ToF), can obtain three-dimensional information of the scene to a certain extent, they still have limitations in terms of measurement range, accuracy, speed, etc.

[0003] Optical phased array laser radar is a laser radar system with an optical phased array unit as the core scanning component. The phase difference between the waveguides on the optical chip is controlled by adjusting the driving voltage value, thereby controlling the deflection direction of the emitted light beam. In the prior art, the camera and the laser radar are usually calibrated together to obtain the external parameters from the camera to the optical phased array laser radar, that is, the rotation matrix and the translation vector; after obtaining the relative posture relationship between the camera and the optical phased array laser radar, the pixel coordinates under the camera field of view can be converted into the spatial coordinates under the optical phased array laser radar coordinate system. However, since the three-dimensional coordinates have been projected to the image plane during the camera imaging process, the lack of depth-related scaling factors will make it impossible to convert the two-dimensional pixel coordinates into three-dimensional space coordinates when the depth is unknown. Therefore, in the target adaptive imaging method, the key issue is to obtain the real-time depth of the scene so as to adopt a more accurate laser radar projection strategy for the areas where different targets are located in the scene.

[0004] Therefore, the art needs a new instant depth acquisition solution to solve the above problems. Summary of the invention

[0005] In order to overcome the above defects, the present invention is proposed to solve the problem of being unable to obtain the real-time depth of the scene.

[0006] In a first aspect, the present invention provides an instant depth acquisition method, comprising: acquiring a first point cloud frame of a scene to be acquired at a current moment, and a first image frame of the scene to be acquired at a current moment; the first point cloud frame includes a first point cloud acquisition and a first depth map; the first image frame includes a plurality of first target areas; acquiring a point cloud acquisition at the next moment, and an image frame at the next moment, wherein the process of acquiring the point cloud acquisition at the next moment includes: performing regional adaptive scanning and global scanning on the scene to be acquired based on a plurality of first target areas of the first image frame, a first depth map, and scanning angle resolutions set for different target areas, to acquire a point cloud acquisition at the next moment; and acquiring a depth map at the next moment based on the point cloud acquisition at the next moment.

[0007] In a technical solution of the above-mentioned instant depth acquisition method, the process of obtaining the first point cloud frame of the scene to be acquired at the current moment and the first image frame of the scene to be acquired at the current moment includes: performing a global scan of the scene to be acquired at the current moment to obtain the first point cloud acquisition at the current moment; obtaining the first depth map at the current moment based on the first point cloud acquisition; performing global image segmentation on the first image frame at the current moment to obtain several first target areas.

[0008] In a technical solution of the above-mentioned instant depth acquisition method, the scene to be acquired is subjected to regional adaptive scanning and global scanning based on several first target areas of the first image frame, the first depth map, and scanning angle resolutions set for different target areas, and the process of acquiring the point cloud acquisition at the next moment includes: based on the resolution set for each first target area, the scene to be acquired is adaptively scanned to obtain a first sub-point cloud acquisition; the resolution set for each first target area is also obtained based on the first depth map; the scene to be acquired is globally scanned to obtain a second sub-point cloud acquisition; and the point cloud acquisition at the next moment is acquired based on the first sub-point cloud acquisition and the second sub-point cloud acquisition.

[0009] In a technical solution of the instant depth acquisition method, the process of globally scanning the scene to be acquired to obtain the second sub-point cloud collection at least includes: globally uniformly scanning the scene at a low resolution to obtain the second sub-point cloud collection.

[0010] In a technical solution of the above-mentioned instant depth acquisition method, the scanning angle resolution set for different target areas is also set based on the target type and size to acquire the optical phased array lidar.

[0011] In a second aspect, the present invention provides a three-dimensional imaging method based on an optical phased array laser radar, the method comprising the instant depth acquisition method of any one of the technical solutions of the instant depth acquisition method described above;

[0012] Repeat the above steps to continuously acquire point cloud collection and image frames. If the preset conditions are met, stop the collection.

[0013] In a third aspect, an electronic device includes a processor and a storage device, the storage device is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by the processor to execute the instant depth acquisition method of any one of the technical solutions of the above-mentioned instant depth acquisition method.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium storing a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the instant depth acquisition method of any one of the technical solutions of the instant depth acquisition method described above.

[0015] The above one or more technical solutions of the present invention have at least one or more of the following beneficial effects:

[0016] In the technical solution for implementing the present invention, the present invention provides an instant depth acquisition method, including: acquiring the first point cloud frame of the scene to be acquired at the current moment, and the first image frame of the scene to be acquired at the current moment; the first point cloud frame includes the first point cloud acquisition and the first depth map; the first image frame includes several first target areas; acquiring the point cloud acquisition at the next moment, and the image frame at the next moment, wherein the process of acquiring the point cloud acquisition at the next moment includes: performing regional adaptive scanning and global scanning on the scene to be acquired based on several first target areas of the first image frame, the first depth map, and the scanning angle resolution set for different target areas, and acquiring the point cloud acquisition at the next moment; acquiring the depth map at the next moment based on the point cloud acquisition at the next moment. Compared with the prior art, the beneficial effects of the instant depth acquisition method provided by the present invention are:

[0017] 1. This method can instantly obtain the point cloud collection and depth map of the next moment based on the point cloud and image frame of the current moment, realizing the real-time acquisition and processing of depth information.

[0018] 2. According to the scanning resolution set for the target area, a combination of regional adaptive scanning and global scanning is used to scan the scene in a targeted manner. For example, more resources can be concentrated on the main target and the scanning of invalid areas can be reduced, thereby improving the utilization efficiency of optical phased array lidar resources and avoiding the generation of redundant data. To a certain extent, it solves the problem of excessive demand for computing resources for OPA lidar distance solution and real-time imaging, and can effectively improve the frame rate of the imaging system. The depth information obtained through global scanning can supplement the depth information that may be missing during the target adaptive scanning process, making the imaging process more complete and accurate. This can avoid the collection and processing of invalid data due to the lack of depth information and reduce the generation of redundant data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The disclosure of the present invention will become more easily understood with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In addition, similar numbers in the figures are used to represent similar components, among which:

[0020] Figure 1 is a schematic flow chart of main steps of a method for instant depth acquisition according to an embodiment of the present invention;

[0021] Figure 2It is a flowchart diagram of main steps of a method for acquiring a first point cloud frame of a scene to be acquired at a current moment and a first image frame of a scene to be acquired at a current moment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] Some embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0023] Embodiment 1

[0024] See attached Figure 1 , Figure 1 FIG. 1 is a flow chart of the main steps of a method for acquiring instant depth according to an embodiment of the present invention. Figure 1 As shown, the instant depth acquisition method in the embodiment of the present invention mainly includes the following steps S1 to S3.

[0025] Step S1, obtaining a first point cloud frame of a scene to be acquired at a current moment and a first image frame of a scene to be acquired at a current moment; the first point cloud frame includes a first point cloud acquisition and a first depth map; the first image frame includes a plurality of first target areas;

[0026] In this embodiment, the scene to be acquired may refer to any scene that requires point cloud data processing and image data analysis, such as an indoor room, a road traffic scene, an electronic production line, etc. The current moment is the moment of current acquisition and processing, or it may be any specified time point. The first point cloud frame refers to the point cloud data acquired for the first time at the current moment, which represents the distribution of the surface of the object in the scene in three-dimensional space at the current moment, including the three-dimensional coordinate information of the object in the scene. The first point cloud acquisition refers to the point cloud data acquired by scanning the scene at the current moment through an optical phased array laser radar. The first depth map refers to the distance information image acquired according to the image data acquired by the camera at the current moment, and its pixel value reflects the distance from each point in the scene to the camera. The first image frame refers to the image frame data acquired for the first time at the current moment, and an ordinary RGB camera or a depth camera is usually used to capture a two-dimensional image of the scene. These images can provide the appearance and texture information of the scene, and are used together with the point cloud data for depth perception and environmental understanding. The first target area refers to the area of ​​objects of interest captured in the scene at the current moment.

[0027] In one embodiment, Figure 2 As shown, the process of step S1, obtaining the first point cloud frame of the scene to be obtained at the current moment and the first image frame of the scene to be obtained at the current moment, includes:

[0028] Step S11, performing a global scan on the scene to be acquired at the current moment to acquire the first point cloud collection at the current moment;

[0029] In this embodiment, the optical phased array laser radar emits a laser beam, which interacts with objects in the scene, is received by the laser radar and converted into an electrical signal, which is then converted into point cloud data. These point cloud data will be organized into a three-dimensional point cloud collection, each point cloud representing a spatial position and its corresponding feature information. Global scanning ensures that the scene point cloud is as complete and accurate as possible.

[0030] Step S12: acquiring a first depth map at a current moment based on the first point cloud acquisition;

[0031] In this embodiment, the distance between each point cloud and the camera can be calculated by processing and analyzing the first point cloud acquisition. Since the optical phased array lidar can quickly collect point cloud data, a more efficient algorithm (such as an algorithm based on the scene wavefront) can be used to improve the calculation speed and accuracy when calculating the depth map. The depth map can provide more accurate depth information and can obtain the geometric shape of obstacles within a certain range.

[0032] Step S13: performing global image segmentation on the first image frame at the current moment to obtain a plurality of first target areas.

[0033] In this embodiment, global image segmentation is a process of segmenting the entire image into different regions or objects. In this process, a target region can be obtained, and this target region can be a pixel region that only contains a target object.

[0034] Step S2, obtaining the point cloud collection at the next moment and the image frame at the next moment, wherein the process of obtaining the point cloud collection at the next moment includes: performing regional adaptive scanning and global scanning on the scene to be acquired based on a plurality of first target areas of the first image frame, the first depth map, and scanning angle resolutions set for different target areas, to obtain the point cloud collection at the next moment;

[0035] In this embodiment, the image frame at the next moment refers to the point cloud acquisition performed after a period of time from the current point cloud acquisition, that is, the point cloud data at the next time point. The image frame at the next moment refers to the image frame acquired after a period of time from the current image, that is, the image data at the next time point. The scanning angle resolution refers to the angular spacing of the laser radar in the horizontal and vertical directions.

[0036] The adaptive scanning process is to scan each target area separately to meet its specific resolution requirements, rather than uniformly scanning the entire scene. According to the resolution set for each target area, regional adaptive scanning can improve the point cloud density of the corresponding target to obtain higher point cloud precision and accuracy. Global scanning is for when the camera captures a new target, because the previous scanning data cannot provide the depth information of the new target, it is impossible to guide the optical phased array lidar to collect the corresponding data. Therefore, a global scan is required to obtain the depth map in the scene to provide the missing depth information during the target adaptive scanning process.

[0037] Alternatively, in an alternative manner, in the regional adaptive scanning and global scanning of the scene to be acquired based on the first target areas of the first image frame, the first depth map, and the scanning angle resolutions set for different target areas, regional adaptive scanning and global sparse scanning can be used. Global sparse scanning acquires depth information by sampling relatively few points in the entire scanning range. This can reduce the sampling density, the amount of data, and the amount of calculation.

[0038] The first image frame provides initial information about the scene to be acquired. By analyzing the first image frame, the target area of ​​interest can be determined and combined with the first depth map to estimate the position and shape of the object. The setting of different scanning angle resolutions allows flexible adjustment of the density and accuracy of point cloud acquisition. By using the data of the previous frame, continuity and consistency can be maintained. In continuous scanning, the position and shape of the object may change, but the data of adjacent frames are usually highly correlated. Therefore, using the data of the previous frame as a reference can better track and analyze the changes of the object. Secondly, the data of the previous frame is already available and closest to the current frame. By using relatively old but already available data for adaptive scanning of the target area, the dependence on immediate data can be reduced, thereby improving real-time performance. Finally, using the data of the previous frame can reduce the amount of calculation and complexity. The processing of camera image frames and lidar frames is usually a complex and time-consuming process. By only using the data of the previous frame, computing resources can be saved and the amount of calculation and delay of the algorithm can be reduced.

[0039] In summary, by performing regional adaptive scanning and global scanning of the scene to be acquired based on several first target areas of the first image frame, the first depth map, and scanning angle resolutions set for different target areas, the point cloud data acquisition process can be optimized, and a more accurate, comprehensive and high-quality point cloud set can be provided at the next moment.

[0040] In one embodiment, based on a plurality of first target areas of the first image frame, the first depth map, and scanning angle resolutions set for different target areas, the scene to be acquired is subjected to regional adaptive scanning and global scanning, and a process of acquiring point cloud acquisition at the next moment includes:

[0041] Based on the resolution set for each first target area, the scene to be acquired is adaptively scanned to obtain a first sub-point cloud collection; the resolution set for each first target area is also obtained based on the first depth map;

[0042] Performing a global scan on the scene to be acquired to obtain a second sub-point cloud collection;

[0043] The point cloud collection at the next moment is acquired based on the first sub-point cloud collection and the second sub-point cloud collection.

[0044] In this embodiment, the resolution set in the target area refers to the density of the laser radar sampling points set around the target object to obtain more accurate imaging of the target object. By setting the resolution of different areas, different densities of laser radar point strategies can be adopted for different target objects. Real-time depth information can provide the precise distance from the target object to the laser radar. When considering the resolution set for each target area, the sampling density of the laser radar can be adjusted according to the position and distance of the target object. For distant target objects, the resolution can be reduced to reduce processing costs, while for close target objects, the resolution can be increased to obtain more detailed information.

[0045] According to the resolution set for each first target area, the scene to be acquired is adaptively scanned to obtain the first part of the point cloud data. This means that according to the set resolution, each target area of ​​the scene will be scanned as needed to obtain the first part of the point cloud acquisition data, which can help improve the acquisition efficiency and reduce the amount of data. Then the entire scene to be acquired is globally scanned to obtain the second part of the point cloud data. Compared with adaptive scanning, global scanning covers a wider range and can obtain more comprehensive point cloud data. The point cloud obtained by performing regional adaptive scanning and global sparse scanning on the scene is: point cloud acquisition at the next moment = first sub-point cloud acquisition ∪ second sub-point cloud acquisition.

[0046] In one embodiment, the process of performing a global scan on the scene to be acquired to obtain the second sub-point cloud collection at least includes: performing a global uniform scan on the scene at a low resolution to obtain the second sub-point cloud collection.

[0047] In this embodiment, low resolution can mean that the number of points in the sub-point cloud is relatively small, or the position information accuracy of each point is low. The reason for performing a global uniform scan of the scene at a low resolution to obtain the second sub-point cloud is to be able to comprehensively and quickly obtain the overall characteristics of the scene. Specifically, the use of a low-resolution scanning method can improve the speed and efficiency of data acquisition, while reducing the amount of data required to be stored and processed, thereby speeding up the acquisition and processing of point cloud data. In addition, within the range of low resolution, the entire scene to be acquired can be covered, thereby obtaining the global characteristics of the scene.

[0048] In one embodiment, the scanning angle resolutions set for different target areas are also acquired by optical phased array laser radar based on the target type and size.

[0049] In this embodiment, different types of targets may require different resolutions to obtain valid data. For example, a small stationary target may require a higher resolution to capture small features or edges, while a large moving target may only require a lower resolution to detect its overall shape and position. The size of the target will also affect the setting of the scanning angle resolution. For larger targets, a lower resolution can be used, while small targets may require a higher resolution to ensure accurate data collection.

[0050] Step S3: acquiring a depth map at the next moment based on the point cloud collection at the next moment.

[0051] In this embodiment, according to the point cloud acquisition data at the next moment obtained in step S2, an instant depth map can be generated by processing the point cloud data, thereby obtaining the depth information of the objects in the scene.

[0052] The azimuth and elevation angles of the target area can be obtained through depth information. The azimuth indicates the position of the target relative to the sensor in the horizontal plane, and the elevation indicates the position of the target relative to the sensor in the vertical direction. These azimuth and elevation information can be used in adaptive scanning to adjust the scanning method of the lidar. For example, when the target is directly in front of the sensor, the scanning range can be set wider to obtain more detailed information about the target. When the target is on the side of the sensor, the scanning range can be reduced to improve scanning efficiency. At the same time, the azimuth and elevation angles can also be used to determine the resolution of the scan. For targets farther away from the sensor, a larger scanning step size can be used to speed up the scanning speed. For targets closer to the sensor, a smaller scanning step size can be used to improve the accuracy of the data. Therefore, depth information can affect the range, angle, and resolution of adaptive scanning by providing the azimuth and elevation angles of the target pixel area, thereby achieving more accurate and efficient lidar scanning.

[0053] The process of calculating the azimuth and elevation angle of the target pixel area is specifically as follows: according to formula (1), based on the pose matrix of the camera and the optical phased array laser radar and the depth information provided by the first point cloud set, the Cartesian coordinate range of the target area is calculated, and then converted into spherical coordinates to obtain the azimuth and elevation angle ranges, where [X, Y, Z] T represents the three-dimensional coordinates of the point in the first point cloud, where X, Y, and Z represent the position of the point on the three coordinate axes, R and t represent the rotation matrix and translation matrix of the camera and the optical phased array lidar, K represents the camera intrinsic parameter matrix, and Z c Represents the distance scale factor in the camera coordinate system, [u,v,1] T Represents the pixel coordinates of the point in the camera image coordinate system, where u and v represent the horizontal and vertical pixel coordinates of the point on the image plane, respectively.

[0054]

[0055] Embodiment 2

[0056] The present invention also provides a three-dimensional imaging method based on an optical phased array lidar, which includes the method for acquiring real-time depth in any of the above embodiments; repeating the above steps to continuously acquire point cloud acquisition and image frames, and stopping acquisition if a preset condition is met.

[0057] In this embodiment, continuously acquiring point cloud data and image frames means continuously acquiring point cloud data and image information around the target during the laser radar scanning process. In this way, the complete three-dimensional shape and appearance characteristics of the target or scene can be captured. Stopping acquisition when a preset condition is reached means setting certain conditions or rules to determine when to end data acquisition, such as reaching a preset number of times or a preset time.

[0058] Embodiment 3

[0059] The present invention also provides an electronic device. In an embodiment of a device according to the present invention, the device includes a processor and a storage device, the storage device can be configured to store a program for executing the instant depth acquisition method of the above method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to the program for executing the instant depth acquisition method of the above method embodiment. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The control device can be a control device device formed by various electronic devices.

[0060] Embodiment 4

[0061] The present invention also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present invention, the computer-readable storage medium can be configured to store a program for executing the instant depth acquisition method of the above-mentioned method embodiment, and the program can be loaded and run by the processor to implement the above-mentioned instant depth acquisition method. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present invention is a non-temporary computer-readable storage medium.

[0062] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the original technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A real-time depth acquisition method, characterized in that: include: Acquire the first point cloud frame of the scene to be acquired at the current moment and the first image frame of the scene to be acquired at the current moment; The first point cloud frame includes a first point cloud acquisition and a first depth map; The first image frame includes a plurality of first target areas; Acquire the next moment point cloud collection and the next moment image frame, wherein the process of acquiring the next moment point cloud collection includes: based on a plurality of first target areas of the first image frame, the first depth map, and scanning angle resolutions set for different target areas, perform regional adaptive scanning and global scanning on the scene to be acquired to acquire the next moment point cloud collection; A depth map at the next moment is acquired based on the point cloud collection at the next moment.

2. The method according to claim 1, characterized in that The process of acquiring the first point cloud frame of the scene to be acquired at the current moment and the first image frame of the scene to be acquired at the current moment includes: Performing a global scan on the scene to be acquired at the current moment to acquire the first point cloud acquisition at the current moment; Acquire a first depth map at a current moment based on the first point cloud acquisition; Perform global image segmentation on the first image frame at the current moment to obtain a plurality of first target areas.

3. The method according to claim 1, characterized in that: Based on a plurality of first target areas of the first image frame, the first depth map, and scanning angle resolutions set for different target areas, the scene to be acquired is subjected to regional adaptive scanning and global scanning, and a process of acquiring point cloud acquisition at the next moment includes: Based on the resolution set for each first target area, the scene to be acquired is adaptively scanned to obtain a first sub-point cloud collection; the resolution set for each first target area is also obtained based on the first depth map; Performing a global scan on the scene to be acquired to obtain a second sub-point cloud collection; The point cloud collection at the next moment is acquired based on the first sub-point cloud collection and the second sub-point cloud collection.

4. The method according to claim 3, characterized in that: The process of performing a global scan on the scene to be acquired to obtain the second sub-point cloud acquisition at least includes: The scene is globally scanned uniformly at a low resolution to obtain the second sub-point cloud collection.

5. The method according to claim 3, characterized in that: The scanning angle resolutions set for different target areas are also acquired by optical phased array laser radar based on the target type and size.

6. A three-dimensional imaging method based on optical phased array laser radar, characterized in that: The method comprises an instantaneous depth acquisition method based on any one of claims 1 to 5; Repeat the above steps to continuously acquire point cloud collection and image frames. If the preset conditions are met, stop the collection.

7. An electronic device comprising a processor and a storage device, the storage device being adapted to store a plurality of program codes, characterized in that: The program code is suitable for being loaded and executed by a processor to execute the instant depth acquisition method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and executed by a processor to execute the instant depth acquisition method according to any one of claims 1 to 5.

Images