Three-dimensional profile reconstruction method, device, and storage medium

Abstract

The present disclosure provides a three-dimensional profile reconstruction method, device and storage medium, wherein a main processor and a coprocessor are configured for a processor in a 3D camera, the coprocessor controls an optical module to perform image acquisition on a measured object, and transmits the acquired image to the main processor; the main processor performs three-dimensional profile reconstruction on the image to generate a three-dimensional point cloud of the measured object. In the embodiment, the division of labor between the main processor and the coprocessor realizes decoupling of the image acquisition task and the image calculation task, guarantees the stability, real-time performance and synchronization of the image acquisition task, and the image acquisition task and the image calculation task are in pipeline parallelism, which can improve the overall throughput, thereby effectively improving the quality and efficiency of the three-dimensional profile reconstruction, and the acquisition program used for the image acquisition task and the calculation program used for the image calculation task can be independently developed, which also improves the cross-platform and portability.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to methods, devices and storage media for three-dimensional contour reconstruction. Background Technology

[0002] A 3D camera is a three-dimensional contour measurement device that projects a structured light pattern onto the surface of the object being measured, captures the distortion of the structured light pattern caused by the undulations of the object's surface, and then reconstructs the three-dimensional point cloud of the object based on the principle of triangulation.

[0003] Traditional 3D camera processors are typically used for image acquisition and computation. However, with increasing demands for accuracy and speed, multiple high-resolution images need to be processed simultaneously, and heavy image signal processing is required. 3D camera processors face enormous data throughput and real-time pressure, affecting the quality and efficiency of 3D contour reconstruction. Summary of the Invention

[0004] This disclosure provides methods, apparatus, and storage media for three-dimensional contour reconstruction in various aspects, thereby decoupling image acquisition tasks and image computation tasks and improving the quality and efficiency of three-dimensional contour reconstruction.

[0005] The first aspect of this disclosure provides a three-dimensional contour reconstruction method applied to a processor of a 3D camera, wherein the 3D camera includes an optical module and the processor, and the processor includes a main processor and a coprocessor; the method includes:

[0006] The coprocessor controls the optical module to acquire images of the object under test and transmits the acquired images to the main processor.

[0007] The main processor performs three-dimensional contour reconstruction on the image to generate a three-dimensional point cloud of the object under test.

[0008] A second aspect of this disclosure provides a 3D camera, the 3D camera including an optical module and a processor, the processor including a main processor and a coprocessor;

[0009] The coprocessor is used to control the optical module to acquire images of the object under test and to transmit the acquired images to the main processor.

[0010] The main processor is used to reconstruct the three-dimensional contour of the image and generate a three-dimensional point cloud of the object under test.

[0011] A third aspect of this disclosure provides a three-dimensional contour reconstruction system, the three-dimensional contour reconstruction system including a processor optical module, wherein the processor is used to perform the method of the first aspect; or, the three-dimensional contour reconstruction system includes a 3D camera, the 3D camera being the 3D camera of the second aspect.

[0012] A fourth aspect of this disclosure provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of the first aspect.

[0013] The fifth aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of the first aspect described above.

[0014] The sixth aspect of this disclosure provides a computer program product comprising: a computer program stored in a readable storage medium, wherein at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the method of the first aspect described above.

[0015] The three-dimensional contour reconstruction method, device, and storage medium disclosed in this embodiment configure a main processor and a coprocessor in the 3D camera. The coprocessor controls the optical module to acquire images of the object under test and transmits the acquired images to the main processor. The main processor then performs three-dimensional contour reconstruction on the images to generate a three-dimensional point cloud of the object under test. This embodiment decouples the image acquisition task and the image calculation task through the division of labor between the main processor and the coprocessor, ensuring the stability, real-time performance, and synchronization of the image acquisition task. Furthermore, the parallel pipeline of the image acquisition and image calculation tasks improves the overall throughput, thereby effectively improving the quality and efficiency of three-dimensional contour reconstruction. Additionally, the acquisition program used for the image acquisition task and the calculation program used for the image calculation task can be developed independently, improving cross-platform compatibility and portability. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0017] Figure 1 An application scenario diagram of a three-dimensional contour reconstruction method provided for an exemplary embodiment of this disclosure;

[0018] Figure 2 A flowchart illustrating the steps of a three-dimensional contour reconstruction method provided as an exemplary embodiment of this disclosure;

[0019] Figure 3 A schematic diagram of a 3D camera architecture provided for an exemplary embodiment of this disclosure;

[0020] Figure 4A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0022] Traditional 3D cameras typically use a single processor to handle image acquisition and computation. However, with increasing demands for accuracy and speed, it is necessary to process multiple high-resolution images simultaneously and perform heavy image signal processing. The single processor of a 3D camera faces enormous data throughput and real-time pressure, affecting the quality and efficiency of 3D contour reconstruction.

[0023] To address the aforementioned technical problems, this disclosure provides a three-dimensional contour reconstruction method. The processor of the 3D camera is configured as both a main processor and a coprocessor. This division of labor between the main processor and the coprocessor decouples the image acquisition task from the image computation task, ensuring the stability, real-time performance, and synchronization of the image acquisition task. Furthermore, the image acquisition and image computation tasks are pipelined in parallel, improving overall throughput and thus effectively enhancing the quality and efficiency of three-dimensional contour reconstruction. Additionally, the acquisition program used for the image acquisition task and the computation program used for the image computation task can be developed independently, improving cross-platform compatibility and portability.

[0024] like Figure 1 As shown, in this embodiment, the 3D camera may include an optical module and a processor. The processor includes a main processor and a coprocessor. The coprocessor controls the optical module to acquire images of the object under test and transmits the acquired images to the main processor. The main processor reconstructs the three-dimensional contour of the images to generate a three-dimensional point cloud of the object under test.

[0025] Figure 2 This is a flowchart illustrating the steps of a three-dimensional contour reconstruction method provided as an exemplary embodiment of the present disclosure. The three-dimensional contour reconstruction method is applied to a processor of a 3D camera. The 3D camera includes an optical module and a processor, which includes a main processor and a coprocessor, such as... Figure 2 As shown, the three-dimensional contour reconstruction method specifically includes the following steps:

[0026] S201. The coprocessor controls the optical module to acquire images of the object under test and transmits the acquired images to the main processor.

[0027] S202. The main processor performs three-dimensional contour reconstruction on the image to generate a three-dimensional point cloud of the object under test.

[0028] In this embodiment, a system-on-a-chip architecture is provided for the 3D camera processor, where a main processor and a coprocessor work together. The coprocessor is mainly used to control the optical module of the 3D camera to acquire images of the object under test, and the acquired images are transmitted to the main processor. The main processor is mainly used for image calculation, that is, to perform three-dimensional contour reconstruction based on the images. This realizes an architecture and data flow that separates acquisition and calculation, that is, the image acquisition task and the image calculation task are decoupled. Its advantages are:

[0029] 1) Ensure the stability, real-time performance, and synchronization of image acquisition tasks, unaffected by image computation tasks. Image acquisition tasks can be ensured to be acquired strictly according to hardware timing, unaffected by other fluctuations in the processor system (such as a sudden increase in the computational load of a certain frame), and avoid frame loss.

[0030] 2) Improved cross-platform compatibility and portability: The acquisition program (Software Development Kit, SDK) used for image acquisition tasks and the computing program used for image computing tasks can be independently optimized and developed. Hardware engineers and driver engineers can focus on the acquisition SDK, while algorithm engineers can focus on the computing SDK (Single Instruction Multiple Data Stream (SIMD) instructions, GPU acceleration, etc.). The acquisition SDK heavily relies on hardware and operating system drivers; when changing cameras or projectors, only the acquisition SDK needs to be replaced or adapted, while the computing SDK requires minimal modification. The computing SDK can be independently optimized and developed for different computing platform architectures, maximizing hardware performance. The acquisition SDK focuses on stable and efficient data acquisition, while the computing SDK focuses on algorithm accuracy and speed. The two communicate through a clearly defined interface, facilitating independent development, testing, debugging, and upgrades.

[0031] 3) Parallel pipeline: When the image computing task is processing the Nth set of images, the image acquisition task can continue to acquire subsequent images, such as the N+1th set of images or the N+2th set of images, which greatly improves the overall throughput.

[0032] Optionally, the coprocessor can be implemented using an FPGA (Field Programmable Gate Array), or other chips such as an ASIC (Application Specific Integrated Circuit). The main processor can be implemented using a CPU, GPU, or other similar devices.

[0033] The optical module includes at least a projection unit and an image sensor. The projection unit is used to project a structured light pattern, such as a sinusoidal stripe pattern, onto the object being measured, while the image sensor is used to acquire a structured light image of the surface of the object being measured, such as a stripe image of a sinusoidal stripe pattern modulated by the surface of the object being measured.

[0034] Specifically, the main processor performs three-dimensional contour reconstruction on the image to generate a three-dimensional point cloud of the object under test. Specifically, the wrapping phase can be calculated based on the stripe image, and the absolute phase can be calculated through unwrapping. Based on the phase and the camera-projection unit calibration model, phase-depth mapping information is generated and finally converted into three-dimensional point cloud coordinates (i.e., X, Y, Z coordinates) to generate the three-dimensional point cloud of the object under test.

[0035] Of course, other acquisition methods and three-dimensional contour reconstruction methods can also be used in this embodiment, and no limitation is made in this embodiment.

[0036] The 3D contour reconstruction method provided in this embodiment uses a coprocessor to control an optical module to acquire images of the object under test, and then transmits the acquired images to the main processor. The main processor then performs 3D contour reconstruction on the images to generate a 3D point cloud of the object under test. This embodiment decouples the image acquisition task and the image computation task by dividing the work between the main processor and the coprocessor, ensuring the stability, real-time performance, and synchronization of the image acquisition task. Furthermore, the parallel pipeline of the image acquisition and image computation tasks improves overall throughput, thereby effectively improving the quality and efficiency of 3D contour reconstruction. Additionally, the acquisition program used for the image acquisition task and the computation program used for the image computation task can be developed independently, improving cross-platform compatibility and portability.

[0037] In one optional embodiment, the optical module includes a projection unit and a monochrome image sensor, and the coprocessor includes a first coprocessor; S201 controls the optical module to acquire images of the object under test through the coprocessor, specifically including:

[0038] The first coprocessor controls the projection unit to project a preset structured light pattern onto the surface of the object being measured, and controls the black and white image sensor to acquire black and white stripe images of the surface of the object being measured.

[0039] In this embodiment, considering that traditional 3D cameras typically use color sensors, the Bayer filter array of a color sensor requires interpolation to obtain the RGB values ​​of each pixel, which leads to image blurring and resolution loss. In contrast, a monochrome sensor, because it lacks color filters (such as Bayer filters), allows each pixel to receive light of all wavelengths, resulting in greater light intake, a higher signal-to-noise ratio, and generally higher resolution and sensitivity, avoiding the spatial resolution loss caused by color filtering. Furthermore, structured light patterns (such as sinusoidal stripes) are typically monochromatic (e.g., blue or white). On a color sensor, if a monochromatic structured light pattern is projected, only the corresponding color channel responds, while the other two channels have weak signals, effectively wasting two-thirds of the pixels. Using a monochrome sensor avoids the problems caused by color channel separation, fully utilizing each pixel. Based on these reasons, this embodiment uses a monochrome sensor to acquire structured light patterns on the surface of the object being measured, obtaining a monochrome striped image. In addition, if high accuracy is required for 3D contour reconstruction, a high-resolution monochrome sensor can be selected. Furthermore, monochrome sensors can be designed for higher frame rates because the amount of data is relatively small (without multiple color channels), making them suitable for high-speed 3D scanning. When high-speed 3D scanning is required, a monochrome sensor can be selected.

[0040] In this embodiment, the first coprocessor can control the projection unit to project a preset structured light pattern onto the surface of the object being measured, and control the black and white image sensor to acquire the structured light pattern on the surface of the object being measured to obtain a black and white striped image.

[0041] More specifically, the first coprocessor integrates a timing generator and waveform generation logic to directly drive optomechanical modules such as digital light processing or laser scanning; the storage unit of the first coprocessor can also cache multiple projection patterns, and the pattern sequence in the storage unit can be loaded through the parallel bus of DLP for projection control, which can realize the pre-loading of projection patterns and custom projection order control.

[0042] In addition, the first coprocessor can also perform synchronization and timing control, and can generate pixel clock, line synchronization (HSYNC), field synchronization (VSYNC) and other signals that are synchronized with the projected pattern data. While projecting each frame of the projected pattern, it generates precise hardware trigger pulses to precisely trigger the exposure of the black and white image sensor, ensuring one-to-one correspondence.

[0043] Furthermore, the black and white image sensor can transmit the acquired black and white striped image to the first coprocessor via the high-speed serial image data interface SLVS-EC (Scalable Low-Voltage Signaling with Embedded Clock, high-speed, long-distance transmission, industrial-grade image transmission interface) or other transmission interfaces.

[0044] Accordingly, S202 performs 3D contour reconstruction on the image through the main processor, generating a 3D point cloud of the measured object, including:

[0045] The main processor performs 3D contour reconstruction on the black and white striped image to generate a 3D point cloud of the object under test.

[0046] In this embodiment, since a black-and-white sensor is used, a black-and-white striped image is obtained. Therefore, when the main processor performs 3D contour reconstruction, 3D contour reconstruction based on the black-and-white striped image can generate a black-and-white 3D point cloud of the measured object. Less color and texture information is retained, making it suitable for scenarios where texture quality requirements are not high. The specific process of 3D contour reconstruction based on the black-and-white striped image can be as follows: calculate the wrapping phase based on the black-and-white striped image, calculate the absolute phase through unwrapping processing, generate phase-depth mapping information based on the phase and the camera-projection unit calibration model, and finally convert it into 3D point cloud coordinates (i.e., X, Y, Z coordinates) to generate a black-and-white 3D point cloud of the measured object.

[0047] In another alternative embodiment, see Figure 3 If the 3D contour reconstruction has high requirements for texture quality, then a black-and-white sensor and a color sensor can be used. That is, the optical module includes a projection unit, a black-and-white image sensor and a color image sensor. The black-and-white image sensor is still used to acquire black-and-white stripe images, while the color sensor is used to acquire two-dimensional color images of the object being measured. It can be used to add color information to the 3D point cloud generated based on the black-and-white stripe image to generate a color 3D point cloud. This retains the advantages of the black-and-white image sensor while also meeting the texture quality requirements, and overcomes the disadvantage of using a color sensor alone to acquire color stripe images to directly generate a color 3D point cloud.

[0048] Furthermore, considering that the coprocessor needs to control the optical module to acquire black-and-white stripe images and two-dimensional color images, i.e., control the projection unit, black-and-white image sensor, and color sensor, this embodiment is equipped with two coprocessors, including a first coprocessor and a second coprocessor. The first coprocessor controls the projection unit to project a preset structured light pattern onto the surface of the object under test and controls the black-and-white image sensor to acquire the black-and-white stripe image of the surface of the object under test; the second coprocessor controls the color image sensor to acquire the two-dimensional color image of the object under test. In this way, the acquisition tasks of acquiring black-and-white stripe images and two-dimensional color images do not interfere with each other, the data paths are decoupled, bandwidth is not contested, high-precision time synchronization can be achieved, and parallel preprocessing is also possible, which also facilitates separate development, upgrades and iterations.

[0049] When the second coprocessor controls the color image sensor to acquire two-dimensional color images, the color image sensor can output multiple pairs of parallel LVDS (Low-Voltage Differential Signaling) differential signals (such as data + clock). The second coprocessor pre-builds an LVDS receiver, connects it to the RGB color image sensor, converts each pair of LVDS differential signals into single-ended signals, and performs serial-to-parallel conversion. Based on the clock and synchronization signals, it performs time-series alignment and data recombination of the multiple parallel data into complete pixel data to obtain a high-quality two-dimensional color image.

[0050] Furthermore, the first coprocessor transmits the black and white striped image to the main processor, and the second coprocessor transmits the two-dimensional color image to the main processor. When the main processor performs three-dimensional contour reconstruction to generate a three-dimensional point cloud of the measured object, it may specifically include:

[0051] The main processor performs three-dimensional contour reconstruction on the black and white striped image to generate a black and white three-dimensional point cloud of the object under test.

[0052] The main processor maps the color information of the two-dimensional color image onto the black and white three-dimensional point cloud to generate the color three-dimensional point cloud of the object being measured.

[0053] In this embodiment, the main processor can calculate the wrapped phase based on the black and white striped image, solve for the absolute phase through unwrapping, and generate phase-depth mapping information based on the phase and the camera-projection unit calibration model. This information is then converted into three-dimensional point cloud coordinates (i.e., X, Y, Z coordinates), generating a black and white three-dimensional point cloud of the object being measured. Furthermore, the main processor can map each pixel of the two-dimensional color image to the black and white three-dimensional point cloud according to a pre-defined transformation relationship, assigning the color information of each pixel to the corresponding point cloud in the black-bordered three-dimensional point cloud, thus generating a color three-dimensional point cloud.

[0054] Based on any of the above embodiments, see Figure 3 When the S201 coprocessor transmits the acquired image to the main processor, it may specifically include:

[0055] The acquired images are cached in the coprocessor's storage unit by the coprocessor;

[0056] The image is retrieved from the coprocessor's storage unit by the first processing unit in the main processor and stored in shared memory.

[0057] In this embodiment, a first processing unit is configured in the main processor as a standardized interface layer with the coprocessor. The coprocessor can cache the acquired images in its storage unit, and the first processing unit in the main processor retrieves the images from the coprocessor's storage unit. Through the first processing unit of the main processor, the main processor's image computing tasks do not need to care about how the coprocessor acquires and stores images, reducing the development and upgrade costs of the computing SDK. Furthermore, the first processing unit of the main processor can synchronously manage multiple coprocessors and retrieve images from multiple coprocessors, achieving unified scheduling and ensuring the synchronization of image retrieval from multiple coprocessors. In addition, the first processing unit of the main processor can control the coprocessor to trigger acquisition as needed, dynamically configure acquisition parameters, and perform data verification to prevent erroneous data from entering the image computing task.

[0058] Optionally, the coprocessor can cache the acquired images in the coprocessor's storage unit. For the first coprocessor, the rapidly flowing black and white striped images can be written into the first coprocessor's storage unit (such as DDR memory) through a FIFO (First Input First Output) mechanism. Through an efficient flow control mechanism, data loss can be prevented.

[0059] Optionally, the first processing unit may be one or more cores in the main processor, and a pre-configured SDK may be deployed on the first processing unit to implement the above functions.

[0060] Furthermore, considering the limited storage space of the coprocessor and main processor, if the first processing unit of the main processor stores the image acquired from the coprocessor in the main processor's storage space, it may occupy the main processor's memory, affecting the image calculation task. Also, if the first processing unit does not acquire the image from the coprocessor in a timely manner, the coprocessor's storage space will fill up, also affecting the image acquisition task. Therefore, in this embodiment, an additional shared memory can be configured. After the first processing unit of the main processor acquires the image from the coprocessor, it can store the image in the shared memory. When the main processor performs 3D contour reconstruction on the image, it can acquire the image from the shared memory and perform 3D contour reconstruction through the second processing unit of the main processor. The second processing unit is the processing unit in the main processor used for image calculation tasks and can deploy a computing SDK. Through shared memory, the image acquisition task and the image calculation task can be further decoupled, allowing more computing power from the main processor to be allocated to the image calculation task. It also allows for unified management of image data from multiple coprocessors, ensuring image synchronization and integrity, and leveraging the maximum transmission speed of shared memory to meet the high throughput requirements of the 3D camera.

[0061] Optionally, when the first processing unit in the main processor retrieves an image from the storage unit of the coprocessor, it can do so based on the Direct Memory Access (DMA) mechanism. This enables efficient data transmission with high bandwidth (>1GB / s), low latency (microseconds), and low CPU usage, further reducing latency.

[0062] Alternatively, the first processing unit in the main processor can also obtain images from the coprocessor's storage unit through other means such as MIPI-SCI or PCIe bus.

[0063] In addition, when the second processing unit of the main processor retrieves the image from shared memory and performs 3D contour reconstruction, it can use a DMA mechanism to write the image to the main processor memory or GPU memory for 3D contour reconstruction. This can provide a high-quality, low-latency data foundation and further reduce latency.

[0064] Based on any of the above embodiments, after S201 controls the optical module to acquire images of the object under test through the coprocessor, the image can also be preprocessed through the coprocessor; furthermore, the preprocessed image is transmitted to the main processor through the coprocessor.

[0065] In this embodiment, the first coprocessor can perform preprocessing on the black and white striped image, including but not limited to one or more of the following:

[0066] Black level correction: The sensor's base noise is subtracted from the sensor's original output value by a configurable register, which is to subtract the preset black level value configured in the register to obtain the corrected value, thus avoiding deviations caused by black level.

[0067] Defect pixel correction: It can repair sensor dead pixels. It can identify image sensor dead pixels in any feasible way, such as pre-storing the coordinates of dead pixels, or judging dead pixels by a preset threshold, and then using the neighboring pixel values ​​(mean or median) of the dead pixels to replace the dead pixel values.

[0068] Denoising filtering: Hardware-optimized filters (such as spatial domain filtering, grayscale domain filtering, etc.) are used to effectively suppress shot noise while preserving image edge information.

[0069] The second coprocessor can complete a full-featured ISP (Image Signal Processor) hardware pipeline, integrating a more complete ISP processing engine, specifically optimized for RGB Bayer format images, and can perform preprocessing on two-dimensional color images, including but not limited to one or more of the following:

[0070] Black level correction: Subtract the sensor's base noise using a configurable register, as above.

[0071] Defective pixel correction: can repair sensor dead pixels, same as above.

[0072] Bayer De-mosaic: Hardware implementation using high-quality interpolation algorithms (such as adaptive Laplacian algorithm) to convert Bayer arrays into complete RGB images. Specifically, high-quality interpolation algorithms such as adaptive Laplacian are implemented in hardware. Based on the neighborhood information and texture features of the Bayer array pixels, high-precision interpolation reconstruction is performed on the missing color channels to restore the single-channel Bayer RAW data into a complete RGB color image, reducing false colors and jagged edges while ensuring sharp edges.

[0073] White balance correction: Based on the preset or automatically detected white point, the gain of the R, G, and B channels is adjusted independently to achieve true color reproduction. Specifically, by using the preset reference white point or automatically detecting the white area in the image, independent gain coefficients are applied to the R, G, and B color channels respectively to compensate for the color shift caused by different light source color temperatures, so that the white area presents a true neutral color and achieves accurate reproduction of scene colors.

[0074] Color correction and gamma correction: These methods use lookup tables to perform non-linear transformations on image data to adapt to the characteristics of display devices. Color correction and gamma correction use hardware lookup tables (LUTs) to perform efficient non-linear transformations on image data. Color correction is used to unify the color response of different sensors and improve color consistency. Gamma correction is used to correct the non-linear relationship of brightness so that the image brightness meets the visual characteristics of the human eye and the input requirements of subsequent display and processing devices.

[0075] After being processed by this complete ISP pipeline, a two-dimensional color image can be obtained as a high-quality RGB image.

[0076] Based on any of the above embodiments, a first configuration instruction can also be sent to the projection unit by the first processing unit in the main processor. The first configuration instruction includes target configuration parameters for the projection unit, used to configure the parameters of the projection unit, thereby realizing parameter configuration of the projection unit, including projecting specific coded patterns (phase-shifted fringes, Gray code, speckle, etc.), enabling switchable projection patterns and synchronously triggering the image sensor to take pictures. Optionally, the first processor can send the first configuration instruction to the projection unit via SPI (Serial Peripheral Interface) / I2C (Inter-Integrated Circuit) or other means to achieve low-speed, short-distance, board-level serial communication.

[0077] In addition, a second configuration command can be sent to the image sensor via the first processing unit in the main processor. This second configuration command includes target configuration parameters for the image sensor, used to configure the image sensor's parameters, including initializing the camera, setting resolution, exposure time, gain, and other parameters. Optionally, the first processor can send the second configuration command to the image sensor via SPI / I2C or other methods to achieve low-speed, short-distance, board-level serial communication.

[0078] Based on any of the above embodiments, the first processing unit in the main processor may also perform one or more of the following:

[0079] The first processing unit in the main processor detects whether the image sensor is abnormal. If so, it controls the coprocessor and the image sensor to restart. The restart can restore the image sensor and the coprocessor. For example, when configuring the parameters of the image sensor, it can determine whether the image sensor is disconnected. Or, during the image sensor acquisition process, it can determine whether the number of images output by the image sensor is consistent with the number of images expected to be acquired by the image sensor, thereby determining whether the image sensor is interrupted.

[0080] The first processing unit in the main processor verifies whether the image is complete. If not, the coprocessor controls the optical module to re-acquire the image. The image completeness can be verified by one or more verification methods such as frame structure verification, data feature verification, pixel value range verification, and stripe feature verification.

[0081] The temperature is acquired by the temperature sensor through the first processing unit in the main processor, and the fan is controlled to dissipate heat according to the temperature. The temperature sensor and the fan are deployed in the optical module and / or the main processor. The temperature acquired by the temperature sensor is compared with a preset temperature threshold. If the temperature exceeds the preset temperature threshold, the fan is controlled to dissipate heat. Optionally, the fan speed can also be controlled to achieve different heat dissipation intensities.

[0082] In addition, the coprocessor can be connected to flash memory, which stores the coprocessor's program. Each time the coprocessor restarts, the program in flash memory can be loaded into the coprocessor. Furthermore, program upgrades can be performed via flash memory, and program version rollback can be performed if the upgrade fails. Optionally, if the coprocessor includes a first coprocessor and a second coprocessor, the first and second coprocessors can be connected to the same flash memory, or they can each be connected to a separate flash memory.

[0083] This exemplary embodiment also provides a 3D camera, which includes an optical module and a processor. The processor includes a main processor and a coprocessor, and can be used to execute the three-dimensional contour reconstruction method in the above embodiments.

[0084] The coprocessor is used to control the optical module to acquire images of the object under test and to transmit the acquired images to the main processor.

[0085] The main processor is used to reconstruct the three-dimensional contour of the image and generate a three-dimensional point cloud of the object being measured.

[0086] Optionally, the optical module includes a projection unit and a monochrome image sensor, and the coprocessor includes a first coprocessor;

[0087] The first coprocessor is used to control the projection unit to project a preset structured light pattern onto the surface of the object being measured, and to control the black and white image sensor to acquire black and white stripe images of the surface of the object being measured.

[0088] The main processor is used to reconstruct the three-dimensional contour of the black and white striped image and generate a three-dimensional point cloud of the object being measured.

[0089] Optionally, the optical module includes a projection unit, a monochrome image sensor, and a color image sensor, and the coprocessor includes a first coprocessor and a second coprocessor;

[0090] The first coprocessor is used to control the projection unit to project a preset structured light pattern onto the surface of the object being measured, and to control the black and white image sensor to acquire black and white stripe images of the surface of the object being measured.

[0091] The second coprocessor is used to control the color image sensor to acquire two-dimensional color images of the object being measured;

[0092] The main processor is used to reconstruct the three-dimensional contour of the black and white striped image to generate a black and white three-dimensional point cloud of the object under test; and to map the color information of the two-dimensional color image to the black and white three-dimensional point cloud to generate a color three-dimensional point cloud of the object under test.

[0093] The 3D camera provided in this embodiment can be used to perform the three-dimensional contour reconstruction method in the above embodiments. Its principle and technical effects will not be described in detail in this embodiment.

[0094] An exemplary embodiment of this disclosure also provides a three-dimensional contour reconstruction system, which includes a processor and an optical module, wherein the processor is used to perform the three-dimensional contour reconstruction method as described in the above embodiments; or, the three-dimensional contour reconstruction system includes the 3D camera described above.

[0095] In this embodiment of the disclosure, in addition to providing a three-dimensional contour reconstruction method, a three-dimensional contour reconstruction apparatus is also provided for performing the above-described three-dimensional contour reconstruction method; the three-dimensional contour reconstruction apparatus includes: an acquisition module and a calculation module.

[0096] The acquisition module is used to control the optical module through the coprocessor to acquire images of the object under test and transmit the acquired images to the main processor.

[0097] The computing module is used to reconstruct the three-dimensional contour of the image through the main processor and generate a three-dimensional point cloud of the object under test.

[0098] Optionally, the optical module includes a projection unit and a monochrome image sensor, and the coprocessor includes a first coprocessor; when the acquisition module controls the optical module to acquire images of the object under test through the coprocessor, it is used for:

[0099] The first coprocessor controls the projection unit to project a preset structured light pattern onto the surface of the object being measured, and controls the black and white image sensor to acquire black and white stripe images of the surface of the object being measured.

[0100] Correspondingly, when the computing module performs 3D contour reconstruction on the image through the main processor to generate a 3D point cloud of the measured object, it is used for:

[0101] The main processor performs 3D contour reconstruction on the black and white striped image to generate a 3D point cloud of the object under test.

[0102] Optionally, the optical module includes a projection unit, a monochrome image sensor, and a color image sensor, and the coprocessor includes a first coprocessor and a second coprocessor; when the acquisition module controls the optical module to acquire images of the object under test through the coprocessor, it is used for:

[0103] The first coprocessor controls the projection unit to project a preset structured light pattern onto the surface of the object being measured, and controls the black and white image sensor to acquire black and white stripe images of the surface of the object being measured.

[0104] The second coprocessor controls the color image sensor to acquire two-dimensional color images of the object under test;

[0105] Correspondingly, when the computing module performs 3D contour reconstruction on the image through the main processor to generate a 3D point cloud of the measured object, it is used for:

[0106] The main processor performs three-dimensional contour reconstruction on the black and white striped image to generate a black and white three-dimensional point cloud of the object under test.

[0107] The main processor maps the color information of the two-dimensional color image onto the black and white three-dimensional point cloud to generate the color three-dimensional point cloud of the object being measured.

[0108] Optionally, when transmitting the acquired images to the main processor, the acquisition module is used for:

[0109] The acquired images are cached in the coprocessor's storage unit by the coprocessor;

[0110] The image is retrieved from the coprocessor's storage unit by the first processing unit in the main processor and stored in shared memory;

[0111] Accordingly, when the computing module performs 3D contour reconstruction of the image via the main processor, it is used for:

[0112] The image is retrieved from shared memory and a 3D contour is reconstructed using the second processing unit of the main processor.

[0113] Optionally, when the acquisition module acquires an image from the storage unit of the coprocessor via the first processing unit in the main processor, it is used to:

[0114] The first processing unit retrieves images from the coprocessor's storage unit using a direct memory access mechanism.

[0115] Optionally, after the acquisition module controls the optical module to acquire images of the object under test via the coprocessor, it is also used for:

[0116] Image preprocessing is performed using a coprocessor;

[0117] Correspondingly, when the acquisition module transmits the acquired image to the main processor, it is used for:

[0118] The preprocessed image is transmitted to the main processor via the coprocessor.

[0119] Optionally, the acquisition module is also used for:

[0120] A first configuration instruction is sent to the projection unit via a first processing unit in the main processor. The first configuration instruction includes target configuration parameters for the projection unit and is used to configure the parameters of the projection unit; and / or

[0121] The first processing unit in the main processor sends a second configuration instruction to the image sensor. The second configuration instruction includes the target configuration parameters of the image sensor and is used to configure the parameters of the image sensor.

[0122] Optionally, the acquisition module is also used for:

[0123] The first processing unit in the main processor detects whether the image sensor is malfunctioning; if so, it controls the coprocessor and image sensor to restart; and / or

[0124] The first processing unit in the main processor verifies whether the image is complete; if not, the coprocessor controls the optical module to re-acquire the image; and / or

[0125] The temperature is acquired by the temperature sensor through the first processing unit in the main processor, and the fan is controlled to dissipate heat according to the temperature. The temperature sensor and the fan are deployed in the optical module and / or the main processor.

[0126] The apparatus provided in this embodiment can be used to execute the technical solution of the above-described three-dimensional contour reconstruction method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0127] Furthermore, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The sequence numbers are merely used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0128] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an example embodiment of this disclosure. For example... Figure 4 As shown, the electronic device 40 includes a processor 41 and a memory 42 communicatively connected to the processor 41, the memory 42 storing computer-executed instructions.

[0129] The processor executes computer execution instructions stored in the memory to implement the three-dimensional contour reconstruction method provided in any of the above method embodiments. The specific functions and technical effects that can be achieved will not be elaborated here.

[0130] This disclosure also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the three-dimensional contour reconstruction method provided in any of the above method embodiments.

[0131] This disclosure also provides a computer program product, which includes a computer program stored in a readable storage medium. At least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the three-dimensional contour reconstruction method provided in any of the above method embodiments.

[0132] In the embodiments provided in this disclosure, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0133] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0134] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0135] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0137] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0138] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A three-dimensional contour reconstruction method, characterized in that, A processor for use in a 3D camera, the 3D camera including an optical module and the processor, the processor including a main processor and a coprocessor; the method includes: The coprocessor controls the optical module to acquire images of the object under test and transmits the acquired images to the main processor. The main processor performs three-dimensional contour reconstruction on the image to generate a three-dimensional point cloud of the object under test.

2. The method according to claim 1, characterized in that, The optical module includes a projection unit and a monochrome image sensor, and the coprocessor includes a first coprocessor; the step of controlling the optical module to acquire images of the object under test through the coprocessor includes: The first coprocessor controls the projection unit to project a preset structured light pattern onto the surface of the object under test, and controls the black and white image sensor to acquire black and white stripe images of the surface of the object under test. Accordingly, the step of reconstructing the three-dimensional contour of the image using the main processor to generate a three-dimensional point cloud of the measured object includes: The main processor performs three-dimensional contour reconstruction on the black and white striped image to generate a three-dimensional point cloud of the object under test.

3. The method according to claim 1, characterized in that, The optical module includes a projection unit, a monochrome image sensor, and a color image sensor; the coprocessor includes a first coprocessor and a second coprocessor; controlling the optical module to acquire images of the object under test via the coprocessor includes: The first coprocessor controls the projection unit to project a preset structured light pattern onto the surface of the object under test, and controls the black and white image sensor to acquire black and white stripe images of the surface of the object under test. The second coprocessor controls the color image sensor to acquire a two-dimensional color image of the object under test; Accordingly, the step of reconstructing the three-dimensional contour of the image using the main processor to generate a three-dimensional point cloud of the measured object includes: The main processor performs three-dimensional contour reconstruction on the black and white striped image to generate a black and white three-dimensional point cloud of the object under test. The main processor maps the color information of the two-dimensional color image onto the black-and-white three-dimensional point cloud to generate the color three-dimensional point cloud of the object under test.

4. The method according to any one of claims 1-3, characterized in that, The step of transmitting the acquired image to the main processor includes: The acquired images are cached in the coprocessor's storage unit by the coprocessor; The image is obtained from the storage unit of the coprocessor by the first processing unit in the main processor and stored in the shared memory; Accordingly, the step of reconstructing the three-dimensional contour of the image using the main processor includes: The image is retrieved from the shared memory and its three-dimensional contour is reconstructed by the second processing unit of the main processor.

5. The method according to claim 4, characterized in that, The step of retrieving the image from the storage unit of the coprocessor through the first processing unit in the main processor includes: The first processing unit retrieves the image from the coprocessor's storage unit using a direct memory access mechanism.

6. The method according to any one of claims 1-3, characterized in that, After the coprocessor controls the optical module to acquire images of the object under test, the method further includes: The image is preprocessed using the coprocessor; Accordingly, transmitting the acquired image to the main processor includes: The preprocessed image is transmitted to the main processor via the coprocessor.

7. The method according to claim 2 or 3, characterized in that, The method further includes: The first processing unit in the main processor sends a first configuration instruction to the projection unit, the first configuration instruction including target configuration parameters of the projection unit, for configuring the parameters of the projection unit; and / or The first processing unit in the main processor sends a second configuration instruction to the image sensor. The second configuration instruction includes target configuration parameters of the image sensor and is used to configure the parameters of the image sensor.

8. The method according to any one of claims 1-3, characterized in that, The method further includes: The first processing unit in the main processor detects whether the image sensor is malfunctioning; if so, it controls the coprocessor and the image sensor to restart; and / or The first processing unit in the main processor verifies whether the image is complete; if not, the coprocessor controls the optical module to re-acquire the image; and / or The temperature is acquired by the temperature sensor through the first processing unit in the main processor, and the fan is controlled to dissipate heat according to the temperature, wherein the temperature sensor and the fan are deployed in the optical module and / or the main processor.

9. A 3D camera, characterized in that, The 3D camera includes an optical module and a processor, the processor including a main processor and a coprocessor; The coprocessor is used to control the optical module to acquire images of the object under test and to transmit the acquired images to the main processor. The main processor is used to reconstruct the three-dimensional contour of the image and generate a three-dimensional point cloud of the object under test.

10. The 3D camera according to claim 9, characterized in that, The optical module includes a projection unit and a monochrome image sensor, and the coprocessor includes a first coprocessor; The first coprocessor is used to control the projection unit to project a preset structured light pattern onto the surface of the object under test, and to control the black and white image sensor to acquire black and white stripe images of the surface of the object under test. The main processor is used to perform three-dimensional contour reconstruction on the black and white striped image to generate a three-dimensional point cloud of the object under test.

11. The 3D camera according to claim 9, characterized in that, The optical module includes a projection unit, a monochrome image sensor, and a color image sensor, and the coprocessor includes a first coprocessor and a second coprocessor. The first coprocessor is used to control the projection unit to project a preset structured light pattern onto the surface of the object under test, and to control the black and white image sensor to acquire black and white stripe images of the surface of the object under test. The second coprocessor is used to control the color image sensor to acquire a two-dimensional color image of the object under test; The main processor is used to perform three-dimensional contour reconstruction on the black and white striped image to generate a black and white three-dimensional point cloud of the object under test; and to map the color information of the two-dimensional color image onto the black and white three-dimensional point cloud to generate a color three-dimensional point cloud of the object under test.

12. A three-dimensional contour reconstruction system, characterized in that, The three-dimensional contour reconstruction system includes a processor and an optical module, wherein the processor is configured to perform the method as described in any one of claims 1-8; or, the three-dimensional contour reconstruction system includes a 3D camera, wherein the 3D camera is the 3D camera as described in any one of claims 9-11.

13. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in any one of claims 1-8.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method described in any one of claims 1-8.

Images