Multi-stage pipeline parallel computing architecture and method based on 3D video sequence data

Abstract

The present application relates to the technical field of image data processing and three-dimensional reconstruction, and particularly relates to a multi-stage pipeline parallel computing architecture and method based on 3D video sequence data. In the process of continuously collecting images of a target to be measured, multiple image data at the same collection time are organized as processing data units and used as basic processing objects for three-dimensional reconstruction calculation. A multi-stage pipeline processing flow including multiple processing stages connected in sequence is constructed, so that the continuously formed processing data units can be executed in parallel on different processing stages, wherein the output of a previous processing stage for the same processing data unit is used as the input of a subsequent processing stage for the processing data unit. In the case of continuous image collection by a 3D camera, continuous real-time output of three-dimensional reconstruction results is achieved. The present application reconstructs the continuous three-dimensional reconstruction calculation flow into a multi-stage pipeline parallel processing, reduces the synchronization overhead, improves the hardware computing power utilization rate, and realizes continuous real-time output of the reconstruction results.

Description

Technical Field

[0001] This invention relates to the field of image data processing and 3D reconstruction technology, and in particular to a multi-stage pipelined parallel computing architecture and method based on 3D video sequence data. Background Technology

[0002] With the development of 3D vision technology, 3D cameras, which can acquire spatial structural information of objects, are widely used in fields such as medical aesthetics, expression recognition, psychological diagnosis and treatment, facial recognition, 3D printing and industrial inspection.

[0003] A typical 3D camera usually completes three core steps in sequence: image acquisition, 3D reconstruction, and data output. It acquires multiple image data through a texture camera, a depth camera, and a structured light projection device under precise timing control; it performs preprocessing, parallax calculation, depth estimation, and other operations on the acquired data to obtain spatial depth information; and it outputs the reconstruction results in a display or storage manner according to application requirements.

[0004] As applications evolve from single point cloud acquisition to continuous 3D video sequence acquisition, the demands for real-time performance have significantly increased. For example, in scenarios such as facial micro-expression recognition, continuous, stable, and high-precision output of 3D video data is required, leading to the evolution of 3D cameras into 3D cameras with continuous acquisition and processing capabilities. In these tasks, the constantly accumulating video frames on the timeline transforms 3D reconstruction from single-frame computation to computationally intensive continuous processing, placing higher demands on system architecture.

[0005] On existing CPU, GPU, or NPU platforms, real-time 3D reconstruction typically relies on general-purpose multithreaded parallel frameworks for acceleration. However, due to the strict data dependencies between the various processing steps in 3D reconstruction, general-purpose parallel frameworks often suffer from high synchronization overhead, uneven resource allocation, and difficulty in balancing accuracy and real-time performance when faced with high-frequency synchronization, large differences in heterogeneous computing loads, and the need for fine-grained multi-step processing, resulting in limited overall processing efficiency.

[0006] Therefore, considering the processing characteristics of 3D cameras in continuous 3D reconstruction scenarios, there is an urgent need for an efficient parallel processing architecture that can reduce synchronization overhead, adapt to task differences, and make full use of hardware computing power to achieve continuous real-time output of 3D reconstruction results. Summary of the Invention

[0007] This invention aims to at least solve the technical problems existing in the prior art. To this end, this invention provides a multi-stage pipelined parallel computing architecture and method based on 3D video sequence data. This architecture can process and organize multi-channel image data continuously acquired by 3D cameras into data units, and complete 3D reconstruction calculations within a multi-stage pipelined processing module. This yields 3D reconstruction results corresponding to each processing data unit and forms a continuously output data stream, thereby achieving the technical effect of continuous real-time output of 3D reconstruction results. In the actual operation of continuous 3D reconstruction, the processing flow can be divided into multiple sequentially connected processing stages according to the processing characteristics of different stages. Data connection and decoupling are achieved between adjacent processing stages through buffer units. A parallel scheduling module enables multiple processing stages to execute in parallel on different processing data units. Simultaneously, the output of the previous processing stage to the same processing data unit serves as the input of the next processing stage to that same processing data unit. Within each stage, sequential execution and necessary parallel output are achieved according to data dependencies, thereby reducing direct waiting and synchronization overhead between stages, improving task adaptability and hardware resource utilization efficiency, and ultimately achieving the technical effect of improving the continuous real-time output capability of real-time 3D reconstruction results.

[0008] In a first aspect, embodiments of the present invention provide a multi-stage pipelined parallel computing architecture based on 3D video sequence data, the architecture including: The image acquisition module is used to acquire multiple image data for 3D reconstruction during the continuous image acquisition of the target under test, and to organize the multiple image data at the same acquisition time into a processing data unit. The multi-stage pipeline processing module is used to perform 3D reconstruction calculations on continuously acquired processing data units to output 3D reconstruction results corresponding to each processing data unit. The parallel scheduling module is used to configure the multi-stage pipeline processing module to include multiple sequentially connected processing stages, and to enable multiple processing stages to be executed in parallel on different processing data units. The output of the previous processing stage to the processing data unit serves as the input of the next processing stage to the same processing data unit, thereby achieving continuous real-time output of 3D reconstruction results during the continuous image acquisition process of the 3D camera.

[0009] Optionally, the multi-stage pipelined processing module includes sequentially connected image acquisition stage, image preprocessing stage, disparity calculation stage, disparity thinning stage, and depth calculation and output stage, used to perform hierarchical pipelined processing on the processing data units formed by continuous acquisition. Adjacent pipelined processing stages are connected by a buffer unit to store the processing results output by the previous pipelined processing stage and provide the corresponding processing data units to the next pipelined processing stage.

[0010] Optionally, the image acquisition stage is used to acquire multiple image data for 3D reconstruction under the synchronous control of structured light projection and imaging, combine the multiple image data at the same acquisition time into a single processing data unit, and write the processing data unit into the corresponding buffer unit for subsequent pipeline processing stages to read.

[0011] Optionally, the image preprocessing stage is used to read the processing data unit from the cache unit corresponding to the previous pipeline processing stage, and to perform data verification, geometric correction and target region extraction processing on the image data in the processing data unit in sequence. The above processing operations are performed in a preset order, and the output of the previous processing operation is used as the input of the next processing operation to generate intermediate data for subsequent disparity calculation, and the intermediate data is written into the corresponding cache unit.

[0012] Optionally, the disparity calculation stage is used to read the intermediate data from the corresponding cache unit and sequentially perform phase acquisition, phase unrolling and multi-view matching processing, wherein there is a data dependency between each processing operation and they are executed in a preset order to obtain disparity information reflecting the correspondence of pixel space, and write the disparity information into the corresponding cache unit.

[0013] Optionally, the disparity refinement stage is used to read the disparity information from the corresponding cache unit and sequentially perform noise suppression, missing data filling and smoothing processing, wherein each processing operation is executed in a preset order to obtain spatially continuous fine disparity data, and the fine disparity data is written into the corresponding cache unit.

[0014] Optionally, the depth calculation and output stage is used to read the fine parallax data from the corresponding cache unit to complete the spatial depth calculation, and after the depth calculation is completed, to perform texture data output, RGBD depth data output, RGB-3D point cloud data output and RGBD data compression operations in parallel, wherein the data is stored after the RGBD data compression operation is completed.

[0015] Secondly, embodiments of the present invention provide a multi-stage pipelined parallel computing method based on 3D video sequence data, the method comprising: During the continuous image acquisition of the target under test, multiple image data for three-dimensional reconstruction are acquired, and the multiple image data at the same acquisition time are organized into processing data units. The continuously acquired processing data units are sequentially fed into a multi-stage pipeline processing flow to perform three-dimensional reconstruction calculations. The multi-stage pipeline processing flow includes multiple sequentially connected processing stages. This allows multiple processing stages to be executed in parallel on different processing data units, and the output of the previous processing stage for the processing data unit is used as the input of the next processing stage for the same processing data unit. As a result, during the continuous image acquisition process of the 3D camera, the 3D reconstruction results corresponding to each processing data unit are continuously output.

[0016] Optionally, the multi-stage pipeline processing flow includes sequentially executed steps of image acquisition, image preprocessing, disparity calculation, disparity thinning, and depth calculation and output processing. The output results of the previous processing step are stored in a cache unit between adjacent processing steps, and the corresponding processing data unit is provided to the next processing step.

[0017] Optionally, in the image preprocessing step, the disparity calculation step, and the disparity thinning step, multiple processing operations for the same processing data unit are executed sequentially in a preset order; in the depth calculation and output step, after the depth calculation is completed, texture data output, RGBD depth data output, RGB-3D point cloud data output, and RGBD data compression operations are executed in parallel, and data storage is performed after the RGBD data compression operation is completed.

[0018] The multi-stage pipelined parallel computing architecture and method based on 3D video sequence data according to the present invention has at least the following beneficial effects: This invention presents a multi-stage pipelined parallel computing architecture based on 3D video sequence data. First, it unifies the organization of multiple image data streams generated during continuous acquisition by a 3D camera, forming processing data units from multiple image data streams acquired at the same time. Based on this, the complete 3D reconstruction calculation process is divided into multiple sequentially connected processing stages, and a multi-stage pipelined processing module is constructed, enabling data connection and decoupling between processing stages through buffer units. Subsequently, a parallel scheduling module allows different processing stages to execute in parallel on different processing data units, while within each processing stage, sequential execution or parallel output is achieved based on data dependencies. Finally, during continuous image acquisition by the 3D camera, each processing data unit can stably flow in the multi-stage pipeline and continuously complete 3D reconstruction calculations, thereby achieving continuous real-time output of 3D reconstruction results. By introducing a multi-stage pipelined parallel structure and buffer decoupling mechanism, the direct synchronization and waiting overhead between processing stages is effectively reduced, the adaptability between different computing tasks is improved, and the computing power advantages of the hardware processing platform are fully utilized, achieving the technical effect of improving the real-time performance of continuous 3D reconstruction and overall processing throughput.

[0019] The multi-stage pipelined parallel computing method based on 3D video sequence data of the present invention first organizes multiple image data streams at the same acquisition moment into processing data units during the continuous image acquisition of the target object. Then, these processing data units are sequentially fed into a multi-stage pipelined processing flow to perform 3D reconstruction calculations, with each processing stage running in parallel on different processing data units. Next, based on the sequential relationship between processing stages, the output of the previous stage is used as the input of the next stage, completing the entire 3D reconstruction calculation flow. Finally, the obtained 3D reconstruction results are output or stored in the form of a continuous data stream. By employing a multi-stage pipelined parallel processing approach, the processing data units formed by continuous acquisition can be processed simultaneously at different processing stages, thereby reducing waiting and blocking in the overall calculation process, improving processing efficiency, and ultimately achieving continuous real-time output of 3D reconstruction results, meeting the application requirements for high real-time performance and continuity.

[0020] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. Attached Figure Description

[0021] Figure 1 This is a structural diagram of the 3D camera system in Example 2; Figure 2 This is a schematic diagram of the structural form and assembly method of the 3D camera in Example 2; Figure 3 This is an overall workflow diagram of the 3D camera system in Example 2; Figure 4 This is a diagram showing the structure and execution timing of the multi-stage pipelined parallel computing architecture in Example 2. Figure 5 This is a flowchart of a single-stage pipeline in the multi-stage pipelined parallel computing architecture of Example 2; Figure 6 This is an interactive display screenshot of the series of data outputs from the 3D camera in Example 2. Detailed Implementation

[0022] The present invention will be further described in detail below with reference to experimental examples and specific embodiments. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following embodiments. All technologies implemented based on the content of the present invention fall within the scope of protection of the present invention.

[0023] Unless otherwise specified, the use of terms such as "upper," "lower," "left," "right," "center," "inner," "outer," and "side" to indicate orientation or positional relationships in the description of specific embodiments of the present invention is based on the orientation or positional relationships shown in the accompanying drawings, or the orientation or positional relationship in which the product / equipment / device is typically placed during use. These terms are merely for the purpose of facilitating the description of the present invention or simplifying the description in specific embodiments, enabling those skilled in the art to quickly understand the solution, and do not indicate or imply that a particular device / component / element must have a specific orientation, or be constructed and operated in a specific positional relationship. Therefore, they should not be construed as limitations on the present invention.

[0024] In the description of the embodiments of this invention, technical terms such as "first" and "second" only distinguish one entity or operation from another, and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary or secondary relationship of the indicated technical features. In the description of the embodiments of this invention, "multiple" means two or more, unless otherwise explicitly defined.

[0025] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0026] Example 1 During their research, the inventors discovered that when using traditional 3D cameras or 3D reconstruction processing methods based on general multi-threaded parallel frameworks, achieving the technical goal of real-time output of continuous 3D reconstruction results typically requires tedious and inconvenient processing such as frequent thread partitioning, synchronization control, and resource coordination for each calculation step. This results in a limited number of 3D reconstruction results being output, sacrificing processing efficiency or reconstruction continuity. In solving practical engineering problems, to achieve stable and high-speed output of 3D reconstruction results under continuous acquisition conditions, existing technologies suffer from high synchronization overhead, insufficient adaptability to different processing tasks, and low hardware computing power utilization, making it difficult to meet the application requirements of real-time 3D reconstruction.

[0027] Therefore, after in-depth research on the above problems, the inventors proposed a multi-stage pipelined parallel computing architecture and method based on 3D video sequence data. In addressing the technical problems of continuous accumulation of computational load and strong data dependence of processing steps in continuous 3D reconstruction scenarios using 3D cameras, the 3D reconstruction calculation process is divided into multiple sequentially connected processing stages, and different processing stages are executed in parallel on different processing data units. At the same time, a cache unit is used to decouple data between adjacent processing stages, thereby achieving stable parallel processing of continuous processing data units in a multi-stage pipeline. This achieves the technical effects of reducing synchronization overhead, improving task adaptability, and realizing continuous real-time output of 3D reconstruction results.

[0028] The multi-stage pipelined parallel computing architecture based on 3D video sequence data provided in this embodiment of the invention may include: The image acquisition module is used to acquire multiple image data for 3D reconstruction during the continuous image acquisition of the target under test, and to organize the multiple image data at the same acquisition time into a processing data unit. The multi-stage pipeline processing module is used to perform 3D reconstruction calculations on continuously acquired processing data units to output 3D reconstruction results corresponding to each processing data unit. The parallel scheduling module is used to configure the multi-stage pipeline processing module to include multiple sequentially connected processing stages, and to enable multiple processing stages to be executed in parallel on different processing data units. The output of the previous processing stage to the processing data unit serves as the input of the next processing stage to the same processing data unit, thereby achieving continuous real-time output of 3D reconstruction results during the continuous image acquisition process of the 3D camera.

[0029] In some specific implementations, this computing architecture may further include: This embodiment provides a multi-stage pipelined parallel computing architecture and method based on 3D video sequence data, applied to a 3D camera, to achieve continuous real-time output of 3D reconstruction results during continuous image acquisition of the target object. Compared with traditional 3D cameras that can only output single 3D data frame by frame, this embodiment reconstructs the 3D reconstruction calculation process in a pipelined manner, enabling data units acquired at different time points to be processed in parallel at different processing stages, thereby significantly improving the overall processing throughput and real-time performance.

[0030] In this embodiment, when the 3D camera continuously acquires images of the target under test, it simultaneously obtains multiple image data streams for 3D reconstruction and organizes these multiple image data streams at the same acquisition moment into a single processing data unit. This processing data unit, as the basic processing object for 3D reconstruction calculations, is sequentially fed into a multi-stage pipeline processing module for processing.

[0031] The multi-stage pipelined processing module includes sequentially connected image acquisition, image preprocessing, disparity calculation, disparity refinement, and depth calculation and output stages. These stages together constitute a complete 3D reconstruction calculation flow. Adjacent pipelined processing stages are connected via buffer units. After the preceding pipelined processing stage completes its processing, it writes the corresponding processing result to the buffer unit. The subsequent pipelined processing stage reads the corresponding processing data from the buffer unit and continues processing, thereby achieving data decoupling between processing stages.

[0032] In the image acquisition stage, under the synchronous control of structured light projection and imaging, multiple image data for 3D reconstruction are acquired respectively, and the multiple image data at the same acquisition time are combined into a single processing data unit. Then, the processing data unit is written into the corresponding buffer unit for subsequent pipeline processing stages to read.

[0033] In the image preprocessing stage, processing data units are read from the cache unit, and data verification, geometric correction, and target region extraction are sequentially performed on the image data in the processing data units. There are clear data dependencies between the above processing operations. The output of the previous processing operation serves as the input of the next processing operation. The processing operations are executed sequentially according to a preset order to generate intermediate data for subsequent disparity calculation, and the intermediate data is written to the cache unit.

[0034] In the disparity calculation stage, the intermediate data is read from the cache unit, and phase calculation, phase unrolling, and multi-view matching are performed sequentially. The above processing operations are also performed in a preset order, with the output of the previous processing operation serving as the input of the next processing operation, thereby obtaining disparity information reflecting the spatial correspondence of pixels, and writing the disparity information into the cache unit.

[0035] In the parallax refinement stage, the parallax information is read from the cache unit, and noise suppression, missing data filling and smoothing are performed in sequence to obtain spatially continuous fine parallax data. Then, the fine parallax data is written to the cache unit.

[0036] The depth calculation and output stage reads the fine parallax data from the cache unit to complete the spatial depth calculation. After the depth calculation is completed, this stage performs texture data output, RGBD depth data output, RGB-3D point cloud data output, and RGBD data compression operations in parallel. Data storage is performed after the RGBD data compression operation is completed, thus simultaneously meeting the needs of real-time display and data storage.

[0037] Throughout the computation process, the parallel scheduling module coordinates the operation of the multi-stage pipeline processing modules, enabling the multiple pipeline processing stages to execute in parallel on different data processing units. Specifically, at any given time, different pipeline processing stages process data processing units from different acquisition times, and the output of the previous processing stage for a data processing unit serves as the input of the next processing stage for the same data processing unit. This creates a stable pipeline parallel processing state during continuous image acquisition by the 3D camera, achieving continuous real-time output of 3D reconstruction results.

[0038] Through the above technical solution, this embodiment can reconstruct the multi-step calculation process that was originally executed serially on a single data processing unit into a multi-level pipeline parallel calculation process without changing the calculation logic of the 3D reconstruction algorithm itself. This effectively improves the system's ability to process continuous 3D video sequence data and is suitable for 3D vision application scenarios with high requirements for real-time performance and continuity.

[0039] The multi-stage pipelined parallel computing architecture and method based on 3D video sequence data provided in this invention can be applied to many technical fields, such as 3D visual perception, intelligent human-computer interaction, computer vision and image processing, including but not limited to 3D face recognition and micro-expression analysis, medical imaging and medical 3D modeling, industrial inspection and dimensional measurement, 3D reconstruction and digital twin modeling, augmented reality and virtual reality, and other application scenarios. In the above implementation, when continuously acquiring and reconstructing 3D data of the target under test, the 3D reconstruction calculation process can be hierarchically divided according to the computational characteristics and data dependencies of different processing stages. Parallel execution of different data processing units at each processing stage can be achieved through multi-stage pipelined parallel processing and decoupling of caching units. This effectively improves processing throughput and real-time performance while ensuring reconstruction accuracy, achieving continuous and stable output of 3D reconstruction results and fully utilizing hardware computing resources.

[0040] Example 2 This embodiment provides a detailed description of the specific implementation of the multi-stage pipelined parallel computing architecture based on 3D video sequence data proposed in this invention.

[0041] The system in this embodiment uses a structured light projection 3D camera as a data acquisition device and runs a five-stage pipeline parallel processing architecture on a hardware processing platform to realize the real-time calculation process from 3D data acquisition to depth result output.

[0042] like Figure 1 As shown, Figure 1This is a structural diagram of a 3D camera system used to capture 3D video sequence data. The 3D camera system mainly includes: a projection device (100) for cyclically projecting a stripe pattern (300), a binocular depth camera (left eye camera 201, right eye camera 202) for acquiring structured light modulated images, an RGB texture camera (400) for acquiring texture images, a control unit (500) for timing synchronization control, and a hardware processing platform (600) for performing parallel computing tasks. The projection device (100) projects a sinusoidal phase-shifted stripe pattern (300) onto the target in a cyclic manner; the depth cameras (201, 202) synchronously acquire the left and right eye images under the corresponding stripe modulation; the RGB texture camera (400) is used to acquire texture images corresponding to the depth images; the control unit (500), such as an FPGA, can uniformly manage projection, exposure, triggering, and acquisition; the hardware processing platform (600) may include a CPU, GPU, NPU, etc., for performing multi-stage pipelined parallel computing.

[0043] Preferably, the depth camera must have short exposure and high frame rate capabilities, and the projection device must support high-speed projection. This embodiment uses a micro-projection matrix (MPM) as the projection system, which combines the advantages of low cost and high-speed projection. It can continuously project three frames of sinusoidal phase-shifted fringes in a cyclic projection mode and achieve camera-level synchronization. Figure 2 As shown, Figure 2 This is a schematic diagram illustrating the structure and assembly method of a 3D camera. Figure 3 As shown, the overall workflow of the 3D camera system provided in this embodiment consists of a hardware workflow (700) and a software workflow (800). In the hardware workflow, after the system enters the 3D acquisition start state, the control unit FPGA sends start and stop trigger signals respectively, and then continuously outputs timed cyclic trigger pulses to synchronously control the exposure action of the binocular depth camera, so that the left and right cameras complete the exposure in a predetermined sequence and write the image to the buffer. This hardware workflow realizes time synchronization between the projection device and the depth camera, thereby ensuring the effective acquisition of structured light modulated images.

[0044] The software flow (800) corresponds to the image data reading, processing, and reconstruction process. When the 3D camera is opened in the software, the system first starts the data cache processing thread and simultaneously starts the camera thread, causing the right eye camera to enter external trigger mode and its working thread to wait until new data is available in the right eye image cache; the RGB texture camera enters preview mode to obtain texture information. After the raw data acquired by the depth camera and RGB texture camera is written to the cache, the data processing thread detects that data already exists in the cache unit, reads a set of raw images from the cache (including one frame of RGB texture image and a pair of left and right eye stereo images), and sends it to the 3D reconstruction module to complete the reconstruction calculation. After reconstruction, the system outputs the corresponding 3D data and continues to wait for the next set of data input after this round of processing, thus forming a complete 3D video stream acquisition and reconstruction process. In such a typical work cycle, the system acquires three frames of structured light modulation images and one frame of RGB texture image to form (3+1 frames) of two-dimensional raw image data, which is then input to the hardware processing platform for real-time processing via the USB 3.0 interface.

[0045] To achieve high-speed processing of continuous 3D video sequences, this embodiment employs the following method: Figure 4 The five-stage pipelined parallel computing architecture shown includes five pipeline stages, from Stage 1 to Stage 5. Each stage is connected sequentially in a front-to-back input manner, and different data processing units are executed in parallel under a parallel scheduling mechanism, enabling the system to maintain a stable 40 GHz even under continuous input conditions. 100 FPS throughput.

[0046] During the Stage 1 image acquisition phase, the RGB texture camera and the stereo depth camera acquire texture images, left-eye structured light images, and right-eye structured light images respectively under unified timing control. Each camera's acquisition task is executed in an independent thread to avoid inter-thread waiting and improve acquisition efficiency. Once multiple images are acquired at the same time, the system combines them into a processing data unit for a single 3D reconstruction.

[0047] In the Stage 2 image preprocessing stage, the system sequentially performs data integrity verification, epipolar correction, and target region extraction on the processed data units. Data integrity verification is used to remove abnormal images; epipolar correction is used to ensure that the left and right views meet the preset alignment relationship; and target region extraction is used to generate intermediate data for disparity calculation. Due to the strict data dependencies between the three operations, they must be completed in a fixed order.

[0048] In the Stage 3 disparity calculation stage, the system sequentially performs truncated phase calculation, phase unwrapping, and phase stereo matching based on the preprocessed image to obtain initial disparity information. Phase acquisition is used to extract the phase corresponding to the fringe modulation pattern; phase unwrapping is used to resolve the phase jump caused by the fringe period; and matching is used to calculate the disparity based on the correspondence between the unwrapped phase and the left and right views.

[0049] In the Stage 4 disparity refinement stage, the system sequentially performs filtering and noise reduction, missing value filling and smoothing on the initial disparity to generate more refined disparity data with stronger spatial continuity and better structural preservation, providing reliable input for the final depth estimation.

[0050] In the Stage 5 depth calculation and output phase, the system first calculates pixel spatial depth based on fine parallax and the corresponding texture image to obtain the data results for output. After completing the depth calculation, the system performs operations such as outputting RGB texture data, outputting RGBD depth data, outputting RGB-3D point cloud data, and RGBD data compression. These four output tasks are run in parallel to improve overall processing efficiency. After compression, storage operations are performed to achieve real-time output and storage of continuous 3D video data.

[0051] like Figure 5 As shown, Figure 5 This is a flowchart of a single-stage pipeline in a multi-stage pipelined parallel computing architecture. Specifically, after each pipeline completes its work, the computation result is saved to the corresponding buffer unit. The next pipeline reads data from the buffer unit of the previous pipeline, performs the corresponding algorithm operation, and then outputs the computation result to a new buffer unit.

[0052] In the five-stage pipeline, the computation time varies between stages. Object detection and region extraction in Stage 2 are the most time-consuming, followed by data output operations in Stage 5. Therefore, this embodiment allocates the number of threads to different pipeline stages based on the computational load, enabling each pipeline stage to execute in parallel on different data processing units. This embodiment allocates two threads to Stage 1 and one thread to Stages 2 through 5 to achieve higher throughput and at least 40 FPS of RGBD depth data output.

[0053] Through the above method, the multi-stage pipelined parallel computing architecture provided in this embodiment can stably output continuous 3D video sequence data under real-time conditions, including a series of data such as continuous RGB texture data, RGBD depth data, and high-precision 3D point cloud data with RGB texture, and store them synchronously in real time. Figure 6 As shown, Figure 6 This is an interactive display screenshot of a series of data outputs from a 3D camera, with the data played in real time as a video stream.

[0054] It should be noted that in this embodiment... Figure 6 The RGB texture video sequences, depth maps, and 3D point cloud video sequences shown are only used to illustrate the technical solution and implementation effects of the present invention, and are not used to identify, analyze, or characterize the identity information of any specific natural person. The data shown has been anonymized or de-identified during generation, processing, and display, and in principle, cannot be mapped to, identified, or restored to a specific natural person, thus not constituting a processing activity of personal identity information. Furthermore, regarding data that may involve the image of a real natural person in this technology, it should be further explained that: the collection and use of the relevant data have obtained explicit authorization from the corresponding natural person, and are limited to the research, testing, and patent illustration purposes of this invention, and have not been used for any purpose beyond the legal, legitimate, and necessary scope, nor have they been provided to external parties or used for commercial identification applications. The above authorization does not affect the universality of the technical solution of this invention. This invention does not rely on specific natural person data for implementation, and its technical effects can also be achieved through non-specific natural persons or simulated data. During the research and verification process, this invention strictly follows the "minimum necessity principle" requirements of the "Personal Information Protection Law of the People's Republic of China," and does not involve the illegal collection, storage, or processing of personal information. Through the above control and explanation, the implementation and disclosure of this invention do not violate current national laws and regulations. The application of this invention is limited to the legal, legitimate, and necessary scope, falling within the category of fair use of technical content. Through the above-described processing and control, the implementation and use of this invention do not violate relevant national laws and regulations.

[0055] Example 3 The system structure and multi-stage pipelined parallel architecture have been described in the foregoing embodiments. To enable those skilled in the art to further understand the specific execution process of the present invention at the method level, this embodiment provides a multi-stage pipelined parallel computing method based on 3D video sequence data, applied to a 3D camera, for achieving continuous real-time output of 3D reconstruction results during continuous image acquisition of the target object. This method can be implemented based on the aforementioned multi-stage pipelined parallel computing architecture, and its computational flow is adapted to the architecture, but the 3D reconstruction process is described from the perspective of time sequence and processing steps.

[0056] In this embodiment, during the continuous image acquisition of the target object, multiple image data streams for 3D reconstruction are simultaneously acquired, and these multiple image data streams at the same acquisition time are organized into a single processing data unit. This processing data unit serves as the basic processing object for subsequent 3D reconstruction calculations, and is continuously generated and enters the calculation process according to the acquisition sequence.

[0057] Subsequently, the continuously acquired processing data units are sequentially fed into a multi-stage pipeline processing flow to perform 3D reconstruction calculations. The multi-stage pipeline processing flow includes sequentially connected steps such as image acquisition, image preprocessing, disparity calculation, disparity thinning, and depth calculation and output processing. Between adjacent processing steps, a buffer unit stores the output result of the previous processing step and provides the corresponding processing data unit to the next processing step, thereby achieving data connection and decoupling between processing steps.

[0058] In the specific processing, the image preprocessing step, disparity calculation step, and disparity refinement step involve multiple processing operations on the same data unit executed sequentially according to a preset order. The output of the previous processing operation serves as the input for the next, thus satisfying the strong data dependencies between the steps in the 3D reconstruction calculation. In the depth calculation and output step, spatial depth calculation is first performed based on fine disparity data. After the depth calculation is completed, texture data output, RGBD depth data output, RGB-3D point cloud data output, and RGBD data compression operations are performed in parallel. Data storage is then performed after the RGBD data compression operation is completed.

[0059] Simultaneously, throughout the computation process, the scheduling and control of the multi-stage pipeline processing flow enables different processing steps to be executed in parallel on different data units. That is, at any given moment, different processing steps process data units from different acquisition times, and the output of the previous processing step for the same data unit serves as the input for the next processing step for that same data unit. Through this method, during the continuous image acquisition process by the 3D camera, the parallel pipeline processing of multiple data units can be stably completed, thereby achieving continuous real-time output of 3D reconstruction results.

[0060] The multi-stage pipeline parallel computing method described in this embodiment effectively reduces the waiting and synchronization overhead between processing steps in continuous 3D reconstruction scenarios without changing the processing logic of the 3D reconstruction algorithm itself, thereby improving the overall processing efficiency. This enables the 3D camera to stably output high temporal density 3D reconstruction results under continuous acquisition conditions, making it suitable for 3D vision application scenarios with high requirements for real-time performance and continuity.

[0061] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0062] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0063] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multi-stage pipelined parallel computing architecture based on 3D video sequence data, characterized in that, include: The image acquisition module is used to acquire multiple image data for three-dimensional reconstruction during the continuous image acquisition of the target under test, and to organize the multiple image data at the same acquisition time into a processing data unit, wherein the processing data unit is a data object for three-dimensional reconstruction calculation. The multi-stage pipeline processing module is used to perform three-dimensional reconstruction calculations on continuously acquired processing data units to output three-dimensional reconstruction results corresponding to each processing data unit. The parallel scheduling module is used to configure the multi-stage pipeline processing module to include multiple sequentially connected processing stages, and to enable multiple processing stages to be executed in parallel on different processing data units. The output of the previous processing stage to the processing data unit serves as the input of the next processing stage to the same processing data unit, thereby achieving continuous real-time output of 3D reconstruction results during the continuous image acquisition process of the 3D camera.

2. The multi-stage pipelined parallel computing architecture based on 3D video sequence data according to claim 1, characterized in that, The multi-stage pipelined processing module includes sequentially connected image acquisition stage, image preprocessing stage, disparity calculation stage, disparity thinning stage, and depth calculation and output stage. It is used to perform hierarchical pipelined processing on the processing data units formed by continuous acquisition. Adjacent pipelined processing stages are connected by a buffer unit to store the processing results output by the previous pipelined processing stage and provide the corresponding processing data units to the next pipelined processing stage.

3. The multi-stage pipelined parallel computing architecture based on 3D video sequence data according to claim 2, characterized in that, The image acquisition stage is used to acquire multiple image data for 3D reconstruction under the synchronous control of structured light projection and imaging, and combine the multiple image data at the same acquisition time into a single processing data unit, and write the processing data unit into the corresponding buffer unit for subsequent pipeline processing stages to read.

4. The multi-stage pipelined parallel computing architecture based on 3D video sequence data according to claim 2, characterized in that, The image preprocessing stage is used to read the processing data unit from the cache unit corresponding to the previous pipeline processing stage, and to perform data verification, geometric correction and target region extraction processing on the image data in the processing data unit in sequence. The above processing operations are executed in a preset order, and the output of the previous processing operation is used as the input of the next processing operation to generate intermediate data for subsequent disparity calculation, and the intermediate data is written into the corresponding cache unit.

5. The multi-stage pipelined parallel computing architecture based on 3D video sequence data according to claim 2, characterized in that, The disparity calculation stage is used to read the intermediate data from the corresponding cache unit and sequentially perform phase acquisition, phase unrolling and multi-view matching processing. Each processing operation has a data dependency relationship and is executed in a preset order to obtain disparity information reflecting the correspondence of pixel space, and write the disparity information into the corresponding cache unit.

6. The multi-stage pipelined parallel computing architecture based on 3D video sequence data according to claim 2, characterized in that, The parallax refinement stage is used to read the parallax information from the corresponding cache unit and perform noise suppression, missing data filling and smoothing in sequence. Each processing operation is performed in a preset order to obtain spatially continuous fine parallax data and write the fine parallax data into the corresponding cache unit.

7. The multi-stage pipelined parallel computing architecture based on 3D video sequence data according to claim 6, characterized in that, The depth calculation and output stage is used to read the fine parallax data from the corresponding cache unit to complete the spatial depth calculation. After the depth calculation is completed, texture data output, RGBD depth data output, RGB-3D point cloud data output and RGBD data compression operations are performed in parallel. The data is stored after the RGBD data compression operation is completed.

8. A multi-stage pipelined parallel computing method based on 3D video sequence data, characterized in that, include: During the continuous image acquisition of the target under test, multiple image data for three-dimensional reconstruction are acquired, and the multiple image data at the same acquisition time are organized into processing data units. The continuously acquired processing data units are sequentially fed into a multi-stage pipeline processing flow to perform three-dimensional reconstruction calculations. The multi-stage pipeline processing flow includes multiple sequentially connected processing stages. This allows multiple processing stages to be executed in parallel on different processing data units, and the output of the previous processing stage for the processing data unit is used as the input of the next processing stage for the same processing data unit. As a result, during the continuous image acquisition process of the 3D camera, the 3D reconstruction results corresponding to each processing data unit are continuously output.

9. The multi-stage pipelined parallel computing method based on 3D video sequence data according to claim 8, characterized in that, The multi-stage pipeline processing flow includes sequentially executed steps such as image acquisition, image preprocessing, disparity calculation, disparity thinning, and depth calculation and output processing. The output results of the previous processing step are stored in a cache unit between adjacent processing steps, and the corresponding processing data unit is provided to the next processing step.

10. The multi-stage pipelined parallel computing method based on 3D video sequence data according to claim 8, characterized in that, In the image preprocessing step, disparity calculation step, and disparity thinning step, multiple processing operations for the same processing data unit are executed sequentially in a preset order; in the depth calculation and output step, after the depth calculation is completed, texture data output, RGBD depth data output, RGB-3D point cloud data output, and RGBD data compression operations are executed in parallel, and data storage is performed after the RGBD data compression operation is completed.

Images