3D image generation method and system based on multi-camera shooting

Abstract

The present application relates to the technical field of 3D image generation, and discloses a 3D image generation method and system based on multi-camera shooting, the method comprising: synchronously triggering and acquiring images acquired by multiple cameras to obtain multi-perspective original images; preprocessing the multi-perspective original images to obtain preprocessed multi-perspective images; performing feature extraction and matching on the preprocessed multi-perspective images to obtain feature matching results; aligning and calculating depth information on the preprocessed multi-perspective images based on the feature matching results to obtain aligned multi-perspective images and depth maps; reconstructing a 3D model based on the aligned multi-perspective images and the depth maps to obtain a scene 3D model; performing stereoscopic parallax correction and rendering processing on the scene 3D model to generate a stereoscopic 3D image, thereby realizing high-precision, high-detail 3D model reconstruction, and generating a high-quality, high-immersion stereoscopic 3D image.

Description

Technical Field

[0001] The present application relates to the technical field of 3D image generation, and in particular to a 3D image generation method and system based on multi-camera shooting. Background Art

[0002] With the rapid development of virtual reality, augmented reality and mixed reality technologies, the demand for high-quality and highly realistic 3D content is growing. Traditional 3D content production methods often rely on complex modeling and rendering processes, which are time-consuming and labor-intensive and difficult to achieve accurate reconstruction of real scenes. 3D image generation methods based on multi-camera shooting have emerged, aiming to capture real scenes simultaneously through multiple cameras and reconstruct high-quality 3D models through computer vision and image processing technology. However, this method still faces many challenges in practical applications.

[0003] In the prior art, the synchronization problem of multi-camera systems has always been a technical difficulty. The time difference between different cameras will cause time inconsistency in the captured images, affecting the subsequent 3D reconstruction effect. Secondly, due to the complexity of the shooting environment, such as lighting changes, occlusion and other factors, the acquired multi-view images often have problems such as noise, chromatic aberration and distortion, which will have an adverse effect on the accuracy of 3D reconstruction. In addition, in key links such as feature extraction and matching, image alignment, and depth information calculation, how to improve computing efficiency while ensuring accuracy is also an important issue. Especially when dealing with dynamic scenes, how to accurately identify and process moving objects and ensure the temporal and spatial consistency of the reconstruction results is still a major challenge facing this field. How to generate high-quality stereoscopic 3D images so that they can present good stereoscopic and immersive effects on various display devices is also a key problem that needs to be solved. Summary of the invention

[0004] The present application provides a 3D image generation method and system based on multi-camera shooting, thereby achieving high-precision, high-detail 3D model reconstruction and generating high-quality, highly immersive stereoscopic 3D images.

[0005] In a first aspect, the present application provides a method for generating 3D images based on multi-camera shooting, and the method for generating 3D images based on multi-camera shooting includes:

[0006] The images collected by multiple cameras are synchronously triggered to obtain multi-view original images;

[0007] Preprocessing the multi-view original image to obtain a preprocessed multi-view image;

[0008] Performing feature extraction and matching on the preprocessed multi-view images to obtain feature matching results;

[0009] Based on the feature matching result, the preprocessed multi-view images are aligned and depth information is calculated to obtain aligned multi-view images and depth maps;

[0010] Reconstructing a 3D model based on the aligned multi-view images and the depth map to obtain a 3D model of the scene;

[0011] The scene 3D model is subjected to stereoscopic parallax correction and rendering processing to generate a stereoscopic 3D image.

[0012] A second aspect of the present application provides a 3D image generation device based on multi-camera shooting, and the 3D image generation device based on multi-camera shooting includes:

[0013] An acquisition module is used to synchronously trigger the acquisition of images acquired by multiple cameras to obtain multi-view original images;

[0014] A preprocessing module, used to preprocess the multi-view original image to obtain a preprocessed multi-view image;

[0015] A matching module, used for performing feature extraction and matching on the preprocessed multi-view images to obtain feature matching results;

[0016] A calculation module, configured to align the preprocessed multi-view images and calculate depth information based on the feature matching result to obtain an aligned multi-view image and a depth map;

[0017] A reconstruction module, used to reconstruct a 3D model based on the aligned multi-view images and the depth map to obtain a scene 3D model;

[0018] The generation module is used to perform stereoscopic parallax correction and rendering processing on the scene 3D model to generate a stereoscopic 3D image.

[0019] The third aspect of the present application provides an electronic device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the electronic device executes the above-mentioned 3D image generation method based on multi-camera shooting.

[0020] A fourth aspect of the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned 3D image generation method based on multi-camera shooting.

[0021] Compared with the prior art, the present application has the following beneficial effects: clock synchronization and microsecond trigger signal generation by the WiFi module and the central control unit, combined with the high-speed LED array to emit light pulses of a specific pattern, sub-microsecond time alignment correction of multiple cameras is realized, and the time consistency of multi-view images is greatly improved. The global color mapping model, adaptive exposure equalization function, polynomial model distortion correction and other technologies are adopted to effectively solve the noise, color difference, uneven exposure and distortion problems of multi-view images. By constructing an affine invariant feature set, global semantic feature extraction and feature fast indexing tree, efficient and accurate matching between multi-view images is achieved. The hierarchical sparse bundle adjustment model, multi-view stereo matching cost volume analysis and other technologies are used to achieve accurate alignment of multi-view images and generation of high-precision depth maps. Through spatiotemporal voxelization, spatiotemporal consistency analysis and fine motion field optimization, dynamic objects are effectively identified and processed, ensuring the spatiotemporal consistency of the reconstruction results. The multi-resolution voxel representation, signed distance field optimization and adaptive surface subdivision and other technologies are adopted to achieve high-precision and high-detail 3D model reconstruction. Based on the human eye stereoscopic vision perception model and adaptive stereo disparity adjustment, combined with dense light field sampling and rendering enhancement technology, high-quality, highly immersive stereoscopic 3D images are generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0023] The structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportion or adjustment of size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical contents disclosed by the present invention.

[0024] Figure 1 is a flow chart of a method for generating 3D images based on multi-camera shooting provided by an embodiment of the present invention;

[0025] Figure 2 It is a schematic block diagram of the structure of a 3D image generation device based on multi-camera shooting provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0028] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0029] It should be further understood that the term "and / or" used in the specification and appended claims of this application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations. Figure 1 In the embodiment of the present application, an embodiment of a 3D image generation method based on multi-camera shooting includes:

[0030] Step 100, synchronously triggering and collecting images collected by multiple cameras to obtain multi-view original images;

[0031] It is understandable that the execution subject of the present application can be a 3D image generation device based on multi-camera shooting, or a terminal or a server, which is not limited here. The embodiment of the present application is described by taking the server as the execution subject as an example.

[0032] Specifically, according to the needs of the scene and the coverage of the viewing angle, the cameras are arranged in an array to maximize the coverage of different viewing angles of the scene and ensure that sufficient information is captured from multiple angles. In the camera array, each camera is equipped with a WiFi module, so as to communicate with the central control unit through the WiFi network to form a camera array with a WiFi module. A central control unit is set up to establish a connection with the camera array with the WiFi module through the WiFi network. The function of the central control unit is to generate and distribute high-precision synchronization trigger signals to ensure that all cameras start shooting at the same time. In order to achieve microsecond-level time synchronization, the central control unit needs to generate a microsecond-level trigger signal. The microsecond-level synchronization trigger signal can ensure that the shooting time difference between cameras is less than 1 microsecond, avoiding motion blur or distortion in the image caused by the time difference. This trigger signal is transmitted to each camera through the WiFi network. The transmitted microsecond-level synchronization trigger signal can ensure that each camera starts to expose and shoot at the moment of receiving the signal, thereby obtaining a preliminarily synchronized multi-view image. In order to improve the accuracy of time synchronization and provide a reliable time reference for subsequent image processing, a high-speed LED array is set in the shooting scene. The high-speed LED array can provide a clear time mark for the scene by emitting light pulses. These light pulses are controlled by the central control unit based on microsecond synchronization trigger signals to ensure that the light pulses are synchronized with the camera's shooting. After the camera receives the synchronization trigger signal and starts shooting, the scene with time stamps is recorded to form a multi-view image with light pulse patterns. These images are transmitted to the cloud APP via the WiFi network for further processing. In the cloud APP, the light pulse marks in the image are used to perform sub-microsecond time alignment correction on the image. The time alignment correction ensures the precise synchronization of each view image on the time axis by analyzing the occurrence time of the light pulses in the image, eliminating the slight time deviation caused by the camera hardware or signal transmission, and finally obtaining a high-precision synchronized multi-view original image.

[0033] Step 200: preprocessing the multi-view original image to obtain a preprocessed multi-view image;

[0034] Specifically, the multi-view original image is denoised to reduce the noise interference in the image and obtain a preliminarily denoised multi-view image. In order to improve the consistency and color accuracy of the image, a standard color card is placed in the shooting scene, and the preliminarily denoised multi-view image is analyzed by the known color value of the standard color card to establish a global color mapping model. The model is used to correct the color difference between the multi-view images to ensure that the images captured by different cameras are consistent in color. Based on the global color mapping model, the preliminarily denoised multi-view image is jointly color corrected to obtain a color-corrected multi-view image. In order to improve the image quality, the color-corrected multi-view image is histogram analyzed to analyze the pixel value distribution characteristics of each image to obtain detailed information about brightness and contrast. Based on the pixel value distribution characteristics, an adaptive exposure equalization function is constructed, which can adjust the exposure and brightness of each image according to the specific situation of each image, so that the final image is more balanced. The pixel value of the color-corrected multi-view image is adjusted based on the adaptive exposure equalization function to obtain an exposure-balanced multi-view image, so that the overall brightness of the image is more uniform, avoiding image quality problems caused by uneven exposure. In order to eliminate the image distortion caused by the characteristics of the camera lens, the multi-view image after exposure equalization is subjected to distortion correction using a polynomial model. The polynomial model can accurately correct the geometric deformation introduced by the lens to obtain a multi-view image after distortion correction, so that the geometric structure of all view images is consistent. The multi-view image after distortion correction is subjected to super-resolution processing to obtain a high-resolution multi-view image. Super-resolution processing technology can improve the resolution of the image through interpolation and detail compensation, making the final generated 3D image clearer and more detailed. In order to ensure the clarity of the image edge and the overall visual effect, the high-resolution multi-view image is subjected to edge sharpening and detail enhancement processing to enhance the key details in the image, and finally the pre-processed multi-view image is obtained.

[0035] Step 300: extract and match the preprocessed multi-view images to obtain feature matching results;

[0036] It should be noted that feature points are extracted from the preprocessed multi-view images, and a preliminary feature point set with unique properties is identified from each image through an algorithm. These feature points usually represent the key parts with high recognition in the image. Based on the preliminary feature point set, an affine invariant feature set is constructed, and image processing technology is used to ensure that these feature sets remain invariant under affine transformation, thereby improving the robustness of feature matching under different viewpoints. Global semantic feature extraction is performed on the affine invariant feature set. During the extraction process, the semantic information of objects in the image is considered, so that these feature points not only rely on geometric shapes, but also have certain semantic understanding capabilities, forming a semantic enhancement feature set. By incorporating semantic information, the accuracy of matching can be effectively improved. Based on the semantic enhancement feature set, a feature fast index tree is constructed. The index tree structure can accelerate the matching process between feature points and reduce the calculation time. Based on the feature fast index tree, rough matching of feature points between multi-view images is performed to obtain a preliminary matching pair set. In the rough matching process, the feature points are effectively screened through the fast index tree to determine the possible corresponding feature pairs. Based on the preliminary matching pair set, a local relationship graph of feature points is constructed. The relationship between feature points is modeled using graph theory methods, and the local geometric relationship between them is analyzed to improve the reliability of matching. An optimized matching pair set is generated based on the local relationship graph of feature points. The optimization algorithm is used to eliminate incorrect matches and add new correct matching pairs to form a more accurate feature point matching set. A global geometric consistency model is established based on the optimized matching pair set. By analyzing the geometric relationship of the entire image set, the feature point matching is ensured to be consistent on a global scale. The matching results are refined based on the global geometric consistency model to obtain a precise matching pair set. By fine-tuning and adjusting the preliminary matching pairs, it is ensured that each matching feature point pair is consistent at the global and local levels. Based on the precise matching pair set, multi-view features are fused and propagated, and the feature information from different perspectives is comprehensively considered to obtain the final feature matching result.

[0037] Step 400: aligning the preprocessed multi-view images and calculating depth information based on the feature matching results to obtain aligned multi-view images and depth maps;

[0038] Specifically, a multi-view geometric relationship graph is constructed based on the feature matching results to obtain the topological structure of the initial view. The topological structure represents the geometric relationship between images from different perspectives. By analyzing these relationships, the relative position and posture of the camera during shooting can be preliminarily understood. The initial view topological structure is optimized, and the optimized view topological structure is obtained by eliminating redundant perspective relationships or enhancing the connection between key perspectives. A hierarchical sparse bundle adjustment model is established based on the optimized view topological structure. The model can effectively handle the complex geometric relationship between multiple perspectives and generate the pose parameters of the initial camera, that is, the spatial position and orientation of each camera during shooting. The initial camera pose parameters are iteratively optimized, and the camera pose is gradually adjusted by minimizing the reprojection error so that all perspective images are optimally aligned in the same three-dimensional space, and the target camera pose parameters are obtained. Based on the target camera pose parameters, the preprocessed multi-view images are geometrically transformed, and the objects in the images are adjusted in space so that they are as consistent as possible under all perspectives, and a roughly aligned multi-view image is obtained. The roughly aligned multi-view images are subjected to multi-view stereo matching cost volume analysis, and the depth of the corresponding pixel points in each perspective image is preliminarily estimated by calculating the disparity information, and the initial depth estimation is obtained. Multi-view based on initial depth estimation Figure 1 Consistency propagation, by comprehensively considering the depth information between multiple perspectives, propagating and adjusting these depth values, to obtain an optimized depth map. The optimized depth map is refined to eliminate noise and errors to obtain a high-precision depth map. Based on the high-precision depth map, the roughly aligned multi-view images are finely aligned, and the images are adjusted so that their details at all perspectives are completely aligned to obtain a precisely aligned multi-view image. The precisely aligned multi-view images and high-precision depth maps are analyzed for spatiotemporal consistency to identify and process dynamic objects in the scene, which may have different positions and shapes at different perspectives. Through special processing, they are ensured to be consistent in the final alignment result. The aligned multi-view images and depth maps are obtained.

[0039] The intrinsic matrix E between the cameras is calculated based on the target camera pose parameters, where E consists of the antisymmetric matrix [t]× of the translation vector t and the rotation matrix R. The intrinsic matrix E is the key element to describe the relative motion of the cameras in the multi-camera system and provides the geometric relationship between the multi-view images. Based on this, the pre-processed multi-view images are subjected to epipolar correction to ensure that the corresponding points of each pair of images are located on the corrected epipolar lines, and the corrected image pairs are obtained. The corrected image pairs are subjected to phase consistency transformation, and the sub-pixel displacement field between the images is extracted by calculating the phase correlation function. This step accurately captures the tiny displacement errors, and the affine transformation matrix is ​​calculated based on this information, so that the images are geometrically adjusted to obtain the roughly aligned multi-view images. The roughly aligned multi-view images are subjected to anisotropic diffusion filtering, and the images are smoothed while retaining the image edges to generate edge-preserving smoothed image sets. Multi-scale Log-Gabor filtering is performed on the smoothed image set. This filtering method can generate feature tensor fields and describe the local structural information of the image by analyzing image features at multiple scales. The structure tensor is calculated based on the feature tensor field. The structure consistency map is obtained by performing eigenvalue decomposition on the structure tensor, revealing the consistency of the image under different viewing angles. The structure consistency map is used to perform non-rigid registration on the preliminarily aligned multi-view image set, and the images are more finely aligned to ensure that they are highly consistent under all viewing angles, generating a finely aligned multi-view image set. Quadrant segmentation is constructed based on the finely aligned multi-view image set, and the image is segmented and decomposed into multiple superpixel sets. Each superpixel set contains a group of pixels with similar features. The multi-view matching cost volume is calculated based on the superpixel set, and the initial depth map is obtained by integrating the image information from multiple viewing angles. The variational method is applied to solve the initial depth map, and the depth information is continuously adjusted and refined through iterative optimization to obtain a high-precision initial depth estimate.

[0040] Bilateral joint upsampling is performed on the high-precision depth map and the coarsely aligned multi-view image to improve the resolution of the depth map while maintaining the edge information, thereby obtaining a high-resolution depth map. The disparity gradient field is calculated based on the high-resolution depth map, and the disparity gradient field can reflect the rate of change of the disparity between different images. On this basis, an affine transformation matrix is ​​constructed to locally deform the coarsely aligned multi-view image so that each view image is more accurately aligned in space, thereby obtaining a finely aligned multi-view image. In order to optimize the aligned image, a structural tensor analysis is performed on the finely aligned multi-view image to extract the edge information of the image while maintaining the smoothness of the image to generate an edge-preserving smooth image. Based on the edge-preserving smooth image, a multi-scale local phase consistency metric is calculated, which can reflect the structural similarity of the image at different scales, and a local structural similarity map is obtained to help identify small differences in the image. Based on the local structural similarity map, a photometric consistency constraint is constructed, and combined with a geometric smoothness constraint, the image is accurately aligned by solving the displacement field to obtain a precisely aligned multi-view image. Perform spatiotemporal voxelization on the precisely aligned multi-view images and high-resolution depth maps, map the image and depth information into four-dimensional space, construct a 4D spatiotemporal voxel grid, and calculate the color and depth feature vectors of each voxel to form a characterized spatiotemporal voxel field. Based on this spatiotemporal voxel field, a spatiotemporal graph is constructed, and the spatiotemporal segmentation results are obtained. Through these segmentation results, object regions in different time and space are identified. Connected domain analysis is performed on the spatiotemporal segmentation results, and the spatiotemporal consistency score of each connected domain is calculated. Through these scores, different dynamic object regions are identified and clustered to obtain a set of dynamic object candidate regions R. The set of dynamic object candidate regions contains regions where dynamic objects may exist. Based on the set of dynamic object candidate regions, optical flow constraints and depth consistency constraints are constructed, and a fine motion field is obtained through joint optimization to capture the movement trajectory and morphological changes of dynamic objects. According to the fine motion field, the precisely aligned multi-view images and high-resolution depth maps are temporally interpolated and extrapolated to ensure consistency when processing dynamic objects on the time axis, and finally the aligned multi-view images and depth maps are obtained.

[0041] Step 500: Reconstruct a 3D model based on the aligned multi-view images and depth map to obtain a scene 3D model;

[0042] Specifically, the aligned multi-view images and depth maps are voxelized, and an initial low-resolution voxel model is obtained by mapping the image and depth information into a three-dimensional voxel grid, so that the scene data can be represented in space. Based on the initial low-resolution voxel model, an octree data structure analysis is performed, and a multi-resolution voxel representation is obtained by recursively dividing the three-dimensional space into smaller subspaces. The multi-resolution voxel representation can not only effectively compress data, but also provide detail information at different resolutions. The signed distance field value is calculated for each voxel in the multi-resolution voxel representation. These values ​​represent the distance between each voxel in the scene and the surface of the object, and the positive and negative signs are used to distinguish whether the voxel is inside or outside the object, so as to obtain the initial SDF model. Based on the initial SDF model, multi-view depth fusion is performed to integrate the depth information from different viewpoints, eliminate the deviations and inconsistencies between the viewpoints, and obtain the optimized SDF model. The optimized SDF model is adaptively subdivided to generate more details in areas where higher resolution is required, so as to obtain a high-precision surface mesh model. Based on the high-precision surface mesh model, multi-view Figure 1 Consistency constraints are set to ensure the consistency of the model surface under different viewing angles, and the objective function of surface optimization is established. The surface optimization objective function is solved iteratively, and the model surface is made more refined by gradually adjusting the vertex positions of the mesh, thus obtaining a refined surface model. A multi-view texture set is extracted based on the refined surface model, and rich details are added to the model surface by obtaining texture information from images of different viewing angles. The multi-view texture set is globally optimized by graph cutting, and the best texture splicing path is selected through the optimization algorithm to eliminate the boundaries between images and obtain a seamless texture mapping. Based on the seamless texture mapping, it is applied to the refined surface model to generate the final scene 3D model.

[0043] Step 600: Perform stereoscopic parallax correction and rendering processing on the scene 3D model to generate a stereoscopic 3D image.

[0044] Specifically, the 3D model of the scene is deeply analyzed based on the human eye stereoscopic vision perception model. By simulating the human eye's perception of depth, the depth distribution characteristics of the scene are obtained to reflect the relative distance and layering of different objects in space. According to the depth distribution characteristics of the scene, an adaptive stereo disparity adjustment model is constructed. The model can dynamically adjust the disparity to adapt to objects in different depth ranges to ensure that the stereo effect of the final image is consistent with the human eye visual experience. The 3D model of the scene is disparity adjusted based on the adaptive stereo disparity adjustment model to obtain a preliminary stereo disparity correction result. Based on the preliminary stereo disparity correction result, a disparity continuity optimization model is constructed. Through the model, the large disparity area is carefully processed to eliminate the unnatural effect caused by excessive disparity changes, and the optimized stereo disparity correction result is obtained. Based on the optimized stereo disparity correction result, dense light field sampling data is generated, which contains rich light information and perspective details in the scene. In order to improve data processing efficiency, the dense light field sampling data is compressed to generate a lightweight light field model. While ensuring the integrity of the light field information, the model greatly reduces the storage and computing overhead, which is convenient for fast processing in practical applications. Based on the lightweight light field model, the initial rendering result is constructed to convert the light field data into a visual image. The initial rendering result is image enhanced to improve the image detail expression and contrast, and obtain a high-quality rendered image. The high-quality rendered image is subjected to spatiotemporal analysis and jitter suppression processing to effectively reduce the jitter caused by changes in viewing angle or display device, and obtain a smooth and stable stereoscopic 3D image.

[0045] In the embodiment of the present application, the clock synchronization and the central control unit generate microsecond trigger signals through the WiFi module, and the light pulses of a specific pattern are emitted by the high-speed LED array to realize the sub-microsecond time alignment correction of multiple cameras, which greatly improves the time consistency of multi-view images. The global color mapping model, adaptive exposure equalization function, polynomial model distortion correction and other technologies are adopted to effectively solve the noise, color difference, uneven exposure and distortion of multi-view images. By constructing an affine invariant feature set, global semantic feature extraction and feature fast indexing tree, efficient and accurate matching between multi-view images is achieved. The hierarchical sparse bundle adjustment model, multi-view stereo matching cost volume analysis and other technologies are used to achieve accurate alignment of multi-view images and generation of high-precision depth maps. Through spatiotemporal voxelization, spatiotemporal consistency analysis and fine motion field optimization, dynamic objects are effectively identified and processed, ensuring the spatiotemporal consistency of the reconstruction results. The multi-resolution voxel representation, signed distance field optimization and adaptive surface subdivision and other technologies are adopted to achieve high-precision and high-detail 3D model reconstruction. Based on the human eye stereoscopic vision perception model and adaptive stereo disparity adjustment, combined with dense light field sampling and rendering enhancement technology, high-quality, highly immersive stereoscopic 3D images are generated.

[0046] In a specific embodiment, the process of executing step 100 may specifically include the following steps:

[0047] Arranging the multiple cameras in an array to obtain a camera array, and configuring a WiFi module for each camera in the camera array to obtain a camera array with a WiFi module;

[0048] A central control unit is provided to establish a connection with the camera array with a WiFi module through a WiFi network to construct a camera system with a central control unit;

[0049] Generate a microsecond trigger signal based on the central control unit to obtain a microsecond synchronization trigger signal, and transmit the microsecond synchronization trigger signal to each camera through a WiFi network;

[0050] The camera system with the central control unit is triggered based on the transmitted microsecond-level synchronization trigger signal to obtain a preliminarily synchronized multi-view image;

[0051] A high-speed LED array is arranged in the shooting scene, and based on the microsecond-level synchronization trigger signal, the high-speed LED array is controlled to emit light pulses of a specific pattern to obtain a scene with a time mark;

[0052] The scene with the time mark is photographed to obtain a multi-view image with a light pulse pattern, and the multi-view image with the light pulse pattern is transmitted to a cloud APP via a WiFi network;

[0053] In the cloud APP, sub-microsecond time alignment correction is performed based on the light pulse pattern in the multi-view image with the light pulse pattern to obtain the multi-view original image.

[0054] Specifically, according to the needs of the scene and the coverage of the viewing angle, the cameras are arranged in an array to maximize the coverage of different viewing angles of the scene and ensure that sufficient information is captured from multiple angles. In the camera array, each camera is equipped with a WiFi module, so that it communicates with the central control unit through the WiFi network to form a camera array with a WiFi module. The connection between the central control unit and the camera array is established through the WiFi network. The function of the central control unit is to generate and distribute high-precision synchronization trigger signals to ensure that all cameras start shooting at the same time. In order to achieve microsecond-level time synchronization, the central control unit needs to generate microsecond-level trigger signals. The microsecond-level synchronization trigger signal can ensure that the shooting time difference between cameras is less than 1 microsecond, avoiding motion blur or distortion in the image due to the time difference. This trigger signal is transmitted to each camera through the WiFi network. Although WiFi transmission may introduce some delays, these delays can be compensated at the receiving end by embedding timestamp information in the signal. The transmitted microsecond-level synchronization trigger signal can ensure that each camera starts to expose and shoot at the moment of receiving the signal, thereby obtaining a preliminarily synchronized multi-view image. In order to improve the accuracy of time synchronization and provide a reliable time reference for subsequent image processing, a high-speed LED array is set in the shooting scene. The high-speed LED array can provide clear time marks for the scene by emitting light pulses. These light pulses are controlled by the central control unit based on microsecond synchronization trigger signals to ensure that the light pulses are synchronized with the camera shooting. The time mark can be used as a reference for time alignment in subsequent image processing by introducing recognizable optical signals into the scene. After the camera receives the synchronization trigger signal and starts shooting, the scene with the time mark is recorded to form a multi-view image with a light pulse pattern. Since these light pulse marks are included in the image captured by each camera, these marks can be used to perform sub-microsecond time alignment correction on the image in the subsequent processing process. The multi-view images with light pulse patterns are transmitted to the cloud APP via the WiFi network. In the cloud APP, the light pulse marks in the image are used to perform sub-microsecond time alignment correction on the image. The time alignment correction ensures the precise synchronization of each view image on the time axis by analyzing the occurrence time of the light pulse in the image, eliminating the slight time deviation caused by the camera hardware, WiFi transmission or signal processing, and finally obtaining a high-precision synchronized multi-view original image. To better understand this time synchronization process, we can describe it through a simple mathematical model. Let the trigger signal time received by each camera i be t i , where i=1, 2, ..., n represents different cameras. By averaging the time signals of all cameras, a system average time T is obtained. avg As a base time:

[0055] T avg =(1 / n)*Σ(t i )(sum of i from 1 to n);

[0056] Among them, T avg is the average time of each camera and is used as the time reference for the entire system. If the clock of a camera i is biased, its internal clock can be adjusted to make t i Equal to T avg , achieving time synchronization for the entire system.

[0057] When processing light pulse markers, the time difference Δt of the light pulse arriving at the camera is defined. This difference is corrected by the corresponding correction algorithm to ensure that all images are recorded at the same time point. This time correction can be expressed as:

[0058] t i =t i +Δt i ;

[0059] where t i ' is the corrected timestamp. By adjusting the timestamp of each camera, the final image is ensured to be completely consistent in time.

[0060] Through the above steps, high-precision time synchronization and perspective alignment can be achieved, and accurate multi-perspective original images can be generated in the cloud APP, laying the foundation for subsequent 3D image or dynamic TIF image generation. This method based on WiFi and cloud processing not only simplifies hardware requirements, but also provides greater flexibility and scalability, making it easier for the system to adapt to different application scenarios and needs.

[0061] In a specific embodiment, the process of executing step 200 may specifically include the following steps:

[0062] Denoising the original multi-view images to obtain preliminarily denoised multi-view images, placing a standard color card in the shooting scene, analyzing the preliminarily denoised multi-view images based on the standard color card, and obtaining a global color mapping model;

[0063] Based on the global color mapping model, the multi-view images with preliminary denoising are jointly color corrected to obtain the color-corrected multi-view images, and the histogram analysis is performed on the color-corrected multi-view images to obtain the pixel value distribution characteristics of each image;

[0064] An adaptive exposure equalization function is constructed based on the pixel value distribution characteristics, and the pixel values ​​of the color-corrected multi-view image are adjusted based on the adaptive exposure equalization function to obtain a multi-view image after exposure equalization;

[0065] Performing distortion correction of a polynomial model on the multi-view image after exposure equalization to obtain a multi-view image after distortion correction, and performing super-resolution processing on the multi-view image after distortion correction to obtain a high-resolution multi-view image;

[0066] The high-resolution multi-view image is processed with edge sharpening and detail enhancement to obtain a preprocessed multi-view image.

[0067] Specifically, the multi-view original image is denoised to eliminate the noise introduced by various factors during the shooting process, such as sensor noise or random noise under low light conditions. Common denoising methods include Gaussian filtering, median filtering, and more advanced non-local mean filtering. By analyzing the similarity or spatial relationship between pixels, the noise is removed while retaining the details and texture of the image as much as possible, and a preliminarily denoised multi-view image is obtained. A standard color card is placed in the shooting scene. The standard color card is a tool with a known color reference value, which is usually used for color correction in image processing. By placing the standard color card in the shooting scene, the preliminarily denoised multi-view image is analyzed using the known color reference value to establish a global color mapping model. The construction of this model is achieved by comparing the actual color value of the standard color card in the image with the expected standard color value, and the color deviation in the image can be identified and corrected accordingly. Based on the established global color mapping model, the preliminarily denoised multi-view image is jointly color corrected to ensure that the color performance of all view images is consistent and avoid color inconsistency caused by different cameras or shooting conditions. By mapping the color value of each image to a unified color space, a color-corrected multi-view image is obtained. Perform histogram analysis on the color-corrected multi-view images. By counting the number of pixels with different brightness levels in each image, the overall brightness distribution and contrast of the image can be reflected. The shape of the histogram can help identify whether the image has underexposure or overexposure problems. By analyzing the histogram of each image, the distribution characteristics of the pixel values ​​are extracted. Based on the pixel value distribution characteristics, an adaptive exposure equalization function is constructed. This function adjusts the exposure of different brightness areas in the image so that the image has a more uniform brightness distribution as a whole. Adaptive exposure equalization stretches or compresses the histogram of the image so that the brightness values ​​are more widely distributed throughout the available brightness range, thereby improving the contrast and clarity of the image. The pixel values ​​of the color-corrected multi-view images are adjusted by this function to obtain a multi-view image with balanced exposure. The multi-view image with balanced exposure is subjected to distortion correction using a polynomial model to eliminate the geometric distortion caused by the camera lens, especially in wide-angle lenses or fisheye lenses, where the distortion is particularly obvious. The polynomial model can be used to approximate this distortion and restore it to a distortion-free perspective by adjusting the position of each pixel in the image. Assume that the coordinates in the original image are The corrected coordinates are , then the polynomial model can be expressed as:

[0068] ;

[0069] ;

[0070] in, are the coefficients of the model. By fitting these coefficients, distortion correction is achieved to obtain a multi-view image after distortion correction. Super-resolution processing is performed on the multi-view image after distortion correction. Low-resolution images are converted into high-resolution images through algorithms. Common methods include traditional methods based on interpolation and modern methods based on deep learning. Super-resolution processing can restore the details and textures in the image, making the final image clearer and obtaining a high-resolution multi-view image. Edge sharpening and detail enhancement are performed on the high-resolution multi-view image. Edge sharpening makes the outline of the object clearer by enhancing the contrast of the edge part of the image. Detail enhancement makes the image look more delicate by improving the clarity of the texture and microstructure in the image, and finally obtains the pre-processed multi-view image.

[0071] In a specific embodiment, the process of executing step 300 may specifically include the following steps:

[0072] Extract feature points from the preprocessed multi-view images to obtain a preliminary feature point set, and construct an affine invariant feature set based on the preliminary feature point set;

[0073] Perform global semantic feature extraction on the affine invariant feature set to obtain a semantically enhanced feature set, and construct a feature fast index tree based on the semantically enhanced feature set;

[0074] Based on the feature fast index tree, the feature points between multi-view images are roughly matched to obtain a preliminary matching pair set, and a local relationship graph of the feature points is constructed based on the preliminary matching pair set;

[0075] Generate an optimized set of matching pairs according to the local relationship graph of feature points, and establish a global geometric consistency model based on the optimized set of matching pairs;

[0076] The matching results are refined based on the global geometric consistency model to obtain a set of exact matching pairs, and multi-view features are fused and propagated based on the set of exact matching pairs to obtain feature matching results.

[0077] Specifically, feature points are extracted from the preprocessed multi-view images. These feature points represent key locations in the image with unique textures or shapes, and are often used for subsequent image matching and 3D reconstruction. Common feature point extraction algorithms include SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (Oriented FAST and RotatedBRIEF), etc. These algorithms can detect feature points at multiple scales and calculate the descriptor for each feature point to obtain a preliminary feature point set. In order to cope with affine transformations between different viewpoints, an affine invariant feature set is constructed based on the preliminary feature point set. Affine invariant features can remain stable when the image undergoes affine transformations such as scaling, rotation, and translation. The preliminary feature point set is transformed by the affine transformation matrix to ensure that the features of the same object under different viewpoints can be described consistently. Assume that the coordinates of a feature point in the image before the transformation are , the transformed coordinates are , then the affine transformation can be expressed as:

[0078] ;

[0079] Where a, b, c, d are the parameters of the affine transformation matrix, and e, f are translation vectors. By applying this transformation to the feature points, an affine invariant feature set is obtained. In order to extract higher-level feature information, global semantic feature extraction is performed on the affine invariant feature set to capture the semantic information of objects in the image, such as identifying the object category, shape, and spatial relationship in the image. Through deep learning technology, the local features of the image are promoted to features with semantic meaning to form a semantically enhanced feature set. The semantically enhanced features not only consider the similarity at the pixel level, but also incorporate the information of object recognition and classification, which helps to improve the accuracy of matching. A feature fast index tree is constructed based on the semantically enhanced feature set. The feature fast index tree is a data structure that can efficiently organize and retrieve feature points and quickly match a large number of feature points. Common index structures include KD trees and LSH, which can map feature points to a specific space through hierarchical or hashing methods, thereby speeding up the matching process. According to the index tree, feature points are roughly matched between multi-view images to obtain a preliminary set of matching pairs. Through fast retrieval, possible corresponding feature point pairs in different images are found. Screen the preliminary matching pair set. Construct a local relationship graph of feature points based on the preliminary matching pair set. The local relationship graph reveals their relative positions and geometric layouts in the image by analyzing the spatial relationships between feature points. Through this graph theory method, those matching pairs that do not conform to the geometric relationship are identified and eliminated to generate an optimized matching pair set. A global geometric consistency model is established based on the optimized matching pair set. The global geometric consistency model ensures that feature point matching is consistent on a global scale by comprehensively considering the matching results of all viewpoint images. The model usually optimizes the matching results by minimizing the reprojection error or geometric error. The reprojection error can be expressed as:

[0080] ;

[0081] in, is the actual matching point position, is the projection point position calculated based on the matching model, Represents the sum of the reprojection errors of all matching points. By optimizing the error function, the optimal global geometric consistency model is obtained. The matching results are refined based on the global geometric consistency model. Through more refined geometric correction and optimization, incorrect matches are eliminated and the confidence of correct matches is enhanced to obtain a set of accurate matching pairs. Multi-view feature fusion and propagation are performed based on the set of accurate matching pairs to obtain feature matching results. Multi-view feature fusion generates more robust and accurate matching results by integrating feature point information from different perspectives. Feature propagation uses known matching information to infer new matching pairs in unmatched viewing angles, thereby expanding the matching coverage.

[0082] In a specific embodiment, the process of executing step 400 may specifically include the following steps:

[0083] A multi-view geometric relationship graph is constructed based on the feature matching results to obtain an initial view topology structure, and the initial view topology structure is optimized to obtain an optimized view topology structure;

[0084] Based on the optimized view topology structure, a hierarchical sparse bundle adjustment model is established to generate initial camera pose parameters, which are iteratively optimized to minimize the reprojection error and obtain the target camera pose parameters.

[0085] Based on the target camera pose parameters, the preprocessed multi-view image is geometrically transformed to obtain a roughly aligned multi-view image, and a multi-view stereo matching cost volume analysis is performed on the roughly aligned multi-view image to obtain an initial depth estimate.

[0086] Multi-view based on initial depth estimation Figure 1 The optimized depth map is obtained by performing consistent propagation, and the optimized depth map is refined to obtain a high-precision depth map;

[0087] Based on the high-precision depth map, the roughly aligned multi-view images are finely aligned to obtain the accurately aligned multi-view images, and the precisely aligned multi-view images and high-precision depth maps are subjected to spatiotemporal consistency analysis to identify and process dynamic objects to obtain the aligned multi-view images and depth maps.

[0088] Specifically, a multi-view geometric relationship graph is constructed based on the feature matching results to describe the relative geometric relationship between different viewpoints. By analyzing the positional relationship between the matching feature points in each viewpoint, a multi-view topological structure is constructed, which represents the initial estimate of the relative position and posture of the camera. The initial view topological structure is optimized to remove unreliable view connections, enhance the connection between key viewpoints, and adjust the relative positions between viewpoints through the optimization algorithm to obtain the optimized view topological structure. A hierarchical sparse bundle adjustment model is established based on the optimized view topological structure. Hierarchical sparse bundle adjustment is a commonly used multi-view reconstruction technology. By globally optimizing the camera pose and the position of the three-dimensional point, it can simultaneously correct the projection error between images and the inaccuracy of the camera pose. The core of this process is to minimize the reprojection error, that is, to make the projection points of the three-dimensional points on all images as consistent as possible with the actually observed feature points. The reprojection error can be expressed by the following formula:

[0089] ;

[0090] in, is the total reprojection error, represents the actual observed position of the i-th 3D point on the j-th image, Represents the rotation matrix through the camera and translation vectors For 3D points Calculate the position after projection. Obtain the optimal camera pose parameters, i.e., the target camera pose parameters, by iteratively optimizing the error function. Perform geometric transformation on the preprocessed multi-view images based on the target camera pose parameters, align all images into a common three-dimensional space, and obtain a roughly aligned multi-view image. Perform multi-view stereo matching cost volume analysis on the roughly aligned multi-view images. Stereo matching cost volume analysis preliminarily estimates the depth information of each pixel by calculating the disparity map. The disparity map represents the pixel offset of objects in the same scene at different perspectives, and this offset is directly related to the depth of the object. Based on the preliminary disparity estimation, obtain the initial depth map. Perform multi-view stereo matching based on the initial depth estimation. Figure 1 Consistency propagation. By integrating and propagating the depth information of multiple perspectives, inconsistent depth values ​​are eliminated to obtain an optimized depth map. The optimized depth map is refined to remove noise and errors, improve the accuracy of the depth map, and obtain a high-precision depth map. Based on the high-precision depth map, the roughly aligned multi-view images are finely aligned. By adjusting the geometric transformation parameters of the image, the image is made completely consistent with the depth information at the pixel level, and a precisely aligned multi-view image is obtained. The precisely aligned multi-view images and high-precision depth maps are analyzed for spatiotemporal consistency to identify and process dynamic objects in the scene. These objects may be in different positions at different time points, resulting in inconsistencies between the depth map and the image. By analyzing these inconsistencies, dynamic objects are accurately identified, and their influence on 3D reconstruction is eliminated through corresponding processing, and finally the aligned multi-view images and depth maps are obtained.

[0091] In a specific embodiment, the execution step performs geometric transformation on the preprocessed multi-view image based on the target camera pose parameters to obtain a coarsely aligned multi-view image, and performs multi-view stereo matching cost volume analysis on the coarsely aligned multi-view image to obtain an initial depth estimation process, which may specifically include the following steps:

[0092] Based on the target camera pose parameters, the essential matrix E between the cameras is calculated, where E=[t]×R, [t]× is the antisymmetric matrix of the translation vector t, and R is the rotation matrix; epipolar correction is performed on the preprocessed multi-view images to obtain the corrected image pairs;

[0093] Performing phase consistency transformation on the corrected image pair, calculating the phase correlation function, obtaining a sub-pixel displacement field, and calculating the affine transformation matrix based on the sub-pixel displacement field to obtain a roughly aligned multi-view image;

[0094] Anisotropic diffusion filtering is performed on the roughly aligned multi-view images to obtain a smoothed image set with edge preservation, and multi-scale Log-Gabor filtering is performed on the smoothed image set to obtain a feature tensor field;

[0095] The structure tensor is calculated based on the feature tensor field, and the eigenvalue decomposition is performed on the structure tensor to obtain a structure consistency map, and the preliminarily aligned multi-view image set is non-rigidly registered based on the structure consistency map to obtain a finely aligned multi-view image set;

[0096] Quadruped segmentation is constructed for the finely aligned multi-view image set to obtain a superpixel set. The multi-view matching cost volume is calculated based on the superpixel set to obtain an initial depth map. The variational method is applied to the initial depth map, and the initial depth estimation is obtained through iterative optimization.

[0097] Specifically, the essential matrix E between the cameras is calculated based on the target camera pose parameters. The essential matrix E describes the relative motion relationship between the two cameras and is closely related to the relationship between points in three-dimensional space and their projections on the image plane. The essential matrix can be expressed as ,in is the translation vector The antisymmetric matrix of , R is the rotation matrix. It can be expressed as:

[0098] ;

[0099] in, are the components of the translation vector t on the x, y, and z axes, respectively. By substituting the camera's rotation matrix R and the translation vector t into the above formula, the essential matrix E is calculated, which is used to describe the geometric relationship between the two cameras. Epipolar correction is performed on the preprocessed multi-view images to constrain the corresponding points in the image to the same epipolar line, simplifying the stereo matching process. Through epipolar correction, the image distortion caused by different shooting angles between cameras is eliminated, so that the corresponding points of each pair of images are aligned on the corresponding epipolar lines to obtain the corrected image pairs. Phase consistency transformation is performed on the corrected image pairs. Phase consistency transformation can effectively extract the edge and texture features of the image by analyzing the phase information of different frequency components in the image. By calculating the phase correlation function, the phase consistency information between the images is obtained, and the displacement field at the sub-pixel level is calculated. The phase correlation function is usually used to compare the similarity between images in the frequency domain, which can provide higher accuracy than traditional pixel-level alignment. With this information, the affine transformation matrix is ​​calculated, the image is geometrically corrected, and a coarsely aligned multi-view image is obtained. Anisotropic diffusion filtering is performed on the coarsely aligned multi-view images. Anisotropic diffusion filtering is a filtering method that can remove noise while retaining image edge information. Through the directional constraints in the diffusion process, the image is kept sharp near the edge, while the noise is removed in the smooth area, and a set of smooth images with edge preservation is obtained. Multiscale Log-Gabor filtering is performed on the edge-preserving smooth image set. Log-Gabor filter is a filter that performs well in the frequency domain and can extract local features of images at different scales. By applying Log-Gabor filtering at multiple scales, a feature tensor field is generated, which contains rich feature information in the image. The structure tensor is calculated based on the feature tensor field. The structure tensor is a mathematical tool that describes the local structural information of an image. By performing eigenvalue decomposition on the structure tensor, a structure consistency map is obtained, which reflects the structural similarity of the image in different regions. Non-rigid registration is performed on the preliminarily aligned multi-view image set based on the structure consistency map. Non-rigid registration allows local deformation between images so that they can achieve higher alignment accuracy at the pixel level. Through this registration method, a finely aligned multi-view image set is obtained. Quadruplet segmentation is constructed for the finely aligned multi-view image set to generate a superpixel set. Superpixel sets are a way of representing an image by dividing it into multiple small regions with similar features. They can effectively reduce the computational complexity of image processing and preserve important edge information. Multi-view matching cost volumes are calculated based on superpixel sets to generate an initial depth map. Multi-view matching cost volume analysis estimates the depth information of each pixel by comparing superpixels in images from different perspectives to obtain an initial depth map. The variational method is applied to the initial depth map to solve it. The variational method defines an energy function and iteratively optimizes the energy function to minimize the overall error of the depth map, thereby obtaining a more accurate initial depth estimate.The core of this process is to find an optimal depth distribution so that the reprojection error of all view images is minimized.

[0100] In a specific embodiment, the execution step finely aligns the roughly aligned multi-view images based on the high-precision depth map to obtain the precisely aligned multi-view images, and performs spatiotemporal consistency analysis on the precisely aligned multi-view images and the high-precision depth map, identifies and processes dynamic objects, and obtains the aligned multi-view images and depth maps. The process may specifically include the following steps:

[0101] Perform bilateral joint upsampling on the high-precision depth map and the coarsely aligned multi-view image to obtain a high-resolution depth map;

[0102] The disparity gradient field is calculated based on the high-resolution depth map, and the affine transformation matrix is ​​constructed based on the disparity gradient field to locally deform the coarsely aligned multi-view image to obtain a finely aligned multi-view image.

[0103] Performing structural tensor analysis on the finely aligned multi-view images to obtain edge-preserving smooth images, and calculating multi-scale local phase consistency metrics based on the edge-preserving smooth images to obtain local structural similarity maps;

[0104] Based on the local structural similarity mapping, the photometric consistency constraint is constructed, and combined with the geometric smoothness constraint, the displacement field is solved to obtain the precisely aligned multi-view images.

[0105] Perform spatiotemporal voxelization on the precisely aligned multi-view images and high-resolution depth maps, construct a 4D spatiotemporal voxel grid, and calculate the color and depth feature vectors of each voxel to obtain a characterized spatiotemporal voxel field;

[0106] Based on the characterized space-time voxel field, a space-time map is constructed to obtain the space-time segmentation result;

[0107] Perform connected domain analysis on the spatiotemporal segmentation results, calculate the spatiotemporal consistency score of each connected domain, cluster the connected domains according to the spatiotemporal consistency score, and obtain the dynamic object candidate region set R;

[0108] Based on the candidate region set of dynamic objects, optical flow constraints and depth consistency constraints are constructed and jointly optimized to obtain a fine motion field.

[0109] The precisely aligned multi-view images and high-resolution depth maps are temporally interpolated and extrapolated according to the fine motion field to obtain the aligned multi-view images and depth maps.

[0110] Specifically, bilateral joint upsampling is performed on the high-precision depth map and the coarsely aligned multi-view image. Bilateral joint upsampling is a filtering method that combines spatial domain and color domain information, which can effectively improve the resolution of the depth map while maintaining the image edge details. The formula for bilateral joint upsampling is:

[0111] ;

[0112] in, is the upsampled depth value, is the original depth value, is the neighborhood of pixel x, and are the color values ​​of pixels x and y, respectively. and Control the smoothing parameters in the spatial and color domains respectively, is the normalization factor. Through this process, a high-resolution depth map consistent with the resolution of the coarsely aligned multi-view image is generated. The disparity gradient field is calculated based on the high-resolution depth map. The disparity gradient field represents the rate of change of depth information and can reveal the three-dimensional structural characteristics of objects in the image. Based on the disparity gradient field, an affine transformation matrix is ​​constructed to locally deform the coarsely aligned multi-view image. The formula of the affine transformation matrix can be expressed as:

[0113] ;

[0114] in, are the original coordinates, are the transformed coordinates, a, b, c, d are the parameters that control the transformation, and e and f are the translations. The local deformation process ensures that images from different perspectives achieve higher alignment accuracy in terms of geometric structure, thereby obtaining finely aligned multi-view images. The structural tensor analysis is performed on the finely aligned multi-view images. The structural tensor is a tool that can capture the local structural features of an image. By analyzing the gradient information in the image, a smooth image with edge preservation is obtained. It can be expressed as:

[0115] ;

[0116] in, and is the gradient of the image in the x and y directions. By analyzing the structure tensor, an edge-preserving smooth image is obtained, and the multi-scale local phase consistency metric is further calculated to generate a local structure similarity map. Based on the local structure similarity map, photometric consistency constraints and geometric smoothness constraints are constructed. Photometric consistency constraints It can be expressed as:

[0117] ;

[0118] in, and are the pixels in the two images The strength value of is the displacement field Will Mapped to a location in another image. Geometric smoothness constraints It is expressed as:

[0119] ;

[0120] in, Represents pixels Neighborhood pixels . By jointly solving these two constraints, a precisely aligned multi-view image is obtained. Spatiotemporal voxelization is performed on the precisely aligned multi-view image and high-resolution depth map, a 4D spatiotemporal voxel grid is constructed, and the color and depth feature vectors of each voxel are calculated to obtain a characterized spatiotemporal voxel field. A spatiotemporal graph is constructed based on the voxel field, and spatiotemporal segmentation is performed to obtain the spatiotemporal segmentation result. A connected domain analysis is performed on the spatiotemporal segmentation result, the spatiotemporal consistency score of each connected domain is calculated, and dynamic objects are identified. Based on dynamic objects, optical flow constraints and depth consistency constraints are constructed, and a refined motion field is obtained through joint optimization. The refined motion field is used to perform temporal interpolation and extrapolation on the precisely aligned multi-view image and high-resolution depth map to obtain the aligned multi-view image and depth map.

[0121] In a specific embodiment, the process of executing step 500 may specifically include the following steps:

[0122] voxelize the aligned multi-view images and depth maps to obtain an initial low-resolution voxel model, and perform octree data structure analysis based on the initial low-resolution voxel model to obtain a multi-resolution voxel representation;

[0123] Calculate the signed distance field value for each voxel in the multi-resolution voxel representation to obtain an initial SDF model, and perform multi-view depth fusion based on the initial SDF model to obtain an optimized SDF model;

[0124] Adaptively subdivide the optimized SDF model to obtain a high-precision surface mesh model, and perform multi-viewing based on the high-precision surface mesh model. Figure 1 Consistency constraints are applied to obtain the surface optimization objective function;

[0125] Iteratively solving the surface optimization objective function to obtain a refined surface model, and extracting a multi-view texture set based on the refined surface model;

[0126] A global optimization of graph cuts is performed on a multi-view texture set to obtain a seamless texture mapping, and a scene 3D model is generated based on the seamless texture mapping.

[0127] Specifically, the aligned multi-view images and depth maps are voxelized. Voxelization is a process of dividing a three-dimensional space into small cubic units (i.e., voxels). An initial low-resolution voxel model is generated by associating each voxel with a pixel or depth information in the image. The low-resolution voxel model can initially capture the three-dimensional structure of the scene at a low computational cost. The octree data structure is analyzed based on the initial low-resolution voxel model. The octree is a tree structure commonly used for three-dimensional spatial data partitioning. It recursively divides the space into eight subspaces to form a hierarchical voxel representation. Each voxel is subdivided into smaller sub-voxels, and the resolution of these sub-voxels can be adjusted as needed. Multi-resolution voxel representation can not only provide higher detail resolution in key areas, but also reduce computational complexity by compressing unimportant areas. A signed distance field value is calculated for each voxel in the multi-resolution voxel representation. The signed distance field value represents the distance from the voxel to the surface of the object, and uses positive and negative signs to distinguish whether the voxel is inside or outside the object. By calculating the signed distance field value of each voxel, an initial SDF model is generated, which describes the shape of the object in the entire space. Based on the initial SDF model, multi-view depth fusion is performed. By integrating depth information from different perspectives, the errors and uncertainties brought by a single perspective are eliminated, and a more accurate three-dimensional shape representation is obtained. By fusing this depth information, an optimized SDF model is generated. The optimized SDF model is adaptively subdivided. By increasing the voxel resolution in areas that require high-precision representation to capture more detailed surface features, a high-precision surface mesh model is generated that can accurately describe the three-dimensional shape of the object. In order to optimize the high-precision surface mesh model, based on multi-view Figure 1 Consistency constraints are used to construct the surface optimization objective function. Figure 1 The consistency constraint can improve the accuracy of the surface model by ensuring that the projections of images from different viewpoints on the mesh surface are consistent. The surface optimization objective function can be expressed as:

[0128] ;

[0129] in, Indicates Perspective at a point The pixel value at Represents the pixel value after projection. The first term measures the projection error, and the second It is the smoothing term of the SDF gradient, which controls the smoothness of the mesh. By iteratively solving the objective function, the mesh surface is optimized to obtain a refined surface model. Based on the refined surface model, a multi-view texture set is extracted. The multi-view texture set contains the texture information corresponding to the surface mesh in the images taken by different cameras. The texture is globally optimized by graph cutting. The graph cutting algorithm can select the best splicing path in the multi-view texture, eliminate the seams between different perspectives, and generate seamless texture mapping. Based on the seamless texture mapping, it is applied to the refined surface model to generate the final scene 3D model.

[0130] In a specific embodiment, the process of executing step 600 may specifically include the following steps:

[0131] Based on the human eye stereoscopic vision perception model, the 3D model of the scene is deeply analyzed to obtain the scene depth distribution characteristics, and an adaptive stereo disparity adjustment model is constructed according to the scene depth distribution characteristics;

[0132] Based on the adaptive stereo disparity adjustment model, the disparity of the scene 3D model is adjusted to obtain a preliminary stereo disparity correction result, and a disparity continuity optimization model is constructed based on the preliminary stereo disparity correction result;

[0133] The large disparity area is processed by a disparity continuity optimization model to obtain an optimized stereo disparity correction result, and dense light field sampling data is generated based on the optimized stereo disparity correction result;

[0134] Compress the dense light field sampling data to obtain a lightweight light field model, and construct the initial rendering result based on the lightweight light field model;

[0135] The initial rendering result is enhanced to obtain a high-quality rendered image, and spatiotemporal analysis and jitter suppression processing are performed based on the high-quality rendered image to obtain a stereoscopic 3D image.

[0136] Specifically, the 3D model of the scene is deeply analyzed through the human eye stereoscopic vision perception model. The human eye stereoscopic vision perception model simulates how the human eyes perceive the depth of objects through parallax. When the human eye observes a scene, the slight position difference between the left and right eyes causes them to see slightly different images, and this difference is called parallax. By calculating these parallaxes, the human brain is able to infer the depth of the object. Based on this model, the 3D model of the scene is deeply analyzed to extract the depth distribution characteristics of the scene. The depth distribution characteristics of the scene reflect the distance hierarchy and depth changes of objects in space. Based on these characteristics, an adaptive stereo disparity adjustment model is constructed. The adaptive stereo disparity adjustment model dynamically adjusts the disparity value according to different depth areas to ensure that the final stereo image visually conforms to the perception habits of the human eye. For example, for distant objects, the disparity should be small, while for close objects, the disparity should be large. The mathematical representation of this adjustment process can be described by the following formula:

[0137] ;

[0138] in, represents the adjusted parallax value, is the adjustment parameter, Represents the depth distance from the object to the camera. The parallax is dynamically adjusted through this formula to obtain a preliminary stereo disparity correction result that adapts to scenes of different depths. A disparity continuity optimization model is constructed based on the preliminary stereo disparity correction result. The disparity continuity optimization model is designed to deal with discontinuity problems caused by excessive parallax changes. Parallax discontinuity can cause visual illusions or unnatural jumps in stereo images. To avoid these problems, the parallax continuity optimization model smoothes the disparity values ​​to make the parallax changes more natural and coherent. This process is usually achieved by minimizing the following objective function:

[0139] ;

[0140] in, is the parallax continuity error, and The adjacent pixels are and The disparity value of Represents pixels neighborhood. By minimizing this error and smoothing the parallax change, an optimized stereo disparity correction result is obtained. Dense light field sampling data is generated based on the optimized stereo disparity correction result. Dense light field represents the light information of each point in the scene in multiple directions. Through dense light field sampling, subtle lighting changes and perspective differences in the scene are captured. The amount of dense light field sampling data is usually very large, so compression processing is required to achieve efficient processing under limited storage and computing resources. The dense light field sampling data is compressed to obtain a lightweight light field model by reducing redundant data. Compression methods may include principal component analysis, transform coding, or other efficient compression algorithms. Through these methods, the amount of data is significantly reduced while retaining important light field information. Based on the lightweight light field model, the initial rendering result is constructed. The initial rendering result is a scene image generated based on the compressed light field data. By reconstructing the light information in the light field, the basic visual effect of the scene is reproduced. Image enhancement processing is performed on the initial rendering result. Image enhancement may include contrast adjustment, color correction, sharpening, etc. to improve the visual effect of the image. For example, the image contrast can be adjusted by the following formula:

[0141] ;

[0142] in, is the enhanced image pixel value, is the original pixel value, is the contrast adjustment factor, is the brightness offset value. Through the enhancement operation, a high-quality rendered image is obtained. Based on the high-quality rendered image, spatiotemporal analysis and jitter suppression processing are performed. Spatiotemporal analysis is used to ensure the consistency of the image in the time series to avoid flickering or instability caused by dynamic changes. Jitter suppression ensures that the image remains smooth and natural when playing stereoscopic 3D images by detecting and compensating for tiny jitters in the image. Through the above steps, a high-quality and stable stereoscopic 3D image is finally obtained.

[0143] The above describes the 3D image generation method based on multi-camera shooting in the embodiment of the present application. The following describes the 3D image generation device 10 based on multi-camera shooting in the embodiment of the present application. Figure 2 In the embodiment of the present application, an embodiment of a 3D image generation device 10 based on multi-camera shooting includes:

[0144] An acquisition module 11 is used to synchronously trigger the acquisition of images acquired by multiple cameras to obtain multi-view original images;

[0145] A preprocessing module 12 is used to preprocess the multi-view original image to obtain a preprocessed multi-view image;

[0146] The matching module 13 is used to extract and match the preprocessed multi-view images to obtain feature matching results;

[0147] A calculation module 14 is used to align the preprocessed multi-view images and calculate the depth information based on the feature matching results to obtain the aligned multi-view images and depth map;

[0148] A reconstruction module 15 is used to reconstruct a 3D model based on the aligned multi-view images and the depth map to obtain a 3D model of the scene;

[0149] The generating module 16 is used to perform stereoscopic parallax correction and rendering processing on the scene 3D model to generate a stereoscopic 3D image.

[0150] Through the cooperation of the above components, the WiFi module is used for clock synchronization and the central control unit generates microsecond trigger signals. Combined with the high-speed LED array to emit light pulses in a specific pattern, the sub-microsecond time alignment correction of multiple cameras is realized, which greatly improves the time consistency of multi-view images. The global color mapping model, adaptive exposure equalization function, polynomial model distortion correction and other technologies are adopted to effectively solve the noise, color difference, uneven exposure and distortion problems of multi-view images. By constructing an affine invariant feature set, extracting global semantic features and fast feature indexing trees, efficient and accurate matching between multi-view images is achieved. The hierarchical sparse bundle adjustment model, multi-view stereo matching cost volume analysis and other technologies are used to achieve accurate alignment of multi-view images and generation of high-precision depth maps. Through spatiotemporal voxelization, spatiotemporal consistency analysis and fine motion field optimization, dynamic objects are effectively identified and processed to ensure the spatiotemporal consistency of the reconstruction results. The multi-resolution voxel representation, signed distance field optimization and adaptive surface subdivision are adopted to achieve high-precision and high-detail 3D model reconstruction. Based on the human eye stereoscopic vision perception model and adaptive stereo disparity adjustment, combined with dense light field sampling and rendering enhancement technology, high-quality, highly immersive stereoscopic 3D images are generated.

[0151] The present application also provides an electronic device, which includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the 3D image generation method based on multi-camera shooting in the above-mentioned embodiments.

[0152] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the 3D image generation method based on multi-camera shooting.

[0153] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0154] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0155] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A 3D image generation method based on multi-camera shooting, characterized in that: The method comprises: The images collected by multiple cameras are synchronously triggered and collected to obtain multi-view original images; specifically, the method includes: arranging the multiple cameras in an array to obtain a camera array, and configuring a WiFi module for each camera in the camera array to obtain a camera array with a WiFi module; setting a central control unit, establishing a connection with the camera array with a WiFi module through a WiFi network, and constructing a camera system with a central control unit; generating a microsecond trigger signal based on the central control unit to obtain a microsecond synchronization trigger signal, and transmitting the microsecond synchronization trigger signal to each camera through a WiFi network; triggering the camera system with the central control unit based on the transmitted microsecond synchronization trigger signal to obtain a preliminarily synchronized multi-view image; setting a high-speed LED array in the shooting scene, and controlling the high-speed LED array to emit a light pulse of a specific pattern based on the microsecond synchronization trigger signal to obtain a scene with a time mark; shooting the scene with the time mark to obtain a multi-view image with a light pulse mark, and transmitting the multi-view image with a light pulse mark to a cloud APP through a WiFi network; performing sub-microsecond time alignment correction based on the light pulse mark in the multi-view image with the light pulse mark in the cloud APP to obtain a multi-view original image; Preprocessing the multi-view original image to obtain a preprocessed multi-view image; Performing feature extraction and matching on the preprocessed multi-view images to obtain feature matching results; Based on the feature matching results, the preprocessed multi-view images are aligned and depth information is calculated to obtain aligned multi-view images and depth maps; specifically, the method comprises: constructing a multi-view geometric relationship graph based on the feature matching results to obtain an initial view topology structure, and optimizing the initial view topology structure to obtain an optimized view topology structure; establishing a hierarchical sparse bundle adjustment model based on the optimized view topology structure and generating initial camera pose parameters, iteratively optimizing the initial camera pose parameters to minimize the reprojection error, and obtaining target camera pose parameters; optimizing the preprocessed multi-view images based on the .... The processed multi-view images are geometrically transformed to obtain a coarsely aligned multi-view image, and a multi-view stereo matching cost volume analysis is performed on the coarsely aligned multi-view image to obtain an initial depth estimate; multi-view consistency propagation is performed based on the initial depth estimate to obtain an optimized depth map, and the optimized depth map is refined to obtain a high-precision depth map; bilateral joint upsampling is performed on the high-precision depth map and the coarsely aligned multi-view image to obtain a high-resolution depth map; a disparity gradient field is calculated based on the high-resolution depth map, and an affine transformation matrix is ​​constructed based on the disparity gradient field to obtain the coarsely aligned multi-view image. The method comprises the following steps: locally deforming the multi-view image to obtain a finely aligned multi-view image; performing structural tensor analysis on the finely aligned multi-view image to obtain an edge-preserving smooth image, and calculating a multi-scale local phase consistency metric based on the edge-preserving smooth image to obtain a local structural similarity map; constructing a photometric consistency constraint based on the local structural similarity map, and solving a displacement field in combination with a geometric smooth constraint to obtain a precisely aligned multi-view image; performing spatiotemporal voxelization on the precisely aligned multi-view image and the high-resolution depth map to construct a 4D spatiotemporal voxel grid, and calculating the color and depth characteristics of each voxel. Eigenvectors are obtained to obtain a characterized spatiotemporal voxel field; a spatiotemporal graph is constructed based on the characterized spatiotemporal voxel field to obtain a spatiotemporal segmentation result; a connected domain analysis is performed on the spatiotemporal segmentation result to calculate the spatiotemporal consistency score of each connected domain, and the connected domains are clustered according to the spatiotemporal consistency score to obtain a set of dynamic object candidate regions R; based on the set of dynamic object candidate regions, optical flow constraints and depth consistency constraints are constructed, and joint optimization is performed to obtain a fine motion field; according to the fine motion field, the precisely aligned multi-view image and the high-resolution depth map are temporally interpolated and extrapolated to obtain an aligned multi-view image and depth map; Reconstructing a 3D model based on the aligned multi-view images and the depth map to obtain a 3D model of the scene; The scene 3D model is subjected to stereoscopic parallax correction and rendering processing to generate a stereoscopic 3D image.

2. The 3D image generation method based on multi-camera shooting according to claim 1, characterized in that: The preprocessing of the multi-view original image to obtain a preprocessed multi-view image includes: Denoising the multi-view original image to obtain a preliminarily denoised multi-view image, placing a standard color card in the shooting scene, and analyzing the preliminarily denoised multi-view image based on the standard color card to obtain a global color mapping model; Performing joint color correction on the preliminarily denoised multi-view images based on the global color mapping model to obtain color-corrected multi-view images, and performing histogram analysis on the color-corrected multi-view images to obtain pixel value distribution characteristics of each image; Constructing an adaptive exposure balancing function based on the pixel value distribution characteristics, and adjusting the pixel values ​​of the color-corrected multi-view image based on the adaptive exposure balancing function to obtain an exposure-balanced multi-view image; Performing distortion correction of a polynomial model on the multi-view image after exposure equalization to obtain a multi-view image after distortion correction, and performing super-resolution processing on the multi-view image after distortion correction to obtain a high-resolution multi-view image; The high-resolution multi-view image is subjected to edge sharpening and detail enhancement processing to obtain a preprocessed multi-view image.

3. The 3D image generation method based on multi-camera shooting according to claim 1, characterized in that: The extracting and matching the preprocessed multi-view images to obtain feature matching results includes: Extracting feature points from the preprocessed multi-view images to obtain a preliminary feature point set, and constructing an affine invariant feature set based on the preliminary feature point set; Performing global semantic feature extraction on the affine invariant feature set to obtain a semantically enhanced feature set, and constructing a feature fast index tree based on the semantically enhanced feature set; Based on the feature fast index tree, the feature points between the multi-view images are roughly matched to obtain a preliminary matching pair set, and a local relationship graph of the feature points is constructed based on the preliminary matching pair set; Generating an optimized set of matching pairs according to the local relationship graph of feature points, and establishing a global geometric consistency model based on the optimized set of matching pairs; The matching results are refined based on the global geometric consistency model to obtain a set of accurate matching pairs, and multi-view feature fusion and propagation are performed based on the set of accurate matching pairs to obtain a feature matching result.

4. The 3D image generation method based on multi-camera shooting according to claim 1, characterized in that: The method of geometrically transforming the preprocessed multi-view image based on the target camera pose parameters to obtain a roughly aligned multi-view image, and performing multi-view stereo matching cost volume analysis on the roughly aligned multi-view image to obtain an initial depth estimate includes: Based on the target camera pose parameters, the intrinsic matrix E between the cameras is calculated, where E=[t]×R, [t]× is the antisymmetric matrix of the translation vector t, and R is the rotation matrix; epipolar correction is performed on the preprocessed multi-view images to obtain a corrected image pair; Performing a phase consistency transformation on the corrected image pair, calculating a phase correlation function to obtain a sub-pixel displacement field, and calculating an affine transformation matrix based on the sub-pixel displacement field to obtain a roughly aligned multi-view image; Performing anisotropic diffusion filtering on the roughly aligned multi-view images to obtain an edge-preserving smoothed image set, and performing multi-scale Log-Gabor filtering on the smoothed image set to obtain a feature tensor field; Calculating a structure tensor based on the feature tensor field, performing eigenvalue decomposition on the structure tensor to obtain a structure consistency map, and performing non-rigid registration on the coarsely aligned multi-view image set based on the structure consistency map to obtain a finely aligned multi-view image set; Quadruplet segmentation is constructed for the finely aligned multi-view image set to obtain a superpixel set, a multi-view matching cost volume is calculated based on the superpixel set to obtain an initial depth map, and a variational method is applied to solve the initial depth map to obtain an initial depth estimate through iterative optimization.

5. The 3D image generation method based on multi-camera shooting according to claim 1, characterized in that: The step of reconstructing a 3D model based on the aligned multi-view images and the depth map to obtain a 3D model of the scene includes: voxelize the aligned multi-view image and the depth map to obtain an initial low-resolution voxel model, and perform octree data structure analysis based on the initial low-resolution voxel model to obtain a multi-resolution voxel representation; Calculating a signed distance field value for each voxel in the multi-resolution voxel representation to obtain an initial SDF model, and performing multi-view depth fusion based on the initial SDF model to obtain an optimized SDF model; Adaptively subdividing the optimized SDF model to obtain a high-precision surface mesh model, and performing multi-view consistency constraints based on the high-precision surface mesh model to obtain a surface optimization objective function; Iteratively solving the surface optimization objective function to obtain a refined surface model, and extracting a multi-view texture set based on the refined surface model; A global optimization of graph cuts is performed on the multi-view texture set to obtain a seamless texture mapping, and a scene 3D model is generated based on the seamless texture mapping.

6. The 3D image generation method based on multi-camera shooting according to claim 1, characterized in that: The performing stereoscopic parallax correction and rendering processing on the scene 3D model to generate a stereoscopic 3D image includes: Performing depth analysis on the scene 3D model based on a human eye stereoscopic vision perception model to obtain scene depth distribution characteristics, and constructing an adaptive stereo disparity adjustment model according to the scene depth distribution characteristics; Performing disparity adjustment on the scene 3D model based on the adaptive stereo disparity adjustment model to obtain a preliminary stereo disparity correction result, and constructing a disparity continuity optimization model based on the preliminary stereo disparity correction result; Processing the large parallax region by using the parallax continuity optimization model to obtain an optimized stereoscopic parallax correction result, and generating dense light field sampling data based on the optimized stereoscopic parallax correction result; Compressing the dense light field sampling data to obtain a lightweight light field model, and constructing an initial rendering result based on the lightweight light field model; The initial rendering result is subjected to image enhancement to obtain a high-quality rendering image, and spatiotemporal analysis and jitter suppression processing are performed based on the high-quality rendering image to obtain a stereoscopic 3D image.

7. A 3D image generation device based on multi-camera shooting, characterized in that: The device is used to execute the 3D image generation method based on multi-camera shooting according to any one of claims 1 to 6, comprising: An acquisition module is used to synchronously trigger the acquisition of images acquired by multiple cameras to obtain multi-view original images; A preprocessing module, used to preprocess the multi-view original image to obtain a preprocessed multi-view image; A matching module, used for performing feature extraction and matching on the preprocessed multi-view images to obtain feature matching results; A calculation module, configured to align the preprocessed multi-view images and calculate depth information based on the feature matching result to obtain an aligned multi-view image and a depth map; A reconstruction module, used to reconstruct a 3D model based on the aligned multi-view images and the depth map to obtain a scene 3D model; The generation module is used to perform stereoscopic parallax correction and rendering processing on the scene 3D model to generate a stereoscopic 3D image.

Images