A multi-camera topology pre-optimization virtual modeling and performance pre-evaluation method

Abstract

The application discloses a kind of multi-camera topology pre-optimization virtual modeling and performance pre-evaluation method, it is related to holographic communication field, this method is combined with holographic communication demand and model algorithm to be used, realize the digital modeling of multi-camera topology;Design and simulate the arrangement structure of a variety of adaptive scene such as matrix, ring, distributed, layered, clear arrangement basis and limiting condition of each arrangement;Multi-objective optimization algorithm is introduced, and the parameters of the arrangement correction parameter of virtual camera and the model algorithm to be used are iteratively optimized, and the quantitative evaluation system associated with holographic transmission quality is simultaneously established, to form "pre-simulation-optimal topology-test transmission" whole-process closed loop, can effectively improve imaging effect, reduce data volume and optimize delay.

Description

Technical Field

[0001] This invention relates to the field of holographic communication, specifically to a method for virtual modeling and performance pre-evaluation of multi-camera topology pre-optimization. Background Technology

[0002] Holographic communication is a communication method that transmits three-dimensional images in real time using holographic technology. It can recreate the spatial sense and interactivity of a real scene, breaking the limitations of traditional planar communication. Its core principle lies in using light field acquisition technology (multi-camera) to capture the three-dimensional information of objects, including shape, texture, and spatial position. This three-dimensional data is then transmitted in real time to the receiving end via a high-speed transmission network. A holographic display device then reconstructs the stereoscopic image, and combined with sensing technology, enables two-way interaction, making the communicating parties appear to be in the same physical space. Holographic communication can present dynamic three-dimensional images, offering a more realistic and immersive experience than video calls. Theoretically, it supports real-time feedback on body movements and spatial position, providing an interactive experience close to face-to-face communication. In telemedicine, doctors can observe patient lesions in real time through holographic images and guide remote surgical procedures. In social entertainment, virtual concerts and holographic video calls provide users with immersive interactive experiences. In industry and education, holographic communication can present the three-dimensional structure of machinery during remote equipment maintenance and recreate the three-dimensional display of complex models in teaching.

[0003] However, when multiple cameras are used to capture images in the field of holographic communication, the traditional "long snake" arrangement results in poor image quality, lack of quantitative basis for adjustment, and transmission quality that does not reach the optimal performance of the hardware. This can lead to problems such as unsatisfactory imaging effects, data redundancy, and / or large delays. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a virtual modeling and performance pre-evaluation method for pre-optimizing multi-camera topology, which solves the problems of unsatisfactory imaging results, data redundancy, and / or significant latency caused by the lack of a quantitative evaluation system for existing multi-camera systems.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows: A method for pre-optimization of virtual modeling and performance pre-evaluation of multi-camera topology is provided, which includes the following steps: Data from a multi-camera system is collected, and a mapping relationship between the data and topology performance of the multi-camera system is established to obtain a parameter-performance correlation dataset for simulation. Based on the parameter-performance correlation dataset used for simulation, a digital model of the holographic communication target scene and a multi-camera virtual array are constructed on the simulation platform. Based on the digital model of the target scene of holographic communication and the multi-camera virtual array, various camera arrangement structures adapted to the needs of holographic communication are simulated to obtain the parallax coverage, blind zone distribution and transmission link bottleneck of each camera arrangement structure. With the objective function of minimizing the holographic blind zone rate, maximizing the peak signal-to-noise ratio of holographic reconstruction, and adapting to the requirements of the proposed model algorithm for multi-camera systems, the arrangement correction parameters of the virtual camera and the parameters of the proposed model algorithm are iteratively optimized to output a camera arrangement structure that satisfies the hardware constraints, thus obtaining the candidate topology scheme. Determine whether the candidate topology scheme meets the requirements of holographic communication. If so, select the candidate topology scheme as the optimal topology scheme; otherwise, recalculate the candidate topology scheme until the selected topology scheme meets the requirements of holographic communication.

[0006] The beneficial effects of this invention are as follows: Before deploying real cameras, this method combines holographic communication requirements with the proposed model algorithm to achieve digital modeling of multi-camera topology; it designs and simulates various adaptive arrangement structures such as matrix, ring, distributed, and hierarchical, clarifying the arrangement basis and constraints of each arrangement; it introduces a multi-objective optimization algorithm to iteratively optimize the arrangement correction parameters of virtual cameras and the parameters of the proposed model algorithm, and simultaneously establishes a quantitative evaluation system for holographic transmission quality correlation, forming a closed loop of the entire process of "pre-simulation - topology optimization - transmission verification", which can effectively improve imaging effect, reduce data volume and optimize latency. Attached Figure Description

[0007] Figure 1 This is a flowchart illustrating the method. Figure 2 This is a schematic diagram of a holographic communication scenario as an example. Detailed Implementation

[0008] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0009] like Figure 1 As shown, the multi-camera topology pre-optimization virtual modeling and performance pre-evaluation method includes the following steps: S1. Collect data from the multi-camera system, establish the mapping relationship between the data and topology performance of the multi-camera system, and obtain the parameter-performance correlation dataset for simulation. S2. Based on the parameter-performance correlation dataset used for simulation, a digital model of the holographic communication target scene and a multi-camera virtual array are constructed on the simulation platform. S3. Based on the digital model of the target scene of holographic communication and the multi-camera virtual array, simulate various camera arrangement structures that are adapted to the needs of holographic communication, and obtain the parallax coverage, blind zone distribution and transmission link bottleneck of each camera arrangement structure. S4. With the objective function of minimizing the holographic blind zone rate, maximizing the peak signal-to-noise ratio of holographic reconstruction, and adapting to the requirements of the proposed model algorithm for multi-camera systems, the arrangement correction parameters of the virtual camera and the parameters of the proposed model algorithm are iteratively optimized to output the camera arrangement structure that satisfies the hardware constraints, thus obtaining the candidate topology scheme. S5. Determine whether the candidate topology scheme meets the requirements of holographic communication. If so, select the candidate topology scheme as the optimal topology scheme; otherwise, recalculate the candidate topology scheme until the selected topology scheme meets the requirements of holographic communication.

[0010] In some embodiments, the data of the multi-camera system includes the hardware parameters of the multi-camera system, spatial layout parameters, proposed model algorithm parameters, and holographic communication transmission quality parameters. Hardware parameters include: (1) Camera optical parameters (focal length f, sensor size Ws×Hs, aperture value F, ISO sensitivity I), camera parameter type (wide-angle / telephoto / fisheye, color / black and white) and holographic communication hardware parameters (transmission bandwidth, reconstruction computing power).

[0011] (2) Imaging performance parameters: including resolution Res (number of pixels M×N), distortion coefficients (radial k1, k2, k3, tangential p1, p2), and holographic depth of field D. DOF (Foreground / Background blur threshold), In / Out screen parallax deviation Δd in-out .

[0012] Spatial layout parameters include: the camera's position in the world coordinate system O. W X W Y W Z W The position vector t = [XW, YW, ZW] in the vector. T Attitude parameters (yaw angle ψ, pitch angle θ, roll angle) ), arrangement correction parameters (adjacent camera spacing correction Δb, inter-layer height correction Δh, ring radius correction Δr), field of view overlap rate O, blind zone rate S.

[0013] The proposed model algorithm parameters include: configured according to the algorithm type, such as 3D Gaussian (point cloud density threshold ρmin, Gaussian kernel size σ), NeRF (number of feature point matches Nfeat, ray sampling density Dray), SLAM (map point reprojection error REslam, loop closure detection accuracy). loop); Holographic communication transmission quality parameters include: holographic reconstruction PSNR, transmission bit rate R, and parallax data synchronization delay τ.

[0014] The mapping relationship between hardware parameters and topology performance includes "optical parameters → field of view → overlap rate / reconstruction accuracy" and "communication parameters → code rate adaptability". The mapping relationship between spatial layout parameters and topology performance includes "pose / correction amount → field of view frustum modeling → blind zone rate / overlap rate"; The mapping relationship between the proposed model algorithm parameters and topology performance includes "number of feature points / reprojection error → algorithm adaptability"; The mapping relationship between holographic communication transmission quality parameters and topology performance includes "bit rate / delay → transmission adaptability".

[0015] In some embodiments, the specific method for constructing a digital model of a holographic communication target scene and a multi-camera virtual array on a simulation platform includes: The standard human activity range of the holographic conference scene model was imported into Blender for 3D scene modeling; a matrix, ring, distributed, and hierarchical multi-camera virtual array was built using Python + Open3D, and the field of view coverage and blind spots of each virtual array were visualized; virtual camera images, holographic reconstructed point clouds, and disparity maps were output based on the NeRF algorithm and the 3D Gaussian algorithm; the transmission simulation used the NS-3 simulation tool to simulate holographic data transmission, calculate the bit rate R and latency τ, verify the adaptability with the hardware bandwidth Bmax, and visualize the transmission link bottleneck.

[0016] In some embodiments, various camera arrangement structures adapted to holographic communication requirements include: Matrix arrangement: The number of rows and columns of cameras is designed based on the coverage width and height of the holographic communication target; the spacing between adjacent cameras is determined based on the field of view of a single camera to ensure the overlap rate of the field of view, ensuring that the matrix field of view completely covers the target area and the overlap rate is greater than or equal to 50%; the arrangement is subject to constraints including avoiding the display space of the monitor and adapting the overall area occupied by the multi-camera matrix to the scene installation space. Optionally, the geometric constraint model for the matrix arrangement is as follows: Let the matrix have m rows and n columns, and the single-camera horizontal field of view (FOV) be... W Vertical field of view adjacent FOV H， The target coverage width Wtar and height Htar are determined by the formula: L 列 = (Wtar) / (m × tan(FOV)) W / 2)×2) The distance L between adjacent cameras in the column direction is calculated using the formula: L 行 = (Wtar) / (m × tan(FOV))H / 2)×2) Calculate the spacing L rows between adjacent cameras in the row and column directions to ensure that the matrix field of view completely covers the target area and the overlap rate is ≥50%.

[0017] Ring arrangement: The ring radius is determined based on the 3D bounding box of the target object, and the number of cameras is configured according to the requirement of 360° field of view without gaps. The constraints of the arrangement include that the center of the ring coincides with the center of the target object's activity and that the camera installation height avoids obstacles in the scene. Optionally, the geometric constraint model for the ring arrangement is as follows: Let the ring radius be R, the horizontal field of view (FOVW) of a single camera, and the number of cameras be k. The reference value of the ring radius (Wobj is the maximum horizontal dimension of the object being acquired) is calculated using the formula R=(Wobj) / (2×sin (π / k)). Then, it is corrected to R'=R±Δr by combining the scene installation space. At the same time, the minimum number of cameras is calculated using the formula k≥360° / FOVW to ensure gapless coverage of the 360° field of view.

[0018] Distributed arrangement: Cameras are arranged according to the distribution of blind spots in the scene. In addition to the coverage of the main field of view, supplementary cameras are set in the blind spot positions. The arrangement is based on the three-dimensional coordinates and area of ​​the blind spot. The field of view of the supplementary cameras covers the blind spot and the overlap rate with the field of view of the main camera is greater than or equal to 40%. The constraints include that the installation position of the supplementary cameras utilizes the existing structure of the scene, does not occupy the installation and operation space of the display, and the data transmission link is compatible with the main camera. Optionally, the geometric constraint model for the distributed arrangement is as follows: Let the center coordinates of the blind spot be (Xb, Yb, Zb), and the position vector of the blind spot-filling camera be tb = [Xb + ΔX, Yb + ΔY, Zb + ΔZ], where ΔX, ΔY, and ΔZ are determined according to the scene structure using the formula FOV. 盲区 =2×arctan (Sb / (2×D)) Calculate the minimum field of view required by the blind spot camera (Sb is the maximum size of the blind spot, and D is the distance from the blind spot camera to the blind spot) to ensure complete coverage of the blind spot.

[0019] Layered arrangement: Based on the multi-depth-of-field requirements of holographic communication, it is divided into 2-3 layers. Each layer of cameras focuses on the corresponding depth-of-field area. When divided into 3 layers, the upper layer camera covers the background, the middle layer camera covers the main subject, and the lower layer camera covers the foreground details. When divided into 2 layers, the upper layer camera covers the background and the main subject, and the lower layer camera covers the foreground details. The arrangement is based on the distance range of each depth-of-field area. The restriction is that the fields of view of each layer of cameras cannot cross and block the display area, and the layer spacing ensures that the images of each depth of field are stitched together without any gaps. Optionally, the geometric constraint model of the layered arrangement is as follows: assuming the target depth of field distance range of each layer is [D1,D2], the focal length of the camera of the corresponding layer is calculated by the formula f=Ws×D / (Res×P) (P is the pixel size), the height difference Δh between layers is determined according to the depth of field range of each layer, and at the same time, the distance between the field of view of each layer and the display area of ​​the monitor is ≥0.5m.

[0020] Methods for visualizing transmission link bottlenecks through virtual simulation of the above-mentioned various arrangement structures include: The transmission bitrate and synchronization delay of each camera are calculated using the NS-3 simulation tool and compared with the maximum bandwidth and maximum allowable delay of the hardware to obtain the degree of bottleneck in the transmission link. In the simulation scene built with Python and Open3D, the bottleneck degree of the transmission link is marked with a color gradient, with green indicating no bottleneck, yellow indicating a slight bottleneck, and red indicating a severe bottleneck. The specific transmission bitrate and synchronization delay values ​​are also marked.

[0021] In some embodiments, hardware constraints include: Transmission bandwidth Bmax limit: Total transmission bit rate R of multiple cameras ≤ 0.8Bmax, where Bmax is the maximum transmission bandwidth; The proposed algorithm type based on the computational power constraints is as follows: In high-computational-power scenarios, the NeRF algorithm ray sampling density (Dray) ≥ 500 sampling points / ray; in low-computational-power scenarios, the 3D Gaussian algorithm point cloud density threshold (ρmin) ≥ 100 points / m³; and the single-camera imaging data volume ≤ Res × FPS × 8 bits / frame, where Res is the camera hardware resolution and FPS represents frames per second. High-computational-power scenarios refer to scenarios with hardware computing power ≥ 10 TFLOPS or equipped with an NVIDIA RTX 3090 or higher GPU; low-computational-power scenarios refer to scenarios with hardware computing power < 10 TFLOPS or equipped with an NVIDIA RTX 2060 or lower GPU. The arrangement correction parameters include the spacing correction for matrix arrangement, the radius correction for circular arrangement, the position correction for distributed arrangement, and the height correction for hierarchical arrangement. The proposed model algorithms include 3D Gaussian algorithm, NeRF algorithm and SLAM algorithm; The parameters of the proposed model algorithm include the number of feature points matched by the NeRF algorithm (Nfeat), the 3D Gaussian kernel size (σ), and the SLAM map point reprojection error (REslam).

[0022] In some embodiments, the specific methods for performing iterative optimization include: A multi-objective optimization algorithm is introduced, with the objective functions of minimizing the holographic blind zone rate, maximizing the peak signal-to-noise ratio of holographic reconstruction, and adapting to the requirements of the proposed model algorithm for multi-camera systems. The particles of the multi-objective optimization algorithm are initialized, and hardware constraints are converted into parameter boundaries. The arrangement correction parameters of the virtual cameras and the parameters of the proposed model algorithm are iteratively optimized. When parameters exceed hardware capabilities during iterative optimization, the fitness value of the corresponding particles is reduced. When the average comprehensive performance of 10 consecutive generations of particles is greater than 0.8 and all parameters satisfy the hardware constraints, the multi-objective optimization algorithm is considered to have converged, and the corresponding camera arrangement structure is output, thus obtaining the candidate topology scheme.

[0023] Optionally, the multi-objective optimization algorithm is an improved particle swarm optimization algorithm, and its iterative optimization methods include: Set the adaptation factor for each camera arrangement structure; the adaptation factor for matrix arrangement structure is 1.3; the adaptation factor for ring arrangement structure is 1.5; the adaptation factor for distributed arrangement structure is 1.8; and the adaptation factor for hierarchical arrangement structure is 1.2. The larger the adaptation factor, the greater the weight of the parameters that need to be optimized in that arrangement structure. Each set of topological parameters is integrated into particle attributes, and each particle is bound to an adaptation factor corresponding to the arrangement structure. A fitness evaluation function with an adaptation factor is constructed, using comprehensive performance as the core indicator. The adaptation factor is introduced to adjust the optimization weights of the permutation correction parameters to achieve iterative optimization. The expression of the fitness evaluation function is as follows: F = α × P + + (1-α )×C Where F is the fitness value; α is the fitness factor of the permutation correction parameter; and P is the overall performance. is the adaptation weight of the proposed model algorithm parameters; D is the adaptation score of the proposed model algorithm parameters, D = 1 when adapted, and D = 0 when not adapted; C is the compliance score of the arrangement correction parameters, C = 1 when compliant, and C = 0 when non-compliant.

[0024] The algorithm optimizes the arrangement correction parameters by strengthening different arrangement structures with α. Finally, it performs differentiated iterative updates and convergence judgment. During the iteration, the parameter update step size is adjusted according to the α value for different arrangement particle populations, and the blind zone rate S of the directional search is controlled within a low range. When the comprehensive performance evaluation value P of 10 consecutive generations of particles meets the requirements, the algorithm is judged to have converged and a candidate topology scheme is output. The theoretical maximum value of P is 1, which represents that all indicators have reached the optimal state.

[0025] Optionally, the formula for calculating the overall performance is:

[0026] Where P represents overall performance; Reconstructing weights for holographic imaging; Peak signal-to-noise ratio; The maximum peak signal-to-noise ratio that the overall hardware performance can support; Optimize weights for blind spots; Blind spot rate; To adapt the weights to the proposed model algorithm; This represents the number of feature points in the NeRF algorithm. This represents the maximum number of feature points required by the NeRF algorithm. Weights are adapted for transmission.

[0027] Based on the above process, a quantitative performance pre-evaluation system for holographic transmission quality correlation can be established, and then the indicators under the quantitative performance pre-evaluation system can be calculated: holographic reconstruction accuracy, ingress / exgress parallax deviation, adaptability of the proposed model algorithm, scene adaptability, and transmission bit rate adaptability of the candidate topology scheme. These indicators can be used to verify whether the candidate topology scheme meets the requirements of holographic communication; among which: The formula for calculating the accuracy of holographic reconstruction is:

[0028] in is the root mean square error of holographic reconstruction, which is also the holographic reconstruction accuracy; N is the total number of holographic point clouds involved in the calculation; For the first The three-dimensional coordinates of a sample point in the holographic point cloud reconstructed by virtual simulation; For the first The actual 3D coordinates of each sample point in the real scene; when A distance of 0.05 meters or more does not meet the requirements for holographic communication; The expression for calculating the parallax deviation between the ingress and egress screens is:

[0029] in Parallax deviation between input and output screens; The actual ingress and egress parallax is calculated using candidate topology schemes; For holographic comfort parallax; when A value greater than 0.003 does not meet the requirements for holographic communication; The formula for calculating the suitability of the proposed model algorithm is as follows:

[0030] in The feature point matching rate of the NeRF algorithm is also known as the algorithmic suitability of the proposed model. This represents the number of feature points successfully matched by the NeRF algorithm in multi-camera images. The total number of feature points extracted from multi-camera images by the NeRF algorithm; when A value less than 80% does not meet the requirements for holographic communication; When the camera occupies the display space or exceeds the installation range of the scene, the scene adaptability does not meet the requirements of holographic communication. The expression for calculating transmission rate adaptability is: η = Bmax / R in η For transmission rate adaptation; Bmax R is the maximum transmission bandwidth; R is the total transmission bit rate of multiple cameras; when η A value greater than 0.8 does not meet the requirements for holographic communication.

[0031] In one embodiment of the present invention, a system based on the multi-camera topology pre-optimization virtual modeling and performance pre-evaluation method is also provided. This system achieves a closed-loop process through multi-mode collaborative work and specifically includes five core functional modules: The holographic parameter acquisition and modeling module, as the data input and basic modeling unit of the system, is responsible for receiving the hardware parameters, spatial layout parameters, proposed model algorithm parameters, and holographic communication transmission quality parameters of the multi-camera system, forming a four-dimensional parameter coupling model of "hardware-space-algorithm-transmission". After standardizing various parameters, it forms a parameter library that can be directly used for simulation, that is, a parameter-performance correlation dataset for simulation.

[0032] The arrangement structure simulation module receives parameter library data and uses 3D modeling technology to build a virtual camera array that is proportional to the real scene. Based on the geometric constraint model of each arrangement structure, it calculates the blind zone rate and compliance. It uses visualization technology to distinguish and display the field of view coverage area, blind zone, and violation area. Finally, it generates an initial topology defect report containing the location and range of topology defects of each arrangement structure and preliminary optimization suggestions.

[0033] The topology optimization module, as the core computing unit of the system, integrates the improved particle swarm optimization algorithm and the arrangement structure adaptation algorithm. Guided by the problems in the defect report, it performs multiple rounds of iterative optimization on the pose parameters, arrangement correction parameters and optical parameters of virtual cameras with different arrangement structures, gradually improving the overall performance of the topology scheme and outputting one or more candidate topology schemes.

[0034] The holographic performance evaluation module, based on the constructed quantitative evaluation system, comprehensively calculates the core indicators of candidate topology schemes such as holographic reconstruction PSNR, parallax deviation, and transmission efficiency, and generates an evaluation report that includes the indicator compliance status and performance bottleneck analysis. When the indicator fails to reach the preset threshold, the topology optimization module is triggered to restart the iteration process through the data interaction mechanism between modules.

[0035] The solution output module, as the system's result presentation unit, integrates the optimal topology parameters, holographic simulation report, and hardware deployment guidance documents after all evaluation indicators have met the standards, forming a complete solution package that can directly guide physical deployment.

[0036] The permutation structure adaptation algorithm includes an "algorithm adaptation submodule": For the 3D Gaussian algorithm, it automatically configures the point cloud density ρmin and Gaussian kernel σ, and adjusts the camera resolution accordingly; for the NeRF algorithm, it automatically sets the feature point matching threshold. Adjust the camera overlap rate O in conjunction with the camera. 3D ≥60%; For SLAM algorithm: Automatically configure map point reprojection error (RE) slam <0.8 pixels, adjust the camera frame rate FPS=30 accordingly.

[0037] In the specific implementation process, in the arrangement structure simulation module, when the calculated blind zone rate is greater than 10%, it is automatically marked as "needs blind zone optimization" and an initial recommended value for the arrangement correction parameters is generated based on the current arrangement type. When diagnosing scene compliance, by comparing the camera position with the display space coordinates, if the camera coordinates fall into the area, it is marked as "occupying display space" and a position offset is recommended. When diagnosing algorithm adaptability, when the feature point matching rate γ of the NeRF algorithm is less than 60%, it is automatically recommended to reduce the camera field of view overlap rate to O. 3D The optimization direction is to increase the accuracy from the initial 50% to 65% by enhancing the overlap of the field of view to improve the feature point matching effect.

[0038] In one embodiment of this method, such as Figure 2 As shown, in a holographic conferencing market scenario for a certain enterprise, the physical dimensions of the conference room are 10m × 8m × 3m (length × width × height). A distributed "five main and four supplementary" camera topology needs to be deployed to achieve 1:1 life-size holographic reconstruction, with the hardware constraint being the maximum transmission bandwidth. The 3D Gaussian algorithm was selected as the proposed model algorithm, with a preset performance threshold of: reconstruction accuracy. Pixel count, blind spot rate ≤10%, transmission rate adaptation coefficient Single-frame processing latency ≤ 80ms.

[0039] First, a virtual scene is constructed based on the "hardware-space-algorithm-transmission" four-dimensional parametric coupling model of this invention. Initial topology parameters are input: the initial coordinates of the five main cameras are as follows: , , , , ,focal length The field of view is 60°, the initial resolution is 1920×1080, and the frame rate (FPS) is 30; the initial coordinates of the four blind spot cameras are... , , , The parameters are consistent with those of the main camera; the initial algorithm is a 3D Gaussian algorithm with a point cloud density threshold. Gaussian kernel .

[0040] Initial performance metrics, including reconstruction accuracy, are calculated using a pre-evaluation system. = 0.62 pixels (exceeding the threshold), blind zone rate 14.3% (concentrated in the corner area of ​​the conference room, exceeding the threshold), single camera single frame data volume = 1920×1080×3×8bit = 49.76Mbps, total transmission bit rate R = 9.5Gbps, adaptation coefficient (Exceeding the threshold) Single frame processing latency of 92ms (exceeding the threshold) is determined to be "Initial topology does not meet usage requirements", triggering the improved particle swarm algorithm optimization process.

[0041] Next, the particle swarm optimization algorithm was improved through iterative optimization. First, the algorithm parameters were initialized: the number of particles m = 50, the maximum number of iterations K = 100, and the learning factor... , arrangement of fit factors Algorithm parameter adaptation weights Inertial weight Adaptive adjustment based on iteration stage (number of iterations) hour, hour, Then calculate the fitness function: according to the formula. Calculation, where (Comprehensive performance normalized score), D=1 (3D Gaussian algorithm parameters are within a reasonable range), C=1 (initial topology does not occupy display area, compliant), initial population optimal fitness value Then, particle iterative updates are performed according to the velocity formula. and position formula When the iteration reaches the 30th iteration, the parameters corresponding to the optimal particle in the population are: the distance between the main cameras is adjusted to 1.2m, and the blind spot camera is offset horizontally by 0.3m. ,at this time 82, = 0.48 pixels, blind spot rate 9.8%, = 0.81, only the transmission adaptation coefficient slightly exceeds the threshold; after 65 iterations, the algorithm converges, and the final optimal parameters are adjusted to the main camera coordinates. , , , , The coordinates of the blind spot camera were adjusted to , , , , .in This indicates the generation of a uniformly distributed random number within the interval (0, 1).

[0042] Finally, after optimization, performance verification was performed. The optimal topology parameters and algorithm parameters were input into the pre-evaluation system to obtain the final performance metric: reconstruction accuracy. = 0.38 pixels (meets the threshold), blind zone rate reduced to 7.1% (no obvious blind zone), total transmission bit rate R = 7.8Gbps, adaptation coefficient = 0.78 (meets the condition R≤0.8) (Constraints) The single-frame processing latency is 72ms (meeting real-time requirements). Simultaneously, virtual simulation verifies that the camera's field of view is unobstructed, the imaging data stitching is seamless, and the 3D Gaussian algorithm reconstructs complete human holographic images with facial features and limb movements restored with ≥95% accuracy, fully meeting the immersive communication needs of a holographic conference room. This fully demonstrates the effectiveness and usability of the pre-optimization modeling and performance evaluation method of this invention.

[0043] In summary, this invention constructs a four-dimensional parameter coupling model of "hardware-space-algorithm-transmission" to achieve digital modeling of multi-camera topology before real camera deployment, combining holographic communication requirements with the proposed model algorithm. It designs and simulates various adaptable arrangement structures such as matrix, ring, distributed, and hierarchical structures, clarifying the arrangement basis and constraints of each arrangement. A multi-objective optimization algorithm is introduced to iteratively optimize the arrangement correction parameters of the virtual cameras and the parameters of the proposed model algorithm, simultaneously establishing a quantitative evaluation system for holographic transmission quality correlation. This forms a closed-loop process of "pre-simulation-topology optimization-transmission verification," effectively improving imaging quality, reducing data volume, and optimizing latency.

Claims

1. A method for virtual modeling and performance pre-evaluation of multi-camera topology pre-optimization, characterized in that, Includes the following steps: Data from a multi-camera system is collected, and a mapping relationship between the data and topology performance of the multi-camera system is established to obtain a parameter-performance correlation dataset for simulation. Based on the parameter-performance correlation dataset used for simulation, a digital model of the holographic communication target scene and a multi-camera virtual array are constructed on the simulation platform. Based on the digital model of the target scene of holographic communication and the multi-camera virtual array, various camera arrangement structures adapted to the needs of holographic communication are simulated to obtain the parallax coverage, blind zone distribution and transmission link bottleneck of each camera arrangement structure. With the objective function of minimizing the holographic blind zone rate, maximizing the peak signal-to-noise ratio of holographic reconstruction, and adapting to the requirements of the proposed model algorithm for multi-camera systems, the arrangement correction parameters of the virtual camera and the parameters of the proposed model algorithm are iteratively optimized to output a camera arrangement structure that satisfies the hardware constraints, thus obtaining the candidate topology scheme. Determine whether the candidate topology scheme meets the requirements of holographic communication. If so, select the candidate topology scheme as the optimal topology scheme; otherwise, recalculate the candidate topology scheme until the selected topology scheme meets the requirements of holographic communication.

2. The method for pre-optimization of multi-camera topology virtual modeling and performance pre-evaluation according to claim 1, characterized in that, The data for the multi-camera system includes the hardware parameters, spatial layout parameters, proposed model algorithm parameters, and holographic communication transmission quality parameters of the multi-camera system. The mapping relationship between hardware parameters and topology performance includes "optical parameters → field of view → overlap rate / reconstruction accuracy" and "communication parameters → code rate adaptability". The mapping relationship between spatial layout parameters and topology performance includes "pose / correction amount → field of view frustum modeling → blind zone rate / overlap rate"; The mapping relationship between the proposed model algorithm parameters and topology performance includes "number of feature points / reprojection error → algorithm adaptability"; The mapping relationship between holographic communication transmission quality parameters and topology performance includes "bit rate / delay → transmission adaptability".

3. The method for pre-optimization of multi-camera topology virtual modeling and performance pre-evaluation according to claim 1, characterized in that, Specific methods for constructing digital models and multi-camera virtual arrays of holographic communication target scenes on simulation platforms include: The standard human activity range of the holographic conference scene model was imported into Blender for 3D scene modeling; a multi-camera virtual array was built using Python + Open3D to visualize the field of view coverage and blind spots of each virtual array; virtual camera images, holographic reconstructed point clouds and disparity maps were output based on the NeRF algorithm and 3D Gaussian algorithm; the transmission simulation used the NS-3 simulation tool to simulate holographic data transmission and visualize the transmission link bottlenecks.

4. The method for pre-optimization of multi-camera topology virtual modeling and performance pre-evaluation according to claim 3, characterized in that, Various camera arrangement structures adapted to holographic communication requirements include: Matrix arrangement: The number of rows and columns of cameras is designed based on the coverage width and height of the holographic communication target; the spacing between adjacent cameras is determined based on the field of view of a single camera to ensure the overlap rate of the field of view, ensuring that the matrix field of view completely covers the target area and the overlap rate is greater than or equal to 50%; the arrangement is subject to constraints including avoiding the display space of the monitor and adapting the overall area occupied by the multi-camera matrix to the scene installation space. Ring arrangement: The ring radius is determined based on the 3D bounding box of the target object, and the number of cameras is configured according to the requirement of 360° field of view without gaps. The constraints of the arrangement include that the center of the ring coincides with the center of the target object's activity and that the camera installation height avoids obstacles in the scene. Distributed arrangement: Cameras are arranged according to the distribution of blind spots in the scene. In addition to the coverage of the main field of view, supplementary cameras are set in the blind spot positions. The arrangement is based on the three-dimensional coordinates and area of ​​the blind spot. The field of view of the supplementary cameras covers the blind spot and the overlap rate with the field of view of the main camera is greater than or equal to 40%. The constraints include that the installation position of the supplementary cameras utilizes the existing structure of the scene, does not occupy the installation and operation space of the display, and the data transmission link is compatible with the main camera. Layered arrangement: Based on the multi-depth-of-field requirements of holographic communication, it is divided into 2-3 layers. Each layer of cameras focuses on the corresponding depth-of-field area. When divided into 3 layers, the upper layer camera covers the background, the middle layer camera covers the main subject, and the lower layer camera covers the foreground details. When divided into 2 layers, the upper layer camera covers the background and the main subject, and the lower layer camera covers the foreground details. The arrangement is based on the distance range of each depth-of-field area. The restriction is that the field of view of each layer of cameras cannot cross and block the display area, and the layer spacing ensures that the images of each depth of field are stitched together without gaps.

5. The method for pre-optimization of multi-camera topology virtual modeling and performance pre-evaluation according to claim 3, characterized in that, Methods for visualizing transmission link bottlenecks include: The transmission bitrate and synchronization delay of each camera are calculated using the NS-3 simulation tool, and compared with the maximum bandwidth and maximum allowable delay of the hardware to obtain the degree of bottleneck in the transmission link. In the constructed simulation scenario, the bottleneck degree of the transmission link is marked with a color gradient, with green indicating no bottleneck, yellow indicating a slight bottleneck, and red indicating a severe bottleneck. The specific transmission bitrate and synchronization delay values ​​are also marked.

6. The method for pre-optimization of multi-camera topology virtual modeling and performance pre-evaluation according to claim 4, characterized in that, Hardware constraints include: The total transmission bit rate of multiple cameras is R ≤ 0.8Bmax, where Bmax is the maximum transmission bandwidth; In high-computing-power scenarios, the NeRF algorithm's ray sampling density (Dray) is ≥ 500 sampling points / ray; in low-computing-power scenarios, the 3D Gaussian algorithm's point cloud density threshold (ρmin) is ≥ 100 points / m³; and the single-camera imaging data volume is ≤ Res × FPS × 8 bits / frame, where Res is the camera's hardware resolution and FPS represents frames per second. High-computing-power scenarios refer to scenarios with hardware computing power ≥ 10 TFLOPS or equipped with an NVIDIA RTX 3090 or higher GPU; low-computing-power scenarios refer to scenarios with hardware computing power < 10 TFLOPS or equipped with an NVIDIA RTX 2060 or lower GPU. The arrangement correction parameters include the spacing correction for matrix arrangement, the radius correction for circular arrangement, the position correction for distributed arrangement, and the height correction for hierarchical arrangement. The proposed model algorithms include 3D Gaussian algorithm, NeRF algorithm and SLAM algorithm; The parameters of the proposed model algorithm include the number of feature points matched by the NeRF algorithm (Nfeat), the 3D Gaussian kernel size (σ), and the SLAM map point reprojection error (REslam).

7. The camera topology pre-optimization virtual modeling and performance pre-evaluation method according to claim 6, characterized in that, Specific methods for iterative optimization include: A multi-objective optimization algorithm is introduced, with the objective functions of minimizing the holographic blind zone rate, maximizing the peak signal-to-noise ratio of holographic reconstruction, and adapting to the requirements of the proposed model algorithm for multi-camera systems. The particles of the multi-objective optimization algorithm are initialized, and hardware constraints are converted into parameter boundaries. The arrangement correction parameters of the virtual cameras and the parameters of the proposed model algorithm are iteratively optimized. When parameters exceed hardware capabilities during iterative optimization, the fitness value of the corresponding particles is reduced. When the average comprehensive performance of 10 consecutive generations of particles is greater than 0.8 and all parameters satisfy the hardware constraints, the multi-objective optimization algorithm is considered to have converged, and the corresponding camera arrangement structure is output, thus obtaining the candidate topology scheme.

8. The camera topology pre-optimization virtual modeling and performance pre-evaluation method according to claim 7, characterized in that, Multi-objective optimization algorithms are improvements on particle swarm optimization algorithms. Their iterative optimization methods include: Set the adaptation factor for each camera arrangement structure; Each set of topological parameters is integrated into particle attributes, and each particle is bound to an adaptation factor corresponding to the arrangement structure. A fitness evaluation function with an adaptation factor is constructed, using comprehensive performance as the core indicator. The adaptation factor is introduced to adjust the optimization weights of the permutation correction parameters and the parameters of the proposed simulation algorithm, respectively, to achieve iterative optimization. The expression of the fitness evaluation function is as follows: F=α×P + + (1-a )×C Where F is the fitness value; α is the fitness factor of the permutation correction parameter; and P is the overall performance. is the adaptation weight of the proposed model algorithm parameters; D is the adaptation score of the proposed model algorithm parameters, D = 1 when adapted, and D = 0 when not adapted; C is the compliance score of the arrangement correction parameters, C = 1 when compliant, and C = 0 when non-compliant.

9. The camera topology pre-optimization virtual modeling and performance pre-evaluation method according to claim 7 or 8, characterized in that, The formula for calculating overall performance is: Where P represents overall performance; Reconstructing weights for holographic imaging; Peak signal-to-noise ratio; The maximum peak signal-to-noise ratio that the overall hardware performance can support; Optimize weights for blind spots; Blind spot rate; To adapt the weights to the proposed model algorithm; This represents the number of feature points in the NeRF algorithm. This represents the maximum number of feature points required by the NeRF algorithm. Weights are adapted for transmission.

10. The camera topology pre-optimization virtual modeling and performance pre-evaluation method according to claim 1, characterized in that, Specific methods for determining whether a candidate topology scheme meets the requirements of holographic communication include: By calculating the holographic reconstruction accuracy, in-screen and out-of-screen parallax deviation, suitability of the proposed model algorithm, scene suitability, and transmission bit rate suitability of the candidate topology schemes, it is possible to verify whether the candidate topology schemes meet the requirements of holographic communication; among which: The formula for calculating the accuracy of holographic reconstruction is: in is the root mean square error of holographic reconstruction, which is also the holographic reconstruction accuracy; N is the total number of holographic point clouds involved in the calculation; For the first The three-dimensional coordinates of a sample point in the holographic point cloud reconstructed by virtual simulation; For the first The actual 3D coordinates of each sample point in the real scene; when A distance of 0.05 meters or more does not meet the requirements for holographic communication; The expression for calculating the parallax deviation between the ingress and egress screens is: in Parallax deviation between input and output screens; The actual ingress and egress parallax is calculated using candidate topology schemes; For holographic comfort parallax; when A value greater than 0.003 does not meet the requirements for holographic communication; The formula for calculating the suitability of the proposed model algorithm is as follows: in The feature point matching rate of the NeRF algorithm is also known as the algorithmic suitability of the proposed model. This represents the number of feature points successfully matched by the NeRF algorithm in multi-camera images. The total number of feature points extracted from multi-camera images by the NeRF algorithm; when A value less than 80% does not meet the requirements for holographic communication; When the camera occupies the display space or exceeds the installation range of the scene, the scene adaptability does not meet the requirements of holographic communication. The expression for calculating transmission rate adaptability is: η = Bmax / R in η For transmission rate adaptation; Bmax R is the maximum transmission bandwidth; R is the total transmission bit rate of multiple cameras; when η A value greater than 0.8 does not meet the requirements for holographic communication.

Images