Multi-path parallax generation cooperative image generation method and apparatus

Abstract

The application relates to a multi-path disparity generation cooperative image generation method and device, which comprises the following steps: acquiring a multi-view target video; performing motion analysis and feature extraction on the target video to determine a sub-pixel edge consistency field of a high-speed motion target; inputting the sub-pixel edge consistency field and a brightness frame of the target video into a multi-disparity generation network, performing feature fusion and disparity calculation to output a preliminary disparity tensor; performing pixel-level correction on the preliminary disparity tensor based on a generated three-dimensional deformation kernel to obtain a fine disparity array; and performing pixel-level registration and synthesis on original images in the target video based on the fine disparity array to output a multi-view aligned target image. The application improves the accuracy of cross-view image generation of a high-speed motion target.

Description

Technical Field

[0001] This application relates to the field of image technology, and in particular to an image generation method and apparatus for multi-path parallax generation collaboration. Background Technology

[0002] In fields such as live sports broadcasts, high-speed industrial inspection, and environmental perception for autonomous driving, multi-view collaborative image generation technology has become a core support. It can provide comprehensive visual information for target tracking, 3D reconstruction, and accurate analysis by integrating data captured by different cameras.

[0003] Existing publicly available technologies typically employ a sequential process of motion analysis, depth reconstruction, and multi-view rendering to meet such requirements. The core implementation path of existing technical solutions is as follows: First, pixel-level displacement information is extracted between consecutive video frames using optical flow estimation algorithms (such as sparse or dense optical flow algorithms) to capture the motion trajectory of high-speed targets. Then, the extracted two-dimensional motion information is input into a depth or disparity estimation network. Through feature learning and geometric inference of the network model, it is elevated to a three-dimensional geometric level, generating depth data reflecting the distance relationship between the target and the camera. Finally, based on the obtained depth or disparity information, rendering techniques such as image resampling and voxel deformation are used to translate and map the color frames of the central viewpoint to other viewpoints, thereby obtaining multi-view output images. To adapt to the processing requirements of high-speed scenes, the entire process typically relies on the parallel computing power of GPUs and multi-level caching mechanisms to achieve near real-time processing at 4K resolution, ensuring smooth visual output.

[0004] However, existing technical solutions still have technical defects when dealing with high-speed shooting scenes. The core problem is that cross-view misalignment is prone to occur during the multi-view image generation process, which ultimately affects the accuracy of the output image. Summary of the Invention

[0005] This application provides a multi-path parallax generation collaborative image generation method and apparatus to solve the problem of inaccuracy in multi-view image generation.

[0006] In a first aspect, this application provides a multi-path disparity generation collaborative image generation method, the method comprising: Acquire target videos from multiple perspectives; Motion analysis and feature extraction are performed on the target video to determine the sub-pixel edge consistency field of the high-speed moving target, wherein the sub-pixel edge consistency field is used to define the adjustable range of the pixel edges of the high-speed moving target; The subpixel edge consistency field and the luminance frame of the target video are input into a multi-disparity generation network. After feature fusion and disparity calculation, a preliminary disparity tensor is output. The preliminary disparity tensor is used to indicate the pixel position differences of the high-speed moving target under multiple viewpoints. The preliminary disparity tensor is corrected at the pixel level based on a generative 3D deformation kernel to obtain a fine disparity array, wherein the fine disparity array is used to indicate the disparity result after multi-view geometric alignment. Based on the fine parallax array, pixel-level registration and synthesis are performed on the original image in the target video to output a target image aligned from multiple perspectives.

[0007] Secondly, this application provides a multi-path parallax generation collaborative image generation apparatus, the apparatus comprising: The acquisition module is used to acquire target videos from multiple perspectives; The analysis module is used to perform motion analysis and feature extraction on the target video to determine the sub-pixel edge consistency field of the high-speed moving target, wherein the sub-pixel edge consistency field is used to define the adjustable range of the pixel edges of the high-speed moving target; The output module is used to input the subpixel edge consistency field and the brightness frame of the target video into the multi-disparity generation network, and output a preliminary disparity tensor after feature fusion and disparity calculation. The preliminary disparity tensor is used to indicate the pixel position difference of the high-speed moving target under multiple views. The correction module is used to perform pixel-level correction on the preliminary disparity tensor based on a generative 3D deformation kernel to obtain a fine disparity array, wherein the fine disparity array is used to indicate the disparity result after multi-view geometric alignment. The compositing module is used to perform pixel-level registration and compositing of the original images in the target video based on the fine parallax array, and output the target image after multi-view alignment.

[0008] Thirdly, this application provides an electronic device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus.

[0009] Fourthly, this application also provides a computer storage medium storing computer-executable instructions for executing the multi-path parallax generation collaborative image generation method described in any of the preceding claims of this application.

[0010] The technical solutions provided in this application have the following advantages compared with the prior art: The server first acquires target videos from multiple perspectives, then performs motion analysis and feature extraction on the videos to identify high-speed moving targets and generate a sub-pixel edge consistency field. This clarifies the adjustable range of pixel edges at the boundary between the high-speed moving target and the background, defining a reasonable variation range for cross-viewpoint registration of pixel edges under different perspectives. Based on this, the server inputs the sub-pixel edge consistency field and the brightness frame of the target video into a multi-disparity generation network. Through feature fusion and disparity calculation, it outputs a preliminary disparity tensor, accurately capturing the pixel positional relationship of the high-speed moving target under multiple perspectives. Next, using the adjustment range defined by the sub-pixel edge consistency field as a benchmark, the server calls a generative 3D deformation kernel to perform pixel-level correction on the preliminary disparity tensor, ensuring that the disparity data from different perspectives all point to the same spatial position of the high-speed moving target, thus obtaining a fine disparity array. Finally, using the fine disparity array as the registration benchmark, the server accurately maps the pixels of the original images from each perspective to a unified spatial coordinate system, achieving pixel-level alignment of the high-speed moving target and outputting the multi-view aligned target image. This application achieves multi-path parallax collaborative generation by unifying constraints of sub-pixel edge consistency fields and optimizing parallax correction, effectively improving the accuracy of cross-view image generation for high-speed moving targets. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0012] Figure 1 A flowchart of a multi-path parallax generation collaborative image generation method provided in this application embodiment; Figure 2 A schematic diagram of the diffusion-based depth estimation process provided in the embodiments of this application; Figure 3 This is a schematic diagram of the overall process provided for an embodiment of this application; Figure 4 A schematic diagram of the structure of a multi-path parallax generation collaborative image generation device provided in this application embodiment; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0015] The following will describe in detail, with reference to specific implementation methods, a multi-path parallax generation collaborative image generation method provided in this application embodiment, taking its application to a server as an example, such as... Figure 1 As shown, the specific steps are as follows: Step 101: Acquire target videos from multiple perspectives; Step 102: Perform motion analysis and feature extraction on the target video to determine the sub-pixel edge consistency field of the high-speed moving target. The sub-pixel edge consistency field is used to define the adjustable range of the pixel edges of the high-speed moving target. Step 103: Input the subpixel edge consistency field and the brightness frame of the target video into the multi-disparity generation network, and output the preliminary disparity tensor after feature fusion and disparity calculation. The preliminary disparity tensor is used to indicate the pixel position difference of the high-speed moving target under multiple views. Step 104: Perform pixel-level correction on the preliminary disparity tensor based on the generative 3D deformation kernel to obtain a fine disparity array, wherein the fine disparity array is used to indicate the disparity result after multi-view geometric alignment; Step 105: Perform pixel-level registration and synthesis on the original image in the target video based on the fine parallax array, and output the target image after multi-view alignment.

[0016] In step 101, the server uses multiple high-definition cameras deployed in different spatial locations to simultaneously capture video of the same scene, forming a multi-view data set, i.e., the target video. All cameras have undergone pre-synchronization and parameter calibration to ensure that video frames from different perspectives precisely correspond to the same moment. For example, the moment a race car crosses the finish line captured by camera A is perfectly aligned with the moment a race car crosses the finish line captured by camera B, avoiding subsequent registration errors caused by timing discrepancies.

[0017] In step 102, in this application, the pixel edge is the boundary between the surface pixels of a high-speed moving target and the background pixels, such as the boundary between the edge pixels of an athlete's arm and the background field pixels, or the boundary between the edge pixels of a mechanical part and the background pixels of a workbench.

[0018] The server then uses technologies such as pyramid optical flow and feature point matching to capture the motion trajectory of pixels in video frames, filter out high-speed moving targets with fast displacement speeds, such as running athletes, and extract key visual features such as pixel edges and textures of high-speed moving targets, focusing on locking the pixel edge regions of the targets.

[0019] Based on the extracted features, the server generates a sub-pixel edge consistency field. This field is a set of parameters that precisely describes the pixel edge characteristics of high-speed moving targets, strictly defining the adjustable range of pixel edges for these targets. "Sub-pixel" signifies precision exceeding that of ordinary pixels, capable of capturing subtle positional changes in pixel edges that are imperceptible to the human eye. "Consistency" refers to its provision of a unified reference standard for the pixel edges of the same high-speed moving target from different viewpoints. For example, the pixel edges of the same race car captured by camera A and camera B are aligned using this parameter set.

[0020] The subpixel edge consistency field transforms dispersed motion trajectories and visual features into unified constraint rules that focus on the edges of target pixels. This unified constraint rule provides a common alignment boundary for the pixel edges of high-speed moving targets from different viewpoints, clarifying the reasonable variation range of pixel edges during cross-viewpoint registration.

[0021] In step 103, the server extracts luminance frames from the target video. Luminance frames are the base images that retain the image's luminance information, which clearly presents the edge features of the target pixels, effectively reducing the interference of color differences on the calculation. Subsequently, the server inputs the luminance frames and the subpixel edge consistency field into the multi-parallax generation network to achieve deep fusion of the image's basic features and pixel edge constraint rules, ensuring that the network calculation always revolves around the edges of the real pixels.

[0022] The multi-disparity generation network (MDGN) performs disparity calculations under the constraint of a sub-pixel edge consistency field. The MMGN prioritizes the pixel edge regions of high-speed moving targets to ensure more accurate disparity calculations in these key areas, thus obtaining a preliminary disparity tensor. Disparity refers to the positional difference of the same target pixel under different viewpoints. For example, the shoulder pixel of the same athlete might be located on the left side in camera A's view and in the middle in camera B's view; this positional difference is disparity. The preliminary disparity tensor is a structured data integrating this positional difference from all viewpoints, clearly reflecting the pixel positional relationship of high-speed moving targets under multiple viewpoints, providing a direct basis for subsequent alignment.

[0023] In step 104, the server calls a generative 3D deformation kernel to optimize the initial disparity tensor. Due to factors such as high-speed motion blur and shooting noise, the initial disparity tensor may have disparity jumps, that is, the positional differences between adjacent pixels suddenly change significantly. For example, pixel edges that should be aligned may show positional shifts in the disparity data. These problems can directly lead to cross-viewpoint image misalignment.

[0024] The generative 3D deformation kernel uses the pixel edge adjustment range defined by the sub-pixel edge consistency field as a benchmark to perform pixel-level correction on the preliminary disparity tensor: for the pixel edge region of high-speed moving targets, the position difference data is finely adjusted pixel by pixel to make the disparity connection of the pixel edge natural and consistent with reality; for static background regions, the disparity data is kept smooth to avoid over-adjustment and the introduction of new errors.

[0025] The refined disparity array obtained after correction enables geometric matching of disparity data from different perspectives, meaning that the disparities from all perspectives point to the same spatial position, fully reflecting the three-dimensional geometric structure of high-speed moving targets.

[0026] In step 105, the server uses a fine parallax array as the registration reference and, based on the positional difference data of each pixel, precisely maps the original image pixels from different viewpoints to a unified spatial coordinate system. This process achieves pixel-level alignment of images from different viewpoints.

[0027] After registration, the server uses subpixel interpolation to fill in the subtle pixel gaps near the target pixel edges, ensuring continuous and complete pixel edges; smoothing brightness differences from different viewing angles to avoid obvious color blocks in the image; and preserving the detailed texture of high-speed moving targets, making the synthesized image both clear and natural. The entire process strictly follows the geometric constraints of a fine parallax array, always closely matching the actual shape of the target.

[0028] The final output multi-view aligned image solves the common cross-view misalignment problem in existing technologies, making pixel edges free of ghosting and foreground and background offset. This high-accuracy image can fully meet the needs of scenarios with extremely high visual precision requirements.

[0029] In this application, the server first acquires target videos from multiple perspectives, then performs motion analysis and feature extraction on the videos to identify high-speed moving targets and generate a sub-pixel edge consistency field. This clarifies the adjustable range of pixel edges at the boundary between the high-speed moving target and the background, defining a reasonable variation range for cross-viewpoint registration of pixel edges under different perspectives. Based on this, the server inputs the sub-pixel edge consistency field and the brightness frame of the target video into a multi-disparity generation network. Through feature fusion and disparity calculation, it outputs a preliminary disparity tensor, accurately capturing the pixel positional relationship of the high-speed moving target under multiple perspectives. Next, using the adjustment range defined by the sub-pixel edge consistency field as a benchmark, the server calls a generative 3D deformation kernel to perform pixel-level correction on the preliminary disparity tensor, ensuring that the disparity data from different perspectives all point to the same spatial position of the high-speed moving target, thus obtaining a fine disparity array. Finally, using the fine disparity array as a registration benchmark, the server accurately maps the pixels of the original images from each perspective to a unified spatial coordinate system, achieving pixel-level alignment of the high-speed moving target and outputting the multi-view aligned target image. This application achieves multi-path parallax collaborative generation by unifying constraints of sub-pixel edge consistency fields and optimizing parallax correction, effectively improving the accuracy of cross-view image generation for high-speed moving targets.

[0030] As an optional implementation, in step 102, motion analysis and feature extraction are performed on the target video to determine the sub-pixel edge consistency field of the high-speed moving target, including: Moving target detection is performed on adjacent frames of the target video, temporary interference is removed, high-speed moving targets are extracted, semantic anchors with unique numbers and motion prior masks are generated for high-speed moving targets, and the sparse optical flow of high-speed moving targets is determined. Among them, semantic anchors are used to indicate the locking of high-speed moving targets across frames, motion prior masks are used to identify the pixel range of high-speed moving targets, and sparse optical flow is used to record the pixel motion direction and amplitude data of high-speed moving targets. Semantic anchors, motion prior masks, and sparse optical flow are concatenated and input into a diffusion generation network to generate a depth probability map. The depth probability map is used to quantify the distance between each pixel in a high-speed moving target and the camera. By performing edge optimization and consistency constraint processing on the depth probability map, a sub-pixel edge consistency field is obtained.

[0031] The server performs motion target detection on adjacent frames of the target video. Through inter-frame grayscale difference and connected component analysis, it locates areas with drastic pixel changes. Then, it combines three-frame sliding window verification to remove temporary interference such as flash and lens noise, and accurately extracts high-speed moving targets from these areas.

[0032] For the extracted high-speed moving targets, the server generates two types of core identifiers: first, semantic anchors with unique numbers, each high-speed moving target corresponding to a unique number, ensuring stable locking of the same high-speed moving target in consecutive frames and avoiding target tracking loss during motion; second, motion prior masks, which identify the pixel range of the high-speed moving target in binary matrix form (binary mask), clarifying the effective area for subsequent calculations and reducing background interference. Simultaneously, the server calculates the sparse optical flow of the high-speed moving target using the Lucas-Kanade algorithm. This sparse optical flow, in units of feature points, records the motion direction and amplitude data of key pixels of the high-speed moving target. The server concatenates the semantic anchors, motion prior masks, and sparse optical flow of the high-speed moving target along the channel dimension to form a multi-dimensional feature tensor containing the target's identity, spatial range, and motion state. This feature tensor is then input into a diffusion generation network, also known as a sparse-continuous coupled diffusion generation network. The diffusion generation network gradually recovers the depth information from the feature tensor through a multi-step noise removal process. Combined with the motion prior mask, higher computational weights are assigned to the high-speed moving target region, ensuring that depth calculations focus on the high-speed moving target itself. The final generated depth probability map quantifies the distance between each pixel in the high-speed moving target and the camera using pixel-level probability values. A higher probability value indicates that the target part corresponding to that pixel is closer to the camera, and vice versa. In this application, the depth probability map transforms the two-dimensional motion information of a high-speed moving target into a three-dimensional geometric reference, breaking the viewpoint deviation problem caused by relying solely on two-dimensional features in the prior art and avoiding cross-viewpoint edge misalignment caused by depth blur. The server first extracts depth abrupt change regions using the Sobel gradient operator to locate the initial position of the target pixel edge. Then, it combines sparse optical flow to correct inter-frame phase drift of the edge, ensuring the stability of the edge position in the time dimension. Finally, it uses structure tensor-guided smoothing to eliminate isolated noise while preserving edge sharpness. After these processes, the server obtains a sub-pixel edge consistency field. This consistency field defines the adjustable range of pixel edges for high-speed moving targets with sub-pixel precision, defining a unified and reasonable variation range for edge registration under different viewpoints. Subpixel edge consistency field is the core bridge connecting depth information and disparity calculation. The unified edge constraint standard it provides allows edge data from different perspectives to have common constraints, solving the problems of scattered edge features and lack of registration basis in existing technologies.

[0033] Optionally, moving target detection is performed on adjacent frames of the target video, and temporary interference is removed to extract high-speed moving targets, including: Pyramid optical flow is used to perform sparse displacement estimation on adjacent frames in the target video, generating a set of feature point displacement vectors for capturing preset displacement and sub-pixel level motion. Bilinear interpolation is performed on all sampling points in the feature point displacement vector set to obtain the displacement amplitude map of the covered frame image; The displacement amplitude map and the inter-frame grayscale difference map are fused with adaptive weights to generate a dynamic saliency map. The inter-frame grayscale difference map is used to reflect brightness changes, and the dynamic saliency map is used to highlight high-speed moving areas under complex lighting conditions. The dynamic saliency map is subjected to dual-threshold segmentation and connected component filtering to obtain a set of candidate motion blocks. Dual-threshold segmentation is used to distinguish high-speed motion regions from the background based on the highest and lowest thresholds, and connected component filtering is used to eliminate lens noise and shake effects. Within a three-frame sliding window, Kalman prediction and nearest neighbor matching are performed on candidate moving blocks to filter out high-speed moving targets after excluding temporary interference.

[0034] S11: The server uses pyramidal Lucas-Kanade optical flow for the current frame. Compared to the previous frame Perform sparse displacement estimation.

[0035] To maintain the same resolution while simultaneously ensuring large displacement and sub-pixel accuracy, the server constructs a three-layer Gaussian pyramid, processing the current frame at each layer. Compared to the previous frame Corner detection is performed separately; then, displacement initialization is completed at the top coarse scale, and the displacement results are recursively refined layer by layer until the original resolution is restored. Finally, the server obtains the feature point displacement vector set. This vector set can accurately capture large displacements and sub-pixel-level minute movements of high-speed moving targets. The server calculates each vector in the vector set... The 2-norm, i.e. The motion intensity of each feature point is obtained, and then bilinear interpolation is performed on all sampling points to generate a displacement amplitude map on the entire image. The displacement amplitude map can reflect the instantaneous motion intensity at the pixel level, forming a continuous high-value band at the edge of a high-speed moving target, providing a basic metric for subsequent dynamic saliency detection.

[0036] S12: Generate a dynamic saliency map by fusing optical flow amplitude and grayscale difference.

[0037] The optical flow vectors introduced by high-speed moving targets are often synchronized with abrupt changes in brightness, but a single feature is insufficient to guarantee detection robustness in complex lighting environments. Therefore, the server performs a weighted fusion of the displacement amplitude map and the grayscale difference map to obtain a dynamic saliency map. Fusion weights Based on the global mean adaptive calculation of the two features, it can automatically amplify the proportion of optical flow information when the brightness of the image changes drastically, and increase the contribution of grayscale difference in stable lighting scenes, thereby stably highlighting high-speed moving areas under various lighting conditions.

[0038] The fusion formula is: .

[0039] in, This is a high-speed movement area. These are the pixel values ​​of the displacement amplitude map after linear normalization; These are the pixel values ​​after linear normalization of the grayscale difference image; Based on and The fusion weights are generated adaptively based on the global mean.

[0040] S13: Candidate motion blocks are obtained by dual-threshold segmentation and connected component filtering.

[0041] For dynamic saliency maps Due to the non-Gaussian distribution characteristics, the server uses a histogram peak-valley adaptive algorithm to determine the high threshold. With low threshold Through dual-threshold segmentation, seed pixels are first delineated in the high-threshold region, and then eight-neighborhood growing is performed in the low-threshold region to ensure that the edges of high-speed moving target blocks are sufficiently intact. Subsequently, the server calculates the area of ​​each region based on connected component labels, and identifies regions with areas smaller than a preset minimum value. Alternatively, spots with discontinuous intensity are removed to eliminate lens noise and the effects of slight camera shake, ultimately outputting a set of candidate motion blocks. ,in, For the candidate moving block set, Let t be the centroid of the j-th candidate moving block in the current frame, where t is the sequence number of the current frame and j is the number of the candidate moving block. Let be the total number of candidate motion blocks in frame t.

[0042] S14: Perform Kalman prediction and nearest neighbor matching within a three-frame sliding window.

[0043] To avoid occasional flickering interference entering the motion prior, the server sets the centroid of each candidate moving block. A Kalman filter with a one-dimensional uniform velocity model is established, and its theoretical position in the next frame is predicted through the state transition equation. The server searches for the theoretical position within the set of valid moving blocks from the previous frame. The Euclidean distance is the smallest and the direction angle is the smallest. Less than the threshold Paired blocks; if the speed ratio is satisfied simultaneously Belongs to the interval The candidate moving block is then deemed to have a consistent motion state and is promoted to a priori moving block. .in, It is the centroid of the candidate block of the k-th candidate moving block in the previous frame. It is the centroid of the candidate motion block of the kth candidate motion block in the previous frame.

[0044] This progressive verification method is based on short-term speed and direction consistency, and can quickly eliminate false motion caused by lens flash or noise jitter within 2 to 3 frames, ensuring that subsequent processing is only for real high-speed moving targets.

[0045] S15: Generate cross-frame semantic anchor points and record trajectory metadata.

[0046] The server provides each advancement's motion prior block. Assign a unique number At the same time, record the frame number of its first appearance. Current frame number And accumulate cross-frame displacement to form a trajectory vector. To facilitate rapid indexing during the deep generation stage, the server compresses the above information into anchor vectors. This vector is used as a conditional encoding input in the subsequent diffusion network to maintain semantic coherence of the same high-speed moving target in the time dimension and avoid deep discontinuities at the generation stage. The vector expression is:

[0047] in, For anchor vectors, A unique identifier is assigned to the prior block of motion. t is the frame number where block i first appears; t is the current frame number. For block Cumulative displacement since its appearance; Let be the average displacement vector of block i in the kth frame.

[0048] S16: Cache the motion prior mask and push it synchronously to the diffusion generation pipeline.

[0049] The server will process each motion prior block. binary mask Its semantic anchor Write to a lightweight cache as key-value pairs, along with timestamp information. The cache pointer is linked to the original frame. By maintaining address alignment within video memory, the subsequent discrete inverse diffusion network can quickly retrieve the mask and fuse it into the denoising-back inference process, ensuring that the edges of high-speed moving targets are aligned in the depth probability map. The geometric stability of the multi-parallax network is constantly enhanced.

[0050] S17: Monitor anchor point activity and promptly cancel outdated priors.

[0051] At the end of each frame processing, the server counts the number of silent frames for all semantic anchors since the last update. If the silent count of anchor i reaches a preset threshold... This indicates that the corresponding high-speed moving target has left the field of view, and the server immediately deletes its associated binary mask. With anchor vector This avoids invalid data consuming computing resources. Simultaneously, the server determines the sparse optical flow of the high-speed moving target through the above process. This sparse optical flow records the motion direction and amplitude data of key pixels of the high-speed moving target, using feature points as units.

[0052] In this application, S11 employs pyramidal Lucas-Kanade optical flow to achieve a balance between large displacement and sub-pixel accuracy, generating a displacement amplitude map that provides a reliable motion intensity measure for high-speed moving target detection; S2 uses adaptive weighted fusion of optical flow amplitude and grayscale difference to ensure that the dynamic saliency map can still stably highlight high-speed moving regions under complex lighting conditions, improving detection robustness; S3's dual-threshold segmentation and connected component screening specifically eliminate lens noise and shaky interference, obtaining a set of candidate moving blocks with complete edges; S14 utilizes Kalman prediction and nearest neighbor matching within a three-frame sliding window to quickly... Temporary interference such as flashes is removed to ensure that only genuine high-speed moving targets are retained and promoted to motion prior blocks. The cross-frame semantic anchors generated in S15, through unique numbers and trajectory metadata, achieve stable locking and semantic coherence of high-speed moving targets in the temporal dimension. S16 caches the motion prior mask and semantic anchors and synchronizes them to the diffusion generation pipeline, strengthening the feature weights of high-speed moving target edges for subsequent depth calculations. Anchor activity monitoring in S17 optimizes computational resource usage by promptly deregistering outdated priors, while the final determined sparse optical flow accurately records the pixel motion data of the high-speed moving targets. The cross-frame semantic anchors, motion prior masks, and sparse optical flow generated by this entire process form a complementary, high-quality feature set, solving the problem of weak cross-frame correlation of high-speed moving targets in existing technologies.

[0053] Optionally, the semantic anchors, motion prior masks, and sparse optical flow are concatenated and input into the diffusion generation network to generate a depth probability map, including: Based on semantic anchors, motion prior masks, and sparse optical flow, a spatiotemporal weight matrix is ​​generated, in which the weight of high-speed motion regions is greater than the weight of static background. The probability map of semantics and edges is obtained by multi-step inverse diffusion. In each step of inverse diffusion, the spatiotemporal weight matrix is ​​used to modulate the noise sample so that the number of iterations in the high-speed motion region is more than the number of iterations in the static background. The probability map of the current frame is mapped to the coordinate system of the previous frame by sparse optical flow, and an exponentially weighted average is performed with the probability map of the previous frame to generate a discrete depth prior. The discrete depth prior is used to suppress inter-frame phase drift caused by jitter. A subpixel-level mesh is constructed based on the input frame resolution. The discrete depth prior is resampled on the subpixel-level mesh through cubic spline interpolation to obtain the initial depth draft in the continuous coordinate domain. Anisotropic edge-preserving filtering is applied to the initial depth draft, and texture noise is suppressed by the brightness similarity term to obtain a depth probability map with sharp edges and no step artifacts. If the gradient direction of the depth probability map is opposite to or discontinuous with the motion direction of the sparse optical flow, the reverse diffusion process is backtracked and repeated until the gradient direction is consistent with the motion direction.

[0054] Figure 2 The flowchart of the diffusion-based depth estimation provided in this application includes the following.

[0055] S21: Conditional coding.

[0056] The server will obtain the binary motion prior mask. With cross-frame anchor vector Mapped to a channel stacking tensor and associated with sparse optical flow The amplitude and direction fields are concatenated by channel; then, a spatiotemporal weight matrix is ​​generated through 1×1 convolution and batch normalization. In the subsequent diffusion inference process, the high-speed moving region has a larger weight in the matrix, while the static background region has a smaller weight. This creates an explicit modulation of the salience of the high-speed moving target, ensuring that subsequent computing resources are tilted towards the core region.

[0057] S22: Discrete reverse diffusion.

[0058] The server injects pixels with a mean of zero and a variance of into a pixel grid at the input frame resolution. Gaussian noise is used to recover a probability map containing semantic information and edge features through a multi-step reverse diffusion process. In each step of the dedivergence process, a spatiotemporal weight matrix is ​​used. Pixel-level multiplication modulation is applied to noisy samples. High-speed moving target regions, due to their higher weights, receive more reverse iterations, which can effectively eliminate noise interference. Meanwhile, static background regions, with lower weights, can converge quickly to reduce invalid computations and ensure that the generated focus is completely consistent with the actual moving focus.

[0059] S23: Cross-frame alignment.

[0060] The server is based on sparse optical flow. The probability map of the current frame Forward mapping to the coordinate system of the previous frame, and compared with the cached probability map of the previous frame. Perform an exponentially weighted average to generate discrete deep priors that are consistent across the time dimension. This cross-frame alignment step can effectively suppress inter-frame phase drift caused by handheld shooting shake or image vibration, prevent flickering at the depth boundaries of high-speed moving targets, and ensure the temporal continuity of depth information.

[0061] S24: Continuous resampling.

[0062] The server will use discrete depth priors. Treating it as an implicit surface function, we directly resample on a pre-constructed sub-pixel-level grid using cubic spline interpolation to obtain a preliminary depth draft in the continuous coordinate domain. This resampling process achieves sub-pixel-level filling of the edge regions of high-speed moving targets, while increasing the sampling density for the fine line structures of the targets, providing high-precision input data for subsequent edge-preserving filtering processing, and laying the foundation for improving depth detail accuracy.

[0063] S25: Edge-preserving filter.

[0064] The server is in the deep draft. Anisotropic Gaussian-bilateral composite filtering is implemented: a smaller standard deviation is used along the depth gradient normal to maintain the sharpness of edges of high-speed moving targets; a larger standard deviation is used along the depth gradient tangent to achieve a smooth transition and eliminate noise. Simultaneously, a brightness similarity term is introduced to suppress the interference of texture noise on depth determination by measuring the brightness difference between pixels. The filtered result is a depth probability map. The image forms a probability fogging zone with controllable thickness along the depth axis, which not only preserves the fine outline of the high-speed moving target, but also completely avoids the generation of staircase artifacts.

[0065] S26: Occlusion check.

[0066] The server uses a depth probability graph. gradient direction and sparse optical flow The system performs pixel-by-pixel consistency checks based on the direction of motion. If the depth gradient direction of a pixel is detected to be opposite to the direction of motion, or if there is a significant discontinuity, it is determined to be an occlusion order anomaly. For such anomaly regions, the server will backtrack through a round of reverse diffusion and re-execute the inference until the gradient direction of the depth probability map is consistent with the direction of motion of the sparse optical flow, ensuring that the generated depth result conforms to the occlusion logic of the real physical scene.

[0067] S27: Result caching.

[0068] The server will eventually obtain the depth probability map. The uncertainty mask, along with the timestamp information, is written into the video memory cache and pushed to the subsequent multi-disparity generation network via the shared memory interface; at the same time, the depth probability map is retained in the displacement amplitude map. sparse optical flow The gradient path is used to adjust the diffusion weights during subsequent end-to-end training, so as to achieve the coordinated convergence of depth information and disparity data in high-speed motion scenes and further improve the accuracy of cross-view image alignment.

[0069] Optionally, by performing edge optimization and consistency constraint processing on the depth probability map, the sub-pixel edge consistency field is obtained, including: A symbolic distance map is generated based on the depth probability map, where a positive sign in the symbolic distance map indicates the foreground closer to the camera, and a negative sign indicates the background farther away from the camera. By mapping the symbolic distance map of the previous frame to the coordinate system of the current frame through sparse optical flow, and performing pixel-by-pixel difference with the symbolic distance map of the current frame, the sub-pixel precision edge phase shift amount is obtained. Determine the difference between the current frame and the previous frame in the symbol distance map, and recalculate the edge phase drift based on the residual between the difference and the edge phase drift. The edge phase drift is weighted by an exponential decay function, and a preliminary sub-pixel edge consistency field is generated by fusing the weighting result with the gradient magnitude of the depth probability map. The initial subpixel edge consistency field is subjected to structure tensor-guided anisotropic smoothing to obtain the smoothed subpixel edge consistency field. The smoothed subpixel edge consistency field and the timestamp of the current frame are input into the multi-disparity generation network to obtain the predicted disparity. If the offset of the predicted disparity in the edge normal exceeds the range of the subpixel edge consistency field, the local gradient update amplitude is reduced to achieve real-time edge correction and reduce flicker artifacts. Determine the variance of the subpixel edge consistency field. If the variance exceeds the variance threshold, re-perform sampling on the subpixel-level grid until the variance does not exceed the variance threshold, thus obtaining the final subpixel edge consistency field.

[0070] S31: Calculate the symbolic distance graph.

[0071] The server first processes the depth probability map. Apply the Sobel gradient operator and simultaneously obtain the gradient magnitude. With unit normal direction Then along the unit normal direction Directional tracing is performed to the nearest depth abrupt change surface. The sign of the abrupt change surface is recorded based on its proximity to the camera, ultimately forming a signed distance map. The foreground region closer to the camera is marked with positive values, while the background region farther from the camera is marked with negative values. The signed distance map provides a unified physical dimension for subsequent calculations of edge phase drift, ensuring that phase data from different regions have a comparable basis.

[0072] S32: Calculate the edge phase shift.

[0073] The server uses the symbol distance map of the current frame. Symbol distance map from the previous frame As the original input, first follow the obtained sparse optical flow The symbol distance map of the previous frame is forward-mapped to fit the coordinate system of the current frame. Then, pixel-wise difference is performed between the symbol distance map of the current frame and the mapped symbol distance map of the previous frame to finally obtain the sub-pixel precision edge phase shift. The calculation formula is as follows:

[0074] in, This value accurately reflects the subtle positional changes of the edge of a high-speed moving target between adjacent frames; Represents pixel coordinates; Let be the symbol distance value of the t-th frame; To utilize sparse optical flow to the t-th The value of 1 frame symbol distance mapped to the current pixel coordinate x; Let be the sparse optical flow vector of the t-th frame.

[0075] S33: Closed-loop residual correction for optical flow error.

[0076] The server reserves a dedicated frame symbolic distance map cache in the video memory, and first calculates the difference between the symbolic distance map of the current frame and the previous frame before sparse optical flow mapping. Then, the difference is compared with the obtained edge phase shift. Find the residual, where, This is the symbol distance value for the (t-1)th frame. If the residual is greater than a preset threshold... This indicates that there is a local error in the sparse optical flow, and the server immediately uses the least squares method to fine-tune the sparse optical flow. The local displacement components are recalculated, the mapping coordinates are updated, and the edge phase drift is recalculated. This ensures the closed-loop consistency of edge phase drift measurement in high-speed motion scenarios, preventing optical flow errors from propagating to subsequent stages.

[0077] S34: Fusion deep gradient confidence.

[0078] The server first normalizes the gradient magnitude of the depth probability map to obtain the edge confidence. The calculation method is as follows The edge confidence value ranges from [0,1], and the higher the value, the more reliable the edge at that location.

[0079] in, For edge confidence, Let x be the gradient magnitude of the depth probability map corresponding to the pixel at coordinate x in frame t. Let y be the gradient magnitude of the depth probability map corresponding to the pixel with coordinate y in frame t.

[0080] The attenuation function weights the edge phase shift Et(x) to construct a preliminary sub-pixel edge consistency field, as shown in the formula:

[0081] in For subpixel edge consistency field, The edge confidence level is α, which is a preset exponential decay coefficient used to adjust the degree of influence of phase drift on the uniformity field. This represents the edge phase shift.

[0082] S35: Anisotropic smoothing.

[0083] The server employs a structure tensor-guided anisotropic diffusion algorithm to perform preliminary sub-pixel edge consistency field analysis. Smoothing is performed by setting a smaller diffusion coefficient along the normal direction of the edge of the high-speed moving target to maximize edge sharpness and detail, and a larger diffusion coefficient along the tangential direction of the edge to effectively suppress noise. After a finite number of iterations, isolated noise in the initial consistency field is eliminated, while the main edge gradients of the high-speed moving target remain consistent, ultimately resulting in a smooth and detailed sub-pixel edge consistency field. .

[0084] S36: Soft-constraint injection multi-parallax generation network.

[0085] The server will smooth the subpixel edge consistency field. The timestamp of the current frame, along with the additional feature channel, is encapsulated and input to the encoding end of the subsequent multi-disparity generation network through the feature concatenation interface. During the network decoding stage, the server monitors in real time the offset of the predicted disparity in the normal direction of the edge of the high-speed moving target. If this offset exceeds the sub-pixel edge consistency field... Within the specified tolerance range, the gradient update amplitude of the region is automatically reduced. This soft constraint method enables real-time correction of the edges of high-speed moving targets, effectively reducing flicker artifacts in cross-view images.

[0086] S37: Closed-loop monitoring and in-depth refined triggering.

[0087] The server has a length of Within the sliding window, the sub-pixel edge consistency field is continuously calculated. variance When the variance Exceeding the preset threshold If this occurs, it indicates that the stability of the current uniformity field is insufficient and there may be depth data errors. The server will automatically call back the continuous resampling and edge-preserving filtering process to update the depth probability map of the corresponding spatial region. Refine and simultaneously refresh the subpixel edge consistency field. This closed-loop monitoring mechanism ensures that the depth data and edge consistency field remain stable throughout long-term sequence processing, providing a continuous and reliable constraint for cross-view image alignment.

[0088] The final sub-pixel edge consistency field generated by the closed-loop optimization in steps S31 to S37 of this application is a set of core constraint parameters specifically for high-speed moving targets. This consistency field clarifies the adjustable range of pixel edges of high-speed moving targets with sub-pixel precision, defining a unified and reasonable variation range for edge registration under different viewpoints. The unified edge constraint standard it provides allows edge data from multiple viewpoints to have a common range. At the same time, this consistency field integrates depth information, motion trajectory, and inter-frame stability verification, completely solving the problems of scattered edge features and lack of registration basis for high-speed moving targets in existing technologies. It effectively avoids edge misalignment and flicker artifacts in cross-viewpoint image alignment, laying a solid foundation for the subsequent output of high-precision disparity data by the multi-disparity generation network, and further improving the accuracy of cross-viewpoint image generation from the perspective of edge constraints.

[0089] Optionally, the subpixel edge consistency field and the luminance frame of the target video are input into a multi-disparity generation network, and the preliminary disparity tensor is output after feature fusion and disparity calculation, including: The luminance frames of the target video and the final subpixel edge consistency field are fed into the main coding branch and the auxiliary coding branch of the multi-disparity generation network, respectively, and the fused initial feature map is output. Based on the amplitude of the sub-pixel edge consistency field, channel-wise multiplicative modulation is performed on the initial feature map to output a modulated feature map that suppresses motion blur texture and focuses on the true contour; Based on the modulation feature map, a cost volume containing multi-view brightness difference matching information is constructed, and the pixel normal direction of the sub-pixel edge consistency field is read. A finite step size search is set along the pixel normal direction to generate the cost volume of the focused edge. Using the gradient and confidence of the subpixel edge consistency field as weights, pixel-level weight adjustment is performed on the brightness difference of the cost volume of the focused edge to generate sparse disparity anchor points, and the candidate tensors of sharp edges and in-plane smoothness are obtained based on the sparse disparity anchor points. Based on sparse disparity anchor points, residual propagation is carried out in dense disparity candidate tensors. During the propagation process, sub-pixel edge consistency fields are used as soft gating, and an initial disparity tensor is output after multiple iterations.

[0090] S41: Explicitly embedded consistency field.

[0091] The server is designed with a dual-coding branch structure for the multi-parallax generation network, with the main coding branch specifically receiving the luminance frames of the target video. The first branch is responsible for extracting basic brightness features and texture details of the image, providing raw image information for disparity calculation; the second branch is specifically connected to the generated final sub-pixel edge consistency field. The key focus is on preserving core constraint data such as the edge normal direction and confidence level of high-speed moving targets. In the first layer of network processing, the server stitches and fuses the feature maps output from the two branches at the channel level, enabling... The edge constraint information is fully preserved from the network input, effectively avoiding the weakening of such key information by subsequent high-frequency convolution operations, and providing continuous and accurate constraints for the entire process of disparity calculation.

[0092] S42: Pixel-level modulation of feature maps.

[0093] The server uses a subpixel edge consistency field. The amplitude is the modulation coefficient. Channel-by-channel multiplication is performed on the initial feature map output by the main coding branch. At the edge pixel positions of high-speed moving targets, through... Amplification of feature response intensity enhances the discernibility of edge features; at the planar pixel location of a high-speed moving target, through... The amplitude attenuation feature response suppresses redundant texture information. This modulation operation filters out motion blur texture interference commonly found in high-speed moving images from the encoder input, directly guiding the feature extraction process to focus on the true contours of high-speed moving targets, significantly improving the effective information density of the feature map.

[0094] S43: Implementation of search window limited by normal direction.

[0095] During the cost volume construction phase, the server calls the sub-pixel edge consistency field C in real time. The provided pixel normal direction data strictly limits the disparity search range to within this normal direction, setting only candidate displacements with a finite step size, completely eliminating the stacking of invalid displacements in the edge tangential direction. This not only compresses the invalid space of the disparity search by more than 60%, but also fundamentally suppresses the lateral drift problem that easily occurs at the edges of high-speed moving targets in disparity calculation, ensuring that the cost volume is accurately focused on the effective matching of the edge region.

[0096] S44: Generate confidence-weighted parallax anchor points.

[0097] During the brightness difference matching measurement phase, the server will apply the sub-pixel edge consistency field. The gradient and confidence level are fused into a weighting factor, and pixel-level weighted adjustment is performed on the brightness difference in the cost volume of the focused edge. In the edge region of high-speed moving target with a confidence level higher than 0.8, the contribution of the effective brightness difference is enhanced and sparse disparity anchors are generated (anchor spacing is controlled at 2-3 pixels). Then, shape-preserving interpolation is performed on these sparse anchors. By combining bilinear interpolation with edge gradient constraints, it is ensured that the edge shape of high-speed moving target is not distorted during the interpolation process. Finally, a dense disparity candidate tensor that takes into account both the sharp edges and in-plane smoothness of high-speed moving target is obtained.

[0098] S45: Controlled transfer of residuals.

[0099] The server uses sparse disparity anchor points as a stable feature diffusion source to perform disparity residual propagation in dense disparity candidate tensors, achieving refined diffusion of disparity information; during the propagation process, it reads the sub-pixel edge consistency field in real time. As a soft gating mechanism, when residual updates cause a decrease in the edge consistency of high-speed moving targets (such as an edge gradient change rate exceeding 15%), the residual propagation step size is automatically reduced to half of the original step size, and the lateral propagation path is frozen, effectively preventing distortion problems such as false expansion or compression of high-speed moving targets in the disparity tensor.

[0100] S46: Output the initial disparity tensor and write it to the buffer.

[0101] After 3-5 rounds of residual-controlled iterations, the multi-disparity generation network outputs an initial disparity tensor. Simultaneously, the server uses the subpixel edge consistency field... Derived parallax uncertainty mask (mask value and) (Confidence level positively correlated) will be used to determine the initial disparity tensor The uncertainty mask is written into the video memory cache, which provides a stable depth basis for subsequent parallax fine-tuning correction and also retains the data from... to The gradient path supports parameter fine-tuning in subsequent end-to-end training.

[0102] In this application, S41 explicitly embeds the consistency field through dual-encoding branches to ensure that key constraints such as edge normals and confidence scores are fully preserved from the network input, avoiding information loss from high-frequency convolution; S42 relies on the amplitude of the consistency field to achieve feature pixel-level modulation, suppressing motion blur textures from the source and guiding feature extraction to focus on the true contours of high-speed moving targets; S43 limits the disparity search window along the normal provided by the consistency field, compressing more than 60% of the invalid space while suppressing the lateral drift of the edges of high-speed moving targets from the root; S44 generates sparse disparity anchor points by weighting the gradient of the consistency field with confidence scores, and combines shape-preserving interpolation to obtain a dense disparity candidate tensor that takes into account both sharp edges and in-plane smoothness; S45 achieves controlled propagation of residuals through soft gating of the consistency field, effectively preventing false expansion or compression of high-speed moving targets; S46 outputs the initial disparity tensor and caches it, while retaining the gradient path to support end-to-end fine-tuning. The initial disparity tensor generated by the entire process accurately captures the pixel positional relationship of high-speed moving targets under multiple viewpoints, completely avoiding problems such as disparity drift and edge blurring. It provides high-quality initial data for subsequent disparity correction of generative 3D deformation kernels, strengthening the foundation for accurate cross-viewpoint image alignment from the source of disparity generation.

[0103] Optionally, in step 104, pixel-level correction is performed on the preliminary disparity tensor based on the generative three-dimensional deformation kernel to obtain a fine disparity array, including the following:

[0104] S51: Gated disparity gradient extraction.

[0105] The server will initially generate the parallax tensor. Feeding horizontal gradient operator With vertical gradient operator Two-dimensional gradient pairs are obtained pixel by pixel. In high-speed motion scenes, high-speed moving targets (such as vehicle body reflections or high-speed wind turbine blades) often exhibit dense stripes. Directly using gradients can easily misinterpret texture undulations as true depth undulations. This step calls the sub-pixel edge consistency field. Confidence level of normalized Chinese economics Pixel-level gating is performed, multiplying low-confidence pixels by a decay factor to weaken their gradient influence, while high-confidence pixels retain their original amplitude values. Ultimately, a stable gradient that matches the true geometric edges of the high-speed moving target is preserved, resulting in a gradient template. This template acts as a filter in the subsequent deformation kernel generation, ensuring that the deformation direction unfolds along the true contour. The formula for calculating the gradient template is:

[0106] in, The gradient template is edge-locked, and x is the voxel coordinate; For the reason Marginal confidence obtained by normalization; These are the weighting coefficients for the horizontal and vertical directions, respectively. Let x be the initial disparity tensor of the x-th pixel coordinate in frame t. It is the horizontal gradient operator. It is the vertical gradient operator.

[0107] S52: Generative 3D Deformation Kernel Construction.

[0108] The server will use gradient templates With the sparse light flow of history The data is concatenated by channel and input to the spatial transformation deformation kernel generator Φ. Considering that high-speed motion can easily cause large-scale shifts in the parallax distribution between consecutive frames, it is difficult to accurately infer the deformation trend by relying solely on the current frame. The generator extracts the local texture principal direction of the high-speed moving target through convolution, gating, and normalization sequences, and then fuses the average displacement vector of historical optical flow to decode a three-dimensional displacement kernel for each voxel. The displacement kernel size is controlled within the range of ≤5×5×5, and the corresponding weights are normalized by softmax and take values ​​between [0,1]. This ensures both the smoothness of local deformation and the differentiability of the weights, making it compatible with the subsequent gradient backpropagation process.

[0109] S53: Voxel-level remapping implementation.

[0110] The server is based on a three-dimensional displacement kernel. For the initial parallax tensor Voxel-level resampling is performed, which is equivalent to performing convolution with a 3D offset on the multi-view parallax volume. During resampling, the resampling is performed along the left and right viewing directions according to... The amplitude is applied to the source voxels with weighted stretching or compression, so that the parallax planes of each viewpoint are... The calibrated edge normals are aligned for the first time using a common reference plane, effectively eliminating the parallax skipping problem caused by high-speed motion. The remapping output is denoted as... The calculation formula is:

[0111] in, Here, Δ represents the disparity value after the first remapping, Δ represents the displacement kernel support domain, and x represents the three-dimensional volume coordinates. The corresponding voxel center; d is the offset vector within the support domain of the generative 3D displacement kernel; The weights of the deformation kernel; It is the sampled value of the preliminary parallax volume at position x+d, providing compensation information for the left and right, up and down, and front and back parallax surfaces.

[0112] S54: Residual adaptive deformation correction operator.

[0113] The server will remap the results. Transform to the coordinate system of individual side views, and perform pixel-by-pixel differencing with the original disparity of the corresponding viewpoint to obtain the geometric consistency error. When a high-speed moving target occludes the scene or rapidly approaches the camera, the error will concentrate in the edge neighborhood. The server will... The spatial gradient is fed back to the deformation kernel generator Φ, driving the generator to update the kernel weight matrix and adding a penalty factor to the displacement direction, causing the deformation kernel to converge finely in the direction of reducing error. Simultaneously, this closed-loop process performs real-time retrieval... If the confidence level of the error is too low (below 0.5), the learning rate is decreased to avoid amplifying the deformation error at the edge of noise.

[0114] S55: Controlled iterative convergence.

[0115] The server repeats the residual back-injection and remapping process until the cross-view parallax average error is less than a preset threshold. Or the number of iterations has reached the limit. Before each iteration begins, the system checks the deformation direction and... The edge locking regularization process determines the angle between the principal normals of the target's edges. If the angle exceeds the limit, an angle penalty is triggered, and the deformation kernel weights of adjacent voxels are frozen to ensure that the deformation does not cross the true edge of the high-speed moving target. This edge locking regularization effectively eliminates the streaking artifacts commonly seen in high-speed pedestrian outlines and high-speed vehicle edges, while maintaining the sharpness of key areas such as vehicle occlusion boundaries.

[0116] S56: Outputs a fine parallax array and caches it.

[0117] After the iteration stops, the server obtains a fine parallax array. and the corresponding confidence mask Actual road tests have verified that... In high-speed driving scenarios, the vehicle edges exhibit a transition width of <0.2 pixels, and the gradient continuity in flat road surface areas is improved by 18% compared to the initial parallax tensor. The server will... and Write to graphics memory to provide a precise and stable depth reference for temporal depth fusion and composite rendering; while preserving the initial parallax tensor. Subpixel edge consistency field To fine parallax array The complete gradient linking supports subsequent end-to-end training and ensures the convergence of the training process.

[0118] The refined disparity array obtained through the above correction further compresses the error of the initial disparity tensor from the sub-pixel level to within 0.1 pixels, accurately matching the three-dimensional geometric shape of high-speed moving targets under multiple perspectives. It completely solves the problems of disparity jump and edge streaking caused by high-speed motion, and provides a millimeter-level disparity benchmark for pixel-level alignment of subsequent cross-view images. Finally, it completes the accuracy closed loop of cross-view image alignment from the dimension of geometric correction.

[0119] In this application, S51 uses confidence gating to extract stable disparity gradients, effectively suppressing texture interference in areas such as vehicle body reflections and high-speed wind blades, providing a gradient template that fits the real geometric edges for deformation correction; S52 fuses the gradient template with historical optical flow to construct a generative 3D deformation kernel, balancing deformation smoothness and differentiability, and adapting to large-scale disparity drift caused by high-speed motion; S53 performs voxel-level remapping along the line of sight, aligning the disparity planes of each viewpoint with a unified normal, completely eliminating disparity jumps caused by high-speed motion; S54 dynamically optimizes the deformation kernel weights through a residual adaptive correction operator, combined with confidence gating to avoid... Excessive deformation at noise-free edges improves geometric consistency; S55 achieves controlled iterative convergence with edge-locking regularization, effectively eliminating streaking artifacts at the outlines of high-speed pedestrians and vehicles while maintaining sharp occlusion boundaries; the fine parallax array output by S56 presents a transition width of <0.2 pixels at the vehicle body edges, and the gradient continuity in flat areas is improved by 18%, providing a precise and stable depth benchmark for temporal depth fusion and synthetic rendering, while retaining a complete gradient link to support end-to-end training throughout the entire process. It thoroughly solves the parallax distortion problem in high-speed motion scenes from the perspective of geometric correction, laying a solid core foundation for pixel-level alignment of cross-view images.

[0120] Optionally, in step 105, the original image in the target video is registered and synthesized at the pixel level based on the fine parallax array, and the output multi-view aligned target image includes the following contents.

[0121] S61: Coordinate unification and pixel-level index construction.

[0122] The server first displays a fine parallax array. Load it into video memory and precisely align its spatial resolution to the current color frame using a linear scaling operation. The coordinate system ensures a one-to-one correspondence between disparity data and the spatial positions of image pixels; simultaneously, a timestamp synchronization module inserts a fixed frame delay to eliminate temporal discrepancies between disparity calculation and image acquisition, guaranteeing consistency between the two in the temporal dimension. Based on the results of dual spatial and temporal synchronization, the server generates a pixel index grid. Each grid point x in this grid can simultaneously index the color brightness value, corresponding to the disparity value and confidence mask information, thus establishing a unified coordinate framework for subsequent sub-pixel displacement calculations and avoiding cross-module data misalignment.

[0123] S62: Jitter threshold retrieval.

[0124] For pixel index grid For each pixel x in the table, the server reads the noise suppression lookup table using a lookup table method. The jitter threshold γt(x) corresponding to the pixel is obtained. This jitter threshold is adaptively generated by step S2 based on the local variance of historical frames. Its core function is to limit the jitter amplitude of sub-pixels accumulated over time, effectively avoiding edge flickering caused by the superposition of small displacements between frames in high-speed motion scenes, and defining a reasonable boundary for displacement calculation.

[0125] S63: Quantization-consistent subpixel displacement vector solver.

[0126] The server employs the quantized consistent sub-pixel displacement vector solver proposed in this application, with a fine disparity matrix. Based on the provided multi-view parallax, while also adopting and using the same... Homogeneous quantization step size For each target perspective (N is the total number of viewpoints) Calculate the horizontal sub-pixel displacement. To ensure edge synchronization of high-speed moving targets and eliminate numerical rounding errors, the displacement is processed using an interval trimming operator. Within the jitter threshold range of S62 retrieval, the calculation formula is as follows:

[0127] in, denoted as the horizontal sub-pixel displacement, where x is the pixel coordinate; The central view index serves as a reference benchmark for multi-view registration. To quantize the step size, Let x be the fine disparity matrix of the x-th pixel coordinates in frame t. This is the jitter threshold, used to limit accumulated error; It is a unit vector in the horizontal direction; Ensure that the displacement is strictly within Within the range, avoid geometric distortion caused by excessive displacement.

[0128] S64: Texture sampling subpixel translation.

[0129] The server will calculate the sub-pixel displacement As a texture coordinate offset, it is fed into the GPU rendering pipeline. Leveraging the parallel computing capabilities of GPU hardware texture sampling, it directly processes color frames. Perform subpixel-level translation operations to obtain initial images for each target viewpoint. The formula for the initial image is shown below.

[0130] .

[0131] in, For the initial image, x For sub-pixel displacement, Let t be the luminance frame of the target video.

[0132] This method uses the GPU's built-in linear filtering mechanism to perform subpixel interpolation, which avoids the additional computational errors caused by software interpolation and fully leverages the bandwidth advantage of parallel rendering to ensure the real-time processing efficiency of high-resolution images.

[0133] S65: Bilinear smoothing with normal constraint.

[0134] The server reads the fine parallax array. Depth normal Its calculation method is as follows This ensures the normalization property of the normal vector. Subsequently, only along the depth normal... For the initial image Perform one bilinear smoothing operation, keeping the pixel values ​​along the edge tangent unchanged. The smoothing formula is:

[0135] in, This is the initial image after smoothing.

[0136] This operation is performed in real time during the GPU shader stage, which can effectively remove jagged edges in the image after subpixel translation, and avoid excessive blurring of the outline of high-speed moving targets, achieving the effect of noise reduction without loss of edge.

[0137] S66: Confidence fallback and multi-view integration.

[0138] The server uses the transmitted confidence mask. For the smoothed image Perform pixel-level filtering: when ( When the confidence threshold is reached, the disparity data corresponding to that pixel is deemed unreliable, and the system automatically reverts to the corresponding pixel value in the original color frame. This prevents flickering or artifacts in low-confidence areas. Finally, the server smooths the images from all target viewpoints. The images are fused using a depth consistency aggregation algorithm to ensure logical consistency in the depth dimension across multiple viewpoints, ultimately yielding a multi-view output of the target. This output It achieves subpixel-level edge alignment in high-speed motion scenarios, eliminating grid jagged edges and providing a stable visual benchmark for downstream applications such as stereoscopic display and light field reconstruction.

[0139] In this application, S61 constructs a unified pixel index grid by synchronizing coordinate scaling and timestamps to ensure accurate matching of disparity data and color frames in the spatiotemporal dimension, laying a unified framework for subsequent registration; S62 generates a jitter threshold based on the variance of historical frames to effectively limit the accumulation of sub-pixel jitter and avoid flickering at the edges of high-speed moving targets; S63 adopts a quantized consistent displacement solver, combined with homogeneous quantization step size and threshold clipping, to eliminate rounding errors and maintain edge synchronization, ensuring the accuracy of displacement calculation; S64 achieves efficient sub-pixel translation by leveraging GPU hardware texture sampling, and avoids additional errors by relying on built-in linear filtering, balancing processing speed and interpolation accuracy; S65 performs bilinear smoothing along the depth normal, removing jagged edges while maintaining the sharpness of the contours of high-speed moving targets, achieving noise reduction without edge loss; S66 filters low-reliability pixels through a confidence backoff mechanism, and completes multi-view fusion by combining depth-consistent aggregation, eliminating flickering and artifacts. The entire process outputs multi-view images with subpixel-level edge alignment and no raster jaggedness defects. It provides a stable visual benchmark for stereoscopic display and light field reconstruction, and meets the requirements of high-resolution real-time processing through GPU parallel optimization. It completely solves the problems of edge misalignment and flickering in multi-view images under high-speed motion scenes, and completes the final closed loop of cross-view image alignment from the registration and synthesis dimension.

[0140] Optionally, after outputting the multi-view aligned target image, the method further includes: A fine parallax array is used to stack target images aligned from multiple viewpoints to form a cross-viewpoint aligned edge tensor. A cross-view difference map is obtained based on the edge magnitude of each view in the cross-view aligned edge tensor, and the cross-view difference map is compressed into a scalar index that reflects the synchronicity of multiple views. If the scalar metric exceeds the adaptive threshold, then random noise and geometric misalignment are distinguished in the cross-view difference plot; Random noise is fed back to the step of generating a depth probability map to optimize the filtering parameters, and geometric misalignment is fed back to the step of generating a fine disparity array to adjust the deformation kernel parameters. The target image after multi-view alignment is regenerated until the scalar index does not exceed the adaptive threshold.

[0141] S71: Cross-view edge stack construction.

[0142] The server receives the rendered N target views. , For the set of multi-path target views in frame t, This is the view of the 0th path in the t-th frame. For the view of the (N-1)th path in the t-th frame, the luminance channel is first extracted to obtain a grayscale frame for each target image. Then, the Sobel operator is directly applied to calculate the gradient magnitude map. Subsequently, non-maximum suppression is used to preserve edges with a single pixel width, generating a clear edge map. To ensure that subsequent difference evaluation only reflects the true geometric error, the server invokes a fine-grained disparity matrix. The pixel-wise parallax, according to the main view reference frame for all edge maps The backprojection operation is performed, during which bicubic interpolation is used to maintain subpixel precision at the edges, and finally, a cross-viewpoint aligned edge tensor is stacked in video memory. This tensor achieves strict alignment in spatial dimensions, providing a reliable benchmark for subsequent cross-view difference calculations.

[0143] S72: Cross-view difference plot generation and index extraction.

[0144] Cross-view aligned edge tensor Within the system, the server iterates through each pixel coordinate x, calculates the difference between the edge amplitudes of any two viewpoints, and sums them to obtain a cross-viewpoint difference map. Subsequently, the cross-view difference map was analyzed. Perform mean-variance statistics on the entire graph and compress it into a scalar index. , Let Dt be the mean pixel value of the entire cross-view difference map. Dt represents the standard deviation of pixels across the entire cross-view difference map. This scalar index comprehensively considers the overall misalignment intensity and local fluctuation amplitude of edges from multiple viewpoints, where the coefficients... During the experimental calibration phase, the scalar index was determined based on human eye sensitivity; a larger value indicates poorer edge synchronization across multiple viewpoints. The entire computation process is completed in shared memory to ensure compatibility with the transmission rate of 4K@60fps video and meet real-time processing requirements.

[0145] S73: Adaptive threshold determination and output decision implementation.

[0146] The threshold controller reads the average motion speed, global brightness and contrast, and historical frame scalar metrics of the current frame from the scene analysis module. Curve, dynamically calculated frame-level adaptive threshold If scalar indicators This indicates that the multi-view edge synchronization requirement is met, and the server directly confirms the validity of the current frame output and proceeds to the next frame processing; if the scalar If a significant geometric misalignment or noise flicker risk is detected, the system immediately enters the feedback optimization branch and marks the output of this frame with a delay to prevent abnormal frames from being sent to the subsequent display stage.

[0147] S74: Random Noise Classification and Feedback.

[0148] Server for cross-view difference map A 3×3 opening operation is performed to remove small connected components, and then scattered pixels and tiny patches are screened out based on an area threshold, which are identified as random noise. For the coordinates corresponding to these random noises, the control unit sends the location list to the noise suppression lookup table manager in the depth probability map generation step. The damping coefficient is increased and the diffusion step size is shortened in the corresponding grid entries to achieve rapid suppression of high-frequency jitter. Moreover, the parameter update is completed through constant memory broadcast, with a delay of less than 1 millisecond, which does not affect the real-time processing link.

[0149] S75: Geometric misalignment classification and feedback completed.

[0150] For cross-view difference maps For regions with disparities exceeding a set threshold and exhibiting a continuous linear or blocky distribution, the server marks them as geometric misalignments. It then encapsulates the region mask, center coordinates, and area ratio into a feedback packet, which is sent back to the deformation kernel scheduler in the process of generating the fine-grained disparity array. Within the corresponding spatial window, the deformation kernel learning rate is locally increased. And increase the maximum number of iterations. To address parallax plane jumps caused by sudden, intense motion, rapid alignment is achieved within three frames to prevent the double image phenomenon from accumulating.

[0151] S76: Parameter refresh and closed-loop convergence monitoring.

[0152] Upon receiving the feedback packet, the steps for generating the depth probability map and the steps for generating the fine disparity matrix immediately refresh the internal parameters on the GPU and apply them at the rendering pipeline synchronization barrier. The new parameters are then used for calculation starting from the next input frame. The controller continuously records scalar metrics. The waveform is calculated and the attenuation slope is determined. If the average descent gradient of the three frames is lower than the preset slope... If the learning rate limit is increased, the learning rate limit will be automatically increased to ensure the closed-loop convergence speed while taking into account the real-time processing requirements.

[0153] S77: Steady-state detection and log archiving.

[0154] When K consecutive frames satisfy the scalar index And the descent slope stabilizes at [ When the range is [0.02, 0.02], the server determines that the closed loop has reached a steady state, freezes the noise suppression lookup table and deformation kernel parameters, retains only the monitoring function, reduces unnecessary write operations, and lowers resource consumption. After video processing is complete, the server outputs a complete adjustment log, which includes a timestamp. The curves, parameter update amounts, and trigger frame numbers provide detailed data support for offline quality assessment and subsequent algorithm version iterations.

[0155] In this application, S71 constructs a spatially strictly aligned cross-view edge tensor through edge extraction, backprojection, and tensor stacking, providing a reliable benchmark for geometric error evaluation; S72 generates a cross-view difference map and compresses it into a scalar index, comprehensively reflecting multi-view synchronization, and its computational efficiency is adapted to the real-time transmission requirements of 4K@60fps; S73 calculates an adaptive threshold based on scene dynamic information to achieve accurate quality judgment and output decision; S74 accurately classifies random noise and feeds it back to the depth generation stage, quickly suppressing high-frequency jitter by optimizing filtering parameters, with parameter update delay of less than 1 millisecond, without affecting the real-time link; S75 locates geometric misalignment and transmits it back to the parallax refinement stage, achieving fast alignment within three frames by adjusting deformation kernel parameters to prevent double image accumulation; S76 refreshes parameters in real time and monitors the convergence slope, balancing the closed-loop convergence speed and real-time processing requirements; S77 implements steady-state detection and log archiving, reducing invalid write operations while providing data support for algorithm iteration. The entire process forms a complete closed loop of detection, classification, feedback, and optimization, effectively solving the problems of random noise interference and geometric misalignment in high-speed motion scenes, ensuring that multi-view images maintain sub-pixel-level edge alignment accuracy over a long period of time, providing stable and reliable visual input for downstream applications such as stereoscopic display and light field reconstruction, and ultimately completing the full-process quality control of cross-view image alignment.

[0156] Figure 3 The overall process diagram provided for this application shows the functions of each module as follows.

[0157] High-speed motion prior extraction: As the starting point of the process, it is responsible for extracting the motion features (such as sparse optical flow and motion mask) of high-speed moving targets from the target video, providing motion constraints for subsequent depth estimation.

[0158] Diffusion-based coarse depth estimation: Based on motion priors, a preliminary depth probability map is generated through a diffusion model to complete the initial modeling of depth information.

[0159] Edge phase shift closed loop: Combining depth map and motion information, calculate and optimize sub-pixel level edge phase shift to generate a sub-pixel edge consistency field, providing edge constraints for subsequent disparity generation.

[0160] Multi-disparity generation: Input edge consistency field and brightness frame to generate preliminary disparity tensor; at the same time output disparity confidence to provide a reliability reference for subsequent correction.

[0161] Generative 3D Deformation Kernel Refinement: Receives a preliminary disparity tensor and uncertainty mask, optimizes the disparity through a deformation kernel, and outputs a refined disparity array to eliminate disparity distortion caused by high-speed motion.

[0162] GPU subpixel multi-view synthesis: Based on a fine parallax array, subpixel-level registration and synthesis are performed on the original image to output a target image after multi-view alignment.

[0163] Cross-pixel differential evaluation: The edge consistency of the synthesized multi-view images is checked to generate a scalar index reflecting synchronization. If the index exceeds the limit, random noise and geometric misalignment are distinguished and fed back to the diffusion-type depth coarse estimation (optimizing filter parameters) and the generative 3D deformation kernel refinement (adjusting deformation kernel parameters) respectively, forming a quality control closed loop.

[0164] The application scenarios of this application cover a wide range of fields, including intelligent driving, unmanned systems, content production, industrial and medical applications, mobile imaging, and emerging digital fields.

[0165] The key advantages of this application compared to various alternatives include the following:

[0166] 1. Compared with frame-by-frame detection + traditional optical flow scheme: leading in both temporal consistency and real-time performance.

[0167] This application inherently guarantees temporal consistency through cross-frame semantic anchor point numbering and trajectory tracking, eliminating the need for additional network structures. It enables collaborative scheduling of data from the same source on the GPU, achieving real-time performance and stability far exceeding that of the proposed solution.

[0168] 2. Compared with end-to-end spatiotemporal Transformer or neural radiation field solutions: it has advantages in deployment flexibility and real-time performance.

[0169] This application adopts segmented optimization and GPU inline design, which makes the memory usage controllable and does not rely on external calibration. While ensuring subpixel accuracy, it can easily meet the real-time requirements of 4K@60fps and has a wider range of deployment scenarios.

[0170] 3. Compared with hardware optimization + conventional multiparallax network solutions: significant advantages in environmental adaptability and cost.

[0171] While hardware combinations such as high-speed shutter, global exposure, and multi-channel synchronous CMOS can be used to reduce inter-frame motion blur, coupled with conventional multi-parallax networks for processing, this approach becomes highly sensitive to lighting conditions and hardware costs. In low-light environments, shutter speeds must be extended, causing motion blur to reappear. Furthermore, multi-channel synchronous CMOS imposes stringent requirements on camera layout, heat dissipation, and power consumption, making it difficult to meet the size and power consumption constraints of consumer-grade and embedded devices. This application achieves a breakthrough at the algorithmic level. Through core technologies such as diffuse depth recovery and edge consistency field constraints, it effectively suppresses motion blur and edge flicker without requiring special hardware support. It can still operate stably in complex lighting and low-light environments, significantly reducing hardware deployment costs and adapting to more terminal scenarios.

[0172] 4. Compared with offline post-processing solutions: it has higher timeliness and scenario adaptability.

[0173] If an offline post-processing solution is adopted, caching all frames and then using optical flow-depth joint optimization to obtain a globally minimized result will result in the loss of the time constraints of online rendering, making it only suitable for offline content production scenarios. For systems with high timeliness requirements, such as streaming media and in-vehicle assisted driving, this offline process is completely unacceptable. The millisecond-level GPU feedback and real-time closed-loop monitoring mechanism in this application ensures that each frame processing can output effective results instantly, perfectly adapting to online application scenarios, while guaranteeing accuracy no less than that of the offline solution, achieving the dual satisfaction of real-time performance and high precision.

[0174] 5. Compared with the two-dimensional guided filtering / Patch-Match local smoothing scheme, the geometric correction accuracy and complexity control are better.

[0175] If the generative 3D deformation kernel is abandoned in the parallax refinement stage and replaced with 2D adaptive guided filtering or patch-Match-based local smoothing techniques, voxel-level 3D elastic correction cannot be achieved. Especially at high-speed foreground-background boundaries, the random search of patch-Match can easily introduce uncontrolled lateral drift. To compensate for this deficiency, additional occlusion detection and confidence reweighting modules are required, without significantly reducing the overall technical complexity. The generative 3D deformation kernel in this application can accurately adapt to the 3D geometry of high-speed moving targets. Combined with edge consistency field constraints, it fundamentally avoids lateral drift without requiring additional redundant modules, achieving higher correction accuracy while simplifying the architecture.

[0176] The aforementioned alternatives either sacrifice spatiotemporal consistency, resulting in edge flickering; rely on high hardware costs and offline processes for limited accuracy; or are difficult to implement in real-time scenarios due to memory and computing power requirements. This application achieves an optimal balance between accuracy, latency, and resource consumption through three core technologies: anchor-driven multi-domain joint constraints, closed-loop self-calibration of a differentially deformable kernel, and millisecond-level GPU feedback paths. Its comprehensive advantages are irreplaceable in high-speed online video processing scenarios, providing a solution with high precision, real-time performance, and deployment flexibility for multiple fields such as autonomous driving, mobile imaging, and content production.

[0177] This application provides a multi-path parallax generation collaborative image generation apparatus, such as... Figure 4 As shown, the device includes: Acquisition module 401 is used to acquire target videos from multiple perspectives; Analysis module 402 is used to perform motion analysis and feature extraction on the target video and determine the sub-pixel edge consistency field of the high-speed moving target. The sub-pixel edge consistency field is used to define the adjustable range of the pixel edges of the high-speed moving target. The output module 403 is used to input the subpixel edge consistency field and the brightness frame of the target video into the multi-disparity generation network, and output a preliminary disparity tensor after feature fusion and disparity calculation. The preliminary disparity tensor is used to indicate the pixel position difference of the high-speed moving target under multiple views. The correction module 404 is used to perform pixel-level correction on the preliminary disparity tensor based on the generative three-dimensional deformation kernel to obtain a fine disparity array, wherein the fine disparity array is used to indicate the disparity result after multi-view geometric alignment. The compositing module 405 is used to perform pixel-level registration and compositing of the original images in the target video based on a fine parallax array, and output the target image after multi-view alignment.

[0178] Optionally, the analysis module 402 is used for: Moving target detection is performed on adjacent frames of the target video, temporary interference is removed, high-speed moving targets are extracted, semantic anchors with unique numbers and motion prior masks are generated for high-speed moving targets, and the sparse optical flow of high-speed moving targets is determined. Among them, semantic anchors are used to indicate the locking of high-speed moving targets across frames, motion prior masks are used to identify the pixel range of high-speed moving targets, and sparse optical flow is used to record the pixel motion direction and amplitude data of high-speed moving targets. Semantic anchors, motion prior masks, and sparse optical flow are concatenated and input into a diffusion generation network to generate a depth probability map. The depth probability map is used to quantify the distance between each pixel in a high-speed moving target and the camera. By performing edge optimization and consistency constraint processing on the depth probability map, a sub-pixel edge consistency field is obtained.

[0179] Optionally, the analysis module 402 is specifically used for: Pyramid optical flow is used to perform sparse displacement estimation on adjacent frames in the target video, generating a set of feature point displacement vectors for capturing preset displacement and sub-pixel level motion. Bilinear interpolation is performed on all sampling points in the feature point displacement vector set to obtain the displacement amplitude map of the covered frame image; The displacement amplitude map and the inter-frame grayscale difference map are fused with adaptive weights to generate a dynamic saliency map. The inter-frame grayscale difference map is used to reflect brightness changes, and the dynamic saliency map is used to highlight high-speed moving areas under complex lighting conditions. The dynamic saliency map is subjected to dual-threshold segmentation and connected component filtering to obtain a set of candidate motion blocks. Dual-threshold segmentation is used to distinguish high-speed motion regions from the background based on the highest and lowest thresholds, and connected component filtering is used to eliminate lens noise and shake effects. Within a three-frame sliding window, Kalman prediction and nearest neighbor matching are performed on candidate moving blocks to filter out high-speed moving targets after excluding temporary interference.

[0180] Optionally, the analysis module 402 is specifically used for: Based on semantic anchors, motion prior masks, and sparse optical flow, a spatiotemporal weight matrix is ​​generated, in which the weight of high-speed motion regions is greater than the weight of static background. The probability map of semantics and edges is obtained by multi-step inverse diffusion. In each step of inverse diffusion, the spatiotemporal weight matrix is ​​used to modulate the noise sample so that the number of iterations in the high-speed motion region is more than the number of iterations in the static background. The probability map of the current frame is mapped to the coordinate system of the previous frame by sparse optical flow, and an exponentially weighted average is performed with the probability map of the previous frame to generate a discrete depth prior. The discrete depth prior is used to suppress inter-frame phase drift caused by jitter. A subpixel-level mesh is constructed based on the input frame resolution. The discrete depth prior is resampled on the subpixel-level mesh through cubic spline interpolation to obtain the initial depth draft in the continuous coordinate domain. Anisotropic edge-preserving filtering is applied to the initial depth draft, and texture noise is suppressed by the brightness similarity term to obtain a depth probability map with sharp edges and no step artifacts. If the gradient direction of the depth probability map is opposite to or discontinuous with the motion direction of the sparse optical flow, the reverse diffusion process is backtracked and repeated until the gradient direction is consistent with the motion direction.

[0181] Optionally, the analysis module 402 is specifically used for: A symbolic distance map is generated based on the depth probability map, where a positive sign in the symbolic distance map indicates the foreground closer to the camera, and a negative sign indicates the background farther away from the camera. By mapping the symbolic distance map of the previous frame to the coordinate system of the current frame through sparse optical flow, and performing pixel-by-pixel difference with the symbolic distance map of the current frame, the sub-pixel precision edge phase shift amount is obtained. Determine the difference between the current frame and the previous frame in the symbol distance map, and recalculate the edge phase drift based on the residual between the difference and the edge phase drift. The edge phase drift is weighted by an exponential decay function, and a preliminary sub-pixel edge consistency field is generated by fusing the weighting result with the gradient magnitude of the depth probability map. The initial subpixel edge consistency field is subjected to structure tensor-guided anisotropic smoothing to obtain the smoothed subpixel edge consistency field. The smoothed subpixel edge consistency field and the timestamp of the current frame are input into the multi-disparity generation network to obtain the predicted disparity. If the offset of the predicted disparity in the edge normal exceeds the range of the subpixel edge consistency field, the local gradient update amplitude is reduced to achieve real-time edge correction and reduce flicker artifacts. Determine the variance of the subpixel edge consistency field. If the variance exceeds the variance threshold, re-perform sampling on the subpixel-level grid until the variance does not exceed the variance threshold, thus obtaining the final subpixel edge consistency field.

[0182] Optionally, the output module 403 is used for: The luminance frames of the target video and the final subpixel edge consistency field are fed into the main coding branch and the auxiliary coding branch of the multi-disparity generation network, respectively, and the fused initial feature map is output. Based on the amplitude of the sub-pixel edge consistency field, channel-wise multiplicative modulation is performed on the initial feature map to output a modulated feature map that suppresses motion blur texture and focuses on the true contour; Based on the modulation feature map, a cost volume containing multi-view brightness difference matching information is constructed, and the pixel normal direction of the sub-pixel edge consistency field is read. A finite step size search is set along the pixel normal direction to generate the cost volume of the focused edge. Using the gradient and confidence of the subpixel edge consistency field as weights, pixel-level weight adjustment is performed on the brightness difference of the cost volume of the focused edge to generate sparse disparity anchor points, and the candidate tensors of sharp edges and in-plane smoothness are obtained based on the sparse disparity anchor points. Based on sparse disparity anchor points, residual propagation is carried out in dense disparity candidate tensors. During the propagation process, sub-pixel edge consistency fields are used as soft gating, and an initial disparity tensor is output after multiple iterations.

[0183] Optionally, the device is also used for: A fine parallax array is used to stack target images aligned from multiple viewpoints to form a cross-viewpoint aligned edge tensor. A cross-view difference map is obtained based on the edge magnitude of each view in the cross-view aligned edge tensor, and the cross-view difference map is compressed into a scalar index that reflects the synchronicity of multiple views. If the scalar metric exceeds the adaptive threshold, then random noise and geometric misalignment are distinguished in the cross-view difference plot; Random noise is fed back to the step of generating a depth probability map to optimize the filtering parameters, and geometric misalignment is fed back to the step of generating a fine disparity array to adjust the deformation kernel parameters. The target image after multi-view alignment is regenerated until the scalar index does not exceed the adaptive threshold.

[0184] like Figure 5 As shown, this application provides an electronic device, including a processor, a high-speed moving target, a communication interface 502, a memory 503, and a communication bus 504, wherein the processor, the high-speed moving target, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0185] Memory 503 is used to store computer programs.

[0186] In one embodiment of this application, when the processor is executing a program stored in memory 503 on a high-speed moving target, it implements the multi-path parallax generation collaborative image generation method provided in any of the aforementioned method embodiments.

[0187] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the multi-path parallax generation cooperative image generation method provided in any of the foregoing method embodiments.

[0188] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A multi-path parallax generation collaborative image generation method, characterized in that, The method includes: Acquire target videos from multiple perspectives; Motion analysis and feature extraction are performed on the target video to determine the sub-pixel edge consistency field of the high-speed moving target, wherein the sub-pixel edge consistency field is used to define the adjustable range of the pixel edges of the high-speed moving target; The subpixel edge consistency field and the luminance frame of the target video are input into a multi-disparity generation network. After feature fusion and disparity calculation, a preliminary disparity tensor is output. The preliminary disparity tensor is used to indicate the pixel position differences of the high-speed moving target under multiple viewpoints. The preliminary disparity tensor is corrected at the pixel level based on a generative 3D deformation kernel to obtain a fine disparity array, wherein the fine disparity array is used to indicate the disparity result after multi-view geometric alignment. Based on the fine parallax array, pixel-level registration and synthesis are performed on the original image in the target video to output a target image aligned from multiple perspectives.

2. The method according to claim 1, characterized in that, Motion analysis and feature extraction are performed on the target video to determine the sub-pixel edge consistency field of the high-speed moving target, including: Moving target detection is performed on adjacent frames of the target video, temporary interference is removed, high-speed moving targets are extracted, and semantic anchors and motion prior masks with unique numbers are generated for the high-speed moving targets, and the sparse optical flow of the high-speed moving targets is determined. The semantic anchor is used to indicate the locking of the high-speed moving target across frames; the motion prior mask is used to identify the pixel range of the high-speed moving target; and the sparse optical flow is used to record the pixel motion direction and amplitude data of the high-speed moving target. The semantic anchors, the motion prior mask, and the sparse optical flow are concatenated and input into a diffusion generation network to generate a depth probability map, wherein the depth probability map is used to quantify the distance between each pixel in the high-speed moving target and the camera. By performing edge optimization and consistency constraint processing on the depth probability map, a sub-pixel edge consistency field is obtained.

3. The method according to claim 2, characterized in that, Moving target detection is performed on adjacent frames of the target video, and temporary interference is removed to extract high-speed moving targets, including: Pyramid optical flow is used to perform sparse displacement estimation on adjacent frames in the target video to generate a set of feature point displacement vectors for capturing preset displacement and sub-pixel level motion. Bilinear interpolation is performed on all sampling points in the feature point displacement vector set to obtain the displacement amplitude map of the covered frame image; The displacement amplitude map and the inter-frame grayscale difference map are fused with adaptive weights to generate a dynamic saliency map. The inter-frame grayscale difference map is used to reflect brightness changes, and the dynamic saliency map is used to highlight high-speed moving areas under complex lighting conditions. The dynamic saliency map is subjected to dual-threshold segmentation and connected component filtering to obtain a candidate motion block set. The dual-threshold segmentation is used to distinguish high-speed motion regions from the background based on the highest and lowest thresholds, and the connected component filtering is used to eliminate lens noise and shake effects. Kalman prediction and nearest neighbor matching are performed on the candidate moving blocks within a three-frame sliding window to filter out high-speed moving targets after excluding temporary interference.

4. The method according to claim 2, characterized in that, The semantic anchors, the motion prior mask, and the sparse optical flow are concatenated and input into a diffusion generation network to generate a depth probability map, including: A spatiotemporal weight matrix is ​​generated based on the semantic anchor, the motion prior mask, and the sparse optical flow, wherein the weight of high-speed motion regions in the spatiotemporal weight matrix is ​​greater than the weight of static background. The probability map of restored semantics and edges is obtained through multi-step inverse diffusion, wherein each step of inverse diffusion uses the spatiotemporal weight matrix to modulate the noise sample so that the number of iterations of the high-speed motion region is more than the number of iterations of the static background. The probability map of the current frame is mapped to the coordinate system of the previous frame by the sparse optical flow, and an exponentially weighted average is performed with the probability map of the previous frame to generate a discrete depth prior, wherein the discrete depth prior is used to suppress inter-frame phase drift caused by jitter. A subpixel-level mesh is constructed based on the input frame resolution. The discrete depth prior is resampled on the subpixel-level mesh using cubic spline interpolation to obtain a preliminary depth draft in the continuous coordinate domain. Anisotropic edge-preserving filtering is performed on the initial depth draft, and texture noise is suppressed by the brightness similarity term to obtain a depth probability map with sharp edges and no step artifacts. If the gradient direction of the depth probability map is opposite to or discontinuous with the motion direction of the sparse optical flow, the reverse diffusion process is backtracked and repeated until the gradient direction is consistent with the motion direction.

5. The method according to claim 2, characterized in that, By performing edge optimization and consistency constraint processing on the depth probability map, the sub-pixel edge consistency field is obtained, including: A symbolic distance map is generated based on the depth probability map, wherein positive signs in the symbolic distance map are used to indicate foregrounds close to the camera, and negative signs are used to indicate backgrounds far from the camera; The symbolic distance map of the previous frame is mapped to the coordinate system of the current frame through the sparse optical flow, and pixel-by-pixel difference is performed between the symbolic distance map of the current frame to obtain the sub-pixel precision edge phase drift. Determine the difference between the current frame and the previous frame in the symbol distance map, and recalculate the edge phase drift amount based on the residual between the difference and the edge phase drift amount; The edge phase drift is weighted by an exponential decay function, and a preliminary sub-pixel edge consistency field is generated by fusing the weighting result with the gradient magnitude of the depth probability map. The preliminary subpixel edge consistency field is subjected to structure tensor-guided anisotropic smoothing to obtain a smoothed subpixel edge consistency field. The smoothed subpixel edge consistency field and the timestamp of the current frame are input into the multi-disparity generation network to obtain the predicted disparity. If the offset of the predicted disparity in the edge normal exceeds the range of the subpixel edge consistency field, the local gradient update amplitude is reduced to achieve real-time edge correction and reduce flicker artifacts. The variance of the subpixel edge consistency field is determined. If the variance exceeds a variance threshold, sampling on the subpixel-level grid is re-executed until the variance does not exceed the variance threshold, thus obtaining the final subpixel edge consistency field.

6. The method according to claim 1, characterized in that, The subpixel edge consistency field and the luminance frame of the target video are input into a multi-disparity generation network. After feature fusion and disparity calculation, the preliminary disparity tensor is output, including: The luminance frame of the target video and the final subpixel edge consistency field are respectively fed into the main coding branch and the auxiliary coding branch of the multi-disparity generation network, and the fused initial feature map is output. Based on the amplitude of the sub-pixel edge consistency field, the initial feature map is subjected to channel-wise multiplication modulation to output a modulated feature map that suppresses motion blur texture and focuses on the true contour. Based on the modulation feature map, a cost volume containing multi-view brightness difference matching information is constructed, and the pixel normal direction of the sub-pixel edge consistency field is read. A finite step size search is set along the pixel normal direction to generate the cost volume of the focused edge. Using the gradient and confidence of the subpixel edge consistency field as weights, pixel-level weight adjustment is performed on the brightness difference of the cost volume of the focused edge to generate sparse disparity anchor points, and sharp edges and dense disparity candidate tensors with in-plane smoothness are obtained based on the sparse disparity anchor points. Based on the sparse disparity anchor points, residual propagation is carried out in the dense disparity candidate tensor. During the propagation process, the sub-pixel edge consistency field is used as a soft gate, and an initial disparity tensor is output after multiple iterations.

7. The method according to claim 2, characterized in that, After outputting the multi-view aligned target image, the method further includes: The fine parallax array is used to stack the multi-view aligned target images to form a cross-view aligned edge tensor; A cross-view difference map is obtained based on the edge magnitude of each view in the cross-view aligned edge tensor, and the cross-view difference map is compressed into a scalar index that reflects the synchronization of multiple views. If the scalar index exceeds the adaptive threshold, then random noise and geometric misalignment in the cross-view difference map are distinguished. The random noise is fed back to the step of generating the depth probability map to optimize the filtering parameters, and the geometric misalignment is fed back to the step of generating the fine disparity array to adjust the deformation kernel parameters. The target image after multi-view alignment is regenerated until the scalar index does not exceed the adaptive threshold.

8. A multi-path parallax generation collaborative image generation device, characterized in that, The device includes: The acquisition module is used to acquire target videos from multiple perspectives; The analysis module is used to perform motion analysis and feature extraction on the target video to determine the sub-pixel edge consistency field of the high-speed moving target, wherein the sub-pixel edge consistency field is used to define the adjustable range of the pixel edges of the high-speed moving target; The output module is used to input the subpixel edge consistency field and the brightness frame of the target video into the multi-disparity generation network, and output a preliminary disparity tensor after feature fusion and disparity calculation. The preliminary disparity tensor is used to indicate the pixel position difference of the high-speed moving target under multiple views. The correction module is used to perform pixel-level correction on the preliminary disparity tensor based on a generative 3D deformation kernel to obtain a fine disparity array, wherein the fine disparity array is used to indicate the disparity result after multi-view geometric alignment. The compositing module is used to perform pixel-level registration and compositing of the original images in the target video based on the fine parallax array, and output the target image after multi-view alignment.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 7.

Images