Video blind area compensation transmission method based on dynamic field of view fusion and server

Abstract

The application provides a video blind area compensation transmission method and server based on dynamic field of view fusion, through acquiring multi-source video stream data, integrating all field of view regions under the same timestamp to a unified global reference space to obtain a global field of view splicing layout, constructing a dynamic field of view topology evolution graph containing a blind area hole outline and a pixel mapping transformation relationship between fields of view according to layout changes of continuous timestamps; performing pixel-by-pixel spatial migration on adjacent field of view video frames according to the pixel mapping transformation relationship of the blind area hole periphery to generate a blind area compensation video segment; performing panorama fusion on the compensation segment and the original video segment to obtain a panoramic video stream; analyzing the distribution pattern of the original collection source and the content remapping source in the panoramic video stream to generate a spatial mask, setting a differentiated encoding transmission strategy for different regions and encoding and distributing. The application realizes high-quality compensation of blind area content and regional differentiated efficient transmission of panoramic video streams.

Description

Technical Field

[0001] This application relates to the field of video processing, and in particular to a video blind spot compensation transmission method and server based on dynamic field fusion. Background Technology

[0002] When multi-camera surveillance systems generate panoramic video by fusing video streams from multiple video acquisition devices, blind spots exist between the fields of view of different devices due to limitations in device deployment location and installation posture. Existing processing methods typically involve directly stitching adjacent video streams to cover blind spots, or filling the blind spots with static background images. Direct stitching easily causes visual discontinuity and interrupted motion trajectories at the boundaries of blind spots in panoramic video. Static filling, because the filling content cannot dynamically change with the scene, results in static artifacts in the panoramic video at the blind spot location that are inconsistent with the surrounding original acquisition area. Summary of the Invention

[0003] This invention provides a video blind spot compensation transmission method and server based on dynamic field fusion.

[0004] In a first aspect, embodiments of the present invention provide a video blind spot compensation transmission method based on dynamic field-of-view fusion, comprising: Acquire multi-source video stream data synchronously collected by the video acquisition device cluster. The multi-source video stream data contains multiple video segment units aligned by timestamps. Each video segment unit is accompanied by the spatial positioning information and field of view description of the corresponding acquisition device. Based on spatial positioning information and field of view description, the field of view regions of all video segment units under the same timestamp are integrated into a unified global reference space to obtain the global field of view stitching layout of the current timestamp. According to the changes in the global field of view stitching layout of consecutive timestamps, a dynamic field of view topology evolution map describing the boundary evolution and occlusion relationship between fields of view is constructed. The dynamic field of view topology evolution map includes the spatial adjacency connection between each field of view region, the blind hole outline, and the pixel mapping transformation relationship between fields of view. The dynamic field of view topology evolution map is input into the field of view content remapping process. The pixel mapping transformation relationship of the region around the blind spot hole contour is extracted. Based on the pixel mapping transformation relationship, a pixel-by-pixel spatial migration operation is performed on the video frames of adjacent fields of view to generate the remapped image content in the blind spot hole. The remapped image content is then smoothly connected with the surrounding original pixels to obtain the blind spot compensation video clip. A panoramic fusion is performed on the blind spot compensation video clip and the original video clip. In the overlapping area of ​​adjacent fields of view, dynamic fusion weights are generated based on the acquisition confidence metric and remapping confidence metric of each pixel source. The pixels in the overlapping area are weighted and synthesized based on the dynamic fusion weights to obtain a panoramic video stream that maintains spatiotemporal continuity. This study analyzes the distribution patterns of the original acquisition sources and content remapping sources in different regions of a panoramic video stream, generates spatial mask information to mark the source types of the regions, sets differentiated encoding and transmission strategies for each region based on the spatial mask information, and encodes and distributes the panoramic video stream according to the differentiated encoding and transmission strategies, so that the remapping regions can obtain encoding quality that is independently adjustable from the original acquisition regions.

[0005] Secondly, embodiments of the present invention provide a server, including: a memory for storing computer-executable instructions or computer programs; and a processor for executing the computer-executable instructions or computer programs stored in the memory to implement the above-mentioned video blind spot compensation transmission method based on dynamic field fusion.

[0006] The embodiments of this application have the following beneficial effects: This invention integrates the spatial positioning information and field-of-view descriptions attached to multi-source video stream data into a unified global reference space and constructs a dynamic field-of-view topology evolution map. This allows for the continuous temporal tracking of blind spot hole contours and pixel mapping transformation relationships between fields of view, providing geometric constraints with the same origin as the original acquisition for subsequent compensation. After inputting the dynamic field-of-view topology evolution map into the field-of-view content remapping processing flow, pixel-by-pixel spatial migration and smooth connection are performed on adjacent field-of-view video frames based on the pixel mapping transformation relationships around the holes. This ensures that the blind spot filling content directly originates from the imaging representation of the same scene by adjacent fields of view at the same moment, rather than relying on external generative priors, thereby maintaining the homology of the compensation content with the original acquisition content in terms of texture source and structure. During the panoramic fusion process, dynamic fusion weights are generated for each pixel position based on the acquisition confidence metric and the remapping confidence metric. This allows the original pixels and remapping pixels to participate in the synthesis according to their respective confidence levels. The consistency constraint between the weight spatial distribution and the confidence distribution regulates the multi-source fusion transition behavior. In the transmission stage, the distribution pattern of the original acquisition area and the content remapping area is analyzed and spatial mask information of the area type is generated. Based on this, differentiated coding transmission strategies are set for areas of different source types, so that the remapping area can obtain coding quality that is independently adjustable from the original acquisition area. Thus, under channel-limited conditions, coding precision that is different from that of the original area is allocated to the remapping area. Attached Figure Description

[0007] Figure 1 This is a schematic diagram of the architecture of the application scenario provided in the embodiments of this application; Figure 2 This is a schematic diagram of the server structure provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the video blind spot compensation transmission method based on dynamic field fusion provided in this application embodiment. Detailed Implementation

[0008] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0009] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0010] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of an application scenario provided in this application embodiment. The video acquisition device cluster consists of multiple independent camera devices 400 deployed in different spatial locations within the monitoring scene. Each camera device 400 establishes a communication connection through a wired or wireless network and achieves time synchronization based on a network time protocol or a precise time protocol. Each camera device 400 continuously outputs video stream data after starting acquisition, and this video stream data is aggregated to a central processing node or edge computing node. The camera devices 400 are connected to a server 200 through a network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both, using wireless or wired links for data transmission.

[0011] In some embodiments, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Camera device 400 can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0012] The server implementing the video blind spot compensation transmission method based on dynamic field fusion provided in this application will be described next. See Figure 2 , Figure 2 This is a schematic diagram of the server structure provided in an embodiment of this application. Figure 2The server shown includes at least one processor 210, memory 250, at least one network interface 220, and an external interface 230. The various components in server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 240.

[0013] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0014] External interface 230 may include, for example, one or more speakers and / or one or more visual displays. External interface 230 may also include one or more input devices 232, such as a keyboard, mouse, microphone, touch screen display, camera, etc.

[0015] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices physically located away from the processor 210.

[0016] The memory 250 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 250 described in this application embodiment is intended to include any suitable type of memory.

[0017] In some embodiments, memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0018] Operating system 251 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 252 is used to reach the camera device via one or more (wired or wireless) network interfaces 220, such as Bluetooth, WiFi, and Universal Serial Bus (USB). Presentation module 253 is configured to enable the display of information (e.g., external interface for operating peripheral devices and displaying content and information) via one or more output devices 231 (e.g., display screen, speaker, etc.) associated with external interface 230; The input processing module 254 is used to detect and translate one or more user inputs or interactions from one or more input devices 232.

[0019] Based on the above description of the application scenarios and servers provided in the embodiments of this application, the video blind spot compensation transmission method based on dynamic field-of-view fusion provided in the embodiments of this application is described below. In actual implementation, the video blind spot compensation transmission method based on dynamic field-of-view fusion provided in the embodiments of this application can be implemented by a server. See also Figure 3 , Figure 3 This is a flowchart illustrating the video blind spot compensation transmission method based on dynamic field-of-view fusion provided in an embodiment of this application. Next, it will be combined with... Figure 3 The steps shown are explained.

[0020] Step S100: Obtain multi-source video stream data synchronously collected by the video acquisition device cluster. The multi-source video stream data contains multiple video segment units aligned by timestamps. Each video segment unit is accompanied by the spatial positioning information and field of view description of the corresponding acquisition device.

[0021] In this embodiment, the multi-source video stream data contains multiple video segment units. Each video segment unit corresponds to an image sequence acquired by a camera device within a certain continuous time period. This image sequence consists of several video frames arranged chronologically according to their acquisition time. During the organization of the multi-source video stream data, a unified time reference is used to perform timestamp alignment on the video segment units output by different camera devices, ensuring that the video frames of each video segment unit at the same timestamp correspond to the same acquisition time. Each video segment unit is accompanied by spatial positioning information and a field of view description. The spatial positioning information is used to determine the position and orientation of the acquisition device in physical space, including the device's three-dimensional coordinates in the world coordinate system and attitude parameters such as gimbal pitch angle, horizontal yaw angle, and roll angle. The field of view description is used to determine the spatial area that the acquisition device can cover at the current moment. It is represented by the geometric description of a field of view cone composed of field of view angle parameters and effective imaging distance parameters, or it can be represented as a sequence of vertex coordinates of the projected polygon of the device's imaging plane in a unified global reference space.

[0022] Step S200: Based on spatial positioning information and field of view description, integrate the field of view regions of all video segment units under the same timestamp into a unified global reference space to obtain the global field of view stitching layout of the current timestamp. Based on the changes in the global field of view stitching layout of consecutive timestamps, construct a dynamic field of view topology evolution map describing the boundary evolution and occlusion relationship between fields of view. The dynamic field of view topology evolution map includes the spatial adjacency connection between each field of view region, the blind hole outline, and the pixel mapping transformation relationship between fields of view.

[0023] In one implementation, step S200 may specifically include the following steps S210 to S260: Step S210: For each video segment unit under the current timestamp, project the field of view of the video segment unit to a unified global reference space using its spatial positioning information, generate the field of view projection polygon of the unit, and summarize the field of view projection polygons of all units to obtain the initial global element layout.

[0024] During projection, for each video segment unit at the current timestamp, the 3D coordinates and attitude parameters of the camera device are extracted from its associated spatial positioning information, and the field of view angle parameters and effective imaging distance parameters are extracted from the field of view description. First, starting from the 3D coordinates of the camera device, four boundary rays in 3D space are constructed based on the horizontal and vertical field of view angles. These four boundary rays originate from the origin in the device's local coordinate system and point towards the four corner points of the field of view. Next, using the attitude parameters, a rotation matrix and translation vector are constructed from the device's local coordinate system to the unified global reference space, transforming the four boundary rays from the device's local coordinate system to the unified global reference space. The four boundary rays extend to the effective imaging distance in the unified global reference space, obtaining the 3D coordinates of the four far corner points. The 3D coordinates of the camera device are also transformed into the unified global reference space as near vertices. The near vertices and the four far corner points together form a spatial quadrilateral pyramid. The surface of this spatial quadrilateral pyramid represents the 3D spatial coverage of the field of view in the unified global reference space. To enable stitching and comparison of different field-of-view regions on the same reference plane, the spatial quadrilateral cone is projected onto a preset horizontal reference plane or the scene ground model plane. The projection operation is achieved by taking the planar coordinates of each vertex of the cone on the reference plane, resulting in a field-of-view projection polygon described by a sequence of planar polygon vertices. The internal region of this field-of-view projection polygon is a two-dimensional region that the corresponding camera device can effectively cover in a unified global reference space, and the polygon vertices sequentially record the closed boundary of this region. The field-of-view projection polygons of all video segment units are aggregated into the same planar coordinate space. If multiple field-of-view projection polygons overlap, the independent boundary information of each polygon is preserved, and fusion processing is not performed temporarily. All field-of-view projection polygons together constitute the initial global element layout of the current timestamp. This initial global element layout explicitly marks the regions covered by field-of-view projection polygons and the blank regions not covered by any field-of-view projection polygons.

[0025] Step S220: In the initial global polygon layout, detect the intersection area between each field of view projection polygon, perform boundary fusion on the intersection area, eliminate the repeated expression of the intersection boundary, and obtain a global polygon splicing layout with continuous boundary. At the same time, identify isolated hole areas that are not covered by any field of view projection polygons, and assign a hole label to each isolated hole area.

[0026] During intersection region detection and boundary fusion, Boolean operations in computational geometry are used to process each pair of field-view projected polygons in the initial global element layout. For any two field-view projected polygons with an intersection relationship, the intersection polygon region is first calculated. The boundary of the intersection polygon region is composed of the intersection line segment generated by the intersection of the two field-view projected polygons and the boundary segment that falls inside the other. After calculating the intersection polygon, the two field-view projected polygons involved in the operation are replaced with the difference polygon obtained after removing the intersection region to eliminate the repeated line segments in the boundary representations of the two polygons. After Boolean operation processing is completed for all pairs of field-view projected polygons with an intersection relationship, all the retained field-view projected polygons and intersection polygons together constitute a global element stitching layout with continuous boundaries. Any two boundary line segments in the layout are either collinear and point-connected or do not intersect each other, and there are no overlapping or intersecting boundary representations. Based on this, on a two-dimensional plane covering the entire unified global reference space, continuous regions not covered by any field-view projected polygons are identified and marked as isolated void regions. An isolated hole region is a closed polygonal region enclosed by the outer boundaries of several field-of-view projection polygons or scene extent boundaries, containing no effective field-of-view coverage. Each identified isolated hole region is assigned a globally unique hole identifier, which remains stable throughout the lifecycle of the dynamic field-of-view topology evolution map to track the evolution of the same hole region in subsequent timestamps. Hole identifiers are generated using an assignment method based on a combination of timestamps and spatial hashes, ensuring that hole regions at the same physical location at different timestamps are assigned consistent identifiers, while hole regions at different physical locations have different identifiers.

[0027] Step S230: Analyze the temporal motion differences of the pixel content on both sides of the shared boundary between adjacent field-of-view projection polygons, estimate the local mapping transformation relationship between the two fields of view based on the distribution of motion differences, and generate a field-of-view mapping transformation matrix that records the correspondence rules of pixel positions between fields of view.

[0028] In one implementation, step S230 may specifically include the following steps S231 to S236: Step S231: Extract pairs of local image patch sequences segment by segment along the shared boundary of the adjacent field-of-view projection polygons. Align the pairs of local image patches from the two fields of view according to the timestamp to form a set of contrast image pairs. Each set of image pairs in the set of contrast image pairs reflects the imaging representation of the same scene under different fields of view.

[0029] Along the shared boundary of the adjacent field-of-view projected polygons, a fixed-size sampling window is moved with a preset step size to extract paired local image patch sequences from the two field-of-views segment by segment. The size of the sampling window is determined based on the resolution of the video segment unit and the field-of-view size. The window is square in shape, and its side length is an adapted value of the pixel density within the field-of-view coverage area. Each sampling window contains a local image patch from the first field of view and a local image patch from the second field of view, which spatially correspond to adjacent regions on both sides of the shared boundary. The extraction operation is performed sequentially from one end to the other along the shared boundary. There is partial overlap between adjacent sampling windows to maintain coverage continuity. The overlap ratio is determined by the ratio of the step size to the window size. Each pair of extracted local image patches is generated at the same sampling location and belongs to the video frames of the two field-of-views at the same timestamp. The paired local image patches from the two field-of-views are aligned and arranged according to the timestamps of their respective video frames to construct a set of contrasting image pairs. Each image pair in the comparison image pair set contains two local image patches. These two local image patches are imaging representations of the same physical scene location under different fields of view, recording the different pixel coordinate positions and different degrees of geometric deformation of the same spatial point in the two field-of-view images.

[0030] Step S232: Perform pixel displacement search for each pair of comparison images, find the optimal pixel matching position in the specified search neighborhood, obtain the horizontal offset direction and vertical offset magnitude of the local position between the two fields of view, and form a set of initial offset correspondence records.

[0031] For each image pair in the set of comparison images, a pixel displacement search based on block matching is performed. Using a local image patch from the first field of view as a reference template, the reference template is translated pixel-by-pixel within the search range formed by the local image patch from the second field of view and its extended neighborhood. At each translation position, a pixel intensity difference metric is calculated between the reference template and the corresponding region in the second field of view. The pixel intensity difference metric is calculated by summing the absolute values ​​of the intensity differences of each color channel; a smaller metric indicates that the two image regions are more consistent in content. The search range is centered on the original corresponding position of the local image patch in the second field of view, expanding outwards in both the horizontal and vertical directions to form a search neighborhood. The range of the search neighborhood is pre-defined based on the maximum possible spatial offset between the two fields of view. After calculating the pixel intensity difference metric at all translation positions within the search neighborhood, the translation position that minimizes the difference metric is selected as the optimal pixel matching position. The displacement of this optimal pixel matching position relative to the center of the search range is the mapping offset of this local position between the two fields of view. This mapping offset comprises two components: the horizontal offset direction and the vertical offset magnitude. The horizontal offset direction indicates the direction of the local image patch's horizontal offset relative to the first field of view in the second field of view, while the vertical offset magnitude indicates the vertical offset distance of the local image patch. The sampling position, horizontal offset direction, and vertical offset magnitude are combined and recorded to form a set of initial offset correspondence records for that local location. After completing pixel displacement searches at all sampling positions along the shared boundary, a series of initial offset correspondence records are obtained.

[0032] Step S233: Summarize the initial offset corresponding records of each segment of the shared boundary, remove abnormal offset records whose abrupt change in offset with the neighboring offset exceeds the tolerance range of the abrupt change, and retain the offset records with continuous offset changes and consistent directions as valid offset records.

[0033] The initial offset records obtained from each sampling position of the shared boundary are arranged in tangential order along the shared boundary, forming a one-dimensional offset record sequence. Anomaly detection is performed on the offset record sequence, calculating the changes in offset direction and magnitude between the offset record at each sampling position and its neighboring offset records. The magnitude of abrupt shifts in neighboring offsets is measured by comparing the offset record at the current sampling position with the offset records at its two adjacent sampling positions, including changes in both horizontal offset direction and vertical offset magnitude. If the difference between the offset record at the current sampling position and its neighboring offset records in both dimensions exceeds a preset abrupt shift tolerance range, the offset record is determined to be an abrupt offset record. The abrupt shift tolerance range is determined based on prior knowledge of the local smoothness of the view mapping relationship in the scene. After removing abrupt offset records from the offset record sequence, the resulting gaps are filled using linear interpolation based on the remaining offset records on both sides. The filled offset records inherit the offset change trend of the neighboring valid offset records. The offset records ultimately retained in the offset record sequence have the characteristics of continuous offset change and good directional consistency; these offset records constitute valid offset records.

[0034] Step S234: Perform spatial interpolation propagation on the effective offset records to spread the offset correspondence at the boundary sampling points to the entire overlapping area of ​​the field of view and the area around the hole, generating a pixel mapping correspondence field covering the overlapping area of ​​the field of view and the area around the hole.

[0035] Spatial interpolation propagation is performed to spread effective offset records defined only at discrete sampling points on the shared boundary to the two-dimensional planar regions on both sides of the shared boundary. Spatial interpolation propagation employs radial basis function interpolation or multi-layer B-spline interpolation methods, using the sampling positions corresponding to each effective offset record as control points, and the offset direction and magnitude at the control points as interpolation target values. This generates a dense pixel mapping correspondence field within the continuous coordinate range of the entire overlapping field of view and the region surrounding the hole. During interpolation, for spatial positions located within the common influence range of multiple control points, the interpolation result is determined by a weighted average of the distances of each control point to that spatial position. Control points closer to each other contribute more to the interpolation result, and the weighting is modulated using a Gaussian kernel function to decrease with distance. The overlapping field of view is determined by the intersection of two field-projected polygons. The region surrounding the hole refers to the area extending outwards from the boundary of the isolated hole region; the extension distance is determined by a combination of the spatial influence radius of the effective offset record and the hole size. After interpolation propagation is completed, at any pixel coordinate position in the overlapping area of ​​the field of view and the area surrounding the hole, the corresponding mapping relationship from the first field of view pixel coordinate to the second field of view pixel coordinate can be queried, forming a two-dimensional dense pixel mapping correspondence field.

[0036] Step S235: Construct a mapping transformation matrix from the pixel coordinates of the first field of view to the pixel coordinates of the second field of view through the distribution structure of the pixel mapping correspondence field. The mapping transformation matrix has local adaptability that changes continuously with spatial position in the area surrounding the hole.

[0037] The pixel mapping correspondence field reflects the mapping relationship between the first and second field-of-view pixel coordinates at various spatial locations. A transformation matrix representation describing this mapping relationship is extracted from the pixel mapping correspondence field. For spatial locations where the mapping relationship tends to be globally consistent within the overlapping field of view, a homography transformation matrix is ​​directly fitted. The parameters of the homography transformation matrix are estimated by inputting multiple sets of corresponding coordinates within that region into a direct linear transformation solver. For the region surrounding the hole, since the mapping relationship is more significantly affected by the geometric differences between fields of view and the hole deformation closer to the hole boundary, a piecewise affine transformation matrix set that continuously varies with spatial location is used to describe the mapping relationship in this region. The piecewise affine transformation matrix set is constructed by dividing the region surrounding the hole into several spatial grid cells. Within each grid cell, a local affine transformation matrix is ​​fitted based on the mapping relationship in the pixel mapping correspondence field falling within that cell. The affine transformation matrices between adjacent grid cells achieve a continuous transition through matrix parameter interpolation at the cell vertices. The resulting set of mapping transformation matrices exhibits local adaptability that changes continuously with spatial location within the region surrounding the cavity. The closer the spatial location is to the cavity boundary, the stronger the ability of the mapping transformation matrix to model local deformation.

[0038] Step S236: Bind the completed mapping transformation matrix with the shared boundary segment identifier and hole identifier on which the matrix was generated, and store it in the field-of-view mapping relationship repository.

[0039] For each completed mapping transformation matrix, the shared boundary segment identifier and hole identifier used in its generation process are recorded to establish a ternary binding relationship. The shared boundary segment identifier indicates the two field-of-view regions to which the mapping transformation matrix applies and the specific shared boundary segment location, while the hole identifier indicates the hole region associated with the mapping transformation matrix. This ternary binding relationship is stored in an inter-field-of-view mapping relationship repository, which is a key-value storage structure with the shared boundary segment identifier and hole identifier as a joint index. This repository supports the rapid retrieval of all inter-field-of-view mapping transformation matrices adjacent to the hole boundary based on the hole identifier in subsequent steps, as well as the retrieval of the pixel mapping correspondence between two specific fields of view at a specific boundary segment based on the shared boundary segment identifier. Each record in the repository also includes a timestamp of the time used to generate the mapping transformation matrix, in order to track the evolution of the mapping relationship over time.

[0040] Step S240: Arrange the global element splicing layout in time sequence, compare the position and shape changes of the hole regions corresponding to the hole markers in the global element splicing layouts of adjacent timestamps, extract the hole boundary drift trajectory and hole deformation direction, and generate a record of the dynamic evolution trend of the hole.

[0041] The global field-of-view stitching layouts generated at each time stamp are arranged in ascending order of time stamp value, forming a time series of global element stitching layouts. A hole-by-hole comparison is performed on the global element stitching layouts corresponding to two adjacent time stamps in the time series. The comparison is based on the continuity of the hole markers, i.e., the change in the geometric state of the corresponding hole region for the same hole marker between two consecutive time stamps. For each hole marker that exists in both consecutive time stamps, the polygonal contours of the hole region at the previous time stamp and the next time stamp are extracted. Shape matching analysis is performed on the two polygonal contours to calculate the drift trajectory of the hole boundary and the deformation direction of the hole. The hole boundary drift trajectory refers to the displacement vector of the geometric center of the hole region between two time stamps. This displacement vector is recorded as the starting and ending coordinates of the drift trajectory, as well as the direction and length of the displacement vector. The direction of cavity deformation is extracted by comparing the local expansion or contraction scales of two consecutive polygonal contours in the circumferential direction. For each boundary segment of the polygonal contour, the change in normal distance between the corresponding segment at the previous time step and the corresponding segment at the next time step is calculated. A positive change in normal distance indicates that the boundary segment is expanding outward, and a negative change in normal distance indicates that the boundary segment is contracting inward. The changes in normal distance of all boundary segments and their corresponding expansion or contraction markers are summarized to form a spatial distribution record describing the direction of cavity deformation. The cavity boundary drift trajectory and cavity deformation direction are associated and stored with the cavity identifier and the corresponding timestamp segment to generate a record of the dynamic evolution trend of the cavity.

[0042] Step S250: Associate the dynamic evolution trend record of the hole with the field-of-view mapping transformation matrix around the corresponding hole to obtain the pixel migration path indication that maps adjacent field-of-view pixels to the inside of the hole at the hole boundary. The pixel migration path indication records the mapping direction sequence and transformation amplitude distribution.

[0043] In one implementation, step S250 may specifically include the following steps S251 to S256: Step S251: Obtain the cavity boundary drift trajectory in the cavity dynamic evolution trend record at the current timestamp, infer the expansion and contraction direction of the cavity boundary from the current timestamp to the next timestamp based on the drift trajectory, and determine the active boundary segment of the cavity boundary change.

[0044] The drift trajectory of the cavity boundary at the current timestamp is extracted from the dynamic evolution trend record of the cavity. The trajectory records the displacement vector of the cavity's geometric center. Using the direction of the displacement vector as a reference direction, the boundary segments of the cavity polygon are divided into expansion direction boundary segments and contraction direction boundary segments according to the angle between their normal vector and the direction of the displacement vector. Specifically, if the angle between the normal vector and the displacement vector of a boundary segment is less than a preset angle division threshold, the boundary segment is determined to be located in the expansion direction of the cavity; if the angle between the normal vector and the opposite vector of the displacement vector is less than the angle division threshold, the boundary segment is determined to be located in the contraction direction of the cavity; if the normal vector and the displacement vector are nearly perpendicular, the boundary segment maintains its original position. For boundary segments along the expansion direction, several boundary segments with the smallest angle between the normal and the displacement vector and a segment length exceeding a preset length are extracted. These boundary segments are identified as active boundary segments for hole boundary changes. These active boundary segments represent the leading edge of the hole during outward expansion and will serve as the primary source boundary for pixel injection during pixel migration. Similarly, for boundary segments along the contraction direction, boundary segments with the smallest angle between the normal and the displacement vector are selected as active contraction boundary segments. These active contraction boundary segments serve as the pixel outflow boundary during pixel migration.

[0045] Step S252: Identify the effective field of view region adjacent to the active boundary line segment, retrieve the field-of-view mapping transformation matrix between the field of view region and the adjacent field of view on the other side of the hole, and extend the scope of the mapping transformation matrix to the coordinate range to be filled inside the hole.

[0046] For each active boundary segment, an effective field of view region spatially adjacent to the active boundary segment is retrieved in the global pixel stitching layout. This effective field of view region is covered by a field of view projection polygon, and its boundary shares a spatial contact relationship with the active boundary segment. In the field-to-field mapping relationship repository, the field of view identifier of the effective field of view region and the field of view identifier of the adjacent field of view on the other side of the active boundary segment are used as joint search conditions to retrieve the corresponding field-to-field mapping transformation matrix. This field-to-field mapping transformation matrix originally defines the pixel coordinate correspondence between these two adjacent fields of view in the shared boundary region, and its effective scope is limited to the overlapping area or boundary neighborhood between the two fields of view. The scope of the mapping transformation matrix is ​​spatially extended to the range of coordinates to be filled inside the hole, located on the side of the active boundary segment facing the hole. This range of coordinates to be filled is formed by the active boundary segment pushing inward along the hole expansion direction into the area inside the hole contour. During the domain extension process, the transformation parameters of the mapping transformation matrix remain unchanged at the active boundary line segment. The transformation parameters are adjusted spatially inward along the expansion direction according to the direction of cavity deformation. The adjustment method is based on the relationship between the change of normal distance and position of each boundary line segment in the cavity deformation direction record and linear extrapolation.

[0047] Step S253: Perform vector correction on the mapping transformation matrix according to the contour deformation direction of the cavity, so that the mapping direction is consistent with the cavity expansion direction, and generate an adaptive mapping transformation matrix after direction correction.

[0048] The cavity deformation direction at the current timestamp is extracted from the cavity dynamic evolution trend record. The cavity deformation direction describes the local expansion or contraction scale of each boundary segment of the cavity contour at the next moment. For the mapping transformation matrix extending into the coordinate range to be filled inside the cavity, the mapping direction component of the mapping transformation matrix is ​​vector-corrected according to the cavity deformation direction at the cavity contour position corresponding to the coordinate position it acts on. The specific operation of vector correction is as follows: the mapping vector defined by the mapping transformation matrix from the source pixel coordinates to the target pixel coordinates is decomposed into a component along the tangent of the cavity boundary and a component along the normal of the cavity boundary. The normal component is compared with the rate of change of the normal distance at the corresponding position in the cavity deformation direction. If the two directions are inconsistent, the normal expansion direction in the cavity deformation direction is used as the reference to correct the direction of the normal component of the mapping vector. The correction magnitude is determined by the expansion scale recorded in the cavity deformation direction at that position. The tangential component remains unchanged or is scaled proportionally to accommodate the corrected normal component, ensuring that the corrected mapping vector maintains the original geometric correspondence between the fields of view and conforms to the actual direction of hole expansion. The mapping transformation matrix after direction correction is denoted as the adaptive mapping transformation matrix, which can adaptively adjust the mapping direction at different locations on the hole boundary according to the direction of hole deformation.

[0049] Step S254: Extract the texture direction information of the original pixels on both sides of the hole boundary, and fine-tune the transformation amplitude of the adaptive mapping transformation matrix through the texture direction information, so that the mapping intensity difference between adjacent layers along the main texture direction is less than the mapping intensity difference between adjacent layers along the texture normal, and generate the texture adaptive mapping transformation matrix.

[0050] At the hole boundary, texture direction information of the original pixel neighborhood located on both sides of the hole boundary is extracted from video frames of adjacent effective field of view regions. Texture direction information is obtained by calculating the gradient structure tensor within the pixel neighborhood. The gradient structure tensor consists of the partial derivatives of pixel brightness in the horizontal and vertical directions. Eigenvalue decomposition is performed on this tensor; the direction of the eigenvector corresponding to the largest eigenvalue is the principal texture direction at that location, and the direction orthogonal to this principal texture direction is the texture normal. The transformation amplitude of the adaptive mapping transformation matrix at each spatial level inside the hole is fine-tuned based on the principal texture direction and texture normal at the corresponding level. The fine-tuning principle is to ensure that the difference in mapping intensity between adjacent levels in the principal texture direction is less than the difference in mapping intensity between adjacent levels in the texture normal direction. Mapping intensity refers to the length of the output vector after the mapping transformation matrix acts on a unit length vector. The mapping intensity difference is kept small in the main direction of the texture, so that the pixel migration results along the main direction of the texture transition smoothly between layers, maintaining the continuity of the texture structure; the mapping intensity difference is kept at the original level or moderately amplified in the normal direction of the texture to reflect the reasonable transition along the normal direction during the hole filling process. The fine-tuned mapping transformation matrix is ​​denoted as the texture adaptive mapping transformation matrix.

[0051] Step S255: Based on the distance from the hole boundary pixel to the hole filling position, the mapping intensity of the texture adaptive mapping transformation matrix is ​​set in layers from strong to weak to obtain a pixel migration path indicator that migrates adjacent field pixels to the hole at the hole boundary. The pixel migration path indicator contains a set of transformation matrices distributed in layers.

[0052] Using the hole boundary as a baseline, the interior of the hole is divided into several layers based on the vertical distance from the pixel to the hole boundary. The spacing between layers is adaptively set according to the size of the hole region, ensuring uniform layering within the hole in the depth direction. For each layer, the mapping intensity adjustment factor of the texture adaptive mapping transformation matrix corresponding to that layer is set based on the average distance from all pixel positions within that layer to the hole boundary. The closer the layer is to the hole boundary, the larger its mapping intensity adjustment factor, meaning a stronger transformation amplitude, and the source pixel is mapped to a position closer to the boundary. The closer the layer is to the interior of the hole, the smaller its mapping intensity adjustment factor, resulting in a weaker transformation amplitude, allowing the mapped position of the source pixel to gradually transition to the original spatial structure inside the hole. The decrease in mapping intensity can follow a linear function or a smooth nonlinear function. The nonlinear function ensures that the mapping intensity changes gradually near the boundary and rapidly decays to the baseline level near the hole center. Each layer's texture adaptive mapping transformation matrix, adjusted by the mapping intensity adjustment factor, constitutes the pixel migration transformation matrix for that layer. All pixel migration transformation matrices are arranged hierarchically from the boundary to the interior, forming a layer-distributed set of transformation matrices. This layer-distributed set of transformation matrices, together with the sampling position information on the hole boundary, constitutes the pixel migration path indicator. The pixel migration path indicator simultaneously records the mapping direction sequence and transformation amplitude distribution of each pixel position within each layer.

[0053] Step S256: Bind the generated pixel migration path indicator with the hole identifier and the current timestamp, and add it to the attribute record of the corresponding hole node in the dynamic field of view topology evolution map to complete the update of hole association information.

[0054] The pixel migration path indicator, hole identifier, and current timestamp generated in step S255 are ternarily bound to form a hole association information record. In the dynamic field-of-view topology evolution map, the corresponding hole node is located by the hole identifier, and the pixel migration path indicator bound to the current timestamp is added or updated in the attribute record of the hole node. The attribute record also stores the hole dynamic evolution trend record of the hole node, the list of associated inter-field-of-view mapping transformation matrices, and the global surface element stitching layout context information under the current timestamp. The attribute record of the hole node is continuously added and updated as the timestamp advances, forming a complete attribute history of the hole in the time dimension, providing a retrieval basis for performing pixel migration and content compensation in subsequent steps.

[0055] Step S260: Collect all global element splicing layouts, hole identifiers, field-to-field mapping transformation matrices, and hole dynamic evolution trend records under all timestamps, and construct a dynamic field-to-field topology evolution graph with field nodes and hole nodes as elements and mapping transformation relationships as edges.

[0056] For all timestamps within the processing time range, the global pixel stitching layout, hole identifier set, inter-view mapping transformation matrix set, and hole dynamic evolution trend record set are extracted from the processing results of each timestamp. Using the field-view projection polygon and isolated hole regions as basic elements, field-view nodes and hole nodes are established respectively. Field-view nodes store the geometric contour of the corresponding field-view projection polygon, the spatial positioning information of the acquisition device, and the coverage area at the current timestamp. Hole nodes store the hole identifier, geometric contour, hole dynamic evolution trend record, and associated pixel migration path indication for the corresponding isolated hole region. Edge connections are established between nodes. If a field-view node and a hole node are spatially adjacent and an available inter-view mapping transformation matrix exists, a directed edge is established between the field-view node and the hole node, with the edge pointing from the field-view node to the hole node, indicating that pixel content can migrate from this field of view to the interior of this hole. The edge attributes include the identifier of the inter-view mapping transformation matrix, the identifier of the shared boundary segment, and a validity marker of the mapping transformation relationship at the current timestamp. All field-of-view nodes and hole nodes are organized according to spatial adjacency and timestamp evolution relationships to construct a dynamic field-of-view topology evolution graph. The dynamic field-of-view topology evolution graph supports slice queries by timestamp. In the topology snapshot at the current timestamp, the spatial adjacency relationships and pixel mapping transformation relationships between each field of view and each hole can be obtained. The topology snapshots of consecutive timestamps describe the dynamic evolution process of these relationships.

[0057] Step S300: Input the dynamic field of view topology evolution map into the field of view content remapping process, extract the pixel mapping transformation relationship of the area surrounding the blind spot hole contour, perform pixel-by-pixel spatial migration operation on the video frames of adjacent fields of view according to the pixel mapping transformation relationship, generate the remapped image content in the blind spot hole, and smoothly connect the remapped image content with the surrounding original pixels to obtain the blind spot compensation video segment.

[0058] In one implementation, step S300 may specifically include the following steps S310 to S360: Step S310: Based on the hole identifier of the current timestamp in the dynamic field of view topology evolution diagram, locate the pixel migration path indicator associated with the hole, and read the set of transformation matrices distributed by layer and the corresponding radius of action of each layer from the pixel migration path indicator.

[0059] In the field-of-view content remapping process, all hole nodes in the dynamic field-of-view topology evolution graph are traversed time-stamp by time. For each hole node, its hole identifier is extracted, and the hole identifier along with the current timestamp is used as the retrieval key to locate the pixel migration path indicator associated with the hole at the current timestamp in the attribute record of the hole node. From the pixel migration path indicator, a set of transformation matrices distributed by layers is parsed. This set of transformation matrices is arranged in hierarchical order from the hole boundary to the hole interior. Each layer corresponds to a pixel migration transformation matrix obtained by adjusting a texture adaptive mapping transformation matrix with a mapping intensity adjustment factor. At the same time, the effective radius corresponding to each layer is read from the pixel migration path indicator. The effective radius defines the applicable spatial range of the pixel migration transformation matrix of that layer inside the hole, extending from the hole boundary as the starting reference direction inward to the outer boundary of the layer. Layers closer to the hole boundary have smaller effective radii, and layers closer to the hole interior have larger effective radii. The effective radii of all layers are contiguous, jointly covering the entire unfilled area from the hole boundary to the hole interior.

[0060] Step S320: Obtain the pixel source region located outside the hole boundary and within the range of the transformation matrix from the original video frames of the adjacent field of view. Apply the transformation matrix of the corresponding layer to the pixel source region pixel by pixel, calculate the target migration coordinates of the source pixel inside the hole, and assign the color component of the source pixel to the target migration coordinate.

[0061] In one implementation, step S320 may specifically include the following steps S321 to S326: Step S321: Using the hole boundary as a reference line, expand outward to form a pixel selection band that is consistent with the maximum effective radius of the transformation matrix set. Within this pixel selection band, mark the position of the source pixel to be migrated in a row-by-column order and record its original color composition.

[0062] Using the hole boundary as a reference line, the matrix expands outward along the normal direction at each position of the hole boundary. The expansion distance is set to the maximum value of the effective radius of all levels in the transformation matrix set. The resulting strip-shaped region is the pixel selection band. The inner boundary of the pixel selection band completely fits the hole boundary, and the outer boundary is a curve generated by translating the maximum effective radius outward along the normal direction. A continuous and gapless strip region is formed between the inner and outer boundaries. Within this pixel selection band, following the pixel grid arrangement of the video frame, starting from the pixel coordinates of one endpoint of the selection band, the matrix scans pixel by pixel along the row and column directions. Each pixel falling within the pixel selection band is marked, and its row and column coordinates in the video frame, as well as its color component values ​​in each color channel, are recorded. The color component values ​​are recorded as the intensity values ​​of the red, green, and blue channels in the color space representation. All marked pixels constitute the set of source pixels to be migrated, and each source pixel to be migrated has a clear spatial location record and original color component record.

[0063] Step S322: For each source pixel to be migrated, obtain the corresponding transformation matrix according to its layer index, perform spatial transformation operation on the source pixel coordinates through the transformation matrix, and obtain the migration landing point coordinates of the pixel inside the hole. The migration landing point coordinates are recorded as floating-point precision spatial positions.

[0064] For each source pixel to be migrated within the selected pixel band, the shortest distance from the source pixel to the hole boundary is determined. The layer index to which the source pixel belongs is determined based on the effective radius range within which the shortest distance falls. Different shortest distance ranges correspond to different layers, and the boundary values ​​between layers are defined by the effective radius of each layer. In the layer-distributed set of transformation matrices, the corresponding pixel migration transformation matrix is ​​retrieved using the layer index as the key. The transformation matrix is ​​applied to the source pixel coordinates to perform a spatial transformation operation. The spatial transformation operation takes the row and column coordinates of the source pixel on the two-dimensional plane of the video frame as input, and through the mapping transformation of the transformation matrix, outputs the migration landing point coordinates corresponding to the source pixel inside the hole. The spatial transformation operation process is as follows: the source pixel coordinates are represented in homogeneous coordinate form; the transformation matrix is ​​multiplied by the homogeneous coordinate vector to obtain the transformed homogeneous coordinates; then the homogeneous coordinates are normalized to two-dimensional Cartesian coordinates, which are the migration landing point coordinates. Since the transformation matrix contains sub-pixel level floating-point parameters, the calculated migration landing point coordinates contain a fractional part and are recorded in floating-point precision spatial position, retaining sub-pixel precision in the horizontal and vertical directions.

[0065] Step S323: Map the floating-point precision migration landing point coordinates onto the discrete pixel grid inside the hole. When a discrete grid cell receives multiple source pixel mappings, calculate the merging contribution ratio according to the distance of each source pixel to the grid center, and determine the color component of the grid cell using a weighted merging method.

[0066] The interior region of the hole is composed of a discrete pixel grid, where each discrete grid cell corresponds to a pixel position in the hole-compensated image and has integer row and column coordinates. The migration landing point coordinates of each source pixel are mapped from floating-point precision to the discrete pixel grid. This mapping involves including several discrete grid cells surrounding the floating-point coordinates within the influence range of the source pixel. The influence range is a fixed-size kernel window centered on the floating-point coordinates, and each discrete grid cell within this kernel window receives a portion of the color contribution from the source pixel. For each discrete grid cell, there may be multiple source pixel migration landing point coordinates that include it within their influence range. When multiple source pixels are mapped to the same discrete grid cell, the Euclidean distance from the migration landing point coordinates of each source pixel to the center of that discrete grid cell is calculated. The reciprocal of this distance or the Gaussian function value of the distance is used as the merging contribution ratio of each source pixel. The color components of all source pixels participating in the mapping are weighted and merged using the merging contribution ratio. The weighting operation is to multiply the color intensity value of each source pixel by the corresponding merging contribution ratio for each color channel, sum them up, and then divide by the sum of the merging contribution ratios for normalization. The weighted and merged color component value is used as the color component of the discrete grid cell in the migration assignment stage of this layer.

[0067] Step S324: For the boundary grid cells of the discrete pixel grid, extract the color consistency status between them and the adjacent assigned grid cells. If the color of the boundary grid cell is inconsistent with the extrapolation trend of the neighbors, then the boundary grid cell is re-included in the color diffusion correction queue for local color backtracking adjustment.

[0068] The boundary grid cell of a discrete pixel grid refers to a grid cell located inside a hole that is adjacent to or close to the edge of the hole, and is adjacent to the original pixel outside the hole boundary. For these boundary grid cells, their color component values ​​are extracted, and the color component values ​​of their adjacent pixels outside the hole boundary that already have the original color values ​​are obtained. Simultaneously, the color component values ​​of the boundary grid cell's neighboring assigned grid cells in the direction towards the hole's interior are also obtained. Based on the color value change trends of adjacent pixels and their internal neighboring assigned grid cells, an extrapolation trend line for color change is constructed. This extrapolation trend line reflects the expected continuous change in color channel intensity along the hole boundary towards the interior. The actual color component value of the boundary grid cell is compared with the expected color value extrapolated from the trend line at that grid cell location. If the difference between the two in any color channel exceeds the color consistency threshold, the boundary grid cell's color is determined to be inconsistent with the adjacent extrapolation trend. Boundary grid cells determined to be inconsistent are re-added to the color diffusion correction queue, awaiting subsequent local color backtracking adjustment processing. Local color backtracking adjustment re-estimates and corrects the color composition of boundary grid cells based on reliable color samples around the boundary grid cells, including external original pixels and internal grid cells that have passed the consistency check, using a color diffusion algorithm.

[0069] Step S325: The internal region of the hole after color assignment is formed into an initial filled image block. At the junction of the filled image block and the surrounding original image, low-frequency smoothing of the boundary is performed. The brightness components of the pixels on both sides of the boundary are linearly gradient transitioned to alleviate the edge contrast introduced by the mapping.

[0070] After calculating the migration landing coordinates and assigning color values ​​to all source pixels to be migrated, and adjusting the color backtracking of inconsistent boundary grid cells, all discrete grid cells inside the hole receive color assignments, forming a complete initial filled image block. A clear boundary line exists between the initial filled image block and the original image surrounding the hole. At this boundary, a transition strip of equal width is extracted on both sides, with the width determined based on the hole size and boundary contrast. For each pixel within the transition strip, its luminance component is extracted. This luminance component is calculated by weighted averaging the intensity values ​​of the pixel across all color channels using luminance transformation. A linear gradient transition model is established between the pixel luminance values ​​on both sides of the boundary along the normal direction perpendicular to the boundary line within the transition strip. This adjusts the luminance components of each pixel within the transition strip to the luminance values ​​determined by the linear gradient transition model, while maintaining the chromaticity information of each pixel during the adjustment process. Linear gradient transition enables a continuous monotonic transition from the brightness of the original pixel outside to the brightness of the initial filled pixel inside within the transition strip, alleviating the edge visual contrast caused by the limitations of the transformation matrix precision and the accumulation of color differences during the mapping process.

[0071] Step S326: The filled image block after boundary low-frequency smoothing replaces the original hole region, which is used as the migration assignment result of the hole in the corresponding layer and passed to the subsequent migration omission check step for further repair.

[0072] The filled image patch with low-frequency smoothing of the boundary is used as part of the remapped image content of the hole at the current timestamp, replacing the original hole region. This filled image patch carries color information migrated pixel by pixel from adjacent fields of view and smoothed at the boundary, constituting the migration assignment result for the corresponding layer. This migration assignment result is passed to subsequent steps for migration omission checking, where hole image positions that have not been assigned values ​​due to discrete coordinate values, occlusion, or insufficient coverage of the transformation matrix are further supplemented and repaired.

[0073] Step S330: Perform migration omission check on the internal image of the hole after the initial assignment, identify the hole image position that has not been assigned due to discrete coordinate values, fill the hole image position by color diffusion processing of the adjacent pixels that have been assigned, and generate the preliminary remapping image content with complete migration.

[0074] After completing the pixel migration assignment operation in step S320, each discrete pixel position inside the hole is traversed and checked to identify hole image positions that are not covered by the migration landing point influence range of any source pixel and therefore have not received a color assignment. The migration omission check is performed by traversing all pixel coordinates on the discrete grid inside the hole and querying whether each pixel coordinate was recorded as a valid assignment in the weighted merging assignment stage of step S323. If a pixel coordinate is not affected by any source pixel or the total merging contribution ratio in the weighted merging is lower than the preset valid assignment threshold, then the pixel is marked as an omission hole image position. All omission hole image positions are collected into an omission image position set. For each omission hole image position in the set, a neighborhood window is defined with it as the center. Neighboring pixels that have received valid color assignments are collected within the neighborhood window, and color estimation is performed on the omission hole image position through color diffusion processing. Color diffusion processing employs either a fast-moving method or a priority-driven pixel-filling algorithm. Starting from the neighborhood boundary of the missed hole image location, it progressively advances towards the interior of the missed hole image location along the distances from the pixel to known pixels in the neighborhood. At each step, the color value of the pixel to be filled is obtained by weighted averaging of the color values ​​of the known pixels according to their distances to the pixel to be filled, until all missed hole image locations are assigned color values. The image inside the filled hole is then merged with the transferred and assigned image portion to generate the preliminary remapped image content with complete transfer.

[0075] Step S340: Extract the pixel color distribution at the boundary of the initial remapped image content and the original pixel color distribution outside the corresponding hole, construct the color transfer mapping curve inside and outside the boundary, and adjust the hue and brightness of the pixels at the boundary of the initial remapped image content through the color transfer mapping curve to reduce the distribution difference of the color channels on both sides of the boundary to within the preset tolerance range.

[0076] In one implementation, step S340 may specifically include the following steps S341 to S346: Step S341: Collect pixel samples inside the boundary and pixel samples outside the boundary at equal intervals along the boundary line between the content of the initial remapped image and the original image outside the hole. Record the intensity value of each sample in each color channel and its normal distance and tangential position relative to the boundary line to form a set of cross-boundary color sample pairs.

[0077] Sampling points are laid out along the boundary line between the initial remapped image content and the original image outside the hole, with a preset equal-interval sampling step size. The sampling step size is set according to the total length of the boundary line and the complexity of color changes. At each sampling point, an internal pixel sample is taken from the boundary line inward, and an external pixel sample is taken from the boundary line outward. The normal distance between the internal and external pixel samples and the boundary line is the same to maintain the spatial symmetry of the samples. For each pixel sample, its intensity value in each color channel is recorded, as well as its normal distance relative to the boundary line. Positive values ​​indicate the internal direction, and negative values ​​indicate the external direction. The tangential position of the sample on the boundary line is also recorded, represented by the arc length parameter along the boundary line from the preset starting point to the sampling point. The paired internal and external pixel samples collected at all sampling points together form a cross-boundary color sample pair set. Each sample pair contains the color intensity vector of the internal sample and the color intensity vector of the external sample, as well as the common tangential position and normal distance offset.

[0078] Step S342: Perform color space distribution clustering analysis on the cross-boundary color sample pair set, divide the internal pixel color values ​​into local intervals according to the tangential position, statistically analyze the mapping tendency distribution from internal color to external color in each local interval, and remove sample pairs whose mapping tendency and the offset difference between adjacent intervals exceed a preset consistency threshold to obtain the reliable color correspondence set corresponding to each local interval.

[0079] The sample pairs in the cross-boundary color sample pair set are sorted according to their tangential positions. The entire boundary line is divided into several local intervals along the tangent. The length of each local interval is adaptively determined based on the curvature of the boundary line and the number of sample pairs, ensuring that each interval contains a sufficient number of statistically significant sample pairs. Within each local interval, a mapping tendency distribution from internal pixel color values ​​to external pixel color values ​​is constructed. Specifically, the intensity values ​​of the internal pixel color values ​​in each color channel are used as input spatial coordinates, and the intensity values ​​of the corresponding external pixel color values ​​in each color channel are used as output spatial coordinates. The input-output mapping relationship of all sample pairs is statistically analyzed to form the mapping tendency distribution for that local interval. The mapping tendency distribution of each local interval is compared with the mapping tendency distribution of adjacent local intervals at the interval boundary to determine the continuity of the mapping values ​​and their changes. The offset difference between adjacent intervals is calculated, measuring the magnitude of the difference in mapping output between the two intervals at the boundary. If the offset difference between the mapping tendency of a sample pair's local interval and that of its adjacent intervals at the boundary exceeds a preset consistency threshold, the sample pair is marked as an inconsistent sample pair and removed. The retained sample pairs constitute a reliable color correspondence set for each local interval. The reliable color correspondence set excludes color anomalies caused by sudden changes in local lighting, occlusion, or the passage of moving objects.

[0080] Step S343: For each local interval, construct a piecewise linear mapping curve from the color channel intensity of the inner pixel to the color channel intensity of the outer pixel within the interval using a set of reliable color correspondences, and constrain the mapping curves of adjacent local intervals to maintain consistent mapping values ​​at the interval boundaries, thereby obtaining a set of spatially continuous color transfer mapping curves.

[0081] For each local interval, the color channel intensity values ​​of internal pixels in the trusted color correspondence set are used as input values, and the color channel intensity values ​​of external pixels are used as output values. A piecewise linear mapping curve is constructed channel by channel. The construction process of the piecewise linear mapping curve is as follows: the input value range is divided into several continuous intervals. Within each interval, two adjacent sample data points are connected to form a linear mapping segment. All linear mapping segments are connected end to end to form the piecewise linear mapping curve of the entire channel. The nodes of the piecewise linear mapping curve are determined by the sample points in the trusted color correspondence set sorted by input value. When constructing the piecewise linear mapping curve of each local interval, the output value of the mapping curve at the two tangential boundaries of the interval is constrained to be consistent with the output value of the mapping curve at the corresponding boundary of the adjacent interval, so as to achieve the mapping continuity between intervals. The constraint is implemented by taking the output values ​​of the mapping curves of the two intervals at the tangential boundary position of the adjacent interval. If there is a difference between the two, the difference is eliminated by adjusting the position of the control point of the mapping curve near the boundary of the two intervals in both directions. The adjustment range decreases with the distance from the control point to the boundary. The piecewise linear mapping curves constructed in each local interval and adjusted for continuity constraints are combined to form a set of color transfer mapping curves that are tangentially continuous along the boundary line.

[0082] Step S344: Traverse each pixel within the boundary region of the initial remapped image content, determine the local interval to which the pixel belongs based on the tangential position of the pixel, extract the corresponding mapping curve, input the intensity values ​​of each color channel of the pixel into the mapping curve, and obtain the adjusted pixel color.

[0083] For each pixel within the boundary region of the initially remapped image content, the width of the boundary region is determined based on the influence range of the color transfer mapping curve set, typically a preset distance extending inward from the boundary line into the interior of the initially remapped image content. For each pixel within the boundary region, its projection point on the boundary line is calculated, and the local interval to which the pixel belongs is determined based on the tangential position of the projection point. The piecewise linear mapping curve corresponding to this local interval is retrieved from the color transfer mapping curve set to obtain the input-output mapping relationship for each color channel. The original intensity value of the pixel in each color channel is used as the input value, and input to the corresponding piecewise linear mapping curve for each channel. The curve outputs the mapped intensity value. The mapped intensity values ​​of each channel are combined to obtain the adjusted color of the pixel. After traversal, the colors of all pixels within the boundary region have been adjusted according to their respective mapping curves, and the adjusted pixel colors are more consistent in hue and brightness with the color distribution at the corresponding position in the original image outside the hole.

[0084] Step S345: Calculate the brightness gradient vector of the adjusted boundary region pixels in the cross-boundary direction, and at the same time extract the brightness gradient vector of the corresponding position of the external original image at the junction. Compare the angle between the directions of the two gradient vectors with the preset angle tolerance, and perform compensation correction along the normal direction for the brightness of pixels that exceed the preset angle tolerance.

[0085] For each pixel within the adjusted boundary region, its luminance gradient vector is calculated in the cross-boundary direction, which is the normal direction of the boundary line at the pixel's projection point. The luminance gradient vector is calculated from the luminance difference between the preceding and following pixels along the normal direction. Simultaneously, the normal luminance gradient vector of the external original image at the boundary corresponding to the pixel's projection point is extracted. This extraction method involves calculating the luminance difference between adjacent pixels at the boundary along the same normal direction in the external original image. The angle between the two luminance gradient vectors is calculated and compared to a preset angle tolerance. If the angle is less than or equal to the preset angle tolerance, it indicates that the adjusted boundary pixel's luminance change trend is consistent with the external original image, and the adjustment result for that pixel is retained. If the angle exceeds the preset angle tolerance, it is determined that the pixel has an inconsistent luminance change trend, and the luminance component of that pixel is compensated and corrected along the normal direction. The compensation correction operation involves adjusting the brightness value of the pixel along the normal direction so that the brightness gradient direction formed by the adjusted pixel and the adjacent external original pixel in the normal direction is close to the brightness gradient direction of the external original image at the boundary. The compensation amount is determined by the difference between the directional angle and the directional tolerance, as well as the normal distance of the pixel to the boundary line.

[0086] Step S346: Combine the pixels in the boundary region that have undergone brightness gradient correction with the unadjusted pixels inside the initial remapped image content to generate adjusted remapped image content where the boundary color transition maintains spatial gradient continuity with the original external image.

[0087] All pixels within the boundary region that have undergone color transfer mapping curve adjustment and brightness gradient correction are combined with pixels in the initial remapped image content that are located inside the boundary region and have not undergone any boundary color adjustment. During the combination, the spatial position of each pixel remains unchanged; pixels in the boundary region use the adjusted color values, while pixels in the inner region retain the color values ​​generated during the initial remap. No additional processing is applied at the junction between the boundary region and the inner region, because the color adjustment only applies to the boundary region. The inner region, being far from the boundary, is unaffected by the boundary color difference, and the natural transition between the two is ensured by the brightness gradient correction in step S345. The combined image content is the adjusted remapped image content, which achieves a continuous color transition with a spatial gradient at the boundary with the original image outside the hole.

[0088] Step S350: Detect whether the change in the main direction angle of the texture of adjacent pixels at the boundary of the adjusted remapped image content exceeds the preset angle jump threshold. If so, perform local micro-shifting of the migrated pixels along the break line segment, realign the texture structure, eliminate structural break marks, and obtain remapped image content that maintains structural connection with the surrounding original scene.

[0089] At the boundary of the remapped image content, the principal orientation angles of the texture directions of adjacent pixels on both sides of the boundary are extracted. The extraction of the principal orientation angles is accomplished by calculating the gradient structure tensor within a local neighborhood window centered on the pixel. Eigenvalue decomposition is performed on this tensor, and the orientation angle of the eigenvector corresponding to the largest eigenvalue in the two-dimensional plane is the principal orientation angle. The principal orientation angles of the texture directions of internal pixels are compared with those of the corresponding external pixels at pixel-by-pixel positions along the boundary line, and the absolute difference between the two angles is calculated as the change in the principal orientation angle. If the change in the principal orientation angle at a certain pixel position exceeds a preset angle jump threshold, a texture structure break is marked at that position. Break segments are extended along the principal orientations of the texture direction on both sides of the break, centered on the marked break position. For the migrated pixels located on the broken line segment inside the hole, a local position micro-shift operation is performed. The micro-shift direction is orthogonal to the broken line segment, adjusting the position of the migrated pixel by a sub-pixel displacement in a direction that aligns its texture principal direction with the texture principal direction of the corresponding external pixel. The displacement amount is determined by the difference between the angle change and the angle jump threshold, as well as the ratio of the pixel's distance to the boundary line. This local position micro-shift operation changes the spatial position of the migrated pixels in the image inside the hole. After the micro-shift, image resampling is used to assign the original color values ​​to the new pixel grid positions. After texture alignment, the texture principal directions on both sides of the boundary tend to be consistent, structural break marks are eliminated, and the resulting remapped image content maintains structural continuity with the surrounding original scene.

[0090] Step S360: Use the adjusted remapped image content as the blind zone compensation image frame for the hole at the current timestamp, keep it synchronized with the original video frame at the corresponding timestamp, and integrate all the blind zone compensation image frames of the holes to generate a blind zone compensation video clip.

[0091] For each hole node in the current timestamp dynamic field of view topology evolution graph, the field of view content remapping process described in steps S310 to S350 is executed to obtain the adjusted remapped image content for the corresponding hole. This remapped image content is used as the blind zone compensation image frame for that hole at the current timestamp, and its hole identifier and timestamp are recorded. All blind zone compensation image frames for holes are integrated into the same video frame canvas according to their spatial relationship to generate a complete blind zone compensation image frame corresponding to the current timestamp. Multiple consecutive timestamps of complete blind zone compensation image frames are arranged sequentially on the timeline to form a blind zone compensation video segment, with each frame carrying a timestamp and synchronized with the original video frame at the same timestamp.

[0092] Step S400: Perform panoramic fusion on the blind spot compensation video clip and the original video clip. In the overlapping area of ​​adjacent fields of view, generate dynamic fusion weights based on the acquisition confidence metric and remapping confidence metric of each pixel source. Based on the dynamic fusion weights, perform weighted synthesis on the pixels in the overlapping area to obtain a panoramic video stream that maintains spatiotemporal continuity.

[0093] In one implementation, step S400 may specifically include the following steps S410 to S460: Step S410: Using a unified global reference space as the alignment benchmark, assign each compensated image frame in the blind zone compensation video clip and each original video frame in the original video clip under the same timestamp to the corresponding position on the panoramic canvas, extract the overlapping polygonal region at the boundary of adjacent fields of view, and record at least two source identifiers corresponding to each pixel position in the overlapping polygonal region.

[0094] Using a unified global reference space as the spatial reference for the panoramic canvas, the panoramic canvas is a two-dimensional pixel plane that outputs the panoramic video stream, possessing a defined coverage area and pixel resolution within the unified global reference space. For each compensated image frame in a blind-spot compensation video segment at the same time stamp, based on the geometric position of the hole identifier corresponding to that compensated image frame in the global pixel stitching layout, the pixel content of that compensated image frame is mapped to the corresponding position on the panoramic canvas through affine or perspective transformation. For each original video frame in an original video segment at the same time stamp, based on the geometric position of its corresponding field-of-view projection polygon in the global pixel stitching layout, its pixel content is mapped to the corresponding position on the panoramic canvas using the same spatial mapping method. After all frame content is mapped, overlapping polygonal regions are formed on the panoramic canvas between adjacent field-of-view projection polygons and between field-of-view projection polygons and compensated image frames. The shape of the overlapping polygonal regions consists of the overlapping portions of the coverage areas of two adjacent sources on the panoramic canvas. All overlapping polygonal regions are extracted, and all source identifiers corresponding to each pixel position within the overlapping polygonal regions are recorded, with each source identifier pointing to an original video frame or a compensated image frame.

[0095] Step S420: For each pixel location, obtain the acquisition confidence metric or remapping confidence metric corresponding to its source identifier, input the confidence metrics of each source into the joint confidence evaluation function, determine the fusion contribution ratio between different sources at the pixel location by the ratio of the confidence metrics of each source, and generate pixel-level initial fusion weights.

[0096] For each pixel location within the overlapping polygonal region, the confidence metric is obtained based on the type of its source identifier. If the source identifier corresponds to a pixel in the original video frame, the acquisition confidence metric for that pixel in the original video frame is obtained. The acquisition confidence metric reflects the reliability of that pixel during the original acquisition process, and its value is calculated comprehensively based on factors such as the spatial resolution of the acquisition device at that pixel location, the signal-to-noise ratio, the lens distortion correction residual, and the distance from that pixel to the boundary of the field-of-view projection polygon. If the source identifier corresponds to a pixel in the blind-spot compensation image frame, the remapping confidence metric for that pixel is obtained. The remapping confidence metric reflects the reliability of that pixel after pixel migration and remapping processing, and its value is calculated comprehensively based on factors such as the condition number of the mapping transformation matrix that generated that pixel, the distance from that pixel to the hole boundary, the number of color diffusions involved in the mapping process, and the mapping intensity adjustment factor of the texture adaptive mapping transformation matrix. The confidence metrics of all sources at a certain pixel location are input into the joint confidence evaluation function. The joint confidence evaluation function calculates the normalized ratio of the confidence metrics of each source and outputs the fusion contribution ratio between different sources at that pixel location. This fusion contribution ratio is the pixel-level initial fusion weight.

[0097] Step S430: Based on the initial pixel-level fusion weights, with the boundary of the polygonal region of the overlapping zone as a constraint, the initial pixel-level fusion weights are globally optimized and adjusted along the direction from the boundary to the interior of the overlapping zone. A fusion weight field is constructed with the source confidence at the boundary of the overlapping zone as the anchor point constraint. The fusion weight field satisfies the spatial smoothness condition inside the overlapping zone and is aligned with the anchor point confidence at the boundary of the overlapping zone.

[0098] In one implementation, step S430 may specifically include the following steps S431 to S436: Step S431: Extract the boundary distribution of pixel-level initial fusion weights on each boundary line segment of the overlapping polygonal region, mark the pixels in the boundary distribution that are consistent with the proportion of confidence metrics from each source as anchor points, and obtain the boundary constraint values ​​of the remaining pixels on the boundary line segment through interpolation of the anchor points.

[0099] For each boundary segment of the overlapping polygonal region, this boundary segment serves as the boundary between the overlapping region and the non-overlapping region or another overlapping source region. Pixel-level initial fusion weight values ​​generated in step S420 are extracted pixel-by-pixel along this boundary segment to form a boundary distribution sequence. Simultaneously, the source confidence metric corresponding to each pixel position on the boundary segment is obtained. For the boundary segment portion located within the original field of view coverage, the source confidence metric is the acquisition confidence metric of the original video frame; for the portion located at the boundary of the blind zone compensation region, the source confidence metric can be the remapping confidence metric or jointly determined by the confidence metrics of adjacent fields of view. Pixels in the boundary distribution whose pixel-level initial fusion weight values ​​are completely consistent with the fusion contribution ratio calculated by each source confidence metric according to the joint confidence evaluation function are marked as anchor points. The fusion weights at these anchor points are considered accurate and reliable. For the remaining pixels on the boundary segment not marked as anchor points, the boundary constraint values ​​of these pixels are calculated by linear interpolation or spline interpolation along the boundary segment, using the fusion weight values ​​at two adjacent anchor points as endpoints.

[0100] Step S432: Using the boundary constraint value of the polygonal region of the overlapping zone as the boundary condition, establish the objective function of the fused weight field in the inner region of the overlapping zone. The objective function includes the second-order spatial difference term of the weight field and the boundary deviation penalty term. The second-order spatial difference term constrains the smoothness of the change of the weight field in the inner region, and the boundary deviation penalty term constrains the weight field to approach the boundary constraint value at the boundary.

[0101] A fusion weight field is defined within the interior region of the overlapping polygonal region. This fusion weight field is a discrete scalar field, with a weight value assigned to each pixel grid position within the interior region. An objective function is established for the fusion weight field, comprising two sub-terms. The first term is a second-order difference term in the weight field space. This term calculates the sum of squares of the second-order differences of the weight field at all pixels within the interior region in both the horizontal and vertical directions. The second-order difference reflects the degree of local curvature variation in the weight field; minimizing this term constrains the fusion weight field to exhibit a smooth variation within the interior region. The second term is a boundary deviation penalty term. This term calculates the sum of squares of the deviations between the fusion weight field values ​​at each pixel along the boundary line segment and the boundary constraint values ​​obtained in step S431. Minimizing this term constrains the fusion weight field to approximate the boundary constraint values ​​at the boundaries. The two sub-terms are weighted and combined using predetermined balance coefficients. These balance coefficients control the trade-off between interior smoothness and boundary fidelity, and are selected based on the geometric width of the overlapping band and the degree of spatial variation in the confidence metric.

[0102] Step S433: Iteratively solve the objective function by updating the weight values ​​of the internal pixels one by one, so that the objective function value drops below the preset convergence index, and obtain an initial fusion weight field that satisfies the boundary constraints and has a smooth internal transition.

[0103] The objective function established in step S432 is solved iteratively using either the successive over-relaxation iteration method or the conjugate gradient method. In each iteration, the gradient information of the objective function is calculated based on the current value of the fused weight field. The weight values ​​at each pixel position in the internal region are updated in the direction that makes the objective function value decrease. The pixel weight values ​​on the boundary are reset to the boundary constraint values ​​after each iteration to maintain the boundary conditions. The iteration continues, and the relative change in the objective function value is calculated after each iteration. When the relative change is lower than the preset convergence index, the iteration process terminates. The fused weight field at this point is the initial fused weight field that satisfies the boundary constraints and has a smooth internal transition.

[0104] Step S434: Detect whether there is a local oscillation region in the overlap zone of the initial fused weight field. By extracting the curvature distribution of the contour lines of the weight field, the region where the curvature of the contour lines exceeds the preset contour line smoothing threshold is marked as the oscillation region. The weight field in the oscillation region is locally re-smoothed, and the weight values ​​in the oscillation region are replaced so that they change monotonically along the contour line normal.

[0105] Weight contour lines are extracted from the initial fused weight field. These contour lines are curves connecting pixels with equal weight values. The curvature of each contour line is calculated point-by-point, obtained by the rate of change of the tangent at that point. The curvature distribution is compared to a preset contour smoothing threshold. Continuous regions with curvature exceeding the threshold are marked as local oscillation zones, where the weight field exhibits spatial non-monotonic fluctuations. Local resmoothing is performed on each oscillation zone. This resmoothing method involves extracting a sequence of weight value changes along the contour line normal within the oscillation zone. If this sequence exhibits non-monotonic fluctuations, median filtering or local weighted regression smoothing is applied, replacing the original weight values ​​with the smoothed values. This ensures that the replaced weight values ​​exhibit a strictly monotonically changing or monotonically changing trend along the contour line normal.

[0106] Step S435: Compare the fused weight field after oscillation processing with the original pixel-level initial fused weight in the non-oscillation region. If the average deviation between the two weight fields in the block is lower than the preset deviation tolerance limit, the local detail features of the initial fused weight field are retained first. If the average deviation exceeds the preset deviation tolerance limit, the weight field after oscillation processing is used to maintain global smoothness.

[0107] The overlapping region is divided into several non-overlapping blocks. The size of each block is determined based on the spatial variation scale of the weight field and the geometric width of the overlapping region, typically using a regular rectangular grid. For each block, the average deviation between the oscillating fused weight field and the original pixel-level initial fused weight field is calculated. The average deviation is obtained by averaging the absolute values ​​of the differences between the two weight values ​​pixel by pixel. The average deviation is compared with a preset deviation tolerance limit. If the average deviation is lower than the tolerance limit, it indicates that the local detail features within the block are highly reliable, and the block ultimately adopts the weight values ​​of the original pixel-level initial fused weight field to preserve the local detail features of the fused weights. If the average deviation exceeds the tolerance limit, it indicates that there is a serious spatial inconsistency problem within the block, and the block ultimately adopts the weight values ​​of the oscillating fused weight field to maintain global smoothness. After the selection is completed for each block, the weight values ​​of each block are merged to generate the final fused weight field. The final fused weight field is aligned with the anchor confidence metric at the boundary of the overlap band, maintains spatial smoothness inside the overlap band, and autonomously selects the emphasis between local details and global smoothness based on the block comparison results.

[0108] Step S436: Merge the weight field values ​​of each block to generate a final fused weight field that aligns with the anchor point confidence metric at the overlap zone boundary, maintains spatial smoothness within the overlap zone, and autonomously switches between local details and global smoothness. The specific operations of this step are completed concurrently with the block selection and merging process in step S435; the overall effect is summarized here.

[0109] Step S440: The source color components at each pixel location within the overlapping polygonal region are weighted and synthesized using the fusion weight field to obtain the initial fused pixel color. A color difference compensation term between the sources is then added. The color difference compensation term is generated by the weighted deviation of each source color component at that pixel location and the gradient of the fusion weight field.

[0110] For each pixel location within the overlapping polygonal region, extract the color component values ​​from all its sources in the original video frame or compensated image frame, as well as the fusion weight values ​​assigned to each source at that pixel location by the final fusion weight field generated in step S430. Weight the color components of each source using the fusion weight values. The weighting method involves multiplying the color component value of each source by the corresponding fusion weight value for each color channel, and then summing the weighted results of all sources to obtain the weighted composite color. Based on this, a color difference compensation term is added. The color difference compensation term is generated by calculating the deviation of each source color component from the weighted composite color in each color channel at that pixel location, i.e., subtracting the weighted composite color from each source color component to obtain the deviation amount. Then, the deviation amount is weighted and summed using the fusion weight values ​​of each source to obtain a weighted color deviation vector. This color deviation vector is then multiplied by the gradient vector of the fusion weight field at that pixel location. The gradient vector consists of the first-order differences of the fusion weight field in the horizontal and vertical directions. The result of the multiplication is added to the weighted composite color as the color difference compensation term to obtain the initial fused pixel color.

[0111] Step S450: On the adjacent transition strip between the overlapping polygonal region and the non-overlapping region, extend the fusion weight field continuously along the normal of the transition strip, push the fusion weight field outward to the outer boundary of the non-overlapping region, and trim it with the minimum width of the transition strip to eliminate the weight field cutoff boundary between the fusion region and the non-fusion region.

[0112] In one implementation, step S450 may specifically include the following steps S451 to S456: Step S451: Identify the boundary transition curve between the overlapping polygonal region and the non-overlapping region, and extend a transition strip with a preset initial width along the normal of the boundary transition curve to the non-overlapping region side. The width of the transition strip is inversely proportional to the gradient magnitude of the fused weight field within the overlapping region.

[0113] Identify all boundary transition curves between the overlapping polygonal regions and the non-overlapping regions; these curves represent the outer boundaries of the overlapping regions. For each segment of the boundary transition curve, extend it towards the non-overlapping region along its normal direction at each point. The extension distance serves as the preset initial width of the transition strip. The width of the transition strip is inversely proportional to the gradient magnitude of the fusion weight field near the boundary within the overlapping region. A larger gradient magnitude indicates a more drastic change in the fusion weights near the boundary, requiring a narrower transition strip to control the extrapolation distance. Conversely, a smaller gradient magnitude indicates a smoother change in the fusion weights near the boundary, allowing for a wider transition strip. The gradient magnitude is obtained by calculating the first-order difference component of the fusion weight field near the boundary within the overlapping region along the normal direction.

[0114] Step S452: Establish a weight extrapolation model within the transition strip. The weight extrapolation model is constrained to ensure that the distribution of weight values ​​within the transition strip is consistent with the trend of weight values ​​along the normal direction at the boundary of the overlapping strip. Extrapolate and broadcast the weight values ​​in each normal direction within the transition strip.

[0115] A weighted extrapolation model is established within the transition strip. The constraint of the model is that the distribution of weight values ​​in each normal direction within the transition strip should continue the trend of weight values ​​along the same normal direction at the boundary of the overlapping strip. The operation of extrapolation broadcasting along each normal direction is as follows: on each normal, starting from the boundary pixel of the overlapping strip, extract the weight values ​​of the boundary pixel and several neighboring pixels in the direction inward of the overlapping strip. Fit the variation curve of these weight values ​​along the normal direction to obtain the variation trend function. The variation trend function can be a linear function or a quadratic polynomial function. Based on the variation trend function, the extrapolated weight value at each pixel position along the normal direction within the transition strip is calculated one by one to achieve continuous extrapolation of the weight values.

[0116] Step S453: For the extrapolation result of each normal direction, calculate the rate of change of the weight value in its direction, compare the rate of change with the rate of change of the weight in the corresponding direction inside the boundary of the overlapping zone, and if the relative deviation of the two rate of change exceeds the set continuity deviation limit, then compensate and correct the extrapolated weight value of the normal direction. The correction amount is determined by the relative deviation and the normal distance.

[0117] For each extrapolated weight result sequence along the normal direction, the first-order difference of the sequence along the normal direction is calculated as the weight value change rate. Simultaneously, the weight value change rate along the same normal direction within the overlapping band boundary is extracted by calculating the first-order difference of the weight value sequences of neighboring pixels within the boundary. The extrapolated change rate is compared with the internal change rate, and the relative deviation between the two is calculated. If the relative deviation is lower than or equal to the set continuity deviation limit, it indicates that the extrapolated weight value naturally continues the internal weight change trend, and the extrapolation result does not require correction. If the relative deviation exceeds the continuity deviation limit, it indicates that the weight change trend has deviated during the extrapolation process. In this case, the extrapolated weight value of each pixel along that normal direction is compensated and corrected. The correction amount for each pixel is determined by the relative deviation and the distance of that pixel from the normal to the boundary. The greater the distance, the more the correction amount gradually decreases while following the deviation direction.

[0118] Step S454: Detect whether there are pixel clusters that violate the boundary monotonicity in the weight field after compensation and correction within the transition strip. Identify non-monotonic pixel clusters by checking the monotonicity along the normal direction. Rearrange the local weight values ​​of the non-monotonic pixel clusters and re-interpolate them with the weight value at the boundary of the normal as the starting point and the weight value at the outer boundary of the transition strip as the ending point.

[0119] Within the transition strip, the monotonicity of the compensated weight value sequence is checked along each normal direction. The standard for monotonicity testing is that the weight values ​​should remain non-increasing monotonic along the normal direction from the boundary of the overlapping zone to the interior of the non-overlapping region; that is, the weight values ​​should not fluctuate by rising and then falling or falling and then rising. If a pixel cluster violating monotonicity is identified on a certain normal segment, it is marked as a non-monotonic pixel cluster. For each non-monotonic pixel cluster, a local weight value rearrangement is performed. The rearrangement operation starts with the weight value at the boundary of the overlapping zone on the normal line and ends with a preset target weight value at the outer boundary of the transition strip. The weight values ​​of each pixel position in the non-monotonic pixel cluster are re-interpolated between the start value and the end value according to the principle of equal interval or equal proportion, so that the replaced weight value sequence changes strictly monotonically along the normal direction.

[0120] Step S455: The interpolated values ​​connecting multiple normal directions form a continuous weight field for the transition strip. The weight field is tangentially smoothed along the direction of the transition strip so that the weight value changes between adjacent normals are coordinated with the tangential curvature of the transition curve.

[0121] The interpolation results of weights in all normal directions, after monotonicity testing and local weight value rearrangement, are combined to form a two-dimensional continuous weight field within the transition strip. To prevent tangential discontinuities caused by extrapolation or correction differences between different normals, the weight field is tangentially smoothed along the tangential direction of the transition strip. Tangential smoothing employs one-dimensional Gaussian filtering or bilateral filtering along the tangential direction of the transition curve. The window size of the filter kernel is related to the tangential curvature of the transition curve. A greater curvature indicates a more drastic change in the spatial orientation of the transition curve at that point, and the filter kernel window is appropriately reduced to retain necessary details of weight changes; a smaller curvature indicates a straighter transition curve at that point, and the filter kernel window is appropriately enlarged to enhance the tangential smoothing effect. The tangentially smoothed weight field exhibits coordinated changes between adjacent normals, adapting to the tangential curvature of the transition curve.

[0122] Step S456: With the constraint that no visible seam appears at the splicing point of the overlapping area and the transition strip, the boundary consistency of the transition strip and the overlapping area of ​​the smoothed weight field is checked. After the check passes, the fused weight fields of the transition strip and the overlapping area are spliced ​​together to obtain a complete fused weight field.

[0123] The transition strip weight field, after tangential smoothing, is concatenated with the final fused weight field within the overlapping region at their boundary. Weight values ​​on the transition strip side and the overlapping region side are extracted pixel-by-pixel at the boundary, and the difference between them is calculated. If the difference between consecutive pixels on the boundary exceeds a preset boundary consistency threshold, the weight values ​​of the relevant pixels on the transition strip side are slightly adjusted along the normal direction to reduce the boundary difference until it falls within the boundary consistency threshold range. After the boundary consistency check passes, the transition strip weight field and the overlapping region fused weight field are concatenated into a single data structure. The weight values ​​within the transition strip directly replace the missing definitions in the original non-overlapping regions, forming a complete fused weight field covering the overlapping region and its surrounding transition strips.

[0124] Step S460: Assemble the overlapping region image generated by weighted extrapolation, the original non-overlapping region image, and the remapped region image corresponding to the blind spot compensation video segment into the panoramic canvas, and output the panoramic video stream in time stamp order.

[0125] By performing weighted synthesis and color difference compensation operations in step S440 on all pixels within the overlapping polygonal region and transition strip through a complete fusion weight field, a fused overlapping region image is generated. The fused overlapping region image, the original images of the non-overlapping regions, and the remapped region images corresponding to the blind spot compensation video segments are assembled according to their respective spatial positions on the panoramic canvas to form a complete panoramic video frame. The panoramic video frames with consecutive timestamps on the timeline are output sequentially according to their timestamp order, forming a panoramic video stream that maintains spatiotemporal continuity.

[0126] Step S500: Analyze the distribution pattern of the original acquisition source and content remapping source in different regions of the panoramic video stream, generate spatial mask information to mark the source type of the region, set differentiated encoding and transmission strategies for each region based on the spatial mask information, and encode and distribute the panoramic video stream according to the differentiated encoding and transmission strategies, so that the remapping region obtains encoding quality that can be independently controlled from the original acquisition region.

[0127] In one implementation, step S500 may specifically include the following steps S510-S560: Step S510: Retrospectively record the synthesis source label of each frame pixel during the panoramic video stream generation process, classify the pixels whose synthesis source label is the original acquisition source into the original acquisition area, classify the pixels whose synthesis source label is the remapping source into the content remapping area, and classify the pixels that are in the overlapping zone fusion and have multiple sources into the transition mixing area, thus obtaining a spatial partition map of the three types of regions.

[0128] During the panoramic video stream generation process, each pixel in each frame retains a synthesis source tag record. The synthesis source tag records which original video frame or blind spot compensation image frame the pixel content originated from, and whether the pixel underwent multi-source weighted synthesis during the fusion process in step S400. Tracing back the synthesis source tag record of the first frame or a designated reference frame of the panoramic video stream, spatial division is performed according to the synthesis source tag: pixels whose synthesis source tag indicates a single original acquisition source and has not undergone multi-source weighted synthesis are assigned to the original acquisition area; pixels whose synthesis source tag indicates a single remapping source are assigned to the content remapping area; and pixels whose synthesis source tag indicates that they have undergone multi-source weighted synthesis are assigned to the transition blending area. These three area types form partition boundaries on the panoramic canvas, and these partition boundaries together constitute a spatial division map of the three area types.

[0129] Step S520: Mark the connected components of the spatial partition map, aggregate adjacent pixels of the same type into connected components, filter out fragmented patches with a pixel count lower than the preset scale, and obtain the regularized region type spatial mask information. The spatial mask information includes the polygon outline and type identifier of each region.

[0130] A connected component analysis algorithm is used to label connected components in the spatial partitioning map. This algorithm is performed independently on the original acquisition region, content remapping region, and transitional blending region. It aggregates 4- or 8-adjacent pixels of the same type into a single connected component, assigning each component a unique region number. All connected components are traversed, and the number of pixels contained in each component is counted. Connected components with fewer pixels than a preset size are marked as fragmented patches. These fragmented patches are typically isolated small regions caused by uncertainty or temporary occlusion at the boundary of local pixel source determination. Fragmented patches are reassigned to the adjacent non-fragmented patch connected component with the largest number of pixels, or merged into the region type surrounding the fragmented patch. The polygonal contours of each regularized connected component are extracted using a boundary tracing algorithm, and the polygonal contours are represented as a sequence of vertex coordinates. Each connected component also includes a region type identifier, indicating whether it belongs to the original acquisition region, content remapping region, or transitional blending region. The polygonal contours and type identifiers of all connected components together constitute the region type spatial mask information.

[0131] Step S530: Based on the texture richness of the original acquisition area and the synthesis complexity of the content remapping area, calculate the encoding complexity description for the original acquisition area and the content remapping area respectively. For the transitional mixed area, the encoding complexity description is determined by weighting the complexity of the two areas.

[0132] In one implementation, step S530 may specifically include the following steps S531 to S536: Step S531: Extract the spatial frequency energy distribution of image blocks within the original acquisition area, divide the spatial frequency energy distribution into flat components, detail components, and edge components, and record the texture richness of the area based on the proportion of detail components and edge components in the total energy.

[0133] For the image content in the original acquisition area, a Discrete Cosine Transform (DCT) or Fast Fourier Transform (FFT) is applied to transform the image patch from the spatial domain to the frequency domain, obtaining the spatial frequency energy distribution. In the frequency domain, the energy distribution is divided into three frequency bands based on the radial frequency of the frequency components: the low-frequency band corresponds to the flat component, reflecting slowly changing uniform areas in the image patch; the mid-frequency band corresponds to the detail component, reflecting texture details and fine structures in the image patch; and the high-frequency band corresponds to the edge component, reflecting strong edges and contours in the image patch. The proportion of energy of each of the flat component, detail component, and edge component to the total energy of the image patch is calculated. The proportions of the detail component and the edge component are weighted and summed, with the detail component given a higher weight and the edge component given a medium weight. The flat component is not included in the texture richness calculation. The weighted sum is used as the texture richness index of the image patch. The average or median of the texture richness indices of all image patches in the original acquisition area is taken to obtain the texture richness record of that area.

[0134] Step S532: For the content remapping region, obtain the set of transformation matrices indicated by the pixel migration path used in the generation process of the region, analyze the spatial drasticness of the transformation matrix set, and generate a synthetic transformation complexity record based on the spatial drasticness.

[0135] For each content remapping region, the pixel migration path indication generated in step S300 is traced back to obtain the layered set of transformation matrices upon which the region is based, as well as the positional range of each transformation matrix in the set. The spatial variation drasticness of the transformation matrix set is analyzed by calculating the difference measure between the transformation matrix parameters of spatially adjacent transformation matrices. The difference measure of transformation matrix parameters is measured by the sum of the absolute values ​​of the differences between the elements of the transformation matrices or the shape difference of the output ellipse after the transformation matrix is ​​applied to the unit circle. The difference measure between spatially adjacent transformation matrices is accumulated and statistically analyzed over the entire region to obtain an index of the drasticness of the spatial variation of the transformation matrix in that region. This drasticness index is normalized and used as a synthetic transformation complexity record, which reflects the richness of high-frequency information components introduced by the complex spatial variation of the transformation matrix during the encoding process of the content remapping region.

[0136] Step S533: Simultaneously analyze the differences in local structural orientation at the boundary where the content remapping region connects with the original image, and transform the differences in structural orientation into a synthetic structural consistency compensation consumption record.

[0137] Extract the connection boundary between the content remapping region and its adjacent original acquisition region or transitional blending region. Calculate the local structural orientation of pixels on the content remapping region side and the local structural orientation of pixels on the adjacent region side segment by segment along the connection boundary. The calculation method for the local structural orientation is consistent with the calculation method for the texture principal direction angle in step S350, obtained through gradient structure tensor eigenvalue decomposition. Calculate the directional difference angle of the structural orientation on both sides, and statistically analyze the directional difference angles of different boundary segments. Regions with larger directional difference angles require more bitrate during encoding to mask the structural discontinuity. Calculate the weighted average of the directional difference angles of each segment of the entire boundary according to the segment length, and multiply by the mapping constant to convert it into a synthetic structural consistency compensation consumption record. This record describes the additional encoding consumption required by the content remapping region to maintain visual consistency with the surrounding original content in the form of bit cost or encoding complexity increment.

[0138] Step S534: Merge the synthesis transformation complexity record and the synthesis structure consistency compensation consumption record according to weight to generate a synthesis complexity record for the content remapping region. The synthesis complexity record indicates the degree of additional coding finesse required for the region while maintaining visual quality.

[0139] The synthesis transformation complexity record and the synthesis structure consistency compensation consumption record are normalized to a unified complexity metric space and then weighted and summed according to preset merging weights. The merging weights are determined in advance based on the influence of synthesis transformation complexity and structure consistency compensation consumption on encoding. The sum of the weights of synthesis transformation complexity and structure consistency compensation consumption is 1. The weighted sum is used as the synthesis complexity record of the content remapping region. The larger the value of this synthesis complexity record, the more encoding finesse resources are required for the content remapping region to maintain a given visual quality level.

[0140] Step S535: Record the texture richness of the original acquisition area as its encoding complexity description, and record the synthesis complexity of the content remapping area as its encoding complexity description, so that the encoding complexity descriptions of the two types of areas can be evaluated independently.

[0141] The encoding complexity of the original acquisition area is directly described using the texture richness record obtained in step S531, and its value depends only on the statistical characteristics of the natural texture within the area. The encoding complexity of the content remapping area is described using the synthesis complexity record generated in step S534, and its value comprehensively reflects the complexity of the synthesis transformation and the additional overhead of structural consistency. The encoding complexity descriptions of the two types of areas are numerically comparable, but their evaluation systems are independent and do not interfere with each other, facilitating subsequent targeted adjustments to areas with different properties using differentiated encoding strategies.

[0142] Step S536: Store the obtained encoding complexity descriptions of each region and the corresponding region type identifiers to obtain a region encoding complexity mapping table, which can be used for querying when generating differentiated encoding transmission strategies.

[0143] The region code, region type identifier, and corresponding coding complexity description value for each connected component on the panoramic canvas are stored in a region coding complexity mapping table. The region coding complexity mapping table is indexed using the region ID as the primary key, and each record contains the region polygon outline, type identifier, coding complexity description, and timestamp version information. This mapping table serves as the basic query data source in subsequent coding configuration and dynamic adjustment stages.

[0144] Step S540: Based on the real-time feedback of the transmission channel, assess the total amount of available transmission resources, set the coding quality protection priority of the original acquisition area to the baseline level, and set the coding quality protection priority of the content remapping area to the dynamically adjustable enhancement level. The enhancement level is jointly adjusted by the coding complexity description and the available transmission resources.

[0145] In one implementation, step S540 may specifically include the following steps S541 to S546: Step S541: Receive the current available throughput assessment record and delay jitter record from the transmission channel feedback, compare the available throughput assessment record with the preset throughput baseline, determine the channel margin status, when the available throughput assessment record exceeds the throughput baseline, increase the adjustable upper limit range of the enhancement level, when the available throughput assessment record is lower than the throughput baseline, shrink the adjustable upper limit range of the enhancement level.

[0146] The system receives available throughput assessment records and latency jitter records from the transmission channel via feedback mechanisms from the transmission control protocol or real-time transmission protocol. The available throughput assessment record indicates the data transmission rate that can be supported on the current end-to-end path, while the latency jitter record indicates the fluctuation in the packet arrival interval. The available throughput assessment record is compared with a preset throughput baseline, which is the minimum transmission throughput required to ensure the basic image quality of the panoramic video stream. If the available throughput assessment record is higher than the throughput baseline, the difference represents the channel margin. In this case, the adjustable upper limit of the enhancement level in the content remapping region is expanded upwards, with the expansion amount proportional to the channel margin. If the available throughput assessment record is lower than the throughput baseline, it indicates that the current channel conditions are insufficient to simultaneously guarantee the coding quality of all regions. In this case, the adjustable upper limit of the enhancement level is shrunk, with the shrinkage amount proportional to the throughput gap. In extreme congestion situations, the adjustable upper limit of the enhancement level is reduced to a level that can only maintain the minimum acceptable quality of the content remapping region.

[0147] Step S542: Based on the channel margin status, set the baseline protection lower limit of the original acquisition area so that the original acquisition area can obtain the basic coding precision under any throughput conditions.

[0148] The raw acquisition area is the content region in the panoramic video stream directly captured by the camera, containing the most original and accurate visual information of the scene. Regardless of channel margin status, a baseline protection lower limit is set for the raw acquisition area, which corresponds to a fixed basic coding fineness level. The coding fineness level is defined by the value range of the quantization parameter. The baseline protection lower limit ensures that the quantization parameter of the raw acquisition area does not exceed the preset upper limit of the quantization parameter, thereby ensuring that the raw acquisition area is always encoded with a fineness at least higher than the minimum expected quality. When the channel margin is sufficient, the raw acquisition area can be allocated a higher coding fineness, but it must never be lower than the baseline protection lower limit.

[0149] Step S543: For the content remapping region, combine the region's encoding complexity description with the adjustable upper limit of the current enhancement level, and dynamically determine the upper limit of the encoding resources allocated to the region by balancing the high and low encoding complexity description. The higher the encoding complexity description, the larger the upper limit allocated.

[0150] Under the enhancement-level protection priority, content remapping regions have the ability to dynamically increase coding resources based on channel margins above the baseline quality. For each content remapping region, its coding complexity description value is looked up from the region coding complexity mapping table, and this value is compared with the adjustable upper limit range of the current enhancement level. Regions with higher coding complexity descriptions introduce more high-frequency information and structural complexity during synthesis. Allocating more coding resource upsizing intervals to these regions can achieve more significant improvements in visual quality. Therefore, the allocation of coding resource upsizing intervals is positively correlated with the coding complexity description of the region. Regions with higher coding complexity descriptions obtain a wider space for quantization parameter downsizing, i.e., a lower lower limit for quantization parameter values. The specific value of the upsizing interval is determined by a linear mapping between the normalized coding complexity description and the maximum and minimum values ​​of the adjustable upper limit range.

[0151] Step S544: Establish coding resource inheritance rules for the transitional mixed region, so that the transitional mixed region obtains its own coding precision target by interpolating the coding resource allocation of the adjacent region according to the distance ratio along the boundary direction of the original acquisition region and the content remapping region.

[0152] The transition blending region lies between the original acquisition region and the content remapping region. Its coding granularity target is not set independently from zero, but rather inherited from the coding resource allocation of adjacent regions. Specifically, the coding resource inheritance rule is as follows: for each pixel within the transition blending region, the spatial distance from that pixel to the boundary of the adjacent original acquisition region and to the boundary of the adjacent content remapping region are calculated. The ratio of these two distances is used as an interpolation coefficient to linearly interpolate between the coding granularity targets of the original acquisition region and the content remapping region. If the transition blending region is adjacent only to a single type of region, the coding granularity target of that adjacent region is directly adopted. The coding granularity targets of each pixel in the transition blending region collectively constitute the coding resource allocation scheme for that region. This scheme ensures a smooth transition of coding granularity from the original acquisition region to the content remapping region, avoiding abrupt changes in bitrate allocation in space.

[0153] Step S545: Integrate the baseline protection lower limit of the original acquisition area, the enhancement floating range of the content remapping area, and the interpolation coding precision target of the transitional mixing area to generate a coding resource configuration scheme based on the region.

[0154] The baseline protection lower limit of the original acquisition area set in step S542 is mapped to the corresponding quantization parameter lower limit and basic quantization parameter value. The enhancement level floating interval of each content remapping area set in step S543 is mapped to the corresponding quantization parameter value range. The interpolation coding fineness target of the transition mixing area determined in step S544 is mapped to the pixel-by-pixel quantization parameter value or the average quantization parameter value within the region. The coding resource configurations of these three types of regions are summarized to form a coding resource configuration scheme based on connected components. The coding resource configuration scheme clearly records the region number, type identifier, applicable quantization parameter value or quantization parameter value range, and corresponding bit rate control parameters for each connected component.

[0155] Step S546: Merge the coding resource configuration scheme with the differentiated coding transmission strategy to obtain a set of regional coding control parameters that can be directly executed by the encoder. The encoder maintains the consistency of the regional coding strategy between different frames based on this set.

[0156] The differentiated coding transmission strategy includes macro-level coding quality adjustment strategies for different region types, such as a fixed quantization parameter lower limit strategy for the original acquisition region, a quantization parameter adjustment strategy that fluctuates based on channel margin for the content remapping region, and a spatial inheritance interpolation strategy for the transitional mixing region. The specific quantization parameters implemented in each connected component of the coding resource configuration scheme are merged with the macro-strategies in the differentiated coding transmission strategy to form a region-level coding control parameter set. The region-level coding control parameter set uses each frame of the panoramic video stream as a time unit, recording the region type to which each coding unit within that frame belongs and its corresponding coding parameters. These coding parameters include quantization parameters, coding block partitioning mode preferences, and bitrate control targets. During the encoding of the panoramic video stream, the encoder maintains consistency in the coding strategy for the same region type across different frames according to the instructions in the region-level coding control parameter set, ensuring temporal coding quality stability.

[0157] Step S550: Form a differentiated encoding transmission strategy, wherein the original acquisition area maintains the target bit rate level, the content remapping area has independent adjustment space in terms of encoding quantization granularity according to the enhancement level, and the encoding strategy of the transitional mixing area is generated by the smooth transition of the adjacent area strategy.

[0158] The outputs of each sub-step in step S540 are combined to form the final differentiated coding and transmission strategy. This strategy stipulates that the original acquisition area maintains the target bit rate level throughout the transmission process, and its quantization parameters fluctuate only slightly with channel conditions above the baseline protection lower limit to ensure that the coding quality of the original acquisition content remains stable at a high level. The content remapping area, based on the enhancement protection priority, enjoys a coding quantization granularity adjustment space independent of the original acquisition area. When there is sufficient channel margin, the quantization parameters can be reduced to obtain a higher quality synthetic content presentation, and the quantization parameters can be appropriately increased when the channel is congested to save coding resources. The coding strategy of the transitional mixing area is smoothly generated from the strategies of the adjacent original acquisition area and content remapping area. Its quantization parameters are continuously interpolated between the two strategies along the spatial direction, without any abrupt strategy boundaries.

[0159] Step S560: The encoder performs regional encoding on the current segment of the panoramic video stream according to the differentiated encoding transmission strategy, encapsulates the encoded data stream and sends it, so that the image quality of the remapped area and the original acquisition area can be independently represented according to the strategy when the receiving terminal decodes.

[0160] At the encoder, the panoramic video stream is divided into several coding segments along the timeline, each containing multiple consecutive panoramic video frames. For each frame in the current coding segment, the encoder locks the region to which each coding unit belongs within the frame based on the region type spatial mask information, and queries the corresponding coding parameters for that coding unit from the region-level coding control parameter set, performing a region-specific coding operation. The region-specific coding process is as follows: during intra-frame prediction, inter-frame prediction, transform, and quantization, the encoder applies its own quantization parameters to coding units belonging to different region types. For coding units belonging to content remapping regions, the encoder also appropriately adjusts the transform coefficient truncation threshold and the recursion depth of the coding block division according to its synthesis complexity record to match its coding complexity. After encoding, the encoded data stream is encapsulated and packaged according to the real-time transmission protocol and distributed to the receiving terminal through the transmission channel. During decoding at the receiving terminal, each region is independently decoded and reconstructed according to the coding parameters specified by the sender. The original acquisition region is presented with stable high quality, the content remapping region presents a quality level corresponding to the enhancement level setting according to the bitrate allocation during transmission, and the transitional mixing region presents a smooth transition quality performance. This differentiated encoding and transmission strategy enables the receiving terminal to observe high-quality and spatially smooth panoramic video content at any time, while the transmission resources are optimally allocated among different areas.

[0161] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A video blind spot compensation transmission method based on dynamic field-of-view fusion, characterized in that, The method includes: Acquire multi-source video stream data synchronously collected by a video acquisition device cluster. The multi-source video stream data includes multiple video segment units aligned by timestamps. Each video segment unit is accompanied by the spatial positioning information and field of view description of the corresponding acquisition device. Based on the spatial positioning information and field of view description, the field of view regions of all video segment units under the same timestamp are integrated into a unified global reference space to obtain the global field of view stitching layout of the current timestamp. According to the changes in the global field of view stitching layout of consecutive timestamps, a dynamic field of view topology evolution diagram describing the boundary evolution and occlusion relationship between fields of view is constructed. The dynamic field of view topology evolution diagram includes the spatial adjacency connection between each field of view region, the blind hole outline, and the pixel mapping transformation relationship between fields of view. The dynamic field of view topology evolution map is input into the field of view content remapping process. The pixel mapping transformation relationship of the region surrounding the blind spot hole contour is extracted. Based on the pixel mapping transformation relationship, a pixel-by-pixel spatial migration operation is performed on the video frames of adjacent fields of view to generate the remapped image content in the blind spot hole. The remapped image content is then smoothly connected with the surrounding original pixels to obtain the blind spot compensation video clip. A panoramic fusion is performed on the blind spot compensation video segment and the original video segment. In the overlapping area of ​​adjacent fields of view, dynamic fusion weights are generated based on the acquisition confidence metric and remapping confidence metric of each pixel source. The pixels in the overlapping area are weighted and synthesized based on the dynamic fusion weights to obtain a panoramic video stream that maintains spatiotemporal continuity. The distribution pattern of the original acquisition source and content remapping source in different regions of the panoramic video stream is analyzed, and spatial mask information marking the source type of the region is generated. Based on the spatial mask information, a differentiated encoding and transmission strategy is set for each region. The panoramic video stream is encoded and distributed according to the differentiated encoding and transmission strategy, so that the remapping region obtains encoding quality that can be independently controlled from the original acquisition region.

2. The video blind spot compensation transmission method based on dynamic field-of-view fusion according to claim 1, characterized in that, Based on the spatial positioning information and field of view description, the field of view regions of all video segment units at the same timestamp are integrated into a unified global reference space to obtain the global field of view stitching layout for the current timestamp. Furthermore, based on the changes in the global field of view stitching layout for consecutive timestamps, a dynamic field of view topology evolution diagram describing the boundary evolution and occlusion relationships between fields of view is constructed, including: For each video segment unit under the current timestamp, the field of view of the video segment unit is projected onto the unified global reference space using its spatial positioning information to generate the field of view projection polygon of the unit. The field of view projection polygons of all units are then aggregated to obtain the initial global element layout. In the initial global element layout, the intersection regions between the projected polygons of each field of view are detected, the intersection regions are fused to eliminate the repeated expression of the intersection boundaries, and a global element splicing layout with continuous boundaries is obtained. At the same time, isolated hole regions that are not covered by any projected polygons of the field of view are identified, and a hole label is assigned to each isolated hole region. The temporal motion differences of pixel content on both sides of the shared boundary between adjacent field-of-view projected polygons are analyzed. Based on the distribution of motion differences, the local mapping transformation relationship between the two fields of view is estimated, and a field-of-view mapping transformation matrix that records the correspondence rules of pixel positions between fields of view is generated. The global element splicing layout is arranged in time sequence. The position and shape changes of the hole regions corresponding to the hole markers in the global element splicing layouts of adjacent timestamps are compared. The drift trajectory of the hole boundary and the direction of hole deformation are extracted to generate a record of the dynamic evolution trend of the hole. The dynamic evolution trend record of the hole is associated with the field-of-view mapping transformation matrix around the corresponding hole to obtain the pixel migration path indication that maps adjacent field-of-view pixels to the inside of the hole at the hole boundary. The pixel migration path indication records the mapping direction sequence and transformation amplitude distribution. The system gathers global element splicing layouts, hole identifiers, inter-field mapping transformation matrices, and dynamic evolution trend records of holes under all timestamps, and constructs a dynamic field topology evolution graph with field nodes and hole nodes as elements and mapping transformation relationships as edges.

3. The video blind spot compensation transmission method based on dynamic field-of-view fusion according to claim 2, characterized in that, The analysis examines the temporal motion differences of pixel content on both sides of a shared boundary between adjacent field-of-view projected polygons. Based on the distribution of these motion differences, it estimates the local mapping transformation relationship between the two fields of view and generates an inter-field-of-view mapping transformation matrix that records the correspondence rules of pixel positions between the fields of view. This includes: Along the shared boundary of the adjacent field-of-view projection polygons, pairs of local image patch sequences are extracted segment by segment. The pairs of local image patches from the two fields of view are aligned according to the timestamp to form a set of contrast image pairs. Each set of image pairs in the set of contrast image pairs reflects the imaging representation of the same scene under different fields of view. For each pair of comparison images, perform pixel displacement search, find the optimal pixel matching position in the specified search neighborhood, and obtain the horizontal offset direction and vertical offset magnitude of the local position between the two fields of view to form a set of initial offset correspondence records. Summarize the initial offset records corresponding to each segment of the shared boundary, remove abnormal offset records whose abrupt changes with neighboring offsets exceed the tolerance range of abrupt changes, and retain offset records with continuous offset changes and consistent directions as valid offset records. Spatial interpolation propagation is performed on the effective offset record to spread the offset correspondence at the boundary sampling point to the entire overlapping area of ​​the field of view and the area surrounding the hole, generating a pixel mapping correspondence field covering the overlapping area of ​​the field of view and the area surrounding the hole. By using the distribution structure of the pixel mapping correspondence field, a mapping transformation matrix is ​​constructed from the pixel coordinates of the first field of view to the pixel coordinates of the second field of view. The mapping transformation matrix has local adaptability that changes continuously with spatial position in the area surrounding the hole. The completed mapping transformation matrix is ​​bound to the shared boundary segment identifier and hole identifier on which the matrix was generated, and stored in the field-of-view mapping relationship repository.

4. The video blind spot compensation transmission method based on dynamic field-of-view fusion according to claim 3, characterized in that, The step of associating the dynamic evolution trend record of the hole with the inter-view mapping transformation matrix around the corresponding hole to obtain the pixel migration path indication that maps adjacent field pixels to the inside of the hole at the hole boundary includes: Obtain the cavity boundary drift trajectory in the cavity dynamic evolution trend record at the current timestamp, infer the expansion and contraction direction of the cavity boundary from the current timestamp to the next timestamp based on the drift trajectory, and determine the active boundary line segment of the cavity boundary change. Identify the effective field of view region adjacent to the active boundary line segment, retrieve the field-of-view mapping transformation matrix between the field of view region and the adjacent field of view on the other side of the hole, and extend the scope of the mapping transformation matrix to the coordinate range to be filled inside the hole. The mapping transformation matrix is ​​vector-corrected according to the contour deformation direction of the cavity to make the mapping direction consistent with the cavity expansion direction, thereby generating an adaptive mapping transformation matrix after direction correction. Extract the texture direction information of the original pixels on both sides of the hole boundary, and fine-tune the transformation amplitude of the adaptive mapping transformation matrix through the texture direction information, so that the mapping intensity difference between adjacent layers along the main texture direction is smaller than the mapping intensity difference between adjacent layers along the texture normal, thus generating a texture adaptive mapping transformation matrix. Based on the distance from the boundary pixel of the hole to the filling position inside the hole, the mapping intensity of the texture adaptive mapping transformation matrix is ​​set in layers from strong to weak to obtain a pixel migration path indication that migrates adjacent field pixels to the inside of the hole at the boundary. The pixel migration path indication contains a set of transformation matrices distributed in layers. The generated pixel migration path indicator is bound to the hole identifier and the current timestamp, and added to the attribute record of the corresponding hole node in the dynamic field of view topology evolution graph to complete the update of hole association information.

5. The video blind spot compensation transmission method based on dynamic field-of-view fusion according to claim 1, characterized in that, The process of inputting the dynamic field-of-view topology evolution map into the field-of-view content remapping process, extracting the pixel mapping transformation relationship of the region surrounding the blind spot hole contour, performing a pixel-by-pixel spatial migration operation on the video frames of adjacent fields of view according to the pixel mapping transformation relationship, generating the remapped image content within the blind spot hole, and smoothly connecting the remapped image content with the surrounding original pixels to obtain a blind spot compensated video segment, includes: Based on the hole identifier of the current timestamp in the dynamic field of view topology evolution diagram, locate the pixel migration path indicator associated with the hole, and read the set of transformation matrices distributed by layer and the corresponding radius of action of each layer from the pixel migration path indicator. Obtain the pixel source region located outside the hole boundary and within the range of the transformation matrix from the original video frames of the adjacent field of view. Apply the transformation matrix of the corresponding layer to the pixel source region pixel by pixel, calculate the target migration coordinates of the source pixel inside the hole, and assign the color component of the source pixel to the target migration coordinate. After the initial assignment, the internal image of the hole is checked for migration omissions. The hole image positions that have not been assigned due to discrete coordinate values ​​are identified. The hole image positions are filled by color diffusion processing of the adjacent pixels that have been assigned, and the initial remapping image content with complete migration is generated. Extract the pixel color distribution at the boundary of the initial remapped image content and the original pixel color distribution outside the corresponding hole, construct the color transfer mapping curve inside and outside the boundary, and adjust the hue and brightness of the pixels at the boundary of the initial remapped image content through the color transfer mapping curve to reduce the distribution difference of the color channels on both sides of the boundary to a preset tolerance range. If the change in the main direction angle of the texture of adjacent pixels at the boundary of the remapped image content exceeds the preset angle jump threshold, the local position of the migrated pixels is slightly moved along the broken line segment to realign the texture structure, eliminate the structural break marks, and obtain the remapped image content that maintains structural connection with the surrounding original scene. The adjusted remapped image content is used as the blind zone compensation image frame for the hole at the current timestamp, and is kept synchronized with the original video frame at the corresponding timestamp. All blind zone compensation image frames of the holes are integrated to generate a blind zone compensation video clip.

6. The video blind spot compensation transmission method based on dynamic field-of-view fusion according to claim 5, characterized in that, The process of obtaining the pixel source region located outside the hole boundary and within the range of the transformation matrix from the original video frames of adjacent fields of view, applying the transformation matrix of the corresponding layer to the pixel source region pixel by pixel, calculating the target migration coordinates of the source pixel inside the hole, and assigning the color components of the source pixel to the target migration coordinates includes: Using the hole boundary as a reference line, a pixel selection band is expanded outward with the same maximum radius of action as the transformation matrix set. Within this pixel selection band, the positions of the source pixels to be migrated are marked in a row-by-column order and their original color components are recorded. For each source pixel to be migrated, the corresponding transformation matrix is ​​obtained according to its layer index. The source pixel coordinates are then subjected to spatial transformation operation through the transformation matrix to obtain the migration landing point coordinates of the pixel inside the hole. The migration landing point coordinates are recorded as floating-point precision spatial positions. The floating-point precision migration landing point coordinates are mapped onto the discrete pixel grid inside the hole. When a discrete grid cell receives multiple source pixel mappings, the color composition of the grid cell is determined by calculating the merging contribution ratio according to the distance of each source pixel to the grid center. For the boundary grid cells of the discrete pixel grid, extract their color consistency with the adjacent assigned grid cells. If the color of the boundary grid cell is inconsistent with the extrapolation trend of the neighbors, then the boundary grid cell is re-included in the color diffusion correction queue for local color backtracking adjustment. The internal region of the hole after color assignment is formed into an initial filled image block. At the boundary between the filled image block and the surrounding original image, low-frequency smoothing of the boundary is performed, and the brightness components of the pixels on both sides of the boundary are linearly gradient transitioned to alleviate the edge contrast introduced by the mapping. The filled image block, after low-frequency smoothing at the boundary, replaces the original hole region and serves as the migration assignment result of the hole in the corresponding layer. This result is then passed to the subsequent migration omission check step for further repair.

7. The method according to claim 5, characterized in that, The process of extracting the pixel color distribution at the boundary of the initially remapped image content and the corresponding original pixel color distribution outside the hole, constructing a color transfer mapping curve inside and outside the boundary, and adjusting the hue and brightness of the pixels at the boundary of the initially remapped image content through the color transfer mapping curve includes: Along the boundary line between the content of the initial remapped image and the original image outside the hole, pixel samples inside and outside the boundary are collected at equal intervals. The intensity value of each sample in each color channel and its normal distance and tangential position relative to the boundary line are recorded to form a set of cross-boundary color sample pairs. A color space distribution clustering analysis is performed on the cross-boundary color sample pair set. The internal pixel color values ​​are divided into local intervals according to the tangential position. The mapping tendency distribution from internal color to external color is statistically analyzed in each local interval. Sample pairs whose mapping tendency and the offset difference between adjacent intervals exceed a preset consistency threshold are removed to obtain the reliable color correspondence set corresponding to each local interval. For each local interval, a piecewise linear mapping curve from the color channel intensity of the inner pixel to the color channel intensity of the outer pixel is constructed within the interval using the trusted color correspondence set. The mapping curves of adjacent local intervals are constrained to maintain consistent mapping values ​​at the interval boundaries, resulting in a spatially continuous set of color transfer mapping curves. Traverse each pixel within the boundary region of the initial remapped image content, determine the local interval to which the pixel belongs based on its tangential position, extract the corresponding mapping curve, input the intensity values ​​of each color channel of the pixel into the mapping curve, and obtain the adjusted pixel color. Calculate the brightness gradient vector of the adjusted boundary region pixels in the cross-boundary direction, and simultaneously extract the brightness gradient vector of the corresponding position of the external original image at the junction. Compare the angle between the directions of the two gradient vectors with a preset angle tolerance, and compensate and correct the brightness of pixels that exceed the preset angle tolerance along the normal direction. The pixels in the boundary region that have undergone brightness gradient correction are combined with the unadjusted pixels inside the initial remapped image content to generate an adjusted remapped image content whose boundary color transition maintains spatial gradient continuity with the original external image.

8. The video blind spot compensation transmission method based on dynamic field-of-view fusion according to claim 1, characterized in that, The process of performing panoramic fusion on the blind spot compensation video segment and the original video segment involves generating dynamic fusion weights based on the acquisition confidence metric and remapping confidence metric of each pixel source in the overlapping region of adjacent fields of view. Pixels in the overlapping region are then weighted and synthesized based on these dynamic fusion weights to obtain a panoramic video stream that maintains spatiotemporal continuity. This includes: Using a unified global reference space as the alignment benchmark, each compensated image frame in the blind zone compensation video clip and each original video frame in the original video clip under the same timestamp are assigned to the corresponding positions on the panoramic canvas. The overlapping polygonal region at the boundary of adjacent fields of view is extracted, and at least two source identifiers corresponding to each pixel position in the overlapping polygonal region are recorded. For each pixel location, obtain the acquisition confidence metric or remapping confidence metric corresponding to its source identifier, input the confidence metrics of each source into the joint confidence evaluation function, and determine the fusion contribution ratio between different sources at the pixel location by the ratio of the confidence metrics of each source, and generate pixel-level initial fusion weights. Based on the pixel-level initial fusion weights, the pixel-level initial fusion weights are globally optimized and adjusted along the direction from the boundary to the interior of the overlapping zone, constrained by the boundary of the polygonal region of the overlapping zone. A fusion weight field is constructed with the source confidence at the boundary of the overlapping zone as the anchor point constraint. The fusion weight field satisfies the spatial smoothness condition inside the overlapping zone and is aligned with the anchor point confidence at the boundary of the overlapping zone. The fusion weight field is used to weight and synthesize the source color components at each pixel position in the overlapping polygonal region to obtain the initial fused pixel color, and then a color difference compensation term between the sources is superimposed. The color difference compensation term is generated by the weighted deviation of each source color component at the pixel position and the gradient of the fusion weight field. On the adjacent transition strip between the overlapping polygonal region and the non-overlapping region, the fusion weight field is continuously extended along the normal of the transition strip, pushing the fusion weight field to the outer boundary of the non-overlapping region, and clipping it with the minimum width of the transition strip to eliminate the weight field cutoff boundary between the fusion region and the non-fusion region. The overlapping region image generated by weighted extrapolation, the original non-overlapping region image, and the remapped region image corresponding to the blind spot compensation video segment are assembled into a panoramic canvas and output as a panoramic video stream in time stamp order.

9. The video blind spot compensation transmission method based on dynamic field-of-view fusion according to claim 8, characterized in that, Based on the initial pixel-level fusion weights, and constrained by the boundary of the overlapping polygon region, the initial pixel-level fusion weights are globally optimized and adjusted along the direction from the boundary to the interior of the overlapping zone. This constructs a fusion weight field constrained by the source confidence at the boundary of the overlapping zone, including: Extract the boundary distribution of pixel-level initial fusion weights on each boundary line segment of the overlapping polygonal region, and mark the pixels in the boundary distribution that are consistent with the confidence metric of each source as anchor points. The remaining pixels on the boundary line segment obtain the boundary constraint value by interpolating the anchor points. Using the boundary constraint value of the polygonal region of the overlapping zone as the boundary condition, an objective function for the fusion weight field is established in the inner region of the overlapping zone. The objective function includes a second-order spatial difference term and a boundary deviation penalty term for the weight field. The second-order spatial difference term constrains the smoothness of the change of the weight field in the inner region, and the boundary deviation penalty term constrains the weight field to approach the boundary constraint value at the boundary. The objective function is solved iteratively by updating the weight values ​​of the internal pixels one by one, so that the objective function value drops below the preset convergence index, and an initial fusion weight field that satisfies the boundary constraints and has a smooth internal transition is obtained. The system detects whether there are local oscillation regions within the overlap zone of the initial fused weight field. By extracting the curvature distribution of the contour lines of the weight field, the region where the curvature of the contour lines exceeds the preset contour smoothing threshold is marked as the oscillation region. The weight field in the oscillation region is locally re-smoothed, and the weight values ​​in the oscillation region are replaced so that they change monotonically along the contour line normal. The fused weight field after oscillation processing is compared with the original pixel-level initial fused weight in the non-oscillation region. If the average deviation between the two weight fields in the block is lower than the preset deviation tolerance limit, the local detail features of the initial fused weight field are retained first. If the average deviation exceeds the preset deviation tolerance limit, the weight field after oscillation processing is used to maintain global smoothness. The weight field values ​​of each block are merged to generate a final fused weight field that is aligned with the anchor point confidence metric at the boundary of the overlap band, maintains spatial smoothness within the overlap band, and autonomously switches between local details and global smoothness.

10. A server, characterized in that, include: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the video blind spot compensation transmission method based on dynamic field fusion as described in any one of claims 1 to 9.

Images