A video image automatic segmentation and splicing method and system based on scene semantic recognition

Abstract

The application discloses a video image automatic segmentation splicing method and system based on scene semantic recognition, and belongs to the field of video image processing. In order to solve the problem that in the prior art, the segmentation result and the splicing result lack closed-loop feedback in the semantic segmentation auxiliary video splicing scheme, the abnormal splicing source cannot be distinguished, and the low-confidence area cannot be repaired, the application forms a closed-loop processing link through scene semantic recognition, dynamic granularity segmentation, semantic guided splicing and splicing quality feedback, so that the boundary alignment error and semantic consistency score generated in the splicing stage can reversely correct the segmentation granularity parameter and the scene switching boundary determination threshold, thereby reducing the splicing misplacement and semantic break caused by the disconnection between segmentation and splicing.

Description

Technical Field

[0001] This invention relates to the field of video image processing technology, and in particular to a method and system for automatic segmentation and stitching of video images based on scene semantic recognition. Background Technology

[0002] Video image stitching is typically used to merge video images acquired from different perspectives, at different times, or with different shooting devices into a continuous video stitching sequence. Existing video image stitching schemes are usually based on steps such as feature point extraction, feature point matching, geometric transformation estimation, stitching seam search, and image fusion. For video images with simple scenes, small changes in perspective, and little foreground motion, the above methods can achieve certain stitching effects.

[0003] However, in actual applications such as surveillance, mobile shooting, drone inspection, vehicle-mounted video acquisition, and complex scene reconstruction, video images often have problems such as frequent scene switching, obvious foreground target movement, static background and dynamic foreground interweaving, large changes in lighting, many occluded areas, and inconsistent semantic boundaries between different viewpoints. If only traditional feature point matching and geometric transformation estimation are relied upon, problems such as misaligned splicing boundaries, broken targets, dynamic foreground ghosting, cut-off semantic objects, and abrupt splicing seams are likely to occur.

[0004] To address the aforementioned issues, existing technologies have developed solutions that utilize semantic segmentation results to assist in image stitching. These solutions typically involve first performing semantic segmentation on video frames or images, and then using the semantic segmentation results for feature point selection, feature point classification, stitching seam selection, or image fusion. Such solutions can, to some extent, avoid mismatching regions of different semantic categories and improve the rationality of stitching seam selection.

[0005] However, most existing semantic segmentation-assisted stitching schemes belong to the forward processing chain. That is, after the semantic segmentation module outputs the segmentation result, the stitching module directly performs matching and fusion based on the segmentation result. Once the semantic segmentation boundary shifts, the segmentation granularity is too coarse, the scene switching point is misjudged, or the dynamic foreground is incorrectly included in the geometric registration area, the stitching module can usually only adjust the stitching line, fuse colors, or perform local repairs during the stitching stage. It is difficult to reverse the process to determine whether the stitching abnormality originates from the segmentation module, and it is also difficult to feed back the stitching failure result to the segmentation module for targeted correction.

[0006] To address this, we propose an automatic video image segmentation and stitching method and system based on scene semantic recognition. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for automatic video image segmentation and stitching based on scene semantic recognition, which solves the problems of lack of closed-loop feedback between segmentation results and stitching results in existing semantic segmentation-assisted video stitching schemes, inability to distinguish the source of stitching anomalies, and inability to specifically repair low-confidence areas.

[0008] To achieve the above objectives, the present invention provides the following technical solution: an automatic video image segmentation and stitching method based on scene semantic recognition, comprising the following steps: S1: acquiring the original video stream, parsing its frame sequence to obtain multiple consecutive video images containing time indices; S2: inputting the multiple consecutive video images into a scene semantic recognition network, wherein the scene semantic recognition network outputs a pixel-level scene semantic label map, a scene switching boundary heatmap, and a confidence distribution of scene semantic categories for each frame of video image; S3: using an adaptive video segmentation unit to perform dynamic granular segmentation based on scene semantics: identifying scene switching points according to the boundary probability gradient in the scene switching boundary heatmap, segmenting the video stream into multiple semantic scene segments; adaptively determining the segmentation granularity based on the semantic complexity of each semantic scene segment, segmenting the semantic scene segment according to the determined segmentation granularity, and generating segmented sub-images carrying segmentation semantic description information; S4: using semantics... The guided stitching unit stitches the segmented sub-images: Based on the segmentation semantic description information, segmented sub-images from different perspectives or time periods are matched according to semantic categories; a stitching strategy pool corresponding to each semantic category is constructed, and the corresponding stitching strategy is called from the strategy pool according to the matched semantic category to perform stitching; semantic consistency verification is performed when stitching segmented sub-images of different semantic categories; S5: A closed-loop feedback link between segmentation and stitching is constructed, and the stitching boundary alignment error value and the semantic consistency score value of the stitching overlap area are used as stitching quality evaluation parameters, which are fed back to the adaptive video segmentation unit in real time to dynamically correct the segmentation granularity parameters and scene switching boundary judgment threshold of subsequent frames, thereby forming a deep coupling between segmentation and stitching, and realizing the co-evolution between the adaptive video segmentation unit and the semantically guided stitching unit; S6: The stitched scene segments are fused according to the time sequence to output a complete video stitching sequence.

[0009] Furthermore, the semantic consistency verification in S4 specifically includes: extracting the semantic category labels of each pixel on both sides of the splicing boundary and establishing a semantic consistency matrix of the splicing boundary; when the matching degree of the semantic category labels at corresponding positions on both sides of the splicing boundary is lower than the preset semantic consistency threshold, it is determined to be semantically inconsistent, and the following repair operations are performed: for the segmented sub-image of the static background semantic category, the optimal stitching line is recalculated in the neighborhood of the splicing boundary, and a graph cut algorithm guided by the semantic boundary heatmap is used to find a splicing path that avoids the semantic boundary; for the segmented sub-image of the dynamic foreground semantic category, the splicing overlap region is input into a pre-constructed conditional generative adversarial network, and the semantic category confidence distribution is used as a conditional constraint to generate the completed semantic continuous image content.

[0010] Furthermore, the semantic complexity in S3 is calculated using the following formula: ;in, The total number of semantic categories. The total length of the semantic region boundary. Foreground instance count The weighting coefficients are dynamically optimized by minimizing the mean square error between the stitching boundary alignment error value fed back in S5, thereby quantifying the stitching quality feedback into a correction signal for the segmentation parameters and realizing deep coupling between the adaptive video segmentation unit and the semantically guided stitching unit.

[0011] Further, in step S4, the corresponding stitching strategy is invoked from the strategy pool based on the semantic category to perform stitching. This further includes: for segmented sub-images with a static background semantic category, a first stitching strategy is invoked, and feature point matching is performed based on scale-invariant feature transformation feature point extraction and random sampling consistency algorithm to achieve geometric alignment stitching; for segmented sub-images with a regular geometric shape semantic category, a second stitching strategy is invoked, and the semantic boundary masks on both sides of the stitching boundary are elastically matched, and the matched semantic boundary is used as the stitching line for stitching, and the Poisson equation solution method is used to perform smooth transition processing of color and texture in the neighborhood of the stitching line; for segmented sub-images with a dynamic foreground semantic category, a third stitching strategy is invoked, and the stitching overlap area is input into a pre-constructed conditional generative adversarial network, with the semantic category confidence distribution as a conditional constraint, and the completed semantic continuous image content is generated by feature resampling based on confidence weight.

[0012] Furthermore, the scene semantic recognition network is obtained through multi-task joint training, which includes the joint training of pixel-level semantic segmentation, scene classification, and temporal scene boundary detection tasks. These three tasks share the underlying feature extraction network parameters, and their joint loss function is: ;in: The cross-entropy loss is used for pixel-level semantic segmentation tasks, where N is the total number of pixels and C is the number of semantic categories. Let i be the true label indicator function for pixel i belonging to category c. The probability predicted by the network; For scene classification tasks, the multi-class cross-entropy loss is used, where K is the total number of scene categories. Labeled as real-world scenarios The global scene probability predicted by the network; Let M be the focus loss function for the temporal scene switching boundary detection task, where M is the number of boundary detection locations. These are the boundary probability values ​​predicted by the network. and is a hyperparameter of the focus loss, used to alleviate the class imbalance problem between scene boundary samples and non-boundary samples; The balancing weight coefficients for each loss term are tuned on the validation set using grid search or Bayesian optimization.

[0013] Furthermore, the adaptive video segmentation unit in S3 adopts a graph-based image segmentation algorithm guided by semantic boundary heatmap. Pixels with boundary probability values ​​greater than a first threshold in the scene switching boundary heatmap are used as strong constraint anchors. The segmentation boundaries of the strong constraint anchors and their neighborhoods are forced to align with the scene switching boundary heatmap. When the boundary probability value is between the first threshold and the second threshold, the segmentation boundary is allowed to shift elastically within a certain range, thereby maintaining the semantic boundary alignment accuracy while taking into account the segmentation flexibility of low-confidence regions.

[0014] Furthermore, the system also includes constructing a bidirectional data channel between the adaptive video segmentation unit and the semantically guided stitching unit: the adaptive video segmentation unit transmits segmented sub-images and their segmentation semantic description information to the semantically guided stitching unit; the semantically guided stitching unit feeds back stitching quality evaluation parameters to the adaptive video segmentation unit, and when the semantically guided stitching unit encounters a low-confidence region during stitching, it requests re-segmentation from the adaptive video segmentation unit; the adaptive video segmentation unit responds to this request, adjusts the segmentation granularity parameters to perform secondary segmentation on the video frame to which the low-confidence region belongs, and retransmits the corrected segmentation result to the semantically guided stitching unit; S5 also includes a stitching anomaly attribution step, which includes: obtaining stitching boundary alignment error values ​​and semantic consistency of stitching overlapping areas. The system considers the following parameters: semantic score, semantic category confidence distribution on both sides of the stitching boundary, number of times the stitching seam crosses the semantic boundary, and dynamic foreground optical flow residual. Based on these parameters, a responsibility attribution vector is generated, comprising a boundary offset responsibility component, an insufficient granularity responsibility component, a scene switching misjudgment responsibility component, a dynamic foreground interference responsibility component, and a stitching strategy mismatch responsibility component. When the boundary offset responsibility component or the insufficient granularity responsibility component meets a preset dominant condition, the adaptive video segmentation unit is triggered to perform secondary segmentation on the local region within the stitching seam neighborhood. When the dynamic foreground interference responsibility component meets a preset dominant condition, the feature matching weight of the dynamic foreground region is reduced, and a stitching strategy based on generative semantic repair is invoked. When the stitching strategy mismatch responsibility component meets a preset dominant condition, the current segmentation result remains unchanged, and the priority of the stitching strategy pool corresponding to the semantic category is rearranged.

[0015] This invention also provides a technical solution: an automatic video image segmentation and stitching system based on scene semantic recognition. This system, with the core objective of establishing deep coupling and co-evolution between segmentation and stitching, includes: a video input parsing module for acquiring and parsing the original video stream; a scene semantic collaborative perception module connected to the video input parsing module for simultaneously outputting pixel-level scene semantic label maps, scene transition boundary heatmaps, and confidence distributions of scene semantic categories for the parsed video frames; and an adaptive semantic segmentation module connected to the output of the scene semantic collaborative perception module, comprising a scene boundary detection unit, a semantic complexity calculation unit, and a segmentation granularity decision and execution unit. The system includes a secondary segmentation trigger unit for performing dynamic granular segmentation based on scene semantics; a semantically guided collaborative stitching module connected to the output of the adaptive semantic segmentation module, comprising a semantic matching unit, a strategy pool invocation unit, a semantic consistency verification unit, and a low-confidence feedback unit, for invoking the corresponding stitching strategy according to the semantic category of the segmented sub-images to perform stitching, and for performing semantic consistency verification on the stitching process; a segmentation-stitching closed-loop collaborative bus connecting the semantically guided collaborative stitching module and the adaptive semantic segmentation module, for bidirectional transmission of stitching quality evaluation parameters and re-segmentation request instructions; and a temporal fusion output module for fusing and stitching the scene segments according to the time sequence.

[0016] Furthermore, the segmentation-stitching closed-loop collaborative bus adopts an asynchronous message passing protocol. When the semantically guided collaborative stitching module detects that the stitching quality evaluation parameters exceed the preset threshold, it automatically generates a re-segmentation request command and sends it to the adaptive semantic segmentation module via the segmentation-stitching closed-loop collaborative bus. This triggers the module to adjust the segmentation granularity parameters to perform secondary segmentation on the corresponding video frames. After receiving the re-segmentation result, the semantically guided collaborative stitching module automatically resumes the stitching processing flow, forming a complete closed-loop control link.

[0017] Furthermore, the adaptive semantic segmentation module and the semantically guided collaborative splicing module jointly optimize the following objective function through the segmentation-splicing closed-loop collaborative bus: ;in, To divide the loss, Due to splicing losses, The semantic closure synchronization error between the segmentation boundary and the splicing boundary is defined as the Frobenius norm of the vector difference between the segmented semantic description information and the actual semantic category label after splicing at each splicing boundary. To balance the weights.

[0018] Compared with existing technologies, the beneficial effects of this invention are as follows: The automatic video image segmentation and stitching method and system proposed in this invention, based on scene semantic recognition, forms a closed-loop processing link through scene semantic recognition, dynamic granular segmentation, semantically guided stitching, and stitching quality feedback. This allows the boundary alignment errors and semantic consistency scores generated during the stitching stage to correct the segmentation granularity parameters and scene switching boundary judgment thresholds, thereby reducing stitching misalignment and semantic breaks caused by the disconnect between segmentation and stitching. Secondly, the segmentation granularity is determined by the semantic complexity score, and the weighting coefficients in the semantic complexity are dynamically correlated with the stitching boundary alignment error, enabling the segmentation granularity to adaptively adjust according to the scene complexity and actual stitching effect, thereby improving... The system ensures stable stitching in complex scenes, dynamic foregrounds, and scenes with multiple semantic boundaries. Secondly, it detects the semantic label matching degree on both sides of the stitching boundary using a semantic consistency matrix, and employs different repair methods for static background and dynamic foreground regions. This allows the stitching path to avoid key semantic boundaries and enables semantic continuity completion of dynamic foreground regions, thereby reducing target breakage, foreground incompleteness, and ghosting. Finally, it distinguishes boundary offset, insufficient granularity, scene switching misjudgment, dynamic foreground interference, and stitching strategy mismatch using a stitching anomaly responsibility attribution vector. Based on different responsibility components, it performs local secondary segmentation, dynamic foreground weight reduction, or stitching strategy pool rearrangement, thereby avoiding blind full-frame re-segmentation and improving the targeting and computational efficiency of feedback correction. Attached Figure Description

[0019] Figure 1 This is an overall flowchart of the automatic video image segmentation and stitching method based on scene semantic recognition of the present invention; Figure 2 This is a block diagram of the automatic video image segmentation and stitching system based on scene semantic recognition of the present invention; Figure 3 This is a schematic diagram showing the pixel-level scene semantic label map, scene switching boundary heatmap, and confidence distribution output by the scene semantic recognition network of this invention; Figure 4 This is a flowchart of the adaptive video segmentation unit of the present invention performing dynamic granularity segmentation; Figure 5 This is a flowchart of the semantic guidance splicing unit calling the splicing strategy pool in this invention; Figure 6 This is a flowchart of the semantic consistency verification and repair process for splicing boundaries in this invention; Figure 7 This is a flowchart illustrating the attribution of responsibility and differentiated handling of splicing anomalies in this invention; Figure 8 This is a flowchart illustrating the transmission of re-segmentation requests and splicing quality evaluation parameters in the segmented splicing closed-loop collaborative bus of the present invention. Figure 9 This is a flowchart of the dynamic optimization process for the semantic complexity weighting coefficients of this invention; Figure 10 This is a schematic diagram of the multi-task joint training structure of the scene semantic recognition network of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1: Please refer to Figures 1-10 This embodiment provides an automatic video image segmentation and stitching method based on scene semantic recognition. This method is applicable to application scenarios such as surveillance video stitching, vehicle-mounted multi-view video stitching, drone inspection video stitching, mobile shooting video stitching, and multi-camera scene reconstruction.

[0022] First, step S1 is executed to acquire the original video stream. Frame sequence parsing is then performed on the original video stream to obtain multiple consecutive video images containing time indices. The original video stream can originate from fixed monitoring equipment, mobile shooting equipment, multi-camera arrays, vehicle-mounted cameras, or drone cameras. Frame sequence parsing includes frame decoding, timestamp extraction, frame sequence number establishment, and inter-frame order verification. Each parsed video image has a corresponding time index, enabling subsequent segmentation, stitching, and temporal fusion to be performed according to the video's temporal order.

[0023] Then, step S2 is executed, in which multiple consecutive video images are input into the scene semantic recognition network. The scene semantic recognition network outputs a pixel-level scene semantic label map, a scene transition boundary heatmap, and a confidence distribution of scene semantic categories for each video image. The pixel-level scene semantic label map is used to represent the semantic category to which each pixel belongs. The scene transition boundary heatmap is used to represent the probability that scene boundaries or semantic boundaries appear in the video frame or the region within the video frame. The confidence distribution of scene semantic categories is used to represent the probability that each pixel, each region, or each segmented sub-image belongs to each semantic category.

[0024] In one implementation, the scene semantic recognition network includes a low-level feature extraction structure, a semantic segmentation output branch, a scene classification output branch, and a temporal scene boundary detection output branch. The low-level feature extraction structure is used to extract spatial visual features of video images, the semantic segmentation output branch is used to output pixel-level scene semantic label maps, the scene classification output branch is used to output global scene category probabilities, and the temporal scene boundary detection output branch is used to output scene switching boundary heatmaps.

[0025] Subsequently, step S3 is executed, in which an adaptive video segmentation unit performs dynamic granular segmentation based on scene semantics. The adaptive video segmentation unit identifies scene switching points based on the boundary probability gradient in the scene switching boundary heatmap and segments the video stream into multiple semantic scene segments. The boundary probability gradient can be obtained by the rate of change of the scene switching boundary heatmap between adjacent frames. When the boundary probability gradient is greater than the preset scene switching threshold, the corresponding time index is determined as the scene switching point.

[0026] For each semantic scene segment, the adaptive video segmentation unit adaptively determines the segmentation granularity based on the semantic complexity of the semantic scene segment. The segmentation granularity is used to represent the degree of refinement of the region segmentation of the semantic scene segment. The segmentation granularity can include semantic region-level segmentation, semantic region-plus-boundary refinement segmentation, foreground instance-level segmentation, and local high-granularity segmentation.

[0027] In one implementation, the segmentation semantic description information includes the segmented sub-image number, the time index of the corresponding video frame, the semantic scene segment number, the spatial coordinate range, the semantic category label, the semantic boundary mask, the region confidence, and the region area. The segmentation semantic description information serves as a data interface between the segmented sub-image and the subsequent stitching process.

[0028] Then, step S4 is executed, in which the segmented sub-images are stitched together using a semantically guided stitching unit. The semantically guided stitching unit matches the segmented sub-images from different perspectives or time periods according to semantic categories based on the segmentation semantic description information. The semantic category matching can be determined jointly based on semantic category label consistency, semantic category confidence similarity, spatial proximity, and temporal index continuity.

[0029] The semantically guided splicing unit constructs a splicing strategy pool corresponding to its respective semantic category. The splicing strategy pool includes a first splicing strategy based on the feature point extraction and random sampling consistency algorithm of scale-invariant feature transformation, a second splicing strategy based on semantic boundary constraints, and a third splicing strategy based on conditional generative adversarial network. The semantically guided splicing unit calls the corresponding splicing strategy from the splicing strategy pool according to the matched semantic category to perform splicing.

[0030] When stitching together segmented sub-images of different semantic categories, the semantically guided stitching unit performs semantic consistency verification. The semantic consistency verification is used to determine whether the semantic category labels on both sides of the stitching boundary are consistent or compatible. If the semantic labels on both sides of the stitching boundary are inconsistent, then the stitching line is recalculated, semantic boundary avoidance is performed, generative semantic completion is performed, or the strategy pool is rearranged.

[0031] Subsequently, step S5 is executed to construct a closed-loop feedback link between segmentation and stitching. The closed-loop feedback link uses the stitching boundary alignment error value and the semantic consistency score value of the stitching overlapping area as stitching quality evaluation parameters, and feeds them back to the adaptive video segmentation unit in real time to dynamically correct the segmentation granularity parameters and scene switching boundary judgment threshold of subsequent frames. This enables the stitching quality to inversely constrain the segmentation process.

[0032] The alignment error value of the splicing boundary can be obtained from the reprojection error of the matching points on both sides of the splicing boundary, or from the offset distance between the structural lines on both sides of the splicing boundary. The semantic consistency score of the splicing overlapping area can be obtained from the ratio of the number of pixels with consistent semantic category labels on both sides of the splicing overlapping area to the total number of pixels in the overlapping area, or from the cosine similarity between the semantic category confidence vectors on both sides of the splicing boundary.

[0033] Finally, step S6 is executed to fuse the stitched scene segments in time sequence and output a complete video stitching sequence. Time sequence fusion may include inter-frame smoothing, stitching boundary jitter suppression, scene segment transition processing, and output frame rate consistency processing.

[0034] Example 2: This example further illustrates the semantic complexity calculation and segmentation granularity determination method based on Example 1.

[0035] In step S3, the semantic complexity Scomp is calculated using the following formula; Where Ccls is the total number of semantic categories, Lbdry is the total length of the semantic region boundary, Ninst is the number of foreground instances, and α, β, and γ are weighting coefficients. The total number of semantic categories, Ccls, reflects the complexity of semantic object types in the scene. The total length of the semantic region boundary, Lbdry, reflects the complexity of semantic boundaries in the scene. The number of foreground instances, Ninst, reflects the number of dynamic objects or independent targets.

[0036] In one implementation, Ccls, Lbdry, and Ninst are all normalized to between 0 and 1 before participating in the calculation. The normalization method can be the ratio between the statistical value of the current semantic scene segment and the preset maximum value, or it can be the normalization method of the maximum value in the current video sequence. Through normalization processing, the numerical magnitude of the total length of the semantic region boundary is avoided from being significantly greater than the total number of semantic categories and the number of foreground instances, which would lead to an imbalance in the semantic complexity calculation results.

[0037] The adaptive video segmentation unit determines the segmentation granularity based on the semantic complexity Scomp. If Scomp is less than the first complexity threshold, semantic region-level segmentation is used. If Scomp is greater than or equal to the first complexity threshold and less than the second complexity threshold, semantic region-plus-boundary refinement segmentation is used. If Scomp is greater than or equal to the second complexity threshold, foreground instance-level segmentation is used, and local high-granularity segmentation windows are set in the dynamic foreground region or the neighborhood of the stitching seam.

[0038] The weighting coefficients α, β, and γ are dynamically optimized by minimizing the mean square error between them and the splicing boundary alignment error value fed back in step S5. Specifically, after each semantic scene segment is spliced, the system calculates the splicing boundary alignment error value corresponding to that semantic scene segment and calculates Ccls, Lbdry, and Ninst for that semantic scene segment. The system uses the splicing boundary alignment error value as the target feedback quantity to update α, β, and γ, so that the semantic complexity calculation results of subsequent similar semantic scene segments can more accurately reflect the splicing difficulty.

[0039] In one implementation, if the alignment error value of the splicing boundary of a certain semantic scene segment continues to increase, and the semantic scene segment has a long semantic boundary or many foreground instances, the system increases β or γ so that subsequent similar segments can use finer boundary segmentation or instance-level segmentation. If the splicing of a certain semantic scene segment is stable and the semantic complexity is low, the system reduces the corresponding weight or uses a coarser segmentation granularity to reduce the amount of computation.

[0040] Example 3: This example further illustrates the semantic consistency verification and splicing strategy pool invocation method based on Example 1.

[0041] In step S4, the semantic-guided stitching unit extracts the semantic category labels of each pixel on both sides of the stitching boundary and establishes a semantic consistency matrix for the stitching boundary. The elements in the semantic consistency matrix for the stitching boundary are used to indicate whether the semantic category labels at corresponding positions on both sides of the stitching boundary are consistent, or to indicate the similarity between the semantic category confidence vectors on both sides of the stitching boundary.

[0042] When the semantic category label matching degree at corresponding positions on both sides of the splicing boundary is lower than the preset semantic consistency threshold, it is determined to be semantically inconsistent. At this time, the semantically guided splicing unit performs different repair operations based on the semantic category and region attributes.

[0043] For segmented sub-images with static background semantic categories, the semantically guided stitching unit recalculates the optimal stitch line within the neighborhood of the stitching boundary and uses a graph cut algorithm guided by the semantic boundary heatmap to find a stitching path that avoids the semantic boundary. This approach can reduce the probability of stitching seams crossing building outlines, road boundaries, horizons, or other stable semantic boundaries.

[0044] For segmented sub-images with regular geometric shapes and semantic categories, the semantically guided stitching unit elastically matches the semantic boundary masks on both sides of the stitching boundary, uses the matched semantic boundary as the stitching line, and employs the Poisson equation solution method to smooth the color and texture transition in the neighborhood of the stitching line. This method can improve the geometric alignment stability of regular boundary regions.

[0045] For segmented sub-images with dynamic foreground semantic categories, the semantically guided stitching unit inputs the stitching overlap region into a pre-constructed conditional generative adversarial network (GAN). Using the semantic category confidence distribution as a constraint, it generates complete semantically continuous image content through confidence-weighted feature resampling. This approach can reduce foreground fragmentation, ghosting, and occlusion residue caused by dynamic target motion.

[0046] In one implementation, the splicing strategy pool is not a fixed call table, but rather the strategy priority is dynamically determined based on semantic category, semantic category confidence, historical splicing quality, and responsibility attribution results. Static background regions preferentially invoke the first splicing strategy. Regions with regular geometric shapes preferentially invoke the second splicing strategy. Dynamic foreground regions and low-confidence regions preferentially invoke the third splicing strategy. When a splicing strategy mismatch with the responsibility component meets the dominant condition, the system reorders the priority of the splicing strategy pool corresponding to the semantic category.

[0047] Example 4: This example further illustrates the multi-task joint training method of the scene semantic recognition network based on Example 1.

[0048] The scene semantic recognition network is obtained through multi-task joint training. This multi-task joint training includes pixel-level semantic segmentation, scene classification, and temporal scene boundary detection. All three tasks share the underlying feature extraction network parameters. The joint loss function, Ljoint, is expressed as follows. Where Lseg is the cross-entropy loss for pixel-level semantic segmentation, Lscene is the multi-class cross-entropy loss for scene classification, Lbdry is the focus loss function for temporal scene transition boundary detection, and ω1, ω2, and ω3 are the balancing weight coefficients for each loss term.

[0049] The cross-entropy loss Lseg for pixel-level semantic segmentation tasks is expressed by the following formula. , where N is the total number of pixels, C is the number of semantic categories, yi,c is the true label indicator function of pixel i belonging to category c, and pi,c is the probability that the network predicts that pixel i belongs to category c.

[0050] The multi-class cross-entropy loss Lscene for scene classification tasks is expressed by the following formula. Where K is the total number of scene categories, zk is the real scene label, and qk is the global scene probability predicted by the network.

[0051] The focus loss function Lbdry for the temporal scene transition boundary detection task is expressed by the following formula. Where M is the number of boundary detection locations, bj is the boundary probability value predicted by the network, aj is an indicator variable indicating whether boundary detection location j belongs to the true boundary, and δ is a hyperparameter of the focus loss. The focus loss function can alleviate the class imbalance problem between scene boundary samples and non-boundary samples, improving the recognition accuracy of low-frequency scene transition boundaries.

[0052] In one implementation, ω1, ω2, and ω3 are tuned on the validation set using grid search or Bayesian optimization. The tuning objectives include improving pixel-level semantic segmentation accuracy, scene classification accuracy, and scene transition boundary localization accuracy.

[0053] Example 5: This example further illustrates the semantic boundary heatmap-guided graph-based image segmentation algorithm based on Example 1.

[0054] The adaptive video segmentation unit employs a graph-based image segmentation algorithm guided by a semantic boundary heatmap. This algorithm uses pixels with a boundary probability value greater than a first threshold in the scene transition boundary heatmap as strong constraint anchors, forcing the segmentation boundaries of the strong constraint anchors and their neighborhoods to align with the scene transition boundary heatmap.

[0055] When the boundary probability value is between the first and second thresholds, the adaptive video segmentation unit allows the segmentation boundary to shift flexibly within a certain range. This flexible shift range can be determined based on the boundary probability value, region confidence, and stitching quality evaluation parameters. A higher boundary probability value allows for a smaller allowable shift range, while a lower boundary probability value allows for a larger allowable shift range.

[0056] In this method, high-confidence semantic boundaries are used as stable segmentation criteria, while low-confidence boundary regions retain adjustable space. This allows for maintaining the accuracy of semantic boundary alignment while also ensuring the flexibility of segmenting low-confidence regions, avoiding subsequent splicing misalignment caused by incorrectly fixed boundaries.

[0057] Example 6: This example further illustrates the bidirectional data channel, secondary segmentation request, and attribution of responsibility for splicing anomalies based on Example 1.

[0058] A bidirectional data channel is established between the adaptive video segmentation unit and the semantically guided stitching unit. The adaptive video segmentation unit transmits segmented sub-images and their segmentation semantic descriptions to the semantically guided stitching unit through this bidirectional data channel. The semantically guided stitching unit feeds back stitching quality evaluation parameters to the adaptive video segmentation unit through this bidirectional data channel.

[0059] When the semantically guided stitching unit encounters a low-confidence region during stitching, it requests re-segmentation from the adaptive video segmentation unit. The re-segmentation request includes at least the abnormal frame time index, abnormal region coordinates, responsibility attribution vector, current stitching strategy identifier, stitching boundary alignment error value, and semantic consistency score of the overlapping stitching region. The adaptive video segmentation unit responds to the re-segmentation request, adjusts the segmentation granularity parameters, performs secondary segmentation on the video frame containing the low-confidence region, and retransmits the corrected segmentation result back to the semantically guided stitching unit.

[0060] The attribution process for splicing anomalies includes obtaining the splicing boundary alignment error value, the semantic consistency score of the splicing overlap region, the semantic category confidence distribution on both sides of the splicing boundary, the number of times the splicing seam crosses the semantic boundary, and the dynamic foreground optical flow residual. The system generates an attribution vector R based on these parameters.

[0061] The responsibility attribution vector R is represented in the following form. Where Rb is the boundary offset responsibility component, Rg is the insufficient granularity responsibility component, Rs is the scene switching misjudgment responsibility component, Rm is the dynamic foreground interference responsibility component, and Rp is the splicing strategy mismatch responsibility component.

[0062] The boundary offset responsibility component Rb is determined based on the semantic boundary heatmap offset of the stitching seam neighborhood and the stitching boundary alignment error value. When the stitching boundary alignment error value is high, and the semantic boundary heatmap within the stitching seam neighborhood shows an offset between the true semantic boundary and the current segmentation boundary, Rb increases.

[0063] The insufficient granularity responsibility component Rg is determined based on the semantic complexity score, segmentation granularity level, and splicing boundary alignment error value. When the semantic complexity is high, the current segmentation granularity is coarse, and the splicing boundary alignment error value is high, Rg increases.

[0064] The responsibility component Rs for scene transition misjudgment is determined based on the time distance between the location of the splicing anomaly and the scene transition point, the gradient of the scene transition boundary heatmap change, and the magnitude of the semantic category change in adjacent frames. When splicing anomalies are concentrated before and after the scene transition point, and there are abnormal peaks in the boundary probability gradient, Rs increases.

[0065] The dynamic foreground interference responsibility component Rm is determined based on the dynamic foreground optical flow residual and the area ratio of the dynamic foreground region. The dynamic foreground optical flow residual can be obtained from the difference between the optical flow vector of adjacent frames and the predicted displacement of the stitching transformation. When the dynamic foreground optical flow residual is large, Rm increases.

[0066] The mismatch responsibility component Rp is determined based on the historical splicing quality of the current splicing strategy, the current semantic consistency score, and the current segmentation confidence. Rp increases when the segmentation boundary confidence is high but the semantic consistency score under the current splicing strategy remains below the threshold.

[0067] In one implementation, the dominant condition is that a certain responsibility component in the responsibility attribution vector reaches its maximum value, which is higher than a preset responsibility threshold, and the difference between this maximum value and the second largest responsibility component is greater than a preset separation threshold. When this condition is met, the system determines that the responsibility component is the dominant cause of the current splicing anomaly.

[0068] When the boundary offset responsibility component or the insufficient granularity responsibility component meets the preset dominant condition, the system triggers the adaptive video segmentation unit to perform secondary segmentation on the local region within the stitching seam neighborhood. The stitching seam neighborhood can be defined as a strip-shaped region centered on the stitching seam and extending w pixels to both sides. w can be determined based on the video resolution, average foreground target width, or stitching boundary error magnitude.

[0069] When the dynamic foreground interference responsibility component meets the preset dominant condition, the semantically guided splicing unit reduces the feature matching weight of the dynamic foreground region and calls the splicing strategy based on generative semantic repair.

[0070] When the mismatch between the concatenation strategy and the responsible component meets the preset dominant condition, the system maintains the current segmentation result and rearranges the priority of the concatenation strategy pool for the corresponding semantic category. This avoids mistakenly reporting concatenation anomalies caused by incorrect concatenation strategy selection to the segmentation module.

[0071] Example 7: This example provides an automatic video image segmentation and stitching system based on scene semantic recognition.

[0072] The system includes a video input parsing module, a scene semantic collaborative perception module, an adaptive semantic segmentation module, a semantically guided collaborative splicing module, a segmentation and splicing closed-loop collaborative bus, and a time-series fusion output module.

[0073] The video input parsing module is used to acquire and parse the raw video stream. It can perform video decoding, frame sequence parsing, time indexing, and frame quality detection.

[0074] The scene semantic collaborative perception module is connected to the video input parsing module, and is used to simultaneously output pixel-level scene semantic label maps, scene switching boundary heatmaps, and confidence distributions of scene semantic categories for the parsed video frames.

[0075] The adaptive semantic segmentation module is connected to the output of the scene semantic collaborative perception module. The adaptive semantic segmentation module includes a scene boundary detection unit, a semantic complexity calculation unit, a segmentation granularity decision and execution unit, and a secondary segmentation triggering unit. The scene boundary detection unit identifies scene transition points based on the scene transition boundary heatmap. The semantic complexity calculation unit calculates the semantic complexity score. The segmentation granularity decision and execution unit determines and executes the segmentation granularity based on the semantic complexity score. The secondary segmentation triggering unit performs secondary segmentation on low-confidence regions or regions of responsibility based on re-segmentation requests.

[0076] The semantically guided collaborative stitching module is connected to the output of the adaptive semantic segmentation module. The semantically guided collaborative stitching module includes a semantic matching unit, a policy pool invocation unit, a semantic consistency verification unit, and a low-confidence feedback unit. The semantic matching unit matches segmented sub-images based on the segmentation semantic description information. The policy pool invocation unit invokes the corresponding stitching strategy based on the semantic category and responsibility attribution results. The semantic consistency verification unit detects semantic consistency on both sides of the stitching boundary. The low-confidence feedback unit generates a re-segmentation request when a low-confidence region is detected.

[0077] The segmentation and splicing closed-loop collaborative bus connects the semantically guided collaborative splicing module and the adaptive semantic segmentation module, and is used for bidirectional transmission of splicing quality evaluation parameters and re-segmentation request commands. The segmentation and splicing closed-loop collaborative bus can adopt an asynchronous message passing protocol. The asynchronous message passing protocol enables the splicing module to continue processing other stable regions after generating a re-segmentation request, and to automatically resume the splicing processing flow of abnormal regions after receiving the re-segmentation result.

[0078] The temporal fusion output module is used to fuse and stitch together scene segments according to time sequence. It can smooth the transition between adjacent scene segments, suppress jitter at stitching boundaries, and output a complete video stitching sequence.

[0079] Example 8: This example further illustrates the asynchronous closed-loop control process based on Example 7.

[0080] When the semantically guided collaborative stitching module detects that the stitching quality evaluation parameters exceed a preset threshold, the low-confidence feedback unit automatically generates a re-segmentation request command. The re-segmentation request command is sent to the adaptive semantic segmentation module via the segmentation and stitching closed-loop collaborative bus.

[0081] After receiving a re-segmentation request instruction, the adaptive semantic segmentation module reads the time index of the abnormal frame, the coordinates of the abnormal region, the responsibility attribution vector, and the current segmentation granularity parameters. If the responsibility attribution vector indicates boundary offset or insufficient granularity, the adaptive semantic segmentation module adjusts the segmentation granularity parameters and the scene switching boundary judgment threshold, and performs secondary segmentation on the corresponding video frame or the neighborhood of the stitching seam.

[0082] After receiving the resegmentation result, the semantically guided collaborative stitching module replaces the original segmentation result with the resegmentation result and automatically resumes the stitching process for the abnormal region. This forms a complete closed-loop control link from stitching anomaly detection, resegmentation request, local secondary segmentation, return of segmentation result, to stitching process recovery.

[0083] Example 9: This example further illustrates the joint optimization objective function.

[0084] The adaptive semantic segmentation module and the semantically guided collaborative assembly module jointly optimize the objective function Etotal through a segmentation and assembly closed-loop collaborative bus. Minimizing Etotal is the joint optimization objective, expressed as the following formula. Where Eseg is the segmentation loss, Estich is the splicing loss, Esync is the semantic closure synchronization error between the segmentation boundary and the splicing boundary, and λ1, λ2, and λ3 are the balancing weights.

[0085] The segmentation loss Eseg measures the difference between the pixel-level scene semantic label map output by the scene semantic recognition network and the annotations on the training samples. The stitching loss Estich measures the geometric alignment error, color continuity error, and structural similarity error of the image content on both sides of the stitching boundary. The semantic closure synchronization error Esync measures the difference between the segmented semantic description information at the stitching boundary and the actual semantic category label after stitching.

[0086] In one implementation, the actual semantic category label after splicing is obtained by re-inputting the splicing boundary neighborhood from the splicing result into the scene semantic recognition network. In another implementation, the actual semantic category label after splicing is obtained by weighted fusion of the semantic category confidence distributions on both sides of the splicing boundary.

[0087] The semantic closure synchronization error (Esync) is defined as the Frobenius norm of the vector difference between the segmented semantic description information and the actual semantic category label after concatenation at each concatenation boundary. By introducing the semantic closure synchronization error, the optimization direction of the segmentation boundary can be constrained by the concatenation result, and the concatenation result can also influence the subsequent segmentation granularity and boundary determination threshold, thereby improving the consistency between the segmentation result and the concatenation result.

[0088] Example 10: This example illustrates the method for evaluating the effectiveness of the present invention.

[0089] In one implementation, the processing effect of the present invention can be evaluated by the splicing boundary alignment error, the semantic consistency score of the splicing overlapping area, the number of times the splicing seam crosses the semantic boundary, the dynamic foreground ghosting area, the number of secondary segmentation triggers, the proportion of the local secondary segmentation area to the full frame area, and the video temporal jitter amplitude.

[0090] The lower the alignment error of the splicing boundary, the more consistent the geometric structure on both sides of the splicing boundary; the higher the semantic consistency score of the splicing overlap area, the more consistent the semantic categories on both sides of the splicing overlap area; the fewer times the splicing seam crosses the semantic boundary, the better the splicing path can avoid the boundaries of key semantic objects; the smaller the area of ​​dynamic foreground ghosting, the lower the dynamic foreground splicing interference; the smaller the proportion of the local secondary segmentation area to the full frame area, the better the system can complete targeted back-repair in a smaller area; and the lower the video temporal jitter amplitude, the more stable the continuous video splicing sequence.

[0091] In one implementation, three sets of comparison schemes can be set. The first set of comparison schemes uses ordinary feature point splicing, the second set of comparison schemes uses semantic segmentation-assisted splicing, and the third set of schemes uses the segmentation splicing closed-loop collaborative method of the present invention. By comparing the splicing boundary alignment error, semantic consistency score, dynamic foreground ghosting area and processing latency of the three sets of schemes, the splicing stability and closed-loop repair capability of the present invention in complex semantic scenes can be verified.

[0092] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for automatic segmentation and stitching of video images based on scene semantic recognition, characterized in that, Includes the following steps: S1: Obtain the raw video stream, parse its frame sequence, and obtain a multi-frame continuous video image containing time index; S2: Input the multi-frame continuous video images into the scene semantic recognition network. The scene semantic recognition network outputs a pixel-level scene semantic label map, a scene switching boundary heatmap, and a confidence distribution of scene semantic categories for each frame of video image. S3: Use an adaptive video segmentation unit to perform dynamic granular segmentation based on scene semantics: identify scene switching points according to the boundary probability gradient in the scene switching boundary heatmap, and segment the video stream into multiple semantic scene segments; adaptively determine the segmentation granularity based on the semantic complexity of each semantic scene segment, and segment the semantic scene segment according to the determined segmentation granularity to generate segmented sub-images carrying segmentation semantic description information. S4: The semantically guided stitching unit stitches the segmented sub-images: Based on the segmentation semantic description information, segmented sub-images from different perspectives or time periods are matched according to semantic categories; a stitching strategy pool corresponding to each semantic category is constructed, and the corresponding stitching strategy is called from the strategy pool according to the matched semantic category to perform stitching; semantic consistency verification is performed when stitching segmented sub-images of different semantic categories; S5: A closed-loop feedback link between segmentation and stitching is constructed, using the stitching boundary alignment error value and the semantic consistency score value of the stitching overlap area as stitching quality evaluation parameters, which are fed back to the adaptive video segmentation unit in real time to dynamically correct the segmentation granularity parameters and scene switching boundary judgment threshold of subsequent frames, thereby forming a deep coupling between segmentation and stitching, and realizing the co-evolution between the adaptive video segmentation unit and the semantically guided stitching unit; S6: Merge the spliced ​​scene segments according to time sequence to output a complete video splicing sequence.

2. The automatic video image segmentation and stitching method based on scene semantic recognition as described in claim 1, characterized in that: The semantic consistency verification in S4 specifically includes: extracting the semantic category labels of each pixel on both sides of the splicing boundary and establishing a semantic consistency matrix of the splicing boundary; when the matching degree of the semantic category labels at corresponding positions on both sides of the splicing boundary is lower than the preset semantic consistency threshold, it is determined to be semantically inconsistent, and the following repair operations are performed: for the segmented sub-image of the static background semantic category, the optimal stitching line is recalculated in the neighborhood of the splicing boundary, and a graph cut algorithm guided by the semantic boundary heatmap is used to find a splicing path that avoids the semantic boundary; for the segmented sub-image of the dynamic foreground semantic category, the splicing overlap region is input into a pre-constructed conditional generative adversarial network, and the semantic category confidence distribution is used as a conditional constraint to generate complete semantic continuous image content.

3. The automatic video image segmentation and stitching method based on scene semantic recognition as described in claim 2, characterized in that: The semantic complexity in S3 is calculated using the following formula: ;in, The total number of semantic categories. The total length of the semantic region boundary. Foreground instance count The weighting coefficients are dynamically optimized by minimizing the mean square error between the stitching boundary alignment error value fed back in S5, thereby quantifying the stitching quality feedback into a correction signal for the segmentation parameters and realizing deep coupling between the adaptive video segmentation unit and the semantically guided stitching unit.

4. The automatic video image segmentation and stitching method based on scene semantic recognition as described in claim 3, characterized in that: S4, which involves calling the corresponding stitching strategy from the strategy pool based on semantic category to perform stitching, further includes: for segmented sub-images with a static background semantic category, calling the first stitching strategy, performing feature point matching based on scale-invariant feature transformation feature point extraction and random sampling consistency algorithm to achieve geometric alignment stitching; for segmented sub-images with a regular geometric shape semantic category, calling the second stitching strategy, elastically matching the semantic boundary masks on both sides of the stitching boundary, using the matched semantic boundary as the stitching line for stitching, and using the Poisson equation solution method to perform smooth transition processing of color and texture in the neighborhood of the stitching line; for segmented sub-images with a dynamic foreground semantic category, calling the third stitching strategy, inputting the stitching overlap region into a pre-constructed conditional generative adversarial network, using the semantic category confidence distribution as a conditional constraint, and generating completed semantically continuous image content through confidence-weighted feature resampling.

5. The automatic video image segmentation and stitching method based on scene semantic recognition as described in claim 4, characterized in that: The scene semantic recognition network is obtained through multi-task joint training, which includes the joint training of pixel-level semantic segmentation, scene classification, and temporal scene boundary detection tasks. These three tasks share the underlying feature extraction network parameters, and their joint loss function is: ;in: The cross-entropy loss is used for pixel-level semantic segmentation tasks, where N is the total number of pixels and C is the number of semantic categories. Let i be the true label indicator function for pixel i belonging to category c. The probability predicted by the network; For scene classification tasks, the multi-class cross-entropy loss is used, where K is the total number of scene categories. Labeled as real-world scenarios The global scene probability predicted by the network; Let M be the focus loss function for the temporal scene switching boundary detection task, where M is the number of boundary detection locations. These are the boundary probability values ​​predicted by the network. and is a hyperparameter of the focus loss, used to alleviate the class imbalance problem between scene boundary samples and non-boundary samples; The balancing weight coefficients for each loss term are tuned on the validation set using grid search or Bayesian optimization.

6. The automatic video image segmentation and stitching method based on scene semantic recognition as described in claim 5, characterized in that: The adaptive video segmentation unit in S3 employs a graph-based image segmentation algorithm guided by a semantic boundary heatmap. Pixels with boundary probability values ​​greater than a first threshold in the scene transition boundary heatmap are used as strong constraint anchors. The segmentation boundaries of the strong constraint anchors and their neighborhoods are forcibly aligned with the scene transition boundary heatmap. When the boundary probability value is between the first and second thresholds, the segmentation boundary is allowed to shift flexibly within a certain range, thereby maintaining the semantic boundary alignment accuracy while taking into account the segmentation flexibility of low-confidence regions.

7. The automatic video image segmentation and stitching method based on scene semantic recognition as described in claim 6, characterized in that: The system also includes establishing a bidirectional data channel between the adaptive video segmentation unit and the semantically guided stitching unit: the adaptive video segmentation unit transmits segmented sub-images and their segmentation semantic description information to the semantically guided stitching unit; the semantically guided stitching unit feeds back stitching quality evaluation parameters to the adaptive video segmentation unit, and when the semantically guided stitching unit encounters a low-confidence region during stitching, it requests re-segmentation from the adaptive video segmentation unit. The adaptive video segmentation unit responds to this request, adjusts the segmentation granularity parameters to perform secondary segmentation on the video frame to which the low-confidence region belongs, and retransmits the corrected segmentation result to the semantically guided stitching unit; S5 also includes a stitching anomaly attribution step, which includes: obtaining stitching boundary alignment error values, semantic consistency evaluation of stitching overlapping areas, etc. The system calculates the scores, semantic category confidence distributions on both sides of the stitching boundary, the number of times the stitching seam crosses the semantic boundary, and the dynamic foreground optical flow residual. Based on these parameters, a responsibility attribution vector is generated, which includes a boundary offset responsibility component, a granularity insufficient responsibility component, a scene switching misjudgment responsibility component, a dynamic foreground interference responsibility component, and a stitching strategy mismatch responsibility component. When the boundary offset responsibility component or the granularity insufficient responsibility component meets a preset dominant condition, the adaptive video segmentation unit is triggered to perform secondary segmentation on the local region in the neighborhood of the stitching seam. When the dynamic foreground interference responsibility component meets a preset dominant condition, the feature matching weight of the dynamic foreground region is reduced and a stitching strategy based on generative semantic repair is invoked. When the stitching strategy mismatch responsibility component meets a preset dominant condition, the current segmentation result remains unchanged and the priority of the stitching strategy pool corresponding to the semantic category is rearranged.

8. A video image automatic segmentation and stitching system based on scene semantic recognition, characterized in that, This system, centered on establishing deep coupling and co-evolution between segmentation and stitching, includes: a video input parsing module for acquiring and parsing the original video stream; a scene semantic collaborative perception module connected to the video input parsing module for simultaneously outputting pixel-level scene semantic label maps, scene transition boundary heatmaps, and confidence distributions of scene semantic categories for the parsed video frames; and an adaptive semantic segmentation module connected to the output of the scene semantic collaborative perception module, comprising a scene boundary detection unit, a semantic complexity calculation unit, a segmentation granularity decision and execution unit, and a secondary segmentation triggering unit, for performing segmentation based on scene semantics. The system includes: a dynamic granular segmentation module; a semantically guided collaborative stitching module connected to the output of the adaptive semantic segmentation module, comprising a semantic matching unit, a strategy pool invocation unit, a semantic consistency verification unit, and a low-confidence feedback unit, used to invoke the corresponding stitching strategy according to the semantic category of the segmented sub-images to perform stitching, and to perform semantic consistency verification on the stitching process; a segmentation-stitching closed-loop collaborative bus connecting the semantically guided collaborative stitching module and the adaptive semantic segmentation module, used for bidirectional transmission of stitching quality evaluation parameters and re-segmentation request instructions; and a temporal fusion output module used to fuse the stitched scene segments according to the time sequence.

9. The video image automatic segmentation and stitching system based on scene semantic recognition as described in claim 8, characterized in that: The segmentation-stitching closed-loop collaborative bus adopts an asynchronous message passing protocol. When the semantically guided collaborative stitching module detects that the stitching quality evaluation parameters exceed the preset threshold, it automatically generates a re-segmentation request command and sends it to the adaptive semantic segmentation module via the segmentation-stitching closed-loop collaborative bus. This triggers the module to adjust the segmentation granularity parameters to perform secondary segmentation on the corresponding video frames. After receiving the re-segmentation result, the semantically guided collaborative stitching module automatically resumes the stitching process, forming a complete closed-loop control link.

10. The video image automatic segmentation and stitching system based on scene semantic recognition as described in claim 9, characterized in that: The adaptive semantic segmentation module and the semantically guided collaborative splicing module jointly optimize the following objective function through the segmentation-splicing closed-loop collaborative bus: ;in, To divide the loss, Due to splicing losses, The semantic closure synchronization error between the segmentation boundary and the splicing boundary is defined as the Frobenius norm of the vector difference between the segmented semantic description information and the actual semantic category label after splicing at each splicing boundary. To balance the weights.

Images