Multi-source video dynamic splicing method based on artificial intelligence and space-time grid

Abstract

The application discloses a multi-source video dynamic splicing method based on artificial intelligence and a space-time grid, and relates to the technical field of multi-source video splicing, and comprises the following steps: acquiring multiple types of video sources and establishing a unified space-time reference, including a space grid and a time grid; under the reference, a dynamic time slice is defined for time alignment of the video sources; time allocation and a time matching cost function optimization are performed on frame images of each video source to determine effective frames; further region planning is performed to form a coarse field of view envelope, and semantic feature parameters of the effective frames are extracted; based on candidate covering grids and the feature parameters, a credible observation degree is calculated, overlapping grids are screened, pose and mapping parameter cascade registration is performed, and finally dynamic splicing of the effective frames of each type of video source in the dynamic time slice is realized to generate a single panoramic image; and the actual situation can be better reflected under a dynamic scene.

Description

Technical Field

[0001] This invention relates to the field of multi-source video stitching technology, specifically a method for dynamic stitching of multi-source videos based on artificial intelligence and spatiotemporal grids. Background Technology

[0002] In modern video surveillance, intelligent transportation, virtual reality and other application scenarios, dynamic stitching technology is widely used to integrate image information from multiple video sources into a continuous panoramic view; this technology can effectively improve user experience and provide more comprehensive and accurate visual information. Existing technologies typically rely on static temporal and spatial reference systems to stitch together multi-source videos through image processing and feature matching. These technologies generally employ feature-point-based matching algorithms to analyze the similarity between images and perform corrections through geometric transformations. However, when dealing with dynamic scenes, especially when the video shooting medium exhibits different speeds and perspective changes, this approach often faces problems such as insufficient temporal alignment and low stitching accuracy. This results in flaws in the stitching results and a decrease in stitching accuracy. Current technologies cannot effectively utilize the spatiotemporal information of video sources for optimization when processing dynamic video sources, and often lack intelligent support in video frame selection and stitching strategies. Existing methods cannot respond to the dynamic changes of different video sources in real time, making it difficult to generate high-quality synthetic panoramic images. Therefore, how to accurately and efficiently stitch together different types of video sources has become an urgent technical problem to be solved.

[0003] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide a method for dynamic stitching of multi-source videos based on artificial intelligence and spatiotemporal grids to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A method for dynamic stitching of multi-source videos based on artificial intelligence and spatiotemporal grids, comprising the following steps: Acquire multiple types of video sources to be merged, and establish a unified spatiotemporal reference for each type of video source. The spatiotemporal reference specifically includes a spatial grid reference and a temporal grid reference. Under the spatiotemporal reference, based on the shooting carrier speed of each type of video source, define several dynamic time slices for video source time alignment. For any type of video source, each frame of the video source is allocated to each dynamic time slice in chronological order, and a time matching cost function is set according to the time difference between each frame and the center of the dynamic time slice. By minimizing the value of the time matching cost function, the effective frame images in each dynamic time slice are determined. For any type of video source, region planning is performed on any valid frame image to form a coarse field of view envelope. The spatial grid of the pixels covered by the coarse field of view envelope under the spatiotemporal reference is used as a candidate coverage grid. Based on the candidate coverage grid and the valid frame image, it is input into the trained artificial intelligence model to obtain the semantic feature parameters of the valid frame image. Based on the candidate coverage grid and the semantic feature parameters under the corresponding grid, the observation patches of the effective frame image in each candidate coverage grid are determined. The confidence observation degree of each candidate coverage grid is calculated according to the observation patch. Based on the state of the observation patch, overlapping grids are selected in the candidate coverage grid. The video source pose and image mapping parameters are concatenated and registered for the overlapping grids to obtain the position, pose and mapping parameter optimization values ​​of each overlapping grid. Based on the position and pose of each overlapping grid, the optimized values ​​of the mapping parameters, and the reliability of the observations, the effective frame images of various types of video sources within the dynamic time slice are dynamically stitched together to obtain a single panoramic image within the dynamic time slice.

[0006] Furthermore, the various video sources include fixed surveillance cameras, PTZ cameras, drone videos, robot dog videos, vehicle-mounted videos, mobile terminal videos, and temporary deployment videos; The establishment of a unified spatiotemporal benchmark includes: A multi-level spatial grid benchmark is constructed based on a unified geographic coordinate system to uniformly locate and grid-represent the spatial position, field of view coverage, and candidate coverage areas of different types of video sources. A multi-level time grid reference is constructed based on a unified standard time to uniformly calibrate the acquisition time, time alignment process, and dynamic time slices of different types of video sources; The spatial grid reference has different spatial resolutions at each level, and the temporal grid reference has different temporal resolutions at each level, so as to support frame-level allocation, grid coverage analysis and dynamic stitching of multiple types of video sources under the same spatiotemporal reference system.

[0007] Furthermore, the method for constructing the dynamic time slice includes: Based on the start time of the video source, the video timeline is initially divided according to the preset basic time slice length; For each initial time slice, a time tolerance window is set according to the movement speed of the corresponding shooting vehicle, and the time boundary of the initial time slice is adjusted using the time tolerance window to obtain several dynamic time slices. The time tolerance window is inversely proportional to the speed of the imaging carrier.

[0008] Furthermore, methods for setting the time-matching cost function include: For any type of video source, determine all frame images of that type of video source and extract the timestamp of each frame image; Based on the spatial coordinates and attitude matrix of the video source of this type under the timestamp of each frame image, the time deviation, spatial position deviation and attitude deviation of each frame image relative to the center time of each dynamic time slice are determined respectively. Based on the time deviation, spatial position deviation, and attitude deviation, and combined with preset weights, a time matching cost function is constructed for allocating each frame image to each dynamic time slice.

[0009] Furthermore, the method for determining the specific effective frame images within each dynamic time slice is as follows: For any type of video source, the allocation scheme of each frame image in the dynamic time slice is taken as the optimization object. The time matching cost function value under different frame image allocation schemes is analyzed. The allocation scheme corresponding to the minimum time matching cost function value is taken as the optimal allocation scheme. The frame images of each type of video source are allocated to the corresponding dynamic time slice according to the optimal allocation scheme. Then, the frame images contained in each dynamic time slice are the effective frame images.

[0010] Furthermore, methods for forming a coarse field of view envelope include: For any type of video source, extract the four corners, boundary sampling points and depth change points of its effective frame image, and project the four corners, boundary sampling points and depth change points onto the spatial grid reference to form the coarse field of view envelope of the effective frame image; The spatial grid covered by the coarse field of view envelope is determined as a candidate coverage grid; The effective frame image is input into the artificial intelligence model to extract the semantic feature parameters of the effective frame image. The semantic feature parameters include: semantic anchor points for characterizing the semantic positional relationship of the scene, image patches for characterizing the image content of the candidate coverage area, mapping parameters for mapping the image patches to the panoramic canvas, mask information for characterizing semantic category and occlusion state, quality and credibility information for characterizing image quality and credibility, and feature description information for characterizing local texture and motion state. Based on the candidate coverage grid, the semantic feature parameters are mapped to the corresponding candidate coverage grid to obtain observation patches of the effective frame image at each candidate coverage grid. The observation patch includes the image patch corresponding to the candidate coverage grid, panoramic canvas mapping parameters, semantic mask, occlusion mask, dynamic target mask, quality and credibility information, as well as feature descriptors, semantic anchors and motion vector summaries.

[0011] Furthermore, methods for calculating the confidence level of each candidate coverage grid based on the observation patch include: Extract the quality and reliability information of any frame image within any dynamic time slice in the corresponding observation slice of any candidate coverage grid; The quality and reliability information includes image quality score, geometric registration score, semantic consistency score, time synchronization score, historical stability score, and occlusion probability; The image quality score, geometric registration score, semantic consistency score, time synchronization score, and historical stability score are weighted according to preset weights, and the weighted results are subtracted and corrected based on the occlusion probability to obtain the confidence level of the image in the candidate coverage grid.

[0012] Furthermore, the method for calculating the geometric registration score includes: Extract multiple semantic anchor points from candidate coverage grid observation patches, and map each semantic anchor point to the grid semantic reference coordinate system based on the mapping parameters corresponding to the observation patches; The positional deviation between each semantic anchor point mapping result and the corresponding grid semantic reference anchor point is determined, and combined with the allowable error corresponding to the grid level where the candidate coverage grid is located, the positional deviation is comprehensively evaluated to obtain the geometric registration score of the observation patch corresponding to the candidate coverage grid. The method for calculating the semantic consistency score includes: The current semantic proportion of each semantic category within the observation patch corresponding to the candidate coverage grid is statistically analyzed, and the consistency of the current semantic proportion of each semantic category with the historical semantic proportion corresponding to the candidate coverage grid is compared. Based on the degree of overlap and difference between the proportions of each semantic category, the semantic consistency score of the observation patch corresponding to the candidate coverage grid is obtained.

[0013] Furthermore, for any dynamic time slice, all candidate coverage grids of each valid frame image within the dynamic time slice are determined. Based on the observation slices of the candidate coverage grids, the candidate coverage grids are filtered. If the observation slice of a candidate coverage grid is an empty set, the candidate coverage grid is removed. All candidate coverage grids within a valid frame image are traversed, and the remaining candidate coverage grids are marked as coverage grids. Within any dynamic time slice, the effective frame images corresponding to different types of video sources are determined by the intersection of the overlay grids to identify the overlapping grid set between the different types of video sources within that dynamic time slice. The specific logic underlying the cascaded registration of video source pose and image mapping parameters for overlapping meshes is as follows: the cascaded registration includes a coarse registration stage, a fine registration stage, and a global consistency optimization stage; The coarse registration stage outputs the pose correction amount of the video source based on the visual localization model of each type of video source. The fine registration stage corrects the mapping parameters of the image to the panoramic canvas, and obtains the correction amount of the mapping parameters. The global consistency optimization stage corrects the video source pose and image mapping parameters between different overlapping grids; This allows us to obtain the optimized values ​​of the position, orientation, and mapping parameters of each overlapping mesh.

[0014] Furthermore, dynamic stitching of valid frame images of various types of video sources within a dynamic time slice is performed. Specifically, the overlapping grids of each valid frame image are stitched together, including the color of the overlapping grid output based on the image blocks of the overlapping grid, the optimized value of the mapping parameters, the normalized fusion weight, and the mask set. The image is stitched together according to the color of the overlapping grid, that is, the color of the overlapping grid is used as the color of the corresponding overlapping grid in the stitched single panoramic image. The remaining area in the stitched single panoramic image is consistent with the corresponding valid frame image. The normalized fusion weights are specifically set based on the reliability of the overlapping grids, and the normalized fusion weights are proportional to the reliability of the observations.

[0015] This invention further provides a multi-source video dynamic stitching system based on artificial intelligence and spatiotemporal grids. The multi-source video dynamic stitching system is used to execute the aforementioned multi-source video dynamic stitching method based on artificial intelligence and spatiotemporal grids, comprising: The multi-source spatiotemporal reference module is used to acquire multiple types of video sources to be fused. For each type of video source, a unified spatiotemporal reference is established. The spatiotemporal reference specifically includes a spatial grid reference and a temporal grid reference. Under the spatiotemporal reference, based on the shooting carrier speed of each type of video source, several dynamic time slices for video source time alignment are defined. The dynamic time frame filtering module is used to allocate each frame of a video source to a dynamic time slice in chronological order for any type of video source. It sets a time matching cost function based on the time difference between each frame and the center of the dynamic time slice. By minimizing the value of the time matching cost function, the effective frame images in each dynamic time slice are determined. The candidate grid feature module is used to perform region planning for any type of video source and any valid frame image to form a coarse field of view envelope. The spatial grid of the pixels covered by the coarse field of view envelope under the spatiotemporal reference is used as a candidate coverage grid. Based on the candidate coverage grid and the valid frame image, it is input into the trained artificial intelligence model to obtain the semantic feature parameters of the valid frame image. The overlapping grid registration module is used to determine the observation patches of the effective frame image in each candidate coverage grid based on the semantic feature parameters of the candidate coverage grid and the corresponding grid. It calculates the confidence observation degree of each candidate coverage grid based on the observation patches, and selects overlapping grids in the candidate coverage grid based on the state of the observation patches. It performs cascade registration of video source pose and image mapping parameters on the overlapping grids to obtain the position, pose and optimized mapping parameter values ​​of each overlapping grid. The dynamic panoramic stitching module is used to dynamically stitch together effective frame images of various types of video sources within a dynamic time slice based on the position and pose of each overlapping grid, the optimized value of the mapping parameters, and the reliability of the observations, so as to obtain a single panoramic image within the dynamic time slice.

[0016] The present invention also provides a multi-source video dynamic stitching device based on artificial intelligence and spatiotemporal grids, including a processor and a memory communicatively connected to the processor. The memory stores a computer program that can be executed by the processor. The computer program is executed by the processor to enable the processor to implement the steps of the above-described multi-source video dynamic stitching method based on artificial intelligence and spatiotemporal grids.

[0017] The present invention further provides a computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps of the above-described method for dynamic stitching of multi-source videos based on artificial intelligence and spatiotemporal grids.

[0018] Compared with the prior art, the beneficial effects of the present invention are: This solution employs an artificial intelligence-based and spatiotemporal grid-based multi-source video dynamic stitching method. By establishing a unified spatiotemporal reference, it improves the accuracy and efficiency of multi-source video stitching. Firstly, the combination of spatial and temporal grid references makes the temporal alignment and spatial position correction of different types of video sources more scientific and systematic. Secondly, by defining dynamic time slices, optimization can be performed on the shooting speed of each video source, allowing for the effective aggregation of image frames within the same time period, thus significantly improving the temporal consistency and spatial continuity of the stitching result. This dual optimization in both time and space ensures that the final synthesized panoramic image is more realistic and can better reflect the actual situation in dynamic scenes. In the selection of effective frame images, this scheme utilizes a time-matching cost function to ensure that the selected frame images within each dynamic time slice are of high quality and high relevance by minimizing the cost function value. In addition, by forming a coarse field of view envelope through region planning and extracting semantic feature parameters with the help of a trained artificial intelligence model, a deeper understanding and processing of the effective frame images can be achieved. The introduction of semantic analysis makes the stitching process not only limited to the geometric feature matching of images, but also takes into account the semantic consistency of image content, thereby further improving the visual effect and information integrity of the synthesized panoramic image. Furthermore, the screening of overlapping grids and the cascaded registration of pose and mapping parameters based on reliable observation provide a more reliable basis for dynamic stitching; and the optimization of the mapping relationship between different video sources makes the final synthesized panoramic image more coherent and natural in visual performance; in addition, the stitching process within the dynamic time slice ensures the consistency of multi-source videos in the time dimension, enabling the final generated panoramic image to be seamlessly connected and providing a smooth viewing experience. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall method flow of the present invention; Figure 2 This is a schematic diagram of multi-channel surveillance video splicing; Figure 3 This is a diagram illustrating the stitching of surveillance video and drone video. Figure 4 This is a schematic diagram of the overall system structure of the present invention; Figure 5 This is a schematic diagram of the storage device structure and reading process of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0021] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0022] Example: Please see Figures 1-3 The present invention provides a technical solution: A method for dynamic stitching of multi-source videos based on artificial intelligence and spatiotemporal grids, comprising the following steps: Step 1: Obtain multiple types of video sources to be fused. For each type of video source, establish a unified spatiotemporal reference. The spatiotemporal reference specifically includes a spatial grid reference and a temporal grid reference. Under the spatiotemporal reference, based on the shooting carrier speed of each type of video source, define several dynamic time slices for video source time alignment.

[0023] The various video sources include fixed surveillance cameras, PTZ cameras, drone videos, robot dog videos, vehicle-mounted videos, mobile terminal videos, and temporary deployment videos. For fixed cameras, spatial parameters can be derived from installation logs, calibration documents, map points, and gimbal feedback. For drones, spatial parameters can be derived from GNSS, RTK, IMU, barometer, gimbal attitude, and flight control telemetry. For robot dogs, spatial parameters can be derived from SLAM, inertial navigation, odometry, UWB, BeiDou positioning, visual positioning, or scene map matching. For mobile terminals lacking high-precision positioning, position compensation can be performed using visual landmark recognition, Wi-Fi / Bluetooth positioning, map constraints, and historical trajectories. The spatial parameters specifically include longitude, latitude, altitude, heading angle, pitch angle, roll angle, and gimbal angle.

[0024] The establishment of a unified spatiotemporal benchmark includes: A multi-level spatial grid benchmark is constructed based on a unified geographic coordinate system to uniformly locate and grid-represent the spatial position, field of view coverage, and candidate coverage areas of different types of video sources. A multi-level time grid reference is constructed based on a unified standard time to uniformly calibrate the acquisition time, time alignment process, and dynamic time slices of different types of video sources. Specifically, the spatial grid is planned based on the GeoSOT standard (GB / T40087-2021). The spatial grid reference has different spatial resolutions at each level, and the temporal grid reference has different temporal resolutions at each level, to support frame-level allocation, grid coverage analysis, and dynamic stitching of multiple types of video sources under the same spatiotemporal reference system. Specifically, the spatial grid reference uses the CGCS2000 coordinate system to construct a 33-level spatial grid, with the spatial resolution of each level specifically expressed as follows: In the formula, For the first Spatial resolution of the spatial grid, This is an index for spatial hierarchy; where 20000×10³ meters is an approximate value for the circumference of the Earth's equator. It should be noted that, This is an index for spatial hierarchy, representing the spatial grid level. The higher the level, the finer the grid. The Earth's equatorial circumference is used as the overall reference benchmark for grid division because the Earth is an approximate sphere, and all spatial positioning needs to be based on the Earth's surface. Using the equatorial circumference as the basic value ensures that grid division has global uniformity and is applicable to various geographical locations. pass This indicates each additional level. The spatial grid resolution is halved. Through a recursive secondary partitioning method, the entire Earth's surface can be divided into increasingly smaller grids layer by layer. This ensures that, at any level and resolution, the spatial grid can be seamlessly stitched together and cover images acquired from all video sources. The hierarchical division method avoids the complex problem of calculating Earth coordinates; different levels of spatial grid resolution can be quickly generated through a simple exponential function.

[0025] The time grid reference uses UTC time, and time is divided into 16 levels from grade level to millisecond level. The time resolution of each time grid is as follows: In the formula, For the first The temporal resolution of the time grid is at the level of time. For time-level indexing, The maximum is 16.

[0026] It should be noted that, This is an index for the time hierarchy, representing the level of the time grid. The higher the level, the higher the time resolution and the finer the time division; the time division starts from one year (…). Starting with a year, which is one of the most commonly used basic units of time in the natural world, it is convenient for global time alignment. Using a year as the benchmark for the time grid can adapt to various video splicing application needs from long-term to short-term. pass The recursive partitioning logic can quickly generate time grids of different levels, avoiding complex time partitioning calculations. By setting a fixed 16-level time hierarchy, the number of time grid levels is limited, thereby controlling storage and computational overhead.

[0027] The method for constructing the dynamic time slice includes: Based on the start time of the video source, the video timeline is initially divided according to the preset basic time slice length; For each initial time slice, a time tolerance window is set according to the movement speed of the corresponding shooting vehicle, and the time boundary of the initial time slice is adjusted using the time tolerance window to obtain several dynamic time slices. The time tolerance window is inversely proportional to the speed of the imaging carrier.

[0028] The specific method for constructing dynamic time slices is as follows: For any type of video source, a time tolerance window is set based on the shooting medium speed. Based on this time tolerance window and a preset base time slice length, several dynamic time slices are defined. The specific formula used is as follows: In the formula, For the k-th dynamic time slice, Based on the base time slice length, The start time of the video source. The time tolerance window width is inversely proportional to the speed of the shooting medium, and k is the index of the dynamic time slice.

[0029] It should be noted that the base time slice length The basic time unit for each time slice is defined and is applicable to all video sources. This length should be set according to the actual application requirements and is related to the frame rate or sampling frequency of the video source. It is generally set between 0.017 seconds and 1 second, corresponding to the frame rate of different video sources. Tolerance window width It is set according to the speed of the shooting medium. Since the shooting medium moves faster, the time tolerance window should be relatively small. In a short time, the position of the moving object will change more, requiring more precise time alignment. For video sources with slow movement or static shooting, a larger time tolerance window can be set because the position will not change much in a longer time interval, and the time alignment requirements are relatively relaxed. By index This generates a series of continuous dynamic time slices, ensuring that the entire video time series is covered, which is suitable for subsequent analysis and processing. By setting a time tolerance window, the dynamic time slices can adapt to the time offset of shooting carriers at different speeds, ensuring the accuracy of time alignment.

[0030] Tolerance window width The specific formula used to set up the method is as follows: In the formula, For reference to the tolerance window width, Let be the average moving speed of the shooting carrier of the s-th video source. The reference speed of the shooting vehicle. This represents the total number of video source types. It should be noted that the key to tolerating changes in window width lies in the speed difference of the shooting medium. This reflects the differences in motion dynamics between different video sources. near When the speed difference is small, the tolerance window change is small. and When the speed difference is large, the tolerance window changes significantly to accommodate alignment requirements. The smoothing property of the natural logarithmic function effectively reduces the excessive impact of the speed difference on the tolerance window, preventing the tolerance window from becoming too large or too small. The format ensures that the impact of speed differences is relatively mild. Even if the speed differences between some video sources are large, the tolerance window will not change too drastically. Multiple video sources may have different shooting platform speeds, such as drones, ground cameras, and vehicle cameras. If the tolerance window is adjusted simply according to a single video source, the alignment effect will be unbalanced. This is achieved by adjusting the tolerance window based on the speed of the shooting platform. The average speed of each video source is calculated to ensure that the tolerance window setting has global adaptability and meets the fusion requirements of multiple video sources. Among them, as The base tolerance window width ensures that the time tolerance window has a basic width when the speed difference is small, and is the reference movement speed of the shooting vehicle. It is usually set to 0 to indicate a fixed camera or static scene.

[0031] Step 2: For any type of video source, allocate each frame of the video source to each dynamic time slice in chronological order, and set a time matching cost function based on the time difference between each frame and the center of its dynamic time slice. By minimizing the value of the time matching cost function, the effective frame images in each dynamic time slice are determined.

[0032] Methods for setting the time matching cost function include: For any type of video source, determine all frame images of that type of video source and extract the timestamp of each frame image; Based on the spatial coordinates and attitude matrix of the video source of this type under the timestamp of each frame image, the time deviation, spatial position deviation and attitude deviation of each frame image relative to the center time of each dynamic time slice are determined respectively. Based on the aforementioned time deviation, spatial position deviation, and pose deviation, and combined with preset weights, a time matching cost function is constructed for allocating each frame image to each dynamic time slice; the time matching cost function is specifically expressed as follows: In the formula, For the i-th frame of the s-th video source, assign the time-matching cost function value to the k-th dynamic time slice. Let i be the timestamp of the i-th frame of the s-th video source. The center time of the k-th dynamic time slice. Let be the spatial coordinates of the s-th video source in the i-th frame. This represents the center time of the k-th dynamic time slice, and the spatial coordinates of the s-th video source. Let be the pose matrix of the s-th video source in the i-th frame. Let be the center time of the k-th dynamic time slice, and be the pose matrix of the s-th video source. , and These are the weighting coefficients. Let represent the Euclidean norm between the spatial coordinates of the s-th video source in the i-th frame and the spatial coordinates of the s-th video source at the center moment of the k-th time slice. Let represent the norm between the pose matrix of the s-th video source in the i-th frame and the pose matrix of the s-th video source at the center moment of the k-th dynamic time slice, where i is the index of the video frame. It should be noted that time alignment is performed on video frames entering the same time slice. Time alignment is not simply selecting the nearest frame, but rather taking into account motion prediction and spatial coverage changes; This measure is used to measure the absolute time difference between the timestamp of the frame image and the center time of the dynamic time slice. A smaller time difference means that the frame image matches the time slice better, ensuring that the selected frame image and the time slice are consistent in time characteristics within the dynamic time slice, avoiding irrelevant or distorted content due to time misalignment. The term measures the Euclidean distance between the spatial coordinates of a frame image and the spatial coordinates at the center of a dynamic time slice. It reflects the spatial consistency of the frame image. By controlling the difference in spatial coordinates, it ensures that the image content within the dynamic time slice is relatively stable in space and consistent with the context information of the dynamic segment. The term utilizes the Frobenius norm to measure the difference between two pose matrices, reflecting the consistency of the video source in pose, such as shooting direction and angle, ensuring that the pose of the frame image matches the pose at the center of the time slice within the dynamic time slice, thereby avoiding image distortion caused by changes in viewpoint. The introduction of weighting coefficients allows for the setting of weighting coefficients in this embodiment when comprehensively considering time, space, and attitude matching. ,and Because ensuring temporal consistency is more important than spatial consistency within a time slice, therefore, [the following is set]. At its maximum, spatial coordinate matching is crucial for maintaining spatial consistency and contextual coherence within dynamic time slices; however, in many applications, minor spatial deviations typically do not significantly impact the overall effect as long as time alignment is correct. Therefore, [the following text is incomplete and requires further context to translate accurately: "to maximize the time alignment, small spatial deviations usually do not significantly affect the overall effect; therefore, to maximize the time alignment, small spatial deviations usually do not significantly affect the overall effect."] Setting the weight to the middle value reflects the importance of spatial matching, but is slightly less important than temporal matching.

[0033] The specific methods for obtaining spatial coordinates are as follows: If the video source is mobile, such as a drone or vehicle camera, its spatial coordinates can be obtained using GPS or a similar positioning system. Acceleration and angular velocity data are provided by the IMU, and combined with filtering algorithms, such as the Kalman filter, the spatial coordinates can be estimated. If multiple cameras are used, the spatial coordinates of the object can be obtained from images from multiple perspectives through 3D reconstruction technology, and the spatial coordinates can be mapped to a spatial grid reference.

[0034] The specific method for obtaining the attitude matrix is ​​as follows: the angular velocity and acceleration data provided by the IMU can be combined with integral calculations to obtain the attitude, such as roll, pitch and yaw angles, and then form the attitude matrix, or the attitude changes can be calculated from the optical flow between consecutive frames through camera motion estimation. The specific method for obtaining the timestamp of the i-th frame image is as follows: During the video source acquisition process, the acquisition time of each frame image is recorded. The corresponding timestamp can be directly extracted from the metadata of the video file. For multiple cameras, a synchronization mechanism, such as timecode or synchronization signal, is required to ensure the consistency of the timestamps of the frame images.

[0035] The method for determining the effective frame images within each dynamic time slice is as follows: For any type of video source, the allocation scheme of each frame image in the dynamic time slice is taken as the optimization object. The time matching cost function value under different frame image allocation schemes is analyzed. The allocation scheme corresponding to the minimum time matching cost function value is taken as the optimal allocation scheme. The frame images of each type of video source are allocated to the corresponding dynamic time slice according to the optimal allocation scheme. Then, the frame images contained in each dynamic time slice are the effective frame images.

[0036] By comprehensively considering time, space, and pose, the cost function provides a comprehensive matching criterion to improve the selection accuracy of effective frame images, ensure the consistency and relevance of image content within dynamic time slices, and efficiently filter effective content from a large number of frame images by minimizing the cost function, while reducing unnecessary computation and storage and optimizing subsequent processing and analysis.

[0037] Step 3: For any type of video source, perform region planning on any valid frame image to form a coarse field of view envelope. Use the spatial grid of the pixels covered by the coarse field of view envelope under the spatiotemporal reference as a candidate coverage grid. Based on the candidate coverage grid and the valid frame image, input it into the trained artificial intelligence model to obtain the semantic feature parameters of the valid frame image.

[0038] Methods for forming a coarse field of view envelope include: For any type of video source, extract the four corners, boundary sampling points and depth change points of its effective frame image, and project the four corners, boundary sampling points and depth change points onto the spatial grid reference to form the coarse field of view envelope of the effective frame image; The spatial grid covered by the coarse field of view envelope is determined as a candidate coverage grid; The effective frame image is input into the artificial intelligence model to extract the semantic feature parameters of the effective frame image. The semantic feature parameters include: semantic anchor points for characterizing the semantic positional relationship of the scene, image patches for characterizing the image content of the candidate coverage area, mapping parameters for mapping the image patches to the panoramic canvas, mask information for characterizing semantic category and occlusion state, quality and credibility information for characterizing image quality and credibility, and feature description information for characterizing local texture and motion state. It should be noted that the specific methods for determining the four corners, boundary sampling points, and depth abrupt change points of a frame image include: the four corners of a frame image usually refer to the four corner points of the image. If there is a depth map or stereo vision data, the depth information is used to project these corner points into three-dimensional space to form actual spatial coordinates. Boundary sampling points refer to feature points at the edges of an image. Edge detection algorithms, such as Canny edge detection, Sobel operator, and Laplace operator, are used to extract image edges. The algorithm finds edge points in the image and forms a boundary curve. A certain number of sampling points are evenly selected on the boundary curve to ensure that the entire boundary is covered. Depth abrupt change points refer to points in an image where the depth changes significantly. They usually represent the edges or structural changes of an object. If an RGB-D camera or laser rangefinder is used, the depth map can be directly obtained, or a depth map can be calculated using a dual-camera system. After obtaining the three-dimensional spatial coordinates of the four corners, boundary sampling points, and depth abrupt change points of the frame image, the positions of these points in space are combined to form a polygonal or polyhedral envelope that covers the entire field of view. The spatial grid of the pixels covered by the coarse field of view envelope under the spatiotemporal reference is used as the candidate coverage grid.

[0039] The artificial intelligence model specifically includes the YOLOv8 object detection model, the semantic segmentation model, and the Mask R-CNN instance segmentation. The semantic feature parameters specifically include: semantic anchors, image patches, mapping parameters from image patches to the panoramic canvas, semantic masks, occlusion masks and dynamic object masks, quality and credibility vectors, feature descriptors, and motion vector summaries. The semantic anchor points are not limited to human faces or moving objects, but rather prioritize structural elements with spatial stability. For urban road scenes, semantic anchor points can be lane line intersections, curb lines, zebra crossing boundaries, traffic light poles, traffic signs, and building corners. For park inspection scenes, semantic anchor points can be door signs, equipment edges, pipeline supports, wall corners, and ground markings. For natural areas, semantic anchor points can be shorelines, road edges, mountain outlines, and obvious feature boundaries.

[0040] The YOLOv8 object detection model is based on CSPDarknet as the backbone network, takes video frame images as input, and uses the region where the semantic anchor point in the semantic feature parameters is located as the label. Tools such as LabelImg and LabelMe are used to annotate the custom dataset and define the model configuration file, which includes network layer definitions, learning rate, optimizer, loss function, etc. Training parameters such as learning rate, batch size, and number of iterations are configured. Use an object detection dataset for training and optimize the loss function, which is usually a combination of bounding box regression loss and class classification loss. During training, data augmentation techniques, such as random cropping and color jittering, can be used to improve the model's generalization ability. Semantic segmentation models are used to assign each pixel in an image to a semantic category, such as roads, buildings, lane lines, etc. Mainstream semantic segmentation models include UNet, DeepLabv3+, etc. Choose a pre-trained semantic segmentation model architecture, such as DeepLabv3+ or UNet; use the cross-entropy loss function or Dice loss function for training; configure hyperparameters, such as learning rate, batch size, etc., and use an optimizer, such as SGD or Adam. MaskR-CNN is an object detection and instance segmentation model that adds a branch to Faster R-CNN specifically for generating pixel-level object masks. It uses ResNeXt as the backbone network for feature extraction and RPN (Region Proposal Network) to generate candidate regions. After ROI pooling, it performs branch decomposition: bounding box coordinate regression; classification prediction; and pixel-level segmentation mask prediction. The model is initialized with pre-trained weights from COCO or other general datasets, and the bounding box and segmentation mask for each object instance are labeled. Tools such as LabelMe and VGG Image Annotator can be used to generate COCO format annotation files.

[0041] Based on the candidate coverage grid, the semantic feature parameters are mapped to the corresponding candidate coverage grid to obtain observation patches of the valid frame image at each candidate coverage grid. Each observation patch includes the image patch corresponding to the candidate coverage grid, panoramic canvas mapping parameters, semantic mask, occlusion mask, dynamic target mask, quality and credibility information, as well as feature descriptors, semantic anchors, and motion vector summaries. Therefore, the specific representation of the observation patches of a valid frame image at each candidate coverage grid within any dynamic time slice is as follows: In the formula, This represents the observation patch of the i-th frame image within the k-th dynamic time slice to the g-th candidate overlay grid. This represents the image patch of the i-th frame within the k-th dynamic time slice, located in the g-th candidate overlay grid. Let be the mapping parameters of the panoramic canvas for the i-th frame image within the k-th dynamic time slice, corresponding to the g-th candidate overlay grid. This represents the set of masks for the i-th frame within the k-th dynamic time slice, located at the g-th candidate overlay grid. Specifically, it includes semantic masks, occlusion masks, and dynamic target masks. This represents the quality and confidence vector of the i-th frame within the k-th dynamic time slice at the g-th candidate coverage grid, specifically including image quality score, time synchronization score, historical stability score, and occlusion probability. This represents the set of feature descriptions for the i-th frame within the k-th dynamic time slice, specifically including feature descriptors, semantic anchors, and motion vector summaries, where g is the index of the candidate coverage grid within the frame image.

[0042] Step 4: Based on the candidate coverage grid and the semantic feature parameters under the corresponding grid, determine the observation patches of the effective frame image in each candidate coverage grid, calculate the confidence observation degree of each candidate coverage grid according to the observation patches, and filter out the overlapping grids in the candidate coverage grids based on the state of the observation patches. Perform concatenated registration of video source pose and image mapping parameters on the overlapping grids to obtain the optimized values ​​of position pose and mapping parameters of each overlapping grid.

[0043] Methods for calculating the confidence level of each candidate coverage grid based on observation patches include: Extract the quality and reliability information of any frame image within any dynamic time slice in the corresponding observation slice of any candidate coverage grid; The quality and reliability information includes image quality score, geometric registration score, semantic consistency score, time synchronization score, historical stability score, and occlusion probability; The image quality score, geometric registration score, semantic consistency score, temporal synchronization score, and historical stability score are weighted according to preset weights, and the weighted results are subtracted and corrected based on the occlusion probability to obtain the confidence level of the image frame in the candidate coverage grid; the confidence level in the candidate coverage grid is specifically expressed as: In the formula, This represents the confidence level of the i-th frame image within the k-th dynamic time slice at the g-th candidate overlay grid. The image quality score for the i-th frame within the k-th dynamic time slice at the g-th candidate overlay grid. The geometric registration score for the i-th frame within the k-th dynamic time slice at the g-th candidate overlay grid is calculated. The semantic consistency score of the i-th frame within the k-th dynamic time slice is given in the g-th candidate overlay grid. For the time synchronization score of the i-th frame within the k-th dynamic time slice at the g-th candidate overlay grid. The historical stability score of the i-th frame within the k-th dynamic time slice at the g-th candidate overlay grid. Let g be the occlusion probability of the i-th frame image in the k-th dynamic time slice in the g-th candidate coverage grid. , , , , and Here are the weighting coefficients, where , , , , and The sum of the two is 1. It should be noted that this formula is derived by multiplying each score by its corresponding weighting coefficient. The weighted summation reflects the relative importance of each score to the reliability of observations, while allowing the influence of each score to be adjusted according to the needs of different application scenarios. By subtracting the occlusion probability The weighted values ​​reasonably reflect the negative impact of occlusion on the credibility of observations. Occlusion usually significantly reduces the usability and information content of an image, so it needs to be deducted when calculating credibility. By combining scores from multiple dimensions such as image quality, geometric registration, semantic consistency, time synchronization, and historical stability, the formula can comprehensively evaluate the credibility of an image. Multi-dimensional evaluation helps to reduce the bias that a single indicator may bring and improve the accuracy of credibility.

[0044] The calculation method for the geometric registration score includes: Extract multiple semantic anchor points from candidate coverage grid observation patches, and map each semantic anchor point to the grid semantic reference coordinate system based on the mapping parameters corresponding to the observation patches; The positional deviations between the mapping results of each semantic anchor point and the corresponding semantic reference anchor point of the grid are determined, and combined with the allowable error corresponding to the grid level of the candidate coverage grid, a comprehensive evaluation of each positional deviation is performed to obtain the geometric registration score of the observation patch corresponding to the candidate coverage grid; the specific calculation formula of the geometric registration score is as follows: In the formula, Let j be the semantic anchor point of the g-th candidate coverage grid observation patch. Indicates the semantic reference anchor point of the grid. Let be the allowable error corresponding to the grid level of the g-th candidate coverage grid, and j be the index of the semantic anchor point within the observation patch. Let g be the total number of semantic anchor points within the observation patch of the g-th candidate coverage grid. Represents a mapping function; It should be noted that in the formula It measures the square of the Euclidean distance between the anchor point obtained through the mapping function and the reference anchor point; it reflects the geometric deviation between the semantic features of the current image and the reference features. The errors of all semantic anchors are averaged to obtain the overall registration quality; the comprehensive evaluation of multiple semantic anchors helps improve the stability of the score and avoids score distortion due to outliers of a single anchor; the entire error is normalized to... This ensures comparability of grids at different levels during scoring; tolerance error It is dynamically set based on the characteristics of the grid, making the scoring reasonable at different spatial resolutions; among which the grid semantic reference anchor points Specifically, the historical semantic anchor point is set according to the corresponding grid, that is, it is defined as the historical semantic anchor point of the previous video fusion moment; Using an exponential function allows the score to remain relatively high when the error is small, but decreases rapidly as the error increases; this non-linear characteristic makes the score more sensitive to deviations, which helps in fine-tuning during quality control; the allowable error... It is dynamically set according to different spatial grid levels. The larger the spatial grid level, the larger its allowable error. The smaller the value, the more adaptable the scoring mechanism can be to different resolutions and scales, ensuring the flexibility of the application. The method for calculating the semantic consistency score includes: The current semantic proportion of each semantic category within the observation patch corresponding to the candidate coverage grid is statistically analyzed, and the consistency of the current semantic proportion of each semantic category with the historical semantic proportion corresponding to the candidate coverage grid is compared. Based on the degree of overlap and difference between the proportions of each semantic category, the semantic consistency score of the observation patch corresponding to the candidate coverage grid is obtained; the formula for calculating the semantic consistency score is as follows: In the formula, For the i-th frame image within the k-th dynamic time slice, within the g-th candidate overlay grid observation slice, the i-th frame image... The semantic proportion of class semantics, For the g-th candidate coverage grid observation patch The historical semantic proportion of class semantics To prevent division by zero constant, For image semantic type index, This represents the total number of image semantic types.

[0045] It should be noted that, The use of this ensures that its contribution is included in the score only when both the current frame image and the historical proportion exist, thus avoiding the situation where the score is too high because a certain type of semantic has a very small proportion in a certain frame image. In the denominator This ensures that the denominator always includes the larger value of the current frame and the historical proportion, providing a normalized basis so that the score can reflect the relative relationship between the two; it also ensures that the score is not too low because a certain type of semantics did not appear in the history; and it adds... To prevent the denominator from being zero, ensure that the formula can be calculated stably in all cases, and avoid division by zero errors when the semantic content is extremely small, the formula comprehensively considers the influence of multiple semantic features by evaluating all semantic categories, ensuring that the score can fully reflect the similarity between the current frame image and historical semantics.

[0046] For any dynamic time slice, determine all candidate coverage grids for each valid frame image within the dynamic time slice. Based on the observation slices of the candidate coverage grids, filter the candidate coverage grids. If the observation slice of a candidate coverage grid is an empty set, then remove the candidate coverage grid. Traverse all candidate coverage grids within a valid frame image and mark the remaining candidate coverage grids as coverage grids. Within any dynamic time slice, the effective frame images corresponding to different types of video sources are determined by the intersection of the overlay grids to identify the overlapping grid set between the different types of video sources within that dynamic time slice. The specific logic underlying the cascaded registration of video source pose and image mapping parameters for overlapping meshes is as follows: the cascaded registration includes a coarse registration stage, a fine registration stage, and a global consistency optimization stage; The coarse registration stage outputs the pose correction amount of the corresponding frame image of the video source based on the visual localization model of each type of video source. The fine registration stage corrects the mapping parameters of the image to the panoramic canvas, and obtains the correction amount of the mapping parameters. The global consistency optimization stage corrects the video source pose and image mapping parameters between different overlapping grids; This allows us to obtain the optimized values ​​of the position, orientation, and mapping parameters of each overlapping mesh.

[0047] The formula used to calculate the pose correction in the coarse registration stage is as follows: In the formula, Let represent the pose correction vector of the i-th frame image. This represents the image of the i-th frame within the k-th dynamic time slice. For map / historical anchor point information within the h-th overlapping grid, It is the feature description set of the i-th frame image in the k-th dynamic time slice at the h-th overlapping grid, where h is the index of the overlapping grid; This represents a visual localization model, specifically constructed based on a convolutional neural network, whose input is... Its output is the pose correction vector of the i-th frame image. A visual localization model is established based on a convolutional neural network. This convolutional neural network consists of an input layer, convolutional layers, pooling layers, fully connected layers, and an output layer. The activation function in the convolutional layers is... function; For the fully connected layer, the number of neurons in the fully connected layer is set to 32, the initial neural network learning rate is set to 0.001, and the number of training rounds is 100.

[0048] The loss function is typically chosen as mean squared error loss. Backpropagation and optimization algorithms, such as Adam and SGD, are used to iteratively update the model parameters and gradually reduce the loss function value until the ideal performance is achieved. The formula used to determine the correction amount of the mapping parameters obtained in the fine registration stage is as follows: In the formula, The correction amount for the panoramic canvas mapping parameters of the i-th frame image in the h-th overlapping grid within the k-th dynamic time slice. Represents the set of matching point pairs within overlapping grids. Represents the robust loss function. The mapping parameters for the panoramic canvas of the i-th frame image in the h-th overlapping grid within the k-th dynamic time slice; This represents the correction amount of the panoramic canvas mapping parameters for the corresponding points of the frame image within the overlapping grid. For the matching points within the overlapping grid, where For a point in a frame image, This represents a point on the corresponding panoramic canvas.

[0049] It should be noted that the choice of robust loss function is crucial to the quality of registration. Common robust loss functions include Huber Loss, which uses squared loss for small residuals and linear loss for large errors. This loss function performs well in handling outliers. It uses squared loss within a certain range and switches to linear loss outside that range, thereby reducing the impact of outliers.

[0050] The matching point pairs within the overlapping grid specifically refer to the corresponding points on the corresponding panoramic canvas for each type of video source frame image. The matching points can be determined using descriptor matching algorithms such as FLANN and BFMatcher. Precise matching point pairs are the foundation for achieving high-quality image stitching and registration. By finding accurate matching points in the overlapping areas, the images are aligned to generate seamless panoramic images.

[0051] The specific formula used in the global consistency optimization phase is as follows: In the formula, Let be the edge weight of the observation patch between the u-th and v-th overlapping grids. This represents the geometric error between the u-th and v-th overlapping grids. This refers to the semantic boundary inconsistency error between the observation patches of the u-th and v-th overlapping grids. The temporal continuity error between the u-th and v-th overlapping grids is... The prior pose of the video source. For the source video pose covariance; The edge set of observation patches between overlapping grids; , and Here are the weighting coefficients; ,and ; It should be noted that the prior pose of the video source This refers to previously known pose information from the source video, used as a reference for optimization; by comparing it with the optimized pose... Comparisons can be made to reduce errors, specifically through prior pose estimation, SLAM (Simultaneous Localization and Mapping), or other sensor fusion techniques. Source video pose covariance This represents the covariance of the pose in the source video, used to quantify the uncertainty of the pose and ensure that the uncertainty of the pose is taken into account during the optimization process. The variance and covariance of the position at each time step are calculated by statistical methods, which can usually be obtained through an uncertainty model. The edge weights of the observation patches between overlapping grids Specifically, it is expressed as follows: In the formula, , , , and These represent the similarity scores for geometric registration, semantic consistency, temporal synchronization, and image quality between the observation patches of the u-th and v-th overlapping grids, respectively. , , and These are the weighting coefficients, where , , and All are greater than 0 and their sum is 1; Geometric registration score similarity, semantic consistency score similarity, temporal synchronization score similarity, and image quality score similarity are specifically characterized based on the Euclidean distance between each score of the observation patch between the u-th and v-th overlapping grids.

[0052] Step 5: Based on the position and pose of each overlapping grid, the optimized value of the mapping parameters, and the reliability of the observation, perform dynamic stitching of the effective frame images of each type of video source within the dynamic time slice to obtain a single panoramic image within the dynamic time slice.

[0053] Dynamic stitching of valid frame images from various video sources within a dynamic time slice is performed. Specifically, this involves stitching the overlapping grids of each valid frame image. This includes outputting the color of the overlapping grid based on image blocks, optimized mapping parameters, normalized fusion weights, and a mask set. Image stitching is then performed based on the color of this overlapping grid, meaning the color of the overlapping grid is used as the color of the corresponding overlapping grid in the stitched single panoramic image. The remaining areas in the stitched single panoramic image are consistent with the corresponding valid frame images. The specific formula used to calculate the color of the overlapping grid is as follows: In the formula, Let y be the color corresponding to the panoramic canvas pixel in the k-th dynamic time slice. Represents the fusion function. Let be the normalized fusion weight of the i-th frame image in the k-th dynamic time slice at the h-th overlapping grid. For the image of the i-th frame in the k-th dynamic time slice, the image patch in the h-th overlapping grid is... This represents the mapping parameter of the panoramic canvas pixel y corresponding to the i-th frame of the k-th dynamic time slice in the h-th overlapping grid. The mask set for the i-th frame of the k-th dynamic time slice at the h-th overlapping grid; This represents the number of video source types contained in the h-th overlapping grid of the k-th dynamic time slice; The normalized fusion weights are specifically set based on the confidence level of the overlapping grid. The normalized fusion weights are proportional to the confidence level. The confidence levels of all image blocks are normalized to ensure that the weight of each image block is reasonable relative to the weights of other image blocks. The normalized fusion weights are obtained by dividing the confidence level of each image block by the sum of the confidence levels of all image blocks.

[0054] Please see Figure 4 The present invention further provides a multi-source video dynamic stitching system based on artificial intelligence and spatiotemporal grids. This multi-source video dynamic stitching system is used to execute the aforementioned multi-source video dynamic stitching method based on artificial intelligence and spatiotemporal grids, comprising: The multi-source spatiotemporal reference module is used to acquire multiple types of video sources to be fused. For each type of video source, a unified spatiotemporal reference is established. The spatiotemporal reference specifically includes a spatial grid reference and a temporal grid reference. Under the spatiotemporal reference, based on the shooting carrier speed of each type of video source, several dynamic time slices for video source time alignment are defined. The dynamic time frame filtering module is used to allocate each frame of a video source to a dynamic time slice in chronological order for any type of video source. It sets a time matching cost function based on the time difference between each frame and the center of the dynamic time slice. By minimizing the value of the time matching cost function, the effective frame images in each dynamic time slice are determined. The candidate grid feature module is used to perform region planning for any type of video source and any valid frame image to form a coarse field of view envelope. The spatial grid of the pixels covered by the coarse field of view envelope under the spatiotemporal reference is used as a candidate coverage grid. Based on the candidate coverage grid and the valid frame image, it is input into the trained artificial intelligence model to obtain the semantic feature parameters of the valid frame image. The overlapping grid registration module is used to determine the observation patches of the effective frame image in each candidate coverage grid based on the semantic feature parameters of the candidate coverage grid and the corresponding grid. It calculates the confidence observation degree of each candidate coverage grid based on the observation patches, and selects overlapping grids in the candidate coverage grid based on the state of the observation patches. It performs cascade registration of video source pose and image mapping parameters on the overlapping grids to obtain the position, pose and optimized mapping parameter values ​​of each overlapping grid. The dynamic panoramic stitching module is used to dynamically stitch together effective frame images of various types of video sources within a dynamic time slice based on the position and pose of each overlapping grid, the optimized value of the mapping parameters, and the reliability of the observations, so as to obtain a single panoramic image within the dynamic time slice.

[0055] Please see Figure 5 The present invention also provides a multi-source video dynamic stitching device based on artificial intelligence and spatiotemporal grids, including a processor and a memory communicatively connected to the processor. The memory stores a computer program that can be executed by the processor. The computer program is executed by the processor to enable the processor to implement the steps of the above-described multi-source video dynamic stitching method based on artificial intelligence and spatiotemporal grids.

[0056] The present invention further provides a computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps of the above-described method for dynamic stitching of multi-source videos based on artificial intelligence and spatiotemporal grids.

[0057] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0058] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0059] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0060] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for dynamic stitching of multi-source videos based on artificial intelligence and spatiotemporal grids, characterized in that, The specific steps include: Acquire multiple types of video sources to be merged, and establish a unified spatiotemporal reference for each type of video source. The spatiotemporal reference specifically includes a spatial grid reference and a temporal grid reference. Under the spatiotemporal reference, based on the shooting carrier speed of each type of video source, define several dynamic time slices for video source time alignment. For any type of video source, each frame of the video source is allocated to each dynamic time slice in chronological order, and a time matching cost function is set according to the time difference between each frame and the center of the dynamic time slice. By minimizing the value of the time matching cost function, the effective frame images in each dynamic time slice are determined. For any type of video source, region planning is performed on any valid frame image to form a coarse field of view envelope. The spatial grid of the pixels covered by the coarse field of view envelope under the spatiotemporal reference is used as a candidate coverage grid. Based on the candidate coverage grid and the valid frame image, it is input into the trained artificial intelligence model to obtain the semantic feature parameters of the valid frame image. Based on the candidate coverage grid and the semantic feature parameters under the corresponding grid, the observation patches of the effective frame image in each candidate coverage grid are determined. The confidence observation degree of each candidate coverage grid is calculated according to the observation patch. Based on the state of the observation patch, overlapping grids are selected in the candidate coverage grid. The video source pose and image mapping parameters are concatenated and registered for the overlapping grids to obtain the position, pose and mapping parameter optimization values ​​of each overlapping grid. Based on the position and pose of each overlapping grid, the optimized values ​​of the mapping parameters, and the reliability of the observations, the effective frame images of various types of video sources within the dynamic time slice are dynamically stitched together to obtain a single panoramic image within the dynamic time slice.

2. The method for dynamic stitching of multi-source videos based on artificial intelligence and spatiotemporal grids according to claim 1, characterized in that, The various video sources include fixed surveillance cameras, PTZ cameras, drone videos, robot dog videos, vehicle-mounted videos, mobile terminal videos, and temporary deployment videos. The establishment of a unified spatiotemporal benchmark includes: A multi-level spatial grid benchmark is constructed based on a unified geographic coordinate system to uniformly locate and grid-represent the spatial position, field of view coverage, and candidate coverage areas of different types of video sources. A multi-level time grid reference is constructed based on a unified standard time to uniformly calibrate the acquisition time, time alignment process, and dynamic time slices of different types of video sources; The spatial grid reference has different spatial resolutions at each level, and the temporal grid reference has different temporal resolutions at each level, so as to support frame-level allocation, grid coverage analysis and dynamic stitching of multiple types of video sources under the same spatiotemporal reference system.

3. The multi-source video dynamic stitching method based on artificial intelligence and spatiotemporal grids according to claim 2, characterized in that, The method for constructing the dynamic time slice includes: Based on the start time of the video source, the video timeline is initially divided according to the preset basic time slice length; For each initial time slice, a time tolerance window is set according to the movement speed of the corresponding shooting vehicle, and the time boundary of the initial time slice is adjusted using the time tolerance window to obtain several dynamic time slices. The time tolerance window is inversely proportional to the speed of the imaging carrier.

4. The method for dynamic stitching of multi-source videos based on artificial intelligence and spatiotemporal grids according to claim 1, characterized in that, Methods for setting the time matching cost function include: For any type of video source, determine all frame images of that type of video source and extract the timestamp of each frame image; Based on the spatial coordinates and attitude matrix of the video source of this type under the timestamp of each frame image, the time deviation, spatial position deviation and attitude deviation of each frame image relative to the center time of each dynamic time slice are determined respectively. Based on the time deviation, spatial position deviation, and attitude deviation, and combined with preset weights, a time matching cost function is constructed for allocating each frame image to each dynamic time slice.

5. The multi-source video dynamic stitching method based on artificial intelligence and spatiotemporal grids according to claim 4, characterized in that, The method for determining the effective frame images within each dynamic time slice is as follows: For any type of video source, the allocation scheme of each frame image in the dynamic time slice is taken as the optimization object. The time matching cost function value under different frame image allocation schemes is analyzed. The allocation scheme corresponding to the minimum time matching cost function value is taken as the optimal allocation scheme. The frame images of each type of video source are allocated to the corresponding dynamic time slice according to the optimal allocation scheme. Then, the frame images contained in each dynamic time slice are the effective frame images.

6. The multi-source video dynamic stitching method based on artificial intelligence and spatiotemporal grids according to claim 4, characterized in that, Methods for forming a coarse field of view envelope include: For any type of video source, extract the four corners, boundary sampling points and depth change points of its effective frame image, and project the four corners, boundary sampling points and depth change points onto the spatial grid reference to form the coarse field of view envelope of the effective frame image; The spatial grid covered by the coarse field of view envelope is determined as a candidate coverage grid; The effective frame image is input into the artificial intelligence model to extract the semantic feature parameters of the effective frame image. The semantic feature parameters include: semantic anchor points for characterizing the semantic positional relationship of the scene, image patches for characterizing the image content of the candidate coverage area, mapping parameters for mapping the image patches to the panoramic canvas, mask information for characterizing semantic category and occlusion state, quality and credibility information for characterizing image quality and credibility, and feature description information for characterizing local texture and motion state. Based on the candidate coverage grid, the semantic feature parameters are mapped to the corresponding candidate coverage grid to obtain observation patches of the effective frame image at each candidate coverage grid. The observation patch includes the image patch corresponding to the candidate coverage grid, panoramic canvas mapping parameters, semantic mask, occlusion mask, dynamic target mask, quality and credibility information, as well as feature descriptors, semantic anchors and motion vector summaries.

7. The multi-source video dynamic stitching method based on artificial intelligence and spatiotemporal grids according to claim 6, characterized in that: Methods for calculating the confidence level of each candidate coverage grid based on observation patches include: Extract the quality and reliability information of any frame image within any dynamic time slice in the corresponding observation slice of any candidate coverage grid; The quality and reliability information includes image quality score, geometric registration score, semantic consistency score, time synchronization score, historical stability score, and occlusion probability; The image quality score, geometric registration score, semantic consistency score, time synchronization score, and historical stability score are weighted according to preset weights, and the weighted results are subtracted and corrected based on the occlusion probability to obtain the confidence level of the image in the candidate coverage grid.

8. The multi-source video dynamic stitching method based on artificial intelligence and spatiotemporal grids according to claim 7, characterized in that, The calculation method for the geometric registration score includes: Extract multiple semantic anchor points from candidate coverage grid observation patches, and map each semantic anchor point to the grid semantic reference coordinate system based on the mapping parameters corresponding to the observation patches; The positional deviation between each semantic anchor point mapping result and the corresponding grid semantic reference anchor point is determined, and combined with the allowable error corresponding to the grid level where the candidate coverage grid is located, the positional deviation is comprehensively evaluated to obtain the geometric registration score of the observation patch corresponding to the candidate coverage grid. The method for calculating the semantic consistency score includes: The current semantic proportion of each semantic category within the observation patch corresponding to the candidate coverage grid is statistically analyzed, and the consistency of the current semantic proportion of each semantic category with the historical semantic proportion corresponding to the candidate coverage grid is compared. Based on the degree of overlap and difference between the proportions of each semantic category, the semantic consistency score of the observation patch corresponding to the candidate coverage grid is obtained.

9. The multi-source video dynamic stitching method based on artificial intelligence and spatiotemporal grids according to claim 8, characterized in that, For any dynamic time slice, determine all candidate coverage grids for each valid frame image within the dynamic time slice. Based on the observation slices of the candidate coverage grids, filter the candidate coverage grids. If the observation slice of a candidate coverage grid is an empty set, then remove the candidate coverage grid. Traverse all candidate coverage grids within a valid frame image and mark the remaining candidate coverage grids as coverage grids. Within any dynamic time slice, the effective frame images corresponding to different types of video sources are determined by the intersection of the overlay grids to identify the overlapping grid set between the different types of video sources within that dynamic time slice. The specific logic underlying the cascaded registration of video source pose and image mapping parameters for overlapping meshes is as follows: the cascaded registration includes a coarse registration stage, a fine registration stage, and a global consistency optimization stage; The coarse registration stage outputs the pose correction amount of the video source based on the visual localization model of each type of video source. The fine registration stage corrects the mapping parameters of the image to the panoramic canvas, and obtains the correction amount of the mapping parameters. The global consistency optimization stage corrects the video source pose and image mapping parameters between different overlapping grids; This allows us to obtain the optimized values ​​of the position, orientation, and mapping parameters of each overlapping mesh.

10. A multi-source video dynamic stitching method based on artificial intelligence and spatiotemporal grids according to claim 9, characterized in that, Dynamic stitching of valid frame images from various types of video sources within a dynamic time slice is performed. Specifically, the overlapping grids of each valid frame image are stitched together. This includes outputting the color of the overlapping grid based on the image blocks of the overlapping grid, the optimized values ​​of the mapping parameters, the normalized fusion weights, and the mask set. The image is stitched together based on the color of the overlapping grid, that is, the color of the overlapping grid is used as the color of the corresponding overlapping grid in the stitched single panoramic image. The remaining area in the stitched single panoramic image is consistent with the corresponding valid frame image. The normalized fusion weights are specifically set based on the reliability of the overlapping grids, and the normalized fusion weights are proportional to the reliability of the observations.

11. A multi-source video dynamic stitching system based on artificial intelligence and spatiotemporal grids, characterized in that, The multi-source video dynamic stitching system is used to execute the multi-source video dynamic stitching method based on artificial intelligence and spatiotemporal grids according to any one of claims 1 to 10, including: The multi-source spatiotemporal reference module is used to acquire multiple types of video sources to be fused. For each type of video source, a unified spatiotemporal reference is established. The spatiotemporal reference specifically includes a spatial grid reference and a temporal grid reference. Under the spatiotemporal reference, based on the shooting carrier speed of each type of video source, several dynamic time slices for video source time alignment are defined. The dynamic time frame filtering module is used to allocate each frame of a video source to a dynamic time slice in chronological order for any type of video source. It sets a time matching cost function based on the time difference between each frame and the center of the dynamic time slice. By minimizing the value of the time matching cost function, the effective frame images in each dynamic time slice are determined. The candidate grid feature module is used to perform region planning for any type of video source and any valid frame image to form a coarse field of view envelope. The spatial grid of the pixels covered by the coarse field of view envelope under the spatiotemporal reference is used as a candidate coverage grid. Based on the candidate coverage grid and the valid frame image, it is input into the trained artificial intelligence model to obtain the semantic feature parameters of the valid frame image. The overlapping grid registration module is used to determine the observation patches of the effective frame image in each candidate coverage grid based on the semantic feature parameters of the candidate coverage grid and the corresponding grid. It calculates the confidence observation degree of each candidate coverage grid based on the observation patches, and selects overlapping grids in the candidate coverage grid based on the state of the observation patches. It performs cascade registration of video source pose and image mapping parameters on the overlapping grids to obtain the position, pose and optimized mapping parameter values ​​of each overlapping grid. The dynamic panoramic stitching module is used to dynamically stitch together effective frame images of various types of video sources within a dynamic time slice based on the position and pose of each overlapping grid, the optimized value of the mapping parameters, and the reliability of the observations, so as to obtain a single panoramic image within the dynamic time slice.

12. A multi-source video dynamic stitching device based on artificial intelligence and spatiotemporal grids, characterized in that, The method includes a processor and a memory communicatively connected to the processor, the memory storing a computer program executable by the processor, the computer program being executed by the processor to enable the processor to implement the steps of the multi-source video dynamic stitching method based on artificial intelligence and spatiotemporal grids according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps of the multi-source video dynamic stitching method based on artificial intelligence and spatiotemporal grids according to any one of claims 1 to 10.

Images