Data coupling system and method for satellite image data and video information

Abstract

The invention relates to the field of image data processing, and discloses a satellite image data and video information data coupling system and method, and the system comprises a data calibration module which is used for carrying out the space-time calibration processing of satellite image data, and obtaining a reference remote sensing image; the anchor point frame modeling module is used for carrying out background modeling processing on the video anchor point frame to obtain a monitoring background image; the ground feature fusion rendering module is used for identifying landmark ground features in the reference remote sensing image and performing feature fusion rendering on the landmark ground features to obtain a rendered remote sensing image; the lacking object compensation module is used for determining a lacking object in the reference remote sensing image and performing compensation rendering on the reference remote sensing image to obtain a compensation remote sensing image; and the image coupling generation module is used for carrying out coupling processing on the rendered remote sensing image and the compensated remote sensing image to obtain an accurate monitoring graph of the target monitoring area. According to the invention, the coupling accuracy of the satellite image data and video information data coupling system can be improved.

Description

Technical Field

[0001] This invention relates to a data coupling system and method for satellite image data and video information, belonging to the field of image data processing. Background Technology

[0002] With the deepening of smart city construction, accurate and real-time monitoring of urban elements has become the key to improving urban governance. Satellite remote sensing technology, with its advantages of wide coverage and periodic observation, has played an important role in the field of macro-monitoring, providing valuable data support for urban master planning, environmental monitoring, disaster assessment and other purposes.

[0003] However, satellite imagery data also has limitations. For example, under specific weather conditions (such as cloud cover) or sensor resolution limitations, some details of ground features in the images are blurred, small targets are difficult to identify, and there are even areas with missing information. This results in insufficient ability to depict the micro-dynamic changes of the city, making it difficult to meet the urgent needs of smart cities for high-precision, fine-grained situational awareness. Currently, the processing and application of satellite imagery data are mostly focused on the analysis of single data sources. Conventional technical solutions usually directly identify and classify targets from satellite images or use multiple images for change detection. Although these methods can reflect the surface conditions to a certain extent, they fail to effectively integrate other near-ground, high-time-sensitivity sensing data, which leads to deviations between the processed satellite data and the actual ground conditions. Summary of the Invention

[0004] This invention provides a data coupling system and method for satellite image data and video information, the main purpose of which is to improve the coupling accuracy of the data coupling system for satellite image data and video information.

[0005] To achieve the above objectives, the present invention provides a data coupling system for satellite imagery data and video information, comprising:

[0006] The module includes a data calibration module, an anchor frame modeling module, a ground feature fusion rendering module, a missing object compensation module, and an image coupling generation module.

[0007] The data calibration module is used to acquire satellite image data and ground video data of the target monitoring area, perform spatiotemporal calibration processing on the satellite image data, and obtain a reference remote sensing image.

[0008] Anchor frame modeling module is used to extract video anchor frames from the ground video data, perform background modeling processing on the video anchor frames, and obtain a monitoring background image;

[0009] The feature fusion rendering module is used to identify landmark features in the reference remote sensing image, and perform feature fusion rendering on the landmark features based on the monitoring background image to obtain a rendered remote sensing image;

[0010] The missing object compensation module is used to determine the missing objects in the reference remote sensing image based on the monitoring background image, and to perform compensation rendering on the reference remote sensing image based on the missing objects to obtain a compensated remote sensing image.

[0011] The image coupling generation module is used to couple the rendered remote sensing image and the compensated remote sensing image to obtain a precise monitoring map of the target monitoring area.

[0012] Optionally, the step of performing spatiotemporal calibration processing on the satellite image data to obtain a reference remote sensing image includes:

[0013] The satellite imagery data is subjected to spatiotemporal frame parsing to obtain a standard spatiotemporal frame;

[0014] The standard spatiotemporal frame is subjected to control point registration processing to obtain an accurate spatial grid;

[0015] The precise spatial grid is subjected to error correction processing to obtain an optimized coordinate system;

[0016] The optimized coordinate system is subjected to image resampling processing to obtain a reference remote sensing image.

[0017] Optionally, the step of performing control point registration on the standard spatiotemporal frame to obtain an accurate spatial grid includes:

[0018] Extract the set of high-quality control points from the standard spatiotemporal framework;

[0019] Analyze the spatial distribution characteristics of the high-quality control point set, and based on the spatial distribution characteristics, determine the regional representative point set of the high-quality control point set;

[0020] Identify the terrain fit degree corresponding to the representative point set of the region, and filter the terrain matching point set in the representative point set of the region based on the terrain fit degree;

[0021] Calculate the registration contribution degree corresponding to the terrain matching point set, and determine the core registration point set in the terrain matching point set based on the registration contribution degree;

[0022] Based on the core registration point set, the standard spatiotemporal frame is subjected to control point registration processing to obtain an accurate spatial grid.

[0023] Optionally, calculating the registration contribution corresponding to the terrain matching point set includes:

[0024] Extract the point accuracy features corresponding to the terrain matching point set, and determine the basic contribution value corresponding to the terrain matching point set based on the point accuracy features.

[0025] Extract the spatial distribution attributes of the terrain matching point set, analyze the coverage of the spatial distribution attributes in the registration area, and calculate the area coverage ratio corresponding to the terrain matching point set;

[0026] Based on the spatial distribution attributes, assign distribution weights to the terrain matching point set;

[0027] The current registration requirements of the standard spatiotemporal framework are detected, and based on the registration requirements, the accuracy adjustment tolerance and allowable error limit of the terrain matching point set are calculated.

[0028] The registration contribution of the terrain matching point set is calculated by combining the basic contribution value, the distribution weight, the accuracy adjustment tolerance, the regional coverage ratio, and the allowable error limit.

[0029] Optionally, extracting video anchor frames from the ground video data includes:

[0030] The ground video data is processed into frames to obtain ground video frames;

[0031] Calculate the scene change degree corresponding to the ground video frame, and extract candidate video frames corresponding to the ground video frame based on the scene change degree;

[0032] Extract video frame features from the candidate video frames and calculate the feature entropy corresponding to the video frame features;

[0033] Based on the feature entropy, characteristic frame features are selected from the candidate video frames;

[0034] The anchor frame is identified by performing anchor frame recognition processing on the features of the characterization frame to obtain the video anchor frame.

[0035] Optionally, the step of performing background modeling processing on the video anchor frame to obtain a monitoring background image includes:

[0036] Extract the inter-frame motion features from the video anchor frames and analyze the scene stability corresponding to the inter-frame motion features;

[0037] Based on the scene stability, determine the stable anchor frames in the video anchor frames;

[0038] The stable anchor frame is subjected to background construction processing to obtain an initial background base map;

[0039] The initial background image is subjected to illumination equalization processing to obtain an optimized background image;

[0040] The optimized background image is then sharpened to obtain the monitoring background image.

[0041] Optionally, the step of performing feature fusion rendering on the landmark features based on the monitored background image to obtain a rendered remote sensing image includes:

[0042] Texture features are extracted from the monitored background image to obtain detailed texture features of ground features;

[0043] The outline features of the landmark features are extracted to obtain the feature outline features;

[0044] The detailed texture features and the outline features of the ground features are subjected to coordinate registration processing to obtain a registered feature pair;

[0045] The registered feature pairs are then pixel-synthesized to obtain an initial rendered image;

[0046] The initial rendered image is subjected to color correction processing to obtain a rendered remote sensing image.

[0047] Optionally, the step of pixel synthesis of the registration feature pairs to obtain an initial rendered image includes:

[0048] The registered feature pairs are subjected to feature separation processing to obtain texture feature maps and contour feature maps;

[0049] The texture feature map is subjected to gradient fusion processing to obtain a fused texture layer;

[0050] The contour feature map is subjected to edge smoothing processing to obtain a smooth contour layer;

[0051] The fused texture layer and the smooth contour layer are processed to generate an initial rendered image.

[0052] Optionally, the step of performing compensatory rendering on the baseline remote sensing image based on the missing object to obtain a compensated remote sensing image includes:

[0053] Based on the lack of objects, an object mask generation process is performed on the reference remote sensing image to obtain the object mask region.

[0054] Contextual feature extraction is performed on the object mask region to obtain the surrounding feature vector;

[0055] Analyze the object semantics corresponding to the missing object, and generate compensation content corresponding to the reference remote sensing image based on the object semantics and the surrounding feature vectors;

[0056] Based on the compensation content, the baseline remote sensing image is subjected to compensation rendering to obtain a compensated remote sensing image.

[0057] To address the aforementioned problems, the present invention also provides a data coupling method for satellite imagery data and video information, the method comprising:

[0058] Acquire satellite imagery data and ground video data of the target monitoring area, and perform spatiotemporal calibration processing on the satellite imagery data to obtain a reference remote sensing image;

[0059] Extract video anchor frames from the ground video data, perform background modeling processing on the video anchor frames, and obtain a monitoring background image;

[0060] Identify landmark features in the baseline remote sensing image, and perform feature fusion rendering on the landmark features based on the monitoring background image to obtain a rendered remote sensing image;

[0061] Based on the monitoring background image, the missing objects in the reference remote sensing image are determined, and the reference remote sensing image is compensated and rendered based on the missing objects to obtain a compensated remote sensing image.

[0062] The rendered remote sensing image and the compensated remote sensing image are coupled to obtain a precise monitoring map of the target monitoring area.

[0063] Compared to the problems described in the background art, this invention, by performing spatiotemporal calibration processing on the satellite imagery data, can unify the temporal and spatial reference benchmarks of multi-source data, eliminating data inconsistencies caused by differences in sensors and observation conditions, and providing accurate data support for subsequent processing. Furthermore, by extracting video anchor frames from the ground video data and performing background modeling processing on these video anchor frames, this invention obtains a monitoring background image, enabling the capture of key scene information from continuous video streams, eliminating interference factors in the video, and thus generating a stable background reference image. Furthermore, by identifying landmark features in the benchmark remote sensing image, this invention can identify salient buildings in the benchmark remote sensing image, based on the monitoring... The background image is used to perform feature fusion rendering on the landmark features to obtain a rendered remote sensing image, which provides a high-quality image for subsequent coupling processing. Furthermore, based on the monitoring background image, the present invention identifies missing objects in the reference remote sensing image, locating information-deficient areas in the image. Based on these missing objects, the reference remote sensing image is compensated for with rendering, effectively filling in missing feature information and improving image completeness. This provides accurate data for subsequent image coupling processing. Finally, the present invention couples the rendered remote sensing image and the compensated remote sensing image to obtain a precise monitoring map of the target monitoring area, resulting in a high-precision and high-completeness monitoring image of the target monitoring area. Therefore, the data coupling system and method for satellite image data and video information provided by this invention can improve the coupling accuracy of data coupling systems for satellite image data and video information. Attached Figure Description

[0064] Figure 1 A functional block diagram of a data coupling system for satellite imagery data and video information provided in an embodiment of the present invention;

[0065] Figure 2 This is a flowchart illustrating a data coupling method for satellite imagery data and video information according to an embodiment of the present invention.

[0066] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0069] In practice, the server-side equipment deployed in the satellite imagery data and video information data coupling system may consist of one or more devices. This system can be implemented as a business instance, a virtual machine, or hardware devices. For example, it can be implemented as a business instance deployed on one or more devices in a cloud node. Simply put, this system can be understood as software deployed on a cloud node to provide portable lithotripsy services to various users. Alternatively, it can be implemented as a virtual machine deployed on one or more devices in a cloud node, with application software installed to manage various users. Or, it can also be implemented as a server composed of numerous identical or different types of hardware devices, with one or more devices configured to provide portable lithotripsy services to various users.

[0070] In terms of implementation, the data coupling system for satellite imagery data and video information and the user terminal are mutually compatible. That is, if the data coupling system for satellite imagery data and video information is implemented as an application installed on a cloud service platform, then the user terminal is implemented as a client that establishes a communication connection with the application; or if the data coupling system for satellite imagery data and video information is implemented as a website, then the user terminal is implemented as a webpage; or if the data coupling system for satellite imagery data and video information is implemented as a cloud service platform, then the user terminal is implemented as a mini-program in an instant messaging application.

[0071] Reference Figure 1 The diagram shown is a functional block diagram of a data coupling system for satellite image data and video information provided in an embodiment of the present invention.

[0072] The data coupling system 100 for satellite imagery data and video information described in this invention can be located on a cloud server. In terms of implementation, it can function as one or more service devices, or as an application installed in the cloud (e.g., a portable rock-sorting server, a server cluster, etc.), or it can be developed into a website. Depending on the functions implemented, the data coupling system 100 for satellite imagery data and video information includes a data calibration module 101, an anchor frame modeling module 102, a ground feature fusion rendering module 103, a missing object compensation module 104, and an image coupling generation module 105.

[0073] The data calibration module 101 is used to acquire satellite image data and ground video data of the target monitoring area, perform spatiotemporal calibration processing on the satellite image data, and obtain a reference remote sensing image.

[0074] This invention performs spatiotemporal calibration on the satellite image data, which can unify the reference benchmarks of multi-source data in time and space, eliminate data inconsistencies caused by differences in sensors and observation conditions, and provide accurate data support for subsequent processing.

[0075] The target monitoring area is an area subject to refined monitoring, such as ecologically sensitive protection zones, key urban control areas, major engineering construction areas, and natural disaster hazard monitoring areas (such as river basins and landslide early warning areas), which are areas undertaking specific monitoring tasks. The satellite image data refers to image data containing multispectral features and spatial geometric information acquired by remote sensing satellites. The ground video data refers to continuous dynamic video stream data collected by fixed or mobile ground monitoring equipment. The reference remote sensing image refers to satellite imagery with a standard spatiotemporal reference system after complete calibration. Furthermore, it can be obtained by connecting to the interface of a commercial remote sensing satellite data service platform and the corresponding video monitoring system or mobile acquisition equipment, thereby obtaining satellite image data and ground video data of the target monitoring area.

[0076] As an embodiment of the present invention, the step of performing spatiotemporal calibration processing on the satellite image data to obtain a reference remote sensing image includes:

[0077] The satellite imagery data is subjected to spatiotemporal frame parsing to obtain a standard spatiotemporal frame;

[0078] The standard spatiotemporal frame is subjected to control point registration processing to obtain an accurate spatial grid;

[0079] The precise spatial grid is subjected to error correction processing to obtain an optimized coordinate system;

[0080] The optimized coordinate system is subjected to image resampling processing to obtain a reference remote sensing image.

[0081] The standard spatiotemporal frame is a coordinate system with a unified spatiotemporal reference established through frame analysis; the precise spatial grid refers to a spatial reference grid with sub-pixel accuracy established after control point registration; and the optimized coordinate system refers to a spatial reference system that meets the accuracy requirements after error correction.

[0082] Furthermore, the spatiotemporal frame parsing process can be performed using metadata parsing algorithms. For example, the GeoTIFF tag information of the image can be read using the GDAL library, and projection parameters and acquisition time can be automatically extracted to generate a standard spatiotemporal frame. The control point registration process can be performed using feature point matching technology. For example, the SIFT algorithm can be used to automatically identify corresponding feature points in the image and reference data, and the transformation parameters can be calculated through least squares adjustment to establish an accurate spatial grid. The error correction process can be performed using error model compensation methods. For example, a comprehensive correction model for sensor attitude error, orbital error, and atmospheric refraction error can be established to systematically optimize the spatial grid. The image resampling process can be performed using bilinear interpolation or cubic convolution interpolation algorithms. For example, each pixel can be repositioned based on the optimized coordinate system to generate a reference remote sensing image with a unified spatiotemporal reference.

[0083] Furthermore, as an optional embodiment of the present invention, the step of performing control point registration processing on the standard spatiotemporal frame to obtain an accurate spatial grid includes:

[0084] Extract the set of high-quality control points from the standard spatiotemporal framework;

[0085] Analyze the spatial distribution characteristics of the high-quality control point set, and based on the spatial distribution characteristics, determine the regional representative point set of the high-quality control point set;

[0086] Identify the terrain fit degree corresponding to the representative point set of the region, and filter the terrain matching point set in the representative point set of the region based on the terrain fit degree;

[0087] Calculate the registration contribution degree corresponding to the terrain matching point set, and determine the core registration point set in the terrain matching point set based on the registration contribution degree;

[0088] Based on the core registration point set, the standard spatiotemporal frame is subjected to control point registration processing to obtain an accurate spatial grid.

[0089] The high-quality control point set is a collection of control points with high-precision coordinates and stable characteristics selected from a standard spatiotemporal framework; the spatial distribution characteristics are the spatial arrangement patterns of the high-quality control point set within the target area, including features such as distribution density, coverage uniformity, and geometric configuration; the regional representative point set is a subset of control points that can fully represent the characteristics of different geographical zones; the terrain fit is the degree of matching between the control points and the terrain features of the area, including indicators such as elevation consistency and slope adaptability; the terrain matching point set is a subset of control points that highly matches the actual terrain features; the registration contribution is a quantitative indicator that measures the improvement of overall registration accuracy by a single control point; and the core registration point set is the optimal combination of control points whose registration contribution reaches a preset threshold and can guarantee regional registration accuracy.

[0090] Furthermore, high-quality control point sets in the standard spatiotemporal framework can be extracted using spatial data quality assessment methods. For example, a combination of accuracy verification, timeliness analysis, and source reliability assessment can be used to screen high-quality control points from the original control point data with horizontal accuracy better than 1 meter and vertical accuracy better than 2 meters. Spatial statistical analysis techniques can be used to analyze the spatial distribution characteristics of the high-quality control point sets. For example, Thiessen polygon analysis can be used to analyze the control point distribution density, Moran's index can be used to assess spatial autocorrelation, and based on the analysis results, the target area can be divided into several registration sub-regions. Representative control points can be selected from each sub-region to form a regional representative point set. The terrain fit degree corresponding to the regional representative point set can be identified through digital elevation model overlay analysis. For example, parameters such as elevation differences and slope changes between control points and surrounding terrain can be calculated, a terrain matching degree evaluation function can be established, and a terrain matching point set can be screened. Based on the core registration point set, a multinomial transformation model or a triangular mesh distortion model can be used to perform control point registration processing on the standard spatiotemporal framework. The transformation parameters can be calculated through least squares adjustment to establish a spatial grid with sub-pixel accuracy.

[0091] Furthermore, as an optional embodiment of the present invention, calculating the registration contribution corresponding to the terrain matching point set includes:

[0092] Extract the point accuracy features corresponding to the terrain matching point set, and determine the basic contribution value corresponding to the terrain matching point set based on the point accuracy features.

[0093] Extract the spatial distribution attributes of the terrain matching point set, analyze the coverage of the spatial distribution attributes in the registration area, and calculate the area coverage ratio corresponding to the terrain matching point set;

[0094] Based on the spatial distribution attributes, assign distribution weights to the terrain matching point set;

[0095] The current registration requirements of the standard spatiotemporal framework are detected, and based on the registration requirements, the accuracy adjustment tolerance and allowable error limit of the terrain matching point set are calculated.

[0096] The registration contribution of the terrain matching point set is calculated by combining the basic contribution value, the distribution weight, the accuracy adjustment tolerance, the regional coverage ratio, and the allowable error limit.

[0097] The point accuracy characteristics refer to the accuracy indicators of each control point in the terrain matching point set, including quantitative data such as planar accuracy and elevation accuracy. The basic contribution value is the initial contribution score calculated based on the point accuracy characteristics; the higher the value, the greater the potential contribution of point accuracy to registration. The spatial distribution attributes refer to the distribution characteristics of control points in the registration area, such as distribution uniformity, density, and geometric configuration. The area coverage ratio is the ratio of the area of ​​the registration area effectively covered by the control points to the total area, reflecting the spatial representativeness of the control points. The distribution weight is the weight value assigned based on the spatial distribution attributes; the higher the value, the stronger the importance of the distribution to registration. The registration requirements refer to the specific accuracy requirements of the current registration task, such as sub-pixel accuracy or meter-level accuracy. The accuracy adjustment tolerance is the remaining capacity of the current registration system in terms of accuracy adjustment; the higher the value, the greater the adjustment space. The allowable error limit is the maximum allowable error threshold for registration, such as a planar error not exceeding 1 pixel or an elevation error not exceeding 2 meters.

[0098] Furthermore, the point accuracy features corresponding to the terrain matching point set can be extracted using the error ellipse model and confidence interval algorithm. Based on these point accuracy features, the basic contribution value corresponding to the terrain matching point set can be determined using a piecewise function mapping method. The spatial distribution attributes of the terrain matching point set can be extracted using Voronoi diagram spatial segmentation technology. The coverage of the spatial distribution attributes in the registration area can be analyzed using spatial autocorrelation index analysis. The regional coverage ratio corresponding to the terrain matching point set can be calculated using a grid statistical method. Based on the spatial distribution attributes, the distribution weights corresponding to the terrain matching point set can be allocated using a combined weighting method. The current registration requirements of the standard spatiotemporal framework can be detected using a technical specification parser. Based on the registration requirements, the accuracy adjustment tolerance and allowable error limit of the terrain matching point set can be calculated using system performance test reports.

[0099] Furthermore, as another embodiment of the present invention, the registration contribution degree corresponding to the terrain matching point set is calculated using the following formula by combining the basic contribution value, the distribution weight, the accuracy adjustment tolerance, the regional coverage ratio, and the allowable error limit:

[0100]

[0101] Where P represents the registration contribution of the terrain matching point set. Indicates the basic contribution value. Represents the distribution weights. Indicates the accuracy adjustment tolerance. Indicates the regional coverage ratio. This indicates the permissible error limit.

[0102] The principle behind this formula is: By employing geometric mean to balance the effects of accuracy and distribution, we can avoid the dominance of a single indicator. The diminishing marginal utility of the adjustment tolerance is described using an exponential decay function. This indicates that the ratio of the coverage ratio constrained by the hyperbolic tangent function to the error limit is within a reasonable range.

[0103] The formula has the following effects: In remote sensing image registration scenarios, it enables nonlinear evaluation of the contribution of control points, enhances registration stability, and optimizes the quality of spatial grid construction.

[0104] The anchor frame modeling module 102 is used to extract video anchor frames from the ground video data, perform background modeling processing on the video anchor frames, and obtain a monitoring background image.

[0105] This invention extracts video anchor frames from the ground video data and performs background modeling processing on the video anchor frames to obtain a monitoring background image. It can capture key scene information from a continuous video stream, eliminate interference factors in the video, and thus generate a stable background reference image.

[0106] The ground video data refers to a continuous dynamic video sequence collected by ground monitoring equipment. The video anchor frame is a key frame image selected from the ground video data that is representative and can reflect the static characteristics of the scene. The background modeling process refers to the dynamic process of performing statistical analysis on the video anchor frame to separate the background and foreground. The monitoring background image is a static scene image obtained after background modeling process, after removing temporary dynamic objects.

[0107] As an embodiment of the present invention, the step of extracting video anchor frames from the ground video data includes:

[0108] The ground video data is processed into frames to obtain ground video frames;

[0109] Calculate the scene change degree corresponding to the ground video frame, and extract candidate video frames corresponding to the ground video frame based on the scene change degree;

[0110] Extract video frame features from the candidate video frames and calculate the feature entropy corresponding to the video frame features;

[0111] Based on the feature entropy, characteristic frame features are selected from the candidate video frames;

[0112] The anchor frame is identified by performing anchor frame recognition processing on the features of the characterization frame to obtain the video anchor frame.

[0113] The terrestrial video frame segmentation is a set of single-frame images obtained by cutting a continuous terrestrial video stream at fixed time intervals; the scene variability is a numerical value that quantifies the difference in scene content between adjacent video frames; the candidate video frames are frames selected from the terrestrial video frames whose scene variability is lower than a preset threshold; the video frame features are information extracted from the candidate video frames that reflects the essential attributes of the image, such as edge features, texture features, and grayscale distribution of key areas; the feature entropy is a quantitative value that measures the complexity of video frame features, and the lower the entropy value, the more stable the features and the less redundant information; the characterization frame features are a set of core features selected from the candidate video frames whose feature entropy is lower than a set entropy value.

[0114] Furthermore, the ground video data can be segmented into frames using a timestamp-based uniform sampling algorithm to obtain ground video frames; the scene variability can be obtained by calculating the structural similarity index of adjacent frames; video frame features can be extracted from the candidate video frames using the SURF algorithm, such as extracting interest points and descriptors in the image; the feature entropy corresponding to the video frame features can be calculated using the Shannon entropy algorithm; based on the comparison between the feature entropy and a set entropy value (which can be 0.8), stable representation frame features can be selected; the representation frame features can be processed by the density peak clustering algorithm to identify anchor frames, clustering frames with high feature similarity into one class, and selecting the frame with the lowest feature entropy and the most uniform temporal distribution in each class as the video anchor frame.

[0115] This invention extracts video anchor frames from the ground video data and performs background modeling on the video anchor frames to obtain a monitoring background image. This allows for the acquisition of a static scene benchmark corresponding to the ground video data, thus providing a data foundation for subsequent motion detection or anomaly analysis. The monitoring background image is a comprehensive representation of static elements in the video scene, eliminating the influence of temporary dynamic objects.

[0116] As an embodiment of the present invention, the step of performing background modeling processing on the video anchor frame to obtain a monitoring background image includes:

[0117] Extract the inter-frame motion features from the video anchor frames and analyze the scene stability corresponding to the inter-frame motion features;

[0118] Based on the scene stability, determine the stable anchor frames in the video anchor frames;

[0119] The stable anchor frame is subjected to background construction processing to obtain an initial background base map;

[0120] The initial background image is subjected to illumination equalization processing to obtain an optimized background image;

[0121] The optimized background image is then sharpened to obtain the monitoring background image.

[0122] The inter-frame motion features are a set of pixel-level motion parameters obtained by analyzing consecutive video anchor frames, including quantitative indicators such as displacement vectors and motion intensity. The scene stability is a comprehensive indicator that analyzes the degree of change of dynamic elements in the video scene; the higher the stability, the more suitable the scene is for background modeling. The stable anchor frames are a set of frame images with highly static features selected from the video anchor frames. The initial background base map is a preliminary background image constructed using statistical analysis methods. The optimized background base map is a background image after illumination correction processing. The monitoring background image is the final static scene image with good visual effects and accuracy.

[0123] Furthermore, inter-frame motion features in the video anchor frames can be extracted using optical flow methods, such as calculating the motion vector and motion intensity of each pixel between adjacent frames to obtain inter-frame motion features. Scene stability corresponding to these inter-frame motion features can be analyzed using clustering analysis methods, such as using the DBSCAN algorithm to cluster motion features and identify stable scene regions with low motion intensity. Based on the scene stability analysis results, stable anchor frames in the video anchor frames can be determined by setting a stability threshold, such as selecting frames with motion intensity below 5 pixels / second as stable anchor frames. Background construction processing of the stable anchor frames can be performed using a Gaussian mixture model, estimating the probability distribution of background pixels using the expectation-maximization algorithm to obtain an initial background image. Illumination equalization processing of the initial background image can be performed using a histogram equalization algorithm to adjust image brightness and contrast, resulting in an optimized background image. Image sharpening processing of the optimized background image can be performed using the Laplacian operator to enhance image edge details, ultimately obtaining a high-resolution monitoring background image.

[0124] The ground feature fusion rendering module 103 is used to identify landmark ground features in the reference remote sensing image, and perform feature fusion rendering on the landmark ground features based on the monitoring background image to obtain a rendered remote sensing image.

[0125] This invention identifies landmark features in a benchmark remote sensing image, thereby determining salient buildings within the image. Based on the monitoring background image, the landmark features are fused and rendered to obtain a rendered remote sensing image, providing a high-quality image for subsequent coupling processing. The landmark features refer to geographical elements in the benchmark remote sensing image that possess significant spatial characteristics and identification value, including key features such as road intersections, water system nodes, and building outlines. The rendered remote sensing image is a geographical image obtained after feature fusion, possessing enhanced visual effects and rich spatial information. Furthermore, landmark features in the benchmark remote sensing image can be identified using a target detection model, such as the YOLOv8 model.

[0126] As an embodiment of the present invention, the step of performing feature fusion rendering on the landmark features based on the monitored background image to obtain a rendered remote sensing image includes:

[0127] Texture features are extracted from the monitored background image to obtain detailed texture features of ground features;

[0128] The outline features of the landmark features are extracted to obtain the feature outline features;

[0129] The detailed texture features and the outline features of the ground features are subjected to coordinate registration processing to obtain a registered feature pair;

[0130] The registered feature pairs are then pixel-synthesized to obtain an initial rendered image;

[0131] The initial rendered image is subjected to color correction processing to obtain a rendered remote sensing image.

[0132] The features include: texture details extracted from the monitoring background image, reflecting surface details of ground features, including roughness, directionality, and periodicity; outline features extracted from landmark ground features, including boundary point coordinates, outline curvature, and corner positions; coordinate registration, aligning texture and outline features in a spatial coordinate system; a registration feature pair, combining texture and outline features after spatial alignment; pixel synthesis, fusing the registered texture and outline information at the pixel level; an initial rendered image, the result of preliminary fusion; and color correction, adjusting the color balance and tone of the fused image.

[0133] Furthermore, texture features can be extracted from the monitored background image using a local binary mode combined with a Gabor filter to obtain multi-scale and multi-directional texture features, forming detailed texture features of ground objects; contour features can be extracted from the landmark ground objects using the Canny edge detection algorithm combined with Douglas-Peucker contour simplification to obtain accurate boundary contour information of ground objects; coordinate registration processing can be performed on the detailed texture features and contour features of ground objects using feature point matching combined with affine transformation to establish a spatial correspondence between texture and contour; color correction processing can be performed on the initial rendered image using histogram matching combined with white balance adjustment to optimize the image color distribution, ultimately obtaining a visually natural rendered remote sensing image.

[0134] Furthermore, as an optional embodiment of the present invention, the step of pixel synthesis of the registration feature pairs to obtain an initial rendered image includes:

[0135] The registered feature pairs are subjected to feature separation processing to obtain texture feature maps and contour feature maps;

[0136] The texture feature map is subjected to gradient fusion processing to obtain a fused texture layer;

[0137] The contour feature map is subjected to edge smoothing processing to obtain a smooth contour layer;

[0138] The fused texture layer and the smooth contour layer are processed to generate an initial rendered image.

[0139] The texture feature map is a grayscale image containing details of the surface texture of ground features; the contour feature map is a binary image containing only information about the boundaries of ground features; the fused texture layer is a texture image with a continuous transition after gradient fusion; and the smooth contour layer is a contour image after smoothing.

[0140] Furthermore, the feature separation processing of the registration feature pair can be implemented through a channel separation algorithm, separating the RGB channels and Alpha channels of the registration feature pair to obtain texture feature maps and contour feature maps respectively; the gradient fusion processing of the texture feature map can be implemented through a Poisson fusion algorithm to maintain the continuity and consistency of the texture in the gradient domain; the edge smoothing processing of the contour feature map can be implemented through Gaussian filtering combined with morphological closing operations to eliminate the jaggedness of the contour; the image generation processing can be implemented through an Alpha blending algorithm, using the blended texture layer as the foreground and the smoothed contour layer as the transparency mask, and performing layer compositing to obtain the initial rendered image.

[0141] The missing object compensation module 104 is used to determine the missing objects in the reference remote sensing image based on the monitoring background image, and to perform compensation rendering on the reference remote sensing image based on the missing objects to obtain a compensated remote sensing image.

[0142] This invention identifies missing objects in a baseline remote sensing image based on the monitoring background image, locating information-deficient areas. Based on these missing objects, the baseline remote sensing image is compensated and rendered, effectively filling in missing ground feature information and improving image integrity. This provides accurate data for subsequent image coupling processing. The missing objects refer to elements in the baseline remote sensing image that are not fully presented due to external factors. The compensated remote sensing image is an enhanced image with complete ground feature information obtained after object compensation. Furthermore, based on the monitoring background image, missing objects in the baseline remote sensing image can be determined using a spatially aligned ground feature comparison method. First, coordinate transformation is used to achieve spatial matching between the two images. Then, small ground feature features in the monitoring image are extracted and matched one by one with a remote sensing image ground feature database to determine the missing objects in the baseline remote sensing image.

[0143] As an embodiment of the present invention, the step of performing compensatory rendering on the reference remote sensing image based on the missing object to obtain a compensated remote sensing image includes:

[0144] Based on the lack of objects, an object mask generation process is performed on the reference remote sensing image to obtain the object mask region.

[0145] Contextual feature extraction is performed on the object mask region to obtain the surrounding feature vector;

[0146] Analyze the object semantics corresponding to the missing object, and generate compensation content corresponding to the reference remote sensing image based on the object semantics and the surrounding feature vectors;

[0147] Based on the compensation content, the baseline remote sensing image is subjected to compensation rendering to obtain a compensated remote sensing image.

[0148] Wherein, the object mask region is a binary image region in the reference remote sensing image that marks the boundary corresponding to the missing object; the surrounding feature vector is a multi-dimensional vector containing features such as color distribution, texture pattern and geometric structure around the object mask region; the object semantics is the meaning expressed by the missing object; and the compensation content is the repair image content generated for the missing object.

[0149] Furthermore, the object mask generation process can be achieved by combining a region growing algorithm with edge detection, generating an accurate object mask region based on the spatial distribution of missing objects; the context feature extraction process can be achieved by a convolutional neural network, extracting multi-scale features from the area surrounding the mask region to form a surrounding feature vector; the generation of compensation content can be achieved by a conditional generative adversarial network, generating compensation content corresponding to the surrounding environment based on object semantics and surrounding feature vectors; the compensation rendering can be achieved by a multi-scale fusion algorithm, fusing the compensation content with the reference remote sensing image at multiple resolution levels, ensuring visual consistency through color correction and texture optimization, and finally obtaining the compensated remote sensing image.

[0150] The image coupling generation module 105 is used to couple the rendered remote sensing image and the compensated remote sensing image to obtain a precise monitoring map of the target monitoring area.

[0151] This invention obtains a precise monitoring map of the target monitoring area by coupling the rendered remote sensing image and the compensated remote sensing image, thereby obtaining a high-precision and high-completeness monitoring image of the target monitoring area. The precise monitoring map is a high-quality regional monitoring image that combines the global spatiality of satellite imagery with the detail integrity of video images.

[0152] Furthermore, pixel-level alignment of the two images is first achieved through spatial coordinate anchor points, such as using fixed features like road intersections and building corners as benchmarks to correct positional deviations between images. Then, feature filtering is performed on overlapping areas to retain the outlines of macroscopic features without missing elements in the rendered remote sensing image, such as the spatial distribution of an entire building complex. Detailed elements are then embedded and filled in the compensation remote sensing image, such as streetlights and green belt boundaries within the building complex. Alternatively, the edge features of the two images can be dynamically adapted. When the edge of a road segment in the rendered image is blurred, it is corrected based on the clear edge of the corresponding road segment in the compensation image, while retaining stable regional tones in the rendered image. Finally, a high-precision monitoring map of the target monitoring area is obtained.

[0153] like Figure 2 The diagram shown is a flowchart illustrating a data coupling method for satellite imagery data and video information according to an embodiment of the present invention. In this embodiment, the method includes:

[0154] Acquire satellite imagery data and ground video data of the target monitoring area, and perform spatiotemporal calibration processing on the satellite imagery data to obtain a reference remote sensing image;

[0155] Extract video anchor frames from the ground video data, perform background modeling processing on the video anchor frames, and obtain a monitoring background image;

[0156] Identify landmark features in the baseline remote sensing image, and perform feature fusion rendering on the landmark features based on the monitoring background image to obtain a rendered remote sensing image;

[0157] Based on the monitoring background image, the missing objects in the reference remote sensing image are determined, and the reference remote sensing image is compensated and rendered based on the missing objects to obtain a compensated remote sensing image.

[0158] The rendered remote sensing image and the compensated remote sensing image are coupled to obtain a precise monitoring map of the target monitoring area.

[0159] In the several embodiments provided by this invention, it should be understood that the provided systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0160] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0161] Finally, it should be noted that in the above embodiments, each embodiment can be combined with each other or independent. Deleting any one of them will not affect the technical implementation of other embodiments. The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A system for data coupling of satellite imagery data and video information, characterized in that, The system comprises a data calibration module, an anchor point frame modeling module, a ground object fusion rendering module, a lack object compensation module, and an image coupling generation module. The data calibration module is configured to acquire satellite image data and ground video data of a target monitoring area, perform space-time calibration on the satellite image data, and obtain a reference remote sensing image. The anchor point frame modeling module is configured to extract a video anchor point frame from the ground video data, perform background modeling on the video anchor point frame, and obtain a monitoring background image. The ground object fusion rendering module is configured to identify a landmark ground object in the reference remote sensing image, perform feature fusion rendering on the landmark ground object based on the monitoring background image, and obtain a rendered remote sensing image. The lack object compensation module is configured to determine a lack object in the reference remote sensing image based on the monitoring background image, perform compensation rendering on the reference remote sensing image based on the lack object, and obtain a compensated remote sensing image. The image coupling generation module is configured to perform coupling processing on the rendered remote sensing image and the compensated remote sensing image, and obtain a precise monitoring image of the target monitoring area.

2. The system for data coupling of satellite image data and video information according to claim 1, wherein, The space-time calibration on the satellite image data comprises: performing space-time frame analysis on the satellite image data to obtain a standard space-time frame; performing control point registration on the standard space-time frame to obtain an accurate spatial grid; performing error correction on the accurate spatial grid to obtain an optimized coordinate system; and performing image resampling on the optimized coordinate system to obtain a reference remote sensing image.

3. The system for data coupling of satellite image data and video information of claim 2, wherein, The control point registration on the standard space-time frame comprises: extracting a high-quality control point set from the standard space-time frame; analyzing the spatial distribution characteristics corresponding to the high-quality control point set, determining a regional representative point set in the high-quality control point set based on the spatial distribution characteristics; identifying the terrain matching degree corresponding to the regional representative point set, and selecting a terrain matching point set in the regional representative point set based on the terrain matching degree; calculating the registration contribution degree corresponding to the terrain matching point set, and determining a core registration point set in the terrain matching point set based on the registration contribution degree; and performing control point registration on the standard space-time frame based on the core registration point set to obtain an accurate spatial grid.

4. The system for data coupling of satellite image data and video information of claim 3, wherein, The calculation of the registration contribution degree corresponding to the terrain matching point set comprises: extracting point position accuracy characteristics corresponding to the terrain matching point set, and determining a basic contribution value corresponding to the terrain matching point set based on the point position accuracy characteristics; extracting the spatial distribution attributes of the terrain matching point set, analyzing the coverage degree of the spatial distribution attributes in the registration area, and calculating the regional coverage ratio corresponding to the terrain matching point set; allocating distribution weights corresponding to the terrain matching point set based on the spatial distribution attributes; detecting the current registration requirement of the standard space-time frame, and calculating the accuracy adjustment tolerance and the allowed error limit value of the terrain matching point set based on the registration requirement. The registration contribution degree corresponding to the set of terrain matching points is calculated in combination with the basic contribution value, the distribution weight, the precision adjustment tolerance, the area coverage ratio, and the allowable error limit value.

5. The system for data coupling of satellite image data and video information of claim 1, wherein, The video anchor frame in the ground video data is extracted, including: The ground video data is frame-processed to obtain ground video frames; The scene change degree corresponding to the ground video frames is calculated, and based on the scene change degree, candidate video frames corresponding to the ground video frames are extracted; Video frame features in the candidate video frames are extracted, and feature entropy corresponding to the video frame features is calculated; Based on the feature entropy, a representative frame feature is screened out from the candidate video frames; The video anchor frame is obtained by performing anchor frame identification processing on the representative frame feature.

6. The system for data coupling of satellite image data and video information of claim 1, wherein, The background modeling processing of the video anchor frame is performed to obtain a monitoring background image, including: Inter-frame motion features in the video anchor frame are extracted, and scene stability corresponding to the inter-frame motion features is analyzed; Based on the scene stability, a stable anchor frame in the video anchor frame is determined; The stable anchor frame is processed to construct a background to obtain an initial background base map; The initial background base map is processed to balance the light to obtain an optimized background base map; The optimized background base map is processed to sharpen the image to obtain a monitoring background image.

7. The system for data coupling of satellite image data and video information of claim 1, wherein, The feature fusion rendering of the landmark ground object is performed based on the monitoring background image to obtain a rendered remote sensing image, including: Texture features of the monitoring background image are extracted to obtain ground object detail texture features; Contour features of the landmark ground object are extracted to obtain ground object contour features; Coordinate registration processing is performed on the ground object detail texture features and the ground object contour features to obtain a registration feature pair; Pixel synthesis is performed on the registration feature pair to obtain an initial rendering image; Color correction processing is performed on the initial rendering image to obtain a rendered remote sensing image.

8. The system for data coupling of satellite image data and video information according to claim 7, wherein, The pixel synthesis of the registration feature pair to obtain an initial rendering image includes: Feature separation processing is performed on the registration feature pair to obtain a texture feature map and a contour feature map; Gradient fusion processing is performed on the texture feature map to obtain a fused texture layer; Edge smoothing processing is performed on the contour feature map to obtain a smoothed contour layer; Image generation processing is performed on the fused texture layer and the smoothed contour layer to obtain an initial rendering image.

9. The system for data coupling of satellite image data and video information of claim 1, wherein, The compensation rendering of the reference remote sensing image based on the lack of object to obtain a compensated remote sensing image includes: Based on the lack of object, object mask generation processing is performed on the reference remote sensing image to obtain an object mask area; Context feature extraction processing is performed on the object mask area to obtain a surrounding feature vector; The object semantics corresponding to the lack of object are analyzed, and based on the object semantics and the surrounding feature vector, compensation content corresponding to the reference remote sensing image is generated; Based on the compensation content, the reference remote sensing image is compensated and rendered to obtain a compensated remote sensing image.

10. A method for data coupling of satellite imagery data and video information, characterized in that, The method includes: Satellite image data and ground video data of a target monitoring area are acquired, the satellite image data is subjected to spatio-temporal calibration processing to obtain a reference remote sensing image; Video anchor point frames in the ground video data are extracted, and the video anchor point frames are subjected to background modeling processing to obtain a monitoring background image; Landmark features in the reference remote sensing image are identified, and the landmark features are subjected to feature fusion rendering based on the monitoring background image to obtain a rendered remote sensing image; Based on the monitoring background image, a lack of object in the reference remote sensing image is determined, and the reference remote sensing image is subjected to compensation rendering based on the lack of object to obtain a compensated remote sensing image; The rendered remote sensing image and the compensated remote sensing image are subjected to coupling processing to obtain a precise monitoring map of the target monitoring area.

Images