A panoramic fusion and AI change monitoring method and system based on video images

Abstract

The application provides a panoramic fusion and AI change monitoring method and system based on video images, belonging to the technical field of computer vision and geographic information processing. Firstly, the road network, land data and dual-spectrum video of the target area are obtained, and a dual-spectrum panoramic image is generated by using a gimbal attitude correction fusion. Secondly, a road network land block topology graph is constructed, a spatial risk probability matrix is generated by using a space-time graph neural network, and a semantic mask is generated in combination with the land category. Subsequently, the risk probability and the semantic mask are subjected to a weighted product operation with the infrared and visible light features of the panoramic image to generate an enhanced feature matrix. Finally, the difference distribution is calculated by comparing the feature matrix of the historical period, and the area exceeding the threshold value is locked as the target change area, thereby realizing accurate monitoring based on space-time logic driving. The application can realize adaptive weighted perception of high-risk area changes, and effectively solve the false alarm and missed alarm problems caused by traditional full-image unified threshold monitoring.

Description

Technical Field

[0001] This application belongs to the field of computer vision and geographic information processing technology, and in particular relates to a panoramic fusion and AI change monitoring method and system based on video images. Background Technology

[0002] With the strengthening of natural resource supervision, panoramic fusion and AI-based change monitoring technologies based on video imagery are being increasingly widely applied in areas such as farmland protection and ecological red line supervision. This technology integrates multiple monitoring video streams to construct a panoramic view and uses artificial intelligence algorithms to automatically identify changes in land cover, effectively improving the real-time nature and coverage of supervision.

[0003] Existing technologies typically utilize pan-tilt-zoom (PTZ) cameras to capture images during patrols, generate panoramic images through image feature matching and stitching, and employ background subtraction or general deep learning object detection models to compare images from different time periods to detect changes. While some methods attempt to incorporate geofencing techniques to overlay static vector boundaries onto the video, they primarily focus on simple visual and vector overlay displays within the geofenced area.

[0004] However, most existing methods rely solely on visual features for indiscriminate full-map monitoring, ignoring the varying risk levels across different geographical locations due to differences in road accessibility or historical violation habits, and failing to adequately consider the impact of land use characteristics on change sensitivity. This monitoring model, lacking spatial business logic constraints, is prone to false alarms in non-priority areas due to environmental noise, while in high-risk areas, insufficient feature extraction weights may cause hidden violation clues to be missed. Therefore, existing technologies suffer from low accuracy and poor targeting in AI change monitoring due to the lack of guidance from geographic and spatiotemporal risk logic. Summary of the Invention

[0005] The purpose of this application is to provide a panoramic fusion and AI change monitoring method and system based on video images, so as to solve the problems of low accuracy and poor targeting of AI change monitoring in the prior art.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a panoramic fusion and AI-based change monitoring method based on video images, comprising:

[0007] Acquire road network vector data, land category data, and land violation records of the target area; collect visible light and infrared video of the target area from the PTZ camera; and obtain the attitude data of the PTZ camera.

[0008] After resampling the pixel coordinates of visible light video and infrared video using attitude data, feature point matching and image fusion are performed to generate bispectral panoramic images.

[0009] Based on road network vector data, a topology graph is constructed with land parcels as nodes and roads as edges. Land parcel violation records are associated with corresponding nodes. The topology graph is input into a trained spatiotemporal graph neural network to output the risk probability values ​​of the nodes. The risk probability values ​​are then mapped to a bispectral panoramic image to generate a probability matrix.

[0010] Based on land category data, a semantic mask with the same resolution as the bispectral panoramic image is generated through projection transformation. The element values ​​of the semantic mask are variation coefficients set according to the land category.

[0011] The first matrix is ​​obtained by element-wise multiplication of the infrared feature matrix extracted from the bispectral panoramic image with the probability matrix, and the second matrix is ​​obtained by multiplying the visible light feature matrix extracted from the bispectral panoramic image with the probability matrix and the semantic mask.

[0012] The pixel differences between the first and second matrices and the historical first and second matrices of the historical period are calculated to obtain the intensity distribution map. The set of pixels with pixel values ​​greater than the preset intensity threshold in the intensity distribution map is determined as the target change area.

[0013] Optionally, the method further includes:

[0014] The panoramic image slices corresponding to the target change area in the bispectral panoramic image are input into a trained multi-scale target detection model that includes a backbone feature extraction network and a multi-scale detection head to determine whether there is a target special machinery in the target change area. The target special machinery includes at least one of excavators, bulldozers, cement mixers and dump trucks.

[0015] When a special-purpose machine is present in the target change area, the target tracking algorithm is used to determine the motion trajectory of the special-purpose machine in the visible light video and calculate the dwell time of the special-purpose machine in the target change area.

[0016] Based on the land category corresponding to the target change area, determine whether the target change area belongs to the prohibited operation area;

[0017] If the dwell time exceeds the preset time threshold and the target change area is a prohibited area, it is determined that there is a violation of the operation in the target change area and a regulatory alarm event is triggered. The preset time threshold is determined according to the equipment type of the target special machinery.

[0018] Optionally, after resampling the pixel coordinates of the visible light video and infrared video using pose data, feature point matching and image fusion are performed to generate a dual-spectral panoramic image, including:

[0019] Extract the horizontal rotation angle, vertical pitch angle, and lens focal length from the attitude data, and calculate the homography mapping matrix of the current video frame relative to the preset horizontal plane;

[0020] Using a homography mapping matrix, visible light video and infrared video are projected onto a preset horizontal plane and resampled to obtain visible light image sequences and infrared image sequences, respectively.

[0021] Feature matching was performed on the visible light image sequence and the infrared image sequence respectively, and the geometric transformation parameters of adjacent video frames of the visible light image sequence and the infrared image sequence on the preset horizontal plane were calculated respectively.

[0022] Using the corresponding geometric transformation parameters, the visible light image sequence and the infrared image sequence are projected onto the preset panoramic spherical coordinate space and multi-scale weighted fusion is performed to generate visible light panoramic image and infrared panoramic image.

[0023] Based on the spatial correspondence of the panoramic spherical coordinate space, the visible light panoramic image and the infrared panoramic image are pixel-level superimposed and aligned to obtain a dual-spectral panoramic image.

[0024] Optionally, a topology graph is constructed based on road network vector data, with land parcels as nodes and roads as edges. Violation records for each land parcel are associated with the corresponding nodes. The topology graph is input into the risk probability values ​​of the output nodes of a trained spatiotemporal graph neural network, and these risk probability values ​​are mapped to a probability matrix generated from a bispectral panoramic image, including:

[0025] Extract the geometric center coordinates of each plot in the road network vector data as topology nodes, and extract the road segment data connecting different plots in the road network vector data as topology edges. Associate the number of violations in each area of ​​the plot violation record with the corresponding topology nodes to construct a topology graph.

[0026] The trained spatiotemporal graph neural network is used to perform message passing and feature aggregation operations on the topology graph, extracting the spatial topological association features and evolutionary features over time between the plot nodes, and outputting the normalized risk probability value of each topology node.

[0027] Based on the projection parameters of the bispectral panoramic image, the predicted risk probability of each topological node is mapped from its geospatial location to the pixel coordinates of the bispectral panoramic image. Spatial probability smoothing is then performed on the pixel regions in the bispectral panoramic image to generate a probability matrix with the same resolution as the bispectral panoramic image.

[0028] Optionally, based on land category data, a semantic mask with the same resolution as the bispectral panoramic image is generated through projection transformation. The element values ​​of the semantic mask are variation coefficients set according to the land category, including:

[0029] Establish a mapping relationship between the geospatial coordinate system where land category data is located and the coordinate space where the bispectral panoramic image is located;

[0030] Based on the mapping relationship, the land parcel attribute labels in the land category data are projected onto the corresponding pixel positions in the bispectral panoramic image to generate an initial mask for each pixel, including the land category label.

[0031] Based on the preset land parcel control level and weight mapping rules, the land category labels in the initial mask are converted into normalized weight values ​​representing the sensitivity of land parcel changes, which are then used as change coefficients to fill the corresponding pixels to generate a semantic mask.

[0032] Optionally, the first matrix is ​​obtained by element-wise multiplication of the infrared feature matrix extracted from the bispectral panoramic image and the probability matrix, and the second matrix is ​​obtained by multiplying the visible light feature matrix extracted from the bispectral panoramic image, the probability matrix, and the semantic mask, including:

[0033] Using a preset edge detection operator, spatial feature extraction processing is performed on the visible light image layer and the infrared image layer in the dual-spectrum panoramic image. A visible light feature matrix is ​​generated by identifying the edge distribution and color distribution information of each pixel region in the visible light image layer, and an infrared feature matrix is ​​generated by extracting the thermal radiation brightness distribution information of each pixel region in the infrared image layer.

[0034] The risk weight matrix is ​​obtained by multiplying the risk probability value of each coordinate point in the probability matrix with the change coefficient of the corresponding coordinate point in the semantic mask at the corresponding pixel position.

[0035] The first matrix is ​​obtained by multiplying the feature value of each channel in the infrared feature matrix with the value at the corresponding coordinate position in the probability matrix.

[0036] The second matrix is ​​obtained by multiplying the feature value of each channel in the visible light feature matrix with the value at the corresponding coordinate position in the risk weight matrix.

[0037] Optionally, the intensity threshold is determined based on the value of the preset percentile in the statistical distribution of pixel values ​​in the differential intensity distribution map of historical monitoring periods;

[0038] Calculate the pixel differences between the first and second matrices and the historical first and second matrices from historical periods to obtain a difference intensity distribution map, including:

[0039] The first matrix of the current monitoring period is spatially mapped and aligned with the first matrix of the historical period, and the values ​​of the mapped and aligned pixels are subtracted to obtain the first pixel difference matrix.

[0040] The second matrix of the current monitoring period is spatially mapped and aligned with the historical second matrix of the historical period, and the values ​​of the mapped and aligned pixels are subtracted to obtain the second pixel difference matrix.

[0041] The values ​​of the first pixel difference matrix and the second pixel difference matrix at the same pixel coordinates are extracted respectively, and then weighted and summed using a weight ratio preset based on monitoring requirements to obtain the intensity distribution map.

[0042] Secondly, this application provides a panoramic fusion and AI change monitoring system based on video images, including:

[0043] The acquisition module is used to acquire road network vector data, land category data, and land violation records of the target area, as well as visible light video and infrared video of the target area collected by the PTZ camera, and the attitude data of the PTZ camera.

[0044] The generation module is used to resample the visible light video and infrared video by pixel coordinates using attitude data, and then perform feature point matching and image fusion to generate a dual-spectral panoramic image.

[0045] The generation module is also used to construct a topology map based on road network vector data, with land parcels as nodes and roads as edges, associate land parcel violation records with corresponding nodes, input the topology map into the risk probability values ​​of the output nodes of the trained spatiotemporal graph neural network, and map the risk probability values ​​to the bispectral panoramic image generation probability matrix.

[0046] The generation module is also used to generate a semantic mask with the same resolution as the bispectral panoramic image by projection transformation based on land category data. The element values ​​of the semantic mask are variation coefficients set according to the land category.

[0047] The computation module is used to obtain the first matrix by element-wise multiplication of the infrared feature matrix and probability matrix extracted from the bispectral panoramic image, and to obtain the second matrix by multiplying the visible light feature matrix, probability matrix and semantic mask extracted from the bispectral panoramic image.

[0048] The determination module is used to calculate the pixel difference between the first matrix and the second matrix and the historical first matrix and the historical second matrix of the historical period to obtain the difference intensity distribution map, and to determine the set of pixels in the difference intensity distribution map whose pixel values ​​are greater than the preset intensity threshold as the target change area.

[0049] Thirdly, this application provides an electronic device, comprising:

[0050] Memory, used to store computer programs;

[0051] A processor is configured to execute the computer program to implement the steps of the panoramic fusion and AI change monitoring method based on video images as described in the first aspect above.

[0052] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the panoramic fusion and AI change monitoring method based on video images as described in the first aspect above.

[0053] The panoramic fusion and AI change monitoring method based on video images provided in this application generates a probability matrix that can quantify geospatial risks by constructing a spatiotemporal graph neural network based on road network topology and historical violation records, and combines it with a semantic mask generated from land category data, transforming the traditional simple visual feature extraction into a feature enhancement process driven by business logic.

[0054] This method utilizes probability matrices and semantic masks to weight the infrared and visible light features of panoramic images. Essentially, it establishes an attention mechanism at the algorithm's underlying layer, tailored to different geographical locations. This enables the monitoring model to automatically suppress environmental noise interference in low-risk areas while significantly enhancing the feature response to subtle changes in high-risk and sensitive areas. This risk-adaptive perception approach effectively overcomes the shortcomings of existing technologies that suffer from a one-size-fits-all monitoring approach due to the lack of geographic and spatiotemporal risk logic guidance. While ensuring monitoring coverage, it achieves high sensitivity in capturing violations and effective filtering of false alarms. Therefore, this application effectively solves the technical problem of low accuracy and poor targeting in AI change monitoring due to the lack of geographic and spatiotemporal risk logic guidance in existing technologies.

[0055] Furthermore, this application introduces multi-scale target detection and motion trajectory analysis within the target change area to accurately identify special engineering machinery entities such as excavators and dump trucks, and combines dwell time with land use type for multi-dimensional compliance assessment. This method uses trajectory tracking to calculate dwell time, effectively distinguishing between normal transit of engineering vehicles and substantive illegal operations. By determining whether the current plot belongs to a prohibited operation area, it elevates simple visual change detection to semantic understanding of human-vehicle-ground interaction behavior.

[0056] This secondary verification mechanism, based on behavioral temporal characteristics and spatial ownership constraints, can largely filter out non-violation interference caused by temporary vehicle stops or passage, ensuring that regulatory alerts focus on genuine destructive activities. Therefore, this application effectively solves the technical problem of high false alarm rates and insufficient identification accuracy in existing technologies due to a lack of in-depth correlation analysis between operational behavior characteristics and spatial compliance. Attached Figure Description

[0057] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 A flowchart illustrating a panoramic fusion and AI change monitoring method based on video images, provided for an embodiment of this application;

[0059] Figure 2 A flowchart illustrating a method for generating a probability matrix provided in an embodiment of this application;

[0060] Figure 3 A flowchart illustrating a method for generating a semantic mask provided in an embodiment of this application;

[0061] Figure 4 A schematic diagram of the structure of a panoramic fusion and AI change monitoring system based on video images provided in an embodiment of this application;

[0062] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0063] In the field of natural resource video monitoring, existing panoramic change detection technologies mainly rely on pixel-level visual comparison of images or general target recognition algorithms, and generally adopt a monitoring mode with a uniform threshold across the entire area. This mode does not integrate key geospatial logic such as road accessibility, historical violation distribution, and land use characteristics into the visual analysis process, resulting in a lack of perception of risk differences in different areas. This deficiency leads to an irreconcilable contradiction in existing technologies: if the detection threshold is lowered to capture hidden clues, environmental noise such as wind rustling in forest areas will cause a large number of false alarms; if the threshold is raised to filter noise, it often misses subtle changes in violations in high-risk areas, making it difficult to meet the dual requirements of accuracy and targeting in regulatory work.

[0064] To address the aforementioned issues, this application proposes a panoramic fusion and AI-based change monitoring method based on video imagery. The core of this method lies in constructing a road network topology-driven spatiotemporal risk adaptive perception mechanism. Specifically, based on the dual-spectral panoramic image generated by fusion of gimbal attitude data, this method constructs a road network plot topology map and inputs it into a spatiotemporal graph neural network to calculate a risk probability matrix representing the region's violation tendency. Furthermore, by combining a semantic mask generated from land categories, pixel-level weighted intervention is applied to the infrared and visible light features of the video imagery.

[0065] This method abandons the traditional indiscriminate visual monitoring mode. Through the dot product operation of risk probability and feature matrix, it achieves targeted enhancement of features in high-risk areas and active suppression of noise in low-risk areas at the algorithm level. It solves the problems of high false alarm rate and low accuracy caused by the lack of geographic and spatiotemporal logic guidance in existing technologies, and significantly improves the level of intelligence in natural resource supervision.

[0066] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0067] To address the problems of existing technologies, embodiments of this application provide a method, apparatus, device, computer storage medium, and computer program product for panoramic fusion and AI change monitoring based on video images. The panoramic fusion and AI change monitoring method based on video images provided in this application embodiment will be described first below.

[0068] Figure 1 A flowchart illustrating a panoramic fusion and AI change monitoring method based on video imagery, according to an embodiment of this application, is shown. Figure 1 As shown, the method includes:

[0069] S101. Acquire road network vector data, land category data, and land violation records of the target area, as well as visible light video and infrared video of the target area collected by the PTZ camera, and attitude data of the PTZ camera.

[0070] Road network vector data refers to geospatial vector files describing the location of road centerlines, topological connections, and road grades within a target area. It can include linear elements with different traffic capacities, such as highways, rural roads, and field paths. Land category data refers to spatial distribution data representing the land use attributes and control requirements of land parcels. It can include classification information such as cultivated land, forest land, construction land, and permanent basic farmland protection zones, as defined by national land survey standards. Land parcel violation records refer to statistical archives of illegal and irregular activities occurring in various administrative land parcels within the target area during historical regulatory periods. This can include structured information such as violation type, occurrence time, and frequency of investigation.

[0071] Visible light video and infrared video refer to RGB color image streams reflecting the visible light band texture characteristics of the earth's surface and thermal imaging image streams reflecting the distribution of thermal energy radiation on the earth's surface, which are synchronously acquired by a dual-spectrum pan-tilt camera. The attitude data of the pan-tilt camera refers to the lens orientation parameters of the camera at the moment of capturing a specific video frame, which may include the horizontal rotation angle of the pan-tilt, the vertical pitch angle, and the current zoom factor of the lens.

[0072] In the specific implementation process, targeting the monitoring area The data processing device establishes a connection with the natural resources database via a dedicated network or wireless transmission protocol, and downloads the latest road network vector data for the area according to the preset monitoring range boundary index. ,in It is a high-grade highway. For field paths and land category data ,in Indicates the basic farmland area. This refers to a general forest area.

[0073] Simultaneously access the law enforcement and supervision business database to retrieve records of land violations in the area. ,in Indicates land parcel Historically This is the first violation. Indicates land parcel It happened This is the first violation. Secondly, at the time of data collection... Control commands or subscription requests are sent to dual-spectrum PTZ cameras deployed at high points in the field via the ONVIF protocol or the device SDK to obtain real-time visible light video streams. With infrared video stream It also uses serial communication or network callback mechanisms to synchronously read the gimbal attitude data corresponding to each frame of video. , These represent the horizontal azimuth, vertical pitch, and focal length magnification, respectively.

[0074] S102. After resampling the pixel coordinates of the visible light video and infrared video using the attitude data, feature point matching and image fusion are performed to generate a dual-spectrum panoramic image.

[0075] Optionally, step S102, which involves resampling the pixel coordinates of the visible light video and the infrared video using the pose data, and then performing feature point matching and image fusion to generate a dual-spectrum panoramic image, may specifically include:

[0076] S1021. Extract the horizontal rotation angle, vertical pitch angle, and lens focal length from the attitude data and calculate the homography mapping matrix of the current video frame relative to the preset horizontal plane.

[0077] The homography mapping matrix is ​​a numerical matrix describing the perspective projection transformation relationship between the camera's imaging plane and a reference plane in the physical world. It is used to correct oblique view images with perspective distortion into orthographic view images. The preset horizontal plane is a virtual reference plane defined in three-dimensional geographic space to eliminate geometric distortions caused by the pan-tilt-zoom (PTZ) angle. The parameters of the preset horizontal plane are defined in Table 1 below:

[0078]

[0079] As shown in Table 1, Table 1 illustrates the key configurations required to construct a preset horizontal plane, where the reference altitude sets the vertical position of the plane, the plane normal vector defines the orientation of the plane, which is usually perpendicular to the direction of gravity, and the projection resolution specifies the pixel density after resampling.

[0080] In the specific implementation process, the attitude data first obtained from step S101 The horizontal rotation angle was analyzed. Vertical pitch angle and lens focal length Secondly, based on the camera's intrinsic parameter matrix This includes the principal point coordinates, pixel physical dimensions, and a rotation matrix constructed from the pose angles. The homography mapping matrix from the current video frame plane to the preset horizontal plane defined in Table 1 is calculated using the projective geometry principle in computer vision. For example, assume the camera's optical center height is... homography mapping matrix As shown in the following formula (1):

[0081] (1)

[0082] in The displacement vector is mainly determined by the camera's installation height. Decide, The plane normal vectors in Table 1 are... This represents the vertical distance from the camera's optical center to the ground plane. The final calculated homography mapping matrix... It is in the following form:

[0083]

[0084] The matrix Elements in and This reflects the degree of perspective transformation.

[0085] S1022. Using the homography mapping matrix, the visible light video and infrared video are projected onto a preset horizontal plane and resampled to obtain the visible light image sequence and the infrared image sequence.

[0086] Visible light image sequences and infrared image sequences refer to standardized image sets generated after inverse perspective transformation, which eliminate the perspective effect of near objects appearing larger and distant objects appearing smaller due to the shooting angle. The pixel coordinates of each frame of the image correspond to the physical position on a preset horizontal plane.

[0087] In the specific implementation process, the homography mapping matrix calculated in step S1021 is used. For visible light video streams respectively Each original image in and infrared video stream Each original image in Perspective transformation is performed. Specifically, a bilinear interpolation algorithm is used for any pixel in the original image. Through matrix operations Obtain its corresponding coordinates on the preset horizontal plane. Finally, the corrected visible light image sequence is obtained. and infrared image sequences .

[0088] S1023. Perform feature matching on the visible light image sequence and the infrared image sequence respectively, and calculate the geometric transformation parameters of adjacent video frames of the visible light image sequence and the infrared image sequence on the preset horizontal plane respectively.

[0089] Geometric transformation parameters refer to the set of values ​​describing the relative positional relationship between two adjacent frames in the same sequence on a preset horizontal plane due to gimbal rotation. These mainly include translation vectors and rotation angles. Feature matching refers to the process of finding feature point pairs representing the same physical scene points in two images. Commonly used algorithms include scale-invariant feature transformation, accelerated robust feature transformation, or directional fast rotation briefing.

[0090] In the specific implementation process, for visible light image sequences Select two adjacent frames and The algorithm uses a directional fast rotation briefing to extract corner features and generate binary descriptors. Hamming distance is used to measure feature point similarity, and a random sampling consensus algorithm is employed to remove mismatches, resulting in a set of matching point pairs. Based on these matching point pairs, the least squares method is used to calculate the similarity of the feature points. arrive geometric transformation matrix Similarly, for infrared image sequences... Perform the same operation to obtain Finally, the calculated geometric transformation parameters are in the following matrix form:

[0091]

[0092] in, Typically, it is an affine transformation matrix, including translation, rotation, and scaling parameters; , Indicates the amount of pixel translation. The angle is the rotation angle.

[0093] S1024. Using the corresponding geometric transformation parameters, the visible light image sequence and the infrared image sequence are projected onto the preset panoramic spherical coordinate space and multi-scale weighted fusion is performed to generate a visible light panoramic image and an infrared panoramic image.

[0094] The preset panoramic spherical coordinate space refers to a virtually constructed spherical model used to carry panoramic image data. Visible light panoramic images and infrared panoramic images refer to complete large-format images stitched and fused from multiple frames in this spherical coordinate system. The parameters of the preset panoramic spherical coordinate space are defined as shown in Table 2 below:

[0095] Table 2: Preset Panoramic Spherical Coordinate Space Parameter Comparison Table

[0096]

[0097] As shown in Table 2, Table 2 specifies the geometric properties of the sphere and the grid division method, ensuring that the panoramic image can seamlessly cover the 360-degree field of view.

[0098] Furthermore, the preset horizontal plane defined in Table 1 uses orthographic projection, with a projection resolution of 0.1 meters per pixel, which is a linear ground resolution, ensuring that the corrected image has a uniform ground sampling interval. The panoramic spherical coordinate space defined in Table 2 uses spherical projection, with an angular resolution of 0.05 degrees per pixel, which is an angular resolution, ensuring uniform angular sampling during panoramic stitching.

[0099] The transformation between these two coordinate systems is based on the camera's geometric installation parameters. Specifically, the ground linear resolution... With spherical angular resolution The relationship between them is approximately: , where H is the installation height of the camera relative to the ground.

[0100] In the specific implementation process, firstly, a blank panoramic canvas is constructed according to the parameters defined in Table 2. Secondly, the geometric transformation parameters obtained in step S1023 are used. The image sequences are respectively and image sequences Each frame in the image is mapped to a corresponding position in the panoramic spherical coordinate space. For overlapping areas, a multi-scale Laplacian pyramid fusion algorithm is used: the overlapping image is decomposed into sub-images of different frequency bands, and weighting coefficients are calculated for each. Linearly gradient weights are used for low-frequency components to eliminate illumination differences, while maximum value preservation or specific weights are used for high-frequency components to maintain texture details. Finally, the image is reconstructed to eliminate stitching seams. Ultimately, a visible light panoramic image covering monitoring area A is generated. and infrared panoramic images .

[0101] S1025. According to the spatial correspondence of the panoramic spherical coordinate space, the visible light panoramic image and the infrared panoramic image are pixel-level superimposed and aligned to obtain a dual-spectral panoramic image.

[0102] Dual-spectral panoramic images are composite panoramic images that include information from both the visible and infrared bands, typically represented as high-dimensional matrix data with multiple channels such as R, G, B, and Thermal. Pixel-level overlay alignment ensures that images acquired by different sensors correspond to the same geographic target at the same pixel coordinates.

[0103] In the actual implementation process, since there may be slight positional deviations in the physical installation of the visible light lens and the infrared lens, the first step is to use a pre-calibrated binocular extrinsic parameter matrix to... Make fine adjustments to make it conform to the spherical coordinate space defined in Table 2. Precise registration is then performed. The two panoramic images are then stacked and merged along the channel dimension. For example, the pixel width of the panoramic image is first calculated based on the latitude and longitude coverage and angular resolution set in Table 2. With pixel height Secondly, the size is Visible light panoramic image, with a size of The infrared panoramic images were merged to obtain a size of Dual-spectral panoramic image Where 3 represents the number of RGB color channels in the visible light image, and 4 represents the total number of channels after overlaying the infrared channels. In this panoramic image, any pixel coordinate... It also includes color and texture information for that geographical location. and temperature radiation information .

[0104] This embodiment eliminates perspective distortion and achieves pixel-level alignment of visible light and infrared dual spectra in the panoramic spherical coordinate system. It effectively solves the geometric misalignment problem caused by differences in viewing angle in traditional video stitching and generates a high-quality panoramic base map with a unified spatial reference and multi-dimensional attributes.

[0105] S103. Construct a topology graph with land parcels as nodes and roads as edges based on road network vector data, associate land parcel violation records with corresponding nodes, input the topology graph into the risk probability values ​​of the output nodes of the trained spatiotemporal graph neural network, and map the risk probability values ​​to the probability matrix generated by the bispectral panoramic image.

[0106] Optionally, step S103, which involves constructing a topology graph based on road network vector data, with land parcels as nodes and roads as edges, associating land parcel violation records with corresponding nodes, inputting the topology graph into the risk probability values ​​of the output nodes of a trained spatiotemporal graph neural network, and mapping the risk probability values ​​to a probability matrix generated from a bispectral panoramic image, may specifically include:

[0107] Figure 2 A flowchart illustrating a method for generating a probability matrix according to an embodiment of this application is shown. Figure 2 As shown, firstly, the geometric center coordinates of each plot in the road network vector data are extracted as topological nodes, and the road segment data connecting different plots are used as topological edges. Simultaneously, the number of violations in the plot violation records is correlated to construct a topological graph reflecting the spatial adjacency and traffic accessibility between plots. Then, a trained spatiotemporal graph neural network is used to perform message passing and feature aggregation operations on the topological graph to extract the spatial topological association features between plot nodes and their evolutionary features over time, outputting the normalized risk probability value corresponding to each topological node. Finally, based on the projection parameters, the predicted risk probability of the topological nodes is mapped from their geospatial location to the pixel coordinates of the bispectral panoramic image, and spatial probability smoothing is performed on the pixel regions to generate a probability matrix with the same resolution as the bispectral panoramic image.

[0108] S1031. Extract the geometric center coordinates of each plot in the road network vector data as a topology node, and extract the road segment data connecting different plots in the road network vector data as topology edges. Associate the number of violations in each area of ​​the plot violation record with the corresponding topology node to construct a topology graph.

[0109] A topology graph is a data structure used to describe the spatial adjacency and accessibility between land parcels, typically represented as... , where the set of nodes Representing various administrative or natural plots, the edges are grouped together. This represents the road network connecting the plots. The geometric center point coordinates refer to the centroid location of the polygonal outline of the plot.

[0110] In the specific implementation process, the land category data obtained in step S101 is first parsed. and road network vector data Secondly, for Each plot of land The centroid coordinates are calculated using a geometric algorithm. ), and generate a unique topology node. Next, the analysis... If there is a road segment in the middle that directly connects to the plot of land, and , or from arrive If the road network distance is less than a preset threshold, such as 500 meters, then at the node... and Establish a topological edge between them The weight of that edge The weight is set according to the road grade, for example, the weight of a Class I highway is... The weight of the field path is Finally, check the land violation records. The historical number of violations for each plot of land As a node feature, it is bound to the corresponding node. superior.

[0111] S1032. Using a trained spatiotemporal graph neural network, perform message passing and feature aggregation operations on the topology graph, extract the spatial topological association features and evolutionary features that change over time between plot nodes, and output the normalized risk probability value of each topology node.

[0112] A well-trained spatiotemporal graph neural network (SLM) is a deep learning model that has been iteratively optimized using historical data, capable of processing graph-structured data and capturing spatiotemporal dependencies. Common architectures include combinations of graph convolutional networks and long short-term memory networks. Spatial topological association features refer to the mutual influence between land parcels due to road network connections, such as the diffusion effect of violations by adjacent parcels. A normalized risk probability value, with an output value between 0 and 1, represents the likelihood of a violation occurring on that land parcel.

[0113] In the specific implementation process, the first step is to train a well-trained spatiotemporal graph neural network: first, obtain a training sample set, which includes training samples from multiple historical time periods. Each training sample includes the historical road network topology, the historical land violation record sequence, and the corresponding label of the actual violation occurrence in the next time period.

[0114] Secondly, perform the following steps for each training sample: extract the road network topology map from each training sample. and historical violation record sequence The input is fed into a pre-defined spatiotemporal graph neural network model. This model first uses graph convolutional layers to aggregate neighbor node information and calculate spatial features. As shown in the following formula (2):

[0115] (2)

[0116] in, For the first The node feature matrix of the layer It is an adjacency matrix. For degree matrix, For weight parameters. When When the layer is the last graph convolutional layer of the network, its output This can be considered as the extracted spatial features. Next, spatial features The input temporal convolutional layer processes the temporal dimension to obtain spatiotemporal aggregated features. Finally, the spatiotemporal aggregation features are... The input to the fully connected layer undergoes a linear transformation, and the output is mapped to the interval between 0 and 1 using the Sigmoid activation function to obtain the predicted violation probability value for each node. .

[0117] Finally, labels are based on the actual occurrence of violations. and predicted probability value of violation The loss value of the model is calculated using the cross-entropy loss function. In the loss value Training stop conditions not met, such as Furthermore, if the number of iterations is less than 1000, the weight parameters in the model are adjusted using the backpropagation algorithm. The updated prediction model is obtained and the training continues until the stopping condition is met, resulting in a trained spatiotemporal graph neural network.

[0118] Next, using the trained model, the current topology graph and associated features constructed in step S1031 are input into the model. After forward computation, the model sequentially completes spatial feature aggregation, spatiotemporal feature extraction, and fully connected layer classification mapping, finally outputting the value of each node. Risk probability value .

[0119] S1033. Based on the projection parameters of the bispectral panoramic image, the predicted risk probability of each topological node is mapped from its geographic spatial location to the pixel coordinates of the bispectral panoramic image, and spatial probability smoothing is performed on the pixel regions in the bispectral panoramic image to generate a probability matrix with the same resolution as the bispectral panoramic image.

[0120] The probability matrix is ​​a two-dimensional numerical matrix with the same size as the bispectral panoramic image, where each element represents the violation risk level of the corresponding pixel location. Projection parameters are transformation coefficients that establish the mapping relationship between geographic coordinates (latitude and longitude) and panoramic image pixel coordinates, typically based on the panoramic spherical coordinates defined in step S1024. Spatial probability smoothing processing refers to using Gaussian blur or interpolation algorithms to eliminate boundary abrupt changes caused by discrete node mapping, thereby generating a continuous risk heat map distribution.

[0121] In the specific implementation process, firstly, based on the panoramic spherical coordinate system parameters determined in Table 2, all areas falling on the plot in the panoramic image are located. The set of pixels within the range. Secondly, for each pixel coordinate in this set ( The initial risk value of each plot of land was set at that value. Predicted probability value Finally, considering that the impact of violations or regulatory oversight may not be strictly limited to the land parcel boundaries, a Gaussian filter is used to smooth the initial matrix to simulate the continuous spatial transition of risk and avoid unstable responses in the algorithm due to sudden probability changes at boundary pixels, generating the final probability matrix. , is represented as:

[0122]

[0123] Among the elements Representing high-risk areas, elements Representing low-risk areas, intermediate elements This demonstrates the effect of a smooth transition.

[0124] This embodiment deeply explores the road adjacency effect and historical violation evolution patterns between land parcels, transforming abstract geographic business data into a pixel-level risk probability matrix that is precisely aligned with panoramic imagery. This endows the monitoring system with the ability to perceive high-risk areas in advance, effectively enabling the focus on high-risk targets and the proactive suppression of environmental noise in non-key areas.

[0125] S104. Based on the land category data, generate a semantic mask with the same resolution as the bispectral panoramic image through projection transformation. The element values ​​of the semantic mask are variation coefficients set according to the land category.

[0126] Optionally, step S104, which generates a semantic mask with the same resolution as the bispectral panoramic image through projection transformation based on land category data, wherein the element values ​​of the semantic mask are variation coefficients set according to the land category, may specifically include:

[0127] Figure 3 A flowchart illustrating a method for generating semantic masks according to an embodiment of this application is shown. Figure 3As shown, the process first establishes a mapping relationship between the geospatial coordinate system of the land category data and the coordinate space of the bispectral panoramic image. To accommodate the large-scale rotation of the pan-tilt camera and avoid edge distortion in planar projection, a spherical coordinate system can be chosen as the basic reference frame for the panoramic image. A transformation function is constructed using projection parameters to accurately calculate the geographic coordinate points into pixel coordinates on the panoramic image. Subsequently, based on the mapping relationship, the land parcel attribute labels in the land category data are projected onto the corresponding pixel points in the bispectral panoramic image. A scanline filling algorithm or a polygon rasterization algorithm is used to find the set of all pixels within the outline, generating an initial mask for each pixel, including the land category label. Finally, according to the preset land parcel control level and weight mapping rules, the land category labels in the initial mask are converted into normalized weight values ​​representing the sensitivity of land parcel changes, which are then used as change coefficients to fill the corresponding pixels, ultimately generating a semantic mask.

[0128] S1041. Establish the mapping relationship between the geospatial coordinate system where the land category data is located and the coordinate space where the bispectral panoramic image is located.

[0129] A geospatial coordinate system refers to a standard coordinate system used to define the location reference of land category vector data, commonly such as WGS84 latitude and longitude coordinates or CGCS2000 projected coordinates. A mapping relationship refers to a mathematical transformation model that accurately converts geographic points on the Earth's surface to pixel points in a panoramic image. The mapping relationships are shown in Table 3 below:

[0130]

[0131] In this embodiment, to accommodate the large-scale rotation of the PTZ camera and avoid edge distortion of the planar projection, a spherical coordinate system is ultimately selected as the basic reference frame for the panoramic image. Its precise mapping to geographic coordinates is defined by the parameters in Table 3.

[0132] As shown in Table 3, Table 3 shows the key variables required to achieve this conversion, where the origin latitude and longitude anchors the geographic center of the panorama, the projection scale determines the corresponding ratio of angles to pixels, and the distortion correction coefficient is used to fine-tune the nonlinear deviations generated by the optical system.

[0133] In the specific implementation process, the land category data in step S101 is read first. The source data is used to determine its source coordinate system, such as EPSG:4326, and the header file of the bispectral panoramic image is read to obtain the panoramic spherical definition. Next, a transformation function is constructed using the parameters in Table 3. Specifically, let's assume the source coordinates are... The center origin of the panoramic image is ( ), projection scale is The rotation offset is Then the transformation function The linearized expression in the local region is shown in the following formula (3):

[0134] (3)

[0135] in This is the correction term calculated based on the distortion coefficient. Using the above formula (3), any geographic coordinates... Each can be uniquely calculated as a pixel coordinate on the panoramic image. .

[0136] S1042. Based on the mapping relationship, project the land parcel attribute labels in the land category data to the corresponding pixel positions in the dual-spectral panoramic image to generate an initial mask for each pixel, including the land category label.

[0137] Land parcel attribute labels refer to the classification identifiers assigned to each land parcel in land category data, such as 0101 representing paddy fields and 0301 representing wooded land. The initial mask is a two-dimensional matrix the same size as the panoramic image, storing the classification code of the land parcel covered by each pixel.

[0138] In the specific implementation process, the land category data is first traversed. Each polygonal plot in Secondly, regarding the land parcel The boundary vertex sequence is converted into a polygonal outline in the panoramic coordinate system using the above formula (3). Then, a scanline filling algorithm or a polygon rasterization algorithm is used to find the set of all pixels included within the outline. For each pixel in the set... The land parcel attribute label corresponding to this pixel. Set it in the initial mask matrix The value in the middle. Finally, for monitoring area A, assume that an initial mask is generated. for:

[0139]

[0140] in Represents the code for basic farmland. Represents forest land code, This represents the code for urban residential land use.

[0141] S1043. According to the preset land parcel control level and weight mapping rules, the land category labels in the initial mask are converted into normalized weight values ​​representing the sensitivity of land parcel changes, and used as change coefficients to fill the corresponding pixels to generate a semantic mask.

[0142] The semantic mask refers to the final generated numerical matrix used to guide the AI ​​algorithm in feature weighting, with each element being a change coefficient. The change coefficient is a floating-point number between 0 and 1; a higher value indicates greater sensitivity to change, i.e., a higher level of regulation. The land parcel control level and weight mapping rule refers to a logical table that maps administrative land use types to algorithm weights. The land parcel control level and weight mapping rule is shown in Table 4 below:

[0143]

[0144] As shown in Table 4, Table 4 lists the control levels and quantitative weights corresponding to different land use codes. Level 1 is the highest level of control, and the corresponding weight value, i.e., the coefficient of change, is also the highest, reflecting the business logic of prioritizing arable land, followed by ecological land, and moderately regulating construction land.

[0145] In the specific implementation process, the initial mask generated in step S1042 is first processed. First, perform a pixel-by-pixel traversal. Next, for each pixel's value (land class code), look up the corresponding weight value in Table 4 and write it into a new matrix to generate the final semantic mask. It can be:

[0146]

[0147] Among them, elements Corresponding to the high sensitivity of basic farmland, elements Corresponding to moderate sensitivity in woodlands, elements The corresponding low sensitivity to construction land.

[0148] This embodiment enables the precise projection of vectorized land use data into the video pixel space. It establishes a land use-driven feature weighting mechanism at the algorithm's underlying level, significantly improving the targeting and intelligence of law enforcement and supervision.

[0149] S105. The first matrix is ​​obtained by element-wise multiplication of the infrared feature matrix and the probability matrix extracted from the bispectral panoramic image. The second matrix is ​​obtained by multiplying the visible light feature matrix, the probability matrix, and the semantic mask extracted from the bispectral panoramic image.

[0150] Optionally, step S105, which involves element-wise multiplication of the infrared feature matrix and probability matrix extracted from the bispectral panoramic image to obtain the first matrix, and multiplying the visible light feature matrix, probability matrix, and semantic mask extracted from the bispectral panoramic image to obtain the second matrix, may specifically include:

[0151] S1051. Using a preset edge detection operator, spatial feature extraction processing is performed on the visible light image layer and infrared image layer in the dual-spectrum panoramic image respectively. A visible light feature matrix is ​​generated by identifying the edge distribution and color distribution information of each pixel region in the visible light image layer, and an infrared feature matrix is ​​generated by extracting the thermal radiation brightness distribution information of each pixel region in the infrared image layer.

[0152] Edge detection operators are mathematical convolution kernels used to identify points of abrupt brightness changes (edges) in an image, such as the Sobel or Laplacian operators, to extract texture contours. The visible light feature matrix is ​​a multidimensional numerical matrix representing texture details, color differences, and edge intensity in a visible light panoramic image. The infrared feature matrix is ​​a data matrix representing the surface temperature radiation intensity and its spatial gradient changes in an infrared panoramic image.

[0153] In the specific implementation process, firstly, the dual-spectral panoramic image generated in step S1025 is used... Separate the visible light image layer and infrared image layer Secondly, for the visible light image layer... The Sobel operator is used to calculate the gradients in the horizontal and vertical directions respectively, and combined with the original RGB color channels to generate a visible light feature matrix. For infrared image layers The grayscale values ​​are directly extracted and the local variance is calculated to reflect the non-uniformity of thermal energy distribution, generating an infrared feature matrix. .

[0154] S1052. Multiply the risk probability value of each coordinate position point in the probability matrix with the change coefficient of the corresponding coordinate position point in the semantic mask by the corresponding pixel position to obtain the risk weight matrix.

[0155] The risk weight matrix is ​​a joint weighted map that integrates road network topology risk and land control level, used to quantify the overall importance of each pixel in change monitoring.

[0156] In the specific implementation process, firstly, the probability matrix generated in step S1033 is read. With the semantic mask generated in step S1043 Secondly, for each coordinate in the matrix... ,Will Risk probability value and coefficient of variation in Multiply to get the corresponding This generates a risk weight matrix. It can be:

[0157]

[0158] S1053. Multiply the feature values ​​of each channel in the infrared feature matrix with the values ​​at the corresponding coordinate positions in the probability matrix to obtain the first matrix.

[0159] The first matrix refers to the infrared radiation feature map after being weighted by the road network risk probability. It highlights the thermal energy anomalies in high-risk areas while suppressing thermal noise in low-risk areas.

[0160] In the specific implementation process, the infrared feature matrix will be used. With probability matrix Perform element-wise multiplication. Specifically, for any coordinates... ,calculate , and Infrared feature matrix With probability matrix The element value at the corresponding position. The resulting first matrix. It can be:

[0161]

[0162] in, This represents the weighted infrared difference intensity.

[0163] S1054. Multiply the feature values ​​of each channel in the visible light feature matrix with the corresponding coordinate values ​​in the risk weight matrix to obtain the second matrix.

[0164] The second matrix refers to the visible light feature map after being weighted by both spatiotemporal risk and land nature, which is used to capture the visual morphological changes of land cover.

[0165] In the specific implementation process, the visible light feature matrix will be used. With the risk weight matrix generated in step S1052 Perform element-wise multiplication. Specifically, for any coordinates... ,calculate The final second matrix It manifests as:

[0166]

[0167] in This represents the weighted difference in visible light intensity.

[0168] This embodiment creatively constructs a feature weighting mechanism, which effectively suppresses background noise in low-risk areas, significantly enhances the feature response of subtle changes in high-risk plots, and greatly improves the sensitivity and accuracy of violation monitoring.

[0169] S106. Calculate the pixel difference between the first matrix and the second matrix and the historical first matrix and the historical second matrix of the historical period to obtain the intensity distribution map, and determine the set of pixels in the intensity distribution map whose pixel values ​​are greater than the preset intensity threshold as the target change area.

[0170] The historical first matrix and historical second matrix refer to the baseline feature data representing historical infrared thermal characteristics and historical visible light visual characteristics, respectively, generated using the same methodology in the previous monitoring period. The differential intensity distribution map is a two-dimensional heat map that integrates the amplitude of changes in both spectra; higher pixel values ​​indicate a greater likelihood of surface changes occurring at that location.

[0171] The target change region refers to the set of connected pixels ultimately determined to have suspected violations, typically corresponding to a specific physical land parcel. The preset intensity threshold is a critical value used to truncate significantly changing regions from continuously varying intensity values.

[0172] Optionally, the intensity threshold in step S106 is determined based on the preset percentile value of the pixel value statistical distribution in the differential intensity distribution map of the historical monitoring period.

[0173] The preset intensity thresholds are shown in Table 5 below:

[0174]

[0175] As shown in Table 5, the threshold calculation logic is based on statistical principles. The basic confidence level sets the minimum filtering threshold, the dynamic percentile specifies the method of adaptively adjusting the threshold according to historical data fluctuations, and the environmental correction factor is used to appropriately relax the standard in severe weather to reduce false alarms.

[0176] In the specific implementation process, the first historical matrix stored in the time series database is retrieved first. And the second historical matrix Secondly, using simple matrix subtraction and weighted summation operations, the first matrix of the current monitoring period is calculated. Second matrix The difference between the data and historical benchmarks generates a strength distribution map. Next, the preset intensity threshold for this monitoring is dynamically determined according to the rules in Table 5. .

[0177] For example, assuming a histogram of pixel values ​​from statistical historical periods such as the past 7 days' intensity distribution maps, find the 95th percentile value m, and take an environmental correction factor of... Then the final threshold Finally, regarding the current intensity distribution map... Perform binarization: [convert all] Pixels marked as 1 are considered "change," while the rest are marked as 0 (background). Connectivity analysis is performed on the pixels marked as 1, removing small noise pixels with an area smaller than 50 pixels. The remaining connected regions are then identified as the target change regions. .

[0178] The process of calculating the pixel differences between the first and second matrices and the historical first and second matrices of historical periods to obtain the intensity distribution map in step S106 can specifically include:

[0179] The first matrix of the current monitoring period is spatially mapped and aligned with the first matrix of the historical period, and the values ​​of the mapped and aligned pixels are subtracted to obtain the first pixel difference matrix.

[0180] The first pixel difference matrix is ​​a numerical matrix representing the degree of difference between the current moment and historical moments in the infrared thermal radiation dimension. Its value reflects the addition or disappearance of surface heat sources such as vehicle engines and construction equipment. Spatial location mapping alignment refers to eliminating pixel offset caused by minor gimbal reset errors between two shots, ensuring that the same physical location is being compared.

[0181] In the specific implementation process, the first historical matrix is ​​retrieved from the database. Secondly, using image registration algorithms such as phase-correlation-based registration, the current first matrix is ​​calculated. Compared to The small translation amount ( ), and on Perform a reverse translation correction. Next, perform matrix subtraction: .

[0182] The second matrix of the current monitoring period is spatially mapped and aligned with the historical second matrix of the historical period, and the values ​​of the mapped and aligned pixels are subtracted to obtain the second pixel difference matrix.

[0183] The second pixel difference matrix is ​​a numerical matrix representing the degree of difference between the current time and historical time in the visible light visual texture dimension. Its values ​​reflect the morphological changes of surface cover such as bulldozing and building construction. This matrix has also undergone strict spatial alignment processing.

[0184] In the specific implementation process, the historical second matrix is ​​first retrieved from the database. Secondly, using image registration algorithms such as phase-correlation-based registration, the current first matrix is ​​calculated. Compared to The small translation amount ( ), and on Perform a reverse translation correction. Then perform a subtraction operation: .

[0185] The values ​​of the first pixel difference matrix and the second pixel difference matrix at the same pixel coordinates are extracted respectively, and then weighted and summed using a weight ratio preset based on monitoring requirements to obtain the intensity distribution map.

[0186] The differential intensity distribution map refers to the final generated comprehensive change detection result map that integrates dual-spectral change information. The weighting ratio refers to the weighting coefficient determined according to different monitoring task focuses, such as whether to focus on nighttime illegal sampling or daytime illegal construction. The preset weighting ratios are shown in Table 6 below:

[0187]

[0188] As shown in Table 6, Table 6 presents the configuration strategies for different scene modes. Infrared weight is used to adjust the proportion of thermal anomalies in the final result, while visible light weight is used to adjust the proportion of visual texture changes.

[0189] In the specific implementation process, first identify the current monitoring environment or user-defined mode, such as the daytime routine mode, and refer to Table 6 to obtain the corresponding infrared weights. and visible light weight Then, for the first pixel difference matrix Second pixel difference matrix Perform weighted fusion for any pixel coordinates Calculate the final difference value The final generated differential intensity distribution map It is in the following form:

[0190]

[0191] in, This indicates the overall difference intensity of the corresponding pixel.

[0192] This embodiment overcomes the blind spots of single-spectrum monitoring at night or in severe weather, ensuring that it can automatically adapt to background fluctuations under different seasons or lighting conditions. While ensuring an extremely low false alarm rate, it achieves high-precision capture of all-weather, multi-type violation changes.

[0193] Optionally, the method further includes:

[0194] The panoramic image slices corresponding to the target change area in the bispectral panoramic image are input into a trained multi-scale target detection model that includes a backbone feature extraction network and a multi-scale detection head to determine whether there is a target special machinery in the target change area. The target special machinery includes at least one of excavators, bulldozers, cement mixers, and dump trucks.

[0195] A panoramic image slice refers to a local image patch, including the area of ​​change of the target and the surrounding background, cropped from a complete bispectral panoramic image. A multi-scale target detection model refers to a deep learning neural network, such as YOLOv5 or Faster R-CNN, which includes a backbone network, such as ResNet, for extracting image features, and a multi-scale detection head for predicting the target's location and category on feature maps of different resolutions. Specialized target machinery refers to vehicles operating illegally, which are a key focus of natural resource regulation; their visual characteristics, such as robotic arms and mixing tanks, are significantly different from ordinary vehicles.

[0196] In the specific implementation process, firstly, a well-trained multi-scale target detection model is obtained: firstly, a training sample set is acquired, which includes multiple labeled images of engineering vehicles. Each training sample includes visible light and infrared image slices, vehicle category labels such as excavator, bulldozer, etc., and corresponding ground truth bounding box coordinate labels.

[0197] Next, the following steps are performed on each training sample: Each training sample is input into a pre-defined detection model. The model first uses a backbone feature extraction network to extract semantic features of the image layer by layer, generating a multi-level feature pyramid. Then, these features are input into a multi-scale detection head to predict the class probability and bounding box offset of candidate targets at large, medium, and small scales, respectively, to obtain the predicted detection results.

[0198] Finally, based on the ground truth bounding box coordinate labels and the predicted detection results, the total loss value of the model is calculated using the loss function and the classification loss function. In the loss value Training stop conditions not met, such as In the case of an error, the weight parameters of the backbone network and the detection head are adjusted by the gradient descent algorithm to obtain the updated model, and then the training continues until the stopping condition is met, resulting in a well-trained multi-scale object detection model.

[0199] Using the trained model, the target change region determined in step S106 is... Corresponding slice image Input model. Model output: list of detection results. .

[0200] When a special type of machinery is present in the target change area, a target tracking algorithm is used to determine the movement trajectory of the special type of machinery in the visible light video, and the dwell time of the special type of machinery in the target change area is calculated.

[0201] Target tracking algorithms refer to techniques used to lock and record the sequence of positions of the same target in consecutive video frames. Common examples include deep learning and simple online real-time tracking or byte-level correlation tracking. In specific implementation, firstly, when special machinery is detected, the corresponding original visible light video stream is automatically retrieved. Secondly, the deep learning algorithm and a simple online real-time tracker are initialized to assign a unique ID to the detected machinery.

[0202] Subsequently, the video was analyzed frame by frame to record the centroid coordinate sequence, i.e., the motion trajectory, of that ID. Finally, the trajectory is calculated within the target's changing region. The duration of stay within the area. For example, assuming the excavator enters the area at 10:00 and has not left by 10:45, the calculated dwell time is... minute.

[0203] Based on the land category corresponding to the target change area, determine whether the target change area belongs to a prohibited operation area.

[0204] Prohibited work areas refer to land parcels where non-agricultural construction or destructive operations are strictly prohibited by laws and regulations. These typically include permanent basic farmland and ecological protection red line areas. The determination is based on land category data obtained in step S101.

[0205] In the specific implementation process, query the target change area. The center coordinates in land category data The attributes of the area are considered. If the area is classified as basic farmland or forest land, it is determined to be a prohibited area; if it is construction land or temporary storage yard, it is determined to be a non-prohibited area.

[0206] If the dwell time exceeds the preset time threshold and the target change area is a prohibited area, it is determined that there is a violation of the operation in the target change area and a regulatory alarm event is triggered. The preset time threshold is determined according to the equipment type of the target special machinery.

[0207] A regulatory alert event refers to an electronic work order that generates evidence of violation, including panoramic views, slices, video clips, and location information, and is pushed to law enforcement personnel. The preset time threshold refers to the minimum duration standard used to define a violation. The preset time thresholds are shown in Table 7 below:

[0208]

[0209] As shown in Table 7, Table 7 presents the judgment criteria set for different types of machinery, reflecting the differentiated consideration of different operating efficiencies and degrees of harm.

[0210] In the specific implementation process, assuming the detected machine type is an excavator, the dwell time is... The target area changed within minutes and is classified as basic farmland. Referring to Table 7, the corresponding time threshold is 30 minutes. Because... If the area is a prohibited work zone, the judgment logic is valid, an alarm event for illegal occupation of farmland for construction is generated, and a package including close-up photos of the excavator, a panoramic location map, and a 45-minute trajectory video clip is sent to the monitoring platform.

[0211] This embodiment not only accurately identifies special engineering machinery entities such as excavators, but also effectively eliminates interference from transit vehicles and compliant construction through the dual constraints of dwell time and land use boundary rules, greatly improving the accuracy of discovering clues of violations and the effectiveness of law enforcement evidence collection.

[0212] Figure 4 This is a schematic diagram illustrating a specific implementation of a panoramic fusion and AI change monitoring system based on video images, provided in an embodiment of this application. (Refer to...) Figure 4 The system may include:

[0213] The 410 acquisition module is used to acquire road network vector data, land category data, and land violation records of the target area, as well as visible light video and infrared video of the target area collected by the PTZ camera, and the attitude data of the PTZ camera.

[0214] The 420 generation module is used to resample the visible light video and infrared video by pixel coordinates using attitude data, and then perform feature point matching and image fusion to generate a dual-spectrum panoramic image.

[0215] The 420 generation module is also used to construct a topology map based on road network vector data, with land parcels as nodes and roads as edges, associate land parcel violation records with corresponding nodes, input the topology map into the risk probability values ​​of the output nodes of the trained spatiotemporal graph neural network, and map the risk probability values ​​to the bispectral panoramic image generation probability matrix.

[0216] The 420 generation module is also used to generate a semantic mask with the same resolution as the bispectral panoramic image by projection transformation based on land category data. The element values ​​of the semantic mask are variation coefficients set according to the land category.

[0217] The 430 operation module is used to obtain the first matrix by element-wise multiplication of the infrared feature matrix and probability matrix extracted from the dual-spectrum panoramic image, and to obtain the second matrix by multiplying the visible light feature matrix, probability matrix and semantic mask extracted from the dual-spectrum panoramic image.

[0218] The 440 determination module is used to calculate the pixel difference between the first matrix and the second matrix and the historical first matrix and the historical second matrix of the historical period to obtain the difference intensity distribution map, and to determine the set of pixels in the difference intensity distribution map whose pixel values ​​are greater than the preset intensity threshold as the target change area.

[0219] The panoramic fusion and AI change monitoring system based on video images in this application is used to implement the aforementioned panoramic fusion and AI change monitoring method based on video images. Therefore, the specific implementation of the panoramic fusion and AI change monitoring system based on video images can be found in the embodiment section of the panoramic fusion and AI change monitoring method based on video images above. The specific implementation can be referred to the description of the corresponding embodiments, and will not be repeated here.

[0220] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application is shown.

[0221] The electronic device may include a processor 510 and a memory 520 storing computer program instructions.

[0222] Specifically, the processor 510 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0223] Memory 520 may include mass storage for data or instructions. For example, and not limitingly, memory 520 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 520 may include removable or non-removable (or fixed) media. Where appropriate, memory 520 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 520 is non-volatile solid-state memory.

[0224] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of this disclosure.

[0225] The processor 510 reads and executes computer program instructions stored in the memory 520 to implement any of the panoramic fusion and AI change monitoring methods based on video images in the above embodiments.

[0226] In one example, the electronic device may also include a communication interface 530 and a bus 540. Wherein, such as Figure 5 As shown, the processor 510, memory 520, and communication interface 530 are connected through bus 540 and complete communication with each other.

[0227] The communication interface 530 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0228] Bus 540 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 540 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0229] The electronic device can execute the panoramic fusion and AI change monitoring method based on video images in the embodiments of this application, thereby realizing the panoramic fusion and AI change monitoring method based on video images described in conjunction with the accompanying drawings.

[0230] Furthermore, in conjunction with the panoramic fusion and AI change monitoring methods based on video images in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the panoramic fusion and AI change monitoring methods based on video images in the above embodiments.

[0231] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0232] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0233] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0234] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0235] The above provides a detailed description of a panoramic fusion and AI change monitoring method and system based on video images provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A panoramic fusion and AI-based change monitoring method based on video images, characterized in that, include: Acquire road network vector data, land category data, and land violation records of the target area; collect visible light and infrared video of the target area from the PTZ camera; and obtain the attitude data of the PTZ camera. After resampling the pixel coordinates of the visible light video and the infrared video using the attitude data, feature point matching and image fusion are performed to generate a dual-spectrum panoramic image. Based on the road network vector data, a topology graph is constructed with land parcels as nodes and roads as edges. The violation records of the land parcels are associated with the corresponding nodes. The topology graph is input into the risk probability value of the output nodes of the trained spatiotemporal graph neural network, and the risk probability value is mapped to the probability matrix generated by the bispectral panoramic image. Based on the land category data, a semantic mask with the same resolution as the bispectral panoramic image is generated through projection transformation. The element values ​​of the semantic mask are variation coefficients set according to the land category. The first matrix is ​​obtained by element-wise multiplying the infrared feature matrix extracted from the dual-spectral panoramic image with the probability matrix, and the second matrix is ​​obtained by multiplying the visible light feature matrix extracted from the dual-spectral panoramic image with the probability matrix and the semantic mask. The pixel differences between the first matrix and the second matrix and the historical first matrix and the historical second matrix of the historical period are calculated to obtain a difference intensity distribution map, and the set of pixels in the difference intensity distribution map whose pixel values ​​are greater than a preset intensity threshold are determined as the target change area.

2. The method according to claim 1, characterized in that, The method further includes: The panoramic image slices corresponding to the target change area in the bispectral panoramic image are input into a trained multi-scale target detection model that includes a backbone feature extraction network and a multi-scale detection head to determine whether there is a target special machinery in the target change area. The target special machinery includes at least one of excavators, bulldozers, cement mixers and dump trucks. When the target special machinery is present in the target change area, the motion trajectory of the target special machinery is determined in the visible light video using a target tracking algorithm, and the dwell time of the target special machinery in the target change area is calculated. Based on the land category corresponding to the target change area, determine whether the target change area belongs to a prohibited operation area; If the dwell time exceeds a preset time threshold and the target change area is a prohibited area, it is determined that there is a violation of the operation in the target change area and a regulatory alarm event is triggered. The preset time threshold is determined according to the equipment type of the target special machinery.

3. The method according to claim 1, characterized in that, The process of resampling pixel coordinates of the visible light video and the infrared video using the attitude data, and then performing feature point matching and image fusion to generate a dual-spectrum panoramic image includes: Extract the horizontal rotation angle, vertical pitch angle, and lens focal length from the attitude data and calculate the homography mapping matrix of the current video frame relative to a preset horizontal plane; Using the homography mapping matrix, the visible light video and the infrared video are projected onto the preset horizontal plane and resampling is performed to obtain a visible light image sequence and an infrared image sequence. Feature matching is performed on the visible light image sequence and the infrared image sequence respectively, and the geometric transformation parameters of adjacent video frames of the visible light image sequence and the infrared image sequence on the preset horizontal plane are calculated respectively. Using the corresponding geometric transformation parameters, the visible light image sequence and the infrared image sequence are projected onto a preset panoramic spherical coordinate space and multi-scale weighted fusion is performed to generate a visible light panoramic image and an infrared panoramic image. According to the spatial correspondence of the panoramic spherical coordinate space, the visible light panoramic image and the infrared panoramic image are pixel-level superimposed and aligned to obtain the dual-spectral panoramic image.

4. The method according to claim 1, characterized in that, The process of constructing a topology graph based on the road network vector data, with land parcels as nodes and roads as edges, associating the land parcel violation records with corresponding nodes, inputting the topology graph into the risk probability values ​​of the output nodes of a trained spatiotemporal graph neural network, and mapping the risk probability values ​​to the probability matrix generated by the dual-spectral panoramic image includes: The geometric center coordinates of each plot in the road network vector data are extracted as topological nodes, and the road segment data connecting different plots in the road network vector data are extracted as topological edges. The number of violations in each area of ​​the plot violation record is associated with the corresponding topological node to construct the topological graph. The trained spatiotemporal graph neural network is used to perform message passing and feature aggregation operations on the topology graph, extracting the spatial topological association features and evolutionary features over time between land parcel nodes, and outputting the normalized risk probability value of each topology node. Based on the projection parameters of the bispectral panoramic image, the predicted risk probability of each topological node is mapped from its geographic spatial location to the pixel coordinates of the bispectral panoramic image, and spatial probability smoothing is performed on the pixel regions in the bispectral panoramic image to generate a probability matrix with the same resolution as the bispectral panoramic image.

5. The method according to claim 1, characterized in that, The step of generating a semantic mask with the same resolution as the bispectral panoramic image through projection transformation based on the land category data, wherein the element values ​​of the semantic mask are variation coefficients set according to the land category, including: Establish a mapping relationship between the geospatial coordinate system where the land category data is located and the coordinate space where the bispectral panoramic image is located; Based on the mapping relationship, the land parcel attribute labels in the land category data are projected onto the corresponding pixel positions of the dual-spectral panoramic image to generate an initial mask for each pixel including the land category label; According to the preset land parcel control level and weight mapping rules, the land category labels in the initial mask are converted into normalized weight values ​​representing the sensitivity of land parcel changes, and these values ​​are used as change coefficients to fill the corresponding pixels to generate the semantic mask.

6. The method according to claim 1, characterized in that, The step of obtaining a first matrix by element-wise multiplying the infrared feature matrix extracted from the dual-spectrum panoramic image with the probability matrix, and obtaining a second matrix by multiplying the visible light feature matrix extracted from the dual-spectrum panoramic image with the probability matrix and the semantic mask, includes: Spatial feature extraction processing is performed on the visible light image layer and infrared image layer of the dual-spectrum panoramic image using a preset edge detection operator. The visible light feature matrix is ​​generated by identifying the edge distribution and color distribution information of each pixel region in the visible light image layer, and the infrared feature matrix is ​​generated by extracting the thermal radiation brightness distribution information of each pixel region in the infrared image layer. The risk probability value of each coordinate position in the probability matrix is ​​multiplied by the change coefficient of the corresponding coordinate position in the semantic mask at the corresponding pixel position to obtain the risk weight matrix. The first matrix is ​​obtained by multiplying the feature value of each channel in the infrared feature matrix with the value at the corresponding coordinate position in the probability matrix. The second matrix is ​​obtained by multiplying the feature value of each channel in the visible light feature matrix with the value at the corresponding coordinate position in the risk weight matrix.

7. The method according to claim 1, characterized in that, The intensity threshold is determined based on the preset percentile value in the statistical distribution of pixel values ​​in the differential intensity distribution map of historical monitoring periods. The step of calculating the pixel differences between the first and second matrices and the historical first and second matrices from historical periods to obtain a difference intensity distribution map includes: The first matrix of the current monitoring period is spatially mapped and aligned with the historical first matrix of the historical period, and the values ​​of the mapped and aligned pixels are subtracted to obtain the first pixel difference matrix. The second matrix of the current monitoring period is spatially mapped and aligned with the historical second matrix of the historical period, and the values ​​of the mapped and aligned pixels are subtracted to obtain the second pixel difference matrix. The values ​​of the first pixel difference matrix and the second pixel difference matrix at the same pixel coordinates are extracted respectively, and then weighted and summed using a weight ratio preset based on monitoring requirements to obtain the difference distribution map.

8. A panoramic fusion and AI-based change monitoring system based on video images, characterized in that, include: The acquisition module is used to acquire road network vector data, land category data, and land violation records of the target area, as well as visible light video and infrared video of the target area collected by the PTZ camera, and the attitude data of the PTZ camera. The generation module is used to resample the visible light video and the infrared video using the attitude data, and then perform feature point matching and image fusion to generate a dual-spectrum panoramic image. The generation module is also used to construct a topology map with land parcels as nodes and roads as edges based on the road network vector data, associate the land parcel violation records with the corresponding nodes, input the topology map into the risk probability value of the output node of the trained spatiotemporal graph neural network, and map the risk probability value to the bispectral panoramic image generation probability matrix. The generation module is also used to generate a semantic mask with the same resolution as the bispectral panoramic image by means of projection transformation based on the land category data, wherein the element values ​​of the semantic mask are variation coefficients set according to the land category; The computation module is used to perform element-wise multiplication of the infrared feature matrix extracted from the dual-spectrum panoramic image with the probability matrix to obtain a first matrix, and to perform multiplication of the visible light feature matrix extracted from the dual-spectrum panoramic image with the probability matrix and the semantic mask to obtain a second matrix. The determination module is used to calculate the pixel differences between the first matrix and the second matrix and the historical first matrix and the historical second matrix of the historical period to obtain a difference intensity distribution map, and to determine the set of pixels in the difference intensity distribution map whose pixel values ​​are greater than a preset intensity threshold as the target change area.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the panoramic fusion and AI change monitoring method based on video images as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the panoramic fusion and AI change monitoring method based on video images as described in any one of claims 1 to 7.

Images