Regional panoramic monitoring method and device

Abstract

This application discloses a method and apparatus for regional panoramic monitoring. The method includes: acquiring an image set composed of monitoring images collected by multiple image acquisition devices within a target area; repeatedly executing the following image fusion process until multiple monitoring images are fused into a single panoramic image: randomly selecting two images from the image set, one as a reference image and the other as an image to be registered; extracting feature points and descriptors from the reference image and the image to be registered; enhancing the descriptors twice using a feature enhancement network; matching feature points based on the enhanced descriptors, and registering the images based on the matching results; fusing the reference image and the registered image to obtain a fused image, which is then returned to the image set; and analyzing the panoramic image using a target detection algorithm to identify monitoring images within the target area. This application solves the technical problem in related video surveillance schemes where it is difficult to efficiently and accurately stitch panoramic images together, resulting in unsatisfactory monitoring effects.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to a method and apparatus for regional panoramic monitoring. Background Technology

[0002] With the acceleration of urbanization and the advancement of smart city construction, urban management departments have an increasingly urgent need for real-time monitoring and management of urban environments, roads, traffic, public facilities, natural resources, and the environment. Video surveillance systems, as an important means of urban governance informatization, have been widely used, enabling urban managers to promptly detect incidents and handle emergencies. The massive volume, multi-modal nature, and strong correlation of video data make it possible to utilize video data for urban governance. Therefore, constructing a video twin system to integrate video data with GIS (Geographic Information System) three-dimensional spatial data is of significant value and importance. A video twin system can merge GIS scenes with video scenes, achieving real-time interaction between virtual and real spaces. It provides a visualized, interactive, and analyzable decision support platform for urban governance. It is a product of combining GIS technology and video surveillance technology, enabling seamless integration of virtual and real spaces and providing effective decision support for urban governance. Video twin systems can meet the needs of urban management in areas such as real-time monitoring, event decision-making, spatial prediction, environmental governance, smart transportation, intelligent security, and resource management, making it a key technology for urban governance.

[0003] Currently, video twin systems mainly consist of five steps: video stream acquisition, feature matching, image fusion, target detection, and 3D GIS fusion. The video stream acquisition module needs to acquire real-time video streams to meet the real-time monitoring needs of urban governance; the feature matching module needs to register images to achieve real-time fusion of video streams; the image fusion module needs to achieve real-time fusion of video streams so that urban managers can observe the urban environment anytime, anywhere; the target detection module needs to perform real-time target detection to enable urban managers to monitor events in real time; and the 3D GIS fusion module needs to fuse the video stream with the GIS 3D scene in real time to achieve seamless integration between virtual and real spaces. However, existing video twin systems have the following shortcomings: 1. They do not support the acquisition of video streams from a large number of cameras, failing to meet the real-time monitoring and event monitoring needs of urban governance; 2. They do not support real-time processing of image registration, image fusion, and target detection, failing to meet the real-time processing needs of urban governance and hindering timely event detection and response; 3. They do not support real-time fusion of video streams with the GIS 3D scene, failing to meet the needs of urban governance for seamless integration between virtual and real spaces.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a method and apparatus for regional panoramic monitoring, which at least solves the technical problem that it is difficult to perform panoramic image stitching efficiently and accurately in related video surveillance schemes, resulting in unsatisfactory monitoring effects.

[0006] According to one aspect of the embodiments of this application, a method for panoramic monitoring of a region is provided, comprising: acquiring an image set composed of multiple monitoring images synchronously acquired by multiple image acquisition devices within a target area; repeatedly executing the following image fusion process until the multiple monitoring images are fused into a single panoramic image: randomly selecting two images from the image set, one as a reference image and the other as an image to be registered; extracting multiple first feature points from the reference image and a first descriptor corresponding to each first feature point, and multiple second feature points from the image to be registered and a second descriptor corresponding to each second feature point; and using a feature enhancement network to enhance the representation of each first descriptor to obtain multiple third descriptors. Each second descriptor is augmented to obtain multiple fourth descriptors. The feature augmentation network first augments the descriptor of each feature point based on its own geometric information, and then augments it a second time based on the correlation information between multiple feature points. Multiple first and second feature points are matched using the third and fourth descriptors, and the images to be registered are registered based on the matching results to obtain a registered image. The reference image and the registered image are fused to obtain a fused image, which is then placed back into the image set. Finally, a target detection algorithm is used to analyze the panoramic image to identify the target monitoring objects within the target area.

[0007] Optionally, extracting multiple first feature points of the reference image and a first descriptor corresponding to each first feature point, and multiple second feature points of the image to be registered and a second descriptor corresponding to each second feature point, includes: extracting multiple first feature points in the reference image using a fast orientation and rotation-invariant feature algorithm, and determining a first descriptor corresponding to each first feature point; extracting multiple second feature points in the image to be registered, and determining a second descriptor corresponding to each second feature point.

[0008] Optionally, the feature enhancement network includes a self-enhancing subnetwork and a cross-enhancing subnetwork. The self-enhancing subnetwork includes two multilayer perceptrons, and the cross-enhancing subnetwork includes a transformer based on a multi-head attention mechanism. The feature enhancement network enhances the representation of each first descriptor to obtain multiple third descriptors, and enhances the representation of each second descriptor to obtain multiple fourth descriptors. This includes: inputting multiple first feature points and their corresponding first descriptors into the feature enhancement network to obtain a third descriptor corresponding to each first feature point. For each first feature point, the self-enhancing subnetwork encodes the geometric information of the first feature point using one multilayer perceptron to obtain a first vector, projects the first descriptor corresponding to the first feature point onto the target space using another multilayer perceptron to obtain a second vector, and concatenates the first and second vectors to form a third vector. The cross-enhancing subnetwork combines multiple third vectors into a first vector sequence and determines the first query matrix corresponding to the first vector sequence. The first key matrix and the first value matrix are used to calculate the third descriptor corresponding to each first feature point using an attention formula. Multiple second feature points and their corresponding second descriptors are input into a feature enhancement network to obtain a fourth descriptor corresponding to each second feature point. For each second feature point, a self-enhancing subnetwork encodes the geometric information of the second feature point using a multilayer perceptron to obtain a fourth vector. Another multilayer perceptron projects the second descriptor corresponding to the second feature point onto the target space to obtain a fifth vector. The fourth and fifth vectors are concatenated to form a sixth vector. A cross-enhancing subnetwork combines multiple sixth vectors to form a second vector sequence and determines the second query matrix, second key matrix, and second value matrix corresponding to the second vector sequence. An attention formula is then used to calculate the fourth descriptor corresponding to each second feature point.

[0009] Alternatively, the attention formula is: In the formula, Let N represent a vector sequence, where N is the number of vectors in the sequence. This represents the augmented descriptor corresponding to the i-th vector in the vector sequence. These represent the query matrix, key matrix, and value matrix corresponding to the vector sequence, respectively. This indicates the i-th row of the query matrix. These represent the j-th row of the key matrix and the value matrix, respectively.

[0010] Optionally, matching multiple first feature points and multiple second feature points based on the third and fourth descriptors includes: inputting multiple first feature points and their corresponding third descriptors, and multiple second feature points and their corresponding fourth descriptors into an improved SuperGlue model to obtain matching results between multiple first feature points and multiple second feature points; wherein, the improved SuperGlue model includes multi-layer matching sub-models with identical structures, each matching sub-model including: a feature encoding module, an attention module, and a classifier; the feature encoding module is used to encode the geometric information of the input feature points and their corresponding descriptors; the attention module includes: a self- The system consists of an attention module and a cross-attention module. The self-attention module enhances the encoding information of feature points based on the association information between feature points in the same image. The cross-attention module calculates the matching degree between feature points in different images based on the enhanced encoding information of each feature point. The classifier determines the matching result between feature points in different images based on the calculation result of the cross-attention module. The output of each layer of the self-attention module serves as the input of the next layer of the feature encoding module. When the self-attention module and the cross-attention module calculate for any two feature points, the feature point position information used is the relative position information between the two feature points.

[0011] Optionally, each classifier layer is used to determine the confidence level of the matching prediction result corresponding to each feature point of the current layer input, and to determine the feature points with a confidence level greater than the first preset threshold corresponding to the layer number of the classifier as reliable feature points, wherein the earlier the layer number of the classifier, the smaller the corresponding first preset threshold; to determine the ratio of the number of reliable feature points to the total number of input feature points; if the ratio is greater than the second preset threshold, to stop the iteration of the improved SuperGlue model and output the matching result of the current layer classifier; if the ratio is not greater than the second preset threshold, to determine the unnecessary feature points with a confidence level greater than the first preset threshold and a matching prediction result of no matching feature points, and to instruct the current layer self-attention module not to output the information of the unnecessary feature points to the next layer feature encoding module.

[0012] Optionally, the image to be registered is registered based on the matching result to obtain a registered image, including: analyzing the position information of multiple pairs of matched first and second feature points in the matching result using a direct linear transformation algorithm to obtain the global homography matrix corresponding to the image to be registered; dividing the image to be registered into multiple sub-images to be registered; for each sub-image to be registered, determining the centroid of the sub-image to be registered, and determining the weight coefficients corresponding to the sub-image to be registered based on the distance relationship between the centroid and each second feature point; determining the local homography matrix corresponding to the sub-image to be registered based on the weight coefficients and the global homography matrix; performing a linear transformation on the sub-image to be registered based on the local homography matrix to obtain the registered sub-image corresponding to the sub-image to be registered; and stitching multiple registered sub-images together to form a registered image.

[0013] Optionally, the reference image and the registered image are fused to obtain a fused image, including: determining the target stitching line when stitching the reference image and the registered image; dividing the reference image into layers according to multiple preset frequency bands to obtain a first Gaussian pyramid, and determining the first Laplacian pyramid corresponding to the first Gaussian pyramid; dividing the registered image into layers according to multiple preset frequency bands to obtain a second Gaussian pyramid, and determining the second Laplacian pyramid corresponding to the second Gaussian pyramid; fusing the same frequency band layers of the first Laplacian pyramid and the second Laplacian pyramid according to the target stitching line to obtain a fused Laplacian pyramid; and performing an inverse Laplacian transform on the fused Laplacian pyramid to obtain the fused image.

[0014] Optionally, determining the target stitching line when stitching the reference image and the registered image includes: determining a color energy function based on the color difference of the stitching line on the reference image and the registered image, and determining a structure energy function based on the gradient difference of the stitching line on the reference image and the registered image, wherein the stitching line is the variable to be solved, and the gradient difference is the combined gradient difference in the horizontal and vertical directions; determining the first weight and the second weight corresponding to the color energy function and the structure energy function respectively, and performing a weighted summation of the color energy function and the structure energy function based on the first weight and the second weight to obtain the target energy function; and solving for the target stitching line that minimizes the target energy function.

[0015] Optionally, the panoramic image is analyzed using a target detection algorithm to obtain the target monitoring objects within the target area, including: dividing the panoramic image into multiple local images; analyzing each local image using a target detection algorithm to obtain the monitoring objects in each local image; and using a non-maximum suppression algorithm to remove duplicate monitoring objects from the multiple local images to obtain each target monitoring object within the target area.

[0016] Optionally, the method further includes: acquiring a 3D scene model and geographic information corresponding to the target area; fusing the panoramic image, the 3D scene model, and the geographic information to obtain a panoramic monitoring model corresponding to the target area; and highlighting each target monitoring object in the panoramic monitoring model.

[0017] According to another aspect of the embodiments of this application, a regional panoramic monitoring device is also provided, comprising: an acquisition module, configured to acquire an image set composed of multiple monitoring images synchronously acquired by multiple image acquisition devices within a target area; and an image fusion module, configured to repeatedly execute the following image fusion process until the multiple monitoring images are fused into a single panoramic image: randomly selecting two images from the image set, one as a reference image and the other as an image to be registered; extracting multiple first feature points of the reference image and a first descriptor corresponding to each first feature point, and multiple second feature points of the image to be registered and a second descriptor corresponding to each second feature point; and using a feature enhancement network to enhance the representation of each first descriptor to obtain multiple third feature points. The descriptor is enhanced for each second descriptor to obtain multiple fourth descriptors. The feature enhancement network first enhances the descriptor of each feature point based on its own geometric information, and then enhances it a second time based on the correlation information between multiple feature points. Multiple first and second feature points are matched using the third and fourth descriptors, and the image to be registered is registered based on the matching results to obtain a registered image. The reference image and the registered image are fused to obtain a fused image, which is then placed back into the image set. The object detection module analyzes the panoramic image using an object detection algorithm to identify target monitoring objects within the target area.

[0018] According to another aspect of the embodiments of this application, a computer program product is also provided, the computer program product comprising: a computer program, wherein the computer program, when executed by a processor, implements the above-described regional panoramic monitoring method.

[0019] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described regional panoramic monitoring method through the computer program.

[0020] In this embodiment, an image set is formed by acquiring multiple monitoring images simultaneously acquired by multiple image acquisition devices within a target area; the following image fusion process is executed iteratively until the multiple monitoring images are fused into a single panoramic image: two images are randomly selected from the image set, one as a reference image and the other as the image to be registered; multiple first feature points of the reference image and the first descriptor corresponding to each first feature point, and multiple second feature points of the image to be registered and the second descriptor corresponding to each second feature point are extracted; each first descriptor is enhanced using a feature enhancement network to obtain multiple third descriptors, and each second descriptor is further enhanced using a feature enhancement network. The enhancement process involves obtaining multiple fourth descriptors. The feature enhancement network first enhances the descriptor of each feature point based on its own geometric information, and then enhances it a second time based on the correlation information between multiple feature points. Multiple first and second feature points are matched using the third and fourth descriptors, and the image to be registered is registered based on the matching results, resulting in a registered image. The reference image and the registered image are then fused to obtain a fused image, which is then placed back into the image set. Finally, a target detection algorithm is used to analyze the panoramic image to identify the target monitoring object within the target area. In this way, a feature enhancement network is introduced into image fusion to perform secondary enhancement on the feature points and descriptors extracted by directional fast and rotation-invariant feature algorithms, thereby achieving fast and accurate feature matching with limited computing resources. In addition, by using local homography matrix and grid partitioning techniques, combined with weight coefficient optimization, the registration problem of large parallax and image overlap regions is effectively handled, eliminating stitching artifacts and ghosting. Furthermore, in the large-size image target detection module, the strategy of first segmenting, then integrating, and then deduplicating enhances the detection capability of small targets and reduces the false negative rate. This solves the technical problem in related video surveillance solutions that it is difficult to perform panoramic image stitching efficiently and accurately, resulting in unsatisfactory monitoring effects. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0022] Figure 1 This is a flowchart illustrating an optional regional panoramic monitoring method according to an embodiment of this application;

[0023] Figure 2 This is a schematic diagram of the structure of an optional regional panoramic monitoring device according to an embodiment of this application;

[0024] Figure 3 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] Example 1

[0028] According to an embodiment of this application, a method for panoramic monitoring of a region is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] Figure 1 This is a flowchart illustrating a regional panoramic monitoring method according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps:

[0030] Step S102: Obtain an image set composed of multiple monitoring images synchronously acquired by multiple image acquisition devices within the target area;

[0031] Step S104: Repeat the following image fusion process until multiple surveillance images are fused into a single panoramic image: Randomly select two images from the image set, one as the reference image and the other as the image to be registered; Extract multiple first feature points and the first descriptor corresponding to each first feature point from the reference image, and multiple second feature points and the second descriptor corresponding to each second feature point from the image to be registered; Enhance each first descriptor using a feature enhancement network to obtain multiple third descriptors, and enhance each second descriptor to obtain multiple fourth descriptors. The feature enhancement network is used to enhance the descriptor of each feature point based on its own geometric information and to enhance the descriptor of each feature point a second time based on the correlation information between multiple feature points; Match multiple first feature points and multiple second feature points based on the third and fourth descriptors, and register the image to be registered based on the matching results to obtain a registered image; Fuse the reference image and the registered image to obtain a fused image, and put the fused image back into the image set.

[0032] Step S106: Analyze the panoramic image using a target detection algorithm to obtain the target monitoring object within the target area.

[0033] The following section explains each step of the regional panoramic monitoring method in conjunction with the specific implementation process.

[0034] First, an image set is formed by acquiring multiple monitoring images simultaneously collected by multiple image acquisition devices within the target area.

[0035] The following image fusion process is executed iteratively until multiple surveillance images are fused into a single panoramic image: Two images are randomly selected from the image set, one as the reference image and the other as the image to be registered; multiple first feature points and their corresponding first descriptors are extracted from the reference image, and multiple second feature points and their corresponding second descriptors are extracted from the image to be registered; each first descriptor is enhanced using a feature enhancement network to obtain multiple third descriptors, and each second descriptor is enhanced to obtain multiple fourth descriptors. The feature enhancement network performs a first enhancement on the descriptor of each feature point based on its own geometric information, and a second enhancement on the descriptor based on the correlation information between multiple feature points; multiple first and second feature points are matched based on the third and fourth descriptors, and the image to be registered is registered based on the matching results to obtain a registered image; the reference image and the registered image are fused to obtain a fused image, which is then returned to the image set.

[0036] For example, the number of cameras is a power of 2, with a maximum of 16 channels. Each camera acquires video frames synchronously. Two adjacent cameras form a group. In each group of image data, one image is used as a reference image and the other is used as the image to be registered.

[0037] Optionally, the extraction of multiple first feature points from the reference image and the first descriptor corresponding to each first feature point, and multiple second feature points from the image to be registered and the second descriptor corresponding to each second feature point, can be performed in the following manner: the Oriented Fast and Rotated BRIEF (ORB) algorithm is used to extract multiple first feature points from the reference image and determine the first descriptor corresponding to each first feature point; the algorithm is also used to extract multiple second feature points from the image to be registered and determine the second descriptor corresponding to each second feature point.

[0038] The ORB algorithm is used to find multiple key points from the reference image. These key points are called "first feature points". These points are usually located in the image with features such as corners and edges. Because the local structure of the image at these locations has high descriptiveness and discriminativeness, it is suitable for subsequent feature matching. For each detected first feature point, the algorithm calculates a set of numerical vectors, namely the first descriptor, to describe the image features around the point. The ORB algorithm uses the BRIEF descriptor, which is a binary descriptor. By comparing the intensity of pixels in the neighborhood of the feature point, a series of binary strings of 0s and 1s are obtained to represent the local features of the feature point.

[0039] This application introduces a feature enhancement network based on the traditional directional fast and rotation-invariant feature algorithms. The idea of ​​the feature enhancement network is to optimize through two stages, self-enhancement and cross-enhancement, without requiring the original image to participate in the calculation.

[0040] Specifically, the feature enhancement network is used to enhance the representation of each first descriptor to obtain multiple third descriptors, and the feature enhancement network is used to enhance the representation of each second descriptor to obtain multiple fourth descriptors. The feature enhancement network is used to enhance the descriptor of each feature point for the first time based on the geometric information of each feature point itself, and to enhance the descriptor of each feature point for the second time based on the association information between multiple feature points.

[0041] Optionally, the feature enhancement network includes a self-enhancing subnetwork and a cross-enhancing subnetwork. The self-enhancing subnetwork includes two multilayer perceptrons, and the cross-enhancing subnetwork includes a transformer based on a multi-head attention mechanism.

[0042] The process involves using a feature enhancement network to enhance the representation of each first descriptor, resulting in multiple third descriptors, and enhancing the representation of each second descriptor, resulting in multiple fourth descriptors. The steps include: inputting multiple first feature points and their corresponding first descriptors into the feature enhancement network to obtain a third descriptor for each first feature point. For each first feature point, a self-enhancing sub-network encodes the geometric information of the first feature point using a multilayer perceptron to obtain a first vector; another multilayer perceptron projects the first descriptor corresponding to the first feature point onto the target space to obtain a second vector; and concatenates the first and second vectors to form a third vector. A cross-enhancing sub-network combines multiple third vectors into a first vector sequence and determines the first query matrix, first key matrix, and first value matrix corresponding to the first vector sequence. An attention formula is then used to calculate the first vector sequence, first query matrix, first key matrix, and first value matrix to obtain a third descriptor for each first feature point. The attention formula is:

[0043]

[0044] In the formula, Let N represent a vector sequence, where N is the number of vectors in the sequence. This represents the augmented descriptor corresponding to the i-th vector in the vector sequence. These represent the query matrix, key matrix, and value matrix corresponding to the vector sequence, respectively. This indicates the i-th row of the query matrix. These represent the j-th row of the key matrix and the value matrix, respectively.

[0045] The obtained second feature points and their corresponding second descriptors are input into the feature enhancement network to obtain the fourth descriptor for each second feature point. For each second feature point, the self-enhancing sub-network encodes the geometric information of the second feature point using a multilayer perceptron to obtain the fourth vector. Another multilayer perceptron projects the second descriptor corresponding to the second feature point onto the target space to obtain the fifth vector. The fourth and fifth vectors are concatenated to form the sixth vector. The cross-enhancing sub-network combines multiple sixth vectors into a second vector sequence and determines the second query matrix, second key matrix, and second value matrix corresponding to the second vector sequence. The attention formula is used to calculate the second vector sequence, second query matrix, second key matrix, and second value matrix to obtain the fourth descriptor for each second feature point.

[0046] The above process combines the efficiency of traditional directional fast and rotation-invariant feature algorithms with the accuracy of feature enhancement networks, enabling real-time image registration on embedded GPU (Graphics Processing Unit) devices. Even in complex scenes, such as images with fast-moving objects causing blurring or dragging marks, it can maintain good matching results.

[0047] After feature point detection and descriptor enhancement, image matching can be performed. Image matching is done by comparing the descriptors of key points in two images, with the goal of determining which points correspond to the same real-world locations in the two images. Traditional matching algorithms, such as Nearest Neighbor Matching (NNM), are a basic feature matching method. They determine the correspondence between feature points in two images by calculating the distance between descriptors (such as Euclidean distance or Hamming distance). However, the simplicity of NNM also brings problems; it may generate a large number of incorrect matching points, especially when there are similar local features in the images, which reduces the accuracy of the matching. The RANSAC (Random Sample Consensus) algorithm estimates the matching model by randomly selecting the smallest dataset (i.e., point pairs), and then uses the entire dataset to verify the accuracy of the model. Through an iterative process, it filters out inliers to eliminate incorrect matching point pairs, thereby improving the accuracy of the matching. However, the RANSAC algorithm is inefficient, requiring a large number of iterations to obtain a good model estimate. It is also sensitive to the setting of model parameters (such as the inlier threshold) and may not yield a globally optimal model estimate. The SuperGlue algorithm, as an end-to-end deep learning solution, learns feature point matching and inlier filtering simultaneously through a neural network. Compared to traditional methods, it can more accurately match two sets of local features and eliminate those that cannot be matched, exhibiting stronger robustness and generalization ability. SuperGlue is based on the Transformer architecture and can handle complex matching tasks. However, its drawbacks include the need for a large amount of training data and computational resources, and the computational complexity increases quadratically with the number of keypoints. Therefore, it can only achieve real-time matching results when running on a desktop GPU.

[0048] To address the limitations of the SuperGlue algorithm, this application proposes two key improvements: First, it replaces absolute encoding with relative encoding. The traditional SuperGlue algorithm uses absolute point location encoding, while this application improves upon it by using relative location encoding. This method considers that relative location information is more important than absolute location information during the matching process because relative positions have better consistency across different images, while absolute positions may be affected by variables such as camera position and viewpoint. Second, it controls accuracy and inference time by assigning confidence scores and thresholds to predictions. At the end of each network layer, the improved method calculates the prediction score confidence score for each feature point. A higher confidence score indicates a reliable matching result for that feature point. By setting a threshold, the network inference process can be stopped early, reducing unnecessary computation. Furthermore, when feature points are predicted to be unmatchable, they are early discarded, retaining only the remaining feature points for subsequent computation. This significantly reduces computational load without affecting the final matching accuracy.

[0049] The matching process using the improved SuperGlue model is as follows:

[0050] Matching multiple first feature points and multiple second feature points based on the third and fourth descriptors, and registering the image to be registered based on the matching results, can be done in the following way: input multiple first feature points and their corresponding third descriptors, multiple second feature points and their corresponding fourth descriptors into the improved SuperGlue model to obtain the matching results between multiple first feature points and multiple second feature points.

[0051] The improved SuperGlue model includes multiple matching sub-models with identical structures. Each matching sub-model includes a feature encoding module, an attention module, and a classifier. The feature encoding module encodes the geometric information of the input feature points and their corresponding descriptors. The attention module includes a self-attention module and a cross-attention module. The self-attention module enhances the encoded information of the feature points based on the association information between feature points in the same image, while the cross-attention module calculates the matching degree between feature points in different images based on the enhanced encoded information of each feature point. The classifier determines the matching result between feature points in different images based on the calculation result of the cross-attention module. The output of each self-attention module serves as the input to the next layer's feature encoding module. When calculating any two feature points, the self-attention module and the cross-attention module use the relative position information between the two feature points.

[0052] For example, according to SuperGlue's method, partial allocation is predicted between two sets of local features extracted from images A and B, with each local feature... The position of a 2D (two-dimensional) point after image size normalization. and a visual descriptor The images A and B are composed of M and N local features, respectively, and their index sets are respectively... and A partial distribution matrix is ​​obtained between the local features of A and B, and the correspondence is extracted from it.

[0053] Image Each local feature in With a state The association is based on the corresponding visual descriptor. Initialization is performed, followed by updates for each layer, which consists of a continuous self-attention module and a cross-attention module.

[0054] For the self-attention module, each point focuses on all points in the same image, performing the same steps for each image I, therefore the superscript I is omitted. For each point... Current state Decomposed into key vectors through different linear transformations and query vector ,point and points The attention scores between them are:

[0055]

[0056] in, It is a rotational encoding of the relative positions between points, dividing the space into Each subspace is divided into two dimensions, and each subspace is rotated by an angle based on the Fourier features, the angle being related to the learned basis vectors. The projection on corresponds to:

[0057]

[0058] Positional encoding plays a crucial role in attention mechanisms, allowing different elements to be processed based on their location. In projection camera geometry, the position of a visual observation is equivalent to the camera's translation within the image plane: 2D points from a 3D point translate in the same way on the same parallel plane, their relative distances remaining constant. Encoding only needs to capture the relative position of points, not their absolute position. Rotational encoding enables the model to... Retrieved points located at the learned relative positions. Position coding should not be applied to numerical values. Therefore, it will not affect the state. The encoding is the same for all layers, so it only needs to be computed once and cached.

[0059] Optionally, each classifier layer is used to determine the confidence level of the matching prediction result corresponding to each feature point of the current layer input, and to determine the feature points with a confidence level greater than the first preset threshold corresponding to the layer number of the classifier as reliable feature points, wherein the earlier the layer number of the classifier, the smaller the corresponding first preset threshold; to determine the ratio of the number of reliable feature points to the total number of input feature points; if the ratio is greater than the second preset threshold, to stop the iteration of the improved SuperGlue model and output the matching result of the current layer classifier; if the ratio is not greater than the second preset threshold, to determine the unnecessary feature points with a confidence level greater than the first preset threshold and a matching prediction result of no matching feature points, and to instruct the current layer self-attention module not to output the information of the unnecessary feature points to the next layer feature encoding module.

[0060] For example, the backbone of the feature enhancement network enhances the input visual descriptors through context, which helps improve the quality of feature point descriptors and makes feature matching more reliable. At the end of each layer, the network attempts to predict feature points and evaluates the confidence of these predictions using the following formula:

[0061]

[0062] In the formula, This represents the prediction confidence level for each point. Indicates the first A higher confidence value in the feature vector of a given point indicates... The representation is a reliable final result, meaning that the point either matches successfully or is determined not to match.

[0063] For a given layer, if the confidence level of a point is greater than a threshold, it is considered reliable. If, after processing a certain layer, a sufficient proportion of feature points have been confirmed as reliable matches, the inference stops. The specific calculation formula is as follows:

[0064]

[0065] In the formula, and These represent the number of points in the two images, respectively. Let A be a set representing two images A and B. In image I, the first... Confidence level at each point This represents the first preset threshold. This indicates that the confidence level in either of the two images exceeds a threshold. The proportion is greater than the preset second preset threshold. ,but If true, it means that the reasoning process can be terminated early.

[0066] It should be noted that the classifier's confidence level is inherently low in the early layers. Therefore, as the number of layers increases, the classifier's confidence level may decrease. Thus, it's advisable to gradually lower the threshold. To adapt to changes in confidence levels across different layers, if a point is predicted as reliable but cannot be matched in earlier layers, it is no longer considered in subsequent layers because it is typically located in a clearly non-co-existent region of the image. This reduces unnecessary computation and improves efficiency. Although inference is terminated early, high matching accuracy is still maintained due to the confidence assessment and threshold adjustment mechanisms.

[0067] The process involves registering the image to be registered based on the matching results, resulting in a registered image. This includes: analyzing the positional information of multiple pairs of matched first and second feature points in the matching results using a direct linear transformation algorithm to obtain the global homography matrix corresponding to the image to be registered; dividing the image to be registered into multiple sub-images; for each sub-image, determining its centroid and weighting coefficients based on the distance between the centroid and each second feature point; determining the local homography matrix based on the weighting coefficients and the global homography matrix; performing a linear transformation on the sub-image based on the local homography matrix to obtain the registered sub-image; and stitching the multiple registered sub-images together to form the registered image.

[0068] For example, suppose a reference image I and an image to be registered. The corresponding matching point is , The transformation relationship is as follows , These are the homogeneous coordinates of X. yes homogeneous coordinates, matrix The column vector is , , , can be represented as:

[0069]

[0070] The homogeneous equation for the projection transformation is as follows:

[0071]

[0072] For two images with N sets of matching points, the objective function for calculating the global homography matrix H using direct linear transformation is:

[0073]

[0074] The image to be registered is divided into x There are 1 grid, and each grid corresponds to a homography matrix. By introducing weighting coefficients Determine the local homography matrix The center point of the grid is The matching point is The local homography matrix of any grid is:

[0075]

[0076]

[0077]

[0078] Traverse each grid and perform a linear transformation on the sub-image to be registered based on the local homography matrix to obtain the registered sub-image corresponding to the sub-image to be registered. Then, stitch multiple registered sub-images together to form a registered image.

[0079] The fusion of the reference image and the registered image to obtain the fused image can be performed as follows: First, determine the target stitching line for stitching the reference image and the registered image; second, divide the reference image into layers according to multiple preset frequency bands to obtain a first Gaussian pyramid, and determine the corresponding first Laplacian pyramid; third, divide the registered image into layers according to multiple preset frequency bands to obtain a second Gaussian pyramid, and determine the corresponding second Laplacian pyramid; fourth, fuse the layers of the same frequency bands of the first and second Laplacian pyramids according to the target stitching line to obtain a fused Laplacian pyramid; fifth, perform an inverse Laplacian transform on the fused Laplacian pyramid to obtain the fused image.

[0080] Determining the target stitching line when stitching the reference image and the registered image includes the following steps: determining the color energy function based on the color difference of the stitching line on the reference image and the registered image, and determining the structure energy function based on the gradient difference of the stitching line on the reference image and the registered image, wherein the stitching line is the variable to be solved, and the gradient difference is the combined gradient difference in the horizontal and vertical directions; determining the first weight and the second weight corresponding to the color energy function and the structure energy function respectively, and performing a weighted summation of the color energy function and the structure energy function based on the first weight and the second weight to obtain the target energy function; and solving for the target stitching line that minimizes the target energy function.

[0081] First, the optimal splicing line is found by minimizing the energy function. Taking into account both color and structure, the calculation formula is as follows:

[0082]

[0083]

[0084] In the formula, Indicates a reference image. This represents the image to be registered. This represents the color difference at point X in the overlapping area. This represents the gradient difference between the horizontal and vertical directions at the seam point X in the overlapping region. Represents the Sobel operator and its energy function. Weight By applying the idea of ​​maximum flow and minimum cut, the energy function is minimized, thereby finding the optimal splicing line.

[0085] The panoramic image is analyzed using a target detection algorithm to obtain the target monitoring objects within the target area. The steps are as follows: Divide the panoramic image into multiple local images; analyze each local image using a target detection algorithm to obtain the monitoring objects in each local image; use a non-maximum suppression algorithm to remove duplicate monitoring objects from the multiple local images to obtain each target monitoring object within the target area.

[0086] For example, the fused image is divided into four local images, and each local image is used for target detection by an improved YOLOv5m model, which adds three smaller anchor boxes for small targets. After detection, the results of all local images are integrated on the original image, and the non-maximum suppression algorithm is used to remove duplicate detection boxes to obtain all target monitoring objects within the target area.

[0087] In addition to the method in this embodiment, a three-dimensional scene model and geographic information corresponding to the target area can also be obtained; the panoramic image, the three-dimensional scene model and the geographic information are fused to obtain a panoramic monitoring model corresponding to the target area; and each target monitoring object is highlighted in the panoramic monitoring model.

[0088] For example, it's necessary to collect detailed geographic information about a target area (such as an industrial park), including terrain, the 3D structure of buildings, roads, etc. This information can be obtained based on high-precision GIS data or through technologies such as LiDAR scanning and drone aerial photography to create a detailed 3D digital model. Besides the 3D model, it's also necessary to acquire information such as geographic coordinates, orientation, and altitude related to the model. This information is used to accurately align the 3D model with the actual geographic environment, ensuring a realistic and reliable fusion of video and model. After image registration and fusion, a high-quality panoramic image is obtained, stitched together from videos from multiple cameras, covering the entire field of view of the park. Subsequently, using the intrinsic and extrinsic parameters of the cameras, the model-view matrix and projection matrix in 3D space are calculated. These matrices describe the spatial relationship between the panoramic image and the 3D model. Using the model-view matrix and projection matrix, and through projection texture technology, each frame in the panoramic video is reprojected onto the corresponding position in the GIS 3D model. A key step here is using GPU-accelerated video and model rendering to update texture data in real time, ensuring that the video content on the 3D model is synchronized with the actual scene. The Graph (Open Source 3D Graphics, OSG) engine is an open-source, high-performance 3D graphics engine that can efficiently handle complex 3D scenes and supports real-time rendering and scene management. Through this engine, a 3D GIS model that integrates panoramic video can be presented, realizing intuitive visualization of the 3D scene model and geographic information of the park. This process realizes the real-time dynamic fusion of panoramic video and 3D geographic information model, ultimately resulting in an intuitive, 3D-based panoramic monitoring model of the target area.

[0089] In this embodiment, an image set is formed by acquiring multiple monitoring images simultaneously acquired by multiple image acquisition devices within a target area; the following image fusion process is executed iteratively until the multiple monitoring images are fused into a single panoramic image: two images are randomly selected from the image set, one as a reference image and the other as the image to be registered; multiple first feature points of the reference image and the first descriptor corresponding to each first feature point, and multiple second feature points of the image to be registered and the second descriptor corresponding to each second feature point are extracted; each first descriptor is enhanced using a feature enhancement network to obtain multiple third descriptors, and each second descriptor is further enhanced using a feature enhancement network. The enhancement process involves obtaining multiple fourth descriptors. The feature enhancement network first enhances the descriptor of each feature point based on its own geometric information, and then enhances it a second time based on the correlation information between multiple feature points. Multiple first and second feature points are matched using the third and fourth descriptors, and the image to be registered is registered based on the matching results, resulting in a registered image. The reference image and the registered image are then fused to obtain a fused image, which is then placed back into the image set. Finally, a target detection algorithm is used to analyze the panoramic image to identify the target monitoring object within the target area. In this way, by introducing a feature enhancement network into image registration, the feature points and descriptors extracted by directional fast and rotation-invariant feature algorithms are enhanced a second time, thereby achieving fast and accurate feature matching with limited computing resources. In addition, by using local homography matrix and grid partitioning techniques, combined with weight coefficient optimization, the registration problems of large parallax and image overlap regions are effectively handled, eliminating stitching artifacts and ghosting. Furthermore, in the large-size image target detection module, the strategy of first segmenting, then integrating, and then deduplicating enhances the detection capability of small targets and reduces the false negative rate. This solves the technical problem in related video surveillance solutions that it is difficult to perform panoramic image stitching efficiently and accurately, resulting in unsatisfactory monitoring effects.

[0090] Example 2

[0091] According to an embodiment of this application, a regional panoramic monitoring device for implementing the regional panoramic monitoring method in Embodiment 1 is also provided, such as... Figure 2 As shown, the panoramic monitoring device for this area includes at least: an acquisition module 21, an image fusion module 22, and an object detection module 23, wherein:

[0092] The acquisition module 21 is used to acquire an image set composed of multiple monitoring images synchronously acquired by multiple image acquisition devices within the target area;

[0093] Image fusion module 22 is used to repeatedly execute the following image fusion process until multiple monitoring images are fused into a single panoramic image: Two images are randomly selected from the image set, one as the reference image and the other as the image to be registered; multiple first feature points and their corresponding first descriptors are extracted from the reference image, and multiple second feature points and their corresponding second descriptors are extracted from the image to be registered; each first descriptor is enhanced using a feature enhancement network to obtain multiple third descriptors, and each second descriptor is enhanced to obtain multiple fourth descriptors. The feature enhancement network enhances the descriptor of each feature point based on its own geometric information and enhances it a second time based on the correlation information between multiple feature points; multiple first and second feature points are matched using the third and fourth descriptors, and the image to be registered is registered based on the matching results to obtain a registered image; the reference image and the registered image are fused to obtain a fused image, and the fused image is returned to the image set.

[0094] The object detection module 23 is used to analyze the panoramic image using the object detection algorithm to obtain the target monitoring object within the target area.

[0095] The following section explains the functions of each module of the regional panoramic monitoring device in conjunction with the specific implementation process.

[0096] First, the acquisition module acquires a set of multiple monitoring images simultaneously acquired by multiple image acquisition devices within the target area.

[0097] The image fusion module iteratively executes the following image fusion process until multiple surveillance images are fused into a single panoramic image: Two images are randomly selected from the image set, one as the reference image and the other as the image to be registered; multiple first feature points and their corresponding first descriptors are extracted from the reference image, and multiple second feature points and their corresponding second descriptors are extracted from the image to be registered; each first descriptor is enhanced using a feature enhancement network to obtain multiple third descriptors, and each second descriptor is enhanced to obtain multiple fourth descriptors. The feature enhancement network performs a first enhancement on the descriptor of each feature point based on its own geometric information, and a second enhancement on the descriptor based on the correlation information between multiple feature points; multiple first and second feature points are matched based on the third and fourth descriptors, and the image to be registered is registered based on the matching results to obtain a registered image; the reference image and the registered image are fused to obtain a fused image, which is then returned to the image set.

[0098] Optionally, extracting multiple first feature points of the reference image and a first descriptor corresponding to each first feature point, and multiple second feature points of the image to be registered and a second descriptor corresponding to each second feature point, includes: extracting multiple first feature points in the reference image using a fast orientation and rotation-invariant feature algorithm, and determining a first descriptor corresponding to each first feature point; extracting multiple second feature points in the image to be registered, and determining a second descriptor corresponding to each second feature point.

[0099] Optionally, the feature enhancement network includes a self-enhancing subnetwork and a cross-enhancing subnetwork. The self-enhancing subnetwork includes two multilayer perceptrons, and the cross-enhancing subnetwork includes a transformer based on a multi-head attention mechanism. The feature enhancement network enhances the representation of each first descriptor to obtain multiple third descriptors, and enhances the representation of each second descriptor to obtain multiple fourth descriptors. This includes: inputting multiple first feature points and their corresponding first descriptors into the feature enhancement network to obtain a third descriptor corresponding to each first feature point. For each first feature point, the self-enhancing subnetwork encodes the geometric information of the first feature point using one multilayer perceptron to obtain a first vector, projects the first descriptor corresponding to the first feature point onto the target space using another multilayer perceptron to obtain a second vector, and concatenates the first and second vectors to form a third vector. The cross-enhancing subnetwork combines multiple third vectors into a first vector sequence and determines the first query matrix corresponding to the first vector sequence. The first key matrix and the first value matrix are used to calculate the third descriptor corresponding to each first feature point using an attention formula. Multiple second feature points and their corresponding second descriptors are input into a feature enhancement network to obtain a fourth descriptor corresponding to each second feature point. For each second feature point, a self-enhancing subnetwork encodes the geometric information of the second feature point using a multilayer perceptron to obtain a fourth vector. Another multilayer perceptron projects the second descriptor corresponding to the second feature point onto the target space to obtain a fifth vector. The fourth and fifth vectors are concatenated to form a sixth vector. A cross-enhancing subnetwork combines multiple sixth vectors to form a second vector sequence and determines the second query matrix, second key matrix, and second value matrix corresponding to the second vector sequence. An attention formula is then used to calculate the fourth descriptor corresponding to each second feature point.

[0100] Alternatively, the attention formula is: In the formula, Let N represent a vector sequence, where N is the number of vectors in the sequence. This represents the augmented descriptor corresponding to the i-th vector in the vector sequence. These represent the query matrix, key matrix, and value matrix corresponding to the vector sequence, respectively. This indicates the i-th row of the query matrix. These represent the j-th row of the key matrix and the value matrix, respectively.

[0101] Optionally, matching multiple first feature points and multiple second feature points based on the third and fourth descriptors includes: inputting multiple first feature points and their corresponding third descriptors, and multiple second feature points and their corresponding fourth descriptors into an improved SuperGlue model to obtain matching results between multiple first feature points and multiple second feature points; wherein, the improved SuperGlue model includes multi-layer matching sub-models with identical structures, each matching sub-model including: a feature encoding module, an attention module, and a classifier; the feature encoding module is used to encode the geometric information of the input feature points and their corresponding descriptors; the attention module includes: a self- The system consists of an attention module and a cross-attention module. The self-attention module enhances the encoding information of feature points based on the association information between feature points in the same image. The cross-attention module calculates the matching degree between feature points in different images based on the enhanced encoding information of each feature point. The classifier determines the matching result between feature points in different images based on the calculation result of the cross-attention module. The output of each layer of the self-attention module serves as the input of the next layer of the feature encoding module. When the self-attention module and the cross-attention module calculate for any two feature points, the feature point position information used is the relative position information between the two feature points.

[0102] Optionally, each classifier layer is used to determine the confidence level of the matching prediction result corresponding to each feature point of the current layer input, and to determine the feature points with a confidence level greater than the first preset threshold corresponding to the layer number of the classifier as reliable feature points, wherein the earlier the layer number of the classifier, the smaller the corresponding first preset threshold; to determine the ratio of the number of reliable feature points to the total number of input feature points; if the ratio is greater than the second preset threshold, to stop the iteration of the improved SuperGlue model and output the matching result of the current layer classifier; if the ratio is not greater than the second preset threshold, to determine the unnecessary feature points with a confidence level greater than the first preset threshold and a matching prediction result of no matching feature points, and to instruct the current layer self-attention module not to output the information of the unnecessary feature points to the next layer feature encoding module.

[0103] Optionally, the image to be registered is registered based on the matching result to obtain a registered image, including: analyzing the position information of multiple pairs of matched first and second feature points in the matching result using a direct linear transformation algorithm to obtain the global homography matrix corresponding to the image to be registered; dividing the image to be registered into multiple sub-images to be registered; for each sub-image to be registered, determining the centroid of the sub-image to be registered, and determining the weight coefficients corresponding to the sub-image to be registered based on the distance relationship between the centroid and each second feature point; determining the local homography matrix corresponding to the sub-image to be registered based on the weight coefficients and the global homography matrix; performing a linear transformation on the sub-image to be registered based on the local homography matrix to obtain the registered sub-image corresponding to the sub-image to be registered; and stitching multiple registered sub-images together to form a registered image.

[0104] Optionally, the reference image and the registered image are fused to obtain a fused image, including: determining the target stitching line when stitching the reference image and the registered image; dividing the reference image into layers according to multiple preset frequency bands to obtain a first Gaussian pyramid, and determining the first Laplacian pyramid corresponding to the first Gaussian pyramid; dividing the registered image into layers according to multiple preset frequency bands to obtain a second Gaussian pyramid, and determining the second Laplacian pyramid corresponding to the second Gaussian pyramid; fusing the same frequency band layers of the first Laplacian pyramid and the second Laplacian pyramid according to the target stitching line to obtain a fused Laplacian pyramid; and performing an inverse Laplacian transform on the fused Laplacian pyramid to obtain the fused image.

[0105] Optionally, determining the target stitching line when stitching the reference image and the registered image includes: determining a color energy function based on the color difference of the stitching line on the reference image and the registered image, and determining a structure energy function based on the gradient difference of the stitching line on the reference image and the registered image, wherein the stitching line is the variable to be solved, and the gradient difference is the combined gradient difference in the horizontal and vertical directions; determining the first weight and the second weight corresponding to the color energy function and the structure energy function respectively, and performing a weighted summation of the color energy function and the structure energy function based on the first weight and the second weight to obtain the target energy function; and solving for the target stitching line that minimizes the target energy function.

[0106] Optionally, the object detection module analyzes the panoramic image using a target detection algorithm to obtain the target monitoring objects within the target area, including: dividing the panoramic image into multiple local images; analyzing each local image using a target detection algorithm to obtain the monitoring objects in each local image; and using a non-maximum suppression algorithm to remove duplicate monitoring objects from the multiple local images to obtain each target monitoring object within the target area.

[0107] Optionally, the method further includes: acquiring a 3D scene model and geographic information corresponding to the target area; fusing the panoramic image, the 3D scene model, and the geographic information to obtain a panoramic monitoring model corresponding to the target area; and highlighting each target monitoring object in the panoramic monitoring model.

[0108] It should be noted that each module in the regional panoramic monitoring device in this embodiment corresponds one-to-one with each implementation step of the regional panoramic monitoring method in Embodiment 1. Since Embodiment 1 has been described in detail, some details not shown in this embodiment can be referred to Embodiment 1, and will not be elaborated further here.

[0109] Example 3

[0110] According to an embodiment of this application, a computer program product is also provided, which includes a computer program, wherein when the computer program is executed by a processor, it implements the regional panoramic monitoring method in Embodiment 1.

[0111] According to an embodiment of this application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device containing the non-volatile storage medium executes the regional panoramic monitoring method in Embodiment 1 by running the computer program.

[0112] According to an embodiment of this application, a processor is also provided for running a computer program, wherein the computer program executes the regional panoramic monitoring method in Embodiment 1 during runtime.

[0113] According to an embodiment of this application, an electronic device is also provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the regional panoramic monitoring method of Embodiment 1 through the computer program.

[0114] Specifically, the computer program executes the following steps during runtime: It acquires an image set composed of multiple surveillance images simultaneously acquired by multiple image acquisition devices within a target area; it iteratively executes the following image fusion process until the multiple surveillance images are fused into a single panoramic image: it randomly selects two images from the image set, one as the reference image and the other as the image to be registered; it extracts multiple first feature points from the reference image and the first descriptor corresponding to each first feature point, and multiple second feature points from the image to be registered and the second descriptor corresponding to each second feature point; it uses a feature enhancement network to enhance the representation of each first descriptor, obtaining multiple third descriptors, and then applies these third descriptors to each... The second descriptor is enhanced to obtain multiple fourth descriptors. A feature enhancement network first enhances the descriptor of each feature point based on its own geometric information, and then enhances it a second time based on the correlation information between multiple feature points. Multiple first and second feature points are matched using the third and fourth descriptors, and the image to be registered is registered based on the matching results to obtain a registered image. The reference image and the registered image are fused to obtain a fused image, which is then placed back into the image set. Finally, a target detection algorithm is used to analyze the panoramic image to identify the target monitoring object within the target area.

[0115] As an alternative implementation, the above-mentioned electronic device may exist in the form of a mobile terminal, a computer terminal, or a similar computing device. Figure 3 A hardware structure block diagram of an electronic device for implementing a regional panoramic monitoring method is shown. Figure 3 As shown, the electronic device 30 may include one or more processors 302 (shown as 302a, 302b, ..., 302n in the figure) (processor 302 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 304 for storing data, and a transmission device 306 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 3 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, electronic device 30 may also include... Figure 3 The more or fewer components shown, or having the same Figure 3 The different configurations shown.

[0116] It should be noted that the aforementioned one or more processors 302 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element of the electronic device 30. As involved in the embodiments of this application, the data processing circuit serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0117] The memory 304 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the regional panoramic monitoring method in this embodiment. The processor 302 executes various functional applications and data processing by running the software programs and modules stored in the memory 304, thereby implementing the above-mentioned application vulnerability detection method. The memory 304 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 304 may further include memory remotely located relative to the processor 302, and these remote memories can be connected to the electronic device 30 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0118] The transmission device 306 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device 30. In one example, the transmission device 306 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 306 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0119] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the electronic device 30.

[0120] The sequence numbers of the above embodiments are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0121] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0123] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0125] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0126] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for panoramic regional monitoring, characterized in that, include: A collection of multiple surveillance images simultaneously acquired by multiple image acquisition devices within a target area is obtained. The following image fusion process is executed repeatedly until the multiple surveillance images are merged into a single panoramic image: two images are randomly selected from the image set, one as the reference image and the other as the image to be registered; Extract multiple first feature points from the reference image and a first descriptor corresponding to each first feature point, and multiple second feature points from the image to be registered and a second descriptor corresponding to each second feature point; Each first descriptor is enhanced using a feature enhancement network to obtain multiple third descriptors. Each second descriptor is then enhanced to obtain multiple fourth descriptors. The feature enhancement network performs a first enhancement on the descriptor of each feature point based on its own geometric information, and a second enhancement on the descriptor of each feature point based on the association information between multiple feature points. Multiple first and second feature points are matched using the third and fourth descriptors, and the image to be registered is registered based on the matching results to obtain a registered image. The reference image and the registered image are then fused to obtain a fused image, which is then placed back into the image set. The panoramic image is analyzed using a target detection algorithm to obtain the target monitoring objects within the target area.

2. The method according to claim 1, characterized in that, Extracting multiple first feature points from the reference image and a first descriptor corresponding to each first feature point, and multiple second feature points from the image to be registered and a second descriptor corresponding to each second feature point, includes: Multiple first feature points in the reference image are extracted using a directional fast and rotation-invariant feature algorithm, and a first descriptor corresponding to each first feature point is determined. Multiple second feature points in the image to be registered are extracted, and a second descriptor corresponding to each second feature point is determined.

3. The method according to claim 1, characterized in that, The feature enhancement network includes a self-enhancing subnetwork and a cross-enhancing subnetwork. The self-enhancing subnetwork includes two multilayer perceptrons, and the cross-enhancing subnetwork includes a transformer based on a multi-head attention mechanism. The feature enhancement network is used to enhance the representation of each first descriptor to obtain multiple third descriptors, and to enhance the representation of each second descriptor to obtain multiple fourth descriptors, including: Multiple first feature points and their corresponding first descriptors are input into the feature enhancement network to obtain the third descriptor corresponding to each first feature point. For each first feature point, the self-enhancing sub-network encodes the geometric information of the first feature point using a multilayer perceptron to obtain a first vector, projects the first descriptor corresponding to the first feature point onto the target space using another multilayer perceptron to obtain a second vector, and concatenates the first vector and the second vector to form a third vector. The cross-enhancing sub-network forms a first vector sequence from multiple third vectors, determines the first query matrix, first key matrix, and first value matrix corresponding to the first vector sequence, and calculates the first vector sequence, the first query matrix, the first key matrix, and the first value matrix using an attention formula to obtain the third descriptor corresponding to each first feature point. Multiple second feature points and their corresponding second descriptors are input into the feature enhancement network to obtain the fourth descriptor corresponding to each second feature point. For each second feature point, the self-enhancing sub-network encodes the geometric information of the second feature point using a multilayer perceptron to obtain a fourth vector, and projects the second descriptor corresponding to the second feature point onto the target space using another multilayer perceptron to obtain a fifth vector. The fourth vector and the fifth vector are concatenated to form a sixth vector. The cross-enhancing sub-network forms a second vector sequence from multiple sixth vectors and determines the second query matrix, the second key matrix, and the second value matrix corresponding to the second vector sequence. The attention formula is used to calculate the second vector sequence, the second query matrix, the second key matrix, and the second value matrix to obtain the fourth descriptor corresponding to each second feature point.

4. The method according to claim 3, characterized in that, The attention formula is: In the formula, X represents the vector sequence, N represents the number of vectors in the vector sequence, and f i (X) represents the enhanced descriptor corresponding to the i-th vector in the vector sequence, and Q, K, and V represent the query matrix, key matrix, and value matrix corresponding to the vector sequence, respectively. i Represents the query matrix for the i-th row, K j V j These represent the j-th row of the key matrix and the value matrix, respectively.

5. The method according to claim 1, characterized in that, Matching multiple first feature points and multiple second feature points based on the third descriptor and the fourth descriptor includes: Input multiple first feature points and their corresponding third descriptors, multiple second feature points and their corresponding fourth descriptors into the improved SuperGlue model to obtain matching results between multiple first feature points and multiple second feature points; The improved SuperGlue model includes multiple matching sub-models with identical structures. Each matching sub-model includes a feature encoding module, an attention module, and a classifier. The feature encoding module encodes the geometric information of the input feature points and their corresponding descriptors. The attention module includes a self-attention module and a cross-attention module. The self-attention module enhances the encoded information of the feature points based on the association information between feature points in the same image. The cross-attention module calculates the matching degree between feature points in different images based on the enhanced encoded information of each feature point. The classifier determines the matching result between feature points in different images based on the calculation result of the cross-attention module. The output of each self-attention module serves as the input of the feature encoding module in the next layer. When the self-attention module and the cross-attention module perform calculations on any two feature points, the feature point position information used is the relative position information between the two feature points.

6. The method according to claim 5, characterized in that, Each layer of the classifier is used to determine the confidence level of the matching prediction result corresponding to each feature point input in the current layer, and to determine feature points with a confidence level greater than a first preset threshold corresponding to the layer number of the classifier as reliable feature points, wherein the earlier the layer number of the classifier, the smaller the corresponding first preset threshold; to determine the ratio of the number of reliable feature points to the total number of input feature points; if the ratio is greater than a second preset threshold, to stop the iteration of the improved SuperGlue model and output the matching result of the current layer classifier; if the ratio is not greater than the second preset threshold, to determine unnecessary feature points with a confidence level greater than the first preset threshold and a matching prediction result of no matching feature points, and to instruct the self-attention module of the current layer not to output the information of the unnecessary feature points to the feature encoding module of the next layer.

7. The method according to claim 1, characterized in that, Based on the matching results, the image to be registered is registered to obtain a registered image, including: The positional information of multiple pairs of matched first and second feature points in the matching results is analyzed using the direct linear transformation algorithm to obtain the global homography matrix corresponding to the image to be registered. The image to be registered is divided into multiple sub-images to be registered; For each of the sub-images to be registered, the centroid of the sub-image to be registered is determined, and the weight coefficients corresponding to the sub-image to be registered are determined according to the distance relationship between the centroid and each of the second feature points. The local homography matrix corresponding to the sub-image to be registered is determined according to the weight coefficients and the global homography matrix, and the sub-image to be registered is linearly transformed according to the local homography matrix to obtain the registered sub-image corresponding to the sub-image to be registered. The multiple registration sub-images are stitched together to form the registration image.

8. The method according to claim 1, characterized in that, The reference image and the registered image are fused to obtain a fused image, including: Determine the target stitching line when stitching the reference image and the registered image; The reference image is divided into layers according to multiple preset frequency bands to obtain the first Gaussian pyramid, and the first Laplace pyramid corresponding to the first Gaussian pyramid is determined. The registered image is layered according to multiple preset frequency bands to obtain the second Gaussian pyramid, and the second Laplace pyramid corresponding to the second Gaussian pyramid is determined. The same frequency band layers of the first Laplace pyramid and the second Laplace pyramid are merged according to the target splicing line to obtain a merged Laplace pyramid; The fused image is obtained by performing an inverse Laplacian transform on the fused Laplacian pyramid.

9. The method according to claim 8, characterized in that, Determining the target stitching line when stitching the reference image and the registered image includes: A color energy function is determined based on the color difference between the stitching line on the reference image and the registered image, and a structure energy function is determined based on the gradient difference between the stitching line on the reference image and the registered image, wherein the stitching line is the quantity to be solved, and the gradient difference is the combined gradient difference in the horizontal and vertical directions; The first weight and the second weight corresponding to the color energy function and the structure energy function are determined respectively, and the color energy function and the structure energy function are weighted and summed according to the first weight and the second weight to obtain the target energy function; Find the target splicing line that minimizes the target energy function.

10. The method according to claim 1, characterized in that, The panoramic image is analyzed using a target detection algorithm to obtain the target monitoring objects within the target area, including: The panoramic image is divided into multiple local images; Each of the local images is analyzed using a target detection algorithm to obtain the monitored object in each local image; The non-maximum suppression algorithm is used to remove duplicate monitoring objects in multiple local images to obtain each of the target monitoring objects within the target area.

11. The method according to claim 1, characterized in that, The method further includes: Obtain the 3D scene model and geographic information corresponding to the target area; The panoramic image, the 3D scene model, and the geographic information are fused to obtain a panoramic monitoring model corresponding to the target area; Each of the target monitoring objects is highlighted in the panoramic monitoring model.

12. A regional panoramic monitoring device, characterized in that, include: The acquisition module is used to acquire a set of multiple monitoring images simultaneously acquired by multiple image acquisition devices within the target area. The image fusion module is used to repeatedly execute the following image fusion process until the multiple monitoring images are fused into a single panoramic image: randomly select two images from the image set, one as the reference image and the other as the image to be registered; Extract multiple first feature points from the reference image and a first descriptor corresponding to each first feature point, and multiple second feature points from the image to be registered and a second descriptor corresponding to each second feature point; Each first descriptor is enhanced using a feature enhancement network to obtain multiple third descriptors. Each second descriptor is then enhanced to obtain multiple fourth descriptors. The feature enhancement network performs a first enhancement on the descriptor of each feature point based on its own geometric information, and a second enhancement on the descriptor of each feature point based on the association information between multiple feature points. Multiple first and second feature points are matched using the third and fourth descriptors, and the image to be registered is registered based on the matching results to obtain a registered image. The reference image and the registered image are then fused to obtain a fused image, which is then placed back into the image set. The object detection module is used to analyze the panoramic image using a target detection algorithm to obtain the target monitoring objects within the target area.

13. A computer program product, characterized in that, include: A computer program, wherein when executed by a processor, the computer program implements the regional panoramic monitoring method according to any one of claims 1 to 11.

14. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the regional panoramic monitoring method according to any one of claims 1 to 11 through the computer program.

Images