Regional panoramic monitoring method and device

Abstract

The invention discloses a regional panoramic monitoring method and device. The method comprises the following steps: acquiring monitoring images acquired by a plurality of image acquisition devices in a target area to form an image set; circularly executing the following image fusion process until the plurality of monitoring images are fused into one panoramic image: randomly taking out two images from the image set, taking one image as a reference image, and taking the other image as a to-be-registered image; extracting feature points and descriptors of the reference image and the to-be-registered image; enhancing the descriptors twice by using a feature enhancement network; matching the feature points according to the enhanced descriptors, and registering the image according to a matching result; fusing the reference image and the registration image to obtain a fused image, and putting the fused image back to the image set; and analyzing the panoramic image by using a target detection algorithm, and identifying a monitoring image in the target area. According to the method and the device, the technical problems that panoramic image stitching is difficult to efficiently and accurately carry out and the monitoring effect is not satisfactory in a related video monitoring scheme are solved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method and device for panoramic regional monitoring. Background Art

[0002] With the acceleration of urbanization and the advancement of smart city construction, urban management departments have an increasingly urgent need to conduct real-time monitoring and management of urban environment, roads, transportation, public facilities, natural resources, environment, etc. As an important means of urban governance informatization, video surveillance systems have been widely used, enabling urban managers to discover events in a timely manner and handle emergencies. The massiveness, multi-mode, multi-modality, and strong correlation of video data make it possible to use video data for urban governance. Therefore, it is of great value and significance to build a video twin system to realize the integration of video data and GIS (Geographic Information System) three-dimensional spatial data. The video twin system can integrate GIS scenes with video scenes to realize real-time interaction between virtual and reality, and provide a visual, interactive, and analyzable decision support platform for urban governance. It is a product of combining GIS technology with video surveillance technology, which can realize seamless connection between virtual space and real space, and provide effective decision support for urban governance. The video twin system can meet the needs of urban management for real-time monitoring, event decision-making, spatial prediction, environmental governance, smart transportation, intelligent security, resource management, etc., and is a key technology for urban governance.

[0003] At present, the video twin system mainly includes five steps: video stream acquisition, feature matching, image fusion, target detection and 3D GIS fusion. Among them, the video stream acquisition module needs to collect real-time video streams to meet the needs of urban governance for real-time monitoring; the feature matching module needs to register the image to achieve real-time fusion of video streams; the image fusion module needs to achieve real-time fusion of video streams so that urban managers can observe the urban environment anytime and anywhere; the target detection module needs to perform real-time target detection to achieve real-time monitoring of events by urban managers; the 3D GIS fusion module needs to perform real-time fusion of video streams with GIS 3D scenes to achieve seamless connection between virtual space and real space. However, the existing video twin system has the following shortcomings: 1. It does not support the acquisition of a large number of camera video streams, and cannot meet the needs of urban governance for real-time monitoring and event monitoring; 2. It does not support real-time processing of image registration, image fusion and target detection, and cannot meet the needs of urban governance for real-time processing, and cannot achieve timely discovery and response to events; 3. It does not support real-time fusion of video streams and GIS 3D scenes, and cannot meet the needs of urban governance for seamless connection between virtual space and real space.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a method and device for regional panoramic monitoring, so as to at least solve the technical problems in related video monitoring solutions that it is difficult to efficiently and accurately stitch panoramic images and the monitoring effect is unsatisfactory.

[0006] According to one aspect of an embodiment of the present application, a method for panoramic monitoring of an area is provided, comprising: obtaining a plurality of monitoring images synchronously acquired by a plurality of image acquisition devices in a target area to form an image set; cyclically executing the following image fusion process until the plurality of monitoring images are fused into a panoramic image: randomly taking out two images from the image set, one as a reference image and the other as an image to be registered; extracting a plurality of first feature points of the reference image and a first descriptor corresponding to each first feature point, a plurality of second feature points of the image to be registered and a second descriptor corresponding to each second feature point; and using a feature enhancement network to enhance each first descriptor to obtain a plurality of third descriptors. Each second descriptor is enhanced and represented respectively to obtain a plurality of fourth descriptors, wherein a feature enhancement network is used to perform a first enhancement on the descriptor of each feature point according to the geometric information of each feature point itself, and to perform a second enhancement on the descriptor of each feature point according to the association information between the plurality of feature points; the plurality of first feature points and the plurality of second feature points are matched according to the third descriptor and the fourth descriptor, and the image to be registered is registered according to the matching result to obtain a registered image; the reference image and the registered image are fused to obtain a fused image, and the fused image is put back into the image set; the panoramic image is analyzed by a target detection algorithm to obtain a target monitoring object in the target area.

[0007] Optionally, extracting multiple first feature points of the reference image and a first descriptor corresponding to each first feature point, and extracting multiple second feature points of the image to be registered and a second descriptor corresponding to each second feature point includes: using a directional fast and rotation invariant feature algorithm to extract multiple first feature points in the reference image and determine the first descriptor corresponding to each first feature point, extracting multiple second feature points in the image to be registered and determining the second descriptor corresponding to each second feature point.

[0008] Optionally, the feature enhancement network includes: a self-enhancement subnetwork and a cross-enhancement subnetwork, the self-enhancement subnetwork includes two multi-layer perceptrons, the cross-enhancement subnetwork includes a transformer based on a multi-head attention mechanism, and the feature enhancement network is used to enhance each first descriptor to obtain multiple third descriptors, and each second descriptor is enhanced to obtain multiple fourth descriptors, including: inputting multiple first feature points and corresponding first descriptors into the feature enhancement network to obtain the third descriptor corresponding to each first feature point, wherein, for each first feature point, the self-enhancement subnetwork is used to encode the geometric information of the first feature point using a multi-layer perceptron to obtain a first vector, and another multi-layer perceptron is used to project the first descriptor corresponding to the first feature point to the target space to obtain a second vector, and the first vector and the second vector are spliced ​​into a third vector; the cross-enhancement subnetwork is used to combine multiple third vectors into a first vector sequence, and determine the first query matrix corresponding to the first vector sequence. , a first key matrix and a first value matrix, and use the attention formula to calculate the first vector sequence, the first query matrix, the first key matrix and the first value matrix to obtain a third descriptor corresponding to each first feature point; multiple second feature points and the corresponding second descriptors are input into the feature enhancement network to obtain a fourth descriptor corresponding to each second feature point, wherein, for each second feature point, the self-enhancement subnetwork is used to encode the geometric information of the second feature point using a multi-layer perceptron to obtain a fourth vector, and another multi-layer perceptron is used to project the second descriptor corresponding to the second feature point to the target space to obtain a fifth vector, and the fourth vector and the fifth vector are concatenated into a sixth vector; the cross-enhancement subnetwork is used to form a second vector sequence with multiple sixth vectors, and determine the second query matrix, the second key matrix and the second value matrix corresponding to the second vector sequence, and use the attention formula to calculate the second vector sequence, the second query matrix, the second key matrix and the second value matrix to obtain the fourth descriptor corresponding to each second feature point.

[0009] Optionally, the attention formula is: In the formula, X represents the vector sequence, N represents the number of vectors in the vector sequence, and f i (X) represents the enhanced descriptor corresponding to the i-th vector in the vector sequence, Q, K, and V represent the query matrix, key matrix, and value matrix corresponding to the vector sequence, respectively. i represents the i-th row of the query matrix, K j 、V j denote the j-th row of the key matrix and the value matrix respectively.

[0010] Optionally, matching the plurality of first feature points with the plurality of second feature points is performed according to the third descriptor and the fourth descriptor, including: inputting the plurality of first feature points and the corresponding third descriptors, the plurality of second feature points and the corresponding fourth descriptors into the improved SuperGlue model, and obtaining matching results between the plurality of first feature points and the plurality of second feature points; wherein the improved SuperGlue model includes a multi-layer matching sub-model with the same structure, and each layer of the matching sub-model includes: a feature encoding module, an attention module and a classifier; the feature encoding module is used to encode and represent the geometric information of the input feature points and the corresponding descriptors; the attention module includes: a self- Attention module and cross-attention module, the self-attention module is used to enhance the representation of the encoding information of the feature points according to the correlation information between the feature points in the same image, and the cross-attention module is used to calculate the matching degree between the feature points in different images according to the encoding information after the enhanced representation of each feature point; the classifier is used to determine the matching results between the feature points in different images according to the calculation results of the cross-attention module; the output of each layer of self-attention module is used as the input of the feature encoding module of the next layer; wherein, when the self-attention module and the cross-attention module calculate any two feature points, the feature point position information used is the relative position information between the two feature points.

[0011] Optionally, each layer of classifiers is used to determine the confidence of the matching prediction results corresponding to each feature point inputted into the current layer, and determine the feature points whose confidence is greater than the first preset threshold corresponding to the number of layers of the classifier as credible feature points, wherein the earlier the number of layers of the classifier is, the smaller the corresponding first preset threshold is; determine the ratio of the number of credible feature points to the total number of input feature points; when the ratio is greater than the second preset threshold, stop the iteration of the improved SuperGlue model and output the matching results of the classifier of the current layer; when the ratio is not greater than the second preset threshold, determine the non-essential feature points whose confidence is greater than the first preset threshold and whose matching prediction results are non-essential feature points without matching feature points, and instruct the self-attention module of the current layer not to output the information of non-essential feature points to the feature encoding module of the next layer.

[0012] Optionally, the image to be registered is registered based on the matching results to obtain a registered image, including: using a direct linear transformation algorithm to analyze the position information of multiple pairs of matching first feature points and second feature points in the matching results to obtain a global homography matrix corresponding to the image to be registered; dividing the image to be registered into multiple sub-images to be registered; for each sub-image to be registered, determining the center of mass of the sub-image to be registered, and determining the weight coefficient corresponding to the sub-image to be registered based on the distance relationship between the center of mass and each second feature point, determining the local homography matrix corresponding to the sub-image to be registered based on the weight coefficient and the global homography matrix, and performing a linear transformation on the sub-image to be registered based on the local homography matrix to obtain a registered sub-image corresponding to the sub-image to be registered; and splicing multiple registered sub-images into a registered image.

[0013] Optionally, fusing the reference image and the registered image to obtain a fused image includes: determining a target stitching line when stitching the reference image and the registered image; stratifying the reference image according to a plurality of preset frequency bands to obtain a first Gaussian pyramid, and determining a first Laplacian pyramid corresponding to the first Gaussian pyramid; stratifying the registered image according to a plurality of preset frequency bands to obtain a second Gaussian pyramid, and determining a second Laplacian pyramid corresponding to the second Gaussian pyramid; fusing the same frequency band layers of the first Laplacian pyramid and the second Laplacian pyramid according to the target stitching line to obtain a fused Laplacian pyramid; and performing an inverse Laplacian transform on the fused Laplacian pyramid to obtain a fused image.

[0014] Optionally, determining a target stitching line when stitching the reference image and the registered image includes: determining a color energy function based on a color difference between the stitching line on the reference image and the registered image, and determining a structure energy function based on a gradient difference between the stitching line on the reference image and the registered image, wherein the stitching line is a quantity to be solved, and the gradient difference is a comprehensive gradient difference in the horizontal and vertical directions; determining a first weight and a second weight corresponding to the color energy function and the structure energy function, respectively, and performing a weighted summation of the color energy function and the structure energy function based on the first weight and the second weight to obtain a target energy function; and solving the target stitching line that minimizes the target energy function.

[0015] Optionally, a target detection algorithm is used to analyze the panoramic image to obtain the target monitoring objects in the target area, including: dividing the panoramic image into multiple local images; analyzing each local image using a target detection algorithm to obtain the monitoring objects in each local image; and deduplicating the monitoring objects in multiple local images using a non-maximum suppression algorithm to obtain each target monitoring object in the target area.

[0016] Optionally, the method also includes: obtaining a three-dimensional scene model and geographic information corresponding to the target area; fusing the panoramic image, the three-dimensional scene model and the geographic information to obtain a panoramic monitoring model corresponding to the target area; and highlighting each target monitoring object in the panoramic monitoring model.

[0017] According to another aspect of the embodiment of the present application, a regional panoramic monitoring device is also provided, including: an acquisition module, used to acquire multiple monitoring images synchronously acquired by multiple image acquisition devices in a target area to form an image set; an image fusion module, used to cyclically execute the following image fusion process until the multiple monitoring images are fused into a panoramic image: randomly select two images from the image set, one as a reference image, and the other as an image to be registered; extract multiple first feature points of the reference image and a first descriptor corresponding to each first feature point, multiple second feature points of the image to be registered and a second descriptor corresponding to each second feature point; use a feature enhancement network to enhance each first descriptor to obtain multiple third descriptors. descriptor, and each second descriptor is enhanced to obtain multiple fourth descriptors, wherein the feature enhancement network is used to perform a first enhancement on the descriptor of each feature point according to the geometric information of each feature point itself, and a second enhancement on the descriptor of each feature point according to the association information between the multiple feature points; the multiple first feature points and the multiple second feature points are matched according to the third descriptor and the fourth descriptor, and the image to be registered is registered according to the matching result to obtain a registered image; the reference image and the registered image are fused to obtain a fused image, and the fused image is put back into the image set; the object detection module is used to analyze the panoramic image using the target detection algorithm to obtain the target monitoring object in the target area.

[0018] According to another aspect of an embodiment of the present application, a computer program product is further provided, the computer program product comprising: a computer program, wherein when the computer program is executed by a processor, the above-mentioned regional panoramic monitoring method is implemented.

[0019] According to another aspect of an embodiment of the present application, an electronic device is further provided, comprising: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned regional panoramic monitoring method through the computer program.

[0020] In an embodiment of the present application, a plurality of surveillance images synchronously acquired by a plurality of image acquisition devices in a target area are acquired to form an image set; the following image fusion process is executed cyclically until the plurality of surveillance images are fused into a panoramic image: two images are randomly selected from the image set, one as a reference image and the other as an image to be registered; a plurality of first feature points of the reference image and a first descriptor corresponding to each first feature point, a plurality of second feature points of the image to be registered and a second descriptor corresponding to each second feature point are extracted; each first descriptor is enhanced by using a feature enhancement network to obtain a plurality of third descriptors, and each second descriptor is enhanced by using a feature enhancement network to obtain a plurality of third descriptors. The method comprises the following steps: a first enhancement is performed on the descriptor of each feature point according to the geometric information of each feature point, and a second enhancement is performed on the descriptor of each feature point according to the association information between the multiple feature points; a plurality of first feature points and a plurality of second feature points are matched according to the third descriptor and the fourth descriptor, and the image to be registered is registered according to the matching result to obtain a registered image; the reference image and the registered image are fused to obtain a fused image, and the fused image is put back into the image set; the panoramic image is analyzed by the target detection algorithm to obtain the target monitoring object in the target area. In this way, a feature enhancement network is introduced in image fusion to perform secondary enhancement on the feature points and descriptors extracted by the directional fast and rotation invariant feature algorithms, thereby achieving fast and accurate feature matching under limited computing resources. In addition, the local homography matrix and grid division technology are used, combined with weight coefficient optimization, to effectively deal with the registration problems of large parallax and image overlapping areas, eliminate stitching traces and ghosting, and in the large-size image target detection module, the strategy of first segmentation, then integration and then deduplication is adopted to enhance the detection capability of small targets and reduce the missed detection rate, thereby solving the technical problems of difficult to efficiently and accurately stitch panoramic images and unsatisfactory monitoring effects in related video surveillance solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 is a flow chart of an optional method for panoramic regional monitoring according to an embodiment of the present application;

[0023] Figure 2 is a schematic structural diagram of an optional regional panoramic monitoring device according to an embodiment of the present application;

[0024] Figure 3 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification, claims and drawings of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Example 1

[0028] According to an embodiment of the present application, a method for panoramic monitoring of an area is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] Figure 1 is a flow chart of a method for panoramic monitoring of a region provided according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0030] Step S102, obtaining a plurality of monitoring images synchronously captured by a plurality of image acquisition devices in a target area to form an image set;

[0031] Step S104, looping and executing the following image fusion process until the multiple surveillance images are fused into a panoramic image: randomly taking out two images from the image set, one as the reference image, and the other as the image to be registered; extracting multiple first feature points of the reference image and the first descriptor corresponding to each first feature point, and multiple second feature points of the image to be registered and the second descriptor corresponding to each second feature point; using the feature enhancement network to enhance each first descriptor respectively to obtain multiple third descriptors, and enhancing each second descriptor respectively to obtain multiple fourth descriptors, wherein the feature enhancement network is used to enhance the descriptor of each feature point for the first time according to the geometric information of each feature point itself, and to enhance the descriptor of each feature point for the second time according to the association information between the multiple feature points; matching the multiple first feature points and the multiple second feature points according to the third descriptor and the fourth descriptor, and registering the image to be registered according to the matching result to obtain the registered image; fusing the reference image and the registered image to obtain the fused image, and putting the fused image back into the image set;

[0032] Step S106, analyzing the panoramic image using a target detection algorithm to obtain a target monitoring object in the target area.

[0033] The following describes each step of the regional panoramic monitoring method in conjunction with a specific implementation process.

[0034] First, multiple monitoring images synchronously captured by multiple image acquisition devices in the target area are obtained to form an image set.

[0035] The following image fusion process is executed in a loop until the multiple surveillance images are fused into a panoramic image: two images are randomly selected from the image set, one is used as the reference image, and the other is used as the image to be registered; multiple first feature points of the reference image and the first descriptor corresponding to each first feature point, multiple second feature points of the image to be registered and the second descriptor corresponding to each second feature point are extracted; each first descriptor is enhanced and represented by a feature enhancement network to obtain multiple third descriptors, and each second descriptor is enhanced and represented to obtain multiple fourth descriptors, wherein the feature enhancement network is used to perform a first enhancement on the descriptor of each feature point according to the geometric information of each feature point itself, and to perform a second enhancement on the descriptor of each feature point according to the association information between the multiple feature points; multiple first feature points and multiple second feature points are matched according to the third descriptor and the fourth descriptor, and the image to be registered is registered according to the matching result to obtain a registered image; the reference image and the registered image are fused to obtain a fused image, and the fused image is put back into the image set.

[0036] For example, the number of cameras is 2 to the power of n, and does not exceed 16 at most. Each camera acquires video frames synchronously. Two adjacent cameras form a group. In each group of image data, one image is used as a reference image and the other image is used as an image to be registered.

[0037] Optionally, extracting multiple first feature points of the reference image and a first descriptor corresponding to each first feature point, and extracting multiple second feature points of the image to be registered and a second descriptor corresponding to each second feature point can be performed in the following manner: using an oriented fast and rotationally invariant feature (Oriented FAST and Rotated BRIEF, ORB) algorithm to extract multiple first feature points in the reference image and determine the first descriptor corresponding to each first feature point, extracting multiple second feature points in the image to be registered, and determining the second descriptor corresponding to each second feature point.

[0038] Among them, the ORB algorithm is used to find multiple key points from the reference image. These key points are called "first feature points. These points are usually located at locations with features such as corners and edges in the image. Because the local image structure at these locations has high descriptiveness and discrimination, they are suitable for subsequent feature matching. For each detected first feature point, the algorithm calculates a set of numerical vectors, namely the first descriptor, which is used to describe the image features around the point. The ORB algorithm uses the BRIEF descriptor, which is a binary descriptor. By comparing the intensity of pixels in the neighborhood of the feature point, a series of binary strings of 0 and 1 are obtained to represent the local features of the feature point.

[0039] This application introduces a feature enhancement network based on the traditional directional fast and rotation invariant feature algorithms. The idea of ​​the feature enhancement network is to optimize through two stages of self-enhancement and cross-enhancement without the need for the original image to participate in the calculation.

[0040] Specifically, a feature enhancement network is used to enhance each first descriptor to obtain multiple third descriptors, and each second descriptor is enhanced to obtain multiple fourth descriptors, wherein the feature enhancement network is used to perform a first enhancement on the descriptor of each feature point based on the geometric information of each feature point itself, and to perform a second enhancement on the descriptor of each feature point based on the association information between the multiple feature points.

[0041] Optionally, the feature enhancement network includes: a self-enhancement sub-network and a cross-enhancement sub-network, the self-enhancement sub-network includes two multi-layer perceptrons, and the cross-enhancement sub-network includes a transformer based on a multi-head attention mechanism.

[0042] Among them, each first descriptor is enhanced and represented by a feature enhancement network to obtain multiple third descriptors, and each second descriptor is enhanced and represented to obtain multiple fourth descriptors, including the following steps: multiple first feature points and corresponding first descriptors are input into the feature enhancement network to obtain the third descriptor corresponding to each first feature point, wherein, for each first feature point, the self-enhancement sub-network is used to encode the geometric information of the first feature point by using a multi-layer perceptron to obtain a first vector, and another multi-layer perceptron is used to project the first descriptor corresponding to the first feature point to the target space to obtain a second vector, and the first vector and the second vector are spliced ​​into a third vector; the cross-enhancement sub-network is used to form a first vector sequence with multiple third vectors, and determine the first query matrix, the first key matrix and the first value matrix corresponding to the first vector sequence, and use the attention formula to calculate the first vector sequence, the first query matrix, the first key matrix and the first value matrix to obtain the third descriptor corresponding to each first feature point, wherein the attention formula is:

[0043]

[0044] In the formula, X represents the vector sequence, N represents the number of vectors in the vector sequence, and f i (X) represents the enhanced descriptor corresponding to the i-th vector in the vector sequence, Q, K, and V represent the query matrix, key matrix, and value matrix corresponding to the vector sequence, respectively. i represents the i-th row of the query matrix, K j 、V j denote the j-th row of the key matrix and the value matrix respectively.

[0045] The obtained multiple second feature points and the corresponding second descriptors are input into the feature enhancement network to obtain a fourth descriptor corresponding to each second feature point, wherein, for each second feature point, the self-enhancement sub-network is used to encode the geometric information of the second feature point using a multi-layer perceptron to obtain a fourth vector, and another multi-layer perceptron is used to project the second descriptor corresponding to the second feature point to the target space to obtain a fifth vector, and the fourth vector and the fifth vector are concatenated into a sixth vector; the cross-enhancement sub-network is used to form a second vector sequence with multiple sixth vectors, and determine the second query matrix, second key matrix and second value matrix corresponding to the second vector sequence, and use the attention formula to calculate the second vector sequence, the second query matrix, the second key matrix and the second value matrix to obtain the fourth descriptor corresponding to each second feature point.

[0046] The above process combines the efficiency of traditional directional fast and rotation-invariant feature algorithms with the accuracy of feature enhancement networks, enabling real-time image registration on embedded GPU (Graphics Processing Unit) devices, and maintaining good matching effects even in complex scenes, such as when there are fast-moving objects in the image that cause blur or dragging marks.

[0047] After feature point detection and descriptor enhancement, image matching can be performed. Image matching is performed by comparing the descriptors of key points in two images. Its goal is to determine which points correspond to the same real-world location in the two images. Traditional matching algorithms, such as Nearest Neighbor Matching (NNM), are a basic feature matching method that determines the correspondence between feature points in two images by calculating the distance between descriptors (such as Euclidean distance or Hamming distance). However, the simplicity of NNM also brings problems. It may generate a large number of incorrect matching points, especially when there are similar local features in the images, which will reduce the accuracy of matching. The RANSAC (Random Sample Consensus) algorithm estimates the matching model by randomly selecting a minimum data set (i.e., point pairs), and then uses the entire data set to verify the accuracy of the model. It screens out inliers through an iterative process to eliminate incorrect matching point pairs, thereby improving the accuracy of matching. However, the RANSAC algorithm is inefficient and requires a large number of iterations to obtain a good model estimate. It is also sensitive to the setting of model parameters (such as the inlier threshold) and may not obtain the globally optimal model estimate. The SuperGlue algorithm, as an end-to-end deep learning solution, simultaneously learns feature point matching and inlier screening through a neural network. Compared with traditional methods, it can more accurately match two sets of local features and remove those points that cannot be matched. It has stronger robustness and generalization capabilities. SuperGlue is based on the Transformer architecture and can handle complex matching tasks, but its disadvantage is that it requires a large amount of training data and computing resources, and as the number of key points increases, the computational complexity increases quadratically. Therefore, it can only achieve real-time matching when running on a desktop GPU.

[0048] In view of the limitations of the SuperGlue algorithm, this application proposes two key improvements: changing absolute coding to relative coding. The traditional SuperGlue algorithm uses absolute point position coding, while this application improves it to use relative position coding. This method takes into account that relative position information is more important than absolute position information in the matching process, because relative position has better consistency between different images, while absolute position may be affected by variables such as camera position and viewing angle; the accuracy and reasoning time are controlled by the confidence and threshold of the predicted allocation. At the end of each layer of the network, the improved method of this application calculates the predicted allocation confidence of each feature point. A higher confidence indicates that the matching result of the feature point is reliable. By setting the threshold, the network reasoning process can be stopped in advance to reduce unnecessary calculations. In addition, when feature points are predicted to be unmatched, they will be eliminated early, and only the remaining feature points will be retained for subsequent calculations, which can significantly reduce the amount of calculation without affecting the final matching accuracy.

[0049] The matching process using the improved SuperGlue model is as follows:

[0050] The multiple first feature points and the multiple second feature points are matched according to the third descriptor and the fourth descriptor, and the image to be registered is registered according to the matching result to obtain the registered image. This can be done in the following way: the multiple first feature points and the corresponding third descriptors, the multiple second feature points and the corresponding fourth descriptors are input into the improved SuperGlue model to obtain the matching results between the multiple first feature points and the multiple second feature points.

[0051] Among them, the improved SuperGlue model includes multiple layers of matching sub-models with the same structure, and each layer of the matching sub-model includes: a feature encoding module, an attention module and a classifier; the feature encoding module is used to encode and represent the geometric information of the input feature points and the corresponding descriptors; the attention module includes: a self-attention module and a cross-attention module, the self-attention module is used to enhance the encoding information of the feature points according to the association information between the feature points in the same image, and the cross-attention module is used to calculate the matching degree between the feature points in different images according to the encoding information after the enhanced representation of each feature point; the classifier is used to determine the matching results between the feature points in different images according to the calculation results of the cross-attention module; the output of each layer of the self-attention module is used as the input of the feature encoding module of the next layer; wherein, when the self-attention module and the cross-attention module calculate any two feature points, the feature point position information used is the relative position information between the two feature points.

[0052] For example, according to the SuperGlue method, the partial distribution is predicted between two sets of local features extracted from images A and B, where each local feature i is composed of a 2D (2Dimensions, two-dimensional) point position p normalized to the image size. i :=(x,y) i ∈[0,1] 2 and a visual descriptor d i ∈R d The images A and B have M and N local features respectively, and the index sets are A: = {1, ..., M} and B: = {1, ..., N} respectively. A partial allocation matrix is ​​obtained between the local features of A and B, and the corresponding relationship is extracted from it.

[0053] Each local feature i in the image I∈{A,B} is associated with a state The state is associated with the corresponding visual descriptor is initialized and then updated by each layer, each layer consisting of a consecutive self-attention module and a cross-attention module.

[0054] For the self-attention module, each point pays attention to all points in the same image. The same steps are performed for each image I, so the superscript I is omitted. For each point i, the current state x i Decompose into key vector k through different linear transformations i and the query vector q i , the attention score between point i and point j is:

[0055]

[0056] in, It is a rotation encoding of the relative position between points, dividing the space into d / 2 two-dimensional subspaces, and rotating each subspace by an angle corresponding to the learned basis vector according to the Fourier feature method. The projection on corresponds to:

[0057]

[0058] Position encoding plays a key role in the attention mechanism, allowing different elements to be processed depending on where they are located. In projective camera geometry, the position of a visual observation is equivalent to the translation of the camera within the image plane: 2D points from 3D points on the same parallel plane are translated in the same way, and their relative distances remain unchanged. The encoding only needs to capture the relative position of the points rather than the absolute position. Rotation encoding enables the model to retrieve point j from i at a learned relative position. Position encoding is not applied to the numerical value v. j , so it will not affect the state xi , the encoding is the same for all layers, so it only needs to be computed once and cached.

[0059] Optionally, each layer of classifiers is used to determine the confidence of the matching prediction results corresponding to each feature point inputted into the current layer, and determine the feature points whose confidence is greater than the first preset threshold corresponding to the number of layers of the classifier as credible feature points, wherein the earlier the number of layers of the classifier is, the smaller the corresponding first preset threshold is; determine the ratio of the number of credible feature points to the total number of input feature points; when the ratio is greater than the second preset threshold, stop the iteration of the improved SuperGlue model and output the matching results of the classifier of the current layer; when the ratio is not greater than the second preset threshold, determine the non-essential feature points whose confidence is greater than the first preset threshold and whose matching prediction results are non-essential feature points without matching feature points, and instruct the self-attention module of the current layer not to output the information of non-essential feature points to the feature encoding module of the next layer.

[0060] For example, the backbone of the feature enhancement network helps improve the quality of feature point descriptors by contextually enhancing the input visual descriptors, making feature matching more reliable. The network attempts to predict feature points at the end of each layer and evaluates the confidence of these predictions using the following formula:

[0061] c i =Sigmoid(MLP(x i ))∈[0,1]

[0062] In the formula, c i Represents the prediction confidence of each point, x i The feature vector representing the i-th point, a higher confidence value indicates that the representation of i is a reliable final result, that is, the point is either successfully matched or determined to be unmatched.

[0063] For a given layer, if the confidence of a point is greater than the threshold, it is considered to be credible. If a sufficient proportion of feature points have been confirmed as reliable matches after processing a certain layer, the reasoning is stopped. The specific calculation formula is:

[0064]

[0065] In the formula, M and N represent the number of points in the two images respectively, I is a set, representing two images A and B, represents the confidence of the i-th point in image I, λ l represents the first preset threshold, and exit represents that the confidence in the two images exceeds the threshold λ l , and the ratio is greater than the preset second preset threshold α, then exit is true, indicating that the reasoning process can be terminated early.

[0066] It should be noted that the confidence of the classifier itself is low in the first few layers. Therefore, as the number of layers increases, the confidence of the classifier may decrease, so the threshold λ is gradually reduced. l , to adapt to the changes in confidence at different layers, if a point is predicted to be credible and cannot be matched in the previous layers, these points are no longer considered in subsequent layers because they are usually located in areas of the image that are obviously not co-viewed. This can reduce unnecessary calculations and improve efficiency. Although the reasoning is terminated early, the confidence evaluation and threshold adjustment mechanism can still maintain a high matching accuracy.

[0067] The image to be registered is registered according to the matching result to obtain the registered image, including: using a direct linear transformation algorithm to analyze the position information of multiple pairs of matching first feature points and second feature points in the matching result to obtain a global homography matrix corresponding to the image to be registered; dividing the image to be registered into multiple sub-images to be registered; for each sub-image to be registered, determining the center of mass of the sub-image to be registered, and determining the weight coefficient corresponding to the sub-image to be registered according to the distance relationship between the center of mass and each second feature point, determining the local homography matrix corresponding to the sub-image to be registered according to the weight coefficient and the global homography matrix, and performing a linear transformation on the sub-image to be registered according to the local homography matrix to obtain the registered sub-image corresponding to the sub-image to be registered; and splicing multiple registered sub-images into a registered image.

[0068] For example, suppose the reference image I and the image to be registered I ′ The corresponding matching point is X = [xy] T , X ′ =[x ′ y ′ ] T , the transformation relationship is is the homogeneous coordinate of X, Yes X ′ Homogeneous coordinates, matrix H∈R 3x3 , the column vector is h1=[h1 h4 h7] T , h2=[h2 h5 h8] T , h3=[h3 h6 h9] T , which can be expressed as:

[0069]

[0070] The homogeneous equation for the projection transformation is as follows:

[0071]

[0072] For two images with N sets of matching points, the objective function for calculating the global homography matrix H based on direct linear transformation is:

[0073]

[0074] Divide the image to be registered into C1xC2 grids, each grid corresponds to a homography matrix H * , by introducing the weight coefficient W * Determine the local homography matrix H * , the center point of the grid is x * , the matching point is x i , the local homography matrix of any grid is:

[0075]

[0076] Traverse each grid and perform linear transformation on the sub-image to be registered according to the local homography matrix to obtain the registration sub-image corresponding to the sub-image to be registered, and then splice multiple registration sub-images into a registration image.

[0077] The reference image and the registered image are fused to obtain a fused image, which can be performed in the following manner: determining a target stitching line when stitching the reference image and the registered image; stratifying the reference image according to a plurality of preset frequency bands to obtain a first Gaussian pyramid, and determining a first Laplacian pyramid corresponding to the first Gaussian pyramid; stratifying the registered image according to a plurality of preset frequency bands to obtain a second Gaussian pyramid, and determining a second Laplacian pyramid corresponding to the second Gaussian pyramid; fusing the same frequency band layers of the first Laplacian pyramid and the second Laplacian pyramid according to the target stitching line to obtain a fused Laplacian pyramid; and performing an inverse Laplacian transform on the fused Laplacian pyramid to obtain a fused image.

[0078] Determining a target stitching line when stitching a reference image and a registered image comprises the following steps: determining a color energy function according to a color difference between the stitching line on the reference image and the registered image, and determining a structure energy function according to a gradient difference between the stitching line on the reference image and the registered image, wherein the stitching line is a quantity to be solved, and the gradient difference is a comprehensive gradient difference in the horizontal and vertical directions; determining a first weight and a second weight corresponding to the color energy function and the structure energy function respectively, and performing a weighted summation of the color energy function and the structure energy function according to the first weight and the second weight to obtain a target energy function; solving a target stitching line that minimizes the target energy function.

[0079] Among them, the optimal splicing line is first found by the method of minimizing the energy function, and the color and structure conditions are comprehensively considered, and the calculation formula is as follows:

[0080] E C (X)=‖I1(X)-I2(X)‖

[0081]

[0082] In the formula, I1 represents the reference image, I2 represents the image to be registered, and E C Indicates the color difference of the overlapping area seam point X, E G Represents the gradient difference between the horizontal and vertical directions of the seam point X in the overlapping area. represents the Sobel operator, energy function E(X) = λ1E C (X)+λ2E G (X), weight λ1+λ2=1, use the maximum flow minimum cut idea to minimize the energy function, so as to find the optimal splicing line.

[0083] The target detection algorithm is used to analyze the panoramic image to obtain the target monitoring objects in the target area. The steps are as follows: divide the panoramic image into multiple local images; use the target detection algorithm to analyze each local image to obtain the monitoring objects in each local image; use the non-maximum suppression algorithm to deduplicate the monitoring objects in multiple local images to obtain the target monitoring objects in the target area.

[0084] For example, the fused image is divided into four local images, and each local image is subjected to target detection through the improved YOLOv5m model. The model adds three smaller anchor frames for small targets. After detection, the results of all local images are integrated on the original image, and the non-maximum suppression algorithm is used to remove duplicate detection frames to obtain all target monitoring objects in the target area.

[0085] In addition to the method of this embodiment, a three-dimensional scene model and geographic information corresponding to the target area can also be obtained; the panoramic image, three-dimensional scene model and geographic information are fused to obtain a panoramic monitoring model corresponding to the target area; and each target monitoring object is highlighted in the panoramic monitoring model.

[0086] For example, it is necessary to collect detailed geographic information of the target area (such as an industrial park), including terrain, three-dimensional structure of buildings, roads, etc. This information can be obtained based on high-precision GIS data or through technologies such as lidar scanning and drone aerial photography to create a fine three-dimensional digital model. In addition to the three-dimensional model, it is also necessary to obtain geographic coordinates, orientation, altitude and other information related to the model. This information is used to accurately align the three-dimensional model with the actual geographical environment to ensure that the fusion effect of the video and the model is true and reliable. After the image registration and fusion modules, a high-quality panoramic image is obtained. The image is stitched together from videos of multiple cameras, covering the complete field of view of the park. Subsequently, the model-view matrix and projection matrix in three-dimensional space are calculated through the internal and external parameters of the camera. These matrices describe the spatial relationship between the panoramic image and the three-dimensional model. Using the model-view matrix and projection matrix, each frame in the panoramic video is reprojected to the corresponding position of the GIS three-dimensional model through projection texture technology. The key step here is to use GPU-accelerated video and model rendering to update texture data in real time to ensure that the video content on the three-dimensional model is synchronized with the actual scene. Open Scene The Graph (Open Source 3D Graph, OSG) engine is an open source, high-performance 3D graphics engine that can efficiently process complex 3D scenes and support real-time rendering and scene management. Through this engine, the 3D GIS model integrated with panoramic video can be presented to achieve intuitive visualization of the park's 3D scene model and geographic information. This process realizes the real-time dynamic fusion of panoramic video and 3D geographic information model, and finally obtains an intuitive, 3D space-based panoramic monitoring model of the target area.

[0087] In an embodiment of the present application, a plurality of surveillance images synchronously acquired by a plurality of image acquisition devices in a target area are acquired to form an image set; the following image fusion process is executed cyclically until the plurality of surveillance images are fused into a panoramic image: two images are randomly selected from the image set, one as a reference image and the other as an image to be registered; a plurality of first feature points of the reference image and a first descriptor corresponding to each first feature point, a plurality of second feature points of the image to be registered and a second descriptor corresponding to each second feature point are extracted; each first descriptor is enhanced by using a feature enhancement network to obtain a plurality of third descriptors, and each second descriptor is enhanced by using a feature enhancement network to obtain a plurality of third descriptors. The method comprises the following steps: a first enhancement is performed on the descriptor of each feature point according to the geometric information of each feature point, and a second enhancement is performed on the descriptor of each feature point according to the association information between the multiple feature points; a plurality of first feature points and a plurality of second feature points are matched according to the third descriptor and the fourth descriptor, and the image to be registered is registered according to the matching result to obtain a registered image; the reference image and the registered image are fused to obtain a fused image, and the fused image is put back into the image set; the panoramic image is analyzed by the target detection algorithm to obtain the target monitoring object in the target area. In this way, by introducing a feature enhancement network in image registration, the feature points and descriptors extracted by the directional fast and rotation invariant feature algorithms are secondary enhanced, thereby achieving fast and accurate feature matching under limited computing resources. In addition, the local homography matrix and grid division technology are used, combined with the optimization of weight coefficients, to effectively deal with the registration problems of large parallax and image overlapping areas, eliminate stitching traces and ghosting, and in the large-size image target detection module, the strategy of first segmentation, then integration and then deduplication is adopted to enhance the detection capability of small targets and reduce the missed detection rate, thereby solving the technical problems of difficult to efficiently and accurately stitch panoramic images and unsatisfactory monitoring effects in related video surveillance solutions.

[0088] Example 2

[0089] According to an embodiment of the present application, a regional panoramic monitoring device for implementing the regional panoramic monitoring method in embodiment 1 is also provided. Figure 2 As shown, the regional panoramic monitoring device at least includes: an acquisition module 21, an image fusion module 22 and an object detection module 23, wherein:

[0090] An acquisition module 21 is used to acquire a plurality of surveillance images synchronously acquired by a plurality of image acquisition devices in a target area to form an image set;

[0091] The image fusion module 22 is used to cyclically execute the following image fusion process until the multiple surveillance images are fused into a panoramic image: randomly select two images from the image set, one as the reference image and the other as the image to be registered; extract multiple first feature points of the reference image and the first descriptor corresponding to each first feature point, multiple second feature points of the image to be registered and the second descriptor corresponding to each second feature point; use the feature enhancement network to enhance each first descriptor to obtain multiple third descriptors, and enhance each second descriptor to obtain multiple fourth descriptors, wherein the feature enhancement network is used to enhance the descriptor of each feature point for the first time according to the geometric information of each feature point itself, and enhance the descriptor of each feature point for the second time according to the association information between the multiple feature points; match the multiple first feature points and the multiple second feature points according to the third descriptor and the fourth descriptor, and register the image to be registered according to the matching result to obtain a registered image; fuse the reference image and the registered image to obtain a fused image, and put the fused image back into the image set;

[0092] The object detection module 23 is used to analyze the panoramic image using a target detection algorithm to obtain the target monitoring object in the target area.

[0093] The functions of each module of the regional panoramic monitoring device are explained below in conjunction with a specific implementation process.

[0094] First, the acquisition module acquires multiple monitoring images synchronously acquired by multiple image acquisition devices in the target area to form an image set.

[0095] The image fusion module cyclically executes the following image fusion process until multiple surveillance images are fused into a panoramic image: randomly select two images from the image set, one as the reference image and the other as the image to be registered; extract multiple first feature points of the reference image and the first descriptor corresponding to each first feature point, multiple second feature points of the image to be registered and the second descriptor corresponding to each second feature point; use the feature enhancement network to enhance each first descriptor respectively to obtain multiple third descriptors, and enhance each second descriptor respectively to obtain multiple fourth descriptors, wherein the feature enhancement network is used to enhance the descriptor of each feature point for the first time according to the geometric information of each feature point itself, and enhance the descriptor of each feature point for the second time according to the association information between the multiple feature points; match the multiple first feature points and the multiple second feature points according to the third descriptor and the fourth descriptor, and register the image to be registered according to the matching result to obtain the registered image; fuse the reference image and the registered image to obtain the fused image, and put the fused image back into the image set.

[0096] Optionally, extracting multiple first feature points of the reference image and a first descriptor corresponding to each first feature point, and extracting multiple second feature points of the image to be registered and a second descriptor corresponding to each second feature point includes: using a directional fast and rotation invariant feature algorithm to extract multiple first feature points in the reference image and determine the first descriptor corresponding to each first feature point, extracting multiple second feature points in the image to be registered and determining the second descriptor corresponding to each second feature point.

[0097] Optionally, the feature enhancement network includes: a self-enhancement subnetwork and a cross-enhancement subnetwork, the self-enhancement subnetwork includes two multi-layer perceptrons, the cross-enhancement subnetwork includes a transformer based on a multi-head attention mechanism, and the feature enhancement network is used to enhance each first descriptor to obtain multiple third descriptors, and each second descriptor is enhanced to obtain multiple fourth descriptors, including: inputting multiple first feature points and corresponding first descriptors into the feature enhancement network to obtain the third descriptor corresponding to each first feature point, wherein, for each first feature point, the self-enhancement subnetwork is used to encode the geometric information of the first feature point using a multi-layer perceptron to obtain a first vector, and another multi-layer perceptron is used to project the first descriptor corresponding to the first feature point to the target space to obtain a second vector, and the first vector and the second vector are spliced ​​into a third vector; the cross-enhancement subnetwork is used to combine multiple third vectors into a first vector sequence, and determine the first query matrix corresponding to the first vector sequence. , a first key matrix and a first value matrix, and use the attention formula to calculate the first vector sequence, the first query matrix, the first key matrix and the first value matrix to obtain a third descriptor corresponding to each first feature point; multiple second feature points and the corresponding second descriptors are input into the feature enhancement network to obtain a fourth descriptor corresponding to each second feature point, wherein, for each second feature point, the self-enhancement subnetwork is used to encode the geometric information of the second feature point using a multi-layer perceptron to obtain a fourth vector, and another multi-layer perceptron is used to project the second descriptor corresponding to the second feature point to the target space to obtain a fifth vector, and the fourth vector and the fifth vector are concatenated into a sixth vector; the cross-enhancement subnetwork is used to form a second vector sequence with multiple sixth vectors, and determine the second query matrix, the second key matrix and the second value matrix corresponding to the second vector sequence, and use the attention formula to calculate the second vector sequence, the second query matrix, the second key matrix and the second value matrix to obtain the fourth descriptor corresponding to each second feature point.

[0098] Optionally, the attention formula is: In the formula, X represents the vector sequence, N represents the number of vectors in the vector sequence, and f i(X) represents the enhanced descriptor corresponding to the i-th vector in the vector sequence, Q, K, and V represent the query matrix, key matrix, and value matrix corresponding to the vector sequence, respectively. i represents the i-th row of the query matrix, K j 、V j denote the j-th row of the key matrix and the value matrix respectively.

[0099] Optionally, matching the plurality of first feature points with the plurality of second feature points is performed according to the third descriptor and the fourth descriptor, including: inputting the plurality of first feature points and the corresponding third descriptors, the plurality of second feature points and the corresponding fourth descriptors into the improved SuperGlue model, and obtaining matching results between the plurality of first feature points and the plurality of second feature points; wherein the improved SuperGlue model includes a multi-layer matching sub-model with the same structure, and each layer of the matching sub-model includes: a feature encoding module, an attention module and a classifier; the feature encoding module is used to encode and represent the geometric information of the input feature points and the corresponding descriptors; the attention module includes: a self- Attention module and cross-attention module, the self-attention module is used to enhance the representation of the encoding information of the feature points according to the correlation information between the feature points in the same image, and the cross-attention module is used to calculate the matching degree between the feature points in different images according to the encoding information after the enhanced representation of each feature point; the classifier is used to determine the matching results between the feature points in different images according to the calculation results of the cross-attention module; the output of each layer of self-attention module is used as the input of the feature encoding module of the next layer; wherein, when the self-attention module and the cross-attention module calculate any two feature points, the feature point position information used is the relative position information between the two feature points.

[0100] Optionally, each layer of classifiers is used to determine the confidence of the matching prediction results corresponding to each feature point inputted into the current layer, and determine the feature points whose confidence is greater than the first preset threshold corresponding to the number of layers of the classifier as credible feature points, wherein the earlier the number of layers of the classifier is, the smaller the corresponding first preset threshold is; determine the ratio of the number of credible feature points to the total number of input feature points; when the ratio is greater than the second preset threshold, stop the iteration of the improved SuperGlue model and output the matching results of the classifier of the current layer; when the ratio is not greater than the second preset threshold, determine the non-essential feature points whose confidence is greater than the first preset threshold and whose matching prediction results are non-essential feature points without matching feature points, and instruct the self-attention module of the current layer not to output the information of non-essential feature points to the feature encoding module of the next layer.

[0101] Optionally, the image to be registered is registered based on the matching results to obtain a registered image, including: using a direct linear transformation algorithm to analyze the position information of multiple pairs of matching first feature points and second feature points in the matching results to obtain a global homography matrix corresponding to the image to be registered; dividing the image to be registered into multiple sub-images to be registered; for each sub-image to be registered, determining the center of mass of the sub-image to be registered, and determining the weight coefficient corresponding to the sub-image to be registered based on the distance relationship between the center of mass and each second feature point, determining the local homography matrix corresponding to the sub-image to be registered based on the weight coefficient and the global homography matrix, and performing a linear transformation on the sub-image to be registered based on the local homography matrix to obtain a registered sub-image corresponding to the sub-image to be registered; and splicing multiple registered sub-images into a registered image.

[0102] Optionally, fusing the reference image and the registered image to obtain a fused image includes: determining a target stitching line when stitching the reference image and the registered image; stratifying the reference image according to a plurality of preset frequency bands to obtain a first Gaussian pyramid, and determining a first Laplacian pyramid corresponding to the first Gaussian pyramid; stratifying the registered image according to a plurality of preset frequency bands to obtain a second Gaussian pyramid, and determining a second Laplacian pyramid corresponding to the second Gaussian pyramid; fusing the same frequency band layers of the first Laplacian pyramid and the second Laplacian pyramid according to the target stitching line to obtain a fused Laplacian pyramid; and performing an inverse Laplacian transform on the fused Laplacian pyramid to obtain a fused image.

[0103] Optionally, determining a target stitching line when stitching the reference image and the registered image includes: determining a color energy function based on a color difference between the stitching line on the reference image and the registered image, and determining a structure energy function based on a gradient difference between the stitching line on the reference image and the registered image, wherein the stitching line is a quantity to be solved, and the gradient difference is a comprehensive gradient difference in the horizontal and vertical directions; determining a first weight and a second weight corresponding to the color energy function and the structure energy function, respectively, and performing a weighted summation of the color energy function and the structure energy function based on the first weight and the second weight to obtain a target energy function; and solving the target stitching line that minimizes the target energy function.

[0104] Optionally, the object detection module uses a target detection algorithm to analyze the panoramic image to obtain the target monitoring object in the target area, including: dividing the panoramic image into multiple local images; analyzing each local image using a target detection algorithm to obtain the monitoring object in each local image; and deduplicating the monitoring objects in multiple local images using a non-maximum suppression algorithm to obtain each target monitoring object in the target area.

[0105] Optionally, the method also includes: obtaining a three-dimensional scene model and geographic information corresponding to the target area; fusing the panoramic image, the three-dimensional scene model and the geographic information to obtain a panoramic monitoring model corresponding to the target area; and highlighting each target monitoring object in the panoramic monitoring model.

[0106] It should be noted that each module in the regional panoramic monitoring device in the embodiment of the present application corresponds one by one to each implementation step of the regional panoramic monitoring method in Example 1. Since a detailed description has been given in Example 1, some details not reflected in this embodiment can be referred to Example 1 and will not be elaborated here.

[0107] Example 3

[0108] According to an embodiment of the present application, a computer program product is also provided, which includes a computer program, wherein when the computer program is executed by a processor, the regional panoramic monitoring method in Example 1 is implemented.

[0109] According to an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the regional panoramic monitoring method in Example 1 by running the computer program.

[0110] According to an embodiment of the present application, a processor is also provided, which is used to run a computer program, wherein the regional panoramic monitoring method in Example 1 is executed when the computer program is running.

[0111] According to an embodiment of the present application, an electronic device is also provided, which includes: a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the regional panoramic monitoring method in Example 1 through the computer program.

[0112] Specifically, the computer program executes the following steps when it is running: obtaining a plurality of surveillance images synchronously acquired by a plurality of image acquisition devices in a target area to form an image set; looping through the following image fusion process until the plurality of surveillance images are fused into a panoramic image: randomly selecting two images from the image set, one as a reference image and the other as an image to be registered; extracting a plurality of first feature points of the reference image and a first descriptor corresponding to each first feature point, a plurality of second feature points of the image to be registered and a second descriptor corresponding to each second feature point; using a feature enhancement network to enhance each first descriptor, respectively, to obtain a plurality of third descriptors, and performing an enhanced representation on each first descriptor, respectively, for each The second descriptor is enhanced to represent the fourth descriptor, wherein the feature enhancement network is used to perform a first enhancement on the descriptor of each feature point according to the geometric information of each feature point itself, and to perform a second enhancement on the descriptor of each feature point according to the association information between the plurality of feature points; the plurality of first feature points and the plurality of second feature points are matched according to the third descriptor and the fourth descriptor, and the image to be registered is registered according to the matching result to obtain a registered image; the reference image and the registered image are fused to obtain a fused image, and the fused image is put back into the image set; the panoramic image is analyzed by using a target detection algorithm to obtain a target monitoring object in the target area.

[0113] As an optional implementation, the electronic device may be in the form of a mobile terminal, a computer terminal or a similar computing device. Figure 3 FIG. 1 shows a hardware structure block diagram of an electronic device for implementing a method for panoramic monitoring of an area. Figure 3 As shown, the electronic device 30 may include one or more (302a, 302b, ..., 302n are used to illustrate) processors 302 (the processor 302 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 304 for storing data, and a transmission device 306 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 3 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 3 More or fewer components as shown, or with Figure 3 Different configurations shown.

[0114] It should be noted that the one or more processors 302 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the electronic device 30. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0115] The memory 304 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the regional panoramic monitoring method in the embodiment of the present application. The processor 302 executes various functional applications and data processing by running the software programs and modules stored in the memory 304, that is, realizing the vulnerability detection method of the above-mentioned application. The memory 304 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 304 may further include a memory remotely arranged relative to the processor 302, and these remote memories may be connected to the electronic device 30 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0116] The transmission device 306 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the electronic device 30. In one example, the transmission device 306 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 306 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0117] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the electronic device 30 .

[0118] The serial numbers of the above embodiments are only for description and do not represent the advantages or disadvantages of the embodiments.

[0119] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0120] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0121] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0122] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0123] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0124] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for panoramic regional monitoring, characterized in that: include: Acquire multiple surveillance images synchronously acquired by multiple image acquisition devices in the target area to form an image set; The following image fusion process is executed cyclically until the multiple surveillance images are fused into a panoramic image: two images are randomly selected from the image set, one as a reference image and the other as an image to be registered; Extracting a plurality of first feature points of the reference image and a first descriptor corresponding to each of the first feature points, and a plurality of second feature points of the image to be registered and a second descriptor corresponding to each of the second feature points; Using a feature enhancement network to enhance each of the first descriptors to obtain a plurality of third descriptors, and enhancing each of the second descriptors to obtain a plurality of fourth descriptors, wherein the feature enhancement network is used to enhance the descriptor of each feature point for the first time according to the geometric information of each feature point itself, and to enhance the descriptor of each feature point for the second time according to the association information between the plurality of feature points; matching the plurality of first feature points and the plurality of second feature points according to the third descriptor and the fourth descriptor, and registering the images to be registered according to the matching results to obtain a registered image; fusing the reference image and the registered image to obtain a fused image, and placing the fused image back into the image set; The panoramic image is analyzed using a target detection algorithm to obtain the target monitoring object in the target area.

2. The method according to claim 1, characterized in that Extracting a plurality of first feature points of the reference image and a first descriptor corresponding to each of the first feature points, and a plurality of second feature points of the image to be registered and a second descriptor corresponding to each of the second feature points, comprises: A plurality of the first feature points in the reference image are extracted using a directional fast and rotation invariant feature algorithm, and the first descriptor corresponding to each of the first feature points is determined; a plurality of the second feature points in the image to be registered are extracted, and the second descriptor corresponding to each of the second feature points is determined.

3. The method according to claim 1, characterized in that The feature enhancement network includes: a self-enhancement sub-network and a cross-enhancement sub-network, the self-enhancement sub-network includes two multi-layer perceptrons, the cross-enhancement sub-network includes a transformer based on a multi-head attention mechanism, and the feature enhancement network is used to enhance each of the first descriptors to obtain multiple third descriptors, and each of the second descriptors is enhanced to obtain multiple fourth descriptors, including: Input a plurality of the first feature points and the corresponding first descriptors into the feature enhancement network to obtain the third descriptor corresponding to each of the first feature points, wherein, for each of the first feature points, the self-enhancement subnetwork is used to encode the geometric information of the first feature point using a multi-layer perceptron to obtain a first vector, and use another multi-layer perceptron to project the first descriptor corresponding to the first feature point to the target space to obtain a second vector, and concatenate the first vector and the second vector into a third vector; the cross-enhancement subnetwork is used to form a first vector sequence with a plurality of the third vectors, and determine the first query matrix, the first key matrix and the first value matrix corresponding to the first vector sequence, and use the attention formula to calculate the first vector sequence, the first query matrix, the first key matrix and the first value matrix to obtain the third descriptor corresponding to each of the first feature points; Input multiple second feature points and corresponding second descriptors into the feature enhancement network to obtain the fourth descriptor corresponding to each second feature point, wherein, for each second feature point, the self-enhancement subnetwork is used to encode the geometric information of the second feature point using a multilayer perceptron to obtain a fourth vector, and use another multilayer perceptron to project the second descriptor corresponding to the second feature point to the target space to obtain a fifth vector, and concatenate the fourth vector and the fifth vector into a sixth vector; the cross-enhancement subnetwork is used to form a second vector sequence with multiple sixth vectors, and determine the second query matrix, second key matrix and second value matrix corresponding to the second vector sequence, and use the attention formula to calculate the second vector sequence, the second query matrix, the second key matrix and the second value matrix to obtain the fourth descriptor corresponding to each second feature point.

4. The method according to claim 3, characterized in that The attention formula is: In the formula, X represents the vector sequence, N represents the number of vectors in the vector sequence, and f i (X) represents the enhanced descriptor corresponding to the i-th vector in the vector sequence, Q, K, and V represent the query matrix, key matrix, and value matrix corresponding to the vector sequence, respectively. i represents the i-th row of the query matrix, K j 、V j denote the j-th row of the key matrix and the value matrix respectively.

5. The method according to claim 1, characterized in that Matching the plurality of first feature points and the plurality of second feature points according to the third descriptor and the fourth descriptor includes: Inputting the plurality of first feature points and the corresponding third descriptors, the plurality of second feature points and the corresponding fourth descriptors into the improved SuperGlue model, and obtaining matching results between the plurality of first feature points and the plurality of second feature points; Among them, the improved SuperGlue model includes multiple layers of matching sub-models with the same structure, and each layer of the matching sub-model includes: a feature encoding module, an attention module and a classifier; the feature encoding module is used to encode and represent the geometric information of the input feature points and the corresponding descriptors; the attention module includes: a self-attention module and a cross-attention module, the self-attention module is used to enhance the encoding information of the feature points according to the association information between the feature points in the same image, and the cross-attention module is used to calculate the matching degree between the feature points in different images according to the encoding information after the enhanced representation of each feature point; the classifier is used to determine the matching result between the feature points in different images according to the calculation result of the cross-attention module; the output of the self-attention module of each layer is used as the input of the feature encoding module of the next layer; Among them, when the self-attention module and the cross-attention module calculate any two feature points, the feature point position information used is the relative position information between the two feature points.

6. The method according to claim 5, characterized in that The classifier in each layer is used to determine the confidence of the matching prediction results corresponding to each feature point input in the current layer, and determine the feature points whose confidence is greater than the first preset threshold corresponding to the number of layers of the classifier as credible feature points, wherein the earlier the number of layers of the classifier is, the smaller the corresponding first preset threshold is; determine the ratio of the number of credible feature points to the total number of input feature points; when the ratio is greater than the second preset threshold, stop the iteration of the improved SuperGlue model and output the matching results of the classifier in the current layer; when the ratio is not greater than the second preset threshold, determine the non-essential feature points whose confidence is greater than the first preset threshold and whose matching prediction results are non-essential feature points without matching feature points, and instruct the self-attention module in the current layer not to output the information of the non-essential feature points to the feature encoding module in the next layer.

7. The method according to claim 1, characterized in that Registering the image to be registered according to the matching result to obtain a registered image includes: Using a direct linear transformation algorithm to analyze the position information of multiple pairs of matching first feature points and second feature points in the matching results, to obtain a global homography matrix corresponding to the image to be registered; Dividing the image to be registered into a plurality of sub-images to be registered; For each of the sub-images to be registered, determine the centroid of the sub-image to be registered, and determine the weight coefficient corresponding to the sub-image to be registered according to the distance relationship between the centroid and each of the second feature points, determine the local homography matrix corresponding to the sub-image to be registered according to the weight coefficient and the global homography matrix, and perform a linear transformation on the sub-image to be registered according to the local homography matrix to obtain a registered sub-image corresponding to the sub-image to be registered; A plurality of the registered sub-images are stitched together into the registered image.

8. The method according to claim 1, characterized in that Fusing the reference image and the registered image to obtain a fused image, comprising: Determining a target stitching line when stitching the reference image and the registered image; The reference image is layered according to a plurality of preset frequency bands to obtain a first Gaussian pyramid, and a first Laplacian pyramid corresponding to the first Gaussian pyramid is determined; Layering the registered image according to a plurality of preset frequency bands to obtain a second Gaussian pyramid, and determining a second Laplacian pyramid corresponding to the second Gaussian pyramid; Fusing the same frequency band layers of the first Laplacian pyramid and the second Laplacian pyramid according to the target splicing line to obtain a fused Laplacian pyramid; Performing an inverse Laplace transform on the fused Laplace pyramid to obtain the fused image.

9. The method according to claim 8, characterized in that Determining a target stitching line when stitching the reference image and the registered image comprises: Determine a color energy function according to a color difference of a stitching line on the reference image and the registration image, and determine a structure energy function according to a gradient difference of the stitching line on the reference image and the registration image, wherein the stitching line is a quantity to be solved, and the gradient difference is a comprehensive gradient difference in the horizontal direction and the vertical direction; Determine a first weight and a second weight corresponding to the color energy function and the structure energy function respectively, and perform weighted summation of the color energy function and the structure energy function according to the first weight and the second weight to obtain a target energy function; Solve the target splicing line that minimizes the target energy function.

10. The method according to claim 1, characterized in that Analyzing the panoramic image using a target detection algorithm to obtain a target monitoring object in the target area includes: Dividing the panoramic image into a plurality of local images; Analyzing each of the local images using a target detection algorithm to obtain a monitored object in each of the local images; A non-maximum suppression algorithm is used to perform deduplication processing on the monitoring objects in the multiple local images to obtain each of the target monitoring objects in the target area.

11. The method according to claim 1, characterized in that: The method further comprises: Acquire a three-dimensional scene model and geographic information corresponding to the target area; Fusing the panoramic image, the three-dimensional scene model and the geographic information to obtain a panoramic monitoring model corresponding to the target area; Each of the target monitoring objects is highlighted in the panoramic monitoring model.

12. A regional panoramic monitoring device, characterized in that: include: An acquisition module is used to acquire multiple surveillance images synchronously acquired by multiple image acquisition devices in a target area to form an image set; The image fusion module is used to cyclically execute the following image fusion process until the multiple monitoring images are fused into a panoramic image: randomly taking two images from the image set, one as a reference image and the other as an image to be registered; Extracting a plurality of first feature points of the reference image and a first descriptor corresponding to each of the first feature points, and a plurality of second feature points of the image to be registered and a second descriptor corresponding to each of the second feature points; Using a feature enhancement network to enhance each of the first descriptors to obtain a plurality of third descriptors, and enhancing each of the second descriptors to obtain a plurality of fourth descriptors, wherein the feature enhancement network is used to enhance the descriptor of each feature point for the first time according to the geometric information of each feature point itself, and to enhance the descriptor of each feature point for the second time according to the association information between the plurality of feature points; matching the plurality of first feature points and the plurality of second feature points according to the third descriptor and the fourth descriptor, and registering the images to be registered according to the matching results to obtain a registered image; fusing the reference image and the registered image to obtain a fused image, and placing the fused image back into the image set; The object detection module is used to analyze the panoramic image using a target detection algorithm to obtain the target monitoring object in the target area.

13. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, the method for panoramic area monitoring described in any one of claims 1 to 11 is implemented.

14. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the regional panoramic monitoring method described in any one of claims 1 to 11 through the computer program.

Images