City appearance comparison detection system and method, computing equipment and storage medium

Abstract

The invention discloses a city appearance comparison detection system and method, computing equipment and a storage medium. The city appearance comparison detection system comprises a plurality of offline acquisition ends and a cloud processing end, each off-line acquisition end comprises a plurality of city cameras and / or patrol vehicle-mounted cameras and a posture storage library, the city cameras and / or patrol vehicle-mounted cameras are / is used for acquiring real-time camera posture data and real-time image data, and the posture storage library is used for storing detection posture data of the off-line acquisition ends; the cloud processing end comprises a comparison detection module and a data storage library, and the comparison detection module is used for comparing and detecting real-time image data and historical image data under the camera pose data matched with the detection pose data and judging whether a condition influencing the city appearance exists or not; and the data storage library is used for storing detection pose data and historical image data of each off-line acquisition end. According to the invention, open set detection of three-dimensional scene changes can be carried out, and high-precision detection can be carried out on the condition that the city appearance is damaged.

Description

Technical Field

[0001] The present invention belongs to the field of computer deep learning technology, and specifically relates to a city appearance comparison detection system, method, computing device and storage medium. Background Art

[0002] With the acceleration of urbanization, the management of city appearance has become an important part of urban management. In particular, the cleanliness of advertising signs, store facades, and public streets in front of stores on the street directly affects the overall image of the city and the quality of life of residents. However, many stores have illegally occupied public areas, changed signs without permission, and the content of advertisements does not comply with regulations, which has brought huge challenges to municipal management. Traditional management methods mainly rely on manual inspections, which are inefficient and prone to omissions. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes a city appearance comparison detection system, method, computing device and storage medium.

[0004] In order to achieve the above object, the technical solution of the present invention is as follows:

[0005] In a first aspect, the present invention discloses a city appearance comparison detection system, including: a plurality of offline collection terminals and a cloud processing terminal;

[0006] Each offline collection end includes: a number of city cameras and / or patrol car cameras and a posture repository, the city cameras and / or patrol car cameras are used to collect real-time camera posture data and real-time image data, and the posture repository is used to store the detection posture data of the offline collection end;

[0007] The cloud processing end includes: a comparison detection module and a data repository, wherein the comparison detection module is used to compare and detect the real-time image data and the historical image data under the camera posture data matching the detection posture data to determine whether there is a situation that affects the city appearance, and the data repository is used to store the detection posture data and the historical image data of each offline acquisition end;

[0008] When and only when the real-time camera pose data collected by the offline acquisition end matches the detection pose data, the real-time image data corresponding to the real-time camera pose data is uploaded to the cloud processing end, replacing the historical image data corresponding to the detection pose data in the updated data repository, and the comparison detection module compares the real-time image data and the corresponding historical image data.

[0009] Based on the above technical solution, the following improvements can be made:

[0010] As a preferred solution, the patrol car is also equipped with a radar, which is used to collect real-time point cloud data.

[0011] As a preferred solution, the data repository is used to store the detection posture data, historical image data and historical point cloud data of each offline acquisition terminal.

[0012] As a preferred solution, the detected posture data in the posture storage repository of the offline acquisition end and the detected posture data in the data storage repository of the cloud processing end are updated synchronously.

[0013] As a preferred solution, the comparison detection module includes:

[0014] A feature extraction encoder is used to extract features from real-time image data and historical image data to obtain feature maps Ft1 and Ft2 respectively;

[0015] The context information module is used to perform multi-level hierarchical upsampling fusion on the feature map Ft1 and the feature map Ft2 to obtain the feature map Ftp1 and the feature map Ftp2 respectively;

[0016] A depth estimation module is used to process the feature map Ftp1 and the feature map Ftp2 through a depth estimation subnetwork to obtain a feature map Dt1 and a feature map Dt2 respectively;

[0017] A feature fusion module is used to concatenate the feature maps Ft1 and Ft2 with the feature maps Dt1 and Dt2 to obtain fused features |Ft1, Dt2| and fused features |Ft2, Dt2| respectively;

[0018] The contrast learning module is used to map the fused features |Ft1, Dt2| and the fused features |Ft2, Dt2| to a low-dimensional feature space to obtain two low-dimensional feature vectors, and obtain a similarity graph C by calculating the cosine similarity between the two low-dimensional feature vectors point by point;

[0019] The output module is used to upsample the similarity map C to the size of the original image data and output it.

[0020] As a preferred solution, the comparison detection module also includes:

[0021] The instantiation module is used to combine the feature map Dt1 and feature map Dt2 obtained by the depth estimation module and the change area features obtained by the output module, split the instance of the change area through the 3D connected domain algorithm, and use the minimum box algorithm to calculate the minimum 3D box of each split change area to identify the change 3D area.

[0022] In a second aspect, the present invention discloses a city appearance comparison detection method, which uses any of the above-mentioned city appearance comparison detection systems for detection, including:

[0023] The city cameras and / or patrol car cameras at the offline collection end collect real-time camera position data and real-time image data;

[0024] The real-time camera pose data is queried and matched with the detection pose data stored in the pose repository of the offline acquisition end.

[0025] When the match is successful, the real-time camera posture data is uploaded to the cloud processing end, and a secondary query match is performed with the detection posture data stored in the data repository of the cloud processing end. If the match is also successful, the cloud processing end sends an upload signal to the corresponding offline acquisition end;

[0026] The offline acquisition end uploads the real-time image data corresponding to the real-time camera posture data to the cloud processing end, replacing and updating the historical image data corresponding to the detection posture data in the data repository;

[0027] The comparison detection module on the cloud processing end compares and detects historical image data with real-time image data to determine whether there are any situations that affect the city appearance.

[0028] As a preferred solution, at fixed intervals, the detection posture data in the data repository of the cloud processing end and the detection posture data of each offline collection end are synchronously updated.

[0029] In a third aspect, the present invention discloses a computing device, comprising:

[0030] one or more processors;

[0031] Memory;

[0032] And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions of any of the above-mentioned city appearance comparison detection methods.

[0033] In a fourth aspect, the present invention further discloses a storage medium storing one or more computer-readable programs, wherein the one or more programs include instructions suitable for being loaded by a memory and executing any of the above-mentioned city appearance comparison detection methods.

[0034] The present invention discloses a city appearance comparison detection system, method, computing device and storage medium, which have the following beneficial effects:

[0035] First, the city appearance comparison detection system of the present invention deploys multiple offline collection terminals and cloud processing terminals, separates data collection to the offline collection terminal, stores and calculates large quantities of data to the cloud processing terminal, and synchronizes the detection posture data in the posture repository and the data repository to realize index update between the two, thereby realizing low-cost city appearance detection.

[0036] Second, the present invention utilizes computer vision technology, combined with contrastive learning change detection networks and depth estimation to compare real-time image data and historical image data, to perform open set detection of three-dimensional scene changes, and to perform high-precision detection of situations that damage the city appearance, such as unauthorized changes to business sign logos and illegal occupation of public street areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 A block diagram of a city appearance comparison detection system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] The expression of “comprising” an element is an “open” expression, which merely means that a corresponding component or step exists, and should not be interpreted as excluding additional components or steps.

[0042] In order to achieve the purpose of the present invention, some embodiments of a city appearance comparison detection system, method, computing device and storage medium include: Figure 1 As shown, the city appearance comparison detection system includes: several offline collection terminals and a cloud processing terminal;

[0043] Each offline collection end includes: a number of city cameras, patrol car cameras and a posture repository, the city cameras and patrol car cameras are used to collect real-time camera posture data and real-time image data, and the posture repository is used to store the detection posture data of the offline collection end;

[0044] The cloud processing end includes: a comparison detection module and a data repository, wherein the comparison detection module is used to compare and detect the real-time image data and the historical image data under the camera posture data matching the detection posture data to determine whether there is a situation that affects the city appearance, and the data repository is used to store the detection posture data and the historical image data of each offline acquisition end;

[0045] When and only when the real-time camera pose data collected by the offline acquisition end matches the detection pose data, the real-time image data corresponding to the real-time camera pose data is uploaded to the cloud processing end, replacing the historical image data corresponding to the detection pose data in the updated data repository, and the comparison detection module compares the real-time image data and the corresponding historical image data.

[0046] The above camera pose data includes internal and external parameter coefficients.

[0047] The above-mentioned comparison detection module is based on depth estimation and change detection network, which can compare real-time image data and historical image data, and automatically detect phenomena that damage the city appearance, such as unauthorized changes to business sign logos and illegal occupation of public street areas.

[0048] The above data storage database consists of a Rectangle Tree dataset structure, which is suitable for storage by location and index search.

[0049] In the above embodiment, the position of the city camera is fixed, while the position of the patrol car camera can change as the vehicle travels. Of course, in other embodiments, the position of the city camera can also change, which is not limited here.

[0050] Furthermore, in some embodiments, the patrol car is also equipped with a radar, which is used to collect real-time point cloud data.

[0051] Furthermore, in some embodiments, the data repository is used to store the detection posture data, historical image data, and historical point cloud data of each offline acquisition terminal.

[0052] Furthermore, in some embodiments, the detected posture data in the posture storage repository of the offline acquisition end and the detected posture data in the data storage repository of the cloud processing end are synchronously updated.

[0053] Further, in some embodiments, the contrast detection module is based on a depth estimation and change detection deep learning network, including: a feature extraction encoder, a context information module, a depth estimation module, a feature fusion module, a contrast learning module and an output module.

[0054] The feature extraction encoder is used to extract features from real-time image data and historical image data to obtain feature maps Ft1 and Ft2 respectively;

[0055] Specifically, the real-time image data and the historical image data are used as input, and a convolutional neural network with shared weights is used as the backbone encoder to obtain the feature maps Ft1 and Ft2 of the two images respectively. The encoder selects the existing network ResNet50, which will not be described here.

[0056] The context information module is used to perform multi-level hierarchical upsampling fusion on the feature map Ft1 and the feature map Ft2 using the FPN structure to obtain a high-resolution feature map Ftp1 and a high-resolution feature map Ftp2, respectively.

[0057] The depth estimation module is used to process the feature map Ftp1 and the feature map Ftp2 through the depth estimation subnetwork to obtain the feature map Dt1 and the feature map Dt2 respectively;

[0058] Specifically, the depth estimation subnetwork can adopt the existing network DepthNet structure, which will not be described here. In addition, the uploaded updated LiDAR point cloud data can be used to train this module, and the LiDAR point cloud data is projected onto the image data collected by the camera for supervised training of the depth estimation module.

[0059] The feature fusion module is used to splice the feature maps Ft1, Ft2 with the feature maps Dt1, Dt2 to obtain the fused features |Ft1, Dt2| and the fused features |Ft2, Dt2| respectively. The above fused features contain the texture, structure and depth information of the scene, making the perception of 3D changes more accurate.

[0060] The contrastive learning module is used to map the fused features |Ft1, Dt2| and the fused features |Ft2, Dt2| to a low-dimensional feature space to obtain two low-dimensional feature vectors, and to identify the 3D changes of the current image scene at different time points by calculating the cosine similarity between the two low-dimensional feature vectors point by point to obtain the similarity graph C.

[0061] The output module is used to upsample the similarity map C to the size of the original image data through a bilinear interpolation algorithm and output it.

[0062] Furthermore, in some embodiments, the comparison detection module also includes: an instantiation module.

[0063] The instantiation module is used to combine the feature map Dt1 and feature map Dt2 obtained by the depth estimation module and the change area features obtained by the output module, split the instance of the change area through the 3D connected domain algorithm, and use the minimum box algorithm to calculate the minimum 3D box of each split change area to identify the change 3D area.

[0064] In addition, in some embodiments, the present invention discloses a city appearance comparison detection method, which uses any of the above-mentioned city appearance comparison detection systems to perform detection, including:

[0065] The position and posture of the city camera at the offline collection end is fixed, and the real-time camera position and posture data and real-time image data are collected at regular intervals (in some embodiments, the real-time camera position and posture data and real-time image data can be collected at the same time every day);

[0066] Patrol cars equipped with radar and on-board cameras collect real-time camera position data, real-time image data, and real-time point cloud data at regular intervals. The patrol car's camera position data changes as the car drives.

[0067] The real-time camera pose data is queried and matched with the detection pose data stored in the pose repository of the offline acquisition end.

[0068] When the match is successful, the real-time camera posture data is uploaded to the cloud processing end, and a secondary query match is performed with the detection posture data stored in the data repository of the cloud processing end. If the match is also successful, the cloud processing end sends an upload signal to the corresponding offline acquisition end;

[0069] The offline acquisition end uploads the real-time image data corresponding to the real-time camera posture data to the cloud processing end, replaces the historical image data corresponding to the detection posture data in the updated data repository, and pops up the original matching point data (in addition, whether it is in the cloud end or the offline acquisition end, the data points just queried for matching and updating are placed in a cold state within a specified time range and are no longer used for query matching and updating);

[0070] The real-time image data just uploaded and the historical image data in the original matching point data that pops up are combined into an image pair as the input of the subsequent change detection and depth estimation neural network. The comparison detection module on the cloud processing end compares the historical image data with the real-time image data to determine whether there are any situations that affect the city appearance.

[0071] Furthermore, at fixed intervals, the detection posture data in the data repository of the cloud processing end is synchronously updated with the detection posture data of each offline acquisition end.

[0072] The present invention combines positioning information, image data, and radar point cloud data, and detects street-side stores illegally occupying public roads, unauthorized changes in signboards, and other phenomena based on city cameras and patrol car-mounted cameras, thereby achieving comparative management of the city appearance.

[0073] In some other embodiments, the present invention discloses a computing device, including:

[0074] one or more processors;

[0075] Memory;

[0076] And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions of any of the above-mentioned city appearance comparison detection methods.

[0077] In addition, the present invention also discloses a storage medium, which stores one or more computer-readable programs, and the one or more programs include instructions, and the instructions are suitable for being loaded by a memory and executing any of the above-mentioned city appearance comparison detection methods.

[0078] The present invention discloses a city appearance comparison detection system, method, computing device and storage medium, which have the following beneficial effects:

[0079] First, the city appearance comparison detection system of the present invention deploys multiple offline collection terminals and cloud processing terminals, separates data collection to the offline collection terminal, stores and calculates large quantities of data to the cloud processing terminal, and synchronizes the detection posture data in the posture repository and the data repository to realize index update between the two, thereby realizing low-cost city appearance detection.

[0080] Second, the present invention utilizes computer vision technology, combined with contrastive learning change detection networks and depth estimation to compare real-time image data and historical image data, to perform open set detection of three-dimensional scene changes, and to perform high-precision detection of situations that damage the city appearance, such as unauthorized changes to business sign logos and illegal occupation of public street areas.

[0081] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which shall fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. City appearance comparison detection system, characterized in that: include: Several offline collection terminals and cloud processing terminals; Each of the offline acquisition terminals comprises: a plurality of city cameras and / or patrol car cameras and a posture repository, wherein the city cameras and / or patrol car cameras are used to collect real-time camera posture data and real-time image data, and the posture repository is used to store the detection posture data of the offline acquisition terminal; The cloud processing end includes: a comparison detection module and a data storage library, wherein the comparison detection module is used to compare and detect real-time image data and historical image data under camera posture data that matches the detection posture data to determine whether there is a situation that affects the city appearance, and the data storage library is used to store the detection posture data and historical image data of each offline acquisition end; When and only when the real-time camera posture data collected by the offline collection end matches the detection posture data, the real-time image data corresponding to the real-time camera posture data is uploaded to the cloud processing end, replacing the historical image data corresponding to the detection posture data in the updated data repository, and the comparison detection module compares the real-time image data and the corresponding historical image data.

2. The city appearance comparison detection system according to claim 1, characterized in that: The patrol car is also equipped with a radar, which is used to collect real-time point cloud data.

3. The city appearance comparison detection system according to claim 2, characterized in that: The data repository is used to store the detection posture data, historical image data and historical point cloud data of each offline acquisition terminal.

4. The city appearance comparison detection system according to claim 1, characterized in that: The detected posture data in the posture storage repository of the offline acquisition end is synchronously updated with the detected posture data in the data storage repository of the cloud processing end.

5. The city appearance comparison detection system according to any one of claims 1 to 4, characterized in that: The comparison detection module comprises: A feature extraction encoder is used to extract features from real-time image data and historical image data to obtain feature maps Ft1 and Ft2 respectively; The context information module is used to perform multi-level hierarchical upsampling fusion on the feature map Ft1 and the feature map Ft2 to obtain the feature map Ftp1 and the feature map Ftp2 respectively; A depth estimation module is used to process the feature map Ftp1 and the feature map Ftp2 through a depth estimation subnetwork to obtain a feature map Dt1 and a feature map Dt2 respectively; A feature fusion module is used to concatenate the feature maps Ft1 and Ft2 with the feature maps Dt1 and Dt2 to obtain fused features |Ft1, Dt2| and fused features |Ft2, Dt2| respectively; The contrast learning module is used to map the fused features |Ft1, Dt2| and the fused features |Ft2, Dt2| to a low-dimensional feature space to obtain two low-dimensional feature vectors, and obtain a similarity graph C by calculating the cosine similarity between the two low-dimensional feature vectors point by point; The output module is used to upsample the similarity map C to the size of the original image data and output it.

6. The city appearance comparison detection system according to claim 5, characterized in that: The comparison detection module also includes: The instantiation module is used to combine the feature map Dt1 and feature map Dt2 obtained by the depth estimation module and the change area features obtained by the output module, split the instance of the change area through the 3D connected domain algorithm, and use the minimum box algorithm to calculate the minimum 3D box of each split change area to identify the change 3D area.

7. A city appearance comparison detection method, characterized in that: The detection is performed using the city appearance comparison detection system as described in any one of claims 1 to 6, comprising: The city cameras and / or patrol car cameras at the offline collection end collect real-time camera position data and real-time image data; The real-time camera pose data is queried and matched with the detection pose data stored in the pose repository of the offline acquisition end. When the match is successful, the real-time camera posture data is uploaded to the cloud processing end, and a secondary query match is performed with the detection posture data stored in the data repository of the cloud processing end. If the match is also successful, the cloud processing end sends an upload signal to the corresponding offline acquisition end; The offline acquisition end uploads the real-time image data corresponding to the real-time camera posture data to the cloud processing end, replacing and updating the historical image data corresponding to the detection posture data in the data repository; The comparison detection module on the cloud processing end compares and detects historical image data with real-time image data to determine whether there are any situations that affect the city appearance.

8. The city appearance comparison detection method according to claim 7, characterized in that: At regular intervals, the detection pose data in the data repository of the cloud processing end is synchronized with the detection pose data of each offline acquisition end.

9. A computing device, characterized in that include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and one or more of the programs include instructions of the urban appearance comparison detection method described in claim 7 or 8 above.

10. A storage medium, characterized in that The storage medium stores one or more computer-readable programs, and the one or more programs include instructions, and the instructions are suitable for being loaded by the memory and executing the urban appearance comparison detection method described in claim 7 or 8 above.

Images