A method and system for positioning a patrol scene

Abstract

The present invention relates to the field of intelligent transportation technology, and in particular to a method and system for positioning patrol scenes, the method comprising: real-time acquisition of image data and positioning data of patrol scenes; semantic segmentation and feature extraction of the image data to obtain scene classification results and feature vectors; defect identification based on the image data, scene classification results, and feature vectors, and if a defect is identified, a hybrid search is performed in a preset scene vector database based on the positioning data and feature vectors to obtain a hybrid search result of the patrol scene; the hybrid search result is updated to the scene vector database to complete the positioning of the defect of the patrol scene. The present invention uses an edge-cloud combined system architecture and utilizes an artificial intelligence image analysis algorithm to achieve accurate and rapid scene positioning of patrol equipment.

Description

Technical Field

[0001] The present invention relates to the fields of intelligent transportation technology, intelligent road patrol and detection technology, and in particular to a patrol scene positioning method and system. Background Art

[0002] As the peak of infrastructure construction has passed, urban elevated roads, urban expressways, highways, tunnels and other road infrastructure are facing huge pressure of management and maintenance. After a certain number of years of service, facilities such as pavement and bridge structures will have various defects and diseases under the influence of multiple factors such as temperature, rain, and load. Minor defects will affect the appearance of the facilities, while serious defects will not only affect the appearance, but also affect the safety of slow traffic.

[0003] To extend the service life of road infrastructure, the most important thing is to improve the ability to detect diseases and events on the infrastructure, and accurately describe the location, size, type and other information of the disease. Among them, positioning data is an important information of the disease. After the maintenance team obtains the positioning data, it can find the corresponding disease and event according to the positioning and carry out repairs. In addition, positioning data can also help further analysis and application: positioning data can be used to analyze the spatial distribution of diseases and events, such as analyzing the clustering density and distribution patterns of diseases and events in different spaces; it can also analyze the time and category of diseases and events at the same location, perform spatial deduplication on diseases reported multiple times at the same location, or analyze whether the location is in a state of "frequent repair and frequent supplementation".

[0004] At present, routine inspections of road facilities are mainly conducted by inspectors on foot or in a car through visual inspection. When a disease or incident is found during the inspection, the inspector collects images of the disease or incident through a mobile phone APP, and uses the positioning function of the mobile phone to record and report the location data of the point. With the advancement of technology, some inspections have begun to use professional inspection equipment to collect images, use artificial intelligence algorithms to identify diseases and incidents, and use high-precision positioning sensors to obtain accurate location information, reduce the error of positioning data, and improve positioning accuracy.

[0005] The positioning data collected during routine inspections are usually two-dimensional longitude and latitude data, and can be displayed on an electronic map as dots, and the road section location can be matched according to the longitude and latitude. However, in special cases such as elevated-ground roads, tunnels-ground roads, main lines-ramp roads, there is an overlap between the upper and lower layers at the same longitude and latitude. Simply using longitude and latitude cannot determine whether the current location belongs to the upper or lower layer, and cannot achieve accurate positioning and road section matching, which may lead to problems in identifying dispatch orders. At present, some existing solutions and methods include: (1) Manual recording: During inspections and tests, the inspector observes and records the current inspection scene location on the APP, or adds a switch to the inspection device, and the inspector clicks the switch to record the inspection scenes corresponding to different inspection times. This method mainly relies on manual subjective judgment and operation, and is easily affected by the inspector's supervisor, resulting in inaccurate data. (2) Collect trajectory, altitude and acceleration data: Add high-precision navigation and positioning receivers and three-axis acceleration sensors to the patrol equipment to record the trajectory, speed, altitude and three-axis acceleration data of the patrol vehicle. Use the vehicle's trajectory and three-axis acceleration data to mark the vehicle's driving route and uphill and downhill time points. According to the vehicle's altitude and driving status, determine which layer of the overlapping road the vehicle may currently be on. This method relies more on the acquisition of sensor data and basic data: On the one hand, when the satellite signal is not good, the collected trajectory, speed, altitude and other data have a large deviation, and may even be invalid data, which leads to inaccurate positioning; on the other hand, the altitude distinction threshold is different for the same facility, and the altitude distinction threshold of elevated roads and tunnels is difficult to obtain, which interferes with accurate scene distinction and subsequent analysis. (3) Use classification neural network to process data: Some patrol equipment is equipped with cameras, and the images collected by the camera are input into the classification neural network, which determines the scene to which the image belongs. Although this method can make a judgment on the image scene, the image is only classified into limited categories and cannot further distinguish different individuals. For example, if the auxiliary roads of two bridges of the same bridge are very close to each other, have the same direction, and are both bridge auxiliary roads, the results of the image classification model alone cannot distinguish between the two roads, which leads to inaccurate road matching. Summary of the invention

[0006] The purpose of the present invention is to provide a method and system for positioning patrol scenes, which uses an edge-cloud combined system architecture and an artificial intelligence image analysis algorithm to achieve accurate and rapid scene positioning of patrol equipment. At the same time, a vector database is used to create feature files for road scenes, thereby improving the accuracy of image scene classification, which is helpful for subsequent further data matching and analysis.

[0007] To achieve the above object, the present invention provides the following solutions:

[0008] A method for positioning a patrol scene, comprising:

[0009] Collect image data and positioning data of inspection scenes in real time;

[0010] Performing semantic segmentation and feature extraction on the image data to obtain scene classification results and feature vectors;

[0011] Performing disease identification based on the image data, scene classification results, and feature vectors, and if a disease is identified, performing a mixed search in a preset scene vector database based on the positioning data and feature vectors to obtain a mixed search result of the inspection scene;

[0012] The mixed search results are updated to the scene vector database to complete the disease location of the inspection scene.

[0013] Optionally, performing semantic segmentation and feature extraction on the image data to obtain a scene classification result and a feature vector includes:

[0014] Input the original image into the semantic segmentation neural network model M1 and output the semantic segmentation image;

[0015] Combining and splicing the semantic segmentation image with the original image to obtain a combined image;

[0016] The combined image is input into the neural network model M2 to obtain the scene classification result and feature vector.

[0017] Optionally, the semantic segmentation neural network model M1 adopts an image coding neural network, including an encoder and a decoder, the encoder is used to extract the original image into a high-dimensional feature coding vector, and the decoder is used to decode the high-dimensional feature coding vector into a semantic segmentation image of the same size as the original image.

[0018] Optionally, the neural network model M2 adopts a convolutional neural network, including a backbone network part and a head network part. The backbone network part uses multiple layers of convolution and pooling layers to process the combined image to obtain a high-dimensional feature map, and then uses a global average pooling layer to process the high-dimensional feature map to obtain the feature vector; the head network part uses a multi-layer neural network containing a residual block to process the feature vector to obtain the scene classification result.

[0019] Optionally, performing the hybrid search includes:

[0020] Performing preliminary screening in the scene vector database according to the latitude, longitude and azimuth in the positioning data to obtain a set of recorded data;

[0021] The cosine similarity between the feature vector and the record data set is calculated, and the record data set is screened again based on the cosine similarity to obtain a mixed search result set.

[0022] Optionally, updating the mixed search results to the scene vector database includes:

[0023] If the hybrid search result set is an empty set, the image data, scene classification results, feature vectors, and positioning data are manually reviewed and added to the scene vector database;

[0024] If the hybrid search result set is not an empty set, and the individual IDs and scene categories in the hybrid search result set are the same, then updating the image data, scene classification results, feature vectors, and positioning data to the scene vector database;

[0025] If the hybrid search result set is not an empty set, and there are different individual IDs in the hybrid search result set, the feature vectors in the hybrid search result set are arithmetic averaged, a new feature vector is obtained and a hybrid search is performed again in the scene vector database, a new hybrid search result set is obtained and voting is performed, and the corrected individual ID and scene category are obtained and updated to the scene vector database.

[0026] Optionally, after updating the mixed search results to the scene vector database, the method further includes manually cleaning and auditing the scene vector database on a regular basis.

[0027] To further achieve the above-mentioned purpose, the present invention also provides a positioning system for patrol scenes, including: an edge device and a cloud device, wherein the edge device is used to collect image data and positioning data of the patrol scene in real time, and perform feature extraction on the image data to obtain scene classification results and feature vectors; the cloud device is used to process the image data, positioning data, scene classification results and feature vectors to obtain disease results and disease positioning of the patrol scene.

[0028] Optionally, the edge device includes an industrial computer, a high-definition camera and a navigation and positioning receiver, the high-definition camera and the navigation and positioning receiver are connected to the industrial computer, and the industrial computer provides data transmission, image feature extraction and power supply functions.

[0029] Optionally, the cloud device includes a disease recognition module, a hybrid search module, and a dispatch module. The disease recognition module is used to identify diseases based on the image data, scene classification results, and feature vectors; the hybrid search module is used to perform a hybrid search in a preset scene vector database based on the positioning data and feature vectors when a disease is identified, to obtain a hybrid search result of the inspection scene; the dispatch module is used to update the hybrid search result to the scene vector database and send it to the corresponding maintenance personnel in combination with the disease recognition result.

[0030] The beneficial effects of the present invention are:

[0031] (1) Combining edge computing with cloud computing to improve efficiency: The combination of edge computing and cloud computing eliminates the need for on-site manual recording and measurement, making real-time data processing more efficient and faster. Edge devices are responsible for on-site data collection and preliminary processing, reducing the pressure of large amounts of raw data transmission while ensuring data security and privacy protection.

[0032] (2) High-precision image recognition and manual verification complement each other: By integrating high-definition cameras and intelligent algorithms, semantic target extraction of traffic facilities in the image is performed, and the extraction results are combined with the original image, so that subsequent analysis models can obtain more information about the on-site infrastructure. At the same time, the present invention retains the manual review link to ensure the accuracy of the results, especially in the face of uncertain or ambiguous situations, human intervention can provide the final decision.

[0033] (3) Multi-level data analysis: When performing scene and ID matching, a hybrid search is used to make full use of the collected multivariate data such as longitude and latitude, azimuth, and feature vector data, which helps to improve the search speed and accuracy. Compared with the simple use of classification and discrimination, the present invention takes into account the differences between different individuals of the same category. For situations where the physical distance is close, the scene category is the same, but they belong to different facilities, effective distinction can be made.

[0034] (4) Database optimization management: The vector database used in the present invention is a composite relational database that supports efficient query and storage functions. When used, it does not require recording special field thresholds for classification, which facilitates the management and maintenance of large amounts of historical data. It also provides a flexible data update mechanism to adapt to changing road conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A flow chart of a positioning method for a patrol scene according to an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of the installation of the intelligent inspection device according to an embodiment of the present invention;

[0038] Figure 3 Schematic diagram of the vanishing point height of a captured image according to an embodiment of the present invention;

[0039] Figure 4 The data processing flow chart of the embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] This embodiment provides a method for positioning a patrol scene, such as Figure 1 As shown, including:

[0043] Collect image data and positioning data of inspection scenes in real time;

[0044] Perform semantic segmentation and feature extraction on image data to obtain scene classification results and feature vectors;

[0045] Defect recognition is performed based on image data, scene classification results, and feature vectors. If a defect is identified, a hybrid search is performed in a preset scene vector database based on positioning data and feature vectors to obtain hybrid search results for the inspection scene.

[0046] The mixed search results are updated to the scene vector database to complete the disease location of the inspection scene.

[0047] Specifically, this embodiment collects image data and positioning data of the patrol scene in real time, which is achieved by installing intelligent patrol equipment on the patrol car, and the intelligent patrol equipment is equipped with intelligent algorithms to perform semantic segmentation, feature extraction, and image classification on the image, and upload the collected and analyzed data to the cloud. After receiving the collected data, the cloud performs a feature vector hybrid search, obtains the corresponding analysis results based on post-processing, and stores the relevant data in the database.

[0048] Intelligent patrol equipment composition and installation Figure 2 As shown, it includes one or more high-definition cameras and an industrial computer, to which a high-precision navigation and positioning receiver is connected, an Internet card is provided, and there is a certain amount of AI computing power, and recognition algorithm programs can be deployed. In terms of power supply source, the intelligent patrol equipment uses direct current power supply; in terms of waterproof performance, the intelligent patrol equipment has good waterproof performance.

[0049] The industrial computer is installed inside the vehicle and provides power for the vehicle. It has the fourth generation (4G) or fifth generation (5G) mobile communication technology, connected to a high-precision navigation and positioning receiver and a high-definition camera. It can send instructions to the high-definition camera to collect images, and continuously receive, process, and upload the collected data. In addition, the central industrial computer can also contain a micro-graphics processing unit (GPU), and the deep neural network algorithm can use the GPU to accelerate image processing.

[0050] The high-definition camera is powered by an industrial computer or a vehicle. The camera pixel is ≥ 1 million pixels. The image quality is clear and there is no delay during acquisition. The acquisition frame rate is ≥ 25Hz. It can be installed inside or outside the vehicle. After installation, the viewing angle is fixed and it can be used to capture images in front of or behind the vehicle. The captured image requires clear image quality, no obvious distortion, no obvious obstruction, and the ratio of the height h' from the vanishing point to the upper edge of the image to the total height h of the image, such as Figure 3 As shown, it is required to be limited to a certain range, that is, 1 / 3≤h' / h≤ 3 / 4.

[0051] The vanishing point of an image can be determined by deploying a convolutional neural network classification model on an industrial computer, dividing the image vertically into n equal intervals, and predicting which interval has the highest probability of the image vanishing point being in.

[0052] The positioning error of the high-precision navigation and positioning receiver is ≤1m, and it is equipped with an inertial navigation system. The collection frequency of positioning data is ≥10Hz.

[0053] Furthermore, the data recognition and processing in this implementation uses edge device perception recognition + cloud data post-processing execution. The processing flow and logic are as follows: Figure 4 shown.

[0054] Performing semantic segmentation and feature extraction on image data to obtain scene classification results and feature vectors includes: inputting the original image into a semantic segmentation neural network model M1 to output a semantic segmentation image; combining and splicing the semantic segmentation image with the original image to obtain a combined image; and inputting the combined image into a neural network model M2 to obtain scene classification results and feature vectors.

[0055] The semantic segmentation neural network model M1 adopts an image coding neural network, including an encoder and a decoder. The encoder is used to extract the original image into a high-dimensional feature coding vector, and the decoder is used to decode the high-dimensional feature coding vector into a semantic segmentation image with the same size as the original image.

[0056] The neural network model M2 adopts a convolutional neural network, including a backbone network part and a head network part. The backbone network part uses multi-layer convolution and pooling layers to process the combined image to obtain a high-dimensional feature map, and then uses a global average pooling layer to process the high-dimensional feature map to obtain a feature vector; the head network part uses a multi-layer neural network containing a residual block to process the feature vector to obtain the scene classification result.

[0057] The hybrid search includes: performing preliminary screening in the scene vector database according to the latitude, longitude and azimuth in the positioning data to obtain a record data set; calculating the cosine similarity between the feature vector and the record data set, and performing another screening in the record data set based on the cosine similarity to obtain a hybrid search result set.

[0058] Updating the hybrid search results to the scene vector database includes:

[0059] If the hybrid search result set is an empty set, the image data, scene classification results, feature vectors, and positioning data are manually reviewed and added to the scene vector database;

[0060] If the hybrid search result set is not an empty set, and the individual IDs and scene categories in the hybrid search result set are the same, then the image data, scene classification results, feature vectors, and positioning data are updated to the scene vector database;

[0061] If the hybrid search result set is not an empty set, and there are different individual IDs in the hybrid search result set, the feature vectors in the hybrid search result set are arithmetic averaged, a new feature vector is obtained and a hybrid search is performed again in the scene vector database, a new hybrid search result set is obtained and voting is performed, and the corrected individual ID and scene category are obtained and updated to the scene vector database.

[0062] Specifically, when the patrol vehicle of this embodiment performs patrol, it collects the original image m i , and use the semantic segmentation neural network model M1 to perform semantic segmentation on the image to obtain the semantic segmentation image m i '; Then the original image m i With the semantic segmentation image m i 'Combined and spliced ​​to get the integrated image m i *; The integrated combined image m i * Input to the neural network model M2 to obtain the scene classification result ci , and output a corresponding feature vector v i ; After completing the image processing, the image will be collected, c i 、v i The corresponding latitude, longitude and azimuth data are uploaded to the cloud server.

[0063] Original image m i The image size is generally [u, v, 3], that is, it contains three channels, RGB, and the aspect ratio of the image is generally 16:9 or 4:3.

[0064] The azimuth angle, also known as the heading angle, is a reference direction for collecting images for high-definition cameras. The north direction is 0° and increases in the clockwise direction. The data value range is [0, 360).

[0065] The semantic segmentation neural network model M1 refers to an image encoding and decoding type of neural network. It first extracts the original image m through the feature extraction encoder neural network layer. i Extract it into a high-dimensional feature encoding vector, and then decode the feature vector into a semantic segmentation image m with the same pixel size as the original image through the feature decoder neural network layer. i '. Semantic segmentation mainly classifies the road surface area, traffic safety facilities such as guardrails, roadside facilities such as signs, overpasses, elevated structures and other road facilities in the image, and assigns a specified value in [100, 255] according to the category. Other unlabeled areas are uniformly classified as background categories and marked as 0, realizing the category labeling of all pixels in the image, and then generating a semantic segmentation image m i ', the image size is [u, v]. Assigning a specified value in [100, 255] according to the category means selecting j integers from [100, 255], corresponding to the j types of facilities that can be recognized in the image semantic recognition network, and corresponding to one category and one integer.

[0066] Original image m i With semantic image m i 'Combined stitching means combining two images according to the channel direction to form a new combined image m i *, the image size is [u, v, 4], and the 4 channels are RGB image and semantic segmentation image respectively.

[0067] The neural network model M2 consists of two parts: the backbone network is a convolutional neural network, which is trained according to Re-Identification (ReID); on the basis of the backbone network, a fully connected classification head network is added for category classification.

[0068] 1) Backbone network part: The model first combines the image m i * Perform preprocessing such as shrinking and normalization, and use a multi-layer convolution (CNN) + pooling network to combine the image m i * is processed to obtain a high-dimensional feature map [u', v', k], and then the high-dimensional feature map is processed using global average pooling to obtain a 512-dimensional feature vector v i .

[0069] 2) Head network part: v i The image is input into a multi-layer neural network containing a residual block for further processing, and the neural network finally outputs k classifications and corresponding confidences, where the k classifications correspond to the k scene classifications to be performed by the model, including but not limited to "ground road", "expressway", "auxiliary road", "tunnel", "road under elevated road" and other categories, and the category with the largest confidence is used as the scene classification result of the M2 model, denoted as c i .

[0070] After completing the analysis, the industrial computer will m i 、c i 、v i , and the longitude and latitude corresponding to the time [lng i , lat i ]、azimuth i , and upload them to the cloud server. i , lat i ] has a numerical precision of 0.000001.

[0071] After receiving the new collected data, the cloud server will use the longitude and latitude data [lng i , lat i ]、azimuth dataazi i , feature vector v i , Classification category c i The data is collected and a hybrid search is performed in the scene vector database in the cloud. According to the results of the hybrid search, the corresponding processing logic is executed to complete the scene judgment of the collected image and update the scene vector database in the cloud.

[0072] The scene vector database is a composite relational database that contains multiple records. Each record in the database contains a URL link to the captured image, the latitude and longitude of the image [lng, lat], the azimuth angle of the image, the scene classification result c of the image, the individual ID of the image, the feature vector v of the image after feature extraction, the creation time CREATETIME of the record, whether it has been checked by the auditor CHECK, and whether it has been marked for deletion DELETE. The data in the vector database changes dynamically. The verified data can be added to the vector database. The auditor can also clean and check the vector database regularly to correct the scene category of the image.

[0073] Hybrid search refers to the preliminary screening of data in the database based on the latitude, longitude and azimuth of the uploaded data, and screening out i , lat i ], the azimuth and azi i Similar record data set Vset i , and then use the cosine distance to calculate the feature vector v i and Vset i Similarity, set the threshold T of cosine similarity, from Vset i Select a new vector set with similarity ≥ T as the mixed search result set Vset i '.

[0074] According to the mixed search set Vset i ', perform corresponding business post-processing:

[0075] 1) Vset i ' is an empty set, which means that the vector database is in the longitude and latitude [lng i , lat i ] nearby, no i Similar vectors, then the v uploaded by the device i 、c i 、m i The data is pushed to the queue for confirmation and added to the vector database V after verification and confirmation by the auditor. db middle.

[0076] 2) If Vset i 'Not an empty set, and Vset i 'The individual IDs are the same, and the scene categories are all c i , then the scene classification result corresponding to the collected data is determined to be c i , and v i 、c i And individual ID and other data are updated and added to V dbmiddle.

[0077] 3) If Vset i 'Not an empty set, and Vset i 'If the individual IDs are different, Vset i ', perform arithmetic averaging on the eigenvectors in the equation to obtain a new eigenvector v'', and then use [lng i , lat i ]、azi i 、v'' remove V db Perform new feature retrieval in , obtain the e vectors with the highest similarity, and form a set to obtain a new feature vector set Vset i '', vote according to the individual ID corresponding to the vector, and use the scene category c'' corresponding to the most likely individual ID in the voting result as the scene category of the collected image, and set c'', [lng i , lat i ]、azi i , the most likely individual ID and other data are updated to V db middle.

[0078] Voting methods include but are not limited to:

[0079] i. Majority voting: The scene category with the largest number of statistics among the e vectors is taken as the voting result.

[0080] ii. Weighted voting: Weighted voting is performed based on cosine distance. The smaller the distance, the more votes can be cast. Finally, the individual ID with the most votes in the e vectors is counted.

[0081] Furthermore, after the mixed search results are updated to the scene vector database, it also includes regular cleaning and review of the scene vector database manually.

[0082] Specifically, auditors can use auxiliary tools to regularly check the existing vector database V db Cleaning can include but is not limited to the following methods:

[0083] 1) Use a new neural network model M2' that is different from the neural network model M2 to db Re-identify the data and select the recognition results and V db The different data records are pushed to the confirmation queue for verification by the auditor, and the mismatched scene categories are corrected and updated in the database.

[0084] 2) Set V db The images with the same individual ID are extracted, and the record data that does not belong to the ID is cleaned.

[0085] To further optimize the technical solution, this embodiment also provides a positioning system for patrol scenes, including: edge devices and cloud devices. The edge devices are used to collect image data and positioning data of patrol scenes in real time, and perform feature extraction on the image data to obtain scene classification results and feature vectors; the cloud devices are used to process the image data, positioning data, scene classification results and feature vectors to obtain disease results and disease locations of patrol scenes.

[0086] Furthermore, the edge device includes an industrial computer, a high-definition camera and a navigation and positioning receiver. The high-definition camera and the navigation and positioning receiver are connected to the industrial computer, and the industrial computer provides data transmission, image feature extraction and power supply functions.

[0087] Specifically, the edge devices are mainly vehicle-mounted intelligent patrol devices. The device-related description is as follows:

[0088] 1) Equipped with an industrial computer equipped with an NVIDIA GT1030 graphics card, which can run deep learning algorithm programs. The industrial computer is powered by a DC power supply to ensure stable operation. The industrial computer has good waterproof performance and is suitable for various outdoor conditions.

[0089] 2) The industrial computer has a built-in high-precision GPS module and Internet card, which can collect longitude, latitude and azimuth. The acquisition frame rate is 10Hz, and it supports real-time data transmission to the cloud server.

[0090] 3) A high-definition camera is installed on the roof and connected to the industrial computer. The camera captures images with a resolution of 2 million pixels and a frame rate of 25 Hz. The image is clear and unobstructed. The camera faces the front of the vehicle and the camera viewing angle is horizontal.

[0091] The process of edge device data collection and data processing is as follows:

[0092] 1) After the edge device is powered on, it will collect GPS data from the GPS sensor and determine whether the current vehicle is patrolling based on the GPS status. When the vehicle is in a moving state, the industrial computer will accumulate the displacement of the vehicle. When the accumulated distance reaches 10m, the industrial computer will issue an instruction to collect images and GPS data, and start a new distance accumulation from 0.

[0093] 2) The image m after acquisition i , will be input into a semantic segmentation neural network model M1, which mainly performs semantic perception of traffic safety facilities, road surface areas, bridges and other facilities in the image, performs image processing, and generates a semantic segmentation image m i ', pixel size and original image m i Same, m i' is to assign a category to each pixel in the image range, including guardrails, anti-glare panels, road surfaces, bridges, traffic signs, sky, warning poles, background, etc., and assign a specific [0,255) integer according to the category.

[0094] 3) Image m i With the semantic segmentation image m i ', stitch according to the channel to obtain a new combined image m i *, m i * Scaling, when input to the neural network model M2, the model will output a 512-dimensional vector v i , and will output a k-dimensional confidence array, corresponding to k scene categories such as "expressway", "ground road", and "tunnel". The scene category with the highest confidence is the scene category inferred by the model, recorded as c i .

[0095] 4) The image data m collected above i , longitude and latitude data [lng i , lat i ]、azimuth i And the result data v i 、c i Summarize and upload to cloud devices.

[0096] Furthermore, the cloud device includes a disease recognition module, a hybrid search module, and a post-processing module. The disease recognition module is used to identify diseases based on image data, scene classification results, and feature vectors; the hybrid search module is used to perform a hybrid search in a preset scene vector database based on positioning data and feature vectors when a disease is identified, to obtain hybrid search results of the inspection scene; the post-processing module is used to update the hybrid search results to the scene vector database and send them to the corresponding maintenance personnel in combination with the disease recognition results.

[0097] Specifically, the cloud device is mainly a cloud server, on which a data receiving program, a scene vector database, a data processing and analysis program, an order dispatching program, etc. are installed.

[0098] 1) Disease identification module:

[0099] When the cloud server receives new data, including image data m i , longitude and latitude data [lng i , lat i ]、azimuth i And the result data v i 、c i, the image data will be input into the disease recognition model. If the image is identified to have a disease, the collected data will be sent to the post-processing program for further analysis. In this embodiment, the disease recognition model adopts a target recognition neural network, which can mark the identified disease category and recognition box position and other information on the image.

[0100] 2) Hybrid search module:

[0101] After receiving the data, the longitude and latitude [lng i , lat i ]、azimuth dataazi i , feature vector data v i , perform a hybrid search in the vector database MILVUS. The MILVUS vector database contains the following fields:

[0102] lng: longitude;

[0103] lat: latitude;

[0104] azi: azimuth, the value range is [0,360);

[0105] vec: feature vector;

[0106] CHECK: whether the auditor has performed verification, 1 means verified, 0 means not verified;

[0107] DELETE: whether the data is marked for deletion, 1 for marked for deletion, 0 for unmarked for deletion;

[0108] CREATETIME: collection time;

[0109] c: scene category;

[0110] ID: Individual ID.

[0111] Among them, when conducting a mixed search, the following search conditions are mainly followed:

[0112] i、lng i - 0.00001 * 8 < lng i + 0.00001 * 8;

[0113] ii.lat i - 0.00001 * 8 < lat i + 0.00001 * 8;

[0114] iii、azi i - 5 < azi < azi i + 5 (azi i∈[5,355] );

[0115] 0 < azi <azi i + 5 ∪ azi > azi i – 5 + 360 (azi i ∈[0,5) );

[0116] azi i - 5 < azi < 360 ∪ azi < azi i + 5 - 360 (azi i ∈(355,360) );

[0117] iv. DELETE=0, CHECK=1;

[0118] v. The searched vector has a cosine distance > 0.8.

[0119] According to the restriction conditions of hybrid search, the search result set Vset is obtained i , for the search result set Vset i The analysis includes:

[0120] i. Vset i ' is an empty set, which means that the vector database is in the longitude and latitude [lng i , lat i ] nearby, no i Similar vectors, then the v uploaded by the device i 、c i 、m i The data is pushed to the queue for confirmation and added to the vector database V after verification and confirmation by the auditor. db middle.

[0121] ii. If Vset i 'Not an empty set, and Vset i 'The individual IDs are the same, and the scene categories are all c i , then the scene classification result corresponding to the collected data is determined to be c i , and v i 、c i And individual ID and other data are updated and added to V db middle.

[0122] iii. If Vset i 'Not an empty set, and Vset i 'If the individual IDs are different, Vset i ', perform arithmetic averaging on the eigenvectors in the equation to obtain a new eigenvector v'', and then use [lngi , lat i ]、azi i 、v'' remove V db Perform new feature retrieval in , obtain the 10 vectors with the highest similarity, and form a set to obtain a new feature vector set Vset i '', vote according to the individual ID corresponding to the vector, and use the scene category c'' corresponding to the most likely individual ID in the voting result as the scene category of the collected image, and set c'', [lng i , lat i ]、azi i , the most likely individual ID and other data are updated to V db The voting logic used is to vote weighted by cosine distance. The smaller the distance, the more votes can be cast. The number of votes that can be cast is [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]. Finally, the individual ID with the most votes in the 10 vectors is counted.

[0123] 3) Order dispatch module:

[0124] If the post-processed data does not need to be verified by the auditor, it will be directly pushed to the dispatch module; if it needs to be verified by the auditor, the relevant data will be pushed to the dispatch module after the auditor passes the verification. In the dispatch module, the disease data will be matched with the road section through the scene category and the individual facility ID, and then pushed to the specific maintenance team and archived for further analysis.

[0125] Furthermore, auditors can use auxiliary tools to regularly review the existing scene vector database V db Cleaning can include the following methods:

[0126] 1) A deeper neural network model M2', for V db Re-identify the image data and select the recognition results and V db The different data records are pushed to the confirmation queue for verification by the auditor, and the mismatched scene categories are corrected and updated in the database.

[0127] 2) Set V db The images with the same individual ID are extracted, and the record data that does not belong to the ID is cleaned.

[0128] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for positioning a patrol scene, characterized in that: include: Collect image data and positioning data of inspection scenes in real time; Performing semantic segmentation and feature extraction on the image data to obtain scene classification results and feature vectors; Performing disease identification based on the image data, scene classification results, and feature vectors, and if a disease is identified, performing a mixed search in a preset scene vector database based on the positioning data and feature vectors to obtain a mixed search result of the inspection scene; Updating the mixed search results to the scene vector database to complete the disease location of the inspection scene; The scene vector database is a composite relational database, which contains multiple records, each of which contains a link to a captured image. url , the latitude and longitude of the collected image[ lng , lat ], the azimuth of the acquired image azi , scene classification results of the collected images c , the individual ID of the collected image, the feature vector of the collected image after feature extraction v , the creation time of the record CREATETIME, whether it has been checked by the auditor CHECK and whether it has been marked for deletion DELETE; Performing semantic segmentation and feature extraction on the image data to obtain scene classification results and feature vectors includes: Input the original image into the semantic segmentation neural network model M1 and output the semantic segmentation image; Combining and splicing the semantic segmentation image with the original image to obtain a combined image; Inputting the combined image into the neural network model M2 to obtain the scene classification result and feature vector; The semantic segmentation neural network model M1 adopts an image coding neural network, including an encoder and a decoder, wherein the encoder is used to extract the original image into a high-dimensional feature coding vector, and the decoder is used to decode the high-dimensional feature coding vector into a semantic segmentation image with the same size as the original image; The neural network model M2 adopts a convolutional neural network, including a backbone network part and a head network part. The backbone network part uses multiple layers of convolution and pooling layers to process the combined image to obtain a high-dimensional feature map, and then uses a global average pooling layer to process the high-dimensional feature map to obtain the feature vector; the head network part uses a multi-layer neural network containing a residual block to process the feature vector to obtain the scene classification result; Performing the hybrid search includes: Performing preliminary screening in the scene vector database according to the latitude, longitude and azimuth in the positioning data to obtain a set of recorded data; Calculating the cosine similarity between the feature vector and the record data set, and re-screening the record data set based on the cosine similarity to obtain a mixed search result set; Updating the mixed search results to the scene vector database includes: If the hybrid search result set is an empty set, the image data, scene classification results, feature vectors, and positioning data are manually reviewed and added to the scene vector database; If the hybrid search result set is not an empty set, and the individual IDs and scene categories in the hybrid search result set are the same, then updating the image data, scene classification results, feature vectors, and positioning data to the scene vector database; If the hybrid search result set is not an empty set, and there are different individual IDs in the hybrid search result set, the feature vectors in the hybrid search result set are arithmetic averaged, a new feature vector is obtained and a hybrid search is performed again in the scene vector database, a new hybrid search result set is obtained and voting is performed, and the corrected individual ID and scene category are obtained and updated to the scene vector database.

2. The method for positioning a patrol scene according to claim 1, characterized in that: After the mixed search results are updated to the scene vector database, the process also includes manually cleaning and auditing the scene vector database on a regular basis.

3. A positioning system for patrol scenes, characterized in that: include: Edge devices and cloud devices, wherein the edge devices are used to collect image data and positioning data of the inspection scene in real time, and perform feature extraction on the image data to obtain scene classification results and feature vectors; the cloud devices are used to process the image data, positioning data, scene classification results and feature vectors to obtain the disease results and disease positioning of the inspection scene; Performing semantic segmentation and feature extraction on the image data to obtain scene classification results and feature vectors includes: Input the original image into the semantic segmentation neural network model M1 and output the semantic segmentation image; Combining and splicing the semantic segmentation image with the original image to obtain a combined image; Inputting the combined image into the neural network model M2 to obtain the scene classification result and feature vector; The semantic segmentation neural network model M1 adopts an image coding neural network, including an encoder and a decoder, wherein the encoder is used to extract the original image into a high-dimensional feature coding vector, and the decoder is used to decode the high-dimensional feature coding vector into a semantic segmentation image with the same size as the original image; The neural network model M2 adopts a convolutional neural network, including a backbone network part and a head network part. The backbone network part uses multiple layers of convolution and pooling layers to process the combined image to obtain a high-dimensional feature map, and then uses a global average pooling layer to process the high-dimensional feature map to obtain the feature vector; the head network part uses a multi-layer neural network containing a residual block to process the feature vector to obtain the scene classification result; The cloud device includes a disease recognition module, a hybrid search module, and a dispatch module. The disease recognition module is used to perform disease recognition based on the image data, scene classification results, and feature vectors; the hybrid search module is used to perform a hybrid search in a preset scene vector database based on the positioning data and feature vectors when a disease is recognized to obtain a hybrid search result of the inspection scene; the dispatch module is used to update the hybrid search result to the scene vector database and send it to the corresponding maintenance personnel in combination with the disease recognition result; The scene vector database is a composite relational database, which contains multiple records, each of which contains a link to a captured image. url , the latitude and longitude of the collected image[ lng , lat ], the azimuth of the acquired image azi , scene classification results of the collected images c , the individual ID of the collected image, the feature vector of the collected image after feature extraction v , the creation time of the record CREATETIME, whether it has been checked by the auditor CHECK and whether it has been marked for deletion DELETE; Performing the hybrid search includes: Performing preliminary screening in the scene vector database according to the latitude, longitude and azimuth in the positioning data to obtain a set of recorded data; Calculating the cosine similarity between the feature vector and the record data set, and re-screening the record data set based on the cosine similarity to obtain a mixed search result set; Updating the mixed search results to the scene vector database includes: If the hybrid search result set is an empty set, the image data, scene classification results, feature vectors, and positioning data are manually reviewed and added to the scene vector database; If the hybrid search result set is not an empty set, and the individual IDs and scene categories in the hybrid search result set are the same, then updating the image data, scene classification results, feature vectors, and positioning data to the scene vector database; If the hybrid search result set is not an empty set, and there are different individual IDs in the hybrid search result set, the feature vectors in the hybrid search result set are arithmetic averaged, a new feature vector is obtained and a hybrid search is performed again in the scene vector database, a new hybrid search result set is obtained and voting is performed, and the corrected individual ID and scene category are obtained and updated to the scene vector database.

4. The positioning system for patrol scenes according to claim 3, characterized in that: The edge device includes an industrial computer, a high-definition camera and a navigation and positioning receiver. The high-definition camera and the navigation and positioning receiver are connected to the industrial computer, and the industrial computer provides data transmission, image feature extraction and power supply functions.

Images