Positioning method and system for inspection scene

Abstract

The present application relates to the field of intelligent transportation. Disclosed are a positioning method and system for an inspection scene. The method comprises: acquiring image data and positioning data of an inspection scene in real time; performing semantic segmentation and feature extraction on the image data, so as to acquire a scene classification result and a feature vector; performing defect identification on the basis of the image data, the scene classification result and the feature vector, and if a defect is identified, performing hybrid search in a preset scene vector database on the basis of the positioning data and the feature vector, so as to obtain a hybrid search result of the inspection scene; and updating the hybrid search result to the scene vector database so as to complete defect positioning for the inspection scene. The present application uses an edge-cloud integrated system architecture and uses an artificial intelligence-based image analysis algorithm to implement accurate and fast positioning for scenes by inspection devices.

Description

A positioning method and system for patrol scenarios

[0001] This application claims priority to Chinese Patent Application No. 202411464133.4, filed on October 21, 2024, entitled "A Positioning Method and System for Patrol Scenes", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of intelligent transportation technology, and in particular to a positioning method and system for patrol scenarios. Background Technology

[0003] With the peak of infrastructure construction over, urban elevated roads, expressways, highways, tunnels, and other road infrastructure are facing enormous maintenance pressure. After a certain number of years of service, road surfaces, bridge structures, and other facilities will develop various defects and damages under the influence of multiple factors such as temperature, rain, and load. Minor defects will affect the aesthetics of the facilities, while serious defects will not only affect the appearance but may even affect the safety of slow-moving traffic.

[0004] To extend the service life of road infrastructure, the most important thing is to improve the ability to detect defects and incidents on the infrastructure and to accurately describe information such as the location, size, and type of defects. Location data is a crucial piece of information about defects; after obtaining location data, maintenance teams can locate the corresponding defects and incidents for repair. Furthermore, location data can aid in further analysis and applications: it can be used to analyze the spatial distribution of defects and incidents, such as analyzing the clustering density and distribution patterns of defects and incidents in different spaces; it can also be used to analyze defects and incidents at the same location by time and category, spatially deduplicate defects reported multiple times at the same location, or analyze whether a location is experiencing a "frequent repair and patching" situation.

[0005] Currently, routine road infrastructure inspections primarily rely on visual inspections by patrol officers on foot or by car. When defects or incidents are discovered, officers use a mobile app to capture images and record and report the location data using the phone's GPS. However, with technological advancements, some patrols are beginning to use specialized equipment to collect images, employ artificial intelligence algorithms to identify defects and incidents, and utilize high-precision positioning sensors to obtain accurate location information, reducing errors and improving positioning accuracy.

[0006] The location data collected during daily patrols is usually two-dimensional latitude and longitude data, which can be displayed on electronic maps and matched with road segments based on latitude and longitude. However, in special cases such as elevated-ground roads, tunnel-ground roads, and mainline-ramp roads, there is overlap at the same latitude and longitude. Simply using latitude and longitude cannot determine whether the current location belongs to the upper or lower layer, and accurate positioning and road segment matching cannot be achieved, which may lead to problems in identification and dispatch. Currently, some existing solutions and methods include: (1) Manual recording: During patrols and inspections, the patrolman observes and records the current patrol scene location on the APP, or a switch is added to the patrol equipment, and the patrolman clicks the switch to record the patrol scene corresponding to different patrol times. This method mainly relies on the subjective judgment and operation of the human, and is easily affected by the supervisor of the patrolman, resulting in inaccurate data. (2) Collecting trajectory, altitude, and acceleration data: High-precision navigation and positioning receivers and three-axis accelerometers are added to the patrol equipment to record the trajectory, speed, altitude, and three-axis acceleration data of the patrol vehicles. The vehicle's trajectory and three-axis acceleration data are used to mark the vehicle's driving route and uphill / downhill time points. Based on the vehicle's altitude and driving status, it is determined which layer of the overlapping road the vehicle may be on. This method relies heavily on the acquisition of sensor data and basic data: On the one hand, when the satellite signal is poor, the collected trajectory, speed, altitude, and other data have large deviations, or may even be invalid data, leading to inaccurate positioning; on the other hand, the altitude distinction thresholds are different for the same facilities, and the altitude distinction thresholds for elevated roads and tunnels are difficult to obtain, all of which interfere with accurate scene distinction and subsequent analysis. (3) Using classification neural networks to process data: Some patrol equipment is equipped with cameras. The images collected by the cameras are input into a classification neural network, which determines the scene to which the image belongs. While this method can make a judgment on the image scene, the image can only be classified into a limited number of categories and cannot further distinguish between different individuals. For example, if the auxiliary roads of two bridges are very close to each other, have the same direction, and both belong to the auxiliary roads of the bridge, the results of the image classification model alone cannot distinguish between the two roads, which leads to inaccurate road matching. Summary of the Invention

[0007] The purpose of this application is to provide a method and system for locating patrol scenes. It uses an edge-cloud combined system architecture and artificial intelligence image analysis algorithms to achieve accurate and rapid scene location of patrol equipment. At the same time, it uses a vector database to create feature files for road scenes, improves the accuracy of image scene classification, and helps with subsequent data matching and analysis.

[0008] To achieve the above objectives, this application provides the following solution:

[0009] Firstly, this application provides a method for locating a patrol scene, the method comprising:

[0010] Real-time acquisition of image and location data of the patrol scene;

[0011] The image data is semantically segmented and feature extracted to obtain scene classification results and feature vectors;

[0012] Based on the image data, the scene classification results, and the feature vector, disease identification is performed. If a disease is identified, a mixed search is performed in a preset scene vector database based on the positioning data and the feature vector to obtain the mixed search results of the inspection scene.

[0013] The mixed search results are updated to the scene vector database to complete the location of the disease in the inspection scene.

[0014] In an exemplary embodiment, performing semantic segmentation and feature extraction on the image data to obtain scene classification results and feature vectors includes:

[0015] The original image is input into the semantic segmentation neural network model M1, and the output is a semantically segmented image;

[0016] The semantic segmentation image is combined and stitched with the original image to obtain a combined image;

[0017] The combined image is input into the neural network model M2 to obtain the scene classification result and the feature vector.

[0018] In an exemplary embodiment, the semantic segmentation neural network model M1 employs an image encoding neural network, including an encoder and a decoder; the encoder is used to extract the original image into a high-dimensional feature encoding vector; the decoder is used to decode the high-dimensional feature encoding vector into a semantic segmentation image with the same size as the original image.

[0019] In an exemplary embodiment, the neural network model M2 employs a convolutional neural network, including a backbone network and a head network. The backbone network uses multiple convolutional and pooling layers to process the combined image to obtain a high-dimensional feature map, and then uses a global average pooling layer to process the high-dimensional feature map to obtain the feature vector. The head network uses a multi-layer neural network containing residual blocks to process the feature vector to obtain the scene classification result.

[0020] In one exemplary embodiment, the hybrid search includes:

[0021] Based on the latitude, longitude and azimuth in the positioning data, a preliminary screening is performed in the scene vector database to obtain a set of recorded data.

[0022] Calculate the cosine similarity between the feature vector and the record data set, and then further filter the record data set based on the cosine similarity to obtain a mixed search results set.

[0023] In one exemplary embodiment, updating the scene vector database with the hybrid search results includes:

[0024] If the mixed search results set is empty, the image data, the scene classification results, the feature vectors, and the positioning data are manually reviewed and then added to the scene vector database.

[0025] If the mixed search results set is not empty, and the individual IDs and scene categories in the mixed search results set are the same, then the image data, the scene classification results, the feature vectors, and the location data are updated to the scene vector database.

[0026] If the mixed search results set is not empty and the individual IDs in the mixed search results set are different, then the feature vectors in the mixed search results set are arithmetically averaged to obtain new feature vectors, and a new mixed search is performed in the scene vector database to obtain a new mixed search results set and vote on it. The corrected individual IDs and scene categories are then obtained and updated in the scene vector database.

[0027] In one exemplary embodiment, after updating the hybrid search results to the scene vector database, the method further includes periodically cleaning and reviewing the scene vector database manually.

[0028] Secondly, this application also provides a positioning system for an inspection scene, which includes an edge device and a cloud device. The edge device is used to collect image data and positioning data of the inspection scene in real time, and to extract features from the image data to obtain scene classification results and feature vectors. The cloud device is used to process the image data, the positioning data, the scene classification results and feature vectors to obtain the disease results and disease location of the inspection scene.

[0029] In one exemplary embodiment, the edge device includes an industrial control computer, a high-definition camera, and a navigation and positioning receiver; the high-definition camera and the navigation and positioning receiver are connected to the industrial control computer; the industrial control computer is used to provide data transmission, image feature extraction, and power supply functions.

[0030] In an exemplary embodiment, the cloud device includes a disease identification module, a hybrid search module, and a dispatch module; the disease identification module is used to identify diseases based on the image data, the scene classification results, and the feature vectors; the hybrid search module is used to perform a hybrid search in a preset scene vector database based on the location data and the feature vectors when a disease is identified, to obtain hybrid search results for the inspection scene; the dispatch module is used to update the hybrid search results to the scene vector database and send them to the corresponding maintenance personnel in conjunction with the disease identification results.

[0031] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0032] (1) Combining edge computing with cloud computing to improve efficiency: By adopting an edge-cloud combined approach, there is no need for manual on-site recording and measurement, making real-time data processing more efficient and faster. Edge devices are responsible for on-site data collection and preliminary processing, reducing the pressure of transmitting large amounts of raw data, while ensuring data security and privacy protection.

[0033] (2) High-precision image recognition and manual verification complement each other: By integrating high-definition cameras and intelligent algorithms, semantic targets of traffic facilities in images are extracted, and the extraction results are combined with the original images, allowing subsequent analysis models to obtain more information about on-site infrastructure. At the same time, this application retains the manual review process to ensure the accuracy of the results. Especially when facing uncertain or ambiguous situations, manual intervention can provide a final decision.

[0034] (3) Multi-level data analysis: When matching scenes and IDs, a hybrid search was used, which made full use of the collected latitude and longitude, azimuth, feature vector data and other multi-dimensional data, which helped to improve search speed and accuracy. Compared with simply using classification, this application considers the differences between different individuals of the same category. For cases where the physical distance is close, the scene category is the same, but they belong to different facilities, effective differentiation can be made.

[0035] (4) Database optimization and management: The vector database used in this application is a composite relational database that supports efficient query and storage functions. It does not require recording special field thresholds for classification, making it easy to manage and maintain a large amount of historical data. It also provides a flexible data update mechanism to adapt to constantly changing road conditions. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 is a flowchart of a positioning method for a patrol scene according to an embodiment of this application;

[0038] Figure 2 is a schematic diagram of the installation of the intelligent patrol device according to an embodiment of this application;

[0039] Figure 3 is a schematic diagram of the vanishing point height of the acquired image according to an embodiment of this application;

[0040] Figure 4 is a data processing flowchart of an embodiment of this application. Detailed Implementation

[0041] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0042] The purpose of this application is to provide a method and system for locating patrol scenes. It uses an edge-cloud combined system architecture and artificial intelligence image analysis algorithms to achieve accurate and rapid scene location of patrol equipment. At the same time, it uses a vector database to create feature files for road scenes, improves the accuracy of image scene classification, and helps with subsequent data matching and analysis.

[0043] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] This embodiment provides a positioning method for a patrol scene, as shown in Figure 1, including:

[0045] Real-time acquisition of image and location data of the patrol scene;

[0046] Perform semantic segmentation and feature extraction on image data to obtain scene classification results and feature vectors;

[0047] Disease identification is performed based on image data, scene classification results, and feature vectors. If a disease is identified, a mixed search is performed in a preset scene vector database based on location data and feature vectors to obtain mixed search results for the inspection scene.

[0048] The mixed search results are updated to the scene vector database to complete the location of defects in the inspection scene.

[0049] Specifically, this embodiment collects image and location data of the patrol scene in real time by installing intelligent patrol equipment on the patrol vehicle. This intelligent patrol equipment is equipped with intelligent algorithms that can perform semantic segmentation, feature extraction, and image classification on the images, and upload the collected and analyzed data to the cloud. After receiving the collected data, the cloud performs a feature vector hybrid search, obtains the corresponding analysis results based on post-processing, and stores the relevant data in a database.

[0050] The intelligent patrol equipment consists of one or more high-definition cameras and an industrial control computer. The industrial control computer is also connected to a high-precision navigation and positioning receiver, has an internet card, and has a certain amount of AI computing power, which can be used to deploy recognition algorithm programs. In terms of power supply, the intelligent patrol equipment uses DC power. In terms of waterproof performance, the intelligent patrol equipment has good waterproof performance.

[0051] The industrial control computer is installed inside the vehicle and provides power to the vehicle. It features fourth-generation (4G) or fifth-generation (5G) mobile communication technology and connects to a high-precision navigation and positioning receiver and a high-definition camera. It can send commands to the high-definition camera to acquire images and continuously receive, process, and upload the acquired data. Furthermore, the central industrial control computer may also contain a miniature graphics processing unit (GPU), allowing deep neural network algorithms to utilize the GPU to accelerate image processing.

[0052] A high-definition camera, powered by an industrial computer or vehicle, with ≥1 megapixel resolution, provides clear image quality with no delay during acquisition and a frame rate ≥25Hz. It can be installed inside or outside the vehicle, with a fixed viewing angle after installation, and can be used to acquire images of the front or rear of the vehicle. The acquired images must be clear, without significant distortion, and not significantly occluded. The ratio of the height h' from the vanishing point to the top edge of the image to the total image height h, as shown in Figure 3, must be limited to a certain range: 1 / 3 ≤ h' / h ≤ 3 / 4.

[0053] Image vanishing points can be identified by deploying a convolutional neural network classification model on an industrial control computer, which divides the image vertically into n equal intervals and predicts which interval the vanishing point is most likely to be in.

[0054] The high-precision navigation and positioning receiver has a positioning error of ≤1m and is equipped with an inertial navigation system, with a positioning data acquisition frequency of ≥10Hz.

[0055] Furthermore, the data identification and processing in this implementation uses edge device perception and identification + cloud data post-processing execution, and the processing flow and logic are shown in Figure 4.

[0056] The process of semantic segmentation and feature extraction of image data to obtain scene classification results and feature vectors includes: inputting the original image into the semantic segmentation neural network model M1 to output a semantic segmented image; combining and stitching the semantic segmented image with the original image to obtain a combined image; and inputting the combined image into the neural network model M2 to obtain scene classification results and feature vectors.

[0057] The semantic segmentation neural network model M1 uses an image encoding neural network, including an encoder and a decoder. The encoder is used to extract high-dimensional feature encoding vectors from the original image, and the decoder is used to decode the high-dimensional feature encoding vectors into a semantic segmentation image with the same size as the original image.

[0058] The neural network model M2 uses a convolutional neural network, which includes a backbone network and a head network. The backbone network uses multiple convolutional and pooling layers to process the combined image to obtain a high-dimensional feature map, and then uses a global average pooling layer to process the high-dimensional feature map to obtain a feature vector. The head network uses a multi-layer neural network containing residual blocks to process the feature vector and obtain the scene classification result.

[0059] The hybrid search includes: performing preliminary filtering in the scene vector database based on the latitude, longitude and azimuth of the positioning data to obtain a set of record data; calculating the cosine similarity between the feature vector and the set of record data, and further filtering in the set of record data based on the cosine similarity to obtain a set of hybrid search results.

[0060] Updating the scene vector database with mixed search results includes:

[0061] If the mixed search results set is empty, the image data, scene classification results, feature vectors, and location data will be manually reviewed and then added to the scene vector database.

[0062] If the mixed search results set is not empty, and the individual IDs and scene categories in the mixed search results set are the same, then the image data, scene classification results, feature vectors, and location data will be updated to the scene vector database.

[0063] If the mixed search results set is not empty and the individual IDs in the mixed search results set are different, then the feature vectors in the mixed search results set are arithmetically averaged to obtain new feature vectors. A new mixed search is then performed in the scene vector database to obtain a new mixed search results set and vote on it. The corrected individual IDs and scene categories are then obtained and updated in the scene vector database.

[0064] Specifically, in this embodiment, the patrol vehicle acquires raw images m during patrol operations. iThe semantic segmentation neural network model M1 is used to perform semantic segmentation on the image, obtaining the semantically segmented image m. i ';Then convert the original image m i With semantic segmentation image m i 'By combining and stitching the images together, we obtain the integrated image m.' i *; The integrated combined image m i * Input the data into neural network model M2 to obtain the scene classification result c. i And output a corresponding feature vector v i After image processing is completed, the acquired image, c i v i The corresponding latitude, longitude, and azimuth data are uploaded to the cloud server.

[0065] Original image m i The image size is generally [u, v, 3], which means it contains three channels: RGB. The aspect ratio of the image is generally 16:9 or 4:3.

[0066] Azimuth angle, also known as heading angle, is a reference direction for high-definition cameras when acquiring images. It is set to 0° in true north and increases clockwise. The data range is [0, 360°].

[0067] Semantic segmentation neural network model M1 refers to a type of image encoding and decoding neural network that first processes the original image m through a feature extraction encoder neural network layer. i The feature vector is extracted into a high-dimensional feature encoding vector, and then decoded into a semantic segmentation image m with the same pixel size as the original image through a feature decoder neural network layer. i Semantic segmentation primarily involves classifying road surfaces, traffic safety facilities such as guardrails, roadside facilities such as signs, overpasses, and elevated structures in an image. Each category is assigned a specified value from [100, 255]. Unlabeled areas are uniformly classified as the "background" category and marked with 0. This process achieves category labeling for all pixels in the image, thereby generating a semantically segmented image m. i The image size is [u, v]. A specified value from [100, 255] is assigned according to the category. This means selecting j integers from [100, 255], corresponding to j facilities that the image semantic recognition network can recognize, with one integer assigned to each category.

[0068] Original image m i With semantic image m i Combining images refers to combining two images along their channel directions to form a new combined image m. i* The image size is [u, v, 4], and the 4 channels are the RGB image and the semantic segmentation image, respectively.

[0069] The neural network model M2 consists of two parts: the backbone network, which is a convolutional neural network trained according to Re-Identification (ReID); and a fully connected classification head network added on the basis of the backbone network for category classification.

[0070] 1) Backbone network part: The model first combines the images m i * Preprocessing such as downsizing and normalization is performed, and a multi-layer convolutional neural network (CNN) + pooling network is used to process the combined image m. i The process involves processing the data to obtain a high-dimensional feature map [u', v', k]. Global average pooling is then used to further process this high-dimensional feature map, resulting in a 512-dimensional feature vector v. i .

[0071] 2) Header network section: v i The input is further processed into a multi-layer neural network containing residual blocks, and the network outputs k categories and their corresponding confidence scores. These k categories correspond to k scene classifications that the model needs to perform, including but not limited to categories such as "ground road," "expressway," "auxiliary road," "tunnel," and "road under elevated road." The category with the highest confidence score is taken as the scene classification result determined by the M2 model, denoted as c. i .

[0072] After completing the analysis, the industrial control computer will... i c i v i And the latitude and longitude corresponding to the time [lng] i lat i ], azimuth i This information is uploaded to the cloud server. The latitude and longitude coordinates are included. i lat i The numerical precision is 0.000001.

[0073] After receiving new collected data, the cloud server will use latitude and longitude data [lng] i lat i Azim data i , eigenvector v i Category c i The data is subjected to a hybrid search in the scene vector database in the cloud. Based on the results of the hybrid search, the corresponding processing logic is executed to complete the scene judgment of the acquired image and update the scene vector database in the cloud.

[0074] The scene vector database is a composite relational database containing multiple records. Each record includes the URL of the captured image, the image's latitude and longitude [lng, lat], the azimuth angle of the captured image (azi), the scene classification result (c), the individual ID of the image, the feature vector (v) after feature extraction, the record's creation time (CREATETIME), whether it has been checked by an auditor, and whether it has been marked for deletion (DELETE). The data in the vector database is dynamically changing; verified data can be added to the vector database, and auditors can also periodically clean and verify the vector database to correct the scene categories of the images.

[0075] Hybrid search refers to initially filtering data in the database based on the latitude, longitude, and azimuth of the uploaded data, selecting [lng] i lat i Nearby, azimuth and azi i Similar record datasets Vset i Then, using the cosine distance, the eigenvector v is calculated. i and Vset i The similarity is set by setting a threshold T for cosine similarity from Vset. i Select a new set of vectors with similarity ≥ T from the results to form the mixed search results set Vset. i '.

[0076] Based on the mixed search set Vset i In the case of ', corresponding post-processing should be performed:

[0077] 1) Vseti' is an empty set, meaning the vector database is located in latitude and longitude [lng] i lat i Nearby, there is no [v] i Similar vectors will then be the v uploaded by the device. i c i m i Once the data is pushed to the confirmation queue and verified by the auditor, it is added to the vector database Vdb.

[0078] 2) If Vseti' is not an empty set, and all individual IDs of Vseti' are the same, then the scene category is c. i Then the scene classification result corresponding to the collected data is determined to be c. i and v i c i Data such as individual IDs are updated and appended to Vdb.

[0079] 3) If Vseti' is not an empty set and the individual IDs of Vseti' are not identical, then the eigenvectors in Vseti' are arithmetically averaged to obtain a new eigenvector v", and then [lng] is used. i lat i ]、azi i 'v' performs new feature retrieval in Vdb, obtains the e vectors with the highest similarity, and forms a set to obtain a new feature vector set Vset. i ", voting is conducted based on the individual IDs corresponding to the vectors, and the scene category c" corresponding to the most likely individual ID in the voting results is taken as the scene category of the acquired image, and c" and [lng i lat i ]、azi i Data such as the most likely individual ID is updated in Vdb.

[0080] Voting methods include, but are not limited to:

[0081] i. Majority voting: Among the e vectors, the scenario category with the highest number of votes is used as the voting result.

[0082] ii. Weighted voting: Voting is weighted based on cosine distance. The smaller the distance, the more votes can be cast. Finally, the individual ID with the most votes is counted among the e vectors.

[0083] Furthermore, after updating the mixed search results to the scene vector database, the process also includes regularly cleaning and reviewing the scene vector database manually.

[0084] Specifically, auditors can use auxiliary tools to periodically clean the existing vector database Vdb, including but not limited to the following methods:

[0085] 1) Using a new neural network model M2' that is different from the neural network model M2, the data in Vdb is re-identified, and data records whose identification results are different from those in Vdb are selected and pushed to the confirmation queue. The auditor will verify the data, correct any mismatched scenario categories, and update the database.

[0086] 2) Extract images with the same individual ID from Vdb and clean the record data that does not belong to that ID.

[0087] To further optimize the technical solution, this embodiment also provides a positioning system for patrol scenarios, including: an edge device and a cloud device. The edge device is used to collect image data and positioning data of the patrol scenario in real time, and to extract features from the image data to obtain scene classification results and feature vectors. The cloud device is used to process the image data, positioning data, scene classification results and feature vectors to obtain the disease results and disease location of the patrol scenario.

[0088] Furthermore, the edge devices include an industrial computer, a high-definition camera, and a navigation and positioning receiver. The high-definition camera and the navigation and positioning receiver are connected to the industrial computer, which provides data transmission, image feature extraction, and power supply functions.

[0089] Specifically, the edge devices are mainly vehicle-mounted intelligent patrol devices, and the relevant descriptions of the devices are as follows:

[0090] 1) Equipped with an industrial PC featuring an NVIDIA GT1030 graphics card, capable of running deep learning algorithms. The industrial PC uses a DC power supply to ensure stable operation. It also boasts excellent waterproofing, making it suitable for various outdoor conditions.

[0091] 2) The industrial control computer has a built-in high-precision GPS module and Internet card, which can collect latitude, longitude and azimuth angles at a frame rate of 10Hz and support real-time data transmission to the cloud server.

[0092] 3) A high-definition camera is installed on the roof of the vehicle and connected to the industrial control computer. The camera captures images with a resolution of 2 million pixels and a frame rate of 25Hz. The images are clear and unobstructed. The camera is facing directly in front of the vehicle and has a horizontal viewing angle.

[0093] The data acquisition and processing flow for edge devices is as follows:

[0094] 1) After the edge device is powered on, it will collect GPS data from the GPS sensor and determine whether the vehicle is currently patrolling based on the GPS status. When the vehicle is in motion, the industrial control computer will accumulate the vehicle's displacement. When the accumulated distance reaches 10m, the industrial control computer will issue a command to collect images and GPS data and start a new distance accumulation from 0.

[0095] 2) Images m after acquisition i The input will be given to a semantic segmentation neural network model M1. This model mainly performs semantic perception on traffic safety facilities, road surfaces, bridges, and other facilities in the image, performs image processing, and generates a semantic segmentation image mi' with the same pixel size as the original image m. i Same, m i'Assigns a category to each pixel within the image range, including guardrails, anti-glare panels, road surfaces, bridges, traffic signs, sky, warning poles, background, etc., and assigns a specific integer [0, 255) based on the category.

[0096] 3) Transfer image m i With semantic segmentation image m i ', stitching together according to channels to obtain a new combined image m i *, m i After scaling, the input to the neural network model M2 will output a 512-dimensional vector v. i It will output a k-dimensional confidence array, corresponding to k scene categories such as "expressway", "surface road", and "tunnel". The scene category with the highest confidence is the one used by the model for inference, denoted as c. i .

[0097] 4) Transfer the collected image data m i Latitude and longitude data [lng] i lat i ], azimuth i And the analysis results data v i c i The data is then aggregated and uploaded to the cloud device.

[0098] Furthermore, the cloud-based device includes a disease identification module, a hybrid search module, and a post-processing module. The disease identification module is used to identify diseases based on image data, scene classification results, and feature vectors. The hybrid search module is used to perform a hybrid search in a preset scene vector database based on location data and feature vectors when a disease is identified, to obtain hybrid search results for the inspection scene. The post-processing module is used to update the hybrid search results to the scene vector database and send them to the corresponding maintenance personnel in combination with the disease identification results.

[0099] Specifically, cloud devices mainly consist of cloud servers, which are equipped with data receiving programs, scene vector databases, data processing and analysis programs, and order dispatching programs.

[0100] 1) Disease identification module:

[0101] The cloud server received new data, including image data m i Latitude and longitude data [lng] i lat i ], azimuth i And the analysis results data v i c iThe image data will be input into the disease identification model. If a disease is identified in the image, the collected data will be sent to the post-processing program for further analysis. In this embodiment, the disease identification model adopts a target recognition neural network, which can mark the identified disease category and the location of the recognition box on the image.

[0102] 2) Hybrid Search Module:

[0103] After receiving the data, latitude and longitude [lng] will be used. i lat i Azim data i Feature vector data v i A mixed search is performed in the MILVUS vector database. The MILVUS vector database contains the following fields:

[0104] lng: longitude;

[0105] lat: latitude;

[0106] azi: azimuth angle, with a value range of [0, 360).

[0107] vec: Feature vector;

[0108] CHECK: Whether the auditor has performed verification. 1 indicates verification has been performed, and 0 indicates verification has not been performed.

[0109] DELETE: Whether the data is marked for deletion, 1 for marked for deletion, 0 for not marked for deletion;

[0110] CREATETIME: Data collection time;

[0111] c: Scene category;

[0112] ID: Individual ID.

[0113] When performing a hybrid search, the following search criteria are primarily followed:

[0114] i、lng i -0.00001*8 <lng i +0.00001*8;

[0115] ii, lat i -0.00001*8 <lat i +0.00001*8;

[0116] iii、azi i -5 <azi<azi i +5(azi i ∈[5, 355]);

[0117] 0 <azi<azi i +5∪azi>azi i –5+360(azi i ∈[0, 5);

[0118] azi i -5 <azi<360∪azi<azi i +5-360(azi i ∈(355, 360));

[0119] iv. DELETE=0, CHECK=1;

[0120] v. The vector obtained from the search has a cosine distance > 0.8.

[0121] Based on the constraints of the hybrid search, the search result set Vset is obtained. i For the search results set Vset i The analysis includes:

[0122] i、Vset i ' is an empty set, meaning the vector database is located in latitude and longitude [lng] i lat i Nearby, there is no [v] i Similar vectors will then be the v uploaded by the device. i c i m i Once the data is pushed to the confirmation queue and verified by the auditor, it is added to the vector database Vdb.

[0123] ii. If Vset i 'Not an empty set, and Vset i All individual IDs are the same, and the scene category is c. i Then the scene classification result corresponding to the collected data is determined to be c. i and v i c i Data such as individual IDs are updated and appended to Vdb.

[0124] iii. If Vset i 'Not an empty set, and Vset i If the individual IDs of 'Vset' are different, then Vset will be... i The eigenvectors in ' are used to perform an arithmetic mean to obtain a new eigenvector v", and then [lng] is used to perform an arithmetic mean. i lat i ]、azi i The function "v" retrieves new features from Vdb, obtains the 10 vectors with the highest similarity, and forms a set to obtain a new feature vector set Vset.i ", voting is conducted based on the individual IDs corresponding to the vectors, and the scene category c" corresponding to the most likely individual ID in the voting results is taken as the scene category of the acquired image, and c" and [lng i lat i ]、azi i Data such as the most likely individual ID is updated in Vdb. The voting logic used is to weight voting based on cosine distance. The smaller the distance, the more votes can be cast. The number of votes is [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]. Finally, the individual ID with the most votes is counted among the 10 vectors.

[0125] 3) Order dispatch module:

[0126] Post-processed data, if not requiring auditor verification, will be directly pushed to the dispatch module; if auditor verification is required, the relevant data will be pushed to the dispatch module after the auditor's verification is approved. Within the dispatch module, the defect data will be matched to road segments through scene category and facility individual ID, then pushed to specific maintenance teams and archived for further analysis.

[0127] Furthermore, auditors can use auxiliary tools to periodically clean the existing scene vector database (Vdb), including the following methods:

[0128] 1) A deeper neural network model M2' re-identifies the image data in Vdb, selects data records whose identification results differ from those in Vdb, pushes them to the confirmation queue, verifies them, corrects the mismatched scene categories, and updates them in the database.

[0129] 2) Extract images with the same individual ID from Vdb and clean the record data that does not belong to that ID.

[0130] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0131] The embodiments described above are merely preferred embodiments of this application and are not intended to limit the scope of this application. Any modifications and improvements made to the technical solutions of this application by those skilled in the art without departing from the spirit of this application shall fall within the protection scope defined by the claims of this application.

Claims

1. A positioning method for a patrol scene, characterized in that, The location methods for the patrol scene include: Real-time acquisition of image and location data of the patrol scene; The image data is semantically segmented and feature extracted to obtain scene classification results and feature vectors; Based on the image data, the scene classification results, and the feature vector, disease identification is performed. If a disease is identified, a mixed search is performed in a preset scene vector database based on the positioning data and the feature vector to obtain the mixed search results of the inspection scene. The mixed search results are updated to the scene vector database to complete the location of the disease in the inspection scene.

2. The positioning method for a patrol scene according to claim 1, characterized in that, Semantic segmentation and feature extraction are performed on the image data to obtain scene classification results and feature vectors, including: The original image is input into the semantic segmentation neural network model M1, and the output is a semantically segmented image; The semantic segmentation image is combined and stitched with the original image to obtain a combined image; The combined image is input into the neural network model M2 to obtain the scene classification result and the feature vector.

3. The positioning method for a patrol scene according to claim 2, characterized in that, The semantic segmentation neural network model M1 employs an image encoding neural network, including an encoder and a decoder; the encoder is used to extract the original image into a high-dimensional feature encoding vector; the decoder is used to decode the high-dimensional feature encoding vector into a semantic segmentation image with the same size as the original image.

4. The positioning method for a patrol scene according to claim 2, characterized in that, The neural network model M2 employs a convolutional neural network, comprising a backbone network and a head network. The backbone network uses multiple convolutional and pooling layers to process the combined image, obtaining a high-dimensional feature map. Then, a global average pooling layer is used to process the high-dimensional feature map to obtain the feature vector. The head network uses a multi-layer neural network containing residual blocks to process the feature vector and obtain the scene classification result.

5. The positioning method for a patrol scene according to claim 1, characterized in that, The hybrid search includes: Based on the latitude, longitude and azimuth in the positioning data, a preliminary screening is performed in the scene vector database to obtain a set of recorded data. Calculate the cosine similarity between the feature vector and the record data set, and then further filter the record data set based on the cosine similarity to obtain a mixed search results set.

6. The positioning method for a patrol scene according to claim 5, characterized in that, Updating the hybrid search results to the scene vector database includes: If the mixed search results set is empty, the image data, the scene classification results, the feature vectors, and the positioning data are manually reviewed and then added to the scene vector database. If the mixed search results set is not empty, and the individual IDs and scene categories in the mixed search results set are the same, then the image data, the scene classification results, the feature vectors, and the location data are updated to the scene vector database. If the mixed search results set is not empty and the individual IDs in the mixed search results set are different, then the feature vectors in the mixed search results set are arithmetically averaged to obtain new feature vectors, and a new mixed search is performed in the scene vector database to obtain a new mixed search results set and vote on it. The corrected individual IDs and scene categories are then obtained and updated in the scene vector database.

7. The positioning method for a patrol scene according to any one of claims 1-6, characterized in that, After updating the mixed search results to the scene vector database, the process also includes periodic cleaning and review of the scene vector database by human intervention.

8. A positioning system for a patrol scenario, characterized in that, The positioning system for the patrol scene includes an edge device and a cloud device. The edge device is used to collect image data and positioning data of the patrol scene in real time, and to extract features from the image data to obtain scene classification results and feature vectors. The cloud device is used to process the image data, the positioning data, the scene classification results, and the feature vectors to obtain the disease results and disease location of the patrol scene.

9. The positioning system for patrol scenarios according to claim 8, characterized in that, The edge device includes an industrial control computer, a high-definition camera, and a navigation and positioning receiver; the high-definition camera and the navigation and positioning receiver are connected to the industrial control computer; the industrial control computer is used to provide data transmission, image feature extraction, and power supply functions.

10. The positioning system for patrol scenarios according to claim 8, characterized in that, The cloud-based device includes a disease identification module, a hybrid search module, and a dispatch module. The disease identification module is used to identify diseases based on the image data, the scene classification results, and the feature vectors. The hybrid search module is used to perform a hybrid search in a preset scene vector database based on the location data and the feature vectors when a disease is identified, to obtain hybrid search results for the inspection scene. The dispatch module is used to update the hybrid search results to the scene vector database and send them to the corresponding maintenance personnel in conjunction with the disease identification results.

Images