Well lid intelligent identification method
Through deep learning technology and timing prediction model, combined with integrated embedded equipment, the automatic detection of manhole covers and future status prediction of manhole covers is solved, and efficient and intelligent manhole cover monitoring is achieved.
Patent Information
- Application Number
- CN202411611802.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-12
- Publication Date
- 2025-03-14
AI Technical Summary
The existing manhole cover detection methods are inefficient and have limited accuracy, and cannot achieve automated and intelligent monitoring, especially when facing environmental changes and complex manhole cover states.
Deep learning technology is used to combine integrated embedded equipment and timing prediction models to realize automated detection of manhole covers and future state prediction. The specific steps include: integrating embedded devices to take photos in real time and transmit image data to the server, the server performs image processing and deep learning model recognition, and combining LSTM models to predict future states.
It improves the efficiency and accuracy of manhole cover detection, realizes automated data acquisition and processing, and can predict the future status of manhole cover, and dynamically adjusts computing resources to improve detection efficiency.
Smart Images
Figure CN119625700A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of manhole cover identification and monitoring, and in particular to a manhole cover intelligent identification method. Background Art
[0002] Existing manhole cover detection methods mainly rely on manual inspection or combined with sensors and image processing technology, such as the Chinese patent application with application number 2023206699601, which uses a sensor system to remotely monitor whether the manhole cover is tilted, and the Chinese patent application with application number 2019109122223, which combines image processing technology to monitor the state of the manhole cover. However, sensors are sensitive to environmental changes, such as light and temperature, which may affect the performance of sensors and the accuracy of data. Moreover, sensors can usually only detect specific physical quantities and may not be able to fully capture the complex state of manhole covers (such as damage, displacement, collapse, etc.). Image processing technology relies on manual feature extraction, which often requires professional knowledge to design and is not applicable to all scenarios. For manhole covers with changing lighting, occlusion, different materials and textures, when facing new or unknown types of manhole covers, the algorithm needs to be redesigned and adjusted. Therefore, although these methods can meet the detection needs to a certain extent, they are often limited by low efficiency, limited accuracy, and difficult to achieve automated and intelligent monitoring. Summary of the invention
[0003] In order to solve the technical problems of low detection efficiency and accuracy in the prior art and inability to predict the future state of manhole covers, a manhole cover intelligent identification method is provided.
[0004] The present invention adopts the following technical solution: Step 1, placing the integrated embedded device above the manhole cover, the integrated embedded device takes a picture of the manhole cover in real time and transmits the image data to the server in real time; Step 2: The server processes the received image data, generates a detection item containing a unique identifier, a comprehensive embedded device ID, and original image information, and adds it to the Redis queue; Step 3: The manhole cover detection module in the server takes out the detection items, puts them into the deep learning model for recognition, updates the manhole cover status data based on the obtained manhole cover status feedback, caches the data, and presents it on the web page.
[0005] Furthermore, the integrated embedded device in step 1 includes an image acquisition device and a communication module. The image acquisition device obtains the manhole cover photo and transmits it to the server via WiFi through the communication module.
[0006] Furthermore, the image data sent to the server in step 1 includes the integrated embedded device number (CameraID) and the taken photo, which are received by the server in the form of a <integrated embedded device number, image> key-value pair.
[0007] Furthermore, step 2 is specifically as follows: After receiving the key-value pair of the integrated embedded device number and the image, the backend data receiving module running on the server first generates a unique ID (UUID) based on the current time through the snowflake algorithm to mark the key-value pair, and then generates a data structure called "detection item" based on the key-value pair. The fields of this data structure are designed as follows: Test items Detect UUID; Comprehensive embedded device ID; Original image; Detect images; time; state; } After the corresponding detection items are generated, they are placed in the detection request queue in Redis.
[0008] Furthermore, step 3 is specifically as follows: After the manhole cover detection module continuously takes out the detection items to be detected from the detection request queue, it reads the original image and performs defogging first, that is, adds AODNet defogging preprocessing to meet the use requirements of multiple scenarios, and then sends it to the target detection model for target recognition and obtains the recognition result. After obtaining the result, the "status" field and "detection image" field in the detection item data structure are updated according to the classification result, presented on the front-end page, and the data is cached as historical data.
[0009] Furthermore, the target detection model described in this application is a Yolov9+botnet model.
[0010] Furthermore, based on the historical data of hidden dangers of urban manhole covers obtained in step 3, the LSTM model is used to identify the patterns and trends of the number and distribution of hidden dangers of urban manhole covers in the past, and to predict the future situation of the number and distribution of hidden dangers of urban manhole covers.
[0011] Furthermore, according to the prediction results, the allocation of computing resources is dynamically adjusted, and more computing resources are allocated to the images transmitted by the integrated embedded devices corresponding to the manhole covers predicted to have problems.
[0012] The beneficial effects of the present invention are: The present invention closely combines deep learning technology with manhole cover application scenarios, uses deep learning technology to perform manhole cover detection, improves the efficiency and accuracy of detection, and also combines the time series prediction model to predict the future state of the manhole cover. Moreover, through close integration with hardware equipment, it realizes automated data collection and processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 A flowchart of the method is shown in FIG. Figure 2 To detect image one; Figure 3 To detect image 2; Figure 4 To detect image three; Figure 5 It is a comparison between the predicted value and the actual value of various states of the manhole cover 1. DETAILED DESCRIPTION
[0014] In order to make the purpose, technical scheme and technical effect of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Example
[0015] like Figure 1 As shown, the method of the present invention comprises: Step 1: Place the integrated embedded device above the manhole cover, and the internal image acquisition device acquires images. After the relevant IP address is automatically configured through the built-in WiFi module of the communication module, it automatically connects to the server. A long connection is established with the server through WebSocket to avoid the communication module frequently requesting data, frequently connecting and disconnecting, and affecting the server performance. After the long connection is established, the communication module sends the integrated embedded device number (CameraID) and the image acquired by the image acquisition device to the server at a certain interval. They are received by the server in the form of <integrated embedded device number, image> key-value pairs.
[0016] Step 2: After receiving the key-value pair of the integrated embedded device number and the image, the backend data receiving module running on the server first generates a unique ID (UUID) to mark the key-value pair based on the current time through the snowflake algorithm, and then generates a data structure called "detection item" based on the key-value pair. The fields of the data structure are designed as follows: Test items Detect UUID; Comprehensive embedded device ID; Original image; Detect images; time; state; } in: The detection UUID is a 64-bit integer generated by the snowflake algorithm that uniquely identifies the data item; The integrated embedded device ID is a 32-bit integer that uniquely identifies the integrated embedded device; The original image is a binary image acquired by the image acquisition device; The detection image is the manhole cover detection result image given by the manhole cover detection module, which is empty when the data structure is constructed by acquiring the comprehensive embedded device data. The time is the time when the communication module sends the image to the server, which is generated by the server; ·The status of the data is marked, which is divided into five states: {Undetected, Normal, Damaged, Missing, Uncovered (Upturned), Manhole Ring Problem, Normal (Ignored)}. "Undetected" corresponds to the detection item that has not been read by the manhole cover detection module and the detection result has not been given; "Normal", "Damaged", "Missing", "Uncovered (Upturned)", and "Manhole Ring Problem" describe the status of the corresponding manhole cover; the normal (ignored) state is to cooperate with the intelligence to reduce the load, and the image is not detected for a period of time, but the manhole cover monitored this time is directly considered to be normal based on the previous detection results, and it is marked as "Normal (Ignored)".
[0017] After the corresponding detection items are generated, they are placed in the detection request queue in Redis.
[0018] Step 3: After the manhole cover detection module in the server continuously takes out the detection items to be detected from the detection request queue, it reads the original image and performs a defogging preprocessing operation, and then sends the defogging image to the target detection model for target recognition to obtain the detection image, such as Figure 2 , 3 As shown in Figure 4, after obtaining the results, the "status" field and "detection image" field in the detection project data structure are updated according to the classification results and presented on the front-end page. They are "uncovered (tilted)", "normal", and "missing" states respectively. This target detection model is an optimized model based on Yolov9.
[0019] YOLOv9 is a regression-based target detection model. The model structure can be divided into the input end, backbone network, neck network, and prediction output end. The network structure is an existing technology and will not be repeated here.
[0020] The present invention improves Yolov9, and the improvement process is as follows: In the present invention, in Yolov9, by modifying the backbone, RepNBottleneck in the RepNCSPELAN4 module is replaced with BottleneckTransformer, wherein BottleneckTransformer is a convolutional block including the MHSA mechanism, thereby forming a Yolov9+botnet target detection model. Yolov9+botnet is an improved Yolov9 model.
[0021] Training Yolov9+botnet model: 1. Collect manhole cover pictures. The sources of manhole cover pictures include open source manhole cover photos or public data sets on the Internet, and manhole cover pictures provided by various competitions.
[0022] 2. Use labellmg to label and classify the manhole cover images, frame each type of manhole cover in the image and label it, and export it into the txt format required by Yolov9. Organize the labeled images into a format that the Yolov9 model can recognize, including images of the training set and validation set and the real label files.
[0023] 3. Input the annotated image into the model, which extracts image features through multiple convolutional layers and activation functions, and finally outputs the prediction results, including the location of the bounding box, the confidence of the object, and the category probability. Compare the model's prediction results with the true label (i.e. the annotated bounding box, confidence, and category). The true label is the label file annotated previously. Use a composite loss function to calculate the loss, which includes positioning loss (such as IoU loss), confidence loss, and classification loss. According to the loss function, calculate the gradient of the model parameters. Use the optimizer to update the model's weights according to the gradient to reduce the loss. After each epoch (a round of training), the model will be evaluated on the validation set to monitor the performance and overfitting of the model.
[0024] 4. According to the training results, gradually adjust the parameters, such as learning rate, to achieve the best training results. The training results are evaluated by mAP50 and mAP50-95, Precision, and Recall.
[0025] The AOD-NET model is used for dehazing. Each convolution layer of the AOD-NET model has only three convolution kernels and few model parameters, so it has the characteristics of small computational complexity, fast speed, and suitable for real-time deployment. The dataset for dehazing model training is based on a simplified atmospheric scattering model. Among them, the atmospheric scattering model and the AOD-NET model are both existing technologies and will not be described in detail here.
[0026] Create multiple lists in Redis, each named after the integrated embedded device number, and each list is specifically used to record the latest 100 detection records corresponding to the integrated embedded device number. After obtaining the test results of the test item and updating the relevant data fields in the "Test Item", read the integrated embedded device number and insert it to the end of the corresponding list in Redis to cache the data. If the number of list elements has reached 100, pop up the head element of the list (that is, the first element to be put in) and continue to insert it to the end of the list. In this way, the head element represents the earliest detection record, and the tail element represents the latest detection record, realizing dynamic update.
[0027] Step 4, use a model based on long short-term memory network (LSTM) to predict the number and distribution of hidden dangers of urban manhole covers, such as Figure 5 shown.
[0028] The LSTM model identifies the number and distribution patterns and trends of hidden dangers of urban manhole covers in the past based on the historical data of hidden dangers of urban manhole covers, and predicts the number and distribution of hidden dangers of urban manhole covers in the future. Based on the prediction results, the allocation of computing resources is dynamically adjusted, and more computing resources are allocated to the images transmitted by the integrated embedded device corresponding to the manhole covers with the prediction results of being damaged, missing, uncovered (lifted), etc.
[0029] The historical detection data of a manhole cover hidden danger is organized into a time series format, and data preprocessing is performed to remove missing values, outliers, and erroneous data to ensure the accuracy and reliability of the data. Determine a time window size, that is, how much historical data of manhole cover hidden dangers are used to predict a future data point. The time window size of this embodiment is 30. The historical detection data sequence of manhole cover hidden dangers is divided into multiple samples according to a fixed time step window, and each sample contains a section of historical detection values of manhole cover hidden dangers and their corresponding target values. In this way, a series of input-output pairs are constructed, in which the input is a window of historical detection data of manhole cover hidden dangers, and the output is a future state of the manhole cover. Using the constructed input-output pairs, the LSTM neural network is trained. The hidden layer of the LSTM neural network prediction model contains three LSTM layers, two Dropout layers, and one Dense Layer layer.
[0030] The specific structure of the LSTM neural network prediction model includes: The first layer has 256 neurons, and the output value of the first layer completely enters the next layer. The second layer discards neurons with a probability of 0.2. The third layer has 128 neurons, and the output values of the third layer all go to the next layer. The fourth layer discards neurons with a probability of 0.2. The fifth layer has 64 neurons, and all the output values of the fifth layer go to the next layer. The sixth layer is a fully connected layer that outputs a predicted value.
[0031] During the model training phase, the difference between the model output and the actual data is evaluated, and this difference is quantified by the loss function. Based on this loss value, the gradients of the model parameters are calculated, which indicate how the parameters should be adjusted to reduce the loss. Then, the model parameters are updated through the gradient descent algorithm in order to obtain a smaller loss value in the next iteration. This process is repeated until the model training reaches a predetermined number of iterations or the performance is no longer significantly improved. After training is completed, the LSTM model can use the learned parameters to predict the state of a specific manhole cover at the next moment.
[0032] It should be understood that the parts not elaborated in detail in this invention belong to the prior art. The above combined with the accompanying drawings are only the specific implementation methods and processes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the art should understand that this is only an example, and various changes and substitutions can be made to this implementation method without departing from the essence of the present invention. The scope of the present invention is limited only by the attached claims.
Claims
1. A method for intelligent identification of manhole covers, characterized in that: include: Step 1, placing the integrated embedded device above the manhole cover, the integrated embedded device takes a picture of the manhole cover in real time and transmits the image data to the server in real time; Step 2: The server processes the received image data, generates a detection item containing a unique identifier, a comprehensive embedded device ID, and original image information, and adds the detection item to a Redis queue; Step 3: The manhole cover detection module in the server takes out the detection items, puts them into the deep learning model for recognition, updates the manhole cover status data based on the obtained manhole cover status feedback, caches the data, and presents it on the web page.
2. A method for intelligent identification of manhole covers according to claim 1, characterized in that: In step 1, the integrated embedded device includes an image acquisition device and a communication module. The image acquisition device obtains the manhole cover photo and transmits it to the server through the communication module.
3. A method for intelligent identification of manhole covers according to claim 1, characterized in that: In step 1, the image data includes the integrated embedded device number and the taken photo, which are received by the server in the form of a <integrated embedded device number, image> key-value pair.
4. A method for intelligent identification of manhole covers according to claim 1, characterized in that: Step 2 is as follows: After receiving the key-value pair of the integrated embedded device number and the image, the back-end data receiving module running on the server first generates a unique ID to mark the key-value pair based on the current time through the snowflake algorithm, and then generates a "detection item" data structure based on the key-value pair. The fields of the data structure are designed as follows: Test items Detect UUID; Comprehensive embedded device ID; Original image; Detect images; time; state; ｝ After the corresponding detection items are generated, they are placed in the detection request queue in Redis.
5. A method for intelligent identification of manhole covers according to claim 4, characterized in that: In step 3, the manhole cover detection module continuously takes out the detection items to be detected from the detection request queue, reads the original image, performs dehazing processing, and then sends it to the target detection model for target recognition to obtain the recognition result; after obtaining the result, the "status" field and "detection image" field in the detection item data structure are updated according to the classification result, presented on the front-end page, and the data is cached as historical data.
6. A method for intelligent identification of manhole covers according to claim 5, characterized in that: The target detection model is the Yolov9+botnet model.
7. A method for intelligent identification of manhole covers according to claim 5, characterized in that: Based on the historical data of hidden dangers of urban manhole covers obtained in step 3, the LSTM model is used to identify the patterns and trends of the number and distribution of hidden dangers of urban manhole covers in the past, and to predict the future situation of the number and distribution of hidden dangers of urban manhole covers.
8. A method for intelligently identifying manhole covers according to claim 7, characterized in that: According to the prediction results, the allocation of computing resources is dynamically adjusted, and more computing resources are allocated to the images transmitted by the integrated embedded devices corresponding to the manhole covers predicted to have problems.