City well lid state identification method and system based on skynet video

Abstract

The invention relates to the technical field of image target detection, in particular to an urban well lid state recognition method and system based on a skynet video, which monitors and recognizes the well lid state in real time through a deep learning YOLOV5 algorithm, can realize all-weather and dead-corner-free well lid monitoring, improves the intelligent level of well lid management, reduces the workload of manual inspection, and meanwhile, improves the efficiency of well lid management. By using the existing skynet video data, additional installation of a sensor is not needed, the hardware input cost can be reduced, and in combination with the real-time processing capability of the video stream, the system can quickly respond once the abnormal condition of the well lid is found, so that the time difference from problem finding to problem solving is shortened, and the emergency processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image target detection, and in particular to a method and system for identifying the status of a city manhole cover based on Skynet video. Background Art

[0002] With the acceleration of urbanization, the management of urban manhole covers has become an important issue in urban management. The number of manhole covers is huge, the types are diverse and they are widely distributed. The traditional manual inspection method has many disadvantages such as long time consumption, low efficiency and high labor cost when facing tasks such as manhole cover census and status monitoring. In some early traditional urban intelligent manhole cover monitoring systems, the underlying acquisition system collects relevant data of the manhole cover through sensors, processes and analyzes it with 51 single-chip microcomputers to obtain the working status of the manhole cover, and then transmits data through technologies such as Zig-Bee and 51 single-chip microcomputers. There are great limitations, such as the sensor may be affected by the environment and the need for additional installation of equipment.

[0003] At the same time, the number of cameras in cities is increasing, especially the huge video surveillance network built by the "Skynet Project". The development and application of its massive video data in various civilian fields also needs further development. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method and system for identifying the status of urban manhole covers based on Skynet video. By utilizing the video data of the "Skynet Project" and combining computer vision and artificial intelligence technology to identify the status of urban manhole covers, it can reduce the workload of manual inspections, reduce the investment in hardware equipment sensors, effectively control costs, and collect and analyze the operating data of manhole covers through the system, providing scientific decision-making support for urban managers.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for identifying the status of a city manhole cover based on Skynet video, comprising:

[0006] S101. Receive Skynet video data and perform data preprocessing, analyze the corresponding position of the video image and convert the image data format so that the data can be directly input into the manhole cover status recognition model;

[0007] S102. Receive the image data processed by S101, use the image recognition neural network model to identify the state of the manhole cover, and obtain the state of the manhole cover corresponding to the current image;

[0008] S103. When the current image manhole cover state is abnormal, a warning is issued and the location of the abnormal manhole cover is indicated;

[0009] S104. Summarize all detection data and detection results, and display them visually in real time. The display includes at least: the number of detection scenes, the number of manhole covers, and abnormal alarm information.

[0010] In particular, the manhole cover status recognition model is a YOLOV5 deep learning model.

[0011] In particular, when the manhole cover state recognition model needs to be deployed in a resource-constrained scenario, the urban manhole cover state recognition method will quantize and prune the manhole cover state recognition model.

[0012] Furthermore, the manhole cover status identified by the manhole cover status recognition model includes at least: normal, lost, damaged, and displaced, among which lost, damaged, and displaced are abnormal states of the manhole cover.

[0013] Furthermore, when an abnormal state of a manhole cover is detected, the manhole cover state identification method also automatically generates a manhole cover repair order and sends it to the terminal of the person in charge of the preset area.

[0014] In particular, the S104 abnormal alarm information display includes at least: the abnormal manhole cover location, the abnormal state of the manhole cover, whether the manhole cover abnormal dispatch has been received, and whether the manhole cover has been repaired.

[0015] Furthermore, the manhole cover identification method also separately records the information of the manhole cover in abnormal state and marks it as "unrepaired". When the maintenance personnel restore the manhole cover, the next time the manhole cover identification method identifies that the state of the manhole cover at the current position is normal, the separately recorded abnormal state manhole cover will be re-marked as "repaired", and the warning information will be eliminated in the corresponding visual display.

[0016] In particular, the manhole cover status recognition method also continuously collects Skynet data and regularly iterates and upgrades the manhole cover status recognition model.

[0017] Preferably, the present invention also provides a city manhole cover status recognition system based on Skynet video, comprising:

[0018] Data receiving module: The data receiving module is connected to the "Skynet Project" to obtain real-time monitoring video data;

[0019] Data processing module: The data processing module analyzes the corresponding positions of the video image and converts the image data format so that the data can be directly input into the manhole cover status recognition model;

[0020] The model inference module uses a manhole cover state recognition neural network model to recognize the manhole cover state of the received image data. The recognition state includes at least: normal, missing, damaged, and displaced;

[0021] Abnormal alarm module: when the system identifies that the manhole cover is in abnormal state, the abnormal alarm module will issue a warning;

[0022] Abnormal dispatching module: when the system identifies that the manhole cover is in abnormal condition, the system automatically generates a repair order and dispatches it to the terminal of the person in charge of the preset area;

[0023] The status display module receives all detection data and test results in real time and displays them visually in real time. The display includes at least: the number of detection scenes, the number of manhole covers, and abnormal alarm information;

[0024] Data storage module, which stores all access data, manhole cover status recognition results, abnormal manhole cover information, abnormal manhole cover status recognition model and its parameters;

[0025] The model updating module regularly iterates and upgrades the manhole cover status recognition model according to the data stored in the data storage module, and the upgraded and optimized model parameters are saved in the data storage module.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. The present invention uses the deep learning YOLOV5 algorithm to monitor and identify the status of manhole covers in real time, which can realize all-weather and no-dead-angle manhole cover monitoring, improve the intelligent level of manhole cover management, reduce the workload of manual inspections, solve the problems of low efficiency of manual inspections, blind spots, and difficulty in all-weather coverage, and timely warn of potential road safety hazards.

[0028] 2. The present invention uses existing Skynet videos for detection and identification, without the need to install additional sensors, thus reducing hardware investment costs, making full use of existing resources, and significantly reducing capital investment, enabling more cities to implement intelligent management solutions.

[0029] 3. The present invention combines the real-time processing capability of video streams to ensure that once an abnormality in the manhole cover is discovered, the system can respond quickly, shortening the time difference from problem discovery to problem resolution, and improving emergency response efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of a method for identifying the state of a city manhole cover based on Skynet video according to the present invention;

[0031] Figure 2 Schematic diagram of the city manhole cover status recognition system based on Skynet video of the present invention;

[0032] Figure 3 This is a training flow chart of the YOLOV5 manhole cover status recognition model of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Embodiment 1:

[0035] like Figure 1 , 2 As shown, the urban manhole cover status recognition method based on Skynet video of the present invention receives Skynet video data and performs video image processing. On the one hand, by interpreting the scene position corresponding to the video, when a problem is found, it can directly correspond to the location of the problematic manhole cover, which is convenient for maintenance personnel to handle; on the other hand, since the recognition model of the present invention adopts a neural network, the input image data needs to be pre-processed, including image size scaling, standardization, etc., to ensure that the image data can be directly input into the manhole cover status recognition model. At the same time, it should be noted that when the collected image data is too large and the target manhole cover accounts for a low proportion in the image, it may be necessary to crop the image and identify it in blocks to improve the model recognition accuracy.

[0036] The processed data is input into the manhole cover status recognition model. In this example, the manhole cover status recognition model is a YOLOV5 neural network model. The manhole cover status that it can detect includes but is not limited to normal, lost, damaged, and displaced. Lost, damaged, and displaced are abnormal states of the manhole cover. When the recognition model detects an abnormality, the present invention immediately retrieves the location of the abnormal manhole cover and issues a warning. At the same time, the abnormal warning information is highlighted on the display terminal, and a maintenance order is generated and automatically distributed to the terminal device of the area responsible personnel. The abnormal warning information includes the type and location of the abnormal manhole cover, whether a maintenance order has been issued, whether the maintenance order has been correctly received, whether the abnormal manhole cover has been restored, etc.

[0037] Subsequently, the present invention continues to track, and when the abnormal manhole cover is detected again, the display status is normal. At this time, it means that the maintenance personnel have restored the manhole cover. Therefore, when it is detected again, the manhole cover will be displayed as normal. At this time, the previously abnormal manhole cover information status is updated to "repaired", and the warning message is crossed out from the display terminal as a processed order.

[0038] Correspondingly, the display terminal will centrally display all the manhole cover information, including the area covered by the manhole covers, the number of manhole covers, the status of the manhole covers, abnormal manhole cover information, repaired manhole cover information, etc. All information can be processed and visualized in line graphs, pie charts, etc. according to the data characteristics, so as to achieve clear management of urban manhole covers.

[0039] Correspondingly, the present invention provides an identification system corresponding to the manhole cover status identification method, including: a data receiving module, a data processing module, a model reasoning module, an abnormal alarm module, an abnormal dispatch module, a status display module, a data storage module, and a model update module. All modules cooperate with each other to realize the above-mentioned urban manhole cover status identification method based on Skynet video.

[0040] Embodiment 2:

[0041] like Figure 3 As shown, the training steps of the manhole cover state recognition model of the present invention are as follows:

[0042] 1. Data preparation:

[0043] Data collection: First, we need to collect image datasets for training, validation, and testing. These images should contain the target objects (manhole covers) to be detected. The data comes from various public image datasets, and also from self-collected image data. In traffic scenarios, we use cameras to collect images containing manhole cover targets. The collected data should be representative and cover various scenarios and angles where the target may appear.

[0044] Data annotation: Annotate the collected images and mark the category and location information of each target. The location information is usually expressed in the form of a bounding box, that is, the coordinates of the upper left and lower right corners of the target object in the image. The quality of annotation has a crucial impact on the training effect of the model.

[0045] Data enhancement: In order to increase the diversity of the data set and improve the generalization ability of the model, data enhancement technology is used. This includes random cropping, rotation, scaling, flipping, and adjusting the brightness and contrast of the image. The image is randomly rotated by a certain angle before being used for training, so that the model can accurately detect targets at different angles.

[0046] 2. Model Construction

[0047] Backbone:

[0048] Convolution layer parameters: The parameter settings of the convolution layer have a great impact on the feature extraction effect. Including parameters such as the size of the convolution kernel, step size, padding, etc. For example, a smaller convolution kernel (3x3) can extract more detailed features, while a larger convolution kernel (5x5) can extract more macro features. The present invention uses a convolution kernel (3x3). The step size determines the stride of the convolution kernel sliding on the image, and padding can control the size of the output feature map.

[0049] Network depth: The depth of the backbone network is also critical. A deeper network can extract more abstract and advanced features, but it is also prone to problems such as gradient vanishing or gradient exploding. To solve these problems, some technologies are used in Yolov5, such as residual connections. Residual connections can make it easier for information to be transmitted in the network and prevent the gradient vanishing problem.

[0050] Neck:

[0051] Feature fusion method: There are many ways to fuse the features of the Neck part. For example, top-down and bottom-up fusion methods can be used. Top-down fusion is to upsample the high-level feature map (rich in semantic information but low in resolution) and then fuse it with the low-level feature map (high in resolution but relatively less in semantic information), while bottom-up fusion is the opposite. This two-way fusion method can make full use of the advantages of feature maps at different levels. This time, bottom-up fusion is used.

[0052] Intermediate layer processing: During the feature fusion process, some processing is performed on the intermediate layer, and a convolutional layer is added to adjust the number of channels of the feature map to better match different feature maps for fusion.

[0053] Head:

[0054] Classification and regression: When predicting the target category, the Head part uses the softmax function method to convert the network output into the category probability. For the regression prediction of the target location, in addition to predicting the coordinates of the bounding box, some other related parameters, such as the aspect ratio of the bounding box, may also be predicted to more accurately locate the target.

[0055] Multi-scale detection: Yolov5 uses multiple feature maps for detection, and the Head part needs to be processed accordingly for feature maps of different scales. For large-scale feature maps, a smaller receptive field is used for detection to improve the detection ability of small targets; for small-scale feature maps, a larger receptive field is used for detection to detect large targets.

[0056] 3. Model Training

[0057] Loss function:

[0058] Position loss: Position loss not only considers the prediction error of bounding box coordinates, but also the weights of coordinate predictions at different scales. For position prediction of small objects, a higher weight is given because the bounding box error of small objects has a greater impact on the detection results.

[0059] Category loss: In addition to the traditional cross entropy loss function used to calculate the loss of category prediction, the loss of different categories may be adjusted according to factors such as the rarity of the target. For categories that appear less frequently in the dataset, the weight of their loss is appropriately increased to improve the model's detection ability for these rare categories.

[0060] Confidence loss: Confidence loss is used to measure the difference between the confidence that the predicted bounding box contains the object and the actual situation. The binary cross entropy method is used to calculate the confidence loss.

[0061] Optimizer:

[0062] Learning rate adjustment: During the training process, the adjustment of the learning rate is very important. Using a learning rate decay strategy, the learning rate is gradually reduced as the number of training rounds increases, which can make the model converge more stably in the later stages of training. Common learning rate decay methods include exponential decay and cosine annealing.

[0063] Momentum parameter: The SGD-momentum optimizer is used. The momentum parameter determines the influence of historical gradients on current gradient updates. Appropriate momentum parameters can speed up training and improve the convergence of the model.

[0064] Training process:

[0065] Batch Normalization: During the training process, batch normalization can accelerate the convergence of the model and improve the stability of the model. It normalizes each batch of data to make the input data distribution of the network more stable.

[0066] Early stopping: To prevent the model from overfitting, an early stopping strategy is used. That is, when the validation loss of the model stops decreasing or starts to increase, the training is stopped. This will result in a model with better performance on the validation set.

[0067] Model Evaluation:

[0068] Confusion Matrix: In addition to the aforementioned indicators such as accuracy, recall, and mAP, the confusion matrix shows the model's predictions on different categories in more detail. It can show the number of correct predictions for each category, the number of incorrect predictions for other categories, and other information, which helps to deeply analyze the performance of the model.

[0069] Evaluation on different datasets: In order to more comprehensively evaluate the generalization ability of the model, it is evaluated on multiple different datasets. In addition to evaluating in the fields where the training and validation datasets are located, it is also evaluated on datasets in other related fields to see whether the model can adapt to different data distributions.

[0070] Model deployment:

[0071] Model Quantization: When deploying to resource-constrained devices (mobile devices), the model is quantized. Quantization is the process of converting the model's parameters from high precision (32-bit floating point numbers) to low precision (8-bit integers), which can greatly reduce the model's storage and computation requirements while maintaining certain performance.

[0072] Model Acceleration: Use some model acceleration technologies, such as model pruning. Model pruning refers to removing unimportant connections or neurons in the model, thereby reducing the amount of model calculations and increasing the model's running speed.

[0073] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying the status of urban manhole covers based on Skynet video, characterized in that: Methods include: S101. Receive Skynet video data and perform data preprocessing, analyze the corresponding position of the video image and convert the image data format so that the data can be directly input into the manhole cover status recognition model; S102. Receive the image data processed by S101, use the image recognition neural network model to identify the state of the manhole cover, and obtain the state of the manhole cover corresponding to the current image; S103. When the current image manhole cover state is abnormal, a warning is issued and the location of the abnormal manhole cover is indicated; S104. Summarize all detection data and detection results, and display them visually in real time. The display includes at least: the number of detection scenes, the number of manhole covers, and abnormal alarm information.

2. The method for identifying the status of urban manhole covers based on Skynet video according to claim 1 is characterized in that: The manhole cover status recognition model is a YOLOV5 deep learning model.

3. The method for identifying the state of urban manhole covers based on Skynet video according to claim 2 is characterized in that: When the manhole cover state recognition model needs to be deployed in a resource-constrained scenario, the urban manhole cover state recognition method will quantize and prune the manhole cover state recognition model.

4. The method for identifying the status of urban manhole covers based on Skynet video according to claim 3 is characterized in that: The manhole cover status identified by the manhole cover status identification model includes at least: normal, lost, damaged, and displaced, among which lost, damaged, and displaced are abnormal states of the manhole cover.

5. The method for identifying the status of urban manhole covers based on Skynet video according to claim 4 is characterized in that: When an abnormal state of a manhole cover is detected, the manhole cover state identification method also automatically generates a manhole cover repair order and sends it to a terminal of a person in charge of a preset area.

6. The method for identifying the status of urban manhole covers based on Skynet video according to claim 5 is characterized in that: S104 The abnormal alarm information display includes at least: the abnormal manhole cover location, the abnormal state of the manhole cover, whether the manhole cover abnormal dispatch order has been received, and whether the manhole cover has been repaired.

7. The method for identifying the status of urban manhole covers based on Skynet video according to claim 6 is characterized in that: The manhole cover recognition method also separately records the abnormal state manhole cover information and marks it as "unrepaired". When the maintenance personnel restore the manhole cover, the next time the manhole cover recognition method recognizes that the state of the manhole cover at the current position is normal, the separately recorded abnormal state manhole cover will be re-marked as "repaired", and the warning information will be eliminated in the corresponding visual display.

8. The method for identifying the status of urban manhole covers based on Skynet video according to claim 7 is characterized in that: The manhole cover status recognition method also continuously collects Skynet data and regularly iterates and upgrades the manhole cover status recognition model.

9. A system for identifying the status of a city manhole cover based on Skynet video, the system running the method for identifying the status of a city manhole cover based on Skynet video according to any one of claims 1 to 8, characterized in that: The system includes: Data receiving module: The data receiving module is connected to the "Skynet Project" to obtain real-time monitoring video data; Data processing module: The data processing module analyzes the corresponding positions of the video image and converts the image data format so that the data can be directly input into the manhole cover status recognition model; The model inference module uses a manhole cover state recognition neural network model to recognize the manhole cover state of the received image data. The recognition state includes at least: normal, missing, damaged, and displaced; Abnormal alarm module: when the system identifies that the manhole cover is in abnormal state, the abnormal alarm module will issue a warning; Abnormal dispatching module: when the system identifies that the manhole cover is in abnormal condition, the system automatically generates a repair order and dispatches it to the terminal of the person in charge of the preset area; The status display module receives all detection data and test results in real time and displays them visually in real time. The display includes at least: the number of detection scenes, the number of manhole covers, and abnormal alarm information; Data storage module, which stores all access data, manhole cover status recognition results, abnormal manhole cover information, abnormal manhole cover status recognition model and its parameters; The model updating module regularly iterates and upgrades the manhole cover status recognition model according to the data stored in the data storage module, and the upgraded and optimized model parameters are saved in the data storage module.

Images