Method of determining abnormal conditions of a manhole cover and computing device

Abstract

This invention provides a method and computing device for determining abnormal conditions of manhole covers, relating to the fields of image processing and traffic safety technology. The method includes: pre-setting an initial circular region based on an image from a camera device, wherein the initial circular region is the manhole opening area in the image; acquiring frame images from a real-time video stream from the camera device, which is used for real-time monitoring of road manhole covers; segmenting the frame images to obtain manhole cover areas; superimposing the initial circular region onto the frame images; and determining whether the manhole cover is closed based on the initial circular region and the manhole cover areas. The method of this invention enables real-time monitoring and early warning of abnormal manhole cover conditions, improving the accuracy of existing detection technologies while reducing equipment and labor installation costs.

Description

Technical Field

[0001] This invention relates to the fields of image processing and traffic safety technology, specifically to a method and computing device for judging abnormal conditions of manhole covers. Background Technology

[0002] With the acceleration of urbanization, the construction of urban underground systems is becoming increasingly sophisticated. As a crucial component of these systems, the safety and stability of manhole covers are receiving increasing attention. Manhole covers serve a wide range of functions and types, and their daily maintenance and management are extremely important. Common types of manhole covers include manhole covers, storm drain covers, fire hydrant covers, cable manhole covers, and more. Manhole covers can cover various forms of underground pipes, equipment, and passageways, effectively preventing pedestrians and vehicles from accidentally stepping on or falling into them, thus ensuring their safety.

[0003] However, due to the high frequency of use of manhole covers and the influence of environmental factors, damage and loss of manhole covers occur frequently, posing significant safety hazards to urban traffic and residents' lives. Therefore, the research and implementation of manhole cover monitoring solutions and measures are particularly important.

[0004] Traditional manhole cover monitoring primarily relies on manual inspections. Inspectors check the condition of manhole covers and repair or replace them promptly if problems are found. This method is low-cost and simple to operate, but its disadvantages are also obvious: low monitoring efficiency, inability to achieve real-time monitoring, and susceptibility to human error and missed detections. With technological advancements, intelligent monitoring solutions are gradually being adopted. These solutions use sensors and monitoring equipment to monitor and collect data on the condition of manhole covers in real time. Through data analysis and processing, damage and loss of manhole covers can be detected promptly. This method offers high monitoring efficiency and real-time monitoring and early warning capabilities, but its disadvantages include higher costs and the need for professional technicians for maintenance and management.

[0005] Therefore, a technical solution is needed that can accurately detect abnormalities in manhole covers and save costs. Summary of the Invention

[0006] The present invention aims to provide a method and computing device for judging abnormal conditions of manhole covers, and to solve the safety hazards to pedestrians and vehicles caused by abnormal conditions of manhole covers on the road surface.

[0007] According to one aspect of the present invention, a method for determining abnormal conditions of manhole covers is provided, comprising:

[0008] An initial circular area is pre-set based on the image from the camera device, wherein the initial circular area is the wellhead area in the image;

[0009] Acquire frame images from a real-time video stream from a camera device used for real-time monitoring of road manhole covers;

[0010] The frame image is segmented to obtain the manhole cover region;

[0011] The initialized circular region is superimposed onto the frame image;

[0012] The manhole cover is determined to be closed based on the initial circular area and the manhole cover area.

[0013] According to some embodiments, the method further includes;

[0014] The longest secant line in the frame image that is parallel to the first coordinate axis and divides the initial circular region is determined as the first secant line;

[0015] The longest secant line in the frame image that divides the manhole cover area parallel to the first coordinate axis is determined as the second secant line;

[0016] Calculate the Euclidean distance between the two endpoints of the first secant and the two endpoints of the second secant;

[0017] If the Euclidean distance is greater than a predetermined threshold, the manhole cover is determined to be not closed and an alarm is issued.

[0018] According to some embodiments, determining whether the manhole cover is closed based on the initialized circular area and the manhole cover area includes:

[0019] Calculate the intersection-union ratio (Iou) of the initial circular region and the manhole cover region.

[0020] According to some embodiments, determining whether the manhole cover is closed based on the initialized circular area and the manhole cover area further includes:

[0021] When the intersection-to-union ratio is 1, the manhole cover is determined to be closed;

[0022] The manhole cover area is segmented into internal damaged areas, and the pixel area of ​​the damaged areas is calculated.

[0023] A maintenance alarm is issued when the pixel area of ​​the damaged region exceeds a preset damage threshold.

[0024] According to some embodiments, determining whether the manhole cover is closed based on the initialized circular area and the manhole cover area further includes:

[0025] When the intersection-to-union ratio is 0 and the minimum distance between the initial circular region and the manhole cover region is greater than a preset displacement threshold, it is determined that the manhole cover has undergone large displacement, and a manhole cover displacement alarm is issued.

[0026] According to some embodiments, determining whether the manhole cover is closed based on the initialized circular area and the manhole cover area further includes:

[0027] When the crossover ratio is greater than 0 and less than 1, or when the crossover ratio is 0 and the minimum distance between the initial circular area and the manhole cover area is less than or equal to a preset displacement threshold, it is determined that the manhole cover is not closed, and the manhole cover area is classified and detected.

[0028] According to some example embodiments, the manhole cover area is classified and detected, including:

[0029] The manhole cover area is classified using a pre-trained neural network model;

[0030] Based on the image classification results, corresponding warnings are issued for manhole covers that are slightly displaced, open, have damaged edges, are bulging, or are sunken.

[0031] According to some example embodiments, the method further includes setting an algorithm recognition region for the frame image.

[0032] Determine the bounding rectangle of the initial circular region on the frame image;

[0033] The circumscribed rectangle is expanded according to a predetermined ratio;

[0034] The region corresponding to the expanded outer rectangle is used as the recognition area of ​​the algorithm.

[0035] Fill the image area outside the algorithm recognition area with any color, or fill it with the average, maximum, minimum, mode, or median of all pixels in the area outside the algorithm recognition area.

[0036] According to some example embodiments, the method further includes:

[0037] The focal length, position, and angle of the camera device are fixed, and the downward viewing angle is 60 to 90 degrees.

[0038] According to another aspect of the present invention, a computing device is provided, including a processor and a memory storing a computer program that, when executed by the processor, causes the processor to perform the method as described in any of the preceding claims.

[0039] According to another aspect of the present invention, a non-transitory computer-readable storage medium is provided, having stored thereon computer-readable instructions that, when executed by a processor, cause the processor to perform the method as described in any of the preceding claims.

[0040] According to embodiments of the present invention, images from a camera device are acquired, and by identifying the manhole opening area and the manhole cover area, it is determined whether an abnormality has occurred in the manhole cover. The technical solution of the present invention can employ a monocular camera, and through image acquisition and relationship judgment, it can accurately detect abnormalities in the manhole cover while significantly saving equipment and labor installation costs.

[0041] According to some embodiments, the cross-union ratio (CUI) is used to determine the abnormality of the manhole cover, and then a pre-trained neural network model is used to classify the damage type and issue corresponding warnings. Compared with existing methods such as manual inspection and sensor detection, this solution has higher technical accuracy in detecting and judging abnormalities of manhole covers, and has a classification alarm function, which can comprehensively and specifically detect abnormalities of manhole covers to avoid safety hazards and traffic safety risks caused by their aging and damage.

[0042] According to some implementation methods, manhole covers are further damaged and repair prompts are provided to ensure the elimination of safety hazards and improve reliability. Furthermore, classifying and issuing alarms based on the condition of manhole covers facilitates the selection of subsequent manual maintenance methods, which is beneficial for the maintenance of manhole cover facilities, reduces maintenance costs, and improves maintenance efficiency.

[0043] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the invention. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0045] Figure 1 This illustrates an application scenario of a camera device according to an example embodiment for determining abnormal conditions of manhole covers.

[0046] Figure 2 A flowchart illustrating a method for determining abnormal conditions of manhole covers according to an example embodiment is shown.

[0047] Figure 3 A flowchart illustrating a method for setting an algorithm recognition region according to an example embodiment is shown.

[0048] Figure 4 A flowchart illustrating a method for determining abnormal conditions of manhole covers according to an example embodiment is shown.

[0049] Figure 5 A schematic diagram illustrating the camera calibration process according to an example embodiment is shown.

[0050] Figure 6 A schematic diagram of a scheme for determining abnormal conditions of manhole covers according to an example embodiment is shown.

[0051] Figure 7A block diagram of a computing device according to an example embodiment is shown. Detailed Implementation

[0052] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that the invention will be thorough and complete, and the concept of the exemplary embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0053] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the invention.

[0054] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0055] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0056] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of the present invention. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0057] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0058] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily essential for implementing the present invention, and therefore cannot be used to limit the scope of protection of the present invention.

[0059] With the acceleration of urbanization, manhole covers are being used more and more frequently, becoming one of the important facilities for urban road safety. Because manhole covers are greatly affected by environmental factors, damage and loss occur frequently, posing safety hazards to urban traffic and residents' lives. Therefore, adopting efficient and accurate manhole cover monitoring schemes and measures is particularly important. To this end, this invention proposes a method for judging abnormal conditions of manhole covers. Based on some example embodiments, real-time images of manhole covers collected by a camera device are analyzed and compared to effectively determine whether the manhole cover is closed and whether it is damaged. According to the specific type of abnormality analyzed from the image, corresponding alarms such as repair and maintenance are issued, which helps to achieve real-time and accurate monitoring and timely repair and maintenance alerts.

[0060] Before describing the embodiments of the present invention, some terms or concepts involved in the embodiments of the present invention will be explained.

[0061] Intrinsic parameter calibration: Camera intrinsic parameter calibration refers to determining the camera's internal parameters, such as focal length, principal point coordinates, and distortion parameters.

[0062] Intrinsic parameter matrix: The intrinsic parameter matrix reflects the camera's own properties. Each camera is different, and calibration is required to know these parameters.

[0063] Distortion coefficient: The distortion coefficient refers to the degree of curvature or distortion of lines or corners in an image due to lens characteristics and other factors when a camera captures an image. The distortion coefficient is a parameter primarily describing the degree of lens distortion in an imaging optical system, and is usually represented by a polynomial function.

[0064] Distortion correction: Distortion correction is a mapping that projects distorted pixels onto the corrected pixel locations.

[0065] Loss function: A function that maps the values ​​of a random event or its related random variables to non-negative real numbers to represent the "risk" or "loss" of that random event.

[0066] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention.

[0067] Figure 1 This illustrates an application scenario of a camera device according to an example embodiment for determining abnormal conditions of manhole covers.

[0068] See Figure 1 The camera device is a monocular camera, installed on a fixed structure on one side of the road, such as a utility pole or streetlight. When selecting a fixed location, the monocular camera is positioned so that it faces the manhole cover area of ​​the road section being monitored. To minimize errors, the camera angle is ensured to be a large downward angle, i.e., 60–90 degrees. Each monocular camera captures a specific range of manhole cover areas. The entire road traffic is monitored by multiple monocular cameras, ensuring full coverage of the entire area to be monitored without blind spots.

[0069] According to an embodiment of the present invention, a monocular camera is used to acquire video frame images of the manhole cover area to provide alarm prompts for abnormal situations. Compared with other manhole cover detection methods, this method is simpler in composition and has lower installation and maintenance costs.

[0070] Figure 2 A flowchart illustrating a method for determining abnormal conditions of manhole covers according to an example embodiment is shown.

[0071] According to an example embodiment, road manhole covers are monitored using a camera device.

[0072] See Figure 2 In S201, an initial circular area is preset based on the image from the camera device, wherein the initial circular area is the wellhead area in the image.

[0073] According to the example embodiment, the focal length, position, and angle of the camera device are fixed. A suitable position is selected to arrange the camera device, ensuring an appropriate detection angle. An initial circular area is set based on the initial image, whereby the initial circular area is the manhole area in the image. Parameters for the initial circular area are set through the software platform's algorithm parameter configuration interface. This initial circular area serves as the basis for determining the location of manholes on the road surface and provides a reference for subsequent judgments of manhole cover anomalies.

[0074] In S203, frame images of a real-time video stream from a camera device used for real-time monitoring of road manhole covers are acquired.

[0075] According to the example embodiment, as described above, after the initial circular area is set, the camera device begins monitoring the road manhole cover and acquires real-time images from the camera device.

[0076] In S205, the frame image is segmented to obtain the manhole cover area.

[0077] According to the example embodiment, the real-time image of the road manhole cover is obtained, and the manhole cover in the current frame of the real-time video stream is segmented by a segmentation algorithm to obtain a separate manhole cover region in the current frame, which is convenient for subsequent calculations.

[0078] According to other embodiments, the HRNet method can be used to process the real-time image to segment the manhole cover region of the frame image.

[0079] The HRNET (High-Resolution Net) architecture consists of parallel high-resolution to low-resolution subnetworks, with repeated information exchange (multi-scale fusion) between the multi-resolution subnetworks. The horizontal and vertical directions correspond to the network depth and feature map scale, respectively. Some image segmentation networks are built by concatenating high-resolution subnetworks, with each subnetwork forming a stage consisting of a series of convolutions, and a downsampling layer between adjacent subnetworks to halve the resolution.

[0080] According to other embodiments, the DEEPLABV3+ method can be used to process the real-time image to segment the manhole cover region of the frame image. According to some embodiments, DEEPLABV3+ semantic segmentation can be divided into semantic segmentation and instance segmentation. Its most significant feature is the introduction of dilated convolution, which increases the receptive field without information loss, allowing each convolution output to contain a larger range of information. Feature point extraction spans pixels, which is beneficial for extracting multi-scale information.

[0081] In S207, the initialized circular region is superimposed onto the frame image.

[0082] According to the example embodiment, in the initial state of the algorithm recognition area, the initialized circular area coincides with the manhole opening area of ​​the manhole cover. Superimposing the initialized circular area onto the frame image allows for clear identification of the manhole opening location in subsequent real-time frame images, preventing recognition errors caused by damage to the manhole opening and improving the accuracy of manhole cover recognition.

[0083] In S209, it is determined whether the manhole cover is closed based on the initial circular area and the manhole cover area.

[0084] According to some embodiments, the determination of whether the manhole cover is closed can be based on the Euclidean distance between the longest secant line between the initial circular area and the manhole cover area. The longest secant line in the frame image that divides the initial circular area parallel to the first coordinate axis is determined as the first secant line, and the longest secant line in the frame image that divides the manhole cover area parallel to the first coordinate axis is determined as the second secant line. The Euclidean distance between the two endpoints of the first secant line and the two endpoints of the second secant line is calculated. If the Euclidean distance is greater than a predetermined threshold, it is determined that the manhole cover is not closed and an alarm is issued.

[0085] According to some embodiments, the determination of whether the manhole cover is closed can also be based on the intersection-union ratio (Iou) of the two image regions, namely the initial circular region and the manhole cover region.

[0086] According to some embodiments, the method for calculating the intersection-union ratio (IUGR) between the manhole cover area and the known manhole area, and determining whether the manhole cover has shifted based on the IUGR value, is as follows: when the IUGR is 1, the manhole cover is determined to be closed. The manhole cover area is segmented into internal damaged areas, and the pixel area of ​​the damaged areas is statistically analyzed. When the pixel area of ​​the damaged areas is greater than a preset damage threshold, a maintenance alarm is issued.

[0087] According to some embodiments, the method for calculating the intersection ratio between the manhole cover area and the known manhole area, and determining whether the manhole cover has shifted based on the value of the intersection ratio, further includes: when the intersection ratio is 0 and the minimum distance between the initial circular area and the manhole cover area is greater than a preset displacement threshold, determining that the manhole cover has shifted significantly, and issuing a manhole cover displacement alarm. When the intersection ratio is greater than 0 and less than 1, or when the intersection ratio is 0 and the minimum distance between the initial circular area and the manhole cover area is less than or equal to the preset displacement threshold, determining that the manhole cover is not closed, and classifying and detecting the manhole cover area.

[0088] According to some embodiments, the classification and detection of the manhole cover area can be performed using a pre-trained neural network model. The image classification is summarized into five types: slight displacement of the manhole cover, open manhole cover, damaged edge of the manhole cover, bulging manhole cover, or dented manhole cover. For each of these five situations, a corresponding warning message is issued.

[0089] Figure 3 A flowchart illustrating a method for setting an algorithm recognition region according to an example embodiment is shown.

[0090] See Figure 3 Based on the collected images of manhole covers on the road surface, the algorithm's recognition area is set.

[0091] In S301, the outer rectangle of the initial circular region is determined on the frame image.

[0092] According to some embodiments, the determination of the initial circular region can be achieved by transmitting parameters to the camera device through the software platform algorithm parameter configuration interface.

[0093] In S303, the circumscribed rectangle is expanded by a predetermined ratio.

[0094] According to some embodiments, the image expansion ratio can be directly set by the user, set in the configuration algorithm parameters, and implemented in the process of inputting the algorithm parameter configuration file into the camera device on the software platform.

[0095] In S305, the area corresponding to the expanded circumscribed rectangle is taken as the recognition area of ​​the algorithm.

[0096] According to an example embodiment, image processing is performed in the algorithm recognition area of ​​the image acquired by the camera device.

[0097] In S307, the image area outside the algorithm recognition area is filled with any color or filled with the average, maximum, minimum, mode, or median of all pixels in the area outside the algorithm recognition area.

[0098] According to the example embodiment, within the algorithm recognition area, the algorithm identifies an initial circular area that coincides with the wellhead area in the real-time monitoring image. This initial circular area is superimposed onto the frame image, clearly defining the wellhead location, preventing recognition errors due to wellhead damage, and improving the accuracy of well cover recognition.

[0099] The initial circular area serves as an important reference for subsequent judgment of abnormal conditions of manhole covers.

[0100] According to the embodiment, based on the positional relationship between the initial circular region or the wellhead region and the well cover region, the occurrence of a well cover abnormality can be determined by calculating the Euclidean distance between the longest secants of the corresponding regions.

[0101] Figure 4 A flowchart illustrating a method for determining abnormal conditions of manhole covers according to an example embodiment is shown.

[0102] See Figure 4 The diagram shows the calculation of the intersection-union ratio Iou of the initial circular region and the manhole cover region, and the determination of whether the manhole cover is closed based on the intersection-union ratio Iou.

[0103] In S401, the intersection-union ratio Iou of the initial circular region and the manhole cover region is calculated.

[0104] According to the example embodiment, based on the real-time image captured by the camera device, the manhole cover in the current frame of the real-time video stream is segmented using a segmentation algorithm to obtain a separate manhole cover region in the current frame, and the intersection-union ratio (Iou) of the initialized circular region and the manhole cover region is calculated.

[0105] In S403, it is determined whether Iou is equal to 1.

[0106] If the intersection-union ratio Iou = 1 is true, proceed to S407. Otherwise, jump to S415.

[0107] In S407, the manhole cover is determined to be closed.

[0108] In S409, after determining that the manhole cover is in a closed state, the manhole cover area is divided into damaged areas inside the manhole cover, and the pixel area S of the damaged area is calculated.

[0109] In S411, it is determined whether the damaged area is greater than a preset damage threshold.

[0110] When the damaged area S is greater than the preset damage threshold, a maintenance alarm is issued at the monitoring device in S413.

[0111] If the crossover-union ratio is not equal to 1, further judgment is made in S415.

[0112] In S415, it is determined whether the intersection-union ratio Iou is equal to 0 and whether the distance between the initialized circular area and the manhole cover area is greater than a preset displacement threshold. If the determination is true, then in S417, it is determined that the manhole cover has undergone large displacement. Furthermore, in S419, a manhole cover displacement alarm is issued.

[0113] In S415, it is determined whether the intersection-union ratio Iou is equal to 0 and whether the minimum distance between the initial circular region and the manhole cover region is greater than a preset displacement threshold. If the determination is false, the next determination is performed in S421.

[0114] In S421, it is determined whether the intersection-union ratio Iou is greater than 0 and less than 1, or whether Iou is 0 and the minimum distance between the initial circular area and the manhole cover area is less than or equal to a preset displacement threshold. When the determination is true, in S423, it is determined that the manhole cover is closed.

[0115] After determining the closed state of the manhole cover, in step S425, the image is classified for detection.

[0116] According to some embodiments, the image classification detection is performed by classifying the manhole cover area using a pre-trained neural network model;

[0117] Based on the image classification results, such as minor displacement of the manhole cover, opening of the manhole cover, damage to the edge of the manhole cover, bulging of the manhole cover, or dent of the manhole cover, the corresponding type of warning is issued in S427.

[0118] According to some embodiments, a pre-trained YOLOv7 neural network model can be used for target object detection, but the present invention is not limited thereto.

[0119] YOLO is an object detection algorithm that provides solutions to many real-world computer vision problems. It is easy to understand, and those skilled in the art can also use other methods for image classification and detection, such as R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CNN, and other models.

[0120] According to some embodiments, a pre-trained Vision Transformer model can be used to classify target objects, but the present invention is not limited thereto.

[0121] Vision Transformer (ViT model), given an H×W×C image and a block size P, can divide the image into N P×P×C blocks, where N = H×W / (P×P). After obtaining the blocks, a linear transformation is used to convert them into D-dimensional feature vectors, and then a position encoding vector is added. ViT also adds a classification flag [class] before the sequence.

[0122] The input sequence is fed into ViT, and then the final output features are classified using the [class] flag. ViT mainly consists of MSA (multi-head self-attention) and MLP (two fully connected layers using the GELU activation function), with LayerNorm and residual connections added before MSA and MLP.

[0123] The loss function of the pre-trained neural network model is:

[0124]

[0125] Where y i,j p represents the label of the i-th sample belonging to the j-th class. i,j Let represent the probability that the i-th sample is predicted to be of class j, n be the number of images in each batch, and C be the number of classes.

[0126] Figure 5 A schematic diagram illustrating the camera calibration process according to an example embodiment is shown.

[0127] In S501, a camera device is set up to capture images.

[0128] The camera device is a monocular camera with a fixed focal length, position, and angle, which monitors road manhole covers and collects images in real time.

[0129] Internal parameter calibration is performed on S503 and S505.

[0130] The camera device is calibrated to obtain the intrinsic parameter matrix and distortion coefficients.

[0131] In S507, it is stored in a configuration file.

[0132] The intrinsic parameter calibrations are stored in a configuration file.

[0133] In S509, distortion correction is performed.

[0134] Distortion correction is performed on images acquired by a monocular camera.

[0135] In S511, the image after distortion correction is obtained.

[0136] The image after distortion correction is stored in a configuration file.

[0137] Figure 6 A schematic diagram of a scheme for determining abnormal conditions of manhole covers according to an example embodiment is shown.

[0138] See Figure 6 First, the camera device is fixed in place, with its focal length, position, and angle fixed, and the angle is a large downward angle, that is, a downward angle of 60-90 degrees.

[0139] According to some embodiments, after the camera is installed, it needs to be connected to the corresponding computer terminal device via wired or wireless network. The camera device acquires video stream frames, and the computer terminal is used for image processing and data processing.

[0140] Set the initial parameters of the camera device, initialize the camera device, and identify the circular area in the monitoring area. The initialized circular area is the wellhead area in the frame image.

[0141] The camera device is calibrated to obtain the intrinsic parameter matrix and distortion coefficients, which are then stored in the configuration file.

[0142] The device acquires frame images from a real-time video stream from a camera device used for real-time monitoring of road manhole covers.

[0143] Obtain the intrinsic parameter matrix and distortion coefficients of the camera device, perform distortion correction on the image, and obtain the distortion-corrected image.

[0144] Based on the acquired images of road manhole covers, an algorithm recognition region is defined. A bounding rectangle is determined on the frame image of the initial circular region. The bounding rectangle is expanded by a predetermined ratio. The region corresponding to the expanded bounding rectangle is used as the algorithm recognition region. The image region outside the algorithm recognition region is filled with any color or using the pixel average, maximum, minimum, mode, or median of all pixels in the region outside the algorithm recognition region.

[0145] According to some embodiments, the determination of the initial circular region can be achieved by transmitting parameters to the camera device through the software platform's algorithm parameter configuration interface. The image expansion ratio can be directly set by the user in the algorithm parameter configuration, and is implemented during the process of inputting the algorithm parameter configuration file into the camera device through the software platform.

[0146] According to the example embodiment, the algorithm recognition area is predetermined, and the area can be set automatically or manually.

[0147] According to an example embodiment, image processing is performed in the algorithm recognition area of ​​the image acquired by the camera device.

[0148] The real-time image of the road manhole cover is acquired, and the manhole cover in the current frame of the real-time video stream is segmented using a segmentation algorithm to obtain individual manhole cover regions in the current frame, which facilitates subsequent calculations.

[0149] According to some embodiments, there are multiple options for segmenting the manhole cover region of the frame image. For example, the HRNet method or the DEEPLABV3+ method can be used, and their specific advantages and disadvantages have been detailed above and will not be repeated here.

[0150] By overlaying the initial circular region onto the frame image, the location of the manhole can be clearly identified in subsequent real-time frame images. This prevents identification errors caused by damage to the manhole and improves the accuracy of manhole cover identification.

[0151] The closure of the manhole cover is determined based on the initial circular area and the manhole cover area. Specifically, the determination can be made by using the Euclidean distance between the longest secant line in the initial circular area and the manhole cover area. The longest secant line in the frame image that divides the initial circular area parallel to the first coordinate axis is identified as the first secant line, and the longest secant line in the frame image that divides the manhole cover area parallel to the first coordinate axis is identified as the second secant line. The Euclidean distance between the two endpoints of the first secant line and the two endpoints of the second secant line is calculated. If the Euclidean distance is greater than a preset threshold, the manhole cover is determined to be open, and an alarm is triggered.

[0152] According to some embodiments, the method of determining whether a manhole cover is closed based on the Euclidean distance of the longest secant between the initial circular area and the manhole cover area can be applied to road sections with manual inspections, where workers perform regular maintenance. The camera system monitors sudden situations such as manhole cover displacement or loss, making up for the lack of real-time performance of manual monitoring.

[0153] The initialization of the circular area and the manhole cover area determines whether the manhole cover is closed. The specific method for determining whether the manhole cover is closed can also be:

[0154] Determine and calculate the intersection-to-intersection ratio Iou of the manhole cover area and the manhole opening area, and calculate the intersection-to-intersection ratio Iou of the initial circular area and the manhole cover area based on the intersection-to-intersection ratio.

[0155] When the intersection-union ratio is 1, the manhole cover is determined to be closed. The manhole cover area is segmented into damaged areas, and the pixel area of ​​the damaged areas is counted. When the pixel area of ​​the damaged areas is greater than a preset damage threshold, a maintenance alarm is issued.

[0156] When the intersection-to-union ratio is 0 and the minimum distance between the initial circular region and the manhole cover region is greater than a preset displacement threshold, it is determined that the manhole cover has undergone large displacement, and a manhole cover displacement alarm is issued.

[0157] When the intersection-over-union ratio (IoU) is greater than 0 and less than 1, or when the IoU is 0 and the minimum distance between the initialized circular region and the manhole cover region is less than or equal to a preset displacement threshold, it is determined that the manhole cover is not closed, and the manhole cover region is classified and detected. A pre-trained neural network model is used to classify the manhole cover region. Based on the image classification results—small displacement of the manhole cover, open manhole cover, damaged edge of the manhole cover, bulging manhole cover, or dented manhole cover—corresponding warnings are issued.

[0158] According to some embodiments, the method for judging manhole cover anomalies by cross-comparison can completely replace manual labor. It performs threshold judgments on various conditions such as aging and damage of manhole covers, and then issues early warnings for repair or corresponding situations. The detection and judgment of manhole cover anomalies are more accurate than current technologies.

[0159] According to some embodiments, this invention uses image processing technology to monitor manhole cover anomalies in real time, eliminating the need for additional sensors, signal generators, or other equipment. In sections with well-established monitoring systems, existing cameras can be used directly, with program settings configured on a computer terminal. This fully utilizes existing equipment, significantly reducing the equipment costs for manhole cover monitoring and the labor costs associated with installation.

[0160] Figure 7 A block diagram of a computing device according to an exemplary embodiment of the present invention is shown.

[0161] like Figure 7 As shown, the computing device 30 includes a processor 12 and a memory 14. The computing device 30 may also include a bus 22, a network interface 16, and an I / O interface 18. The processor 12, memory 14, network interface 16, and I / O interface 18 can communicate with each other via the bus 22.

[0162] Processor 12 may include one or more general-purpose CPUs (Central Processing Units), microprocessors, or application-specific integrated circuits, for executing relevant program instructions. According to some embodiments, computing device 30 may also include a high-performance display adapter (GPU) 20 for accelerating processor 12.

[0163] Memory 14 may include a machine-readable medium in the form of volatile memory, such as random access memory (RAM), read-only memory (ROM), and / or cache memory. Memory 14 is used to store one or more programs containing instructions, as well as data. Processor 12 may read the instructions stored in memory 14 to perform the methods described above according to embodiments of the present invention.

[0164] The computing device 30 can also communicate with one or more networks via the network interface 16. The network interface 16 can be a wireless network interface.

[0165] Bus 22 can include address bus, data bus, control bus, etc. Bus 22 provides a path for exchanging information between components.

[0166] It should be noted that, in specific implementations, the computing device 30 may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the device described above may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0167] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), network storage devices, cloud storage devices, or any type of medium or device suitable for storing instructions and / or data.

[0168] This invention also provides a computer program product comprising a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments.

[0169] Those skilled in the art will clearly understand that the technical solutions of the present invention can be implemented by means of software and / or hardware. In this specification, "unit" and "module" refer to software and / or hardware capable of independently performing or cooperating with other components to perform a specific function, wherein the hardware may be, for example, a field-programmable gate array (FPGA), an integrated circuit, etc.

[0170] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0171] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0172] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0173] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0174] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0175] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention.

[0176] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0177] Exemplary embodiments of the present invention have been specifically shown and described above. It should be understood that the present invention is not limited to the detailed structures, arrangements, or implementations described herein; rather, the present invention is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended provisions.

Claims

1. A method for determining abnormal conditions of manhole covers, characterized in that, include: An initial circular area is pre-set based on the image from the camera device, wherein the initial circular area is the wellhead area in the image, and the camera device is a monocular camera installed on a fixed building on one side of the road; The system acquires frame images from a real-time video stream from the camera device used for real-time monitoring of road manhole covers. The frame image is segmented to obtain the manhole cover region; The initialized circular region is superimposed onto the frame image; Determine whether the manhole cover is closed based on the initialized circular area and the manhole cover area: The longest secant line in the frame image that is parallel to the first coordinate axis and divides the initial circular region is determined as the first secant line; The longest secant line in the frame image that divides the manhole cover area parallel to the first coordinate axis is determined as the second secant line; Calculate the Euclidean distance between the two endpoints of the first secant and the two endpoints of the second secant; If the Euclidean distance is greater than a predetermined threshold, the manhole cover is determined to be not closed and an alarm is issued.

2. The method according to claim 1, characterized in that, It also includes setting the algorithm recognition region for the frame image: Determine the bounding rectangle of the initial circular region on the frame image; The circumscribed rectangle is expanded according to a predetermined ratio; The region corresponding to the expanded outer rectangle is taken as the recognition region of the algorithm. Fill the image area outside the algorithm recognition area with any color, or fill it with the average, maximum, minimum, mode, or median of all pixels in the area outside the algorithm recognition area.

3. The method according to claim 1, characterized in that, The camera device has a fixed focal length, position, and angle, and the downward viewing angle is 60 to 90 degrees.

4. A computing device, characterized in that, include: processor; as well as A memory storing a computer program that, when executed by the processor, implements the method as described in any one of claims 1-3.

Images