Video processing method, pipeline defect information display method and corresponding device

Abstract

The embodiment of the present application discloses a video processing method, a method for displaying pipeline defect information, and a corresponding device. The specific technical solution includes: obtaining video data obtained by shooting the inside of the pipeline; performing frame extraction processing on the video data to obtain multiple image frames; using a pipeline defect detection model to identify defects in each image frame to obtain a first recognition result, wherein the pipeline defect detection model is a machine learning model; integrating and / or filtering the defect information in the first recognition result to obtain a second recognition result. The technical solution provided by the present application can accurately and efficiently detect pipeline defects.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video processing method, a method and a device for displaying pipeline defect information. Background Art

[0002] Underground drainage pipe networks are important infrastructure in cities. Every day, tens of thousands of tons of rainwater and sewage flow into water purification plants through the drainage pipe network, and then flow into rivers after treatment. This cycle revitalizes the city. However, since urban drainage pipes are buried deep underground, how to accurately and efficiently detect defects such as damage, leakage, and sedimentation is an important and difficult problem to solve. Summary of the invention

[0003] In view of this, the present application provides a video processing method, a method and a device for displaying pipeline defect information, so as to accurately and efficiently detect pipeline defects.

[0004] This application provides the following solutions:

[0005] According to a first aspect, a video processing method is provided, comprising:

[0006] Obtain video data obtained by shooting the inside of the pipeline;

[0007] Performing frame extraction processing on the video data to obtain multiple image frames;

[0008] Using a pipeline defect detection model to perform defect recognition on each image frame to obtain a first recognition result, wherein the pipeline defect detection model is a machine learning model;

[0009] The defect information in the first recognition result is integrated and / or filtered to obtain a second recognition result.

[0010] According to an achievable manner in an embodiment of the present application, the pipeline defect detection model includes N detection sub-models of different defect types, where N is a positive integer;

[0011] The image frame is identified by using the detection sub-models of the N different defect types to obtain the pipeline defect type contained in the image frame, wherein the defect type includes at least one of sediment, obstacle, scum, scaling, corrosion, material shedding, foreign body penetration, misalignment, disconnection, residual wall, tree root, branch pipe concealed connection, rupture, deformation, undulation and leakage.

[0012] According to an achievable method in an embodiment of the present application, the detection sub-models of sediment, obstacle, scum, scaling, corrosion, residual wall, tree root, branch pipe concealed connection, deformation, undulation, leakage, material shedding, foreign body penetration and rupture are pre-trained based on the target detection model;

[0013] The mismatch and disconnection detection sub-model includes a pre-trained segmentation model and classification model and a preset rule model.

[0014] According to an achievable manner in the embodiment of the present application, the method further includes:

[0015] Using the auxiliary model, at least one of shooting angle recognition, pipeline material recognition, detection well recognition and distance recognition is performed on each image frame, wherein the distance is the distance between the shooting position corresponding to the image frame and the starting shooting position of the video data;

[0016] Using the recognition result of the auxiliary model, the pipeline defect detection model is used to perform defect recognition on each image frame or to integrate and / or filter the defect information in the first recognition result.

[0017] According to an achievable method in an embodiment of the present application, using the recognition result of the auxiliary model, executing the method of using the pipeline defect detection model to perform defect recognition on each image frame includes:

[0018] Acquire shooting angle information of each image frame from the recognition result of the auxiliary model;

[0019] According to the preset correspondence between the shooting angle information and the defect type, the defect type corresponding to the shooting angle information of the image frame is identified for the image frame.

[0020] According to an achievable method in an embodiment of the present application, using the recognition result of the auxiliary model, executing the method of using the pipeline defect detection model to perform defect recognition on each image frame includes:

[0021] Acquire pipeline material information of the image frame from the recognition result of the auxiliary model;

[0022] According to the preset correspondence between the pipeline material information and the model accuracy, a pipeline defect detection model with the accuracy corresponding to the pipeline material information in the recognition result is used to perform defect recognition on the image frame.

[0023] According to an achievable manner in the embodiment of the present application, using the recognition result of the auxiliary model, performing integration and / or filtering processing on the defect information in the first recognition result includes at least one of the following:

[0024] filtering out the specific type of defects identified for the image frames containing the detection wells;

[0025] Integrate multiple defects of the same type that appear within a preset range into one defect;

[0026] If two image frames with the same defect type are separated by M image frames without the defect and the M image frames are all side views, the same type of defects in the two image frames are integrated into one defect, and M is greater than or equal to a preset number threshold.

[0027] According to an achievable manner in the embodiment of the present application, the defect information in the first recognition result is integrated and / or filtered to obtain a second recognition result including:

[0028] Combining defects of the same type that appear in consecutive image frames into one defect; and / or,

[0029] Integrate multiple defects of the same type that appear at intervals in the image frame and whose intervals are less than or equal to a preset first interval threshold into one defect; and / or,

[0030] If the distance between a defect appearing in an image frame and the last and next image frames where the defect appears exceeds a preset second interval threshold, deleting the defect appearing in the image frame; and / or,

[0031] Based on the areas of the identified defects, the defects whose areas do not meet the preset area requirements are filtered out; and / or,

[0032] According to the confidence of the identified defects, the defects with confidence less than or equal to the preset confidence threshold are filtered out.

[0033] According to an achievable manner in the embodiment of the present application, the method further includes:

[0034] An alarm is issued based on the defect information in the second recognition result, and the alarm uses at least one of sound, text and special identification information.

[0035] According to a second aspect, a method for displaying pipeline defect inspection information is provided, comprising:

[0036] Obtain video data obtained by shooting the inside of the pipeline;

[0037] Performing frame extraction processing on the video data to obtain multiple image frames;

[0038] Using a pipeline defect detection model to perform defect recognition on each image frame to obtain a first recognition result, wherein the pipeline defect detection model is a machine learning model;

[0039] Integrating and / or filtering the defect information in the first recognition result to obtain a second recognition result;

[0040] The second recognition result is superimposed on the video data and rendered on a display screen of an augmented reality (AR) device.

[0041] According to a third aspect, a video processing apparatus is provided, comprising:

[0042] A video acquisition unit is configured to acquire video data obtained by shooting the inside of the pipeline;

[0043] A video frame extraction unit is configured to perform frame extraction processing on the video data to obtain multiple image frames;

[0044] A defect recognition unit is configured to perform defect recognition on each image frame using a pipeline defect detection model to obtain a first recognition result;

[0045] The correction processing unit is configured to integrate and / or filter the defect information in the first recognition result to obtain a second recognition result.

[0046] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the first aspect are implemented.

[0047] According to a fifth aspect, an electronic device is provided, characterized in that it includes:

[0048] one or more processors; and

[0049] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the first aspects above.

[0050] According to the specific embodiments provided by the present application, after performing frame extraction processing on the video data, the present application uses a pipeline defect detection model to perform defect recognition on each image frame obtained by frame extraction, and integrates and / or filters the recognition results to obtain a final defect recognition result, thereby realizing automatic recognition of pipeline defects, improving efficiency compared to manual recognition methods, and avoiding the influence of recognition accuracy being dependent on personnel's experience and work attitude. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 An exemplary system architecture to which embodiments of the present application can be applied is shown;

[0053] Figure 2A flow chart of the main method of video processing provided by the embodiment of the present application;

[0054] Figure 3 A schematic diagram of auxiliary identification provided by an embodiment of the present application;

[0055] Figure 4 A schematic diagram of an image frame provided in an embodiment of the present application;

[0056] Figure 5a to Figure 5d Several schematic diagrams of correcting defect information provided in the embodiments of the present application;

[0057] Figure 6 A schematic block diagram showing a video processing apparatus according to an embodiment.

[0058] Figure 7 A schematic diagram of the architecture of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0060] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0061] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0062] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0063] For pipeline defect detection, some companies have already used pipeline robots to enter underground pipelines, use the robot's crawler and camera to capture the pipeline site and store it as a video. Then, the videos collected by the pipeline robot are manually judged to identify pipeline defects. However, this manual identification method is very inefficient, and its accuracy is heavily dependent on the experience and work attitude of the personnel.

[0064] The method provided in this application is based on artificial intelligence technology to replace manual judgment.

[0065] In order to facilitate the understanding of the technical solution provided by this application, the system architecture based on which the embodiment of this application is based is briefly described first. Figure 1 As shown in , the system mainly includes a pipeline robot 102 , a control device 104 and an intelligent recognition device 106 .

[0066] The pipeline robot 102 is composed of at least a crawler and a camera, and can be a wireless device or a wired device. If it is a wireless device, there is a wireless transceiver module, which receives the control instructions of the control device 104 through the wireless transceiver module to move inside the pipeline, thereby capturing video data inside the pipeline; and reports status information, such as operating status, operating position, etc., to the control device 104 through the wireless transceiver module. If the pipeline robot 102 is a wired device, there is a cable, which receives the control instructions of the control device 104 through the cable to move inside the pipeline, thereby capturing video data inside the pipeline; and reports status information, such as operating status, operating position, etc., to the control device 104 through the cable.

[0067] The video data captured by the pipeline robot 102 can be provided to the intelligent recognition device 106 for defect recognition. The video data captured by the pipeline robot 102 can be stored in a storage device, and then the video data in the storage device can be provided to the intelligent recognition device 106 by copying or reading. It is also possible to provide the video data to the intelligent recognition device 106 through a communication connection by establishing a communication connection between the pipeline robot 102 and the intelligent recognition device 106. Alternatively, the pipeline robot 102 can upload the video data to a server, and then the server sends the video data to the intelligent recognition device 106. Other methods can also be used, which are not exhaustive here.

[0068] The intelligent identification device 106 may be a computer terminal with strong computing power, or a server-side device, a single server, a server cluster, a cloud computing server, etc.

[0069] It should be understood that Figure 1The number of pipeline robots 102, control devices 104 and intelligent recognition devices 106 in the embodiment is only for illustration. Any number of pipeline robots 102, control devices 104 and intelligent recognition devices 106 may be provided according to implementation requirements.

[0070] Figure 2 This is a flow chart of the main method of video processing provided by the embodiment of the present application. The method can be performed by Figure 1 The smart recognition device 106 in the system architecture shown is executed. Figure 2 As shown in , the method mainly includes the following steps:

[0071] Step 202: Obtain video data obtained by shooting the interior of the pipeline.

[0072] Step 204: Perform frame extraction processing on the acquired video data to obtain multiple image frames.

[0073] Step 206: Use the pipeline defect detection model to perform defect recognition on each image frame to obtain a first recognition result.

[0074] Step 208: Integrate and / or filter the defect information in the first recognition result to obtain a second recognition result.

[0075] It can be seen that after the video data is extracted, the present application uses the pipeline defect detection model to perform defect recognition on each image frame obtained by the extraction, and integrates and / or filters the recognition results to obtain the final defect recognition result, thereby realizing automatic recognition of pipeline defects, improving efficiency compared to manual recognition methods, and avoiding the influence of recognition accuracy on personnel's experience and work attitude.

[0076] The above steps are described in detail below in conjunction with the embodiments. First, the above step 204, namely "performing frame extraction processing on the acquired video data to obtain multiple image frames", is described in detail in conjunction with the embodiments.

[0077] In this step, after acquiring the video data captured by the pipeline robot, the intelligent recognition device can first identify the video data and intercept the video data captured inside the target pipeline. The so-called target pipeline refers to the pipeline part to be detected. The pipeline part to be detected can be one or more pipelines, or a part of a pipeline, which can be selected according to actual needs.

[0078] The purpose of frame extraction is to decode the video data captured by the pipeline robot into images. When performing frame extraction, the frame extraction frequency can be set according to actual needs. For example, in order to ensure the recall rate of defects, a frame extraction frequency of 1 frame / second can be used. If computing resources allow, a frame extraction frequency as high as possible can be used, such as 5 frames / second.

[0079] After the video data is processed by frame extraction, a series of continuous image frames are obtained. These image frames can be regarded as images of the inside of the pipeline taken by the pipeline robot at multiple consecutive time points. Since the pipeline robot is moving, these image frames reflect the actual conditions of each position inside the pipeline.

[0080] As a preferred implementation, before step 206, an auxiliary model may be further used to assist in identifying each image frame.

[0081] like Figure 3 As shown in , the auxiliary identification may include at least one of the following: shooting angle identification, pipeline material identification, detection well identification, and distance identification. The result of the auxiliary identification may be used to assist defect identification in the subsequent step 206, or to assist in the integration and / or filtering of defect information in the subsequent step 208. The above-mentioned auxiliary identifications are described in detail below.

[0082] Shooting perspective recognition refers to identifying the shooting perspective of an image frame, that is, the shooting perspective used by the pipeline robot to shoot the inside of the pipeline to obtain the image frame. The shooting perspective may include front view of the pipeline, side view of the pipeline, non-pipeline view, etc.

[0083] The so-called front view of the pipeline refers to the angle of view taken along the pipeline axis, and the side view of the pipeline refers to the angle of view taken along the pipeline radial direction. It should be noted that the front view of the pipeline and the test pipeline are not necessarily strictly along the axial direction and radial direction, and angle deviations within a certain range are allowed.

[0084] The above-mentioned recognition of the shooting angle of the image frame can be implemented by using a pre-trained perspective recognition model. The perspective recognition model can recognize the shooting angle of the image frame and output the shooting angle information when the image frame is input. Among them, the perspective recognition model can be implemented based on the classification model, for example, the first training data is obtained in advance, and the first training data includes some image samples and labels for annotating the shooting angle information of the image samples. The image sample is used as the input of the classification model, and the classification model extracts features from the image sample to form a feature vector, and then maps the feature vector to the classification result of the shooting angle. In the process of training the classification model, the shooting angle label of the image sample is used as the target output, and the classification model is trained, that is, the difference between the output of the classification model and the label in the sample is minimized. A loss function can be constructed, and the value of the loss function is used in each round of iteration to update the parameters of the classification model by a method such as gradient descent until the preset training end condition is met. The training end condition may include, for example, the value of the loss function is less than or equal to the preset loss function threshold, the number of iterations reaches the preset number threshold, etc.

[0085] The recognition result of the shooting angle recognition can assist in defect recognition in the subsequent step 206 on the one hand, and can also assist in filtering defect information in the subsequent step 208 on the other hand, which will be described in detail in the subsequent embodiments.

[0086] Pipe material recognition refers to identifying the material of the pipe in the image frame. The pipe material may include cement pipe, corrugated pipe, etc.

[0087] The above pipeline material recognition of the image frame can be implemented by using a pipeline material recognition model obtained by pre-training. The pipeline material recognition model can recognize the pipeline material of the image frame and output the pipeline material information when the image frame is input. Among them, the pipeline material recognition model can be implemented based on the classification model, for example, the second training data is obtained in advance, and the second training data includes some image samples and labels for marking the pipeline material information of the image sample. The image sample is used as the input of the classification model, and the classification model extracts features from the image sample to form a feature vector, and then maps the feature vector to the classification result of the pipeline material. In the process of training the classification model, the pipeline material label of the image sample is used as the target output, and the classification model is trained, that is, the difference between the output of the classification model and the label in the sample is minimized. A loss function can be constructed, and the value of the loss function is used in each round of iteration to update the parameters of the classification model by a method such as gradient descent until the preset training end condition is met. The training end condition may include, for example, the value of the loss function is less than or equal to the preset loss function threshold, the number of iterations reaches the preset number threshold, etc.

[0088] The identification result of the pipeline material identification can assist in defect identification in the subsequent step 206, which will be described in detail in the subsequent embodiments.

[0089] Inspection well identification refers to identifying whether the content captured by the image frame contains an inspection well. Inspection wells are set up for the maintenance of urban underground infrastructure such as power supply, water supply, drainage, sewage, communication, cable TV, gas pipes, street light lines, etc., and are easy to install. They are generally set up at pipeline intersections, turns, places where pipe diameters or slopes change, and at regular intervals on straight pipe sections to facilitate regular inspection of auxiliary structures.

[0090] The detection well identification of the image frame can be implemented by using a detection well identification model obtained by pre-training. The detection well identification model can identify the detection well of the image frame when the image frame is input to output the identification result of whether the detection well is included. Among them, the detection well identification model can be implemented based on the classification model, for example, the third training data is obtained in advance, and the third training data includes some image samples and labels for marking whether the image sample contains the detection well. The image sample is used as the input of the classification model, and the classification model extracts features from the image sample to form a feature vector, and then maps the feature vector to the classification result of whether the detection well is included. In the process of training the classification model, the label of whether the image sample contains the detection well is used as the target output, and the classification model is trained, that is, the difference between the output of the classification model and the label in the sample is minimized. A loss function can be constructed, and the value of the loss function is used in each round of iteration to update the parameters of the classification model by a method such as gradient descent until the preset training end condition is met. The training end condition can include, for example, the value of the loss function is less than or equal to the preset loss function threshold, the number of iterations reaches the preset number threshold, etc.

[0091] The identification result of the detection well identification is mainly used to assist in the filtering process of the defect information in the subsequent step 208, which will be described in detail in the subsequent embodiments.

[0092] Distance recognition refers to the recognition of the distance between the shooting position corresponding to the image frame and the starting shooting position of the image data. Usually, when the pipeline robot is moving and shooting in the pipeline, it will start from the starting shooting position and mark the distance information on the image frame by marking, forming a distance stamp on the image frame. Figure 4 In the image frame shown in , there is a distance stamp "Distance: 27.25m". This distance stamp needs to be identified to obtain distance information.

[0093] As one of the feasible ways, optical character recognition (OCR) may be performed on the image frame to obtain the distance stamp information.

[0094] However, in addition to the distance stamp, the image frame may also contain other information stamps, such as Figure 4 As shown in , it also includes information such as timestamp, location stamp, pipeline number, etc. If OCR is used, it is difficult to distinguish these information stamps and effectively extract distance information. Therefore, as a preferred implementation, the detection model combined with the multi-task classification model can be used in the embodiment of the present application to replace the OCR method to achieve distance recognition.

[0095] For example, the detection model can be used to detect the minimum enclosing rectangular frame that contains "distance", "number" and "m" from the image frame, so as to locate the distance stamp area; then the characters in the distance stamp area are multi-classified (classified to obtain specific numbers and decimal points) to identify the digital content (including the decimal point) and obtain specific time information.

[0096] The recognition result of distance recognition can be used in three aspects. On the one hand, after the defect is detected later, the defect can be associated with the specific distance information to achieve defect location; on the other hand, it can assist in the integration of defect information in step 208, which will be described in detail in subsequent embodiments; on the third hand, it can provide auxiliary information for subsequent service levels. For example, pipeline defect detection services can be charged according to the detection distance, so the detection distance provides a basis for fee calculation.

[0097] The above step 206, namely "using the pipeline defect detection model to perform defect recognition on each image frame to obtain a first recognition result", is described in detail below in conjunction with an embodiment.

[0098] In this embodiment, the pipeline defect detection model is implemented based on a machine learning model. As one of the achievable methods, the pipeline defect detection model may include N detection sub-models of different defect types, which are respectively used to detect N types of pipeline defects, where N is a positive integer. That is, the image frame is identified using the N detection sub-models of different defect types to obtain the pipeline defect type contained in the image frame.

[0099] Among them, the above-mentioned defect types may include at least one of sediment, obstacles, scum, scaling, corrosion, material shedding, foreign body penetration, misalignment, disconnection, residual wall, tree roots, branch pipe concealed connection, rupture, deformation, undulation, leakage, material shedding, foreign body penetration and rupture.

[0100] The above defect types can be trained separately as detection sub-models. During the training process, training data are pre-built. The training data includes image samples and labels for marking whether the image samples contain specific types of defects. Taking sediment as an example, the image samples are labeled to indicate whether they contain sediment.

[0101] As one of the feasible ways, the detection sub-models of sediment, obstacle, scum, scaling, corrosion, residual wall, tree root, branch pipe blind connection, deformation, undulation and leakage are pre-trained based on the target detection model. The target detection model can adopt common CNN (Convolutional Neural Network) target detection models such as YOLO (You Only Look Once) and FCOS (Fully Convolutional One-Stage Object Detection).

[0102] Among them, sediments are formed by the long-term accumulation of fine-grained solids (such as mud and sand). When the amount of sedimentation is large, it will reduce the water flow area. Scum refers to floating objects on the water surface in the pipeline. Obstacles refer to obstacles in the pipeline that affect the flow. Scaling refers to attachments on the inner wall of the pipeline. Corrosion is the erosion of the inner wall of the pipeline, which causes loss or peeling, resulting in rough surfaces or exposed steel bars. Deformation is the shape variation of the pipeline caused by external force. Fluctuation refers to the offset of the interface position and the change of the vertical position of the pipeline. The concealed connection of the branch pipe is the direct side connection of the branch pipe to the main pipe without passing through the inspection well. Leakage refers to the flow of water outside the pipe into the pipeline. Material shedding refers to the entry of similar interface materials such as rubber rings, asphalt, cement, etc. into the pipeline. Foreign body penetration is the penetration of objects that are not ancillary facilities of the pipeline system into the pipe wall.

[0103] When training each detection sub-model, the image sample is used as the input of the target detection model, and the target detection model outputs the detection result of whether it contains the corresponding type of defects. In the process of training the target detection model, the label of the image sample is used as the target output, and the target detection model is trained, that is, the difference between the output of the target detection model and the label in the sample is minimized. A loss function can be constructed, and the value of the loss function is used in each round of iteration to update the parameters of the classification model by methods such as gradient descent until the preset training end conditions are met. The training end conditions may include, for example, the value of the loss function is less than or equal to the preset loss function threshold, the number of iterations reaches the preset number threshold, etc.

[0104] The mismatch and disconnection detection sub-model includes a pre-trained segmentation model and classification model as well as a preset rule model.

[0105] Misalignment refers to the lateral deviation of the two pipe ends of the same interface, which are not in the correct position of the pipeline. Disconnection refers to the inadequate connection of the ends of the two pipes or the detachment of the interface. Misalignment and disconnection occur at the joints between pipes. If the gap is too large or unreasonable, it becomes a defect, and it is not easy to distinguish whether it is a normal joint or a defect. Therefore, we can first perform regional segmentation to divide the image frame into various regions; then use the classification model to classify each region to determine the joint area between pipes; and then use the rule model to identify the joint area to determine whether misalignment or disconnection occurs. For example, for the rule model, the envelope width ratio of the joint area, the maximum inscribed circle diameter and other indicators can be compared with the preset threshold. If the preset threshold is exceeded, it is considered that defects such as misalignment or disconnection have occurred.

[0106] The segmentation model may adopt common CNN segmentation models such as FCN (Fully Convolutional Networks), and the classification model may adopt common CNN classification models such as ResNet (Residual Network) and ResNeXt.

[0107] The detection sub-models corresponding to each defect type can be designed and trained separately. This solution has the advantages of high reliability and easy maintenance. Joint training can also be performed for a small number of similar defects. For example, a joint detection sub-model can be trained for sediment, obstacles and scum to identify these three defects. For another example, a joint detection sub-model can be trained for scaling and corrosion to identify these two defects.

[0108] Furthermore, since different defect types are affected to different degrees by different shooting angles during identification, the shooting angle information obtained by the auxiliary model for each image frame can be obtained, and the image frame can be identified for the defect type corresponding to the shooting angle information of the image frame according to the preset correspondence between the shooting angle information and the defect type. For example, all types of defects can be identified for the image frame of the front view of the pipeline, and the image frame of the side view of the pipeline can be identified for the defect types such as corrosion, material shedding, branch pipe concealed connection, and rupture.

[0109] Furthermore, after the auxiliary model identifies the pipeline material information for each image frame, the image frames obtained by taking different pipeline materials will affect the detection accuracy of the defect detection model. Therefore, the pipeline material information of the image frame can be obtained from the recognition result of the auxiliary model; then, according to the correspondence between the preset pipeline material information and the model accuracy, the pipeline defect detection model with the accuracy corresponding to the pipeline material information in the recognition result is used to perform defect recognition on the image frame. This method can make the recognition of pipeline defects more accurate.

[0110] After the above identification, the defect type information contained in each image frame can be obtained respectively, and the corresponding distance information can be further included. If the image frame does not contain any defects, there may be no output, or information of no defects may be output.

[0111] The above step 208, namely "integrating and / or filtering the defect information in the first recognition result to obtain the second recognition result", is described in detail below in conjunction with an embodiment.

[0112] This step can be regarded as a process of correcting the defect information in the first recognition result to avoid the same defect being reported multiple times or the defect being misjudged, etc. It can include but is not limited to one or any combination of the following methods:

[0113] The first method is to combine the same type of defects that appear in consecutive image frames into one defect.

[0114] For example Figure 5a As shown in , if defect 1 (marked as 1 in the figure) is identified in 5 consecutive image frames, it can be considered as one defect and merged into one defect 1, and only one alarm is required subsequently. Figure 5a Each strip rectangle in the figure represents an image frame, and the shaded ones indicate that defects are recognized, while the unshaded strip rectangles indicate that defects are not recognized.

[0115] The second method is to integrate multiple defects of the same type that appear at intervals in the image frame and whose intervals are less than or equal to a preset first interval threshold into one defect.

[0116] For example Figure 5b As shown in , if defect 1 appears at intervals in the image frame, but the interval is small, for example, less than 4 image frames, it can be considered as one defect and merged into one defect 1, and only one alarm is required subsequently.

[0117] The third method: if the distance between a defect appearing in an image frame and the last and next image frames in which the defect appears exceeds a preset second interval threshold, the defect appearing in the image frame is deleted.

[0118] For example Figure 5c As shown in , if defect 1 appears in a certain image frame, but defect 1 has not been detected in more than 4 image frames before and defect 1 has not been detected in more than 4 image frames after, then defect 1 is likely to be misidentified and can be deleted.

[0119] The fourth method: based on the area of ​​the identified defects, the defects whose area does not meet the preset area requirements are filtered out.

[0120] In the past, target detection was mostly used in the defect detection process. In the target detection process, the defect area is usually located, which reflects the area of ​​the defect to a certain extent. If the area of ​​the defect is less than or equal to the preset area threshold, it can be considered that the defect does not affect the normal operation of the pipeline and can be filtered out. Different defect types can correspond to different area thresholds.

[0121] The fifth method: based on the confidence of the identified defects, the defects with a confidence less than or equal to a preset confidence threshold are filtered out.

[0122] During the defect detection process, the recognition results output by each detection sub-model will include the confidence of the defect. In this method, the confidence can be used for secondary filtering. That is, the confidence threshold can be set based on experience. If the confidence of the identified defect is less than or equal to the confidence threshold, the defect can be filtered out.

[0123] In addition, as mentioned in the above embodiment, part of the recognition results of the auxiliary model can also be used to integrate and / or filter the defect information in the first recognition result. Corresponding to the following methods:

[0124] A sixth method is to filter out the identified specific types of defects for the image frames containing the detection wells.

[0125] If the image frame contains a detection well, it is easy to misidentify some types of defects, such as obstacles, sediments, misalignment, etc. Therefore, if the image frame contains a detection well and the above-mentioned specific type of defect is also identified, the defect in the image frame is filtered out. The above-mentioned specific type of defect can be preset, which can be set to some defect types or all defect types, that is, as long as the image frame contains a detection well, the defects identified in the image frame are filtered out.

[0126] The seventh method: multiple defects of the same type that appear within a preset range of corresponding distances are integrated into one defect.

[0127] The idea of ​​this method is similar to the first method, except that it is no longer based on the number of consecutive image frames, but on the actual distance, which can be understood as the actual position of the defect. That is, if the actual positions of multiple detected defects of the same type are very close, they can be integrated into one defect.

[0128] The eighth method: If there are M image frames without the defect between two image frames with the same defect type and the M image frames are all side views, the same type of defects in the two image frames are integrated into one defect, and M is greater than or equal to a preset number threshold.

[0129] like Figure 5dAs shown in , if there are image frames with the same defect type, although the number of image frames spaced apart exceeds 4, since the 4 image frames spaced apart are all side-view pipes, the image frames of the 4 side-view pipes can be deleted, and it is considered that these defects of the same type are integrated into one defect.

[0130] Furthermore, in the above integration process, the image features of the image frames to be integrated can be extracted, and then based on the feature distance, it can be determined whether the defects in the image frames are of the same type. If so, integration will continue; otherwise, integration will not be performed. This method can further improve the accuracy of defect recognition.

[0131] After obtaining the second recognition result, an alarm may be issued based on the defect information in the second recognition result, wherein the alarm may be issued in the form of, but not limited to, sound, text, special identification information, and the like.

[0132] As one of the more typical application scenarios, after adopting the above-mentioned video processing method embodiment, the above-mentioned second recognition result can be further superimposed with the video data obtained by shooting the inside of the pipeline, and rendered to the display screen of the AR (Augmented Reality) device.

[0133] It can be learned from the above-mentioned video processing method embodiment that the second recognition result includes defect information and distance information corresponding to the defect information, and the video data obtained by shooting the inside of the pipeline also corresponds to distance information. Therefore, as one of the achievable methods, when superimposing the second recognition result with the video data shot inside the pipeline, the distance information corresponding to the defect information and the distance information corresponding to the video data can be matched, and the superimposition processing is performed according to the matching result. That is, the second recognition result corresponding to the same distance information is superimposed with the video data shot inside the pipeline.

[0134] The superimposed data is rendered on the display screen of the AR device, and the industrial workers using the AR device can see the pipeline defects in an immersive way, so as to adopt some targeted repair measures. In addition to AR devices, display devices such as VR (Virtual Reality) devices can also be used.

[0135] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0136] According to an embodiment of another aspect, a video processing apparatus is provided. Figure 6 A schematic block diagram of a video processing device according to an embodiment is shown, wherein the device is arranged at Figure 1 The smart identification device in the architecture shown. Figure 6 As shown, the device 600 includes: a video acquisition unit 601, a video frame extraction unit 602, a defect recognition unit 603 and a correction processing unit 604, and may further include: an auxiliary recognition unit 605 and an alarm unit 606. The main functions of each component unit are as follows:

[0137] The video acquisition unit 601 is configured to acquire video data obtained by shooting the inside of the pipeline.

[0138] The video frame extraction unit 602 is configured to perform frame extraction processing on the video data to obtain multiple image frames.

[0139] The defect recognition unit 603 is configured to perform defect recognition on each image frame using the pipeline defect detection model to obtain a first recognition result.

[0140] The correction processing unit 604 is configured to integrate and / or filter the defect information in the first recognition result to obtain a second recognition result.

[0141] The pipeline defect detection model may include N detection sub-models of different defect types, where N is a positive integer.

[0142] The defect recognition unit 603 can use N detection sub-models of different defect types to identify the image frame to obtain the pipeline defect type contained in the image frame, and the defect type includes at least one of sediment, obstacle, scum, scaling, corrosion, material shedding, foreign body penetration, misalignment, disconnection, residual wall, tree root, branch pipe concealed connection, rupture, deformation, undulation and leakage.

[0143] As one of the feasible ways, detection sub-models for sediment, obstacle, scum, scaling, corrosion, residual wall, tree root, branch pipe concealed connection, deformation, undulation, leakage, material falling, foreign body penetration and rupture are pre-trained based on the target detection model.

[0144] The mismatch and disconnection detection sub-model includes a pre-trained segmentation model and classification model as well as a preset rule model.

[0145] The auxiliary identification unit 605 is configured to use the auxiliary model to perform at least one of shooting angle identification, pipeline material identification, detection well identification and distance identification on each image frame, where the distance is the distance between the shooting position corresponding to the image frame and the starting shooting position of the video data.

[0146] The defect recognition unit 603 may perform defect recognition on each image frame using a pipeline defect detection model based on the recognition result of the auxiliary recognition unit 605 .

[0147] As one of the achievable methods, the defect recognition unit 603 can obtain the shooting angle information of each image frame from the recognition result of the auxiliary model, and recognize the defect type corresponding to the shooting angle information of the image frame according to the preset correspondence between the shooting angle information and the defect type.

[0148] As another achievable method, the defect recognition unit 603 can obtain the pipeline material information of the image frame from the recognition result of the auxiliary model; based on the preset correspondence between the pipeline material information and the model accuracy, a pipeline defect detection model with an accuracy corresponding to the pipeline material information in the recognition result is used to perform defect recognition on the image frame.

[0149] The correction processing unit 604 may integrate and / or filter the defect information in the first recognition result based on the recognition result of the auxiliary recognition unit 605 .

[0150] As one possible implementation, the correction processing unit 604 may filter out the identified specific type of defects for the image frame containing the detection well; and / or,

[0151] Combining multiple defects of the same type that appear within a preset range of corresponding distances into one defect; and / or,

[0152] If two image frames with the same defect type are separated by M image frames without the defect and the M image frames are all side views, the same type of defects in the two image frames are integrated into one defect, and M is greater than or equal to the preset number threshold.

[0153] In addition, the correction processing unit 604 may be specifically configured to: integrate the same type of defects appearing in consecutive image frames into one defect; and / or,

[0154] Integrate multiple defects of the same type that appear at intervals in the image frame and whose intervals are less than or equal to a preset first interval threshold into one defect; and / or,

[0155] If the distance between a defect appearing in an image frame and the last and next image frames where the defect appears exceeds a preset second interval threshold, then deleting the defect appearing in the image frame; and / or,

[0156] Based on the areas of the identified defects, the defects whose areas do not meet the preset area requirements are filtered out; and / or,

[0157] According to the confidence of the identified defects, the defects with confidence less than or equal to the preset confidence threshold are filtered out.

[0158] The alarm unit 606 is configured to issue an alarm based on the defect information in the second recognition result, and the alarm adopts at least one of sound, text and special identification information.

[0159] Furthermore, the above device may further include a rendering unit (not shown in the figure) for superimposing the second recognition result with the video data obtained by shooting the inside of the pipeline, and rendering it to a display screen of an AR (Augmented Reality) device.

[0160] It can be seen from the above method and device embodiments that the technical solution provided by the embodiments of the present application can have the following advantages:

[0161] 1) After extracting frames from the video data, the present application uses a pipeline defect detection model to perform defect recognition on each image frame obtained by the extraction, and integrates and / or filters the recognition results to obtain the final defect recognition result, thereby realizing automatic recognition of pipeline defects. Compared with the manual recognition method, the efficiency is improved and the recognition accuracy is avoided from being affected by the experience and work attitude of the personnel.

[0162] 2) Design and train detection sub-models for each defect type separately. Updating the model of a certain type of defect or adding a new type of defect will not affect the detection of other types of defects. It has strong scalability, high reliability and easy maintainability.

[0163] 3) By identifying auxiliary features such as shooting angle, pipeline material, detection well, distance, etc., it can assist in defect identification and / or defect correction, thereby further improving the accuracy of defect detection.

[0164] 4) By integrating and / or filtering the defect information obtained by the defect detection model, the over-reporting of repeated defects and the false reporting of erroneous defects are reduced, making the final defect detection result more accurate.

[0165] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time.

[0166] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0167] And an electronic device, comprising:

[0168] one or more processors; and

[0169] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0170] in, Figure 7 The architecture of the electronic device is shown as an example, which may include a processor 710, a video display adapter 711, a disk drive 712, an input / output interface 713, a network interface 714, and a memory 720. The processor 710, the video display adapter 711, the disk drive 712, the input / output interface 713, the network interface 714, and the memory 720 may be communicatively connected via a communication bus 730.

[0171] Among them, the processor 710 can be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.

[0172] The memory 720 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 720 can store an operating system 721 for controlling the operation of the electronic device 700, and a basic input and output system (BIOS) 722 for controlling the low-level operation of the electronic device 700. In addition, a web browser 723, a data storage management system 724, and a device for detecting pipeline defects 725, etc. can also be stored. The above-mentioned device for detecting pipeline defects 725 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 720 and is called and executed by the processor 710.

[0173] The input / output interface 713 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0174] The network interface 714 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0175] The bus 730 comprises a pathway for transmitting information between the various components of the device (eg, the processor 710, the video display adapter 711, the disk drive 712, the input / output interface 713, the network interface 714, and the memory 720).

[0176] It should be noted that, although the above device only shows a processor 710, a video display adapter 711, a disk drive 712, an input / output interface 713, a network interface 714, a memory 720, a bus 730, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0177] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0178] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0179] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A video processing method, comprising: Obtain video data obtained by shooting the inside of the pipeline; Performing frame extraction processing on the video data to obtain multiple image frames; Using a pipeline defect detection model to perform defect recognition on each image frame to obtain a first recognition result, wherein the pipeline defect detection model is a machine learning model; Integrating and / or filtering the defect information in the first recognition result to obtain a second recognition result; The method also includes: using the auxiliary model to perform at least one of shooting angle recognition, pipeline material recognition, detection well recognition and distance recognition on each image frame, wherein the distance is the distance between the shooting position corresponding to the image frame and the starting shooting position of the video data; using the recognition result of the auxiliary model, executing the defect recognition of each image frame using the pipeline defect detection model or integrating and / or filtering the defect information in the first recognition result.

2. The method according to claim 1, wherein: The pipeline defect detection model includes N detection sub-models of different defect types, where N is a positive integer; The image frame is identified by using the detection sub-models of the N different defect types to obtain the pipeline defect type contained in the image frame, wherein the defect type includes at least one of sediment, obstacle, scum, scaling, corrosion, material shedding, foreign body penetration, misalignment, disconnection, residual wall, tree root, branch pipe concealed connection, rupture, deformation, undulation and leakage.

3. The method according to claim 2, wherein: The detection sub-models for sediment, obstacle, scum, scaling, corrosion, residual wall, tree root, branch pipe blind connection, deformation, undulation, leakage, material shedding, foreign body penetration and rupture are pre-trained based on the target detection model; The mismatch and disconnection detection sub-model includes a pre-trained segmentation model and classification model and a preset rule model.

4. The method according to claim 1, wherein: Using the recognition result of the auxiliary model, executing the defect recognition of each image frame using the pipeline defect detection model includes: Acquire shooting angle information of each image frame from the recognition result of the auxiliary model; According to the preset correspondence between the shooting angle information and the defect type, the defect type corresponding to the shooting angle information of the image frame is identified for the image frame.

5. The method according to claim 1, wherein: Using the recognition result of the auxiliary model, executing the defect recognition of each image frame using the pipeline defect detection model includes: Acquire pipeline material information of the image frame from the recognition result of the auxiliary model; According to the preset correspondence between the pipeline material information and the model accuracy, a pipeline defect detection model with the accuracy corresponding to the pipeline material information in the recognition result is used to perform defect recognition on the image frame.

6. The method according to claim 1, wherein: Using the recognition result of the auxiliary model, integrating and / or filtering the defect information in the first recognition result includes at least one of the following: filtering out the identified defects for the image frames containing the detection wells; Integrate multiple defects of the same type that appear within a preset range into one defect; If two image frames with the same defect type are separated by M image frames without the defect and the M image frames are all side views, the same type of defects in the two image frames are integrated into one defect, and M is greater than or equal to a preset number threshold.

7. The method according to claim 1, wherein: Integrating and / or filtering the defect information in the first recognition result to obtain a second recognition result includes: Combining defects of the same type that appear in consecutive image frames into one defect; and / or, Integrate multiple defects of the same type that appear at intervals in the image frame and whose intervals are less than or equal to a preset first interval threshold into one defect; and / or, If the distance between a defect appearing in an image frame and the last and next image frames where the defect appears exceeds a preset second interval threshold, deleting the defect appearing in the image frame; and / or, Based on the areas of the identified defects, the defects whose areas do not meet the preset area requirements are filtered out; and / or, According to the confidence of the identified defects, the defects with confidence less than or equal to the preset confidence threshold are filtered out.

8. The method according to any one of claims 1 to 7, further comprising: An alarm is issued based on the defect information in the second recognition result, and the alarm uses at least one of sound, text and special identification information.

9. A method for displaying pipeline defect inspection information, comprising: Obtain video data obtained by shooting the inside of the pipeline; Performing frame extraction processing on the video data to obtain multiple image frames; Using a pipeline defect detection model to perform defect recognition on each image frame to obtain a first recognition result, wherein the pipeline defect detection model is a machine learning model; Integrating and / or filtering the defect information in the first recognition result to obtain a second recognition result; Superimposing the second recognition result with the video data, and rendering the result on a display screen of an augmented reality (AR) device; The method also includes: using the auxiliary model to perform at least one of shooting angle recognition, pipeline material recognition, detection well recognition and distance recognition on each image frame, wherein the distance is the distance between the shooting position corresponding to the image frame and the starting shooting position of the video data; using the recognition result of the auxiliary model, executing the defect recognition of each image frame using the pipeline defect detection model or integrating and / or filtering the defect information in the first recognition result.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 9 are implemented.

11. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 9.

Images