Endoscope-based image analysis method, terminal device and storage medium

Abstract

The present application belongs to the field of image processing technology, and in particular, relates to an endoscope-based image analysis method, terminal equipment and storage medium, wherein the method can identify actual defects inside a pipeline, determine serious actual defects therein, and prompt a user to promptly maintain the serious actual defects. Furthermore, the method can also predict and simulate actual defects with extension trends based on information of the detected actual defects to determine whether composite defects will occur due to defect extension. If so, the user can be prompted to take timely precautions to avoid the occurrence of composite defects. In the present application, timely repair notifications of existing defects on the inner wall of the pipeline can be achieved at the same time, and potential serious defects in the future can be warned, so that the user can take comprehensive maintenance measures, thereby ensuring the continued normal use of the pipeline.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to an endoscope-based image analysis method, a terminal device, and a storage medium. Background Art

[0002] An industrial endoscope is a device used to inspect and monitor internal areas that are difficult to directly access or observe. It is commonly used in industrial repair, quality control, inspection and maintenance.

[0003] The detection of internal defects in pipelines is usually achieved through industrial endoscopes. The existing methods of using industrial endoscopes to detect pipelines can often only identify defects that have actually occurred in the pipelines. For some types of defects (such as cracks), they will extend and develop over time. However, existing methods are difficult to predict and simulate the extension and development of defects that have already occurred, and thus it is difficult to prevent the occurrence of serious defects (such as compound defects formed by the extension and convergence of multiple defects) in a timely manner. Summary of the invention

[0004] In view of this, the embodiments of the present application provide an endoscope-based image analysis method, a terminal device and a storage medium, which can solve the above-mentioned technical problems.

[0005] A first aspect of an embodiment of the present application provides an endoscope-based image analysis method, the endoscope-based image analysis method comprising:

[0006] S1: After the endoscope enters the tested pipe for the first round of inspection and completes the inspection, an inner surface point cloud model of the tested pipe corresponding to the first round of inspection is generated based on the real-scene images collected by the endoscope, and defects on the inner surface point cloud model are identified;

[0007] S2: When a new round of endoscope enters the tested pipe for inspection, a video screen of the endoscope is obtained, wherein a real-life image of the inner surface of the tested pipe is displayed on the video screen;

[0008] S3: for each real-scene image displayed on the video screen, identifying actual defects on the inner surface of the tested pipeline on the real-scene image, and determining whether each actual defect exceeds a preset defect degree, and if so, generating first warning information for the actual defect;

[0009] S4: calling out the previously generated inner surface point cloud model, and performing coordinate alignment and blurring processing on the inner surface point cloud model, so that the real scene image and the blurred inner surface point cloud model are superimposed and displayed in the video screen;

[0010] S5: taking defects on the virtualized inner surface point cloud model as previous defects, taking actual defects that do not exceed a preset defect level as remaining actual defects, and comparing the positions of each remaining actual defect with those of each previous defect to determine whether the remaining actual defect is extended from the previous defect;

[0011] S6: If yes, generating extension prediction information according to the remaining actual defects and the positions of the corresponding previous defects;

[0012] S7: simulate extension of the corresponding remaining actual defects according to each generated extension prediction information, determine whether a composite defect will be generated according to the position of the remaining actual defects of each simulated extension, and if so, generate second warning information;

[0013] S8: After the current round of detection is completed, an inner surface point cloud model corresponding to the current round of detection is generated based on the real scene images collected by the current round of detection, and steps S2 to S8 are executed until the detected pipeline is stopped from being used.

[0014] A second aspect of an embodiment of the present application provides a terminal device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the endoscope-based image analysis method.

[0015] A third aspect of an embodiment of the present application provides a terminal-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the steps of the endoscope-based image analysis method.

[0016] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the method provided by the present invention includes: after the endoscope enters the tested pipeline for the first round of inspection and completes the inspection, generating an inner surface point cloud model of the tested pipeline corresponding to the first round of inspection based on the real-scene images collected by the endoscope, and identifying defects on the inner surface point cloud model; when a new round of endoscope enters the tested pipeline for inspection, obtaining a video screen of the endoscope, wherein the real-scene image of the inner surface of the tested pipeline is displayed on the video screen; for each real-scene image displayed on the video screen, identifying actual defects on the inner surface of the tested pipeline on the real-scene image, and judging whether each actual defect exceeds a preset defect degree, if so, generating a first warning information for the actual defect; calling out the previously generated inner surface point cloud model, and performing coordinate alignment and blurring processing on the inner surface point cloud model, so that the real-scene image and the blurred inner surface point cloud model are superimposed and displayed in the video screen; taking the defects on the blurred inner surface point cloud model as previous defects, taking the actual defects that do not exceed the preset defect degree as remaining actual defects, and for each remaining actual defect, comparing the positions of the remaining actual defects and the previous defects to judge Whether the remaining actual defect is obtained by extending the previous defect; if so, generating extension prediction information based on the remaining actual defect and the corresponding previous defect position; simulating the extension of the corresponding remaining actual defect based on each generated extension prediction information, and determining whether a composite defect will be generated based on the position of each simulated extended remaining actual defect, and if so, generating second warning information; after the completion of this round of detection, generating an inner surface point cloud model corresponding to this round of detection based on the real-scene image collected by this round of detection, executing steps S2 to S8 until the tested pipeline is stopped; in the present application, it is possible to identify the actual defects inside the pipeline and prompt the user to perform maintenance on the serious actual defects in time, and further, it is possible to perform prediction simulation based on the detected actual defects to determine whether a composite defect will be generated due to the extension of the defect, and if so, it is possible to prompt the user to take timely precautions to avoid the occurrence of composite defects, and simultaneously realize timely repair notification of existing defects on the inner wall of the pipeline, and to give early warning of potential serious defects in the future, so that the user can take comprehensive maintenance measures, thereby ensuring the continuous normal use of the pipeline. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 It is a schematic diagram of the implementation process of the endoscope-based image analysis method provided in an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of an implementation environment of an endoscope-based image analysis method provided in an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of a first path of an endoscope-based image analysis method provided in an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of a first coordinate system of an endoscope-based image analysis method provided in an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of a target vector of an endoscope-based image analysis method provided in an embodiment of the present application;

[0023] Figure 6 It is a schematic diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0025] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0026] Figure 1 The endoscope-based image analysis method provided in the first embodiment of the present application is shown, and the endoscope-based image analysis method includes:

[0027] S1: After the endoscope enters the tested pipe for the first round of inspection and completes the inspection, an inner surface point cloud model of the tested pipe corresponding to the first round of inspection is generated based on the real-scene images collected by the endoscope, and defects on the inner surface point cloud model are identified;

[0028] S2: When a new round of endoscope enters the tested pipe for inspection, a video screen of the endoscope is obtained, wherein a real-life image of the inner surface of the tested pipe is displayed on the video screen;

[0029] S3: for each real-scene image displayed on the video screen, identifying actual defects on the inner surface of the tested pipeline on the real-scene image, and determining whether each actual defect exceeds a preset defect degree, and if so, generating first warning information for the actual defect;

[0030] S4: calling out the previously generated inner surface point cloud model, and performing coordinate alignment and blurring processing on the inner surface point cloud model, so that the real scene image and the blurred inner surface point cloud model are superimposed and displayed in the video screen;

[0031] S5: taking defects on the virtualized inner surface point cloud model as previous defects, taking actual defects that do not exceed a preset defect level as remaining actual defects, and comparing the positions of each remaining actual defect with those of each previous defect to determine whether the remaining actual defect is extended from the previous defect;

[0032] S6: If yes, generating extension prediction information according to the remaining actual defects and the positions of the corresponding previous defects;

[0033] S7: simulate extension of the corresponding remaining actual defects according to each generated extension prediction information, determine whether a composite defect will be generated according to the position of the remaining actual defects of each simulated extension, and if so, generate second warning information;

[0034] S8: After the current round of detection is completed, an inner surface point cloud model corresponding to the current round of detection is generated based on the real scene images collected by the current round of detection, and steps S2 to S8 are executed until the detected pipeline is stopped from being used.

[0035] In this embodiment, if Figure 2 As shown, the method is executed in a terminal device, which may be an independent physical server or terminal, or a server cluster composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud servers, cloud databases, cloud storage, and CDN. The method uses an industrial endoscope to detect the pipeline to be tested. The industrial endoscope is connected to the terminal device, and a display is provided on the terminal device. The image captured by the camera of the industrial endoscope can be displayed on the display of the terminal device.

[0036] In this embodiment, each tested pipeline will undergo multiple rounds of inspections and corresponding maintenance throughout its life cycle to ensure its quality of use throughout its life cycle; the first round of inspections may be the inspection conducted when the tested pipeline is just put into use, and the last round of inspections may be the inspection conducted before reaching a preset service life (when the service life is reached, the tested pipeline is stopped from being used); the intervals between the inspections may be the same or different, and are not limited here.

[0037] In this embodiment, the video picture of the endoscope is the picture captured by the endoscope camera. The video picture is composed of several frames of real-scene images, each of which presents a part of the inner surface of the measured pipeline. After the user controls the industrial endoscope to move in the measured pipeline, all local real-scene images of the inner surface of the pipeline can be captured; the camera of the endoscope adopts a depth camera, which can identify the spatial coordinates of each point in the captured image (before the first round of detection, a reference space coordinate system can be established, and the spatial coordinates in the images captured in each subsequent round of detection are the coordinates on the reference space coordinate system), and then the built-in image processing algorithm in the terminal device, such as the accelerated robust feature algorithm (SURF algorithm), can be used to extract feature points in each image, and feature points with the same coordinates in different images can be matched, and then a point cloud can be generated based on the matched feature points through the built-in three-dimensional processing software, such as MeshLab, to reconstruct the corresponding inner surface point cloud model.

[0038] In this embodiment, the actual defect is a defect that has actually occurred on the current real scene image, and the first warning information includes the type, degree and location of the actual defect (the location can be the spatial coordinates of the geometric center of the actual defect); the generated first information can be sent to the user terminal of the staff to prompt them to perform corresponding maintenance work on the actual defect;

[0039] In this embodiment, the coordinate alignment of the inner surface point cloud model is performed, that is, the position of the inner surface point cloud model is adjusted in the reference space coordinate system of the video screen, so that each point of the inner surface point cloud model is located at its corresponding space coordinate; after the coordinate alignment, the inner surface of the pipeline in the real scene image coincides with the inner surface point cloud model (only a part of the inner surface of the pipeline is displayed in the video screen, so only a part of the coincidence is displayed), and in order to facilitate image analysis, the point cloud model is blurred so that the user can see both the point cloud model and the real scene image;

[0040] In this embodiment, some actual defects (such as cracks) will not cause substantial harm at present, but due to their ductility, they may cause harm after extension, especially after the defect is extended to form a composite defect with other extension defects, which will have a serious impact on the use of the pipeline; furthermore, since the defect that will extend will also extend before the current round of detection, and the extension of the defect is continuous, the subsequent extension trend can be predicted from the previous extension trend, and the inner surface point cloud model represents the previous state of the defect. Therefore, the extension prediction information characterizing the extension trend of the defect can be determined from the defects of the inner surface point cloud model and the defects on the current real scene image; by simulating the extension of the corresponding actual defect using the extension prediction information, the future state of the defect can be simulated, and then it can be determined whether a composite defect will be formed. If so, a second warning information is generated to prompt the user to take corresponding remedial measures.

[0041] In the present application, it is possible to identify actual defects inside the pipeline and prompt the user to perform maintenance on serious actual defects in a timely manner. Furthermore, it is possible to perform predictive simulation based on the detected actual defects to determine whether composite defects will occur due to defect extension. If so, the user can be prompted to take timely precautions to avoid the occurrence of composite defects. At the same time, timely repair notifications of existing defects on the inner wall of the pipeline can be achieved, and potential serious defects in the future can be warned, so that users can take comprehensive maintenance measures, thereby ensuring the continued normal use of the pipeline.

[0042] As a preferred embodiment, there are several types of defects, and a corresponding indicator parameter is used to characterize the degree of each type of defect. A corresponding critical value is preset for each indicator parameter. Judging whether any actual defect exceeds the preset defect degree includes:

[0043] Get a screenshot of the actual defect;

[0044] Compare the screenshot with the images in the defect type database to determine the type of the actual defect, wherein the defect type database includes various defect types and a number of images corresponding to each defect type;

[0045] Determine corresponding index parameters according to the type of the actual defect, and detect the index parameters of the actual defect;

[0046] It is determined whether the detection value exceeds the critical value of the indicator parameter. If so, the actual defect exceeds the preset defect level. If not, the actual defect does not exceed the preset defect level.

[0047] In this embodiment, there are several types of defects, such as corrosion, depressions, cracks, etc. For each type of defect, a corresponding indicator parameter is used to characterize the defect degree, such as crack length, depression diameter, etc.; a corresponding critical value is preset for each indicator parameter, such as the critical value of the depression diameter is 2 cm; in this embodiment, the image of the actual defect is first compared and matched with the image in the defect type database to determine the type of the actual defect, and then the indicator parameter corresponding to the type in the actual defect is identified. For example, if the maximum depression diameter is identified to be 1.5 cm and does not exceed the corresponding critical value, then the actual defect does not exceed the preset defect degree, otherwise, the actual defect exceeds the preset defect degree; in this embodiment, defects that may cause harm can be first identified to notify the user to repair them in time.

[0048] As a preferred embodiment, calling out the previously generated inner surface point cloud model, and performing coordinate alignment and blurring processing on the inner surface point cloud model includes:

[0049] Call out each previously generated inner surface point cloud model, and perform coordinate alignment on each inner surface point cloud model;

[0050] The transparency of each inner surface point cloud model is adjusted to a preset transparency, so that each inner surface point cloud model is blurred;

[0051] The defects on each inner surface point cloud model and the real scene image are rendered in different colors.

[0052] Comparing the remaining actual defect with the positions of each previous defect to determine whether the remaining actual defect is extended from the previous defect includes:

[0053] Screening out, from among the previous defects, previous defects that overlap with the remaining actual defects and are of the same type as the remaining actual defects;

[0054] It is determined whether there is a target defect among the previously screened defects that is completely contained in the remaining actual defects and one end point of which coincides with an end point of the remaining actual defect. If so, the remaining actual defect is obtained by extending the target defect.

[0055] In this embodiment, as the number of inspection rounds increases, the number of previous inner surface point cloud models increases. Each time a defect extension trend analysis is performed, all inner surface point cloud models are called out, so that the historical extension trend of the defects can be presented in detail, so that the extension prediction information obtained is more accurate; the preset transparency can be 50%, 60% or other values, which are not limited here; the defects on the inner surface point cloud model and the real-life image are rendered in different colors, so that the defects detected in each period appear in different colors, which is easier to distinguish and their extension path can be intuitively seen.

[0056] In this embodiment, if a remaining actual defect is extended from a previous defect, then the previous defect must overlap with the remaining actual defect and be completely contained in the remaining actual defect (that is, the length of the previous defect is shorter than the remaining actual defect, and the previous defect is only equivalent to a part of the remaining actual defect); furthermore, an extended remaining actual defect often undergoes multiple rounds of detection from the initial stage to the current state, that is, there are often multiple previous defects corresponding to the remaining actual defect, each previous defect starts at the same point and has a different length, so that multiple overlapping previous defects can show the development process of the remaining actual defect from its generation to the current state.

[0057] like Figure 3-Figure 4 As shown, as a preferred embodiment, each actual defect obtained by extending a previous defect is superimposed with at least one previous defect; the extension prediction information includes an extension direction and an extension speed, and the extension prediction information is generated according to the remaining actual defect and the position of the corresponding previous defect, including:

[0058] Identify the extension paths of the remaining actual defects and the corresponding previous defects, and use the extension paths of the remaining actual defects as the first path, wherein the extension path of a defect is the central axis of the defect;

[0059] Identify the starting node and the ending node of the first path, and mark the acquisition time of the remaining actual defects at the ending node of the first path, wherein the starting node of the first path is the common endpoint of the first path and the extension path of the previous defect, and the other endpoint of the first path is the ending node;

[0060] Determine the position where the terminal node of the extended path of each previous defect falls on the first path, mark the intermediate node at the position on the first path, and mark the acquisition time of the corresponding previous defect at each intermediate node, so as to obtain a first path with multiple nodes;

[0061] Taking each intermediate node and the end node of the first path as a calculation node, detecting the length of the line segment between each calculation node and the start node of the first path;

[0062] Generate a first coordinate system whose abscissa is the acquisition time and whose ordinate is the line segment length, and mark corresponding coordinate points on the first coordinate system according to the line segment length and the acquisition time corresponding to each calculation node;

[0063] Determine the regression line of each coordinate point in the first coordinate system, and use the slope of the regression line as the extension speed;

[0064] Determine a target vector closest to each node of the first path, and use the direction of the target vector as the extension direction;

[0065] The obtained extension speed and extension direction are integrated into the extension prediction information of the remaining actual defects.

[0066] In this embodiment, since different defects have different widths, the extension path directly reflected by the shape of the defect is not obvious, so the central axis of the defect is taken to represent its route, which is more accurate; when the industrial camera performs image acquisition, it will record the image acquisition time. The acquisition time of a defect is the time when the defect is first acquired; since the remaining actual defects and the corresponding previous defects are in a relationship of extension sequence, their corresponding extension paths must overlap, but the lengths are different, and the paths of defects closer to the current time are longer; the middle node on the first path represents the position to which the defect extends during each round of detection, and the length of the line segment corresponding to the position represents the length of the defect at the corresponding time point; in this embodiment, the regression line of the coordinate point represents the relationship between the overall extension length of the defect from the beginning to the present and time, and its slope represents the overall extension speed. Simulation at this speed has higher accuracy.

[0067] like Figure 5 As shown, as a preferred embodiment, determining the target vector closest to each node of the first path and using the direction of the target vector as the extension direction includes:

[0068] Generate a vector at the first path;

[0069] The relative position of the vector and the first path is adjusted until the sum of the weighted distances between each node on the first path and the straight line where the vector is located is minimized, and the vector obtained by the adjustment is the target vector;

[0070] The weighted distance sum is calculated by the following formula:

[0071]

[0072] in, is the weighted distance sum, n is the total number of nodes on the first path, is the distance between the i-th node on the first path and the line where the vector is located, For the The weighting coefficient of The value of increases as i increases.

[0073] In this embodiment, since the extension direction is a direction guiding the continued extension from the terminal node of the first path, the direction of the generated vector can be a direction consistent with the direction from the starting node of the first path toward the terminal node. Adjusting the relative position of the vector to the first path mainly adjusts the position of the vector, and only fine-tunes the direction of the vector (the angle deviation from the initial direction does not exceed a set angle, such as 30°, 35° or other angles), so that the direction of the vector is always roughly consistent with the direction of the initial direction, thereby avoiding obtaining a target vector opposite to the initial direction; in addition, the target vector closest to each node of the first path is the vector with the smallest weighted distance and each node on the first path. The first node on the first path is the starting node, and subsequent nodes are calculated in sequence; the later part of the first path (the part of the line segment close to the terminal node in the first path) corresponds to the stronger representation of the extension direction, and the weighting coefficient increases with the increase of i, that is, the nodes in the later part of the first path can play a more important role in determining the extension direction, which is consistent with the actual situation and has higher simulation accuracy.

[0074] As a preferred embodiment, according to each generated extension prediction information, the corresponding remaining actual defects are simulated and extended, and it is determined whether a composite defect will be generated according to the position of the remaining actual defects of each simulated extension. If so, the second warning information is generated, including:

[0075] S71: Obtain a preset unit time length t, and set m=1;

[0076] S72: Determine simulation time T=m*t;

[0077] S73: for each remaining actual defect, obtaining an extension speed v in the extension prediction information of the remaining actual defect;

[0078] S74: Calculate a first length d=T*v for simulating the extension of the remaining actual defect;

[0079] S75: Taking the termination node of the first path corresponding to the remaining actual defect as the starting point, extending the first path by a first length in the extension direction corresponding to the remaining actual defect;

[0080] S76: determining whether each extended first path generates an intersection, and if an intersection is generated, generating second warning information for the intersection;

[0081] S77: Let m=m+1, and execute steps S72 to S77 until T is greater than the set time length.

[0082] Generating the second warning information for the intersection includes:

[0083] Identifying the position information of the intersection and the position information of each first path intersecting the intersection after extension;

[0084] Determine the simulation time T corresponding to the generation of the intersection point;

[0085] The simulation duration and the obtained position information are integrated into the second warning information.

[0086] In this embodiment, the unit time can be 10 hours, 15 hours or other time. This embodiment adopts a step-by-step simulation method, which can determine the simulation time corresponding to the occurrence of any composite defect, that is, the time distance between the time point when the composite defect occurs and the current time point. The generated second warning information includes the time, which can enable the user to know when the potential composite defect will occur approximately, so as to take relevant preventive measures in time. Furthermore, the position information of the intersection is the spatial coordinates of the intersection, and the position information of the first path is the spatial coordinates of the starting node and the ending node of the first path.

[0087] A terminal device provided in a second embodiment of the present application includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the endoscope-based image analysis method, specifically including:

[0088] S1: After the endoscope enters the tested pipe for the first round of inspection and completes the inspection, an inner surface point cloud model of the tested pipe corresponding to the first round of inspection is generated based on the real-scene images collected by the endoscope, and defects on the inner surface point cloud model are identified;

[0089] S2: When a new round of endoscope enters the tested pipe for inspection, a video screen of the endoscope is obtained, wherein a real-life image of the inner surface of the tested pipe is displayed on the video screen;

[0090] S3: for each real-scene image displayed on the video screen, identifying actual defects on the inner surface of the tested pipeline on the real-scene image, and determining whether each actual defect exceeds a preset defect degree, and if so, generating first warning information for the actual defect;

[0091] S4: calling out the previously generated inner surface point cloud model, and performing coordinate alignment and blurring processing on the inner surface point cloud model, so that the real scene image and the blurred inner surface point cloud model are superimposed and displayed in the video screen;

[0092] S5: taking defects on the virtualized inner surface point cloud model as previous defects, taking actual defects that do not exceed a preset defect level as remaining actual defects, and comparing the positions of each remaining actual defect with those of each previous defect to determine whether the remaining actual defect is extended from the previous defect;

[0093] S6: If yes, generating extension prediction information according to the remaining actual defects and the positions of the corresponding previous defects;

[0094] S7: simulate extension of the corresponding remaining actual defects according to each generated extension prediction information, determine whether a composite defect will be generated according to the position of the remaining actual defects of each simulated extension, and if so, generate second warning information;

[0095] S8: After the current round of detection is completed, an inner surface point cloud model corresponding to the current round of detection is generated based on the real scene images collected by the current round of detection, and steps S2 to S8 are executed until the detected pipeline is stopped from being used.

[0096] A terminal-readable storage medium is provided in the third embodiment of the present application. A computer program is stored on the terminal-readable storage medium. When the computer program is executed by a processor, the processor executes the steps of the endoscope-based image analysis method, specifically including:

[0097] S1: After the endoscope enters the tested pipe for the first round of inspection and completes the inspection, an inner surface point cloud model of the tested pipe corresponding to the first round of inspection is generated based on the real-scene images collected by the endoscope, and defects on the inner surface point cloud model are identified;

[0098] S2: When a new round of endoscope enters the tested pipe for inspection, a video screen of the endoscope is obtained, wherein a real-life image of the inner surface of the tested pipe is displayed on the video screen;

[0099] S3: for each real-scene image displayed on the video screen, identifying actual defects on the inner surface of the tested pipeline on the real-scene image, and determining whether each actual defect exceeds a preset defect degree, and if so, generating first warning information for the actual defect;

[0100] S4: calling out the previously generated inner surface point cloud model, and performing coordinate alignment and blurring processing on the inner surface point cloud model, so that the real scene image and the blurred inner surface point cloud model are superimposed and displayed in the video screen;

[0101] S5: taking defects on the virtualized inner surface point cloud model as previous defects, taking actual defects that do not exceed a preset defect level as remaining actual defects, and comparing the positions of each remaining actual defect with those of each previous defect to determine whether the remaining actual defect is extended from the previous defect;

[0102] S6: If yes, generating extension prediction information according to the remaining actual defects and the positions of the corresponding previous defects;

[0103] S7: simulate extension of the corresponding remaining actual defects according to each generated extension prediction information, determine whether a composite defect will be generated according to the position of the remaining actual defects of each simulated extension, and if so, generate second warning information;

[0104] S8: After the current round of detection is completed, an inner surface point cloud model corresponding to the current round of detection is generated based on the real scene images collected by the current round of detection, and steps S2 to S8 are executed until the detected pipeline is stopped from being used.

[0105] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0106] It should be understood that when used in the present application specification, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0107] It should also be understood that the term “and / or” used in the specification of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0108] As used in the present specification, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [the described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [the described condition or event] is detected" or "in response to detecting [the described condition or event]" depending on the context.

[0109] In addition, in the description of the present specification, the terms "first", "second", "third", etc. are only used to distinguish descriptions, and cannot be understood as indicating or implying relative importance. It should also be understood that although the terms "first", "second", etc. are used to describe various elements in some embodiments of the present application in the text, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first table can be named as the second table, and similarly, the second table can be named as the first table, without departing from the scope of the various described embodiments. The first table and the second table are both tables, but they are not the same table.

[0110] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0111] The endoscope-based image analysis method provided in the embodiments of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiments of the present application do not impose any restrictions on the specific types of terminal devices.

[0112] For example, the terminal device can be a station (STAION, ST) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, a vehicle networking terminal, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite wireless device, a wireless modem card, a TV set top box (STB), a customer premises equipment (CPE) and / or other devices for communicating on a wireless system and a next-generation communication system, such as a mobile terminal in a 5G network or a mobile terminal in a future evolved Public Land Mobile Network (PLMN) network, etc.

[0113] As an example but not limitation, when the terminal device is a wearable device, the wearable device can also be a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not just hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are fully functional, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0114] Figure 6 Schematic diagram of the structure of a terminal device provided by an embodiment of the present application. Figure 6 As shown, the terminal device of this embodiment includes: at least one processor, a memory, and a display, wherein the memory stores a computer program that can be run on the processor. When the processor executes the computer program, the steps in the above-mentioned various endoscope-based image analysis method embodiments are implemented, such as Figure 1 Steps S1 to S8 are shown.

[0115] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 6 It is only an example of a terminal device and does not constitute a limitation of the terminal device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include an input sending device, a network access device, a bus, etc.

[0116] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0117] In some embodiments, the memory may be an internal storage unit of the terminal device, such as a hard disk or memory of the terminal device. The memory may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device. Furthermore, the memory may include both an internal storage unit and an external storage device of the terminal device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory may also be used to temporarily store data that has been sent or is to be sent.

[0118] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0119] An embodiment of the present application provides a computer program product. When the computer program product is run on a mobile terminal device, the mobile terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0120] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0121] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0122] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0123] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0124] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An endoscope-based image analysis method, characterized in that: The endoscope-based image analysis method comprises: S1: After the endoscope enters the tested pipe for the first round of inspection and completes the inspection, an inner surface point cloud model of the tested pipe corresponding to the first round of inspection is generated based on the real-scene images collected by the endoscope, and defects on the inner surface point cloud model are identified; S2: When a new round of endoscope enters the tested pipe for inspection, a video screen of the endoscope is obtained, wherein a real-life image of the inner surface of the tested pipe is displayed on the video screen; S3: for each real-scene image displayed on the video screen, identifying actual defects on the inner surface of the tested pipeline on the real-scene image, and determining whether each actual defect exceeds a preset defect degree, and if so, generating first warning information for the actual defect; S4: calling out the previously generated inner surface point cloud model, and performing coordinate alignment and blurring processing on the inner surface point cloud model, so that the real scene image and the blurred inner surface point cloud model are superimposed and displayed in the video screen; S5: taking defects on the virtualized inner surface point cloud model as previous defects, taking actual defects that do not exceed a preset defect level as remaining actual defects, and comparing the positions of each remaining actual defect with those of each previous defect to determine whether the remaining actual defect is extended from the previous defect; S6: If yes, generating extension prediction information according to the remaining actual defects and the positions of the corresponding previous defects; S7: simulate extension of the corresponding remaining actual defects according to each generated extension prediction information, determine whether a composite defect will be generated according to the position of the remaining actual defects of each simulated extension, and if so, generate second warning information; S8: after the current round of detection is completed, an inner surface point cloud model corresponding to the current round of detection is generated based on the real scene images collected by the current round of detection, and steps S2 to S8 are executed until the detected pipeline is stopped; Comparing the remaining actual defect with the positions of each previous defect to determine whether the remaining actual defect is extended from the previous defect includes: Screening out, from among the previous defects, previous defects that overlap with the remaining actual defects and are of the same type as the remaining actual defects; Determine whether there is a target defect among the previously screened defects that is completely contained in the remaining actual defects and one end point of which coincides with an end point of the remaining actual defect. If so, the remaining actual defect is obtained by extending the target defect. Each actual defect obtained by extending a previous defect is superimposed with at least one previous defect; the extension prediction information includes an extension direction and an extension speed, and the extension prediction information is generated according to the remaining actual defect and the position of the corresponding previous defect, including: Identify the extension paths of the remaining actual defects and the corresponding previous defects, and use the extension paths of the remaining actual defects as the first path, wherein the extension path of a defect is the central axis of the defect; Identify the starting node and the ending node of the first path, and mark the acquisition time of the remaining actual defects at the ending node of the first path, wherein the starting node of the first path is the common endpoint of the first path and the extension path of the previous defect, and the other endpoint of the first path is the ending node; Determine the position where the terminal node of the extended path of each previous defect falls on the first path, mark the intermediate node at the position on the first path, and mark the acquisition time of the corresponding previous defect at each intermediate node, so as to obtain a first path with multiple nodes; Taking each intermediate node and the end node of the first path as a calculation node, detecting the length of the line segment between each calculation node and the start node of the first path; Generate a first coordinate system whose abscissa is the acquisition time and whose ordinate is the line segment length, and mark corresponding coordinate points on the first coordinate system according to the line segment length and the acquisition time corresponding to each calculation node; Determine the regression line of each coordinate point in the first coordinate system, and use the slope of the regression line as the extension speed; Determine a target vector closest to each node of the first path, and use the direction of the target vector as the extension direction; The obtained extension speed and extension direction are integrated into the extension prediction information of the remaining actual defects.

2. The method according to claim 1, characterized in that: There are several types of defects. For each type of defect, a corresponding indicator parameter is used to characterize the degree of the defect. For each indicator parameter, a corresponding critical value is preset; Determining whether any actual defect exceeds the preset defect level includes: Get a screenshot of the actual defect; Compare the screenshot with the images in the defect type database to determine the type of the actual defect, wherein the defect type database includes various defect types and a number of images corresponding to each defect type; Determine corresponding index parameters according to the type of the actual defect, and detect the index parameters of the actual defect; It is determined whether the detection value exceeds the critical value of the indicator parameter. If so, the actual defect exceeds the preset defect level. If not, the actual defect does not exceed the preset defect level.

3. The method according to claim 1, characterized in that Retrieving the previously generated inner surface point cloud model, and performing coordinate alignment and blurring processing on the inner surface point cloud model includes: Call out each previously generated inner surface point cloud model, and perform coordinate alignment on each inner surface point cloud model; The transparency of each inner surface point cloud model is adjusted to a preset transparency, so that each inner surface point cloud model is blurred; The defects on each inner surface point cloud model and the real scene image are rendered in different colors.

4. The method according to claim 1, characterized in that: Determining a target vector closest to each node of the first path, and using the direction of the target vector as an extension direction includes: Generate a vector at the first path; The relative position of the vector and the first path is adjusted until the sum of the weighted distances between each node on the first path and the straight line where the vector is located is minimized, and the vector obtained by the adjustment is the target vector; The weighted distance sum is calculated by the following formula: in, is the weighted distance sum, n is the total number of nodes on the first path, is the distance between the i-th node on the first path and the line where the vector is located, For the The weighting coefficient of The value of increases as i increases.

5. The method according to claim 1, characterized in that According to each generated extension prediction information, the corresponding remaining actual defects are simulated and extended, and it is determined whether a composite defect will be generated according to the position of the remaining actual defects of each simulated extension. If so, the second warning information is generated, including: S71: Obtain a preset unit time length t, and set m=1; S72: Determine simulation duration T=m*t; S73: for each remaining actual defect, obtaining an extension speed v in the extension prediction information of the remaining actual defect; S74: Calculate a first length d=T*v for simulating the extension of the remaining actual defect; S75: Taking the termination node of the first path corresponding to the remaining actual defect as the starting point, extending the first path by a first length in the extension direction corresponding to the remaining actual defect; S76: determining whether each extended first path generates an intersection, and if an intersection is generated, generating second warning information for the intersection; S77: Let m=m+1, and execute steps S72 to S77 until T is greater than the set time length.

6. The method according to claim 5, characterized in that Generating the second warning information for the intersection includes: Identifying the position information of the intersection and the position information of each first path intersecting the intersection after extension; Determine the simulation time T corresponding to the generation of the intersection point; The simulation duration and the obtained position information are integrated into the second warning information.

7. A terminal device, characterized in that: The invention comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the endoscope-based image analysis method according to any one of claims 1 to 6.

8. A terminal-readable storage medium, characterized in that: A computer program is stored on the terminal-readable storage medium. When the computer program is executed by a processor, the processor executes the steps of the endoscope-based image analysis method according to any one of claims 1 to 6.

Images