Endoscopic pipeline inner wall crack automatic identification system based on video deep learning

Abstract

The invention discloses an automatic endoscope pipeline inner wall crack identification system based on video deep learning, and relates to the technical field of pipeline detection, and the system comprises an image acquisition module, an image preprocessing module, a pipeline inner wall model establishment module, a deep learning identification module and a result output module. The image acquisition module is used for acquiring images in the pipeline by utilizing the autonomous moving endoscope device, and a multi-wheel driving structure, a self-adaptive rubber track, a high-precision sensor and the like of the image acquisition module ensure that the acquisition process is stable; according to the invention, the automation level and accuracy of pipeline inner wall crack detection are obviously improved, the autonomous moving endoscope device is matched with three-dimensional illumination and multi-angle image acquisition to ensure that high-quality pipeline inner wall images can be obtained in a complex environment, and through calibration registration, noise suppression and contrast enhancement of the image preprocessing module, the pipeline inner wall crack detection accuracy is improved. And high-quality input data is provided for a deep learning model, so that the accuracy and reliability of crack identification are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of pipeline detection, and in particular to an automatic recognition system for cracks on the inner wall of an endoscopic pipeline based on video deep learning. Background Art

[0002] In industrial production, the pipeline system is a key infrastructure for transporting various media (such as liquids, gases, etc.). It is widely used in many fields such as petrochemicals, electricity, water supply and drainage. The integrity of the inner wall of the pipeline is directly related to the safety, efficiency and environmental protection of production. For example, in the petrochemical industry, pipeline leakage may cause serious accidents such as fires and explosions, and pollute the environment; in the water supply system, pipeline damage will lead to waste of water resources and water pollution. Therefore, it is crucial to detect defects such as cracks on the inner wall of the pipeline in a timely and accurate manner. However, traditional detection methods have many shortcomings. Manual visual inspection relies on the experience and skills of the inspectors, is highly subjective, has low detection efficiency, and is prone to fatigue due to long-term work, resulting in missed inspections. Non-destructive testing technologies such as ultrasonic testing and radiographic testing can detect some defects, but the operation is complicated and requires professional personnel to operate the equipment. It has high requirements for the surface of the pipeline, and the detection process may cause certain harm to the health of personnel, which is difficult to meet the needs of large-scale and rapid detection.

[0003] With the rapid development of artificial intelligence technology, video deep learning technology has shown great potential in the field of industrial inspection. Deep learning algorithms have achieved remarkable results in tasks such as image recognition, classification, and target detection. In terms of industrial visual inspection, deep learning-based methods can automatically learn features in images and have stronger adaptability to complex scenes and diverse defect types. For example, in some surface defect detection scenarios, deep learning models can accurately identify tiny scratches, cracks and other defects with much higher accuracy and efficiency than traditional methods. Video deep learning technology can also process large amounts of image data, realize real-time detection, and provide timely feedback on the condition of the inner wall of the pipeline, providing more reliable protection for industrial production. The development trend of this technology has prompted people to explore its application in the detection of cracks on the inner wall of pipelines to overcome the limitations of traditional detection methods.

[0004] In order to solve the above defects, a technical solution is now provided. Summary of the invention

[0005] The purpose of the present invention is to solve the problems of low accuracy, low efficiency and insufficient reliability of existing pipeline inner wall crack detection technology, and to propose an endoscopic pipeline inner wall crack automatic identification system based on video deep learning.

[0006] The purpose of the present invention can be achieved through the following technical solutions:

[0007] The automatic identification system of cracks on the inner wall of endoscope pipe based on video deep learning includes:

[0008] The image acquisition module uses an autonomous mobile endoscope device to collect images inside the pipeline, and cooperates with a stereo lighting device and a multi-angle image acquisition module to ensure the acquisition of high-quality, all-round images of the inner wall of the pipeline;

[0009] The image preprocessing module calibrates and registers the collected multi-angle images to ensure geometric accuracy, suppress noise and enhance contrast, and provide high-quality input data for subsequent deep learning models;

[0010] The pipeline inner wall model building module eliminates duplicate parts of image data through feature point extraction and matching, image registration, selects the optimal image based on evaluation parameters, and then uses algorithms to convert the processed image into point cloud data. After filtering, surface reconstruction and texture mapping, a model reflecting the actual appearance of the pipeline inner wall is constructed;

[0011] The deep learning recognition module is used to build an improved convolutional neural network model, which can be trained with different annotated data sets to identify cracks on the inner wall of the pipeline and improve the ability to identify cracks in different situations;

[0012] The result output module displays the crack identification results in a visual interface, highlights the crack location and range with different colors and marks, and generates a detection report containing detailed crack information to provide data support for pipeline maintenance and repair.

[0013] Furthermore, the specific process performed by the image acquisition module is as follows:

[0014] The image data in the pipeline is collected through an autonomously moving endoscope device. The endoscope device adopts a multi-wheel drive structure and is equipped with adaptive rubber tracks on the wheels to adapt to the inner walls of pipelines with different diameters and materials. The endoscope device maintains a stable speed and posture during the crawling process. The endoscope device is equipped with a gyroscope and accelerometer to monitor its position and movement status in the pipeline in real time, and transmit this information through wireless communication.

[0015] The endoscope device is equipped with a three-dimensional lighting device, which consists of multiple LED light sources with independently controllable angles and brightness. These light sources are distributed around the endoscope device and automatically adjust the lighting angle and brightness through intelligent algorithms according to the geometric shape and reflection characteristics of the inner wall of the pipeline.

[0016] A multi-angle image acquisition module is constructed, including multiple cameras installed at different positions and angles of the endoscope device, to simultaneously capture images of the inner wall of the pipeline from different perspectives; the camera has the characteristics of large depth of field and low distortion, capturing images of the inner wall of the pipeline from close to long distances; at the same time, the camera's acquisition frame rate is adaptively adjusted according to the pipeline crawling speed and detection requirements.

[0017] Furthermore, the specific operation steps of automatically adjusting the illumination angle by the intelligent algorithm in the image acquisition module are as follows:

[0018] Angle adjustment based on normal vector: Assume that the normal vector of the inner wall of the pipe at a certain point is The position coordinates of the endoscope device in the pipeline are (x, y, z); for each LED light source i, its initial direction vector is According to the principle of light reflection, the standard lighting angle makes the angle between the light and the normal vector perpendicular to obtain the strongest reflected light; calculate the dot product of each light source direction vector and the normal vector where θ i is the angle between the light source and the normal vector; by adjusting the angle of the light source, cosθ i Equal to; use the gradient descent optimization algorithm to adjust to minimize |cosθ i |, the update formula is Where α is the learning rate;

[0019] Angle adjustment based on ray tracing: Considering the multiple reflections of light on the inner wall of the pipe, light is emitted from each LED light source to simulate the propagation path of light in the pipe. When the light encounters the inner wall of the pipe, the direction of the reflected light is calculated based on the relationship between the incident angle and the reflection angle. By tracing the propagation path of the light, it is analyzed which areas do not receive illumination that meets the standard or have shadows. The angle of the light source is adjusted based on the analysis results so that the light can cover the inner wall of the pipe more evenly.

[0020] In areas where the inner wall of the pipe is flat, the normal vector-based angle adjustment algorithm is first used to adjust the angle of the light source to a direction perpendicular to the inner wall, providing a good foundation for lighting; when encountering curved parts, connecting parts or areas with geometric shapes in the pipe, the ray tracing algorithm is enabled; considering the reflection and propagation path of light on the inner wall of complex shapes, the light source angle can be adjusted more accurately to ensure that these areas can also receive uniform lighting.

[0021] Furthermore, the specific operation steps of automatically adjusting the light brightness by the intelligent algorithm in the image acquisition module are as follows:

[0022] Brightness adjustment based on illumination model: according to the inverse square law of illumination Where E is the illuminance, I is the intensity of the light source, r is the distance from the light source to the illuminated point, and θ is the angle between the light and the normal of the illuminated surface; for each point P on the inner wall of the pipe, calculate its distance r to each LED light source i and the angle θ i ; Given the initial intensity I0 of each light source, calculate the theoretical illuminance at that point Set a target illuminance E target, according to the actual measured illuminance E actual , calculate the brightness adjustment coefficient Then, adjust the brightness of each light source according to I i = kI0; When encountering a curved pipe section, the distance r close to the inner wall becomes smaller, and the E calculated according to the above formula will increase. At this time, the brightness I of the light source on this side is reduced to maintain the balance of illumination;

[0023] Brightness adjustment based on image grayscale histogram: obtain the grayscale histogram H(k) of the image captured by the camera in real time, where k is the grayscale level; calculate the mean of the grayscale histogram and standard deviation Set the target mean μ target and standard deviation σ target , calculate the brightness adjustment parameters according to the current μ and σ, specifically using the linear mapping function L new =aL old +b, where L old is the brightness of the original image, L new is the adjusted brightness, a and b are based on μ, σ, μ target and σ target The calculated coefficients are used to change the overall brightness of the image by adjusting the brightness parameters, and the brightness of the light source is adjusted to obtain appropriate lighting conditions. For the curved pipe part, the brightness of the light source at the corresponding position is adjusted in a targeted manner according to the changes in the grayscale histogram of different areas in the image.

[0024] In the initial stage, the basic brightness of the light source is set according to the illumination model. By calculating the distance and angle from the light source to each point on the inner wall of the pipeline, the brightness of each LED light source is preliminarily determined according to the inverse square law of illumination, so that the entire inner wall of the pipeline has a uniform illumination basis at a macro level. Then, during the image acquisition process, the grayscale histogram of the image captured by the camera is analyzed in real time. When it is found that the overall brightness or local brightness of the image does not meet expectations, the brightness adjustment method based on the image grayscale histogram is used for fine-tuning. This combination is based on the consideration of the physical illumination model and is dynamically optimized according to the visual effects of the actual captured images, ensuring that images with better clarity and detail integrity can be obtained under different pipeline conditions.

[0025] Furthermore, the image preprocessing module performs the following steps:

[0026] Image calibration and registration: Calibrate and register images collected from multiple angles; use the known camera installation position and angle information to map images from different perspectives into a unified coordinate system through a geometric transformation algorithm; at the same time, correct image deformation caused by lens distortion factors to ensure the geometric accuracy of the image and provide accurate input data for subsequent deep learning models;

[0027] Noise suppression and enhancement: Image noise suppression technology is used, combined with the characteristics of the pipeline inner wall image, to effectively remove the noise in the image; a method based on wavelet transform and local adaptive threshold is used to suppress noise while retaining the edge information of the cracks; at the same time, the contrast of the image is enhanced through histogram equalization and contrast stretching algorithms, making the cracks more prominent in the image and easier to identify.

[0028] Furthermore, the specific operation steps of the pipeline inner wall model building module are as follows:

[0029] Eliminate duplicate image data:

[0030] Use scale-invariant feature transformation or accelerated robust feature algorithm to extract feature points from each image in the image data. Feature points are representative local structural information in the image and have a certain invariance to image rotation, scaling, and illumination changes. Match feature points between adjacent images, find corresponding feature point pairs in different images by calculating the distance and similarity between feature points, use the nearest neighbor algorithm to determine the matching feature points, and set a threshold to exclude false matches.

[0031] According to the matched feature points, the images are registered using the geometric transformation model of affine transformation or perspective transformation, and the images of different perspectives are aligned in the same coordinate system so that the repeated areas can accurately correspond. In the registered images, the repeated parts are identified and removed by comparing the similarity between the images and the content of the overlapping areas. The image difference algorithm is used to calculate the difference between adjacent images and remove the repeated pixel information. For the overlapping areas, the images in the overlapping areas are evaluated to determine the best image to be retained.

[0032] When the image data is in the form of video, the motion estimation technology of the optical flow method is used to estimate the motion of the object by analyzing the motion vector of the pixel points in the image; the repeated area between adjacent frames is determined according to the motion trajectory of the endoscope in the pipeline and the optical flow information, and for the same part in consecutive frames, only the relevant image data of one frame is retained to reduce data redundancy;

[0033] Construct the inner wall model of the pipe:

[0034] According to the characteristics of the image data and the geometric shape of the inner wall of the pipeline, select a 3D reconstruction algorithm based on multi-view geometry, including binocular stereo vision or multi-view stereo vision algorithm; when the texture information contained in the image data meets the standard, use a 3D reconstruction method based on structured light; for pipeline objects, optimize the reconstruction process in combination with its geometric constraints, including using the circular cross-section feature of the pipeline to reconstruct the entire inner wall of the pipeline by analyzing and fitting images of different cross-sections;

[0035] Through the three-dimensional reconstruction algorithm, the processed image data is converted into point cloud data. The point cloud is a collection of a series of points in three-dimensional space, and each point contains spatial coordinates and color and texture information. The generated point cloud is filtered to remove noise points and abnormal points. Specifically, statistical filtering and radius filtering methods are used for filtering.

[0036] Furthermore, the specific operation steps of evaluating the image of the overlapping area in the pipeline inner wall model building module are as follows:

[0037] Image evaluation is performed by obtaining comprehensive judgments based on evaluation parameters of the image. The evaluation parameters include:

[0038] Clarity parameters: including gradient amplitude, high-frequency energy and focusing degree. The gradient of the image in the horizontal and vertical directions is calculated by using the Sobel operator, Scharr operator or Laplace operator. The image is Fourier transformed and converted to the frequency domain. The high-frequency energy is calculated to evaluate the clarity of the image. The focus evaluation function is used to measure the focusing degree of the image. Specifically, the variance method is used to calculate the variance of the grayscale value of the image pixels. The larger the variance, the more focused the image is and the clearer the details are. The variance is quantified by a value of 0-1. After the obtained gradient amplitude, high-frequency energy and focusing degree are normalized, a triangle is established with the gradient amplitude, high-frequency energy and focusing degree as the three sides of the triangle, and the area of ​​the triangle is calculated and recorded as the clarity evaluation value. This clarity evaluation value is used as the standard for measuring clarity.

[0039] Noise level parameter: by calculating the root mean square noise and signal-to-noise ratio of the image, and calculating the ratio of the root mean square noise to the signal-to-noise ratio after normalization, the noise evaluation value is obtained, and this noise evaluation value is used as the standard for measuring the noise level;

[0040] Color parameters: evaluated by color saturation and color consistency. Color saturation measures the color vividness by calculating the saturation component in the HSV color space. Color consistency is measured by counting the entropy of the color histogram. The obtained color saturation and the entropy of the color histogram are normalized and then the ratio is calculated to obtain the color evaluation value, which is used as the standard for measuring color parameters.

[0041] The obtained clarity evaluation value, noise evaluation value and color evaluation value are calibrated as qx, zp and ys respectively, and after normalization, they are entered into the following formula: To obtain the comprehensive evaluation value ZGZ, where β1, β2, and β3 are the preset weight coefficients of the clarity evaluation value, the noise evaluation value, and the color evaluation value, and the comprehensive evaluation value ZGZ is used as the standard for measuring the image selection priority evaluation;

[0042] The different comprehensive evaluation values ​​obtained from the images in the overlapping area are sorted according to their sizes, and the image with the largest comprehensive evaluation value is selected as the optimal image.

[0043] Furthermore, the execution process of the deep learning recognition module is as follows:

[0044] A convolutional neural network model for identifying cracks on the inner wall of a pipeline is constructed. The model is improved based on the classic CNN architecture and adds a convolution kernel suitable for the texture features of the inner wall image of the pipeline. In the network structure, a multi-scale feature fusion method is adopted to capture the crack features of different sizes and shapes at the same time. At the same time, an attention mechanism is introduced to enable the model to automatically focus on the area with cracks when processing images, thereby improving the accuracy of recognition.

[0045] Collect images of pipeline inner wall cracks of different types, sizes and severity as training data sets; annotate these images with information including the location, length, width and direction of the cracks; use data augmentation technology to increase the diversity of the data set; optimize the model parameters by training the customized model for a long time on a large-scale data set, and improve the model's ability to recognize pipeline inner wall cracks under various conditions.

[0046] Furthermore, the execution process of the result output module is as follows:

[0047] The crack identification results obtained by the deep learning identification module are displayed on a visual interface. On the three-dimensional model of the pipeline, the location and range of the cracks are highlighted with different colors and marks. The direction of the cracks is indicated by red lines, and the severity of the cracks is indicated by different shades of colors, so that the inspection personnel can intuitively understand the distribution of cracks on the inner wall of the pipeline.

[0048] At the same time, an inspection report is generated, which includes the number of cracks, average length, maximum width, and location coordinate information in the pipeline; these reports are saved in the form of text files or spreadsheets to provide accurate data support for subsequent pipeline maintenance and repair work.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] (1) The present invention significantly improves the automation level and accuracy of pipeline inner wall crack detection. It uses an autonomous mobile endoscope device in conjunction with stereoscopic lighting and multi-angle image acquisition to ensure that high-quality pipeline inner wall images can be obtained even in complex environments. Through the calibration, registration, noise suppression and contrast enhancement of the image preprocessing module, it provides high-quality input data for the deep learning model, thereby improving the accuracy and reliability of crack identification.

[0051] (2) The present invention automatically adjusts the illumination angle and brightness through an intelligent algorithm, effectively responding to the geometric shape and reflective characteristics of the inner wall of the pipeline, and achieving a uniform and optimized lighting effect. This illumination adjustment strategy improves the clarity and detail integrity of the image, especially in the flat area and complex shape area of ​​the inner wall of the pipeline. It can provide uniform illumination, further improving the accuracy of crack recognition. At the same time, the advanced image processing technology of the image preprocessing module, such as noise suppression and histogram equalization based on wavelet transform, enhances the crack characteristics and improves the recognition efficiency;

[0052] (3) In the present invention, the deep learning recognition module adopts an improved convolutional neural network model, which effectively captures the characteristics of cracks of different sizes and shapes through multi-scale feature fusion and attention mechanism, thereby improving the ability to recognize cracks in complex situations. The result output module displays the crack recognition results in an intuitive manner on the visual interface, and generates a detection report containing detailed crack information, providing accurate data support for pipeline maintenance and repair. This intuitive result display and detailed report generation greatly facilitate the analysis and subsequent processing of cracks on the inner wall of the pipeline by detection personnel, thereby improving work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to facilitate understanding by those skilled in the art, the present invention is further described below in conjunction with the accompanying drawings;

[0054] Figure 1 This is the overall system block diagram of the present invention. DETAILED DESCRIPTION

[0055] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0057] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0058] like Figure 1 As shown, the automatic recognition system for the inner wall cracks of the endoscope pipeline based on video deep learning includes an image acquisition module, an image preprocessing module, a pipeline inner wall model building module, a deep learning recognition module and a result output module;

[0059] The image acquisition module is used to collect images inside the pipeline using an autonomous mobile endoscope device. Its multi-wheel drive structure, adaptive rubber tracks, and high-precision sensors ensure a stable acquisition process. The stereo lighting device and multi-angle image acquisition module ensure the acquisition of high-quality, all-round images of the inner wall of the pipeline.

[0060] The image data in the pipeline is collected through an autonomously moving endoscope device. The endoscope device adopts a multi-wheel drive structure and is equipped with adaptive rubber tracks on the wheels, which can adapt to the inner walls of pipelines with different diameters and materials. The endoscope device can maintain a stable speed and posture during the crawling process to reduce the impact of vibration and shaking on the image quality. The endoscope device is equipped with a high-precision gyroscope and accelerometer to monitor its position and movement status in the pipeline in real time, and transmit this information through wireless communication;

[0061] The endoscope device is equipped with a three-dimensional lighting device, which consists of multiple LED light sources with independently controllable angles and brightness. These light sources are distributed around the endoscope device. According to the geometric shape and reflection characteristics of the inner wall of the pipeline, the lighting angle and brightness are automatically adjusted through intelligent algorithms. The lighting angle adjustment process is as follows:

[0062] Angle adjustment based on normal vector: Assume that the normal vector of the inner wall of the pipe at a certain point is The position coordinates of the endoscope device in the pipeline are (x, y, z); for each LED light source i, its initial direction vector is According to the principle of light reflection, the standard lighting angle makes the angle between the light and the normal vector perpendicular to obtain the strongest reflected light; calculate the dot product of each light source direction vector and the normal vector where θ i is the angle between the light source and the normal vector; by adjusting the angle of the light source, cosθ iApproaches 0 (i.e. the angle approaches 90°); uses the gradient descent optimization algorithm to adjust to minimize |cosθ i |, the update formula is Where α is the learning rate; Angle adjustment based on ray tracing: Considering the multiple reflections of light on the inner wall of the pipe, light is emitted from each LED light source to simulate the propagation path of light in the pipe. When the light encounters the inner wall of the pipe, the direction of the reflected light is calculated based on the relationship between the incident angle and the reflection angle. By tracing the propagation path of the light, it is analyzed which areas do not receive illumination that meets the standard or have shadows; the angle of the light source is adjusted according to the analysis results so that the light can cover the inner wall of the pipe more evenly; in areas with flat inner walls of the pipe, the angle adjustment algorithm based on the normal vector is first used to quickly adjust the angle of the light source to a direction close to perpendicular to the inner wall, providing a good foundation for illumination; when encountering curved parts, connecting parts or areas with special geometric shapes of the pipe, the ray tracing algorithm is enabled; the reflection and propagation path of light on the inner wall of complex shapes is considered more finely, so as to adjust the angle of the light source more accurately to ensure that these special areas can also receive uniform illumination; this combination method fully utilizes the advantages of the two algorithms, which not only ensures the accuracy of the basic illumination angle, but also can cope with complex situations;

[0063] The light intensity adjustment process is as follows: Brightness adjustment based on the light intensity model: According to the inverse square law of light intensity Where E is the illuminance, I is the intensity of the light source, r is the distance from the light source to the illuminated point, and θ is the angle between the light and the normal of the illuminated surface. For each point P on the inner wall of the pipe, calculate its distance r to each LED light source. i and the angle θ i Given the initial intensity I0 of each light source, calculate the theoretical illuminance at that point Set a target illuminance E target , according to the actual measured illuminance E actual (obtained through the light sensor installed on the endoscope device) and calculate the brightness adjustment coefficient Then, adjust the brightness of each light source according to I i = kI0. When encountering a curved pipe section, the distance r close to the inner wall becomes smaller, and the E calculated according to the above formula will increase. At this time, the brightness I of the light source on this side is reduced (i.e., k is reduced) to maintain the balance of illumination; Brightness adjustment based on the image grayscale histogram: The grayscale histogram H(k) of the image captured by the camera is obtained in real time, where k is the grayscale level. Calculate the mean of the grayscale histogram and standard deviation Set the target mean μ target and standard deviation σ target , calculate the brightness adjustment parameters according to the current μ and σ, specifically using the linear mapping function Lnew =aL old +b, where L old is the brightness of the original image, L new is the adjusted brightness, a and b are based on μ, σ, μ target and σ target The calculated coefficient; the overall brightness of the image is changed by adjusting the brightness parameter, and at the same time, it is fed back to the LED light source control system to adjust the brightness of the light source to obtain suitable lighting conditions; for the curved pipe part, the brightness of the light source at the corresponding position is adjusted in a targeted manner according to the changes in the grayscale histogram of different areas in the image; in the initial stage, the basic brightness of the light source is set according to the illumination model, and the brightness of each LED light source is preliminarily determined according to the inverse square law of the illumination by calculating the distance and angle from the light source to each point on the inner wall of the pipe, so that the entire inner wall of the pipe has a uniform illumination basis at a macro level; then, during the image acquisition process, the grayscale histogram of the image captured by the camera is analyzed in real time. When it is found that the overall brightness or local brightness of the image does not meet expectations, a brightness adjustment method based on the image grayscale histogram is used for fine-tuning; this combination can be dynamically optimized based on the visual effect of the actual captured image on the basis of considering the physical illumination model, ensuring that images with good clarity and detail integrity can be obtained under different pipeline conditions.

[0064] A multi-angle image acquisition module is constructed, including multiple high-resolution, high-frame-rate cameras. These cameras are installed at different positions and angles of the endoscope device and can simultaneously capture images of the inner wall of the pipeline from different perspectives. The camera lens adopts a special optical design with the characteristics of large depth of field and low distortion, which can clearly capture images of the inner wall of the pipeline from close to long distances. At the same time, the camera's acquisition frame rate is adaptively adjusted according to the pipeline crawling speed and detection requirements, up to 120 frames / second, to ensure that no possible crack information is missed.

[0065] The image preprocessing module calibrates and registers the collected multi-angle images to ensure geometric accuracy, while suppressing noise and enhancing contrast, providing high-quality input data for subsequent deep learning models;

[0066] Image calibration and registration: Calibrate and register images collected from multiple angles. Using the known camera installation position and angle information, the images from different perspectives are mapped into a unified coordinate system through a geometric transformation algorithm. At the same time, image deformation caused by factors such as lens distortion is corrected to ensure the geometric accuracy of the image and provide accurate input data for subsequent deep learning models.

[0067] Noise suppression and enhancement: Image noise suppression technology is used, combined with the characteristics of the pipeline inner wall image, to effectively remove the noise in the image; a method based on wavelet transform and local adaptive threshold is used to suppress noise while retaining the edge information of the cracks; in addition, the contrast of the image is enhanced through histogram equalization and contrast stretching algorithms, making the cracks more prominent in the image and easier to identify.

[0068] The pipeline inner wall model building module eliminates duplicate parts of image data through feature point extraction and matching, image registration, etc., selects the optimal image based on the evaluation parameters, and then uses a suitable algorithm to convert the processed image into point cloud data. After filtering, surface reconstruction and texture mapping, a model that can accurately reflect the actual appearance of the pipeline inner wall is constructed; the specific process is as follows:

[0069] Eliminate duplicate parts of image data: Use the scale-invariant feature transform (SIFT) or speeded up robust feature (SURF) algorithm to extract feature points from each image in the image data. These feature points are representative local structural information in the image, and have a certain invariance to the rotation, scaling, and illumination changes of the image; match feature points between adjacent images, find the corresponding feature point pairs in different images by calculating the distance and similarity between feature points, use the nearest neighbor algorithm to determine the matched feature points, and set a threshold to exclude incorrect matches; based on the matched feature points, use the geometric transformation model of affine transformation or perspective transformation to register the images. The purpose of registration is to align images of different perspectives in the same coordinate system so that the duplicate areas can accurately correspond; in the registered images, identify and remove duplicate parts by comparing the similarity between images and the content of the overlapping areas; use the image difference algorithm to calculate the difference between adjacent images and remove duplicate pixel information; for overlapping areas, evaluate the images in the overlapping areas and determine that the best image is retained. Image evaluation is comprehensively judged by obtaining evaluation parameters of the image. The evaluation parameters include:

[0070] Clarity parameters: including gradient amplitude, high-frequency energy and focusing degree. The gradient of the image in the horizontal and vertical directions is calculated by using the Sobel operator, Scharr operator or Laplace operator. The image is Fourier transformed and converted into the frequency domain. The high-frequency energy is calculated to evaluate the clarity of the image. The focus evaluation function is used to measure the focusing degree of the image. Specifically, the variance method is used to calculate the variance of the grayscale value of the image pixels. The larger the variance, the more focused the image is and the clearer the details are. The variance is quantified by a value of 0-1. After the obtained gradient amplitude, high-frequency energy and focusing degree are normalized, a triangle is established with the gradient amplitude, high-frequency energy and focusing degree as the three sides of the triangle, and the area of ​​the triangle is calculated and recorded as the clarity evaluation value. As a standard for measuring clarity; Noise level parameters: by calculating the root mean square noise and signal-to-noise ratio of the image, the ratio of the root mean square noise to the signal-to-noise ratio is calculated after normalization to obtain the noise evaluation value, and this noise evaluation value is used as a standard for measuring the noise level; Color parameters: evaluated by color saturation and color consistency, color saturation is measured by calculating the saturation component in the HSV color space to measure the color vividness; Color consistency is measured by counting the entropy of the color histogram; the obtained color saturation and the entropy of the color histogram are normalized and the ratio is calculated to obtain the color evaluation value, and this color evaluation value is used as a standard for measuring color parameters; The obtained clarity evaluation value, noise evaluation value and color evaluation value are calibrated as qx, zp and ys respectively, and after normalization, they are inserted into the following formula: To obtain the comprehensive valuation ZGZ, where β1, β2, and β3 are the preset weight coefficients of the clarity evaluation value, noise evaluation value, and color evaluation value, respectively, and the obtained comprehensive valuation ZGZ is used as the criterion for measuring the image selection priority evaluation; the different comprehensive valuations obtained from the images in the overlapping area are sorted according to size, and the image with the largest comprehensive valuation is selected as the optimal image.

[0071] When the image data is in video format, the optical flow motion estimation technology is used to estimate the motion of the object by analyzing the motion vectors of the pixels in the image. The repeated areas between adjacent frames are determined based on the motion trajectory of the endoscope in the pipe and the optical flow information. For the same parts in consecutive frames, only the relevant image data of one frame is retained, thereby reducing data redundancy.

[0072] Construct the inner wall model of the pipeline: According to the characteristics of the image data and the geometric shape of the inner wall of the pipeline, select a 3D reconstruction algorithm based on multi-view geometry, including binocular stereo vision or multi-eye stereo vision algorithm; if the image data contains enough texture information, use a 3D reconstruction method based on structured light; for pipeline objects, optimize the reconstruction process in combination with its geometric constraints, including using the circular cross-sectional features of the pipeline to reconstruct the entire inner wall of the pipeline by analyzing and fitting images of different cross-sections; convert the processed image data into point cloud data through the 3D reconstruction algorithm. The point cloud is a collection of a series of points in three-dimensional space, each of which contains spatial coordinates and color and texture information; filter the generated point cloud to remove noise points and abnormal points; specific use Statistical filtering, radius filtering and other methods; surface reconstruction algorithms, including Poisson surface reconstruction or moving least squares surface reconstruction, are used to convert point cloud data into a continuous surface model. These algorithms fit a smooth surface to represent the inner wall of the pipe according to the distribution of the point cloud; during the surface reconstruction process, the connectivity and smoothness of each area of ​​the inner wall of the pipe are analyzed to ensure that the model can accurately reflect the actual appearance of the inner wall of the pipe, and the smoothness of the surface and the degree of detail retention are controlled by adjusting the parameters of the algorithm; finally, texture mapping is performed on the generated inner wall model of the pipe, and the texture information in the original image is mapped to the reconstructed surface model, so that the model can more realistically present the specific appearance of each area of ​​the inner wall of the pipe, including color, texture and other detailed features.

[0073] The deep learning recognition module is used to build an improved convolutional neural network model, which is trained with a large number of labeled data sets to enable it to accurately identify cracks on the inner wall of the pipeline and improve the ability to identify cracks in different situations;

[0074] A convolutional neural network (CNN) model for identifying cracks on the inner wall of a pipeline is constructed. The model is improved based on the classic CNN architecture and adds convolution kernels suitable for the texture features of the inner wall image of the pipeline. In the network structure, a multi-scale feature fusion method is adopted to simultaneously capture crack features of different sizes and shapes. At the same time, an attention mechanism is introduced to enable the model to automatically focus on areas where cracks may exist when processing images, thereby improving the accuracy of recognition. A large number of images of cracks on the inner wall of pipelines of different types, sizes, and severity are collected as training data sets. These images are annotated, and the annotated information includes the location, length, width, and direction of the cracks. Data augmentation techniques such as rotation, flipping, and scaling are used to increase the diversity of the data set. The customized CNN model is trained for a long time on a large-scale data set to optimize the parameters of the model and improve the model's ability to recognize cracks on the inner wall of pipelines under various conditions.

[0075] The result output module displays the crack identification results in a visual interface, highlights the crack location and range with different colors and marks, and generates a test report containing detailed crack information to provide data support for pipeline maintenance and repair;

[0076] The crack identification results obtained by the deep learning recognition module are displayed on the visual interface. On the three-dimensional model of the pipeline, the location and range of the cracks are highlighted with different colors and marks; for example, red lines are used to indicate the direction of the cracks, and different shades of colors are used to indicate the severity of the cracks, so that the inspection personnel can intuitively understand the distribution of cracks on the inner wall of the pipeline; at the same time, a detailed inspection report is generated, including the number of cracks, average length, maximum width, position coordinates in the pipeline, and other information; these reports can be saved in the form of text files or spreadsheets to provide accurate data support for subsequent pipeline maintenance and repair work.

[0077] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. An automatic recognition system for cracks on the inner wall of an endoscope tube based on video deep learning, characterized in that: include: The image acquisition module uses an autonomous mobile endoscope device to collect images inside the pipeline, and cooperates with a stereo lighting device and a multi-angle image acquisition module to ensure the acquisition of high-quality, all-round images of the inner wall of the pipeline; The image preprocessing module calibrates and registers the collected multi-angle images to ensure geometric accuracy, suppress noise and enhance contrast, and provide high-quality input data for subsequent deep learning models; The pipeline inner wall model building module eliminates duplicate parts of image data through feature point extraction and matching, image registration, selects the optimal image based on evaluation parameters, and then uses algorithms to convert the processed image into point cloud data. After filtering, surface reconstruction and texture mapping, a model reflecting the actual appearance of the pipeline inner wall is constructed; The deep learning recognition module is used to build an improved convolutional neural network model, which can be trained with different annotated data sets to identify cracks on the inner wall of the pipeline and improve the ability to identify cracks in different situations; The result output module displays the crack identification results in a visual interface, highlights the crack location and range with different colors and marks, and generates a detection report containing detailed crack information to provide data support for pipeline maintenance and repair.

2. The automatic identification system for inner wall cracks of endoscope pipes based on video deep learning according to claim 1 is characterized in that: The specific process performed by the image acquisition module is as follows: The image data in the pipeline is collected through an autonomously moving endoscope device. The endoscope device adopts a multi-wheel drive structure and is equipped with adaptive rubber tracks on the wheels to adapt to the inner walls of pipelines with different diameters and materials. The endoscope device maintains a stable speed and posture during the crawling process. The endoscope device is equipped with a gyroscope and accelerometer to monitor its position and movement status in the pipeline in real time, and transmit this information through wireless communication. The endoscope device is equipped with a three-dimensional lighting device, which consists of multiple LED light sources with independently controllable angles and brightness. These light sources are distributed around the endoscope device and automatically adjust the lighting angle and brightness through intelligent algorithms according to the geometric shape and reflection characteristics of the inner wall of the pipeline. A multi-angle image acquisition module is constructed, including multiple cameras installed at different positions and angles of the endoscope device, to simultaneously capture images of the inner wall of the pipeline from different perspectives; the camera has the characteristics of large depth of field and low distortion, capturing images of the inner wall of the pipeline from close to long distances; at the same time, the camera's acquisition frame rate is adaptively adjusted according to the pipeline crawling speed and detection requirements.

3. The automatic identification system for inner wall cracks of endoscope pipes based on video deep learning according to claim 2 is characterized in that: The specific operation steps of automatically adjusting the illumination angle by the intelligent algorithm in the image acquisition module are as follows: Angle adjustment based on normal vector: Assume that the normal vector of the inner wall of the pipe at a certain point is The position coordinates of the endoscope device in the pipeline are (x, y, z); for each LED light source i, its initial direction vector is According to the principle of light reflection, the standard lighting angle makes the angle between the light and the normal vector perpendicular to obtain the strongest reflected light; Calculate the dot product of each light source direction vector and the normal vector where θ i is the angle between the light source and the normal vector; by adjusting the angle of the light source, cosθ i The difference from 0 is within the preset difference range; The gradient descent optimization algorithm is used to adjust to minimize |cosθ i |, the update formula is Where α is the learning rate; Angle adjustment based on ray tracing: Considering the multiple reflections of light on the inner wall of the pipe, light is emitted from each LED light source to simulate the propagation path of light in the pipe. When the light encounters the inner wall of the pipe, the direction of the reflected light is calculated based on the relationship between the incident angle and the reflection angle. By tracing the propagation path of the light, it is analyzed which areas do not receive illumination that meets the standard or have shadows. The angle of the light source is adjusted based on the analysis results so that the light can cover the inner wall of the pipe more evenly. In areas where the inner wall of the pipe is flat, the angle adjustment algorithm based on the normal vector is first used to adjust the light source angle to a direction perpendicular to the inner wall, providing a good foundation for lighting; when encountering curved parts, connecting parts or areas with geometric shapes of the pipe, the ray tracing algorithm is enabled; Considering the reflection and propagation path of light on the inner walls of complex shapes, the light source angle can be adjusted more accurately to ensure that these areas can also receive uniform lighting.

4. The automatic identification system for inner wall cracks of endoscope pipes based on video deep learning according to claim 2 is characterized in that: The specific operation steps of automatically adjusting the light brightness by intelligent algorithm in the image acquisition module are as follows: Brightness adjustment based on illumination model: according to the inverse square law of illumination Where E is the illuminance, I is the intensity of the light source, r is the distance from the light source to the illuminated point, and θ is the angle between the light and the normal of the illuminated surface; for each point P on the inner wall of the pipe, calculate its distance r to each LED light source i and the angle θ i ; Knowing the initial intensity I0 of each light source, calculate the theoretical illuminance at that point Set a target illuminance E target , according to the actual measured illuminance E actual , calculate the brightness adjustment coefficient Then, adjust the brightness of each light source according to I i = kI0; When encountering a curved pipe section, since the distance r on the side close to the inner wall becomes smaller, E calculated according to the above formula will increase. At this time, the balance of illumination can be maintained by reducing the brightness I of the light source on the side close to the inner wall; Brightness adjustment based on image grayscale histogram: obtain the grayscale histogram H(k) of the image captured by the camera in real time, where k is the grayscale level; calculate the mean of the grayscale histogram and standard deviation Set the target mean μ target and standard deviation σ target , calculate the brightness adjustment parameters according to the current μ and σ, specifically using the linear mapping function L new =aL old +b, where L old is the brightness of the original image, L new is the adjusted brightness, a and b are based on μ, σ, μ target and σ target The calculated coefficients are used to change the overall brightness of the image by adjusting the brightness parameters, and the brightness of the light source is adjusted to obtain matching lighting conditions. For the curved pipe part, the brightness of the light source at the corresponding position is adjusted in a targeted manner according to the grayscale histogram changes in different areas of the image. In the initial stage, the basic brightness of the light source is set according to the illumination model. By calculating the distance and angle from the light source to each point on the inner wall of the pipeline, the brightness of each LED light source is preliminarily determined according to the inverse square law of illumination, so that the entire inner wall of the pipeline has a uniform illumination basis at a macro level. Then, during the image acquisition process, the grayscale histogram of the image captured by the camera is analyzed in real time. When it is found that the overall brightness or local brightness of the image does not meet expectations, the brightness adjustment method based on the image grayscale histogram is used for fine-tuning. This combination is based on the consideration of the physical illumination model and is dynamically optimized according to the visual effects of the actual captured images, ensuring that images with better clarity and detail integrity can be obtained under different pipeline conditions.

5. The automatic identification system for inner wall cracks of endoscope pipes based on video deep learning according to claim 1 is characterized in that: The execution steps of the image preprocessing module are as follows: Image calibration and registration: Calibrate and register images collected from multiple angles; use the known camera installation position and angle information to map images from different perspectives into a unified coordinate system through a geometric transformation algorithm; at the same time, correct image deformation caused by lens distortion factors to ensure the geometric accuracy of the image and provide accurate input data for subsequent deep learning models; Noise suppression and enhancement: Image noise suppression technology is used, combined with the characteristics of the pipeline inner wall image, to effectively remove the noise in the image; a method based on wavelet transform and local adaptive threshold is used to suppress noise while retaining the edge information of the cracks; at the same time, the contrast of the image is enhanced through histogram equalization and contrast stretching algorithms, making the cracks more prominent in the image and easier to identify.

6. The automatic identification system for inner wall cracks of endoscope pipes based on video deep learning according to claim 1 is characterized in that: The specific operation steps of the pipeline inner wall model building module are as follows: Eliminate duplicate image data: Use scale-invariant feature transformation or accelerated robust feature algorithm to extract feature points from each image in the image data. Feature points are representative local structural information in the image and have a certain invariance to image rotation, scaling, and illumination changes. Match feature points between adjacent images, find corresponding feature point pairs in different images by calculating the distance and similarity between feature points, use the nearest neighbor algorithm to determine the matching feature points, and set a threshold to exclude false matches. Based on the matched feature points, the images are registered using the geometric transformation model of affine transformation or perspective transformation, and images of different perspectives are aligned in the same coordinate system so that the repeated areas can accurately correspond. In the registered images, the repeated parts are identified and removed by comparing the similarity between the images and the content of the overlapping areas. The image difference algorithm is used to calculate the difference between adjacent images and remove the repeated pixel information. For the overlapping area, the images in the overlapping area are evaluated to determine that the image retained is the optimal image; When the image data is in the form of video, the motion estimation technology of the optical flow method is used to estimate the motion of the object by analyzing the motion vector of the pixel points in the image; the repeated area between adjacent frames is determined according to the motion trajectory of the endoscope in the pipeline and the optical flow information, and for the same part in consecutive frames, only the relevant image data of one frame is retained to reduce data redundancy; Construct the inner wall model of the pipe: According to the characteristics of the image data and the geometric shape of the inner wall of the pipeline, select a 3D reconstruction algorithm based on multi-view geometry, including binocular stereo vision or multi-view stereo vision algorithm; when the texture information contained in the image data meets the standard, use a 3D reconstruction method based on structured light; for pipeline objects, optimize the reconstruction process in combination with its geometric constraints, including using the circular cross-section feature of the pipeline to reconstruct the entire inner wall of the pipeline by analyzing and fitting images of different cross-sections; Through the three-dimensional reconstruction algorithm, the processed image data is converted into point cloud data. The point cloud is a collection of a series of points in three-dimensional space, and each point contains spatial coordinates and color and texture information. The generated point cloud is filtered to remove noise points and abnormal points. Specifically, statistical filtering and radius filtering methods are used for filtering.

7. The automatic identification system for inner wall cracks of endoscope pipes based on video deep learning according to claim 6 is characterized in that: The specific operation steps for evaluating the image of the overlapping area in the pipeline inner wall model building module are as follows: Image evaluation is performed by obtaining comprehensive judgments based on evaluation parameters of the image. The evaluation parameters include: Clarity parameters: including gradient amplitude, high-frequency energy and focusing degree. The gradient of the image in the horizontal and vertical directions is calculated by using the Sobel operator, Scharr operator or Laplace operator. The image is Fourier transformed and converted to the frequency domain. The high-frequency energy is calculated to evaluate the clarity of the image. The focus evaluation function is used to measure the focusing degree of the image. Specifically, the variance method is used to calculate the variance of the grayscale value of the image pixels. The larger the variance, the more focused the image is and the clearer the details are. The variance is quantified by a value of 0-1. After the obtained gradient amplitude, high-frequency energy and focusing degree are normalized, a triangle is established with the gradient amplitude, high-frequency energy and focusing degree as the three sides of the triangle, and the area of ​​the triangle is calculated and recorded as the clarity evaluation value. This clarity evaluation value is used as the standard for measuring clarity. Noise level parameter: by calculating the root mean square noise and signal-to-noise ratio of the image, and calculating the ratio of the root mean square noise to the signal-to-noise ratio after normalization, the noise evaluation value is obtained, and this noise evaluation value is used as the standard for measuring the noise level; Color parameters: evaluated by color saturation and color consistency. Color saturation measures the color vividness by calculating the saturation component in the HSV color space. Color consistency is measured by counting the entropy of the color histogram. The obtained color saturation and the entropy of the color histogram are normalized and then the ratio is calculated to obtain the color evaluation value, which is used as the standard for measuring color parameters. The obtained clarity evaluation value, noise evaluation value and color evaluation value are calibrated as qx, zp and ys respectively, and after normalization, they are entered into the following formula: To obtain the comprehensive evaluation value ZGZ, where β1, β2, and β3 are the preset weight coefficients of the clarity evaluation value, the noise evaluation value, and the color evaluation value, and the comprehensive evaluation value ZGZ is used as the standard for measuring the image selection priority evaluation; The different comprehensive evaluation values ​​obtained from the images in the overlapping area are sorted according to their sizes, and the image with the largest comprehensive evaluation value is selected as the optimal image.

8. The automatic identification system for inner wall cracks of endoscope pipes based on video deep learning according to claim 1 is characterized in that: The execution process of the deep learning recognition module is as follows: A convolutional neural network model for identifying cracks on the inner wall of a pipeline is constructed. The model is improved based on the classic CNN architecture and adds a convolution kernel suitable for the texture features of the inner wall image of the pipeline. In the network structure, a multi-scale feature fusion method is adopted to capture the crack features of different sizes and shapes at the same time. At the same time, an attention mechanism is introduced to enable the model to automatically focus on the area with cracks when processing images, thereby improving the accuracy of recognition. Collect images of pipeline inner wall cracks of different types, sizes and severity as training data sets; annotate these images with information including the location, length, width and direction of the cracks; use data augmentation technology to increase the diversity of the data set; optimize the model parameters by training the customized model for a long time on a large-scale data set, and improve the model's ability to recognize pipeline inner wall cracks under various conditions.

9. The automatic identification system for inner wall cracks of endoscope pipes based on video deep learning according to claim 1 is characterized in that: The execution process of the result output module is as follows: The crack identification results obtained by the deep learning identification module are displayed on a visual interface. On the three-dimensional model of the pipeline, the location and range of the cracks are highlighted with different colors and marks. The direction of the cracks is indicated by red lines, and the severity of the cracks is indicated by different shades of colors, so that the inspection personnel can intuitively understand the distribution of cracks on the inner wall of the pipeline. At the same time, an inspection report is generated, which includes the number of cracks, average length, maximum width, and location coordinate information in the pipeline; these reports are saved in the form of text files or spreadsheets to provide accurate data support for subsequent pipeline maintenance and repair work.

Images