An endoscope fault image feature extraction method based on artificial intelligence

Abstract

The present application relates to the technical field of image processing, and more particularly to an endoscope fault image feature extraction method based on artificial intelligence, which comprises: bad point detection to determine abnormal pixels and coordinates; density ratio and cluster analysis to determine spatial distribution patterns; identification of fault types to determine the timing evolution trend; extraction of fingerprint feature vectors; prediction of development trend to determine the maintenance time window; and positioning of the physical location of the fault. The present application locates abnormal pixels through bad point detection, determines whether the bad point distribution belongs to edge priority or linear arrangement through density ratio and cluster analysis, and further determines the fault type of sealing failure or cable breakage. Meanwhile, the feature vector is extracted through the fusion of CNN and Transformer, the development trend is predicted in combination with the timing evolution to determine the maintenance window, and finally the physical location is located through coordinate mapping when the maintenance conditions are met, thereby improving the efficiency and accuracy of endoscope fault diagnosis.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an artificial intelligence-based method for extracting features from endoscopic malfunction images. Background Technology

[0002] With the widespread use of medical endoscopes, their image sensors are prone to pixel failure due to repeated bending and the seepage of disinfectant liquids, resulting in dead pixels in fixed locations. Currently, in the repair industry, the detection and correction of dead pixels are mainly achieved through threshold comparison or pixel value replacement, aiming to repair image defects rather than diagnose hardware faults. There are no applications to establish a correlation between the spatial distribution characteristics of dead pixels and equipment failure modes, which means that endoscopic fault diagnosis still faces the technical challenge of moving from symptom identification to root cause localization.

[0003] Chinese Patent Publication No. CN119485037A discloses a method for establishing an exposure correction model for endoscopic images and an image correction method. The method includes: establishing an exposure correction model, which includes a first convolutional module for capturing initial feature information of the input image, multiple exposure correction modules for exposure correction processing, and a second convolutional module for restoring the channel number of the final feature map obtained by the multiple exposure correction modules. The convolutional kernels of the first and second convolutional modules are both 3×3. The multiple exposure correction modules are connected sequentially, and the output of the previous exposure correction module is used as the input of the next exposure correction module. Each exposure correction module includes a first normalization module for normalizing the input image, a multi-scale 2D selective scanning module for extracting features in four directions from the normalized input image, a second normalization module for normalizing the feature extracted image, and a feedforward layer network module for locally modulating the normalized feature extracted image. Adjacent exposure correction modules are connected by a connection module that performs residual connection between the modulated feature map and the input image.

[0004] Therefore, the existing technologies mentioned above still have the following shortcomings: their focus is on exposure correction and image quality enhancement of endoscopic images, with the aim of improving the quality of clinical images. They do not involve the detection and analysis of pixel-level defects in image sensors, nor do they associate the spatial distribution characteristics of defects with the types of hardware faults in the equipment, and they have not established a positioning mechanism from image coordinates to chip physical coordinates. Therefore, they cannot provide direct technical support for the diagnosis, disassembly and repair of hardware faults in endoscopic equipment. Summary of the Invention

[0005] To address this, the present invention provides an artificial intelligence-based method for extracting features from endoscope fault images, thereby overcoming the problems in existing technologies where fault type determination relies on subjective experience, it is impossible to determine whether the fault is worsening or stable, and fault location can only indicate the position on the screen but cannot correspond to the actual position inside the device.

[0006] To achieve the above objectives, this invention provides an artificial intelligence-based method for extracting features from endoscopic malfunction images, comprising: Abnormal pixels and their coordinate positions are determined based on the bad pixel detection results of the fault image, wherein the bad pixel detection results of the fault image are determined based on the fault image output by the acquired endoscope image sensor. The spatial distribution pattern of the fault image is determined based on the edge-to-center density ratio and spatial clustering analysis results, wherein the edge-to-center density ratio and spatial clustering analysis results are determined based on the coordinate positions of the abnormal pixels. The corresponding fault type is determined based on the spatial distribution pattern, and the temporal evolution trend is determined based on the changing trend of the spatial distribution pattern in fault images at multiple historical moments. Based on the fault type and the coordinate position of the abnormal pixel, a hybrid architecture combining CNN and Transformer is used to extract the fingerprint feature vector of the fault image. Based on the fingerprint feature vector and the temporal evolution trend determination result, the fault development trend prediction is determined, and the recommended maintenance time window is determined based on the fault development trend prediction. In response to the recommended maintenance time window meeting the preset maintenance conditions, the physical location of the endoscope failure is determined by combining the coordinate position of the abnormal pixel.

[0007] Furthermore, the process of determining the abnormal pixels and their coordinate locations includes: A multi-frame image sequence is determined based on several frames of fault images continuously acquired under standardized lighting conditions; Preprocessed multi-frame fault images are determined based on denoising and normalization processes performed on each frame of the image. The brightness value and color channel value of each pixel are determined based on the calculation of the pixel value sequence of each pixel position in the multi-frame fault image. Abnormal pixels are identified based on the comparison results of the brightness value and color channel value of the pixel with a preset threshold, and the coordinate position of the abnormal pixel is determined based on the position of the abnormal pixel in the image.

[0008] Furthermore, the process of determining the edge-to-center density ratio includes: The edge region and center region affiliation of each abnormal pixel in the fault image are determined based on the coordinate position of the abnormal pixel. The edge bad pixel density is determined based on the number of abnormal pixels in the edge region and the total number of pixels in the edge region, and the center bad pixel density is determined based on the number of abnormal pixels in the center region and the total number of pixels in the center region. The edge-to-center density ratio is determined based on the ratio of the edge defect density to the center defect density.

[0009] Furthermore, the process of determining the spatial clustering analysis results includes: Based on the spatial distance between the abnormal pixels, abnormal pixels with a spatial distance less than a preset distance threshold are grouped into the same connected region. The orientation of the abnormal pixels in the connected region is used to determine whether the connected region is a linear structure. In response to the connected region being a linear structure, the length of the linear bad pixel is determined based on the arrangement length of the abnormal pixels in the linear structure, and the direction of the linear bad pixel is determined based on the extension direction of the linear structure.

[0010] Furthermore, the process of determining the spatial distribution pattern includes: The edge-priority distribution pattern of the fault image is determined based on the edge-to-center density ratio. The linear arrangement pattern of the fault image is determined based on the spatial clustering analysis results; The edge-priority distribution pattern and the linear arrangement pattern are respectively used as the classification results of the spatial distribution pattern.

[0011] Furthermore, the process of determining the fault type includes: In response to the spatial distribution pattern being an edge-first distribution pattern, the fault type is determined to be seal failure with liquid infiltration and oxidation. In response to the spatial distribution pattern being a linear arrangement, the fault type is determined to be bending fatigue cable breakage.

[0012] Furthermore, the process of determining the time-series evolution trend judgment result includes: Acquire historical fault images captured by the endoscope at multiple historical moments, perform bad spot detection and spatial distribution pattern recognition on each historical fault image, and determine the historical edge to center density ratio and the length of historical linear bad spots at each historical moment; Based on the current edge-to-center density ratio and the historical edge-to-center density ratio at each historical time, the trend of the edge-to-center density ratio over time is determined; Based on the current length of linear defects and the historical lengths of linear defects at various historical moments, the trend of the length of linear defects over time is determined. Based on the changing trends of the edge-to-center density ratio and the linear defect length over time, the temporal evolution trend determination result is determined.

[0013] Furthermore, the process of extracting the fingerprint feature vector includes: Extract abnormal pixel regions from the fault image based on the coordinate positions of the abnormal pixels; The abnormal pixel region is extracted based on CNN convolutional layers to determine the local features of the abnormal pixel, which include texture features, color features and morphological features. Global features of the abnormal pixel region are extracted based on the Transformer encoder layer to determine the global features of the abnormal pixels. The global features include spatial distribution pattern, density distribution pattern and arrangement pattern. The fingerprint feature vector of the fault image is determined based on the fusion result of the local features and the global features.

[0014] Furthermore, the process of determining the predicted fault development trend includes: The current fault severity level is determined based on the failure mode corresponding to the fault type and the fingerprint feature vector, wherein the fault severity level includes mild, moderate and severe. The current fault change rate is determined based on the time-series evolution trend determination results; The fault development trend prediction is determined based on the combination of the current fault severity level and the current fault change rate.

[0015] Furthermore, the process of determining the physical location of the fault includes: In response to the predicted fault development trend as accelerated deterioration or worsening, the chip physical coordinates of the abnormal pixel are determined based on the image coordinates of the abnormal pixel and the obtained physical coordinate mapping matrix of the endoscope, wherein the physical coordinate mapping matrix is ​​a pre-calibrated geometric transformation matrix between the image coordinates and the chip physical coordinates. Based on the superposition analysis of the chip's physical coordinates and the obtained structural layout of the endoscope, the distribution position of the abnormal pixels in the chip's physical space is determined, wherein the structural layout includes a sealing structure layout and a cable routing layout. The physical location of the endoscope malfunction is determined based on the distribution location.

[0016] Compared with existing technologies, the advantages of this invention are as follows: First, this invention establishes a fault type mapping relationship based on the spatial distribution pattern of bad pixels, so that edge-priority distribution corresponds to sealing failure, liquid infiltration and oxidation, and linear arrangement corresponds to bending fatigue and cable breakage, thereby realizing automatic fault type determination; then, it uses a CNN and Transformer fusion architecture to extract local features and global spatial distribution features of abnormal areas, forming a fingerprint-like feature vector that can characterize the fault state, and predicts the fault development trend by combining the temporal evolution trend determination results; further, it determines a recommended maintenance time window based on the fault development trend, and uses the recommended maintenance time window as the execution basis for fault physical location. Only when the recommended maintenance time window meets the preset maintenance conditions is the physical location of the fault determined by combining the abnormal pixel coordinates and the endoscope structure layout, so that fault location is associated with the fault development trend, avoiding premature physical location of early stable faults and improving the effectiveness of fault location results. Attached Figure Description

[0017] Figure 1 This is a flowchart of the endoscope fault image feature extraction method based on artificial intelligence in this embodiment; Figure 2 This is the logic diagram for determining the spatial distribution pattern in this embodiment; Figure 3 This is a flowchart of the fingerprint feature vector extraction process in this embodiment; Figure 4 This is a logic diagram for fault development trend prediction and physical location in this embodiment. Detailed Implementation

[0018] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0019] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0020] Please see Figure 1 The diagram shown is a flowchart of the endoscope malfunction image feature extraction method based on artificial intelligence in this embodiment. This embodiment provides an endoscope malfunction image feature extraction method based on artificial intelligence, including: Abnormal pixels and their coordinate positions are determined based on the bad pixel detection results of the fault image, wherein the bad pixel detection results of the fault image are determined based on the fault image output by the acquired endoscope image sensor. The spatial distribution pattern of the fault image is determined based on the edge-to-center density ratio and spatial clustering analysis results, wherein the edge-to-center density ratio and spatial clustering analysis results are determined based on the coordinate positions of the abnormal pixels. The corresponding fault type is determined based on the spatial distribution pattern, and the temporal evolution trend is determined based on the changing trend of the spatial distribution pattern in fault images at multiple historical moments. Based on the fault type and the coordinate position of the abnormal pixel, a hybrid architecture combining CNN and Transformer is used to extract the fingerprint feature vector of the fault image. Based on the fingerprint feature vector and the temporal evolution trend determination result, the fault development trend prediction is determined, and the recommended maintenance time window is determined based on the fault development trend prediction. In response to the recommended maintenance time window meeting the preset maintenance conditions, the physical location of the endoscope failure is determined by combining the coordinate position of the abnormal pixel.

[0021] Specifically, this invention first establishes a fault type mapping relationship based on the spatial distribution pattern of bad pixels, so that edge-priority distribution corresponds to sealing failure, liquid seepage and oxidation, and linear arrangement corresponds to bending fatigue and cable breakage, thereby realizing automatic fault type determination. Then, a CNN and Transformer fusion architecture is used to extract local features and global spatial distribution features of abnormal areas to form a fingerprint feature vector that can characterize the fault state, and the fault development trend is predicted by combining the temporal evolution trend judgment results. Furthermore, a recommended maintenance time window is determined based on the fault development trend, and the recommended maintenance time window is used as the execution basis for fault physical location. Only when the recommended maintenance time window meets the preset maintenance conditions is the physical location of the fault determined by combining the abnormal pixel coordinates and the endoscope structure layout, so that the fault location is associated with the fault development trend, avoiding premature physical location of early stable faults and improving the effectiveness of fault location results.

[0022] In this embodiment, the process of determining abnormal pixels and their coordinate positions includes: Several frames of fault images acquired continuously under standardized lighting conditions are sized and formed into a multi-frame image sequence.

[0023] Denoising and normalization are performed on each frame of the image. The denoising process uses median filtering to eliminate noise signals in the image, and the normalization process maps the pixel values ​​of each frame of the image to a uniform numerical range, resulting in preprocessed multi-frame fault images, so that each frame of the image is consistent in pixel value scale.

[0024] For each pixel location in each frame of the preprocessed fault image, the brightness value and color channel value of the pixel in multiple frames are extracted to form a pixel value sequence. The brightness value and color channel value of the pixel location in each frame are statistically analyzed to obtain the brightness statistical value and color channel statistical value of the pixel location in multiple frames. Based on the comparison of the brightness statistics and color channel statistics of each pixel position with preset thresholds, if the brightness statistics of a pixel under dark conditions are continuously higher than the first preset threshold, it is determined to be a bright spot abnormal pixel; if the brightness statistics of a pixel under uniform lighting conditions are continuously lower than the second preset threshold, it is determined to be a dark spot abnormal pixel; if the color channel statistics of a pixel under standard white balance conditions continuously deviate from the statistics of neighboring pixels at the same position by more than the third preset threshold, it is determined to be a colored spot abnormal pixel.

[0025] In this embodiment, the first preset threshold is determined based on the distribution of brightness statistics of normal pixels in multiple frames of images under dark conditions, the second preset threshold is determined based on the distribution of brightness statistics of normal pixels in multiple frames of images under uniform illumination conditions, and the third preset threshold is determined based on the distribution of statistical values ​​of normal pixels in each color channel in multiple frames of images under standard white balance conditions.

[0026] The pixel position determined to be an abnormal pixel is recorded as the coordinate position of the abnormal pixel in the image, and the coordinate position is associated with the corresponding abnormal pixel type and stored.

[0027] Specifically, there is a correspondence between the edge-priority distribution pattern and the oxidation caused by liquid seepage due to sealing failure. After the endoscope tip sealing structure fails, liquid seeps in along the chip edge under capillary effect, directly causing preferential oxidation and corrosion of pixel units in the edge area. There is also a correspondence between the linear arrangement pattern and the fracture of bending fatigue cables. After the signal cable is fractured due to fatigue, the pixel signal transmission in the corresponding row or column is interrupted, directly forming a linear arrangement of dead pixels with fixed positions. Within the fault type range defined in this embodiment, a correspondence is established between the spatial distribution pattern and the fault type, so that the spatial distribution pattern can serve as an important basis for fault type judgment, improve the accuracy of fault type identification, and make fault type judgment no longer dependent on the subjective experience judgment of engineers, but based on the automatic judgment of abnormal pixel spatial distribution characteristics.

[0028] In this embodiment, the process of determining the edge-to-center density ratio includes: The fault image is divided into edge regions and center regions according to a preset boundary distance. Each abnormal pixel is classified into its corresponding edge region or center region based on the coordinate position of the abnormal pixel.

[0029] The total number of abnormal pixels in the edge region is counted, and the total number of abnormal pixels in the edge region is divided by the total number of pixels in the edge region to obtain the edge bad pixel density; the total number of abnormal pixels in the center region is counted, and the total number of abnormal pixels in the center region is divided by the total number of pixels in the center region to obtain the center bad pixel density.

[0030] Divide the edge defect density by the center defect density to obtain the edge to center density ratio, where a larger ratio indicates that abnormal pixels are more likely to be distributed in the edge region of the image.

[0031] Specifically, this invention divides faulty images into edge regions and center regions and separately counts the density of abnormal pixels, transforming the distribution characteristics of abnormal pixels in the image space into a quantifiable density ratio. This provides a unified numerical measurement standard for the edge tendency of abnormal pixels. The ratio is obtained by dividing the density of bad pixels at the edge by the density of bad pixels at the center, eliminating the influence of differences in image size and total number of abnormal pixels on the judgment result. It solves the problem of misjudgment that easily occurs when judging edge tendency directly based on the absolute number of abnormal pixels or subjective visual judgment. This provides a repeatable and comparable numerical input basis for judging edge-priority distribution patterns, improving the accuracy and reliability of distribution pattern judgment.

[0032] In this embodiment, the process of determining the spatial clustering analysis results includes: Based on the coordinate positions of the abnormal pixels, the spatial distance between each abnormal pixel is calculated. Abnormal pixels with a spatial distance less than a preset distance threshold are determined to be adjacent to each other, and adjacent abnormal pixels are grouped into the same connected region.

[0033] In this embodiment, the preset distance threshold is determined based on the maximum allowable spacing between adjacent abnormal pixels in the linear structure. When the spatial distance between two abnormal pixels is less than the preset distance threshold, it indicates that the two abnormal pixels belong to adjacent parts of the same linear break region in spatial location and should be classified into the same connected region; when the spatial distance between two abnormal pixels is greater than the preset distance threshold, it indicates that there is an interval between them that exceeds the allowable spacing of the linear structure, and they may belong to abnormal pixels generated by different break regions or different fault types, and should not be classified into the same connected region.

[0034] Linear fitting is performed based on the coordinate positions of each abnormal pixel in the connected region, and the angle between the fitted line and the horizontal direction is calculated. When the vertical distance from each abnormal pixel in the connected region to the fitted line is less than a preset deviation threshold, the connected region is determined to be a linear structure.

[0035] In this embodiment, the preset deviation threshold is determined based on the maximum allowable width of the linear structure. When the vertical distance from each abnormal pixel in a certain connected region to the fitted line is less than the preset deviation threshold, it indicates that the offset of the abnormal pixels in the connected region in the direction perpendicular to the fitted line is within the allowable width range of the linear structure, thus determining that the connected region is a linear structure; when the vertical distance from any abnormal pixel to the fitted line is greater than the preset deviation threshold, it indicates that the connected region has an offset in the direction perpendicular to the fitted line that exceeds the allowable width of the linear structure, thus determining that the connected region is a non-linear structure.

[0036] In response to the fact that the connected region is a linear structure, the coordinates of the endpoints along the direction of the fitted straight line in the connected region are obtained, the length of the linear defect is determined based on the distance between the endpoint coordinates, and the direction of the linear defect is determined based on the angle between the fitted straight line and the horizontal direction.

[0037] Specifically, this invention calculates the spatial distance between abnormal pixels to group neighboring abnormal pixels into the same connected region. Then, it performs linear fitting on this connected region and determines whether it is a linear structure by comparing the vertical distance with a preset deviation threshold. The preset distance threshold is consistent with the maximum allowable spacing between adjacent abnormal pixels in the linear structure, and the preset deviation threshold is consistent with the maximum allowable width of the linear structure. This ensures that the aggregation and shape determination of abnormal pixels match the actual physical characteristics of the linear structure. Simultaneously, by obtaining the endpoint coordinates of the linear structure along the fitted straight line, the length and direction of the linear defect are determined, transforming the geometric features of the linear defect into a numerical description. This provides quantitative input in both length and direction dimensions for subsequent linear arrangement pattern determination.

[0038] Please see Figure 2 As shown, this is a logic diagram for determining the spatial distribution pattern in this embodiment. In this embodiment, the process of determining the spatial distribution pattern includes: The edge-priority distribution pattern of the fault image is determined based on the comparison result of the edge-to-center density ratio and the preset ratio threshold. When the edge-to-center density ratio is greater than or equal to the preset ratio threshold, the spatial distribution pattern of the fault image is determined to be an edge-priority distribution pattern.

[0039] In this embodiment, the preset ratio threshold is determined based on the boundary between edge-preferred distribution and center distribution. When the edge-to-center density ratio is greater than or equal to the preset ratio threshold, it indicates that the abnormal pixel density in the edge region is significantly higher than that in the center region, and the degree of aggregation of abnormal pixels at the image edge has reached the minimum requirement for determining an edge-preferred distribution; when the edge-to-center density ratio is less than the preset ratio threshold, it indicates that the difference in abnormal pixel density between the edge region and the center region is insufficient to constitute an edge-preferred distribution.

[0040] Based on the spatial clustering analysis results, the linear arrangement pattern of the fault image is determined. When there is a linear structure in the spatial clustering analysis results where the length of the linear bad points is greater than or equal to a preset length threshold, the spatial distribution pattern of the fault image is determined to be a linear arrangement pattern.

[0041] In this embodiment, the preset length threshold is determined based on the boundary between linear arrangement and random dispersion. When the length of the linear defective pixel is greater than or equal to the preset length threshold, it indicates that the length of the abnormal pixel continuously extending along the fitted straight line direction within the connected region has reached the minimum requirement for determining a linear arrangement; when the length of the linear defective pixel is less than the preset length threshold, it indicates that the extension length of the abnormal pixel in the fitted straight line direction is insufficient and cannot form a linear arrangement.

[0042] The edge-priority distribution pattern and the linear arrangement pattern are respectively used as the classification results of the spatial distribution pattern. The edge-priority distribution pattern and the linear arrangement pattern of the fault image constitute the spatial distribution pattern. When the spatial distribution pattern is determined to be either the edge-priority distribution pattern or the linear arrangement pattern, the spatial distribution pattern of the fault image is uniquely determined.

[0043] Specifically, this invention determines the edge-priority distribution pattern based on the comparison results of the edge-to-center density ratio and a preset ratio threshold, and determines the linear arrangement pattern based on the comparison results of the linear bad pixel length and a preset length threshold. This allows the spatial distribution characteristics of the two types of abnormal pixels to be determined by independent quantification indicators, and then merges the two determination results into a unified spatial distribution pattern. The edge-priority distribution pattern and the linear arrangement pattern are mutually exclusive. Within any control cycle, the spatial distribution pattern of the fault image is uniquely determined to be one of the two patterns, providing a definite classification input for the subsequent determination of the fault type.

[0044] In this embodiment, the process of determining the fault type includes: In response to the spatial distribution pattern being an edge-priority distribution pattern, the fault type is determined to be sealing failure due to liquid infiltration and oxidation based on the correspondence between the edge-priority distribution pattern and the sealing failure liquid infiltration and oxidation. In the edge-priority distribution pattern, the density of abnormal pixels in the image edge region is significantly higher than that in the center region. After the endoscope tip sealing structure fails, liquid seeps in along the chip edge under the action of capillary effect, causing the pixel units in the edge region to undergo preferential oxidation and corrosion. Therefore, there is a correspondence between the edge-priority distribution pattern and sealing failure liquid infiltration and oxidation.

[0045] In response to the spatial distribution pattern being a linear arrangement pattern, the fault type is determined to be a bending fatigue cable breakage based on the correspondence between the linear arrangement pattern and bending fatigue cable breakage. In the linear arrangement pattern, abnormal pixels form continuous horizontal or vertical line structures in the image. When the signal cable connecting the image sensor experiences fatigue fracture due to repeated bending at the endoscope's bending section, the pixel signal transmission in the corresponding row or column is interrupted, forming a fixed-position linear bad pixel arrangement. Therefore, there is a correspondence between the linear arrangement pattern and bending fatigue cable breakage.

[0046] Specifically, this invention establishes a pre-defined correspondence between spatial distribution patterns and fault types, allowing fault type determination to be completed directly based on the spatial distribution pattern's judgment result. This eliminates the need to re-analyze the distribution characteristics of abnormal pixels when determining the fault type. The correspondence between the edge-priority distribution pattern and the oxidation caused by liquid seepage due to sealing failure is determined by the physical process of liquid seeping along the chip edge. The correspondence between the linear arrangement pattern and the bending fatigue cable breakage is determined by the physical process of the entire row or column of pixels failing after the signal cable breaks. Once the spatial distribution pattern is determined, the corresponding fault type is uniquely locked, and there is no possibility of the same spatial distribution pattern corresponding to multiple fault types.

[0047] In this embodiment, the process of determining the temporal evolution trend determination result includes: Based on the acquired historical fault images collected by the endoscope at multiple historical moments, bad spot detection and spatial distribution pattern recognition are performed on each historical fault image to determine the historical edge-to-center density ratio and the length of historical linear bad spots at each historical moment.

[0048] Based on the current edge-to-center density ratio and the historical edge-to-center density ratio at each historical moment, an edge-to-center density ratio change sequence is formed in chronological order. Based on the change sequence, the trend of the edge-to-center density ratio over time is determined. When the edge-to-center density ratio decreases over time and the number of abnormal pixels in the central region increases synchronously, it is determined to be a trend of edge oxidation expanding into the central region. When the edge-to-center density ratio increases over time, it is determined to be a trend of increasing abnormal proportion in the edge region. When the edge-to-center density ratio remains basically unchanged over time, it is determined to be a stable trend.

[0049] Based on the comparison between the current linear defect length and the historical linear defect lengths at various historical moments, the trend of the linear defect length over time is determined. If the linear defect length shows an increasing trend over time, it is determined that the cable breakage range is expanding. If the linear defect length remains essentially unchanged over time, it is determined that the cable breakage is in a stable period. If multiple new linear structures appear in the linear defect length change sequence, it is determined that multiple signal cables have a new breakage trend. Here, "remains essentially unchanged" means that the change in corresponding parameters between adjacent historical moments is less than a preset change threshold. The preset change threshold is determined based on the allowable fluctuation range of corresponding parameters of the endoscope image sensor under normal operating conditions.

[0050] Specifically, this invention continuously analyzes fault images from multiple historical moments, transforming the spatial distribution features obtained from a single detection into evolutionary features that change over time. The edge-to-center density ratio reflects the expansion process of liquid seeping into the oxidation area due to sealing failure, while the length of linear defects reflects the expansion process of the fracture range of bending fatigue cables. By analyzing the direction and amplitude of change of these two types of features over time, the fault evolution state can be further extended from static fault identification to dynamic trend judgment, providing continuous temporal feature input for subsequent fault development trend prediction.

[0051] Please see Figure 3 The diagram shown is a flowchart of the fingerprint feature vector extraction process in this embodiment. In this embodiment, the process of extracting the fingerprint feature vector includes: Based on the coordinates of the abnormal pixels, the smallest rectangular region containing all the abnormal pixels is cropped from the fault image as the abnormal pixel region.

[0052] The abnormal pixel region is extracted using CNN convolutional layers. Multiple convolutional kernels are used to perform sliding window convolution operations within the abnormal pixel region to extract the texture, color, and morphological features of the abnormal pixels as local features of the abnormal pixels.

[0053] Global feature extraction is performed on the abnormal pixel region based on the Transformer encoder layer. The abnormal pixel region is divided into multiple image blocks. The dependency relationship between each image block is calculated through a multi-head self-attention mechanism. The spatial distribution pattern, density distribution law and arrangement law of the abnormal pixels in the whole image are extracted as global features of the abnormal pixels.

[0054] Based on the fusion of the local features and the global features, a fingerprint-like feature vector of the fault image is generated after feature splicing and dimensionality reduction.

[0055] In this embodiment, the process of constructing and performing feature extraction using the hybrid architecture of CNN and Transformer includes: First, taking the cropped abnormal pixel region as input, the abnormal pixel region is uniformly scaled to a preset input size according to network input requirements; the CNN branch uses a convolutional neural network with residual connections as the backbone network, removes the terminal classification layer, and sequentially inputs the abnormal pixel region into multiple cascaded convolution stages. Each convolution stage extracts local spatial features under different receptive fields through convolution operations and residual connections, and obtains local feature vectors based on the pooling processing of the feature maps output by the intermediate layers; the Transformer branch divides the same abnormal pixel region into multiple image blocks, flattens each image block, and obtains the corresponding block embedding vector through linear mapping, while superimposing position encoding to preserve the relationships between each image block. Spatial location information is used to form a block embedding sequence, which is then input into a Transformer encoder. The Transformer encoder calculates the correlation between different image blocks through a multi-head self-attention mechanism and performs feature transformation through a feedforward network. Combined with residual connections and layer normalization, it outputs a global feature vector. During fusion, the local feature vector and the global feature vector are concatenated along the feature dimension to obtain a fused feature vector. The fused feature vector is then subjected to feature compression and mapping to output the fingerprint feature vector. The local feature vector corresponds to the texture change features in the neighborhood of abnormal pixels, and the global feature vector corresponds to the spatial correlation features between abnormal pixels. By jointly expressing the two, the fused feature vector can simultaneously characterize the local damage features in the bad pixel formation process and the spatial propagation features caused by structural failure.

[0056] In this embodiment, the CNN backbone network and Transformer encoder are initialized with weights pre-trained on a large-scale natural image dataset. The entire network is fine-tuned end-to-end on an endoscope fault image dataset, which contains various fault types and corresponding severity labels labeled by professional engineers. During the fine-tuning training, a gradient-based optimization algorithm is used to jointly optimize all network parameters, and the learning rate is dynamically adjusted in conjunction with a learning rate scheduling strategy until the loss function value of the model on the validation set converges, at which point the training is terminated. After training, the network parameters are frozen, the abnormal pixel region to be tested is input into the network, and the fingerprint feature vector is output after forward propagation.

[0057] Specifically, this invention uses the smallest rectangular region containing the abnormal pixel as the target region for feature extraction, narrowing the computational scope of CNN and Transformer from the entire image to the abnormal pixel region, thus avoiding interference from irrelevant background regions. The abnormal pixel region simultaneously includes the abnormal pixel and its surrounding neighboring pixels, providing a unified spatial range for both local and global feature extraction. The CNN convolutional layer extracts texture, color, and morphological features of the abnormal pixel as local features to reflect the microscopic morphological information of the abnormal pixel within the local image region. Simultaneously, the Transformer encoder layer divides the abnormal pixel region into multiple image blocks and calculates the dependencies between these blocks, extracting the spatial distribution pattern, density distribution, and arrangement pattern of the abnormal pixel as global features to reflect the macroscopic distribution information of the abnormal pixel within the image range. This allows local and global features to describe fault characteristics from both microscopic and macroscopic dimensions. Finally, the local and global features are concatenated and fused, and dimensionality reduced to generate a fingerprint feature vector. This fingerprint feature vector simultaneously encodes local microscopic morphological information and global macroscopic distribution information, providing a complete feature input for subsequent fault development trend prediction.

[0058] Please see Figure 4 As shown, this is a logic diagram for fault development trend prediction and physical location in this embodiment. In this embodiment, the process of determining the fault development trend prediction includes: The severity level of the current fault is determined based on the failure mode corresponding to the fault type and the fingerprint feature vector. Specifically, for liquid seepage and oxidation due to sealing failure, the degree of oxidation is determined based on the edge oxidation texture features, color degradation features, and edge bad spot expansion features in the fingerprint feature vector. For cable breakage due to bending fatigue, the degree of breakage is determined based on the linear texture features, linear continuity, and linear bad spot length in the fingerprint feature vector. The failure severity is classified into mild, moderate, and severe based on the failure mode.

[0059] The current fault change rate is determined based on the changing trends of the edge-to-center density ratio and the linear defect length over time, as determined by the temporal evolution trend judgment results. Specifically, when the rate of decrease of the edge-to-center density ratio is greater than a first rate of change threshold or the rate of increase of the linear defect length is greater than a second rate of change threshold, it is determined to be a rapid deterioration; when the rate of decrease of the edge-to-center density ratio is less than or equal to the first rate of change threshold and is greater than zero, or the rate of increase of the linear defect length is less than or equal to the second rate of change threshold and is greater than zero, it is determined to be a slow change; and when the rate of change of the edge-to-center density ratio is zero and the rate of change of the linear defect length is zero, it is determined to be basically unchanged.

[0060] In this embodiment, the first rate of change threshold and the second rate of change threshold are determined based on the allowable amount of degradation per unit control cycle within the rated service life of the endoscope. The first rate of change threshold is a criterion for determining whether the rate of decrease in the edge-to-center density ratio belongs to rapid deterioration, and the second rate of change threshold is a criterion for determining whether the rate of increase in the length of linear dead spots belongs to rapid deterioration.

[0061] The fault development trend prediction is determined based on a combination of the current fault severity level and the current fault change rate. Specifically, when the severity level is severe and the change rate is rapidly deteriorating, or when the severity level is moderate and the change rate is rapidly deteriorating, the fault development trend prediction is accelerated deterioration. When the severity level is moderate and the change rate is slowly changing, or when the severity level is severe and the change rate is slowly changing, or when the severity level is mild and the change rate is rapidly deteriorating, the fault development trend prediction is deterioration. When the severity level is mild and the change rate is slowly changing or basically unchanged, or when the severity level is moderate and the change rate is basically unchanged, or when the severity level is severe and the change rate is basically unchanged, the fault development trend prediction is stable.

[0062] Specifically, this invention first analyzes the features reflecting the fault morphology in the fingerprint feature vector based on the failure mode corresponding to the fault type. For liquid infiltration and oxidation due to sealing failure, the focus is on analyzing edge oxidation texture features, color degradation features, and edge bad spot expansion features. For bending fatigue cable breakage, the focus is on analyzing linear texture features, linear continuity, and linear bad spot length to obtain the fault severity corresponding to different failure modes. Furthermore, the fault change rate is determined by combining the temporal evolution trend judgment results, and the fault development trend is predicted by integrating the fault severity and the fault change rate, so that the fault development trend simultaneously reflects the current failure state and the subsequent deterioration rate. Subsequently, a recommended maintenance time window is determined based on the fault development trend, and a judgment basis is provided for whether to perform subsequent physical fault location, ensuring that fault location matches the fault development state, avoiding premature physical location of faults that do not require maintenance, and improving the relevance of fault diagnosis results.

[0063] In this embodiment, the process of determining the recommended maintenance time window includes: Based on the predicted fault development trend, a corresponding recommended maintenance time window is determined. Specifically, when the predicted fault development trend is accelerating, the recommended maintenance time window is determined to be immediate maintenance; when the predicted fault development trend is worsening, the recommended maintenance time window is determined to be maintenance within a preset maintenance cycle; and when the predicted fault development trend is stable, the recommended maintenance time window is determined to be the next planned maintenance cycle.

[0064] If the recommended maintenance time window is for immediate maintenance or maintenance within a preset maintenance cycle, it is determined that the recommended maintenance time window meets the preset maintenance conditions; if the recommended maintenance time window is for the next planned maintenance cycle, it is determined that the recommended maintenance time window does not meet the preset maintenance conditions.

[0065] Specifically, by further converting the predicted fault development trend into a recommended maintenance time window, and then determining whether to enter the maintenance phase based on the recommended maintenance time window, fault location is performed only on faults that need maintenance, avoiding repeated physical location of faults that do not need maintenance, improving the efficiency of location analysis, and ensuring that maintenance decisions are consistent with fault location.

[0066] In this embodiment, the process of determining the physical location of the fault includes: In response to the recommended maintenance time window meeting preset maintenance conditions, based on the image coordinates of the abnormal pixel and the obtained physical coordinate mapping matrix of the endoscope, the image coordinates of the abnormal pixel are substituted into the physical coordinate mapping matrix for spatial transformation to obtain the chip physical coordinates of the abnormal pixel on the image sensor chip. The physical coordinate mapping matrix is ​​a pre-calibrated geometric transformation matrix between the image coordinates and the chip physical coordinates. The process of determining the physical coordinate mapping matrix includes: acquiring a calibration image containing known spatial location features collected by the endoscope image sensor; extracting the image coordinates of calibration feature points in the calibration image; and obtaining the chip physical coordinates corresponding to the calibration feature points; based on multiple calibration features... A coordinate correspondence is established between the two-dimensional coordinates of the feature points in the image coordinate system and their corresponding spatial coordinates in the chip physical coordinate system. The geometric transformation relationship between the image coordinate system and the chip physical coordinate system is determined based on the positional constraints between each calibration feature point. The physical coordinate mapping matrix is ​​obtained by solving the parameters of the geometric transformation relationship. The geometric transformation relationship describes the coordinate transformation relationship caused by optical projection, sensor installation offset, and pixel arrangement during image acquisition. The calibration feature points are formed by regularly arranged marker points set on the optical calibration plate. The optical calibration plate is located within the endoscope image acquisition range during the calibration process, so that the pixel position in the calibration image corresponds to the actual spatial position on the calibration plate.

[0067] In this embodiment, the preset maintenance conditions are determined based on the fault development trend prediction results to determine whether the current fault has reached a state that requires physical location and maintenance. When the fault development trend prediction indicates that the fault is in an accelerated deterioration state or a deterioration state, it is determined that the recommended maintenance time window meets the preset maintenance conditions. When the fault development trend prediction indicates that the fault is in a stable state, it is determined that the recommended maintenance time window does not meet the preset maintenance conditions.

[0068] Based on the superposition analysis of the chip's physical coordinates and the obtained structural layout of the endoscope, the chip's physical coordinates are mapped to the corresponding physical region in the structural layout to obtain the distribution position of the abnormal pixels in the chip's physical space. The structural layout is a two-dimensional or three-dimensional structural model pre-established based on the internal assembly structure dimensions of the endoscope, the installation position of the image sensor, and the cable routing path.

[0069] The physical location of the endoscope's fault is determined based on the distribution location. When the fault type is seal failure and liquid seepage oxidation, the distribution location corresponds to the area with the highest density of edge defects in the sealing structure layout, and the physical location of the fault is the seal failure seepage path corresponding to the area. When the fault type is bending fatigue cable breakage, the distribution location corresponds to the cable channel in the corresponding row or column in the cable routing layout, and the physical location of the fault is the breakage point location corresponding to the cable channel.

[0070] Specifically, this invention advances fault location from image coordinate space to chip physical coordinate space, solving the problem that image coordinates cannot directly correspond to the actual position inside the device. Image coordinates reflect the row and column position of pixels in the image, while chip physical coordinates reflect the actual physical position of pixel units on the sensor chip. The conversion between the two is accomplished by a physical coordinate mapping matrix. After superimposing and analyzing the structural position information provided by the sealing structure layout and cable routing layout with the chip physical coordinates, the distribution position of abnormal pixels corresponds to the sealing structure or cable channel. This allows the fault location result to match the physical operation object of disassembly and repair. Since the physical fault location is based on the premise that the recommended repair time window meets the preset repair conditions, physical location analysis is only performed on faults that have entered the repair stage. Repair engineers can identify the infiltration direction of sealing failure or the channel location of cable breakage before disassembly, without having to check the sealing area or cable channel one by one after disassembly.

[0071] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for extracting features from endoscopic malfunction images based on artificial intelligence, characterized in that, include: Abnormal pixels and their coordinate positions are determined based on the bad pixel detection results of the fault image, wherein the bad pixel detection results of the fault image are determined based on the fault image output by the acquired endoscope image sensor. The spatial distribution pattern of the fault image is determined based on the edge-to-center density ratio and spatial clustering analysis results, wherein the edge-to-center density ratio and spatial clustering analysis results are determined based on the coordinate positions of the abnormal pixels. The corresponding fault type is determined based on the spatial distribution pattern, and the temporal evolution trend is determined based on the changing trend of the spatial distribution pattern in fault images at multiple historical moments. Based on the fault type and the coordinate position of the abnormal pixel, a hybrid architecture combining CNN and Transformer is used to extract the fingerprint feature vector of the fault image. Based on the fingerprint feature vector and the temporal evolution trend determination result, the fault development trend prediction is determined, and the recommended maintenance time window is determined based on the fault development trend prediction. In response to the recommended maintenance time window meeting the preset maintenance conditions, the physical location of the endoscope failure is determined by combining the coordinate position of the abnormal pixel.

2. The method for extracting features from endoscopic malfunction images based on artificial intelligence according to claim 1, characterized in that, The process of determining abnormal pixels and their coordinates includes: A multi-frame image sequence is determined based on several frames of fault images continuously acquired under standardized lighting conditions; Preprocessed multi-frame fault images are determined based on denoising and normalization processes performed on each frame of the image. The brightness value and color channel value of each pixel are determined based on the calculation of the pixel value sequence of each pixel position in the multi-frame fault image. Abnormal pixels are identified based on the comparison results of the brightness value and color channel value of the pixel with a preset threshold, and the coordinate position of the abnormal pixel is determined based on the position of the abnormal pixel in the image.

3. The method for extracting features from endoscopic malfunction images based on artificial intelligence according to claim 1, characterized in that, The process of determining the edge-to-center density ratio includes: The edge region and center region affiliation of each abnormal pixel in the fault image are determined based on the coordinate position of the abnormal pixel. The edge bad pixel density is determined based on the number of abnormal pixels in the edge region and the total number of pixels in the edge region, and the center bad pixel density is determined based on the number of abnormal pixels in the center region and the total number of pixels in the center region. The edge-to-center density ratio is determined based on the ratio of the edge defect density to the center defect density.

4. The method for extracting features from endoscopic malfunction images based on artificial intelligence according to claim 1, characterized in that, The process of determining the spatial clustering analysis results includes: Based on the spatial distance between the abnormal pixels, abnormal pixels with a spatial distance less than a preset distance threshold are grouped into the same connected region. The orientation of the abnormal pixels in the connected region is used to determine whether the connected region is a linear structure. In response to the connected region being a linear structure, the length of the linear bad pixel is determined based on the arrangement length of the abnormal pixels in the linear structure, and the direction of the linear bad pixel is determined based on the extension direction of the linear structure.

5. The method for extracting features from endoscopic fault images based on artificial intelligence according to claim 1, characterized in that, The process of determining the spatial distribution pattern includes: The edge-priority distribution pattern of the fault image is determined based on the edge-to-center density ratio. The linear arrangement pattern of the fault image is determined based on the spatial clustering analysis results; The edge-priority distribution pattern and the linear arrangement pattern are respectively used as the classification results of the spatial distribution pattern.

6. The method for extracting features from endoscopic malfunction images based on artificial intelligence according to claim 1, characterized in that, The process of determining the fault type includes: In response to the spatial distribution pattern being an edge-first distribution pattern, the fault type is determined to be seal failure with liquid infiltration and oxidation. In response to the spatial distribution pattern being a linear arrangement, the fault type is determined to be bending fatigue cable breakage.

7. The method for extracting features from endoscopic fault images based on artificial intelligence according to claim 1, characterized in that, The process of determining the time series evolution trend judgment result includes: Acquire historical fault images of the endoscope at multiple historical moments, perform bad spot detection and spatial distribution pattern recognition on each historical fault image, and determine the historical edge to center density ratio and the length of historical linear bad spots at each historical moment. Based on the current edge-to-center density ratio and the historical edge-to-center density ratio at each historical time, the trend of the edge-to-center density ratio over time is determined; Based on the current length of linear defects and the historical lengths of linear defects at various historical moments, the trend of the length of linear defects over time is determined. Based on the changing trends of the edge-to-center density ratio and the linear defect length over time, the temporal evolution trend determination result is determined.

8. The method for extracting features from endoscopic malfunction images based on artificial intelligence according to claim 1, characterized in that, The process of extracting the fingerprint feature vector includes: Extract abnormal pixel regions from the fault image based on the coordinate positions of the abnormal pixels; The abnormal pixel region is extracted based on CNN convolutional layers to determine the local features of the abnormal pixel, which include texture features, color features and morphological features. Global features of the abnormal pixel region are extracted based on the Transformer encoder layer to determine the global features of the abnormal pixels. The global features include spatial distribution pattern, density distribution pattern and arrangement pattern. The fingerprint feature vector of the fault image is determined based on the fusion result of the local features and the global features.

9. The method for extracting features from endoscopic malfunction images based on artificial intelligence according to claim 1, characterized in that, The process of determining the predicted fault development trend includes: The current fault severity level is determined based on the failure mode corresponding to the fault type and the fingerprint feature vector, wherein the fault severity level includes mild, moderate and severe. The current fault change rate is determined based on the time-series evolution trend determination results; The fault development trend prediction is determined based on the combination of the current fault severity level and the current fault change rate.

10. The method for extracting features from endoscopic fault images based on artificial intelligence according to claim 1, characterized in that, The process of determining the physical location of the fault includes: In response to the predicted fault development trend as accelerated deterioration or worsening, the chip physical coordinates of the abnormal pixel are determined based on the image coordinates of the abnormal pixel and the obtained physical coordinate mapping matrix of the endoscope, wherein the physical coordinate mapping matrix is ​​a pre-calibrated geometric transformation matrix between the image coordinates and the chip physical coordinates. Based on the superposition analysis of the chip's physical coordinates and the obtained structural layout of the endoscope, the distribution position of the abnormal pixels in the chip's physical space is determined, wherein the structural layout includes a sealing structure layout and a cable routing layout. The physical location of the endoscope malfunction is determined based on the distribution location.

Images