Industrial failure early warning method, device, computer equipment and program product

Abstract

The application relates to the technical field of industrial computer vision, and in particular to an industrial fault early warning method and device, computer equipment and a program product. The industrial fault early warning method comprises the following steps: obtaining a focusing sequence image of a to-be-read instrument under different focusing parameters, wherein the different focusing parameters make the lens surface contaminants and the to-be-read instrument present distinguishable imaging feature differences in the focusing sequence image; determining particle pollution position information based on the imaging feature differences; performing reading processing on an image of the to-be-read instrument based on the particle pollution position information to generate a reading result, and outputting industrial fault early warning information according to the reading result. The method provided by the application can improve the reliability of the early warning information.

Description

Technical Field

[0001] This application relates to the field of industrial computer vision technology, and in particular to an industrial fault early warning method, device, computer equipment and program product. Background Technology

[0002] In industrial production, real-time monitoring of equipment operating status and early warning of anomalies are crucial for ensuring production safety and efficiency. Existing early warning methods rely on operators conducting regular inspections, recording readings from instruments such as pressure gauges, thermometers, and level gauges to determine whether the equipment is operating normally, and issuing warnings when parameters are abnormal.

[0003] With the development of automation technology, in order to be compatible with older equipment, more and more companies are beginning to use automated inspection robots to replace manual labor for equipment monitoring. These devices collect instrument images and extract readings, compare the readings with preset safety thresholds, and realize automatic early warning of equipment status.

[0004] However, existing technologies still have some problems, which make it impossible to obtain more accurate readings, resulting in low reliability of early warning information. Summary of the Invention

[0005] To solve the above-mentioned technical problems, or at least partially solve them, this application provides an industrial fault early warning method that can improve the reliability of early warning information.

[0006] The first aspect is the industrial fault early warning method, which includes the following steps: Acquire a focus sequence image of the instrument to be read under different focus parameters, wherein the different focus parameters cause the contaminants on the lens surface and the instrument to be read to present distinguishable imaging feature differences in the focus sequence image; The location information of particulate pollution is determined based on the differences in the imaging features. Based on the particulate pollution location information, the image of the instrument to be read is processed to generate reading results, and industrial fault early warning information is output according to the reading results.

[0007] Optionally, the industrial fault early warning method further includes the following steps: Aperture sequence images of the instrument to be read are acquired under different aperture parameters, wherein the different aperture parameters cause transparent contaminants to exhibit distinguishable imaging morphological differences in the aperture sequence images; The location information of transparent contamination is determined based on the differences in imaging morphology; The particulate pollution location information and the transparent pollution location information are fused in a multimodal manner to obtain pollution status information, and the corresponding processing path is executed according to the pollution status information. When the processing path includes an intervention operation, the pollution status information is reacquired after the intervention operation is performed.

[0008] Optionally, the process of processing the image of the instrument to be read based on the particulate pollution location information to generate reading results includes the following steps: The image of the instrument to be read and the reacquired pollution status information are used as inputs. Based on the pollution status information, the features at the corresponding positions in the image are weighted and suppressed. Based on the unsuppressed features and combined with the instrument structure constraints of the instrument to be read, semantic completion recognition is performed to generate the reading value. The physical confidence level corresponding to the reading value is determined based on the spatial distribution of the pollution status information. The output mode of the reading value is determined based on the physical confidence level, and the output mode includes direct adoption mode and review mode.

[0009] Optionally, the generation of the reading value is achieved through a pollution sensing and identification network, which includes the following structure: The input layer includes a first input channel and a second input channel, wherein the first input channel receives the image of the instrument to be read, and the second input channel receives the pollution status information; The feature extraction layer receives image data from the first input channel and performs multi-level feature extraction. The multi-level feature extraction includes layer-by-layer abstraction processing from low-level edge texture to high-level semantic features, and outputs a multi-level feature map. The negative attention modulation layer receives the feature map output by the feature extraction layer and the contamination status information of the second input channel. Based on the information of the second input channel, it performs negative weight modulation on the feature map output by the feature extraction layer, performs weight attenuation on the features corresponding to the contamination positions, and outputs the weight-modulated feature map. The decoding layer receives the modulated feature map output by the negative attention modulation layer, performs feature association and semantic completion through the global receptive field mechanism, and generates reading recognition results in combination with the instrument structure constraints. The output layer receives the reading identification result from the decoding layer and the physical confidence level calculated based on the pollution status information, and outputs the reading value and the physical confidence level.

[0010] Optionally, the process of determining the location information of particulate pollution based on the differences in imaging features includes the following steps: High-frequency components are extracted from the focused sequence images respectively; Calculate the differences between high-frequency components of the image under different focusing parameters, where the differences correspond to the edge features of contaminants on the lens surface; Threshold processing is applied to the differences to locate the spatial position of the pollutants; The spatial location of the pollutant is mapped to the image coordinate system used for reading by coordinate transformation, thereby generating the particulate pollution location information.

[0011] Optionally, the process of determining the location information of transparent contamination based on the imaging morphology differences includes the following steps: The highlight regions of the aperture sequence images under different aperture parameters are extracted to obtain morphological features; Calculate the morphological similarity of the highlight region at the same location under different aperture parameters; When the similarity of the morphological features is lower than a preset threshold, the highlight area is determined to be scattering interference from a transparent contaminant, and the location information of the transparent contaminant is generated. The process of multimodal fusion of the particulate pollution location information and the transparent pollution location information includes: The particulate pollution location information and the transparent pollution location information are spatially superimposed to form a fusion result.

[0012] Optionally, the process of executing the corresponding processing path based on the pollution status information includes the following steps: When the contamination characteristics of the fusion result meet the first condition, conventional reading processing is performed directly; When the contamination characteristics of the fusion result meet the second condition, a physical cleaning operation is performed, and after cleaning, contamination detection and multimodal fusion are re-performed to obtain the re-acquired contamination status information.

[0013] Secondly, this application proposes an industrial fault early warning method apparatus, the apparatus comprising at least one module for performing any of the industrial fault early warning methods described in the first aspect.

[0014] Thirdly, this application proposes a computer device including a processor for executing a computer program stored in a memory to implement the industrial fault early warning method described in any of the first aspects.

[0015] Fourthly, this application proposes a computer program product containing instructions that, when executed by a computer device, cause the computer device to perform the industrial fault early warning method as described in any of the first aspects.

[0016] The technical solution provided in this application has the following advantages compared with the prior art: One of its beneficial effects and its working principle is as follows: In industrial equipment inspection scenarios, inspection robots are deployed. These robots carry camera devices and move within the factory workshop, photographing and reading the instruments on operating equipment. During actual inspections, the camera lenses are constantly exposed to the industrial environment, making them susceptible to the accumulation of contaminants such as dust and oil mist. Dust particles in the industrial environment, after adhering to the lens surface, form diffuse spots in normal focusing mode. The edge features of these spots are often misidentified by image processing algorithms as scale lines or character strokes on a dashboard, leading to incorrect readings. Existing technologies rely on external calibration plates or manual cleaning to diagnose lens contamination, and cannot autonomously perform detection during the inspection process.

[0017] This application acquires a sequence of focused images of the instrument to be read under different focusing parameters. Utilizing the fact that contaminants on the lens surface and the instrument are at different physical distances, the two exhibit distinguishable imaging characteristics in the focused image sequence. When the focusing parameters are adjusted to the closest focusing distance, the distant instrument is blurred into a low-frequency background due to its significant deviation from the focal plane, while only the contaminants on the lens surface remain within the focal plane, exhibiting clear high-frequency edges. By extracting the high-frequency components of the images under different focusing parameters and calculating their differences, particulate contaminants attached to the surface of the optical element can be accurately located. After thresholding this difference to pinpoint the spatial location of the contaminants, coordinate transformation is used to map the spatial location of the contaminants to the image coordinate system used for reading, generating particulate contaminant location information.

[0018] The method provided in this application can autonomously identify particulate contaminants on the lens surface during shooting without the need for external reference objects. It can still complete the reading task and output industrial early warning information even when the lens is contaminated, thus improving the reliability of the warning information.

[0019] Its second beneficial effect and its working principle are as follows: When a transparent oil film adheres to the lens surface, it lacks obvious edge features and does not produce high-frequency component differences in the focused image sequence. This means that particulate contamination identification methods based on high-frequency detection cannot detect the presence of such transparent contaminants. Furthermore, the irregular scattering of incident light by the oil film layer can create localized glare or white spots in the image of the instrument being read. These scattered spots maintain relatively stable high-frequency characteristics at different focusing distances and are misinterpreted by high-frequency difference detection methods as the instrument's own scale lines or character features, thus being retained. This results in false features of the contaminated area being incorrectly treated as valid reading information, exacerbating reading errors.

[0020] This application acquires aperture sequence images of the instrument under different aperture parameters. It utilizes the phenomenon that scattered light spots from transparent contaminants undergo nonlinear morphological changes with aperture variations, enabling the transparent contaminants to exhibit distinguishable imaging morphological differences in the aperture sequence images. Highlight regions are extracted from the aperture sequence images under different aperture parameters, and their morphological features are extracted. The morphological similarity of the highlight regions at the same location under different aperture parameters is calculated. Due to normal highlights generated by ambient diffuse reflection and instrument panel specular reflection, the highlight points only undergo linear size scaling but maintain stable morphological topology when the aperture is reduced. However, oil film scattered light spots are essentially a superposition of thin-film interference and random scattering. Changes in aperture aperture cause changes in the angular distribution of incident light, leading to the recombination of interference fringes, manifested as morphological breaks, directional changes, or topological restructuring of the light spots. Therefore, when the morphological similarity is below a preset threshold, the highlight region is determined to be scattering interference from transparent contaminants, generating transparent contaminant location information.

[0021] The method provided in this application can identify transparent contaminants without edge features, thus overcoming the blind spot of non-particulate contaminant detection in particulate contaminant detection methods based on focus parameters.

[0022] Its third beneficial effect and its working principle are as follows: When the area of ​​lens contamination exceeds a preset threshold, a physical cleaning operation is triggered to reduce the degree of contamination. However, physical cleaning mechanisms, limited by their mechanical structure and the physical properties of the cleaning medium, cannot completely eliminate all contaminants from the lens surface. Particulate contamination and stubborn oil film residue in cleaning dead zones will remain. If these residual contaminants happen to obscure key reading positions on the instrument being read, existing technologies may mistakenly extract the edges of the contaminants as character strokes or scale lines during feature extraction, using these false contamination features as valid semantic features for inference, leading to incorrect output values. When contamination is detected at key locations, only an error message can be displayed, making it impossible to complete the reading under partial obstruction.

[0023] This application uses the image of the instrument to be read and the reacquired contamination status information as inputs, and processes the readings through a contamination perception and recognition network. This network includes a first input channel and a second input channel. The first input channel receives the image of the instrument to be read, and the second input channel receives the contamination status information. After multi-level feature extraction from the data in the first input channel, a negative attention modulation layer modulates the feature map output by the feature extraction layer with negative weights based on the information from the second input channel. This attenuates the weights of features corresponding to contamination locations to near zero, preventing the propagation of false contamination features. The decoding layer, based on the modulated features output by the negative attention modulation layer, performs feature association through a global receptive field mechanism. Semantic completion is performed on the unsuppressed clean area features, combined with instrument structural constraints, to generate the reading value. The output layer simultaneously outputs the reading value and physical confidence score. The physical confidence score is determined based on the spatial distribution of the contamination status information. The actual impact of contamination on the reliability of the reading is quantified by calculating the proportion of the overlap area between the contamination status information and the key reading area.

[0024] The method provided in this application can perform semantic completion and output reading values ​​based on incomplete features when the lens is partially contaminated and cannot be completely cleaned. When the contaminated area does not obscure key positions, the physical confidence level is high and the reading value can be directly adopted. When the contamination is severe and key positions are obscured, the physical confidence level decreases and triggers a manual review process, preventing erroneous readings from being directly adopted and improving the reliability of the warning information. Attached Figure Description

[0025] Figure 1 A schematic diagram illustrating a scenario for the industrial fault early warning method, apparatus, computer equipment, and program products provided in the embodiments of this application; Figure 2 One of the flowcharts for the industrial fault early warning method, apparatus, computer equipment and program product provided in the embodiments of this application; Figure 3 A second schematic flowchart illustrating the industrial fault early warning method, apparatus, computer equipment, and program product provided in the embodiments of this application; Figure 4 The third flowchart illustrates the industrial fault early warning method, apparatus, computer equipment, and program product provided in the embodiments of this application. Figure 5 This is the fourth flowchart illustrating the industrial fault early warning method, apparatus, computer equipment, and program product provided in the embodiments of this application. Detailed Implementation

[0026] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0028] Before providing a detailed explanation of the embodiments of this application, let's first introduce the application scenarios involved in the embodiments of this application.

[0029] Figure 1 This is a schematic diagram illustrating an application scenario of the industrial fault early warning method provided in this application embodiment. For example... Figure 1 As shown, an inspection robot is set up in an industrial equipment inspection scenario. The inspection robot carries a camera device and moves within the factory workshop to photograph the instruments of operating equipment. The industrial fault early warning device (the microcomputer built into the inspection robot) executes the industrial fault early warning method described in the following embodiment. Based on the images captured by the camera device, it determines the location information of lens contamination and can process the readings of the instruments to be read in the presence of contamination, thereby outputting the reading results to obtain industrial fault early warning information for inspection personnel or equipment managers to judge the equipment status and handle abnormalities.

[0030] The industrial fault early warning method provided in this application embodiment can be loaded and executed by an industrial fault early warning device, the device including at least one module, the at least one module being used to execute the industrial fault early warning method described in the following embodiments.

[0031] Reference Figure 2 As shown in the embodiments of this application, the industrial fault early warning method includes the following steps: S201: Acquire a focus sequence image of the instrument to be read under different focus parameters, wherein the different focus parameters cause the contaminants on the lens surface and the instrument to be read to present distinguishable imaging feature differences in the focus sequence image; Specifically, during robot inspection, the camera device is equipped with an adjustable focus motor, which drives the lens to switch between different focusing distances. The focusing parameter is set to the distance to the instrument plane, and the motor drives the lens to focus on the physical plane where the instrument to be read is located, acquiring the first focused image. The focusing parameter is then adjusted to the closest focusing distance, putting the lens into macro mode, and acquiring the second focused image. The two images, the first and second focused images, are combined to form a focused image sequence.

[0032] Because contaminants on the lens surface are at different physical distances from the instrument being read, when the focus parameters change, objects within the focal plane exhibit clear high-frequency characteristics, while objects deviating from the focal plane exhibit blurry low-frequency characteristics. In instrument plane focusing mode, the instrument being read is located within the focal plane and appears as a clear image, while contaminants on the lens surface, being too close to the lens, deviate from the focal plane, forming diffuse spots.

[0033] In the closest focusing distance mode, contaminants on the lens surface appear as sharp edges within the focal plane, while distant instruments are blurred into a low-frequency background due to their significant deviation from the focal plane. By switching focusing parameters, the lens surface contaminants and the instruments being read exhibit distinguishable imaging characteristics in the focused image sequence.

[0034] S202: Determine the location information of particulate pollution based on the differences in imaging features; Reference Figure 3 As shown, specifically, the process of determining the pollution location information based on the differences in imaging features includes the following steps: S301: Extract high-frequency components from the focused sequence images respectively; Specifically, for each image in the focus sequence, the Laplacian operator is used to extract its high-frequency components. The Laplacian operator extracts edge regions with drastic gray-level changes and suppresses background regions with flat gray levels by calculating the second derivative of the image. After processing, the high-frequency components of the instrument plane focus image and the closest focus distance image are obtained, respectively.

[0035] S302: Calculate the difference between high-frequency components of the image under different focusing parameters, the difference corresponding to the edge features of contaminants on the lens surface; Specifically, the two high-frequency component images extracted in step S301 are subjected to pixel-by-pixel difference calculation. For each pixel position in the image, the high-frequency component value of the closest focusing distance image is subtracted from the high-frequency component value of the instrument plane focusing image to obtain the difference image. The calculation formula is as follows: in, This represents the high-frequency components of the image at the closest focusing distance. This represents the high-frequency component of the focused image on the instrument plane, and (x, y) represents the pixel coordinates in the image. The absolute value operation is used to preserve the amplitude information of the difference.

[0036] The non-zero value regions in the difference image correspond to the high-frequency features added at the closest focusing distance. Since only lens surface contaminants are within the focal plane and exhibit clear high-frequency edges in the closest focusing distance mode, while the instrument being read is blurred into a low-frequency background, the high-frequency features retained in the difference image are the edge features of the lens surface contaminants.

[0037] S303: Perform threshold processing on the differences to locate the spatial position of the pollutants; The difference image obtained in step S302 is binarized by setting a threshold. A fixed threshold method is used, and a global threshold T is set based on the statistical characteristics of the difference image. Pixels in the difference image above the threshold are marked as contaminant locations, and pixels below the threshold are marked as background. The calculation formula for binarization is: Where M(x, y) is the binary mask image, D(x, y) is the difference image, and T is the threshold. When the difference value at a certain location is greater than the threshold, it indicates that there are significant high-frequency edge features at that location in macro mode, but no such features in instrument focusing mode. This is determined to be the presence of particulate contaminants at that location on the lens surface, and the mask value is recorded as 1. When the difference value is less than or equal to the threshold, it indicates that there are no significant differences at that location in either focusing mode. This is determined to be the absence of contaminants at that location on the lens surface, and the mask value is recorded as 0.

[0038] The generated binary mask image M(x, y) identifies the spatial distribution of particulate contaminants on the lens surface in the image coordinate system at the closest focusing distance.

[0039] S304: The spatial location of the pollutant is mapped to the image coordinate system used for reading by coordinate transformation, thereby generating the particulate pollution location information.

[0040] Specifically, because changes in focusing parameters cause changes in the field of view, there is a scale difference between the closest focusing distance image and the instrument plane focusing image. The binary mask image M(x, y) generated in step S303 identifies the position of the contaminant in the coordinate system of the closest focusing distance image, while the instrument plane focusing image is actually used for reading processing. Therefore, it is necessary to map the contaminant position to the coordinate system of the instrument plane focusing image.

[0041] Affine transformation is used to map the coordinates of the binary mask image M(x, y). The formula for calculating the affine transformation is: Where (x, y) are the pixel coordinates of the pollutant in the closest focusing distance image coordinate system, (x', y') are the pixel coordinates in the instrument plane focusing image coordinate system after mapping, and k is the scale scaling factor. and This represents the translation offset. The scaling factor k is calculated based on the lens's optical parameters and the ratio of the two focusing distances. First, a calibration is performed during the initialization phase: using a calibration board of known size, an image of the calibration board is first captured at the closest focusing distance, and the pixel distance d between two marked points on the calibration board is measured. near Next, switch the focus distance to the instrument plane focus distance and photograph the same calibration board, measuring the pixel distance d between two identical marker points. focusThe scaling factor k is calculated by the ratio of the pixel distances measured in two measurements, i.e., k = d. focus / d near .

[0042] All pixel locations marked as 1 in the binary mask image M(x, y) are mapped to the instrument plane focusing image coordinate system through the aforementioned affine transformation, generating particulate contamination location information. This particulate contamination location information is represented in pixel-level mask form, identifying which pixel locations in the instrument image used for reading are affected by lens surface particulate contamination.

[0043] S203: Acquire aperture sequence images of the instrument to be read under different aperture parameters, wherein the different aperture parameters cause transparent pollutants to exhibit distinguishable imaging morphological differences in the aperture sequence images; Specifically, after completing the particulate pollution detection, the camera maintains a constant focusing distance at the instrument plane and acquires images at different aperture parameters by adjusting the aperture mechanism. The aperture parameter is set to a large aperture (e.g., F1.8) to acquire the first aperture image; subsequently, the aperture parameter is set to a small aperture (e.g., F8.0) to acquire the second aperture image. These two aperture images are then combined to form an aperture sequence image.

[0044] When a transparent oil film adheres to the lens surface, it creates scattered light spots in the image. Changing the aperture aperture alters the geometric distribution of the incident light beam: a larger aperture results in a wider incident light beam, while a smaller aperture results in a narrower incident light beam.

[0045] For normal highlights generated by the reflection from the dashboard mirror, the reflected light path remains consistent regardless of the aperture. Only the size of the reflected light spot decreases linearly as the aperture decreases, and its outline shape remains similar. However, due to the surface inhomogeneity and random thickness distribution of the transparent oil film layer, the scattered optical path difference produced under different incident beam widths varies, causing changes in the distribution of interference fringes. This manifests as a break, distortion, or change in the outline shape or topological structure of the scattered light spot.

[0046] Therefore, by switching different aperture parameters, normal highlights maintain morphological stability in aperture sequence images, while scattered light spots generated by transparent contaminants exhibit abrupt morphological changes in aperture sequence images, forming distinguishable imaging morphological differences.

[0047] S204: Determine the location information of transparent contamination based on the imaging morphology differences; Specifically, the process of determining the location information of transparent contaminants based on the differences in imaging morphology includes the following steps: The highlight regions of the aperture sequence images under different aperture parameters are extracted to obtain morphological features; Specifically, the highlight region extraction uses a gray-level thresholding method, marking pixels in the image with gray values ​​higher than a preset highlight threshold as highlight regions. After processing, the highlight regions of the large-aperture image and the small-aperture image are obtained respectively.

[0048] Extract the morphological features of the highlight region. The highlight region is segmented using connected component analysis, and the boundary contour of each connected component is extracted. The Hu moment feature is calculated for the extracted contours.

[0049] Calculate the morphological similarity of the highlight region at the same location under different aperture parameters; Specifically, the morphological similarity is calculated using the contour matching method, and the calculation formula is as follows: in, For morphological similarity, This represents the shape distance between two highlight regions. The shape distance is calculated using the Hu moment distance formula: in, and Let i represent the i-th Hu moment invariant (i=1,2,...,7) of the highlight region contour in the large aperture image and the small aperture image, respectively. The Hu moment contains 7 invariant features and is invariant to the translation, rotation and scale changes of the contour.

[0050] Used to quantify the degree of difference between two contour shapes. When the shapes of two highlight areas are similar... Smaller Approaching 1; when the morphological differences are large, Larger Smaller.

[0051] When the similarity of the morphological features is lower than a preset threshold, the highlight area is determined to be scattering interference from a transparent contaminant, and the location information of the transparent contaminant is generated. Specifically, a morphological similarity threshold is set; if If the morphological similarity threshold is exceeded, the highlight area is determined to be caused by scattering from a transparent oil film, and the location of this area in the image is marked as the transparent contamination location. The spatial distribution of transparent contaminants in the image is represented in the form of a pixel-level mask.

[0052] S205: Perform multimodal fusion of the particulate pollution location information and the transparent pollution location information to obtain pollution status information, and execute the corresponding processing path according to the pollution status information; The process of multimodal fusion of the particulate pollution location information and the transparent pollution location information includes: The particulate pollution location information and the transparent pollution location information are spatially superimposed to form a fusion result.

[0053] Specifically, both particulate contamination location information and transparent contamination location information are represented using binary masks, where pixels marked as 1 indicate the presence of contamination, and pixels marked as 0 indicate the absence of contamination. Spatial overlay is achieved by merging the two masks pixel by pixel, using the following formula: in, For the merged pollution status information, For particulate pollution location information, This represents the location information for transparent contamination. The formula states that for each pixel location (x, y) in the image, the maximum value between two masks is taken: if the location is marked as 1 in either the particle contamination location information or the transparent contamination location information, the fusion result is marked as 1; only when the location is 0 in both masks is the fusion result 0. After spatial superposition, a comprehensive contamination status information reflecting both particle contamination and transparent contamination on the lens surface is obtained.

[0054] The process of executing the corresponding processing path based on the status information includes the following steps: When the contamination characteristics of the fusion result meet the first condition, conventional reading processing is performed directly; Specifically, the contamination characteristics of the fusion result are calculated. These characteristics are quantified by the ratio of the total number of pixels marked as contaminated in the contamination status information to the total number of pixels in the image. This ratio reflects the area of ​​contamination covering the lens surface. The calculation formula is: in, The pollution coverage is represented by W and H, which are the width and height of the image, respectively. The numerator represents the total number of pixels marked as 1, and the denominator represents the total number of pixels in the image.

[0055] The pollution characteristics are determined based on the pollution coverage rate to see if they meet preset conditions. A first condition threshold θ1 and a second condition threshold θ2 are set, where θ1 < θ2. When the pollution coverage rate... When the value is ≤θ_1, the contamination characteristics of the fusion result meet the first condition, and conventional reading processing is performed directly. At this time, the contamination area is small and has limited impact on the reading. Conventional optical character recognition algorithms can be directly applied to the instrument plane focusing image to output the reading value.

[0056] When pollution coverage When the value is greater than θ2, the contamination characteristics of the fusion result are determined to meet the second condition, and a treatment path including intervention operations is selected.

[0057] S206: When the processing path includes an intervention operation, the pollution status information is reacquired after the intervention operation is performed.

[0058] When the contamination characteristics of the fusion result meet the second condition, a physical cleaning operation is performed, and after cleaning, contamination detection and multimodal fusion are re-performed to obtain the re-acquired contamination status information.

[0059] Specifically, when step S205 determines that the contamination characteristics of the fusion result meet the second condition, i.e., the contamination coverage rate... When the value is greater than θ2, the cleaning mechanism configured on the camera device is triggered to perform a physical cleaning operation on the lens surface. The cleaning mechanism can be implemented using an air blowing device, an ultrasonic vibration device, or a mechanical wiping device.

[0060] After the physical cleaning operation is completed, the cleaning effect is assessed. If residual contamination remains on the lens surface after cleaning, the contamination detection and multimodal fusion process is repeated to obtain re-acquired contamination status information. The specific process is as follows: Steps S201 and S202 are repeated to acquire focusing sequence images under different focusing parameters, and the location information of particulate contamination after cleaning is determined based on the differences in imaging features.

[0061] Next, steps S203 and S204 are repeated to acquire aperture sequence images under different aperture parameters, and the location information of the transparent contaminant after cleaning is determined based on the differences in imaging morphology.

[0062] Finally, the location information of the cleaned particulate contamination and the location information of the cleaned transparent contamination are fused in a multimodal manner according to the spatial overlay method in step S205 to obtain the reacquired contamination status information.

[0063] S207: Based on the particulate pollution location information, perform reading processing on the image of the instrument to be read to generate reading results, and output industrial fault early warning information according to the reading results.

[0064] Reference Figure 4 As shown, specifically, the process of processing the image of the instrument to be read based on the particulate pollution location information to generate reading results includes the following steps: S401: Take the image of the instrument to be read and the reacquired pollution status information as input, perform weighted suppression on the features at the corresponding positions in the image according to the pollution status information, perform semantic completion recognition based on the unsuppressed features and combined with the instrument structure constraints of the instrument to be read, and generate reading values. S402: Determine the physical confidence level corresponding to the reading value based on the spatial distribution of the pollution status information; S403: Determine the output mode of the reading value based on the physical confidence level, wherein the output mode includes direct adoption mode and verification mode.

[0065] Reference Figure 5 As shown, specifically, steps S401, S402, and S403 above are calculated by the following steps, wherein the generated reading value is achieved through a pollution sensing and identification network, which includes the following structure: S501: Input layer, including a first input channel and a second input channel, wherein the first input channel receives the image of the instrument to be read, and the second input channel receives the pollution status information; Specifically, the input layer of this application adopts a dual-channel input architecture, enabling the network to simultaneously acquire image information and pollution status information. The first input channel receives the instrument plane focused image, which is an RGB three-channel image with dimensions denoted as W×H×3. The input image is preprocessed, normalizing the pixel values ​​from 0 to 255 to the range of 0 to 1. The second input channel receives the reacquired pollution status information M'. total (x, y) is a single-channel binary image with dimensions W×H×1. Pixels marked as 1 indicate that the location is affected by contamination, and pixels marked as 0 indicate that the location is unaffected.

[0066] The input layer passes the normalized image to the feature extraction layer for feature extraction.

[0067] S502: Feature extraction layer, receives image data from the first input channel, performs multi-level feature extraction, the multi-level feature extraction includes layer-by-layer abstraction processing from low-level edge texture to high-level semantic features, and outputs multi-level feature maps; Specifically, in the embodiments of this application, the feature extraction layer adopts a multi-level convolutional neural network structure, and the feature is abstracted layer by layer from the low level to the high level through the residual connection mechanism.

[0068] The feature extraction layer comprises four levels of feature extraction modules. The first level module receives the normalized image from the input layer and contains two convolutional layers (7×7 and 3×3 kernel sizes, 64 output channels), extracting low-level features such as edges and textures, outputting feature map F1 with spatial dimensions of W / 2×H / 2×64. The second level module contains three residual blocks (3×3 kernels, 128 output channels), extracting mid-level features such as local character shapes, outputting feature map F2 with spatial dimensions of W / 4×H / 4×128. The third level module contains four residual blocks (3×3 kernels, 256 output channels), extracting high-level semantic features such as complete character shapes and pointer outlines, outputting feature map F3 with spatial dimensions of W / 8×H / 8×256. The fourth level module contains two residual blocks (3×3 kernels, 512 output channels), extracting the highest-level semantic features such as the overall instrument layout, outputting feature map F4 with spatial dimensions of W / 16×H / 16×512.

[0069] The feature extraction layer outputs multi-level feature maps F1, F2, F3, and F4, which are then passed to the negative attention modulation layer.

[0070] S503: Negative attention modulation layer, receives the feature map output by the feature extraction layer and the contamination status information of the second input channel, performs negative weight modulation on the feature map output by the feature extraction layer according to the information of the second input channel, performs weight attenuation on the features corresponding to the contamination position, and outputs the weight-modulated feature map; Specifically, in this embodiment, two inputs are received: one is the multi-level feature maps F1, F2, F3, and F4 output by the feature extraction layer, and the other is the pollution state information M' transmitted from the input layer. total (x, y) is used for mask preprocessing.

[0071] Due to pollution status information M' total The spatial dimensions are W×H, while the spatial dimensions of the four feature maps output by the feature extraction layer are W / 2×H / 2, W / 4×H / 4, W / 8×H / 8, and W / 16×H / 16, respectively. The contamination state information needs to be downsampled to the same spatial dimensions as each feature map layer. The nearest neighbor interpolation method is used for downsampling. For each position (x) in the feature map... i y i ), find the corresponding original location (x) in the pollution status information. i ×k, y i ×k), where k is the downsampling factor, and the mask value at that position is taken. After downsampling, four contamination masks M1, M2, M3, and M4 of different sizes are obtained, which are aligned with F1, F2, F3, and F4 in terms of spatial dimensions, respectively.

[0072] Perform negative weight modulation. For feature map Fi For each position (x, y) in the image, examine the corresponding contamination mask M. i The value of (x, y). If M i (x, y) = 0, indicating that the location is uncontaminated. The characteristic value F at this location is... i (x, y, c) remain unchanged, where c represents the channel index. If M i (x, y) = 1 indicates that this location is affected by contamination, and the feature value at this location needs to be attenuated. The attenuation is calculated by multiplying the feature values ​​of all channels at this location by a weighting factor (1-α) less than 1. i ), where α i This is the attenuation coefficient. The complete modulation formula is: when When =0, (1- When (×0)=1, the eigenvalue remains unchanged; when When =1, (1- ×1)=(1- The eigenvalues ​​are decayed to (1-) times their original values. ) times. For example, when When the value is 0.9, the eigenvalue of the contaminated location is attenuated to 0.1 times its original value, which is equivalent to reducing the eigenresponse of that location by 90%.

[0073] Different feature levels exhibit varying sensitivities to contamination, thus requiring different attenuation coefficients. Shallow features (F1, F2) primarily contain local edge information, where contaminant edges are easily confused with the instrument's true edges, necessitating strong suppression of features at contamination locations. Therefore, larger preset α values ​​are used: α1 = 0.9, α2 = 0.8. Deep features (F3, F4), after multiple convolutions, already contain more global semantic information. The influence of a single contamination location is diluted by the features of surrounding pixels, exhibiting some robustness to local contamination. Therefore, smaller preset α values ​​are used: α3 = 0.6, α4 = 0.4, thereby preserving some feature responses for global semantic inference.

[0074] Each modulated feature map is normalized by calculating the mean and standard deviation of all channels in the layer and adjusting the feature values ​​to a zero-mean, unit-variance distribution to complete the normalization process.

[0075] The negative attention modulation layer outputs weighted feature maps F'1, F'2, F'3, and F'4, where the feature weights at contaminated locations have been attenuated. These output feature maps are then passed to the decoding layer for feature fusion and semantic completion.

[0076] S504: Decoding layer, receives the modulated feature map output by the negative attention modulation layer, performs feature association and semantic completion through the global receptive field mechanism, and generates reading recognition results in combination with instrument structure constraints; After negative attention modulation, the features of the contaminated location are suppressed, forming feature gaps. Therefore, by utilizing the global receptive field mechanism, and analyzing the features of the uncontaminated area and the contextual semantic information, we can perform fill-in reasoning to fill the feature gaps.

[0077] The decoding layer first performs feature fusion. A top-down fusion strategy is adopted, upsampling F'4 by a factor of 2 and concatenating it with F'3 before convolution to obtain F. fuse3 And so on, to obtain F fuse2 and F fuse1 The global semantic information contained in high-level features provides contextual guidance for low-level features.

[0078] The decoding layer employs a Transformer-based self-attention mechanism to achieve the global receptive field. The fused features are flattened into a sequence and positional encoding is added before being input into the multi-head self-attention module. The self-attention calculation formula is: Through self-attention, features from uncontaminated locations can be associated with those from contaminated locations, achieving semantic completion. For example, when a stroke of a number is occluded by contamination, the network infers the shape of the occluded stroke by analyzing other unoccluded strokes of that number and the contextual information of adjacent numbers.

[0079] The decoding layer upsamples the image to its original size W×H using three transposed convolutional layers (4×4 kernels, stride 2). The number of output channels for the final convolutional layer is determined by the instrument type: 11 channels for digital instruments (digits 0 to 9 and a decimal point), and 1 channel for pointer instruments (pointer angle regression value).

[0080] The decoding layer incorporates instrument structure constraint loss during training, requiring the predicted reading to be within the measurement range. The total loss function during training is a weighted sum of the reading recognition loss and the structure constraint loss, and the network parameters are optimized using a backpropagation algorithm. The decoding layer outputs the reading recognition result, which is then passed to the output layer.

[0081] S505: Output layer, receives the reading identification result of the decoding layer and the physical confidence level calculated based on the pollution status information, and outputs the reading value and the physical confidence level.

[0082] Specifically, firstly, the output layer converts the reading recognition result of the decoding layer into the final reading value, and compares the reading value with the normal instrument preset threshold range. When it exceeds the normal instrument preset threshold range, an industrial fault warning message is output.

[0083] Then, based on the spatial distribution of the pollution status information, the physical confidence level corresponding to the final reading is determined. First, the key reading area is determined: for digital instruments, the key reading area is the rectangular area containing the numerical characters; for pointer instruments, the key reading area is the pointer tip and the scale line it points to. The formula for calculating the physical confidence level is: in, For physical confidence level, This represents the critical reading area. The numerator indicates the number of pixels marked as contaminated within the critical reading area, and the denominator indicates the total number of pixels within the critical reading area. When the critical reading area is completely uncontaminated... =1; when partially contaminated Between 0 and 1.

[0084] The output mode of the read value is determined based on the physical confidence level. A confidence threshold is set, which in this embodiment ranges from 0.7 to 0.9, and can be adjusted according to actual conditions. When the value is greater than or equal to the confidence threshold, the direct adoption mode is adopted, outputting the reading value and confidence level, and the reading value is directly recorded and adopted.

[0085] when When the confidence level is below the threshold, a verification mode is adopted. The output reading value and confidence level are marked as low confidence value, and the output warning information is submitted to the manual verification queue. The instrument image is checked by a human to confirm the accuracy of the reading value.

[0086] In this application, the training process of the pollution perception and recognition network is as follows: A training dataset for training the pollution perception and recognition network is collected. The training dataset consists of instrument images, pollution state masks, and real reading labels. The instrument images are collected from actual industrial scenarios, the pollution state masks are generated using the methods described in steps S201 to S206, and the real reading labels are obtained by manually reading the instruments. When training the pollution perception and recognition network, the instrument images and pollution state masks in the training dataset are used as inputs, and the real reading labels in the training dataset are used as labels. The network parameters are updated using the backpropagation algorithm and the gradient descent optimizer, and the training is iteratively continued until the network converges.

[0087] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0088] In the various embodiments of this application, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments are consistent and can be referenced mutually. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships. In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. In the textual description of the embodiments of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. In this application, "first," "second," and various numerical designations are only for ease of description and are not used to limit the scope of the embodiments of this application. For example, they are used to distinguish different messages, rather than to describe a specific order or sequence.

[0089] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

[0090] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An industrial fault early warning method, characterized in that, The industrial fault early warning method includes the following steps: Acquire a focus sequence image of the instrument to be read under different focus parameters, wherein the different focus parameters cause the contaminants on the lens surface and the instrument to be read to present distinguishable imaging feature differences in the focus sequence image; The location information of particulate pollution is determined based on the differences in the imaging features. Based on the particulate pollution location information, the image of the instrument to be read is processed to generate reading results, and industrial fault early warning information is output according to the reading results.

2. The industrial fault early warning method according to claim 1, characterized in that, The industrial fault early warning method further includes the following steps: Aperture sequence images of the instrument to be read are acquired under different aperture parameters, wherein the different aperture parameters cause transparent contaminants to exhibit distinguishable imaging morphological differences in the aperture sequence images; The location information of transparent contamination is determined based on the differences in imaging morphology; The particulate pollution location information and the transparent pollution location information are fused in a multimodal manner to obtain pollution status information, and the corresponding processing path is executed according to the pollution status information. When the processing path includes an intervention operation, the pollution status information is reacquired after the intervention operation is performed.

3. The industrial fault early warning method according to claim 2, characterized in that, The process of processing the image of the instrument to be read based on the particulate pollution location information to generate reading results includes the following steps: The image of the instrument to be read and the reacquired pollution status information are used as inputs. Based on the pollution status information, the features at the corresponding positions in the image are weighted and suppressed. Based on the unsuppressed features and combined with the instrument structure constraints of the instrument to be read, semantic completion recognition is performed to generate the reading value. The physical confidence level corresponding to the reading value is determined based on the spatial distribution of the pollution status information. The output mode of the reading value is determined based on the physical confidence level, and the output mode includes direct adoption mode and review mode.

4. The industrial fault early warning method according to claim 3, characterized in that, The generated reading value is achieved through a pollution sensing and identification network, which includes the following structure: The input layer includes a first input channel and a second input channel, wherein the first input channel receives the image of the instrument to be read, and the second input channel receives the pollution status information; The feature extraction layer receives image data from the first input channel and performs multi-level feature extraction. The multi-level feature extraction includes layer-by-layer abstraction processing from low-level edge texture to high-level semantic features, and outputs a multi-level feature map. The negative attention modulation layer receives the feature map output by the feature extraction layer and the contamination status information of the second input channel. Based on the information of the second input channel, it performs negative weight modulation on the feature map output by the feature extraction layer, performs weight attenuation on the features corresponding to the contamination positions, and outputs the weight-modulated feature map. The decoding layer receives the modulated feature map output by the negative attention modulation layer, performs feature association and semantic completion through the global receptive field mechanism, and generates reading recognition results in combination with the instrument structure constraints. The output layer receives the reading identification result from the decoding layer and the physical confidence level calculated based on the pollution status information, and outputs the reading value and the physical confidence level.

5. The industrial fault early warning method according to claim 1, characterized in that, The process of determining the location information of particulate pollution based on the differences in imaging features includes the following steps: High-frequency components are extracted from the focused sequence images respectively; Calculate the differences between high-frequency components of the image under different focusing parameters, where the differences correspond to the edge features of contaminants on the lens surface; Threshold processing is applied to the differences to locate the spatial position of the pollutants; The spatial location of the pollutant is mapped to the image coordinate system used for reading by coordinate transformation, thereby generating the particulate pollution location information.

6. The industrial fault early warning method according to claim 2, characterized in that, The process of determining the location information of transparent contamination based on the imaging morphology differences includes the following steps: The highlight regions were extracted from the aperture sequence images under different aperture parameters to obtain morphological features; Calculate the morphological similarity of the highlight region at the same location under different aperture parameters; When the similarity of the morphological features is lower than a preset threshold, the highlight area is determined to be scattering interference from a transparent contaminant, and the location information of the transparent contaminant is generated. The process of multimodal fusion of the particulate pollution location information and the transparent pollution location information includes: The particulate pollution location information and the transparent pollution location information are spatially superimposed to form a fusion result.

7. The industrial fault early warning method according to claim 2, characterized in that, The process of executing the corresponding processing path based on the pollution status information includes the following steps: When the contamination characteristics of the fusion result meet the first condition, conventional reading processing is performed directly; When the contamination characteristics of the fusion result meet the second condition, a physical cleaning operation is performed, and after cleaning, contamination detection and multimodal fusion are re-performed to obtain the re-acquired contamination status information.

8. An industrial fault early warning method and device, characterized in that, The device includes at least one module for performing the industrial fault early warning method according to any one of claims 1-7.

9. A computer device, characterized in that, The computer device includes a processor for executing a computer program stored in a memory to implement the industrial fault early warning method according to any one of claims 1-7.

10. A computer program product containing instructions, characterized in that, When the instruction is executed by a computer device, the computer device performs the industrial fault early warning method as described in any one of claims 1-7.

Images