Image processing method and device, equipment, medium and fault detection method

Abstract

The invention provides an image processing method and device, equipment, a medium and a fault detection method, and the method comprises the steps: carrying out the feature extraction of a to-be-enhanced image, and obtaining the image features of a plurality of scales; performing feature fusion on the image features of the multiple scales by adopting a progressive fusion method to obtain target fusion features, the progressive fusion method being used for gradually fusing the image features of different scales from bottom to top; and obtaining an enhanced target image according to the target fusion feature. According to the method, the device, the equipment and the storage medium, the to-be-enhanced image can be processed in combination with multi-scale feature extraction and progressive feature fusion modes, and local details are enhanced while a global structure is kept, so that the quality of the image is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an image processing method, device, equipment medium and fault detection method. Background Art

[0002] Since belt conveyors are in a long-term high-speed, heavy-load, and harsh operating environment, they are prone to various faults and require real-time monitoring and fault diagnosis. However, the weak light environment makes it difficult for monitoring equipment to collect high-quality images, affecting the accuracy of fault diagnosis.

[0003] Existing low-light image enhancement methods, whether traditional algorithms or deep learning-based methods, have problems such as underexposure, color desaturation, artifacts or obvious noise, and cannot meet the task requirements of fault monitoring. Therefore, the enhanced image quality obtained by using existing low-light image enhancement methods is low. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an image processing method, device, equipment medium and fault detection method to improve image quality in view of the above-mentioned deficiencies in the prior art.

[0005] In a first aspect, the present invention provides an image processing method, the method comprising:

[0006] Perform feature extraction on the image to be enhanced to obtain image features at multiple scales;

[0007] Using a progressive fusion method to fuse the image features of the multiple scales to obtain target fusion features, wherein the progressive fusion method is used to gradually fuse the image features of different scales from bottom to top;

[0008] According to the target fusion features, an enhanced target image is obtained.

[0009] Furthermore, the image features of the multiple scales include high-resolution features, medium-resolution features and low-resolution features.

[0010] The feature extraction of the image to be tested is performed to obtain image features of multiple scales, including:

[0011] Extracting features from the image to be tested to obtain the high-resolution features;

[0012] Performing a first downsampling on the high-resolution feature to obtain the medium-resolution feature;

[0013] The medium-resolution features are downsampled for a second time to obtain the low-resolution features.

[0014] Furthermore, the step of using a progressive fusion method to fuse the image features of the multiple scales to obtain target fusion features includes:

[0015] The image features of the multiple scales are gradually fused from bottom to top through multiple feature fusion modules and multiple feature enhancement modules to obtain the target fusion feature.

[0016] Furthermore, the image features of the multiple scales are gradually fused from bottom to top through multiple feature fusion modules and multiple feature enhancement modules to obtain the target fusion features, including:

[0017] Performing a first feature enhancement and a first upsampling on the low-resolution feature to obtain a first enhanced feature;

[0018] Performing a first feature fusion on the first enhanced feature and the medium-resolution feature to obtain a first fused feature;

[0019] Performing a second feature enhancement and a second upsampling on the first fused feature to obtain a second enhanced feature;

[0020] Performing a second feature fusion on the second enhanced feature and the high-resolution feature to obtain a second fused feature;

[0021] Performing a third feature enhancement and a third upsampling on the second fused feature to obtain a third enhanced feature;

[0022] The first enhanced feature, the second enhanced feature, the third enhanced feature and the high-resolution feature are spliced ​​to obtain the target fusion feature.

[0023] Furthermore, the feature fusion module includes: a 2D convolution module, an average pooling layer, a linear layer, a ReLu activation function and a sigmoid activation function.

[0024] Furthermore, the feature enhancement module includes:

[0025] F IMRB =F IN +IMR2(IMR1(F CM )),

[0026] Among them, F IN Indicates the feature map at the beginning of IMRB, F CM Represents the feature map obtained by applying two sets of 3×3 convolutional layers and LeakyRelu activation function to the input features. IMR2(IMR1()) represents the multi-scale residual recurrent processing.

[0027] In a second aspect, the present invention provides a fault detection method for a belt conveyor, characterized in that the method comprises:

[0028] Acquiring a detection image of the belt conveyor;

[0029] Performing image enhancement processing on the detection image according to the image processing method of the first aspect to obtain an enhanced detection image;

[0030] Fault detection is performed on the belt conveyor according to the enhanced detection image.

[0031] In a third aspect, the present invention provides an image processing device, comprising:

[0032] A feature extraction module is used to extract features from the image to be enhanced and obtain image features at multiple scales;

[0033] A feature processing module, connected to the feature extraction module, for performing feature fusion on the image features of the multiple scales by using a progressive fusion method to obtain a target fusion feature, wherein the progressive fusion method is used to gradually fuse the image features of different scales from bottom to top;

[0034] The image acquisition module, the feature processing module and the feature fusion module are used to obtain an enhanced target image according to the target fusion features.

[0035] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image processing method described in the first aspect is implemented.

[0036] In a fifth aspect, the present invention provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the image processing method as described in the first aspect.

[0037] The image processing method, device, equipment, medium and fault detection method provided by the present invention obtain the details and structural information of the image to be enhanced at different scales through multi-level feature extraction; then, the target fusion feature is obtained by gradually integrating the information of different scales from bottom to top through a progressive fusion strategy, which can better retain the detail information and enhance the large-scale structure at the same time; finally, the enhanced target image is obtained according to the target fusion feature. Thus, the image to be enhanced is processed by combining the multi-scale feature extraction and progressive feature fusion methods, and the local details are enhanced while maintaining the global structure, thereby improving the quality of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1This is a flowchart of an image processing method according to Embodiment 1 of the present invention;

[0039] Figure 2 A flowchart of another image processing method according to Embodiment 1 of the present invention

[0040] Figure 3 This is a framework diagram of Embodiment 1 of the present invention;

[0041] Figure 4 This is a schematic diagram of the structure of a feature enhancement module according to Embodiment 1 of the present invention;

[0042] Figure 5 This is a schematic diagram of the structure of a feature fusion module in Example 1 of the present invention;

[0043] Figure 6 is a structural schematic diagram of an image processing device according to Embodiment 3 of the present invention;

[0044] Figure 7 This is a schematic diagram of the structure of an image processing device according to Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0046] It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention rather than to limit the present invention.

[0047] It can be understood that, in the absence of conflict, the various embodiments of the present invention and the various features in the embodiments can be combined with each other.

[0048] It can be understood that, for the convenience of description, the drawings of the present invention only show the parts related to the present invention, while the parts irrelevant to the present invention are not shown in the drawings.

[0049] It can be understood that each unit and module involved in the embodiments of the present invention may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.

[0050] It can be understood that the terms "first", "second", etc. in the embodiments of the present invention are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.

[0051] It can be understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of the present invention may occur in an order different from that marked in the drawings.

[0052] It is understood that the flowcharts and block diagrams of the present invention illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present invention. Each box in the flowchart or block diagram may represent a unit, module, program segment, or code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based system that implements the specified functions, or may be implemented by a combination of hardware and computer instructions.

[0053] It can be understood that the units and modules involved in the embodiments of the present invention can be implemented by software or hardware. For example, the units and modules can be located in a processor.

[0054] It should be noted that the scenario diagram described in the embodiment of the present application is for the purpose of more clearly illustrating the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can know that with the evolution of network architecture and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0055] Belt conveyor, also known as belt conveyor, is an important equipment for continuous material transportation. It is widely used in various industries such as mining, metallurgy, chemical industry, building materials, energy, transportation and other fields to transport various bulk materials or finished items.

[0056] Downsampling: Downsampling is the process of reducing the spatial size of an image or feature map. Upsampling: Upsampling is the process of increasing the spatial size of an image or feature map.

[0057] Belt conveyors are in high-speed, heavy-loaded operation for a long time, and are prone to belt deviation, slipping, tearing, roller damage, blockage and other faults due to harsh working environment, long transportation lines, and uneven transportation volume. Therefore, it is necessary to monitor the status of belt conveyors and carry out emergency protection to ensure their safe and reliable operation during the production process. However, due to the inevitable problems of weak light intensity in the belt conveyor transportation environment, the monitoring equipment is unable to collect high-quality images, which has a negative impact on the inspection of safety hazards of belt conveyors. Therefore, obtaining high-quality belt monitoring images through technical means is of great significance to ensuring high production, efficiency and safety in the coal industry. Compared with improving hardware equipment to capture high-quality images, software algorithms have the advantages of low cost and easy deployment.

[0058] The existing low-light image enhancement methods mainly include traditional methods such as histogram equalization and Retinex theory, as well as image enhancement methods based on deep learning. Traditional methods improve the brightness and contrast of images in a simple and efficient way by improving the pixel distribution of images. However, due to the limited prior characterization capabilities of the design, these traditional algorithms often cause problems such as underexposure, color desaturation, and obvious artifacts or noise, and cannot meet the task requirements of belt conveyor belt fault monitoring in low-light environments. The method based on deep learning mainly uses a wide range of data to train neural networks, generate network models with strong robustness and generalization, achieve end-to-end optimization of low-light images, and improve the reliability of belt conveyor fault monitoring tasks. However, most methods based on deep learning mainly use the strategy of "divide and conquer", ignoring the correlation between features of different scales. In view of the problems existing in the above prior art, the present application provides a low-light image enhancement method based on deep learning to improve the quality of images and accurately monitor the operating status of belt conveyors in low-light environments.

[0059] Based on the problems existing in the prior art, the present application provides an image processing method to improve image quality.

[0060] Embodiment 1:

[0061] This embodiment provides an image processing method, such as Figure 1 As shown, the method includes the following steps S101 to S103:

[0062] S101: Extract features from the image to be enhanced to obtain image features at multiple scales.

[0063] Images to be enhanced are those original images that are of poor quality due to various factors and need to be processed to improve their visual effects, increase recognizability, or enhance specific features. These images usually have some problems that affect their quality and usability and need to be improved through image enhancement technology. Images to be enhanced can be many types of images, especially those taken under low light or other challenging conditions. For example, in the scenario of belt conveyor monitoring, the image to be enhanced can be a conveyor belt image taken in a complex industrial environment with poor quality but containing key monitoring information.

[0064] Specifically, image processing techniques can be used to capture the features of the image at different resolutions or scales. For example: multi-resolution analysis, using filters or convolution kernels of different sizes to extract features of different scales; multi-scale convolutional neural networks, using convolution kernels of different sizes to extract multi-scale features at the same layer; scale space theory, multi-resolution geometric analysis, image pyramid methods, etc., can be selected according to actual conditions.

[0065] S102: Using a progressive fusion method to fuse image features at multiple scales to obtain target fusion features.

[0066] Among them, the progressive fusion method is used to gradually fuse image features of different scales from bottom to top.

[0067] Specifically, we start from the smallest scale (finest granularity) features and gradually fuse them to larger scale features. Thus, we gradually integrate information of different scales from bottom to top to obtain the target fusion features, which can better preserve the detail information while enhancing the large-scale structure.

[0068] Optionally, convolution fusion (using convolutional layers to learn feature fusion), pyramid pooling, adaptive feature fusion (using learnable parameters to dynamically adjust the fusion strategy), multi-layer perceptron fusion, etc. can be used. In practical applications, these methods can be used alone or in combination to achieve better results.

[0069] S103: Obtain an enhanced target image according to the target fusion features.

[0070] Specifically, it can be implemented in a variety of ways, including direct decoding network, U-Net structure, residual learning, attention mechanism, progressive generation, GAN, multi-scale fusion, conditional generation, pixel-level regression, filter-based methods, image-to-image conversion, autoencoder structure, reinforcement learning-based methods, and meta-learning methods. You can choose a suitable method or combine multiple methods according to specific task requirements, computing resources, and experimental results.

[0071] In the embodiment of the present application, the details and structural information of the image to be enhanced at different scales are obtained through multi-level feature extraction; then, the target fusion feature is obtained by gradually integrating the information of different scales from bottom to top through the progressive fusion strategy, which can better retain the detail information and enhance the large-scale structure at the same time; finally, the enhanced target image is obtained according to the target fusion feature. Thus, the image to be enhanced is processed by combining multi-scale feature extraction and progressive feature fusion, and local details are enhanced while maintaining the global structure, thereby improving the image quality. .

[0072] Optionally, the image features at multiple scales include high-resolution features, medium-resolution features, and low-resolution features. The above S101 may specifically include the following steps:

[0073] Perform feature extraction on the image to be tested to obtain high-resolution features;

[0074] Perform the first downsampling of high-resolution features to obtain medium-resolution features;

[0075] The medium-resolution features are downsampled a second time to obtain low-resolution features.

[0076] Specifically, both local details and global information of an image can be captured by extracting multi-scale features.

[0077] In one example, the expressions for performing the downsampling operation are the following formulas (1) and (2):

[0078] X middle =Down(X 浅层特征 ) (1)

[0079] X low =Down(X middle ) (2)

[0080] Among them, X middle is the feature map of medium resolution flow (medium resolution feature), X 浅层特征 is the feature map of high-resolution flow (high-resolution feature), X low It is a low-resolution feature map (low-resolution feature), and Down() is a downsampling operation.

[0081] In this embodiment, the high-resolution feature is a feature directly extracted from the original image to be enhanced, which retains the details and texture information of the image. The medium-resolution feature is obtained by performing the first downsampling of the high-resolution feature. While maintaining certain details, a part of the high-frequency information is removed. It is usually used to extract regional and medium details in the image. The low-resolution feature is obtained by performing the second downsampling of the medium-resolution feature. It is relatively rough, but it can summarize the overall structure and shape of the image. It is usually used to analyze the global information and low-frequency features of the image.

[0082] In one example, taking the inspection image of a belt conveyor as an example, high-resolution features (e.g., 2048x1536 pixels) can clearly see the fine cracks, wear and material particles on the surface of the conveyor belt, and observe the conveyor belt joints and equipment details; medium-resolution features (e.g., 1024x768 pixels) can be used to evaluate the overall status, analyze material distribution, and detect medium-scale anomalies; low-resolution features (e.g., 512x384 pixels) can be used to quickly determine the overall status, detect large-scale problems, and monitor the overall flow.

[0083] Optionally, the above S102 may specifically include:

[0084] Through multiple feature fusion modules and multiple feature enhancement modules, image features of multiple scales are gradually fused from bottom to top to obtain the target fusion features.

[0085] Among them, the feature fusion module is used to merge features of different scales. Different feature fusion modules can be selected according to actual needs, for example: multiple simple concatenations (Concatenation), which are used to directly concatenate features of different scales in the channel dimension; weighted sum, which is used to weighted sum features of different scales, and the weights can be fixed or learnable; attention mechanism, which uses the attention mechanism to dynamically adjust the importance of features of different scales; pyramid pooling module (Pyramid Pooling Module), such as the module used in PSPNet, which performs multi-scale pooling on features and then fuses them; feature pyramid network (Feature Pyramid Network, FPN), which realizes the fusion of multi-scale features by combining top-down and bottom-up paths.

[0086] Feature enhancement module: used to improve the quality and expression ability of single-scale features, or to improve the quality of fused features. You can choose different feature enhancement modules according to actual needs, for example: Residual Block: Use skip connections to enhance features and avoid the gradient vanishing problem; Dense Connection, such as the dense block in DenseNet, enhance features through dense connections; Spatial Attention Module, learns the attention weights in the spatial dimension to enhance features; Channel Attention Module, such as the Squeeze-and-Excitation module in SENet, learns the dependencies between channels; Non-local Block, captures long-distance dependencies to enhance features; Dilated Convolution, increases the receptive field without increasing the number of parameters, and is used to enhance the representation ability of features.

[0087] These modules can be combined and adjusted according to specific task requirements. In practical applications, multiple such modules may be used in series or in parallel to gradually fuse and enhance features of different scales to finally obtain the target fusion features.

[0088] Specifically, multiple feature fusion modules and feature enhancement modules are used, starting from low-resolution features, and gradually fusing image features of different scales from bottom to top, and the fused features are optimized through feature enhancement modules. It can effectively integrate multi-scale information, maintain the hierarchical structure of features, and continuously improve feature quality through multiple enhancements. It has good flexibility and computational efficiency, and can apply different fusion and enhancement strategies according to different levels, and finally generate a target fusion feature containing rich multi-scale information, providing high-quality input for subsequent image enhancement or other visual tasks.

[0089] Optionally, the step of fusing image features of multiple scales from bottom to top through multiple feature fusion modules and multiple feature enhancement modules to obtain target fusion features may include the following steps:

[0090] Perform a first feature enhancement and a first upsampling on the low-resolution feature to obtain a first enhanced feature;

[0091] Performing a first feature fusion on the first enhanced feature and the medium-resolution feature to obtain a first fused feature;

[0092] Perform a second feature enhancement and a second upsampling on the first fused feature to obtain a second enhanced feature;

[0093] Performing a second feature fusion on the second enhanced feature and the high-resolution feature to obtain a second fused feature;

[0094] Perform a third feature enhancement on the second fused feature to obtain a third enhanced feature;

[0095] The first enhancement feature, the second enhancement feature, the third enhancement feature and the high-resolution feature are spliced ​​to obtain the target fusion feature.

[0096] Among them, the first enhanced feature: improves the expressiveness of the low-resolution feature through the first feature enhancement module (i.e., the first feature enhancement); then performs the first upsampling to improve the resolution of the low-resolution feature, thereby obtaining the first enhanced feature, so that it can be better aligned and fused with the medium-resolution feature.

[0097] The first fusion feature is obtained by fusing the first enhanced feature with the medium-resolution feature through the first feature fusion module, combining the enhanced low-resolution feature with the original medium-resolution feature.

[0098] Second enhanced feature: The feature expression of the first fusion feature is further improved through the second feature enhancement module (i.e., the second feature enhancement); then a second upsampling is performed to obtain the second enhanced feature, and the spatial size of the first fusion feature is increased to match the high-resolution feature, in preparation for the next step of fusion.

[0099] Second fused features: obtained by fusing the second enhanced features with the high-resolution features through the second feature fusion module. This step is similar to the first fusion, combining information from different resolutions through splicing or other fusion operations. The final synthesized features have both the multi-scale features enhanced in the previous steps and the details of the high-resolution features.

[0100] The third enhanced feature is to perform a third feature enhancement on the second fused feature through a third feature fusion module (it can also be upsampled for a third time, which can be determined according to actual conditions), and further improve the description ability of the feature through the enhancement module to obtain the third enhanced feature.

[0101] This embodiment adopts a bottom-up, multiple enhancement and fusion strategy, starting from low-resolution features, and gradually fusing medium-resolution and high-resolution features through three feature enhancements and at least two upsampling operations. Each step includes feature enhancement, upsampling, and fusion with higher-resolution features, ensuring the effective integration and quality improvement of information at different scales. Finally, by splicing all enhanced features with the original high-resolution features, a target fusion feature containing multi-scale information is formed. This method not only realizes the effective information transfer from low-level features to high-level features, but also improves the feature quality through multiple enhancements, while retaining the detail information of the original high-resolution features. This carefully designed fusion process can make full use of image information at different scales, providing a powerful feature representation that contains rich semantic information and retains details for subsequent image processing tasks.

[0102] Optionally, the feature fusion module includes: a 2D convolution module, an average pooling layer, a linear layer, a ReLu activation function and a sigmoid activation function.

[0103] Among them, the 2D convolution module is used to extract the spatial feature from the two input features to obtain the first feature after preliminary fusion;

[0104] The average pooling layer, connected to the 2D convolution module, is used to downsample the preliminary features to obtain the second features;

[0105] The linear layer, connected to the average pooling layer, is used to perform a linear transformation on the second feature to obtain the third feature;

[0106] The ReLu activation function is connected to the linear layer to perform ReLu activation on the third feature to obtain the fourth feature;

[0107] The sigmoid activation function is connected with the ReLu activation function to perform Sigmoid normalization on the fourth feature to obtain the fused feature.

[0108] 2D convolution module: used to extract and transform spatial features, retain spatial information, and learn local feature patterns; average pooling layer: used to reduce the resolution of feature maps and extract regional average features; linear layer: used for feature transformation and dimension adjustment; ReLU activation function: introduces nonlinearity, enhances the expression ability of the model, can solve the gradient disappearance problem, and accelerate convergence; Sigmoid activation function: maps the output to the (0,1) interval to generate attention weights.

[0109] First, the size and number of channels of the spliced ​​features are changed through the 2D convolution module, and then the attention weights are extracted through the average pooling layer, linear layer, activation function and sigmoid function. The harmonious fusion of different features is achieved through the attention weights, providing rich aggregation features for the post-processing steps.

[0110] The feature fusion module provided in this embodiment can effectively process and fuse features from different resolutions, and provide high-quality feature representation for subsequent image processing tasks.

[0111] Optionally, the feature enhancement module includes the following formula (3):

[0112] F IMRB =F IN +IMR2(IMR1(F CM )) (3)

[0113] Among them, F IMRB is the enhanced feature obtained after feature enhancement processing, that is, F IMRB After feature enhancement The enhanced features obtained , F IN Indicates the feature map at the beginning of IMRB, that is, F IN is the input feature of the IMRB module, F CM Represents the feature map obtained by applying two sets of 3×3 convolutional layers and LeakyRelu activation function to the input features. IMR2(IMR1()) represents the multi-scale residual recurrent processing.

[0114] Specifically, the feature enhancement module adopts a highly optimized multi-level processing architecture that integrates multiple deep learning methods. The core structure of the module includes the initial feature extraction layer, the multi-scale residual processing unit, and the final residual connection. First, two sets of 3×3 convolutional layers are used with the LeakyReLU activation function to perform preliminary feature extraction. This step introduces nonlinear transformation and enhances the expressiveness of the model. Subsequently, a two-level intrascale multi-scale residual (IMR) structure is used for deep feature processing. This design allows image features to be captured and enhanced at different scales, effectively integrating multi-scale information. Finally, the processed features are added to the input features through residual connections, which not only retains the original information but also enhances the feature expression ability. In this way, not only can multi-scale features be effectively extracted and enhanced, but also the gradient vanishing problem in deep network training is alleviated through the residual learning mechanism, significantly improving the learning ability and generalization of the model. The use of LeakyReLU further improves the nonlinear activation process and effectively avoids the "death" problem that traditional ReLU may face.

[0115] In this embodiment, different convolution kernels are used to extract features of different scales, and 1x1 convolution kernels are used to fuse these features. At the same time, two short connections are used to improve the loop ability of image features. In general, the proposed deep multi-scale residual feature iterative feature enhancement module better realizes image brightness and detail texture enhancement by acquiring multi-branch deep features and looping different scale residual features, thereby better enhancing the image features.

[0116] Optionally, the above S103 may specifically include the following steps:

[0117] The target fusion features are added to the image to be enhanced to obtain the enhanced target image.

[0118] Specifically, the target fusion features obtained through complex network processing are superimposed on the original image to be enhanced at the pixel level, so that the extracted and enhanced high-level feature information is injected into the original image, achieving a comprehensive enhancement of image details, textures and structures. It can not only improve the clarity and detail richness of the image, but also repair the defective areas in the image to a certain extent, so that the basic content of the original image is retained, and the quality is significantly improved.

[0119] The image processing method provided by the embodiment of the present invention realizes high-quality image processing through progressive feature fusion and multi-scale feature enhancement. The method first extracts multi-scale features from the input image, and then processes them alternately using a feature fusion module (MFB) and a feature enhancement module (IMRB). The feature fusion module gradually fuses features of different scales through an attention mechanism and convolution operations, while the feature enhancement module deepens and enriches feature representation using a multi-scale residual learning structure. This iterative process of "fusion-enhancement-re-fusion" continuously improves feature quality, and finally adds high-quality fused features to the original image to be enhanced to obtain an enhanced target image.

[0120] In a specific embodiment, Figure 2 The image processing method shown in the figure is based on a multi-scale progressive fusion low-light image enhancement method for belt conveyor belt operation monitoring. It takes the original low-light image as input, extracts the shallow features through two layers of convolution, and then obtains image features of different scales through a downsampling module. The multi-feature fusion module (MFB) is responsible for adaptively fusing multi-dimensional features to facilitate the subsequent processing process to obtain richer image information. The residual feature iterative enhancement module (IMRB, feature enhancement module) is responsible for extracting the deep features and multi-scale features of the image after fusion of different dimensions and multi-dimensional information. In order to better enhance the color and texture details of the image, multiple feature enhancement modules are mainly used for cumulative enhancement of different dimensions. Finally, after splicing four different features, the normal light image is estimated through two layers of convolution and added to the original low-light image information to generate the final output image, thereby improving the image quality.

[0121] The entire process from low-light image input to processing through the network structure includes the following steps:

[0122] Step 1: Build a data set;

[0123] The training dataset consists of the training data in PASCAL VOC2007, and the test dataset consists of the collected working images of belt conveyors in real low-light scenes.

[0124] Step 2: Use the multi-scale progressive fusion low-light image enhancement network preset in the training set for training

[0125] The preset multi-scale progressive fusion low-light image enhancement network includes: a residual feature iterative enhancement module (feature enhancement module) and an attention-based feature fusion module.

[0126] Specific as Figure 3 As shown:

[0127] 1. Collect shallow features

[0128] A convolution layer with a convolution kernel of 3×3 is used to extract shallow features of the original input low-light image. The shallow features have more pixel information, including color, texture, edge, corner, etc.;

[0129] 2. Convert the shallow features into feature maps of different scales through downsampling.

[0130] In order to make full use of the brightness, detail texture and other features in low-light images, the input low-light image X is transformed in the spatial dimension. IN ∈R H×W×3 Through two layers of convolution, shallow features X including brightness are extracted 浅层特征 ∈R H×W×c , where the input X 浅层特征 ∈R H×W×C It is downsampled twice before entering the IMRB module to extract medium-resolution features. And low-resolution features While shrinking the feature map, the receptive field is expanded. Each downsampling will reduce the resolution by half and double the number of channels. The low-resolution feature map has rich contextual information, but more spatial information is lost due to the downsampling operation. The opposite is true for the high-resolution feature map.

[0131] Among them, the expression for downsampling operation is:

[0132] X middle =Down(X 浅层特征 ) (1)

[0133] X low =Down(X middle ) (2)

[0134] Among them, X middle is the feature map of medium resolution flow, X 浅层特征 is the characteristic map of high-resolution flow, X low It is the feature map of low-resolution flow, and Down() is the downsampling operation.

[0135] 3. Use the multi-scale residual feature enhancement module to enhance features of different scales and suppress unnecessary features respectively, and use the feature fusion module to fuse the feature maps enhanced by the multi-scale feature enhancement module from bottom to top.

[0136] Specifically, in each multi-scale residual feature enhancement module and feature fusion module:

[0137] (1) fusing the low-resolution feature map after passing through the multi-scale residual feature enhancement module with the medium-resolution feature map bit by bit through a feature fusion module;

[0138] (2) fusing the fused medium-resolution feature map bit by bit with the high-resolution feature map after passing through the multi-scale residual feature enhancement module;

[0139] (3) The fused high-resolution feature map is once again spliced ​​with the upsampled medium-resolution feature map, low-resolution feature map and shallow features to obtain enhanced features with richer information;

[0140] In order to make the image take into account both contextual information and spatial features, the feature map is made to flow from bottom to top, and the low-resolution features with rich contextual information are fused with the high-resolution features with accurate spatial information. The low-resolution features are sent to the IMRB module to obtain richer low-resolution features, and then the low-resolution features are transformed into Through the MFB module and the original After fusion, the IMRB module is used to obtain richer medium-resolution features, and then the high-resolution features are transformed into X′ through the upsampling module. 浅层特征 ∈R H×W×C , through the MFB module and the original X 浅层特征 ∈R H×W×C After fusion, it passes through the IMRB module and finally X 浅层特征 , X middle , X′ high , and X′ 浅层特征 After feature concatenation, the number of channels is restored through two layers of convolution, and then added to the original low-light image to obtain the final output image.

[0141] The structure of the multi-scale residual enhancement module (IMRB): Figure 4 As shown in the figure, 3×3 and 5×5 convolution kernels are used to extract the detail texture and global information of the image. The entire processing process of IMRB is as follows:

[0142] F IMRB =F IN +IMR2(IMR1(F CM )) (3)

[0143] where F IN Represents the feature map at the beginning of IMRB. Represents the feature map obtained by applying two sets of 3×3 convolutional layers and LeakyRelu activation function to the input features. IMR2(IMR1()) represents multi-scale residual loop processing. Different convolution kernels are used to extract features of different scales, and these features are fused using 1x1 convolution kernels. At the same time, two short connections are used to improve the loop ability of image features. Overall, this enhancement module better achieves image brightness and detail texture enhancement by obtaining multi-branch deep features and circulating residual features of different scales.

[0144] The structure of the feature fusion module (MFB): Figure 5 As shown in the figure, the size and number of channels of the spliced ​​features are first changed through the 2D convolution module, and then the attention weights are extracted through the average pooling layer, linear layer, activation function and sigmoid function. The attention weights are used to achieve harmonious fusion of different features and provide rich aggregation features for the post-processing steps.

[0145] Among them, the expression of the feature map fusing low-resolution and high-resolution features from bottom to top is:

[0146] X′ low =Up(IMRB(X low )) (4)

[0147] X′ middle =Up(IMRB(MFB(X middle ,X′ low ))) (5)

[0148] X′ 浅层特征 =Up(IMRB(MFB(X 浅层特征 ,X′ middle ))) (6)

[0149] X′ 增强 =X 低光图像 +CONV(CONCAT(X 浅层特征 ,X′ 浅层特征 ,X′ middle ,X l ' ow ) )(7)

[0150] Among them, IMRB() is a multi-scale residual feature enhancement module, X l ' ow is the low-resolution feature map after IMRB, X′ middle is the medium-resolution feature map after IMRB, X′ 浅层特征 The high-resolution feature map after IMRB, MFB is the feature fusion module, CONCAT() is the serial concatenation operation, and Up() is the upsampling operation.

[0151] 4. Finally, the enhanced features are added to the original low-light image to obtain an enhanced image.

[0152] 5. Based on the loss function, the loss value between the enhanced image and the original image is calculated, and the multi-scale progressive fusion low-light image enhancement network is trained.

[0153] The loss function plays a role in guiding model training, measuring reconstruction errors, improving perceptual quality, and processing specific task requirements in low-light image enhancement tasks. In the field of low-light enhancement, commonly used loss functions include structural loss and smoothness loss. Considering the structural information and semantic information of the image, the loss function designed in this embodiment mainly consists of two parts:

[0154] Structural Loss (SSIM Loss): The calculation of the SSIM loss function is based on the brightness, contrast, and structural similarity of the image. The formula of the SSIM loss function is as follows:

[0155] L ssim (x,y)=(1-ssim(x,y))

[0156] Where x and y are the image to be tested and the output image respectively.

[0157] Total Variation Loss is often used in tasks such as image enhancement and image denoising. Its formula is as follows:

[0158]

[0159] In the formula is the image to be tested. Symbol Represents the pixel value. i and j are the indices of the pixel.

[0160] The loss function used in the training process of the preset low-light image enhancement network based on multi-scale progressive fusion is:

[0161] L total =L ssim +L tv (9)

[0162] In this embodiment, a simple and effective multi-scale residual feature enhancement module (IMRB) is used to enhance the image information of different branches and suppress the influence of noise. At the same time, a simple feature fusion module (MFB) is used to fuse features of different scales to obtain a high-quality output image. After the downsampling operation, the MFB fuses low-resolution and high-resolution features from the bottom up multiple times, overcomes the limitation of the convolution kernel size on the receptive field size, and enables the model to learn the global information of the input image; by proposing a multi-scale residual feature enhancement module (IMRB), multiple residual feature iteration modules and multi-layer convolution blocks are used to filter noise and enhance important features, so that the model has excellent global illumination reconstruction, color restoration and detail reconstruction capabilities, thereby improving the image quality and visual effects; by proposing a low-light image enhancement based on multi-scale progressive fusion composed of a multi-scale residual feature enhancement module (IMRB) and a feature fusion module (MFB), by stacking multiple IMRB modules and MFB modules, top-down feature extraction and bottom-up feature fusion are achieved, and a satisfactory low-light enhancement effect is achieved.

[0163] Embodiment 2: This embodiment provides a fault detection method for a belt conveyor, which specifically includes:

[0164] Acquire detection images of belt conveyors;

[0165] Performing image enhancement processing on the detection image according to the image processing method of any of the above embodiments to obtain an enhanced detection image;

[0166] Fault detection is performed on the belt conveyor according to the enhanced detection image.

[0167] Specifically, first, obtain the inspection image of the belt conveyor. For example, use a camera to collect the image of the belt conveyor as input. Then enhance the inspection image: extract features of multiple resolution levels from the image to be enhanced; use a bottom-up progressive method to gradually fuse image features of different scales, and improve the expression ability and quality of features through feature fusion modules and feature enhancement modules; through multiple feature enhancements and feature fusions, gradually improve image features, starting from low resolution, through the fusion of medium resolution features and the integration of high resolution features, finally obtain the target fusion feature; combine the target fusion feature with the original image to generate an enhanced image. Finally, use the enhanced image to analyze and identify potential faults. This method significantly improves the sensitivity and accuracy of fault detection.

[0168] Among them, fault detection can be done by analyzing images through machine learning models (such as convolutional neural networks) or rule-based algorithms to identify potential conveyor faults. It can detect problems that are usually not easy to detect, such as belt damage, component displacement, surface anomalies, etc.

[0169] By combining image enhancement technology and fault detection algorithms, belt conveyor operation problems and potential faults can be detected more accurately and efficiently. Problems can be detected in a timely manner, reducing downtime and maintenance costs.

[0170] Embodiment 3:

[0171] like Figure 6 As shown, this embodiment provides an image processing device, which is used to execute the above-mentioned image processing method, including:

[0172] A feature extraction module 201 is used to extract features from the image to be enhanced to obtain image features at multiple scales;

[0173] The feature processing module 202 is connected to the feature extraction module and is used to fuse the image features of multiple scales using a progressive fusion method to obtain target fusion features. The progressive fusion method is used to gradually fuse the image features of different scales from bottom to top.

[0174] The image acquisition module 203, the feature processing module and the feature fusion module are used to obtain an enhanced target image according to target fusion features.

[0175] Optionally, the image features of multiple scales include high-resolution features, medium-resolution features and low-resolution features.

[0176] The feature extraction module 201 includes:

[0177] The first extraction unit is used to extract features from the image to be tested to obtain high-resolution features;

[0178] A second extraction unit, connected to the first extraction unit, is used to perform a first downsampling of the high-resolution features to obtain medium-resolution features;

[0179] The third extraction unit is connected to the second extraction unit and is used to perform a second down-sampling on the medium-resolution feature to obtain a low-resolution feature.

[0180] Optionally, the feature processing module 202 includes:

[0181] The feature processing submodule is used to gradually fuse image features of multiple scales from bottom to top through multiple feature fusion modules and multiple feature enhancement modules to obtain target fusion features.

[0182] Optionally, the feature processing submodule includes:

[0183] A first enhancement unit, used for performing a first feature enhancement and a first upsampling on the low-resolution feature to obtain a first enhanced feature;

[0184] A first fusion unit is used to perform a first feature fusion on the first enhanced feature and the medium-resolution feature to obtain a first fused feature;

[0185] A second enhancement unit is used to perform a second feature enhancement and a second upsampling on the first fusion feature to obtain a second enhanced feature;

[0186] A second fusion unit is used to perform a second feature fusion on the second enhanced feature and the high-resolution feature to obtain a second fused feature;

[0187] A third enhancement unit is used to perform a third feature enhancement and a third upsampling on the second fusion feature to obtain a third enhanced feature;

[0188] The splicing unit is used to splice the first enhancement feature, the second enhancement feature, the third enhancement feature and the high-resolution feature to obtain a target fusion feature.

[0189] In the embodiment of the present application, the above-mentioned image processing device obtains the details and structural information of the image to be enhanced at different scales through multi-level feature extraction; then, the target fusion feature is obtained by gradually integrating the information of different scales from bottom to top through a progressive fusion strategy, which can better retain the detail information and enhance the large-scale structure at the same time; finally, the enhanced target image is obtained according to the target fusion feature. Thus, the image to be enhanced is processed by combining multi-scale feature extraction and progressive feature fusion, and local details are enhanced while maintaining the global structure, thereby improving the quality of the image.

[0190] Embodiment 4:

[0191] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0192] The electronic device may include a processor 301 and a memory 302 storing computer program instructions.

[0193] Specifically, the processor 301 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0194] The memory 302 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 302 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 302 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 302 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 302 is a non-volatile solid-state memory.

[0195] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0196] The processor 301 implements any one of the image processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 302 .

[0197] In one example, the electronic device may further include an image processing interface 303 and a bus 304. Figure 7 As shown, the processor 301, the memory 302, and the image processing interface 303 are connected via a bus 304 and perform image processing among each other.

[0198] The image processing interface 303 is mainly used to implement image processing between various modules, devices, units and / or equipment in the embodiments of the present application.

[0199] Bus 304 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 304 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0200] In addition, in combination with the image processing method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the image processing methods in the above embodiment is implemented.

[0201] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0202] The functional blocks shown in the above structured block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or an image processing link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0203] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0204] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0205] The above are only specific implementation methods of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. An image processing method, characterized in that: include: Perform feature extraction on the image to be enhanced to obtain image features at multiple scales; Using a progressive fusion method to fuse the image features of the multiple scales to obtain target fusion features, wherein the progressive fusion method is used to gradually fuse the image features of different scales from bottom to top; According to the target fusion features, an enhanced target image is obtained.

2. The method according to claim 1, characterized in that The image features of the multiple scales include high-resolution features, medium-resolution features and low-resolution features, The feature extraction of the image to be tested is performed to obtain image features of multiple scales, including: Extracting features from the image to be tested to obtain the high-resolution features; Performing a first downsampling on the high-resolution feature to obtain the medium-resolution feature; The medium-resolution features are downsampled for a second time to obtain the low-resolution features.

3. The method according to claim 2, characterized in that The step of using a progressive fusion method to fuse the image features of the multiple scales to obtain target fusion features includes: By using multiple feature fusion modules and multiple feature enhancement modules, the image features of the multiple scales are gradually fused from bottom to top to obtain the target fusion feature. The feature fusion module is used to fuse features of different scales, and the feature enhancement module is used to improve the quality and expression ability of features of a single scale.

4. The method according to claim 3, characterized in that The step of fusing the image features of the multiple scales from bottom to top through multiple feature fusion modules and multiple feature enhancement modules to obtain the target fusion feature includes: Performing a first feature enhancement and a first upsampling on the low-resolution feature to obtain a first enhanced feature; Performing a first feature fusion on the first enhanced feature and the medium-resolution feature to obtain a first fused feature; Performing a second feature enhancement and a second upsampling on the first fusion feature to obtain a second enhanced feature; Performing a second feature fusion on the second enhanced feature and the high-resolution feature to obtain a second fused feature; Performing a third feature enhancement on the second fused feature to obtain a third enhanced feature; The first enhanced feature, the second enhanced feature, the third enhanced feature and the high-resolution feature are spliced ​​to obtain the target fusion feature.

5. The method according to claim 3, characterized in that: The feature fusion module includes: a 2D convolution module, an average pooling layer, a linear layer, a ReLu activation function and a sigmoid activation function.

6. The method according to claim 3, characterized in that The feature enhancement module comprises: F IMRB =F IN +IMR2(IMR1(F CM )), Among them, F IMRB F is the enhanced feature obtained after feature enhancement processing. IN is the input feature of the IMRB module, F IN Indicates the feature map at the beginning of IMRB, F CM represents the feature map obtained by applying two sets of 3×3 convolutional layers and LeakyRelu activation function to the input features, and IMR2 (IMR1()) represents the multi-scale residual cycle processing.

7. The method according to claim 1, characterized in that Obtaining an enhanced target image according to the target fusion features includes: The target fusion feature is added to the image to be enhanced to obtain an enhanced target image.

8. A belt conveyor fault detection method, characterized in that: Acquiring a detection image of the belt conveyor; Performing image enhancement processing on the detection image according to the image processing method according to any one of claims 1 to 7 to obtain an enhanced detection image; Fault detection is performed on the belt conveyor according to the enhanced detection image.

9. An image processing device, characterized in that: include: A feature extraction module is used to extract features of the image to be enhanced and obtain image features at multiple scales; A feature processing module, connected to the feature extraction module, for performing feature fusion on the image features of the multiple scales by using a progressive fusion method to obtain a target fusion feature, wherein the progressive fusion method is used to gradually fuse the image features of different scales from bottom to top; The image acquisition module, the feature processing module and the feature fusion module are used to obtain an enhanced target image according to the target fusion features.

10. The device according to claim 9, characterized in that The image features of the multiple scales include high-resolution features, medium-resolution features and low-resolution features, The feature extraction module comprises: A first extraction unit, configured to extract features from the image to be enhanced to obtain the high-resolution features; A second extraction unit, connected to the first extraction unit, configured to perform a first downsampling on the high-resolution feature to obtain the medium-resolution feature; The third extraction unit is connected to the second extraction unit and is used to perform a second down-sampling on the medium-resolution feature to obtain the low-resolution feature.

11. The device according to claim 10, characterized in that The feature processing module comprises: The feature processing submodule is used to gradually fuse the image features of the multiple scales from bottom to top through multiple feature fusion modules and multiple feature enhancement modules to obtain the target fusion feature.

12. An electronic device, characterized in that: The invention comprises a processor, a memory and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the image processing method according to any one of claims 1 to 7.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.

Images