Image defect segmentation method, device and equipment and computer readable storage medium

Abstract

The application discloses an image defect segmentation method, device and equipment and a computer readable storage medium. The method comprises the following steps: acquiring a target image and inputting the target image into a multi-level feature extraction network; the multi-level feature extraction network comprises a plurality of feature extraction sub-networks connected in sequence; for each feature extraction sub-network, according to sequence data generated by a previous feature extraction sub-network in a feature extraction process, performing feature extraction on a feature map output by the previous feature extraction sub-network by using the feature extraction sub-network, and obtaining a corresponding feature map of the feature extraction sub-network; performing fusion processing on the corresponding feature maps of the feature extraction sub-networks to obtain a fusion feature map, and determining a defect segmentation result of the target image according to the fusion feature map. The technical solution provided by the application can realize more accurate and effective image defect segmentation processing.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image defect segmentation method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Currently, visual segmentation schemes based on neural network models (such as selective state-space models) generally adopt a "stage-by-stage independent operation" paradigm. This means that each stage of the network, after receiving the feature map from the previous stage, starts a selective scan from scratch, failing to effectively inherit the relevant feature information extracted by shallower networks. When processing image data, especially high-resolution industrial network data, this design often leads to the segmentation of multi-scale defects such as fine cracks and mesh-like scratches into discontinuous fragments, resulting in both missed detections and false alarms, thus affecting the accuracy of image defect segmentation results.

[0003] Therefore, how to achieve more accurate and effective image defect segmentation is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this application is to provide an image defect segmentation method that can achieve more accurate and effective image defect segmentation processing; another purpose of this application is to provide an image defect segmentation device, electronic device, and computer-readable storage medium, all of which have the above-mentioned beneficial effects.

[0005] In a first aspect, this application discloses an image defect segmentation method, comprising:

[0006] A target image is acquired and input into a multi-level feature extraction network; the multi-level feature extraction network includes multiple feature extraction sub-networks connected in sequence.

[0007] For each feature extraction subnetwork, based on the sequence data generated by the previous feature extraction subnetwork during the feature extraction process, the feature extraction subnetwork is used to extract features from the feature map output by the previous feature extraction subnetwork to obtain the feature map corresponding to the feature extraction subnetwork.

[0008] The feature maps corresponding to each feature extraction sub-network are fused to obtain a fused feature map, and the defect segmentation result of the target image is determined based on the fused feature map.

[0009] Optionally, based on the sequence data generated by the previous feature extraction sub-network during the feature extraction process, feature extraction is performed on the feature map output by the previous feature extraction sub-network using the previous feature extraction sub-network to obtain the feature map corresponding to the feature extraction sub-network, including:

[0010] The feature map output by the previous feature extraction sub-network is processed using the mapping calculation channel in the feature extraction sub-network to obtain the original information of the feature map;

[0011] The feature map output by the previous feature extraction sub-network is processed using the convolution calculation channel in the feature extraction sub-network to obtain the local features of the feature map;

[0012] Based on the sequence data generated by the previous feature extraction subnetwork during the feature extraction process, the feature map output by the previous feature extraction subnetwork is processed using the sequence calculation channel in the feature extraction subnetwork to obtain the global features of the feature map;

[0013] Based on the original information, the local features, and the global features, a feature map corresponding to the feature extraction sub-network is generated.

[0014] Optionally, based on the sequence data generated by the preceding feature extraction sub-network during the feature extraction process, the feature map output by the preceding feature extraction sub-network is processed using the sequence calculation channel in the feature extraction sub-network to obtain the global features of the feature map, including:

[0015] The feature map output by the previous feature extraction subnetwork is serialized using the sequence calculation channel in the feature extraction subnetwork to obtain initial sequence data.

[0016] The initial sequence data is concatenated with the sequence data generated by the previous feature extraction sub-network during the feature extraction process to obtain the target sequence data.

[0017] Sequence calculations are performed on the target sequence data to obtain the initial global features of the feature map;

[0018] The initial global features are sampled based on the sequence length of the initial sequence data to obtain the global features.

[0019] Optionally, the initial global features are sampled based on the sequence length of the initial sequence data to obtain the global features, including:

[0020] The target sampling rule is determined based on the current image defect segmentation scenario; the target sampling rule is a random sampling rule or a sampling rule based on a preset data range;

[0021] Using the target sampling rule, the initial global features are sampled with reference to the sequence length of the initial sequence data to obtain the global features.

[0022] Optionally, generating a feature map corresponding to the feature extraction sub-network based on the original information, the local features, and the global features includes:

[0023] The original information, the local features, and the global features are concatenated to obtain the initial feature map corresponding to the feature extraction sub-network.

[0024] The initial feature map is downsampled to obtain the feature map corresponding to the feature extraction sub-network.

[0025] Optionally, the feature maps corresponding to each of the feature extraction sub-networks are fused to obtain a fused feature map, including:

[0026] The feature maps corresponding to each of the feature extraction sub-networks are fused to obtain an initial fused feature map;

[0027] The initial fused feature map is upsampled to obtain the fused feature map.

[0028] Optionally, the feature maps corresponding to each of the feature extraction sub-networks are fused to obtain a fused feature map, including:

[0029] A denoising network is used to denoise the feature map corresponding to the first feature extraction sub-network in the multi-level feature extraction network to obtain a denoised feature map.

[0030] The denoised feature map and the feature maps corresponding to other feature extraction sub-networks in the multi-level feature extraction network are fused to obtain the fused feature map.

[0031] Secondly, this application discloses an image defect segmentation apparatus, comprising:

[0032] An acquisition module is used to acquire a target image and input the target image into a multi-level feature extraction network; the multi-level feature extraction network includes multiple feature extraction sub-networks connected in sequence.

[0033] The extraction module is used to extract features from the feature map output by the previous feature extraction sub-network based on the sequence data generated by the previous feature extraction sub-network during the feature extraction process, and to obtain the feature map corresponding to the feature extraction sub-network.

[0034] The fusion module is used to fuse the feature maps corresponding to each of the feature extraction sub-networks to obtain a fused feature map, and to determine the defect segmentation result of the target image based on the fused feature map.

[0035] Thirdly, this application discloses an electronic device, including:

[0036] Memory, used to store computer programs;

[0037] A processor, configured to implement the steps of any of the image defect segmentation methods described above when executing the computer program.

[0038] Fourthly, this application discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the image defect segmentation methods described above.

[0039] This application provides an image defect segmentation method, comprising: acquiring a target image and inputting the target image into a multi-level feature extraction network; the multi-level feature extraction network comprising a plurality of sequentially connected feature extraction sub-networks; for each feature extraction sub-network, based on the sequence data generated by the previous feature extraction sub-network during the feature extraction process, performing feature extraction on the feature map output by the previous feature extraction sub-network to obtain a feature map corresponding to the feature extraction sub-network; performing fusion processing on the feature maps corresponding to each feature extraction sub-network to obtain a fused feature map, and determining the defect segmentation result of the target image based on the fused feature map.

[0040] By applying the technical solution provided in this application, feature extraction of a target image is achieved using a multi-level feature extraction network. This multi-level feature extraction network includes multiple feature extraction sub-networks connected in sequence. The defect segmentation result of the target image can then be obtained by fusing the feature maps output by each feature extraction sub-network. In the process of feature extraction using each feature extraction sub-network, the sequence data generated by the previous feature extraction sub-network during the feature extraction process is fused for feature extraction. This allows the "prior knowledge" of the overall defect morphology obtained by the shallow network from the target image to be injected into the deep network, avoiding repeated learning of each level of feature extraction sub-network, thereby effectively reducing the risk of fragmentation and thus effectively improving the accuracy of the image defect segmentation result.

[0041] The image defect segmentation device, electronic device, and computer-readable storage medium provided in this application also have the above-mentioned technical effects, and will not be described in detail here. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the prior art and the embodiments of this application, the accompanying drawings used in the description of the prior art and the embodiments of this application will be briefly introduced below. Of course, the accompanying drawings described below with respect to the embodiments of this application are only a part of the embodiments in this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort, and such other drawings also fall within the protection scope of this application.

[0043] Figure 1 A flowchart illustrating an image defect segmentation method provided in an embodiment of this application;

[0044] Figure 2 This is a schematic diagram of an image defect segmentation architecture provided in an embodiment of this application;

[0045] Figure 3 This is a schematic diagram illustrating the principle of cross-layer skipping between feature extraction sub-networks provided in an embodiment of this application.

[0046] Figure 4 This is a schematic diagram illustrating the principle of image serialization processing provided in an embodiment of this application;

[0047] Figure 5 A schematic diagram illustrating the principle of cross-layer sequence computation provided in this application embodiment;

[0048] Figure 6 This is a schematic diagram of the structure of an image defect segmentation device provided in an embodiment of this application;

[0049] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0050] The core of this application is to provide an image defect segmentation method that can achieve more accurate and effective image defect segmentation processing; another core aspect of this application is to provide an image defect segmentation device, electronic device, and computer-readable storage medium, all of which have the aforementioned beneficial effects.

[0051] To provide a clearer and more complete description of the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0052] This application provides an image defect segmentation method.

[0053] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating an image defect segmentation method provided in an embodiment of this application. The image defect segmentation method may include, but is not limited to, the following S101~S103.

[0054] S101: Acquire the target image and input the target image into the multi-level feature extraction network; the multi-level feature extraction network includes multiple feature extraction sub-networks connected in sequence.

[0055] This step aims to acquire the target image and input it into a multi-level feature extraction network for subsequent image processing. The target image is the image requiring defect segmentation, such as a high-resolution industrial image. Its acquisition method is not unique; it can be image data acquired by an image acquisition device (such as a camera), image data directly input by the user, or image data loaded / downloaded from a specific storage location. This application does not limit this. The multi-level feature extraction network is a visual neural network used to extract features from the target image. It can include multiple sequentially connected feature extraction sub-networks (the number of feature extraction sub-networks is greater than or equal to 2), where each feature extraction sub-network performs a feature extraction operation.

[0056] In one possible implementation, the multi-level feature extraction network can specifically be a feature extraction network layer in a Selective State Space Model (SSM). In an SSM, the multi-level feature extraction network is connected to both the input and output networks. It can receive the target image through the input network and transmit it to the multi-level feature extraction network for feature extraction, and then output the final feature extraction result through the output network.

[0057] S102: For each feature extraction sub-network, based on the sequence data generated by the previous feature extraction sub-network during the feature extraction process, the feature extraction sub-network is used to extract features from the feature map output by the previous feature extraction sub-network to obtain the feature map corresponding to the feature extraction sub-network.

[0058] This step aims to achieve target image feature extraction based on a multi-level feature extraction network. As mentioned above, the multi-level feature extraction network includes multiple sequentially connected feature extraction sub-networks. Each feature extraction sub-network performs a feature extraction operation. In the implementation process, the target image is first input into the first feature extraction sub-network of the multi-level feature extraction network. This sub-network extracts features from the target image to obtain its corresponding feature map, and simultaneously obtains its corresponding sequence data through serialization during the feature extraction process. Then, the current feature map and the current sequence data are input into the second feature extraction sub-network of the multi-level feature extraction network. This sub-network continues to extract features from the current feature map with reference to the current sequence data to obtain a new feature map, and similarly, obtains a new sequence data through serialization during the feature extraction process. Further, the new feature map and the new sequence data are input into the third feature extraction sub-network of the multi-level feature extraction network, and so on, until the last feature extraction sub-network of the multi-level feature extraction network outputs its corresponding feature map. Thus, the feature map output by each feature extraction sub-network in the multi-level feature extraction network can be obtained.

[0059] In one embodiment of this application, based on the sequence data generated by the previous feature extraction sub-network during the feature extraction process, feature extraction is performed on the feature map output by the previous feature extraction sub-network to obtain the feature map corresponding to the feature extraction sub-network. This may include: processing the feature map output by the previous feature extraction sub-network using the mapping calculation channel in the feature extraction sub-network to obtain the original information of the feature map; processing the feature map output by the previous feature extraction sub-network using the convolution calculation channel in the feature extraction sub-network to obtain the local features of the feature map; processing the feature map output by the previous feature extraction sub-network using the sequence calculation channel in the feature extraction sub-network based on the sequence data generated by the previous feature extraction sub-network during the feature extraction process to obtain the global features of the feature map; and generating the feature map corresponding to the feature extraction sub-network based on the original information, local features, and global features.

[0060] This application provides an implementation scheme for feature extraction based on a feature extraction sub-network. Specifically, any feature extraction sub-network in a multi-level feature extraction network can include a channel splitting module, a channel calculation module, and a channel concatenation module, with the channel splitting module connected to the channel calculation module, and the channel calculation module connected to the channel concatenation module. The calculation module can be divided into mapping calculation channels, convolution calculation channels, and sequence calculation channels. Based on the aforementioned connection relationships between modules, it can be determined that these three calculation channels are respectively connected to the channel splitting module and the channel concatenation module.

[0061] Based on this, during feature extraction based on any feature extraction sub-network, the feature map output by the previous feature extraction sub-network can be obtained through the channel splitting module and channel splitting can be performed on it (it should be noted that if the current feature extraction sub-network is the first feature extraction sub-network in a multi-level feature extraction network, the image data obtained here should be the target image); then, it is input to the mapping calculation channel, convolution calculation channel and sequence calculation channel respectively to perform different types of channel calculations to obtain the calculation results of each channel; finally, the channel splicing module is used to splice the calculation results of each channel to obtain the feature map corresponding to the current feature extraction sub-network.

[0062] The mapping calculation channel is used to achieve an identity mapping between the feature map and the target image, directly reaching the channel stitching module, aiming to preserve some original information of the current feature map / target image. The convolution calculation channel is used to perform convolution calculations on the feature map / target image, aiming to obtain local detail features of the current feature map / target image. In addition, this convolution calculation channel can be further divided into convolution calculation sub-channels based on convolution calculation kernels of different sizes. For example, this convolution calculation channel can use convolution calculation kernels of sizes N×N and M×M to perform convolution calculations on the feature map / target image, obtaining their respective convolution calculation results, which are then input together into the channel stitching module. The sequence computation channel is used to perform sequence computation on the feature map / target image, aiming to obtain the global macroscopic features of the current feature map / target image. It should be noted that this sequence computation process can be achieved by integrating the sequence data (previous sequence data) generated by the previous feature extraction sub-network during the feature extraction process (of course, if the current feature extraction sub-network is the first feature extraction sub-network in a multi-level feature extraction network, there is no need to obtain the previous sequence data here, and the feature extraction operation can be performed directly). At the same time, this sequence computation process will also generate new sequence data for the next feature extraction sub-network to perform the sequence computation operation in the feature extraction process. In this way, the "prior knowledge" of the overall morphology of defects obtained by the shallow network from the target image can be injected into the deep network.

[0063] In one possible implementation, based on the sequence data generated by the previous feature extraction sub-network during feature extraction, the feature map output by the previous feature extraction sub-network is processed using the sequence calculation channel in the feature extraction sub-network to obtain the global features of the feature map. This can include: serializing the feature map output by the previous feature extraction sub-network (the previous feature map) using the sequence calculation channel in the feature extraction sub-network to obtain initial sequence data; concatenating the initial sequence data with the sequence data generated by the previous feature extraction sub-network during feature extraction to obtain target sequence data; performing sequence calculation on the target sequence data to obtain the initial global features of the feature map; and sampling the initial global features according to the sequence length of the initial sequence data to obtain the global features.

[0064] This application provides an implementation scheme for calculating the current sequence based on the previous sequence data. Specifically, the sequence calculation includes serialization processing and sequence scanning calculation. In the implementation process, for any feature extraction subnetwork in the multi-level feature extraction network, it can use its own sequence calculation channel to serialize the previous feature map to obtain initial sequence data. Serialization processing refers to serializing a 2D image into 1D data, which can then be deserialized back into 2D data when entering the channel splicing module. Further, the currently obtained initial sequence data and the previous sequence data are spliced ​​to obtain target sequence data. Then, by performing sequence scanning calculation on the target sequence data, the initial global features of the feature map can be obtained. However, since the target sequence data is obtained by splicing the current initial sequence data and the previous sequence data, the data length of the target sequence data = the data length of the initial global features = the data length of the current initial sequence data + the data length of the previous sequence data. Therefore, it is also necessary to sample the initial global features with reference to the data length of the current initial sequence data to obtain global features with the same data length as the current initial sequence data.

[0065] Furthermore, the initial global features are sampled based on the sequence length of the initial sequence data to obtain global features. This can include: determining a target sampling rule based on the current image defect segmentation scenario; the target sampling rule being a random sampling rule or a sampling rule based on a preset data interval; and using the target sampling rule, the initial global features are sampled with reference to the sequence length of the initial sequence data to obtain global features. In other words, the specific sampling scheme for the initial global features can be determined by referring to the actual image defect segmentation scenario. Specifically, the sampling rule can include random sampling rules and sampling rules based on a preset data interval. It can be understood that the former does not limit the sampling data interval, while the latter does. The former consumes more computational resources than the latter, but it can make the network model more robust. Therefore, the former is suitable for complex defect segmentation scenarios, while the latter is suitable for simple defect segmentation scenarios.

[0066] S103: The feature maps corresponding to each feature extraction sub-network are fused to obtain a fused feature map, and the defect segmentation result of the target image is determined based on the fused feature map.

[0067] This step aims to achieve the fusion of multiple feature maps to facilitate the determination of defect segmentation results for the target image based on the fused feature maps. Specifically, the feature maps output by each feature extraction sub-network in the multi-level feature extraction network can be fused. Based on this fusion result, the defect segmentation result of the target image can be determined, thus completing the defect segmentation of the target image.

[0068] In one embodiment of this application, generating a feature map corresponding to a feature extraction sub-network based on original information, local features, and global features may include: concatenating the original information, local features, and global features to obtain an initial feature map corresponding to the feature extraction sub-network; and downsampling the initial feature map to obtain the feature map corresponding to the feature extraction sub-network.

[0069] Correspondingly, the feature maps corresponding to each feature extraction sub-network are fused to obtain a fused feature map. This can include: fusing the feature maps corresponding to each feature extraction sub-network to obtain an initial fused feature map; and upsampling the initial fused feature map to obtain a fused feature map.

[0070] The image defect segmentation method provided in the above-mentioned embodiments can also perform downsampling processing during feature extraction and upsampling processing after feature extraction. Downsampling processing can effectively reduce the spatial resolution of the feature map to compress spatial information and retain key features, thereby effectively improving the accuracy of feature extraction. Upsampling processing can restore the high-resolution features of the feature map through detail filling, effectively restoring the original image size for output.

[0071] In one embodiment of this application, the feature maps corresponding to each feature extraction sub-network are fused to obtain a fused feature map. This may include: using a denoising network to denoise the feature map corresponding to the first feature extraction sub-network in the multi-level feature extraction network to obtain a denoised feature map; and fusing the denoised feature map with the feature maps corresponding to the other feature extraction sub-networks in the multi-level feature extraction network to obtain a fused feature map.

[0072] To further improve the accuracy of image defect segmentation results, a denoising network can be added to denoise the feature maps. This denoising network can be connected to the first feature extraction sub-network in a multi-level feature extraction network, aiming to remove noise interference from shallow images and improve the robustness of the entire defect segmentation framework. In implementation, this denoising layer can be based on 3×3 or 5×5 convolutional layers.

[0073] As can be seen, the image defect segmentation method provided in this application embodiment utilizes a multi-level feature extraction network to extract features from the target image. This multi-level feature extraction network includes multiple feature extraction sub-networks connected in sequence. The defect segmentation result of the target image can then be obtained by fusing the feature maps output by each feature extraction sub-network. In the process of feature extraction using each feature extraction sub-network, the sequence data generated by the previous feature extraction sub-network during the feature extraction process is fused for feature extraction. This allows the "prior knowledge" of the overall defect morphology obtained by the shallow network from the target image to be injected into the deep network, avoiding repeated learning of each level of feature extraction sub-network, thereby effectively reducing the risk of fragmentation and thus effectively improving the accuracy of the image defect segmentation result.

[0074] For example, embodiments of this application provide an image defect segmentation method.

[0075] The image defect segmentation method provided in this application embodiment is a defect segmentation method that embeds a "hierarchical state transfer" mechanism inside the grouped SSM model. By caching and reusing hidden states across layers, it achieves seamless flow between long-range defect contours and background suppression information, thereby significantly improving the accuracy and robustness of industrial visual inspection without increasing the computational load.

[0076] Specifically, please refer to Figure 2 , Figure 2This is a schematic diagram of an image defect segmentation architecture provided in an embodiment of this application. The image defect segmentation architecture consists of multiple parts, and its structure will be described in detail below. First, the input can be an image captured by an industrial camera, which is fed into multiple feature extraction blocks (i.e., a multi-level feature extraction network, including feature extraction block_1, feature extraction block_2, feature extraction block_3, and feature extraction block_4, i.e., each feature extraction sub-network in the multi-level feature extraction network). Then, between these four feature extraction blocks, based on cross-layer connections, the "prior knowledge" of the overall defect morphology from the shallow network is directly injected into the deep network through learnable hidden state vectors, avoiding repetitive learning at each level and significantly reducing the risk of fragmentation.

[0077] Based on this, please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the principle of cross-layer skipping between feature extraction subnetworks provided in an embodiment of this application. Figure 3 The core structure is illustrated using the connection relationship between feature extraction block_2 and feature extraction block_3 as an example. (Refer to...) Figure 3 Feature extraction block _2, as shown, takes the feature map group extracted from the upper layer as input. First, the channels are split into a certain number of parts (4 parts in this example; this number can be set according to actual needs and is independent of the number of feature extraction blocks), resulting in group _1, group _2, group _3, and group _4. Group _1 is not processed further and is used for identity mapping directly to the channel concatenation layer, preserving some original information. Groups _2 and _3 are used for feature extraction with convolution kernels of N×N and M×M respectively, to obtain local detail features. Group _4 will undergo sequence calculation to extract macroscopic information based on global features. During sequence calculation, the 2D image needs to be serialized into 1D data. The branch entering channel concatenation is deserialized from 1D data back to 2D data after sequence calculation, while the part used for cross-layer jump connections continues to retain the 1D form to pass information to the next layer. Furthermore, the image serialization principle is as follows... Figure 4 As shown, Figure 4 This is a schematic diagram illustrating the principle of image serialization processing provided in an embodiment of this application; deserialization is the reverse operation. Further, after all block operations are completed, the processed channels are concatenated in their original positions, and then this set of feature maps is downsampled to compress features and reduce computational load. In specific implementations, downsampling can use average pooling downsampling or bilinear interpolation downsampling.

[0078] For further information, please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram illustrating the principle of cross-layer sequence computation provided in an embodiment of this application. Specifically, the input consists of sequence data from the upper layer and sequence data from the current layer. Figure 5 The sequence data shown has a length of 4, with the numbers representing the IDs of different sequence elements. First, the sequences from the upper layer and the current layer are concatenated. Then, the concatenated data undergoes sequence computation to obtain a new sequence of the same length. This sequence incorporates global information from different layers, directly injecting the shallow network's "prior knowledge" of the overall defect morphology into the deep network, avoiding repetitive learning at each level and significantly reducing the risk of fragmentation. Second, the data output to the next layer after sequence computation is not all the elements calculated in this operation, but rather a sequence of elements with the same length as the input. For example, if the initial input sequence length in the current illustration is 4, then a sequence of length 4 will be used to enter the next layer. The method for selecting these 4 elements can be chosen based on the actual task, such as random sampling, sampling the middle 4 elements, or sampling the first 4 or last 4 elements.

[0079] Finally, back Figure 2 The outputs of feature extraction blocks 2, 3, and 4 can be fed into a channel fusion layer for data fusion to integrate multi-layer feature information. This fusion layer can be a 1×1 convolutional layer. Simultaneously, to further integrate the information from the shallowest feature extraction block 1, a denoising layer can be used to remove noise interference from the shallow image, improving the robustness of the entire defect segmentation framework. This denoising layer can be based on a 3×3 or 5×5 convolutional layer. Finally, after all information is fused, an upsampling layer is applied to restore the data to the input size, yielding the output of the image defect segmentation model.

[0080] As can be seen, the image defect segmentation method provided in this application embodiment utilizes a multi-level feature extraction network to extract features from the target image. This multi-level feature extraction network includes multiple feature extraction sub-networks connected in sequence. The defect segmentation result of the target image can then be obtained by fusing the feature maps output by each feature extraction sub-network. In the process of feature extraction using each feature extraction sub-network, the sequence data generated by the previous feature extraction sub-network during the feature extraction process is fused for feature extraction. This allows the "prior knowledge" of the overall defect morphology obtained by the shallow network from the target image to be injected into the deep network, avoiding repeated learning of each level of feature extraction sub-network, thereby effectively reducing the risk of fragmentation and effectively improving the accuracy of the image defect segmentation result.

[0081] This application provides an image defect segmentation device.

[0082] Please refer to Figure 6 , Figure 6 This is a schematic diagram of the structure of an image defect segmentation device provided in an embodiment of this application. The image defect segmentation device may include:

[0083] Acquisition module 1 is used to acquire the target image and input the target image into the multi-level feature extraction network; the multi-level feature extraction network includes multiple feature extraction sub-networks connected in sequence;

[0084] Extraction module 2 is used to extract features from the feature map output by the previous feature extraction sub-network based on the sequence data generated by the previous feature extraction sub-network during the feature extraction process, and obtain the feature map corresponding to the feature extraction sub-network for each feature extraction sub-network.

[0085] The fusion module 3 is used to fuse the feature maps corresponding to each feature extraction sub-network to obtain a fused feature map, and to determine the defect segmentation result of the target image based on the fused feature map.

[0086] As can be seen, the image defect segmentation device provided in this application embodiment utilizes a multi-level feature extraction network to extract features from the target image. This multi-level feature extraction network includes multiple feature extraction sub-networks connected in sequence. The defect segmentation result of the target image can then be obtained by fusing the feature maps output by each feature extraction sub-network. In the process of feature extraction using each feature extraction sub-network, the sequence data generated by the previous feature extraction sub-network during the feature extraction process is fused for feature extraction. This allows the "prior knowledge" of the overall defect morphology obtained by the shallow network from the target image to be injected into the deep network, avoiding repeated learning of each level of feature extraction sub-network, thereby effectively reducing the risk of fragmentation and effectively improving the accuracy of the image defect segmentation result.

[0087] In one embodiment of this application, the extraction module 2 may include:

[0088] The first processing unit is used to process the feature map output by the previous feature extraction sub-network using the mapping calculation channel in the feature extraction sub-network to obtain the original information of the feature map.

[0089] The second processing unit is used to process the feature map output by the previous feature extraction sub-network using the convolution calculation channel in the feature extraction sub-network to obtain the local features of the feature map.

[0090] The third processing unit is used to process the feature map output by the previous feature extraction sub-network using the sequence calculation channel in the feature extraction sub-network based on the sequence data generated by the previous feature extraction sub-network during the feature extraction process, so as to obtain the global features of the feature map.

[0091] The generation unit is used to generate feature maps corresponding to the feature extraction subnetwork based on the original information, local features, and global features.

[0092] In one embodiment of this application, the third processing unit may include:

[0093] The serialization subunit is used to serialize the feature map output by the previous feature extraction subnetwork using the sequence computation channel in the feature extraction subnetwork to obtain the initial sequence data.

[0094] The splicing subunit is used to splice the initial sequence data with the sequence data generated by the previous feature extraction subnetwork during the feature extraction process to obtain the target sequence data.

[0095] The sequence computation subunit is used to perform sequence computation on the target sequence data to obtain the initial global features of the feature map;

[0096] The sampling subunit is used to sample the initial global features based on the sequence length of the initial sequence data to obtain the global features.

[0097] In one embodiment of this application, the above-mentioned sampling subunit can be specifically used to determine the target sampling rule according to the current image defect segmentation scenario; the target sampling rule is a random sampling rule or a sampling rule based on a preset data interval; using the target sampling rule, the initial global features are sampled with reference to the sequence length of the initial sequence data to obtain the global features.

[0098] In one embodiment of this application, the above-mentioned generation unit can be specifically used to concatenate the original information, local features, and global features to obtain an initial feature map corresponding to the feature extraction sub-network; and to perform downsampling processing on the initial feature map to obtain a feature map corresponding to the feature extraction sub-network.

[0099] In one embodiment of this application, the fusion module 3 can be specifically used to fuse the feature maps corresponding to each feature extraction sub-network to obtain an initial fused feature map; and to upsample the initial fused feature map to obtain a fused feature map.

[0100] In one embodiment of this application, the fusion module 3 can be specifically used to denoise the feature map corresponding to the first feature extraction sub-network in the multi-level feature extraction network using a denoising network to obtain a denoised feature map; and to fuse the denoised feature map and the feature maps corresponding to the other feature extraction sub-networks in the multi-level feature extraction network to obtain a fused feature map.

[0101] For a description of the apparatus provided in the embodiments of this application, please refer to the above method embodiments; further details will not be repeated here.

[0102] This application provides an electronic device.

[0103] Please refer to Figure 7 , Figure 7 This application provides a schematic diagram of the structure of an electronic device, which may include:

[0104] Memory 11 is used to store computer programs;

[0105] The processor 10 is configured to execute computer programs to implement the steps of any of the image defect segmentation methods described above.

[0106] like Figure 7 The diagram shows the structural composition of an electronic device, which may include a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0107] In this embodiment, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0108] The processor 10 can call the program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiment of the image defect segmentation method.

[0109] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:

[0110] The target image is acquired and input into a multi-level feature extraction network. The multi-level feature extraction network includes multiple feature extraction sub-networks connected in sequence. For each feature extraction sub-network, based on the sequence data generated by the previous feature extraction sub-network during the feature extraction process, the feature map output by the previous feature extraction sub-network is used to extract features, thereby obtaining the feature map corresponding to the feature extraction sub-network. The feature maps corresponding to each feature extraction sub-network are fused to obtain a fused feature map, and the defect segmentation result of the target image is determined based on the fused feature map.

[0111] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0112] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0113] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0114] Of course, it should be noted that, Figure 7 The structure shown does not constitute a limitation on the electronic device in the embodiments of this application. In practical applications, the electronic device may include more than Figure 7 More or fewer components as shown, or combinations of certain components.

[0115] This application provides a computer-readable storage medium.

[0116] The computer-readable storage medium provided in this application embodiment stores a computer program, which, when executed by a processor, can implement the steps of any of the image defect segmentation methods described above.

[0117] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0119] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0120] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0121] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0122] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. An image defect segmentation method, characterized in that, include: Acquire the target image and input the target image into a multi-level feature extraction network; The multi-level feature extraction network includes multiple feature extraction sub-networks connected in sequence; For each feature extraction subnetwork, based on the sequence data generated by the previous feature extraction subnetwork during the feature extraction process, the feature extraction subnetwork is used to extract features from the feature map output by the previous feature extraction subnetwork to obtain the feature map corresponding to the feature extraction subnetwork; the sequence data is obtained by serializing the feature map obtained by the previous feature extraction subnetwork during the feature extraction process. The feature maps corresponding to each of the feature extraction sub-networks are fused to obtain a fused feature map, and the defect segmentation result of the target image is determined based on the fused feature map. Specifically, based on the sequence data generated by the preceding feature extraction sub-network during feature extraction, the feature extraction sub-network is used to extract features from the feature map output by the preceding feature extraction sub-network to obtain the feature map corresponding to the feature extraction sub-network. This includes: processing the feature map output by the preceding feature extraction sub-network using the mapping calculation channel in the feature extraction sub-network to obtain the original information of the feature map; processing the feature map output by the preceding feature extraction sub-network using the convolution calculation channel in the feature extraction sub-network to obtain the local features of the feature map; processing the feature map output by the preceding feature extraction sub-network using the sequence calculation channel in the feature extraction sub-network based on the sequence data generated by the preceding feature extraction sub-network during feature extraction to obtain the global features of the feature map; and generating the feature map corresponding to the feature extraction sub-network based on the original information, the local features, and the global features.

2. The image defect segmentation method according to claim 1, characterized in that, Based on the sequence data generated by the preceding feature extraction subnetwork during the feature extraction process, the feature map output by the preceding feature extraction subnetwork is processed using the sequence calculation channel in the feature extraction subnetwork to obtain the global features of the feature map, including: The feature map output by the previous feature extraction subnetwork is serialized using the sequence calculation channel in the feature extraction subnetwork to obtain initial sequence data. The initial sequence data is concatenated with the sequence data generated by the previous feature extraction sub-network during the feature extraction process to obtain the target sequence data. Sequence calculations are performed on the target sequence data to obtain the initial global features of the feature map; The initial global features are sampled based on the sequence length of the initial sequence data to obtain the global features.

3. The image defect segmentation method according to claim 2, characterized in that, The initial global features are sampled based on the sequence length of the initial sequence data to obtain the global features, including: The target sampling rule is determined based on the current image defect segmentation scenario; the target sampling rule is a random sampling rule or a sampling rule based on a preset data range; Using the target sampling rule, the initial global features are sampled with reference to the sequence length of the initial sequence data to obtain the global features.

4. The image defect segmentation method according to claim 1, characterized in that, Based on the original information, the local features, and the global features, a feature map corresponding to the feature extraction sub-network is generated, including: The original information, the local features, and the global features are concatenated to obtain the initial feature map corresponding to the feature extraction sub-network. The initial feature map is downsampled to obtain the feature map corresponding to the feature extraction sub-network.

5. The image defect segmentation method according to claim 4, characterized in that, The feature maps corresponding to each of the aforementioned feature extraction sub-networks are fused to obtain a fused feature map, including: The feature maps corresponding to each of the feature extraction sub-networks are fused to obtain an initial fused feature map; The initial fused feature map is upsampled to obtain the fused feature map.

6. The image defect segmentation method according to any one of claims 1 to 5, characterized in that, The feature maps corresponding to each of the aforementioned feature extraction sub-networks are fused to obtain a fused feature map, including: A denoising network is used to denoise the feature map corresponding to the first feature extraction sub-network in the multi-level feature extraction network to obtain a denoised feature map. The denoised feature map and the feature maps corresponding to other feature extraction sub-networks in the multi-level feature extraction network are fused to obtain the fused feature map.

7. An image defect segmentation device, characterized in that, include: The acquisition module is used to acquire the target image and input the target image into a multi-level feature extraction network; The multi-level feature extraction network includes multiple feature extraction sub-networks connected in sequence; The extraction module is used to extract features from the feature map output by the previous feature extraction sub-network based on the sequence data generated by the previous feature extraction sub-network during the feature extraction process, thereby obtaining the feature map corresponding to the previous feature extraction sub-network; the sequence data is obtained by serializing the feature map obtained by the previous feature extraction sub-network during the feature extraction process. The fusion module is used to fuse the feature maps corresponding to each of the feature extraction sub-networks to obtain a fused feature map, and to determine the defect segmentation result of the target image based on the fused feature map; Specifically, the extraction module is used to process the feature map output by the previous feature extraction sub-network using the mapping calculation channel in the feature extraction sub-network to obtain the original information of the feature map; to process the feature map output by the previous feature extraction sub-network using the convolution calculation channel in the feature extraction sub-network to obtain the local features of the feature map; to process the feature map output by the previous feature extraction sub-network using the sequence calculation channel in the feature extraction sub-network based on the sequence data generated during the feature extraction process to obtain the global features of the feature map; and to generate the feature map corresponding to the feature extraction sub-network based on the original information, the local features, and the global features.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the image defect segmentation method as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image defect segmentation method as described in any one of claims 1 to 6.

Images