A machine learning model training method, an image processing method, and an electronic device

Abstract

The application relates to the technical field of computers, in particular to a machine learning model training method, an image processing method, a machine learning model training device, an image processing device, electronic equipment and a nonvolatile computer readable storage medium. In the technical scheme provided by the embodiment of the application, a target background distinguishing module about background region segmentation is introduced in the process of training the backbone network, so that the trained model can learn how to identify the background and the target region, and the two can be separated as much as possible, the precision of the model in the process of image processing for target contour extraction is improved, and clearer defect morphology and more accurate positioning results can be obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a machine learning model training method, an image processing method, a machine learning model training device, an image processing device, an electronic device and a nonvolatile computer readable storage medium. BACKGROUND

[0002] In the prior art, the weakly supervised defect recognition method based on image-level label mainly trains a classification network based on a CNN (Convolutional Neural Networks) to generate a class activation map as an initial pixel-level pseudo label. Since the initial pseudo label is relatively rough, a general fully connected CRF (Conditional Random Field) is used to post-process the initial pseudo label to obtain a more refined pseudo label, and the pseudo label is used to locate the target of the image.

[0003] Due to the downsampling operation in the network structure, the object edge of the generated class activation map is relatively rough, and accurate contour positioning of the target cannot be achieved. SUMMARY

[0004] To solve the problems in the prior art, the embodiments of the present application provide a machine learning model training method, an image processing method and an electronic device, which jointly optimize the backbone network and the target-background distinguishing module to guide the model training together, so as to obtain clearer defect morphology and more accurate positioning results.

[0005] To achieve the above purpose, the technical solutions adopted by the embodiments of the present application are as follows:

[0006] In a first aspect, a machine learning model training method is provided, the machine learning model comprising a backbone classification network and a target-background distinguishing module, the training method comprising: training the backbone classification network according to a target training round to obtain an initial backbone classification network, the backbone classification network being a network structure based on a global attention module; extracting a semantic feature map of a sample image according to the initial backbone classification network, the sample image containing a real class label; segmenting the semantic feature map using the target-background distinguishing module, and determining a segmentation loss function according to the segmentation result; training the initial backbone classification network according to the segmentation loss function until the initial backbone classification network is in a convergent state.

[0007] Further, the method further comprises: classifying the first sample image by using the backbone classification network, determining a classification loss function according to a classification result, and training the backbone classification network as an initial backbone classification network based on a target training round according to the classification loss function; and training the initial backbone classification network comprises: training the initial backbone classification network according to the classification loss function and the segmentation loss function.

[0008] Further, the classification of the first sample image by using the backbone classification network comprises: determining a semantic feature map of the first sample image by using a global attention module of the backbone classification network, and classifying the sample image by using a global average pooling layer of the backbone classification network according to the semantic feature map.

[0009] Further, the segmentation of the semantic feature map by using the target background distinguishing module comprises: performing variance processing on the semantic feature map to obtain a first-class activation map and a second-class activation map, the first-class activation map being a target region estimation of the semantic feature map, and the second-class activation map being a background region estimation of the semantic feature map; and performing weight assignment on pixels in a feature map corresponding to a real class to obtain a real feature map; and the determination of the segmentation loss function according to the segmentation result comprises: obtaining a target background distinguishing loss function according to the real feature map and the first-class activation map and the second-class activation map.

[0010] Further, the variance processing of the semantic feature map to obtain the first-class activation map and the second-class activation map comprises: collecting pixel points in the semantic feature map with a variance less than a preset variance threshold to obtain the first-class activation map; and collecting pixel points in the semantic feature map with a variance greater than or equal to the preset variance threshold to obtain the second-class activation map.

[0011] Further, the weight assignment on the pixels in the real class semantic feature map to obtain the real feature map comprises: multiplying the real class semantic feature map by a Sigmoid function to perform weight assignment on pixels of the second semantic feature map to obtain the real feature map.

[0012] Further, the extraction of the semantic feature map of the sample image by using the initial backbone classification network comprises: dividing the sample image into a plurality of image blocks; processing the plurality of image blocks by using the global attention module to determine a plurality of feature vectors; and determining the semantic feature map according to the plurality of feature vectors.

[0013] Further, the determining the semantic feature map according to the plurality of feature vectors comprises: obtaining an attention sequence about the first sample image based on the global attention module; and obtaining the semantic feature map by performing convolution processing on the attention sequence through a convolution kernel.

[0014] In a second aspect, an image processing method is provided, comprising: determining a semantic feature map and an attention feature map in a to-be-processed image by using a global attention module in the machine learning model, wherein the machine learning model is trained by the training method in any one of the preceding aspects; coupling the semantic feature map and the attention feature map semantically to obtain a class activation map; and obtaining a pseudo labeling box of a defect by performing a preset threshold processing on the class activation map.

[0015] Further, the determining the attention feature map in the to-be-processed image by using the global attention module in the machine learning model comprises: dividing the to-be-processed image into a plurality of image blocks; determining a plurality of feature vectors by processing the plurality of image blocks by using the global attention module; and determining the attention feature map according to the plurality of feature vectors.

[0016] Further, the dividing the to-be-processed image into a plurality of image blocks, processing the plurality of image blocks by using the global attention module, and determining a plurality of feature vectors comprises: dividing the to-be-processed image into a plurality of image block sequences of the same size in a two-dimensional form, obtaining a plurality of feature vectors based on embedding blocks from the plurality of image block sequences, and embedding a mark vector at the head of the plurality of feature vectors.

[0017] Further, the determining the attention feature map according to the plurality of feature vectors comprises: embedding a mark vector at the head of the plurality of feature vectors, and obtaining the attention feature map based on the mark vector.

[0018] Further, the obtaining the attention feature map based on the mark vector comprises: obtaining an attention vector corresponding to an attention matrix of the mark vector in the global attention mechanism; and converting the last dimension of the attention vector into a two-dimensional structure by matrix deformation to obtain the attention feature map.

[0019] Further, before the two-dimensional conversion of the attention vector, the method further comprises: obtaining a target attention vector by performing mean value processing on the attention vector in a multi-layer structure; and performing two-dimensional conversion on the last dimension of the target attention vector to obtain a two-dimensional structure, wherein the two-dimensional structure is the attention feature map.

[0020] In a third aspect, a training apparatus of a machine learning model is provided, and the training apparatus comprises: an initial training unit configured to train the backbone classification network according to a target training round to obtain an initial backbone classification network; a feature extraction unit configured to extract a semantic feature map of a sample image according to the initial backbone classification network; a segmentation unit configured to segment the semantic feature map by using the target background distinguishing module, and determine a segmentation loss function according to a segmentation result; and a training unit configured to train the initial backbone classification network according to the segmentation loss function until the initial backbone classification network is in a convergent state.

[0021] In a fourth aspect, an image processing apparatus is provided, and the image processing apparatus comprises: a feature extraction unit configured to determine a semantic feature map and an attention feature map in a to-be-processed image by using a global attention module in the machine learning model, the machine learning model being trained by using any one of the training methods described above; a coupling unit configured to semantically couple the semantic feature map and the attention feature map to obtain a class activation map; and an identification unit configured to obtain a pseudo labeling box of a defect by using a preset threshold on the class activation map.

[0022] In a fifth aspect, an electronic device is provided, and the electronic device comprises: a memory; and a processor coupled to the memory, the processor being configured to execute any one of the training methods described above or any one of the image processing methods described above based on instructions stored in the memory.

[0023] In a sixth aspect, a non-volatile computer-readable storage medium is provided, and the non-volatile computer-readable storage medium stores a computer program, the computer program being executed by a processor to implement any one of the training methods described above or any one of the image processing methods described above.

[0024] In the technical scheme provided by the embodiments of the present application, the target background distinguishing module related to background region segmentation is introduced in the process of training the backbone network, so that the trained model can learn how to distinguish the background region and the target region, and separate the two as much as possible, thereby improving the precision of target contour extraction in the image processing process of the model. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0026] The methods, systems, and / or programs of the various aspects described herein can be implemented on one or more computers. The computers can each include a processor for executing programs and instructions stored in memory. Where computers execute instructions, the execution can be in sequence or in parallel. The computers can be general purpose computers, special purpose computers, microcode, or any other programming hardware or architecture.

[0027] Figure 1 is a machine learning model training method flowchart provided by an embodiment of the application.

[0028] Figure 2 is an image processing method flowchart provided by an embodiment of the application.

[0029] Figure 3 is a machine learning model training device structure block diagram provided by an embodiment of the application.

[0030] Figure 4 is an image processing device structure block diagram provided by an embodiment of the application.

[0031] Figure 5 is an electronic device structure schematic diagram provided by an embodiment of the application. DETAILED DESCRIPTION

[0032] In order to better understand the above technical solutions, the technical solutions of the present application will be described in detail below through the drawings and specific embodiments. It should be understood that the specific features in the embodiments of the present application and the embodiments are detailed descriptions of the technical solutions of the present application, and are not limitations of the technical solutions of the present application. In the case of no conflict, the technical features in the embodiments of the present application and the embodiments can be combined with each other.

[0033] In the following detailed description, many specific details are set forth in order to provide a thorough understanding of the relevant teachings. However, it will be apparent to one skilled in the art that the present application can be practiced without these details. In other instances, well-known methods, procedures, systems, components, and / or circuits have been described at a relatively high level, without detail, in order to avoid unnecessarily obscuring aspects of the present application.

[0034] The flowcharts in the present application illustrate the execution processes performed by the system according to the embodiments of the present application. It should be clearly understood that the execution processes of the flowcharts can not be executed in sequence. On the contrary, these execution processes can be executed in reverse order or simultaneously. In addition, at least one other execution process can be added to the flowchart. One or more execution processes can be deleted from the flowchart.

[0035] Before the embodiments of the present application are further described in detail, the terms and terms involved in the embodiments of the present application are explained, and the terms and terms involved in the embodiments of the present application are applicable to the following explanations.

[0036] (1) Based on, for indicating the condition or state that the operation depends on, when the dependent condition or state is met, one or more operations performed can be real-time, or have a set delay; in the absence of special instructions, there is no limit to the order of execution of multiple operations performed.

[0037] (2) Based on, for indicating the condition or state that the operation depends on, when the dependent condition or state is met, one or more operations performed can be real-time, or have a set delay; in the absence of special instructions, there is no limit to the order of execution of multiple operations performed.

[0038] First, some terms that may appear in the embodiments of the present application are explained.

[0039] Object Detection: Each object in the picture is marked with a rectangular box, and the category of the object is given.

[0040] Feature map: image information obtained after convolution of image and filter; Feature map can be convolved with filter to generate new Feature map.

[0041] Attention feature map: refers to the feature map that is focused on the region containing the target object through the attention mechanism, and this region will have a stronger response.

[0042] RoIPooling: refers to pooling the corresponding region in the feature map into a fixed size feature map according to the position of the input rectangular box.

[0043] In the prior art, the weakly supervised defect recognition method based on image-level label mainly trains a classification network based on CNN (Convolutional Neural Networks, Convolutional Neural Networks) to generate a class activation map as an initial pixel-level pseudo label. Since the initial pseudo label is relatively rough, a general fully connected CRF (Conditional Random Field, Conditional Random Field) is used to post-process the initial pseudo label to obtain a more refined pseudo label, and the image is positioned through the pseudo label.

[0044] Then, the above processing method has the following problems.

[0045] The convolution operation in CNN has a limited receptive field and cannot effectively utilize the global information of the image, resulting in a class activation Figure 1Generally, only a partial region of an object can be contained. That is, a complete object cannot be well labeled in the pseudo label, thereby affecting the accuracy of semantic segmentation.

[0046] Due to operations such as downsampling in the network structure, the object edges of the generated class activation map are relatively rough, and accurate positioning of the target contour cannot be achieved.

[0047] To solve the above technical problems, the embodiment of the present application proposes a weakly supervised semantic classification network based on visual attention learning semantic relationship. The present disclosure introduces the visual Transformer structure into the weakly supervised semantic classification method, and by introducing a background distinguishing module in the training process of the semantic classification network, the trained semantic classification network can learn how to distinguish the background and target regions and separate them as much as possible.

[0048] It is worth noting that the embodiment of the present application mainly explains the Transformer structure, but it does not mean that it can only be applied to the Transformer structure. Other attention mechanism networks can also be applied.

[0049] In some embodiments, a classification network based on a visual Transformer, i.e., the backbone classification network in the embodiment of the present application, is used to extract the feature map of the image. Due to the global attention mechanism in the Transformer structure, the global object information in the image can be well utilized. Thus, the generated CAM (Class Activation Map, class activation map) can discover complete semantic objects, and better overcome the defects of CNN.

[0050] In some embodiments, the Transformer structure is further optimized by setting a background distinguishing module to guide the Transformer structure to further optimize the information of distinguishing the background of the image, learn the difference between the background region and the target region, and further improve the accuracy of the generated CAM.

[0051] For example, the technical scheme of the present disclosure can be implemented through the following embodiments.

[0052] Referring to Figure 1 , Figure 1 a flowchart showing some embodiments of the machine learning model training method of the present disclosure.

[0053] As Figure 2 shown, the machine learning model training method in the embodiment of the present application includes the following steps:

[0054] Step S110. Training the backbone classification network according to the target training round to obtain an initial backbone classification network.

[0055] Specifically, the backbone classification network in the embodiment of the present application is a network structure based on a global attention module.

[0056] In some embodiments, the initial backbone classification network is a network model after preliminary training of the backbone classification network, and is not a final network model. The end of training during this period can be set by setting the target training round.

[0057] The specific processing process includes:

[0058] A training sample set is constructed, wherein the training sample set contains a plurality of first sample images. The first sample images are input into the backbone classification network to obtain corresponding classification results, and a loss function is determined based on the classification results.

[0059] According to the classification loss function, the backbone classification network is trained to be an initial backbone classification network based on the set target training round.

[0060] It is worth noting that the initial backbone classification network obtained by the training process at this stage is not a target network model, and is not in a converged state. This initial backbone classification network is used for subsequent learning of segmentation results. Although its state is not in a converged state, its network is in a stable state.

[0061] In some embodiments, the backbone classification network processes the first sample image to obtain a classification result through the global attention module in the backbone classification network, extracts semantic feature maps of the first sample image, and then classifies the sample image based on the semantic feature maps using the global average pooling layer of the backbone classification network.

[0062] Before feature extraction of the first sample image, the first sample image needs to be divided into a plurality of image blocks, the global attention module is used to extract features of the plurality of image blocks, a plurality of feature vectors about the plurality of image blocks are determined, and a semantic feature map is determined according to the plurality of feature vectors.

[0063] In some embodiments, the extraction process is to obtain an attention sequence about the first sample image through the global attention module, and to obtain a semantic feature map by convolution processing of the attention sequence through a convolution kernel.

[0064] Specifically, the feature extraction process is to randomly scale and randomly crop any one sample image in the training set to obtain an input image with a size of , and then divide the input image with a size of into image blocks.

[0065] The input image with a size of The feature vector of length D corresponding to these N image blocks is obtained through the Patch Embedding (embedding block) in the global attention module , where i represents the sequence number of the data block. And add a CLS marker feature of length D , the image block label and CLS label are represented as a feature vector as follows ,in Size .

[0066] Will Input into L cascaded Transformer blocks connected end to end In the example above, we get the updated feature encoding .

[0067] The last layer of Transformer block Output image patch features Resize to , where D is the feature embedding (embedding block) length, is the number of image patch features.

[0068] Use a 3 × 3 convolution kernel with D input channels and C output channels. Perform convolution operation on the image block features to obtain the semantic feature maps corresponding to C categories .

[0069] Specifically, the calculation method of the semantic feature map for the i-th category is as follows:

[0070] ,in Represents a convolution operation.

[0071] Through the above processing, a semantic feature map of the first sample image is obtained, the semantic feature map is processed based on the global average pooling layer to obtain a classification result, and the classification loss function of the backbone classification network is determined according to the classification result.

[0072] The loss function for classification is:

[0073] , where gt represents the true defect category, represents the semantic representation of the k-th image patch feature on the i-th category.

[0074] Specifically, the backbone classification network is trained for a target number of training rounds based on a determined classification loss function to obtain an initial backbone classification network.

[0075] Step S120. Extracting a semantic feature map of the sample image according to the initial backbone classification network.

[0076] In the embodiment of the present application, the sample image contains a real class label.

[0077] For the retraining of the sample image in step S120 to the initial backbone classification network obtained in step S110, the sample image in this process contains a real class label, which reduces the problem of high computational cost caused by uncertain class in the process of segmenting the target region and the background region in the image.

[0078] The extraction process of the semantic feature map in the embodiment of the present application is the same as step S110, except that the backbone classification network in this process is the initial backbone classification network, that is, the network after iterative training of the backbone classification network, the hyperparameters of each module in the network are different, but the features of the output semantic feature map are the same.

[0079] The above process will not be described in detail in the embodiment of the present application.

[0080] Step S130. Segmenting the semantic feature map using the target-background distinguishing module, and determining a segmentation loss function according to the segmentation result.

[0081] Due to the defects of the class activation method of the visual Transformer itself, the C class semantic feature maps extracted in step S120 often have the problem of irrelevant background activation, which is manifested as low segmentation accuracy of the target detection region and the background region on the model application side. In order to solve this technical problem, the embodiment of the present application learns the target region and the background region for the initial backbone classification network after preliminary training, and learns by setting the target-background distinguishing module in combination with the initial backbone classification network, so that the finally trained network model can realize the rapid segmentation of the target region and the background region.

[0082] The application of the target-background module is only on the training side, and needs to be introduced under the stable state of the backbone classification network. The purpose of such setting is to reduce the interference of the introduction of the target-background module on the training process of the backbone classification network. When the backbone classification network tends to be stable, the backbone classification network will not change the hyperparameters in the model due to the introduction of the target-background module, thereby affecting the finally trained network model.

[0083] Specifically, for the segmentation of the semantic feature map by the target background distinguishing module in the embodiment of the application, it is assumed that the activation value of the target region has obvious differences in the activation results of each channel, and shows high activation in at least one target category, while the activation value of the background region in all category channels should be uniform without obvious differences. The distribution of the activation value on the semantic feature map is determined to determine the target region and the background region.

[0084] And the acquisition of the activation value is obtained by performing variance processing on the C-layer semantic feature map along the category channel direction to obtain a first type of activation map and a second type of activation map on the semantic feature map. The first type of activation map is an estimation of the target region of the semantic feature map, and the second type of activation map is an estimation of the background region of the semantic feature map.

[0085] Specifically, for the determination of the estimation, it is determined whether the pixel variance obtained by performing variance processing on the semantic feature map is less than a preset variance threshold. The pixel points in the semantic feature map whose pixel variance is less than the preset variance threshold are collected to obtain the first type of activation map, and the pixel points in the semantic feature map whose pixel variance is greater than or equal to the preset variance threshold are collected to obtain the second type of activation map. The variance of the C-layer semantic feature map is calculated along the category channel direction When the variance is less than the preset variance threshold, it is identified as a background region estimation , and other regions are identified as target region estimations , as shown in the following formula:

[0086] , ; wherein is a preset variance threshold.

[0087] And in order to improve the segmentation accuracy, the pixels in the real category semantic feature map are weighted to obtain a real feature map. Specifically, this process is performed by multiplying the semantic feature map by a Sigmoid function, and the pixels in the second semantic feature map are weighted to obtain a real feature map. That is, the C category semantic feature maps obtained in step S120 are input feature maps , and the semantic feature map corresponding to the real category gt is processed by function to obtain a real feature map . .

[0088] The target background distinguishing loss function is obtained based on the real feature map, the first type of activation map and the second type of activation map obtained above, wherein the definition of the target background distinguishing loss function is as follows:

[0089] .

[0090] Step S140. The initial backbone classification network is trained according to the segmentation loss function until the initial backbone classification network is in a convergent state.

[0091] The initial backbone classification network is subsequently trained according to the target background distinguishing loss function obtained in step S130, and because the initial backbone classification network has not yet converged in the training for the classification task, the initial backbone classification network is trained by the target background distinguishing loss function in the training process, and the classification loss function in step S110 is also added to form an overall loss function for training the initial backbone classification network.

[0092] The overall loss function is defined as shown in the following formula:

[0093] .

[0094] The initial backbone network is trained by the overall loss function until it converges to obtain the target machine learning model.

[0095] In the above embodiments, by introducing the segmentation function for segmenting the image background region and the target region in the training stage, the trained backbone classification network can enhance the accurate positioning ability of the target contour. Through the joint optimization of the classification loss function and the segmentation loss function, the model training is guided together, so that the model can learn how to identify the background and target regions, and accurately segment the background region and the target region.

[0096] In the model training side, the embodiments of the present application introduce the target background distinguishing module to the backbone classification network in the training process, so that the finally trained backbone classification network can accurately capture the contour information.

[0097] For the trained machine learning model, the embodiments of the present application also provide an image processing method, which is mainly applied to identifying and positioning defects in an image.

[0098] The machine learning model provided by the embodiments of the present application is mainly used in the object surface defect detection scene, and in some embodiments, it is mainly used for identifying and positioning defects on the surface of steel. In other embodiments, it can also be used for identifying defects on the surface of glass materials, plastic materials and other smooth materials. In some other embodiments, it can also be used for identifying defects on the surface of rough and granular materials.

[0099] Figure 2 A flowchart showing some embodiments of the image processing method disclosed by the embodiments of the present application.

[0100] like Figure 3 As shown, this image processing method includes the following steps:

[0101] Step S210. Utilize the global attention module in the machine learning model to determine the semantic feature map and attention feature map in the image to be processed.

[0102] In the embodiment of the present application, the machine learning model used is a machine learning model obtained by training through steps S110 to S140.

[0103] Among them, for the extraction of the semantic feature map, reference can be made to the image feature processing part in step S120, and no further details will be given.

[0104] The acquisition of the attention feature map is achieved by acquiring the attention feature map and generating an attention map with unknown category semantics through the attention branch in the global attention module for the attention vector of the CLS feature.

[0105] Specifically, the image to be processed is divided into multiple image blocks, the global attention module is used to process the multiple image blocks, multiple feature vectors are determined, and the attention feature map is determined based on the multiple feature vectors.

[0106] The plurality of feature vectors are obtained by evenly dividing the image to be processed into a plurality of two-dimensional image block sequences of the same size, and obtaining the plurality of feature vectors based on the embedding blocks of the plurality of image block sequences.

[0107] Among them, a tag vector is embedded in the head of multiple feature vectors, and an attention vector corresponding to the attention matrix in the global attention mechanism based on the embedded tag vector is obtained, and the last dimension of the attention vector is converted into a two-dimensional structure through matrix deformation, and the obtained two-dimensional structure is the attention feature map.

[0108] In some embodiments, the Transformer block has a multi-layer structure, and the acquisition of the attention vector needs to take into account the fact that the attention vectors of each layer of the multi-layer structure are different. In order to obtain the dependency relationship from the cascaded Transformer blocks, the attention vector to be processed is processed by demeaning each layer or the obtained attention vector to obtain the target attention vector, and the last dimension of the target attention vector is converted into a two-dimensional structure, and the obtained two-dimensional structure is used as the attention feature map.

[0109] Step S220: Semantically couple the semantic feature map and the attention feature map to obtain a class activation map.

[0110] The obtained semantic feature map is coupled with the attention feature map by element multiplication to obtain a fused class activation result , wherein the class activation result is an activation map corresponding to different results.

[0111] Step S230. A pseudo-labeled box of the defect is obtained by presetting a threshold for the class activation map.

[0112] Based on the class activation map obtained in step S220, the gt layer fusion activation map of M is taken , and threshold segmentation is performed on to obtain a predicted target position.

[0113] Because the input image to be recognized has undergone scaling processing before processing, the obtained fusion activation map needs to be scaled when performing mapping restoration at the end, that is, the prediction result of the fusion activation map is mapped to the original image size by scaling. .

[0114] In the technical scheme provided by the embodiments of the present application, by introducing the target background distinguishing module for background region segmentation in the training process of the backbone network, the trained model can learn how to distinguish the background and the target region and separate the two as much as possible, thereby improving the accuracy of the model in extracting the target contour in the image processing process.

[0115] Referring to Figure 3 , a block diagram of some embodiments of a machine learning model training apparatus is shown.

[0116] As shown in Figure 4 , the machine learning model training apparatus 300 includes:

[0117] An initial training unit 310 is configured to train the backbone classification network according to a target training round to obtain an initial backbone classification network.

[0118] A feature extraction unit 320 is configured to extract a semantic feature map of a sample image according to the initial backbone classification network.

[0119] A segmentation unit 330 is configured to segment the semantic feature map by using the target background distinguishing module and determine a segmentation loss function according to a segmentation result.

[0120] A training unit 340 is configured to train the initial backbone classification network according to the segmentation loss function until the initial backbone classification network is in a convergent state.

[0121] In some embodiments, the training unit 340 is further configured to determine a classification loss function, and train the initial backbone classification network according to the classification loss function and the segmentation loss function.

[0122] In some embodiments, the determination of the classification loss function is performed by classifying the first sample image using the backbone classification network, and determining according to the classification result.

[0123] In some embodiments, the classification of the first sample image is performed by determining a semantic feature map of the first sample image by a global attention module of the backbone classification network, and classifying the sample image using a global average pooling layer of the backbone classification network according to the semantic feature map.

[0124] In some embodiments, the determination of the segmentation loss function in the segmentation unit 330 is based on a segmentation result, wherein the segmentation result is a target region and a background region on the semantic feature map. The determination process is as follows: obtaining a first class activation map and a second class activation map by segmenting the semantic feature map and by variance processing the semantic feature map, the first class activation map is an estimation of the target region with respect to the semantic feature map, and the second class activation map is an estimation of the background region with respect to the semantic feature map; obtaining a real feature map by assigning weights to pixels in a feature map corresponding to a real class.

[0125] In some embodiments, the first class activation map and the second class activation map respectively represent the target region and the background region, and the determination process is based on variance processing of pixels in the semantic feature map, specifically, collecting pixel points in the semantic feature map with a variance less than a preset variance threshold to obtain the first class activation map; and collecting pixel points in the semantic feature map with a variance greater than or equal to the preset variance threshold to obtain the second class activation map.

[0126] In some embodiments, the real feature map is obtained by processing a real class semantic feature map, wherein the real class semantic feature map is obtained by performing semantic feature processing on an image with a real class label.

[0127] In some embodiments, the real feature map is obtained by multiplying the real class semantic feature map by a Sigmoid function, and assigning weights to pixels of the second semantic feature map to obtain the real feature map.

[0128] Figure 4 Some embodiments of the image processing apparatus of the present application are shown in the block diagrams.

[0129] As shown in Figure 5 The image processing apparatus 400 includes:

[0130] The feature extraction unit 410 is configured to determine semantic feature maps and attention feature maps in the image to be processed by using a global attention module in the machine learning model trained by the training method in steps S110-S140.

[0131] The coupling unit 420 is configured to semantically couple the semantic feature maps and the attention feature maps to obtain a class activation map.

[0132] The recognition unit 430 is configured to obtain a pseudo labeling box of a defect by performing a preset threshold on the class activation map.

[0133] Referring to Figure 1 , an electronic device provided by an embodiment of the present application is shown. The electronic device 500 includes a memory 501 and a processor 502 coupled to the memory 501. The processor 502 is configured to execute a machine learning model training method or an image processing method in any one of the preceding embodiments based on instructions stored in the memory 501.

[0134] The electronic device can further include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input / output interfaces 505, one or more keyboards 506, and the like.

[0135] In one specific embodiment, the electronic device includes a memory and one or more programs, wherein one or more programs are stored in the memory, and the one or more programs can include one or more modules, and each module can include a series of computer executable instructions for guiding the puncture device in the device, and the one or more processors are configured to execute the one or more programs, and the one or more programs include computer executable instructions for:

[0136] training the backbone classification network according to the target training round to obtain an initial backbone classification network;

[0137] extracting semantic feature maps of sample images according to the initial backbone classification network;

[0138] segmenting the semantic feature maps by using the target background distinguishing module, and determining a segmentation loss function according to a segmentation result;

[0139] training the initial backbone classification network according to the segmentation loss function until the initial backbone classification network is in a convergent state.

[0140] and determining semantic feature maps and attention feature maps in the image to be processed by using a global attention module in the machine learning model;

[0141] The semantic feature map and the attention feature map are coupled semantically to obtain a class activation map.

[0142] The class activation map is subjected to a preset threshold to obtain a pseudo label frame of a defect.

[0143] The various components of the processor will be specifically introduced as follows:

[0144] In this embodiment, the processor is an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the application, for example, one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0145] Alternatively, the processor can perform various functions by running or executing software programs stored in the memory and calling data stored in the memory, for example, executing the above ​ method.

[0146] In a specific implementation, as an embodiment, the processor can include one or more microprocessors.

[0147] The memory is used to store software programs for executing the schemes of the application and is controlled by the processor to execute, and the specific implementation can refer to the above method embodiments, which will not be described here.

[0148] Optionally, the memory can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions; a random access memory (RAM), or other type of dynamic storage device that can store information and instructions; an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by the computer, but is not limited to this. The memory can be integrated with the processor or exist independently and be coupled to the processing unit through the interface circuit of the processor, and the embodiments of the present application are not limited in this regard.

[0149] It should be noted that the structure of the processor shown in the embodiments does not constitute a limitation on the device, and the actual device can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0150] In addition, the technical effects of the processor can refer to the technical effects of the methods described in the above method embodiments, which will not be described here.

[0151] It should be understood that the processor in the embodiments of the present application can be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), ready programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0152] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0153] The above-described embodiments can be implemented in whole or in part by software, hardware (e.g., circuitry), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center through a wired (e.g., infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0154] In this application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0155] It should be understood that the size of the sequence number of the above-mentioned processes in various embodiments of the present application does not mean the order of execution, and the execution order of the processes should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0156] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0157] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0158] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0159] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0160] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0161] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the essential part or part of the technical solutions that make contributions to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0162] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A machine learning model training method, characterized in that: The machine learning model includes a backbone classification network and a target-background distinction module, and the training method includes: Training the backbone classification network according to a target training round to obtain an initial backbone classification network, wherein the backbone classification network is a network structure based on a global attention module; and extracting a semantic feature map of a sample image according to the initial backbone classification network, wherein the sample image includes a true category label; comprising: determining a semantic feature map of a first sample image using a global attention module of the backbone classification network; and classifying the sample image according to the semantic feature map using a global average pooling layer of the backbone classification network; The target-background distinction module is used to segment the semantic feature map, and a segmentation loss function is determined according to the segmentation result; the method includes: performing variance processing on the semantic feature map to obtain a first-class activation map and a second-class activation map, wherein the first-class activation map is an estimate of the target region of the semantic feature map, and the second-class activation map is an estimate of the background region of the semantic feature map; and weighting pixels in the feature map corresponding to the true category to obtain a true feature map; According to the segmentation loss function, the initial backbone classification network is trained until the initial backbone classification network is in a converged state; including: obtaining a target background distinction loss function according to the real feature map and the first class activation map and the second class activation map; The method also includes: using the backbone classification network to classify the first sample image, and determining a classification loss function based on the classification result; and according to the classification loss function, training the backbone classification network as an initial backbone classification network based on a target training round; the training of the initial backbone classification network includes: training the initial backbone classification network according to the classification loss function and the segmentation loss function.

2. The machine learning model training method according to claim 1, characterized in that The performing variance processing on the semantic feature map to obtain the first type activation map and the second type activation map includes: The pixel points in the semantic feature map whose pixel variance is less than a preset variance threshold are aggregated to obtain the first type of activation map; The pixel points whose pixel variances in the semantic feature map are greater than or equal to a preset variance threshold are aggregated to obtain the second type of activation map.

3. The machine learning model training method according to claim 1, characterized in that The pixels in the real category semantic feature map are weighted to obtain the real feature map, including: The true category semantic feature map is multiplied by a Sigmoid function, and weights are assigned to the pixels of the second category activation map to obtain a true feature map.

4. The machine learning model training method according to any one of claims 1 to 3, characterized in that: The step of extracting a semantic feature map of a sample image according to the initial backbone classification network includes: Dividing the sample image into a plurality of image blocks; Processing the plurality of image blocks using the global attention module to determine a plurality of feature vectors; The semantic feature map is determined according to the multiple feature vectors.

5. The machine learning model training method according to claim 4, characterized in that Determining the semantic feature graph according to the multiple feature vectors includes: Obtaining an attention sequence for the first sample image based on the global attention module; The attention sequence is convolved by a convolution kernel to obtain the semantic feature map.

6. An image processing method, characterized in that: The method comprises: Determining a semantic feature map and an attention feature map in the image to be processed using a global attention module in the machine learning model, wherein the machine learning model is trained using the training method according to any one of claims 1 to 5; Semantically coupling the semantic feature map and the attention feature map to obtain a class activation map; A pseudo-labeled box of the defect is obtained by using a preset threshold on the class activation map.

7. The image processing method according to claim 6, characterized in that: Determining an attention feature map in an image to be processed by utilizing a global attention module in the machine learning model includes: Dividing the image to be processed into a plurality of image blocks; Processing the plurality of image blocks using the global attention module to determine a plurality of feature vectors; Determine the attention feature map based on the multiple feature vectors.

8. The image processing method according to claim 7, wherein: The step of dividing the image to be processed into a plurality of image blocks, processing the plurality of image blocks using the global attention module, and determining a plurality of feature vectors includes: The image to be processed is evenly divided into a plurality of image block sequences in a two-dimensional form with the same size, and a plurality of feature vectors are obtained based on the embedding blocks of the plurality of image block sequences.

9. The image processing method according to claim 8, characterized in that: Determining the attention feature map according to the multiple feature vectors includes: A tag vector is embedded in the heads of the plurality of feature vectors, and the attention feature map is obtained based on the tag vector.

10. The image processing method according to claim 9, wherein: The obtaining the attention feature map based on the tag vector includes: Obtaining an attention vector corresponding to the attention matrix of the tag vector in the global attention module; The last dimension of the attention vector is converted into a two-dimensional structure through matrix transformation, and the obtained two-dimensional structure is the attention feature map.

11. The image processing method according to claim 10, wherein: Before performing two-dimensional conversion on the attention vector, it also includes obtaining the attention vector in the multi-layer structure and taking the average to obtain the target attention vector, and performing two-dimensional conversion based on the last dimension of the target attention vector to obtain a two-dimensional structure. The obtained two-dimensional structure is the attention feature map.

12. A training device for a machine learning model, characterized in that: Used to perform the training method according to claim 1, comprising: The initial training unit is used to train the backbone classification network according to the target training round to obtain an initial backbone classification network A feature extraction unit, configured to extract a semantic feature map of a sample image based on the initial backbone classification network; a segmentation unit, configured to segment the semantic feature map using the target-background distinction module, and determine a segmentation loss function according to the segmentation result; A training unit is used to train the initial backbone classification network according to the segmentation loss function until the initial backbone classification network is in a converged state.

13. An image processing device, characterized in that: include: a feature extraction unit, configured to determine a semantic feature map and an attention feature map in the image to be processed using a global attention module in the machine learning model, wherein the machine learning model is trained using the training method according to any one of claims 1 to 5; A coupling unit, configured to semantically couple the semantic feature map and the attention feature map to obtain a class activation map; The recognition unit is used to obtain a pseudo-labeled box of the defect by using a preset threshold value for the class activation map.

14. An electronic device comprising: Memory; and A processor coupled to the memory, the processor being configured to execute the training method according to any one of claims 1 to 5, or the image processing method according to any one of claims 6 to 11, based on instructions stored in the memory.

15. A non-volatile computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the training method according to any one of claims 1 to 5 or the image processing method according to any one of claims 6 to 11 is implemented.

Images