Image classification method and device, computer device, readable storage medium and program product

Abstract

The application relates to an image classification method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: obtaining an object detection result of an original image, wherein the object detection result comprises position information of each object in the original image and an object category corresponding to each object; obtaining a mask corresponding to each object category according to the object detection result; performing mixing processing on the masks corresponding to each object category to obtain a mixed mask, wherein the mixed mask is used to represent distribution information of the objects in the original image; obtaining a foreground-background segmentation map corresponding to the original image according to the object detection result; and obtaining an enhanced feature map of the original image according to the original image, an embedded representation of the mixed mask and the foreground-background segmentation map, wherein the enhanced feature map is used to classify the original image to obtain an image category of the original image. The method can improve the accuracy of image classification.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image classification method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Technology

[0002] Image classification is one of the tasks of computer vision technology in the field of artificial intelligence. Image classification refers to the process of classifying images into different categories based on their features. For example, images can be classified into different categories such as human faces, animals, and buildings.

[0003] Image classification methods provided in related technologies are feature-based methods that use computer vision techniques to extract image features and apply them to image classification tasks. Common methods include using filters, color histograms and shape descriptors, or neural networks to extract image features based on texture, color, and shape.

[0004] However, these methods rely solely on image features, resulting in poor accuracy in image classification. Summary of the Invention

[0005] Therefore, it is necessary to provide an image classification method, apparatus, computer device, computer-readable storage medium, and computer program product to address the aforementioned technical problems and improve the accuracy of image classification.

[0006] Firstly, this application provides an image classification method, including:

[0007] Obtain object detection results from the original image, the object detection results including the location information of each object in the original image and the corresponding object category of each object;

[0008] Based on the object detection results, a mask image corresponding to each object category is obtained;

[0009] The mask images corresponding to each object category are blended to obtain a blended mask image, which is used to represent the distribution information of objects in the original image.

[0010] Based on the object detection results, a foreground / background segmentation map corresponding to the original image is obtained;

[0011] An enhanced feature map of the original image is obtained based on the original image, the embedded representation of the hybrid mask map, and the foreground / background segmentation map. The enhanced feature map is used to classify the original image to obtain the image category of the original image.

[0012] Secondly, this application also provides an image classification apparatus, comprising:

[0013] The detection result acquisition module is used to acquire the object detection results of the original image. The object detection results include the location information of each object in the original image and the corresponding object category of each object.

[0014] The detection result encoding module is used to obtain the mask image corresponding to each object category based on the object detection result;

[0015] The detection result encoding module is also used to perform a mixing process on the mask images corresponding to each object category to obtain a mixed mask image, which is used to characterize the distribution information of objects in the original image;

[0016] The detection result encoding module is also used to obtain the foreground and background segmentation map corresponding to the original image based on the object detection result;

[0017] An image classification module is used to obtain an enhanced feature map of the original image based on the original image, the embedding representation of the hybrid mask map, and the foreground / background segmentation map. The enhanced feature map is used to classify the original image to obtain the image category of the original image.

[0018] In some embodiments, the detection result encoding module is further configured to determine the total number of object categories based on the corresponding object categories of each object; set multiple different pixel values ​​according to the total number of object categories; for a target object category, determine the object pixels belonging to the target object category and other pixels not belonging to the target object category in the original image based on the location information, set the pixel value of the object pixels belonging to the target object category to one of the multiple different pixel values, and set the pixel values ​​of the other pixels to background pixel values ​​to obtain a mask image corresponding to the target object category, wherein the target object category is any object category in the object detection result, and the pixel values ​​of object pixels belonging to different object categories are different.

[0019] In some embodiments, the detection result encoding module is further configured to obtain the category weights of each object category; and to perform a weighted average of the mask images corresponding to each object category according to the category weights to obtain a hybrid mask image.

[0020] In some embodiments, the detection result encoding module is further configured to determine the region where the object is located in the original image based on the location information of each object; set the pixel value of the pixel point in the region where the object is located in the original image as the foreground pixel value; set the pixel value of the pixel point in the original image other than the region where the object is located as the background pixel value, thereby obtaining the foreground and background segmentation map corresponding to the original image.

[0021] In some embodiments, the image classification module is further configured to extract features from the original image to obtain a feature map of the original image; perform embedding processing on the hybrid mask to obtain an embedded representation of the hybrid mask, wherein the hybrid mask is used to represent the distribution information of objects in the original image; concatenate the feature map with the embedded representation of the hybrid mask to obtain a concatenated feature; and fuse the foreground / background segmentation map with the concatenated feature to obtain an enhanced feature map of the original image.

[0022] In some embodiments, the image classification module is further configured to segment the feature map, the foreground / background segmentation map, and the embedding representation of the hybrid mask map into sub-blocks to obtain feature sub-blocks, foreground / background sub-blocks, and embedding representation sub-blocks corresponding to each local region in the original image, wherein the embedding representation sub-blocks are used to indicate the distribution information of objects in the corresponding local region; to concatenate the feature sub-blocks and embedding representation sub-blocks corresponding to the same local region to obtain concatenated sub-blocks; to calculate the object coverage of each local region based on the foreground / background sub-blocks corresponding to each local region; and to obtain the enhanced feature map of the original image based on the object coverage and the concatenated sub-blocks using a self-attention mechanism.

[0023] In some embodiments, the image classification module is further configured to, for each stitched sub-block, convert the stitched sub-block into a query vector, a key vector, and a value vector based on three different linear transformation matrices in the self-attention network; determine the mask weight of each local region based on the object coverage of each local region; obtain the attention score of each local region based on the query vector, key vector, and mask weight of each local region; and weight and sum the value vectors of each local region according to the attention scores to obtain the enhanced feature map of the original image.

[0024] In some embodiments, the detection result acquisition module is further configured to acquire an original image, input the original image into an object detection model, and perform object detection on the original image through the object detection model to obtain an object detection result.

[0025] In some embodiments, the image classification module is further configured to input the original image, the embedding representation of the hybrid mask, and the foreground / background segmentation map into a trained image classification model; obtain an enhanced feature map of the original image based on the original image, the embedding representation of the hybrid mask, and the foreground / background segmentation map through the trained image classification model; and obtain an image classification result of the original image based on the enhanced feature map of the original image.

[0026] In some embodiments, the image classification device further includes:

[0027] The training module is used to acquire sample images, object detection results of the sample images, and labeled image categories; based on the object detection results, obtain a hybrid mask map and a foreground / background segmentation map of the sample images; acquire an initial image classification model; input the sample images, the embedding representation of the hybrid mask map of the sample images, and the foreground / background segmentation map into the initial image classification model; output the image category prediction probability of the sample images through the initial image classification model; adjust the model parameters of the initial image classification model based on the image category prediction probability of the sample images and the labeled image categories, until the trained image classification model is obtained when the training stopping condition is met.

[0028] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described image classification method.

[0029] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described image classification method.

[0030] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described image classification method.

[0031] The aforementioned image classification method, apparatus, computer device, computer-readable storage medium, and computer program product acquire object detection results of the original image, the object detection results including the location information of each object in the original image and the corresponding object category of each object; based on the object detection results, obtain a mask image corresponding to each object category; perform a mixing process on the mask images corresponding to each object category to obtain a mixed mask image, which is used to represent the distribution information of objects in the original image; obtain a foreground and background segmentation map corresponding to the original image based on the object detection results; and obtain an enhanced feature map of the original image based on the embedding representation of the original image, the mixed mask image, and the foreground and background segmentation map, which is used to classify the original image to obtain the image category of the original image.

[0032] Since the object detection results provide the location information of various objects in the original image, a hybrid mask can be obtained based on the location information of various objects. The hybrid mask is used to represent the distribution information of objects in the original image, and provides additional information such as the area, quantity, location, category and overlap of objects in the original image. The foreground and background segmentation map provides the location information of objects in the original image. By combining the embedding representation of the original image, the hybrid mask, and the foreground and background segmentation map, the original image and the above-mentioned additional information can be encoded into the enhancement feature map. The enhancement feature map is obtained by combining the original image and the above-mentioned additional information, so that the image category obtained by classifying the original image based on the enhancement feature map is more accurate. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a diagram illustrating the application environment of an image classification method in one embodiment;

[0035] Figure 2 This is a flowchart illustrating an image classification method in one embodiment;

[0036] Figure 3 This is a schematic diagram of the mask images corresponding to various object categories obtained in one embodiment;

[0037] Figure 4 This is a schematic diagram of a hybrid mask pattern in one embodiment;

[0038] Figure 5 This is a schematic diagram of the foreground and background segmentation map corresponding to the original image in one embodiment;

[0039] Figure 6 This is a schematic diagram illustrating the extraction of an embedded representation of a hybrid mask image via an embedded network in one embodiment.

[0040] Figure 7 This is a schematic diagram of the process for obtaining the enhanced feature map of the original image in one embodiment;

[0041] Figure 8 This is a flowchart illustrating the process of obtaining the enhanced feature map of the original image in a specific embodiment.

[0042] Figure 9 This is a schematic diagram of the input and output of an image classification model in one embodiment;

[0043] Figure 10 This is a flowchart illustrating an image classification method in another embodiment;

[0044] Figure 11 This is a structural block diagram of an image classification device in one embodiment;

[0045] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] Convolutional Neural Network (CNN): A deep learning architecture that automatically extracts features from images through convolutional layers, and is often used for visual tasks such as image classification and detection.

[0048] Self-Attention is a mechanism that allows models to consider the relationships between different positions within a sequence when processing sequential data, which helps to capture long-distance dependency information.

[0049] Feature Extraction: The process of extracting useful information from raw data. This information can represent important features or attributes of the data, so as to facilitate further analysis or model training.

[0050] Embedding: The process of converting discrete or continuous data items (such as words or pixel blocks) into low-dimensional and dense vector representations that capture the intrinsic properties of the data items.

[0051] Patch: A local region segmented from an image, used to extract local features in a deep learning model.

[0052] In related technologies, several image feature-based classification methods already exist for image classification problems. These methods typically use computer vision techniques to extract image features and apply them to classification tasks. One common approach is to classify based on image features such as texture, color, and shape. These image features can be extracted using techniques such as filters, color histograms and shape descriptors, or neural networks.

[0053] When image classification is applied to specific fields, relying solely on image features for classification can lead to inaccurate classification. For example, in industrial quality inspection, defect detection results for various defects in part images are typically available, providing information such as the location and category of the defects. However, current technologies fail to fully utilize this additional information to improve the accuracy of part image classification, failing to accurately categorize part images into predefined image categories such as scrapped parts, parts awaiting repair, and parts not requiring repair. Similarly, in scene recognition, target detection results for various objects in a scene image are typically available, providing information such as the location and category of each object (e.g., identifying the location of pedestrians, vehicles, and roads). However, current technologies fail to fully utilize this additional information to improve the accuracy of scene classification, failing to accurately categorize scene images into predefined image scene categories such as outdoor street scenes and indoor shopping mall scenes.

[0054] The image classification method provided in this application comprehensively utilizes image features and other additional information to better classify images. By combining image features with other additional information, such as object location information, category information, area, quantity, etc., and integrating information from multiple dimensions, the accuracy of classification can be improved, and it can play a role in application fields such as industrial quality inspection and scene recognition.

[0055] The image classification method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. In some embodiments, server 104 can obtain object detection results of the original image, including the location information of each object in the original image and the corresponding object category; based on the object detection results, obtain a mask map corresponding to each object category; perform a mixing process on the mask maps corresponding to each object category to obtain a mixed mask map, which is used to characterize the distribution information of objects in the original image; obtain a foreground / background segmentation map corresponding to the original image based on the object detection results; obtain an enhanced feature map of the original image based on the embedding representation of the original image, the mixed mask map, and the foreground / background segmentation map, which is used to classify the original image to obtain the image category of the original image. Optionally, server 104 can classify the original image based on the enhanced feature map to obtain the image category of the original image. Optionally, server 104 can feed back the image category of the original image to terminal 102. Of course, in some embodiments, the image classification method provided in the embodiments of this application can also be executed by the terminal 102 to obtain the image category.

[0056] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0057] The image method provided in this application embodiment can be applied to various application scenarios.

[0058] For example, in the field of industrial quality inspection, the image method provided in the embodiments of this application can be applied. Specifically, the computer device can acquire the defect detection results of the part image. The defect detection results include the location information of each defect in the part image and the corresponding defect category of each defect. The computer device can obtain the mask image corresponding to each defect category based on the defect detection results. The mask images corresponding to each defect category are mixed to obtain a mixed mask image. The mixed mask image is used to characterize the distribution information of defects in the part image. The computer device obtains the foreground and background segmentation map corresponding to the part image based on the defect detection results. The computer device obtains the enhanced feature map of the part image based on the part image, the embedded representation of the mixed mask image and the foreground and background segmentation map. The computer device classifies the part image based on the enhanced feature map to obtain the image category of the part image. The image category is one of scrapped part image, part image to be repaired and part image not to be repaired.

[0059] For example, in the field of scene recognition, the image method provided in the embodiments of this application can be applied. Specifically, the computer device can acquire the object detection results of the scene image to be classified. The object detection results include the position information of each object in the scene image and the corresponding object category of each object. Based on the object detection results, a mask image corresponding to each object category is obtained. The mask images corresponding to each object category are mixed to obtain a mixed mask image. Based on the object detection results, a foreground and background segmentation image corresponding to the original image is obtained. An enhanced feature map of the original image is obtained based on the embedding representation of the original image, the mixed mask image, and the foreground and background segmentation image. The computer device classifies the scene image based on the enhanced feature map to obtain the image category of the scene image. The image category is one of the preset image scene categories such as outdoor street scene image and indoor shopping mall scene image.

[0060] In one exemplary embodiment, such as Figure 2 As shown, an image classification method is provided, which can be applied to... Figure 1 The following steps are used as an example of computer equipment (server or terminal), including steps 202 to 210. Wherein:

[0061] Step 202: Obtain the object detection results of the original image. The object detection results include the location information of each object in the original image and the corresponding object category of each object.

[0062] The original image is the image to be classified. The object detection result of the original image is the result obtained by performing object detection on the original image, including the location information of each object in the original image and the corresponding object category. For example, in the field of industrial quality inspection, the original image can be a part image obtained by image acquisition of industrial parts, and correspondingly, the object detection result of the original image is the location information of each defect in the part image and the defect category of each defect. In the field of scene recognition, the original image can be a scene image obtained by image acquisition of a real-time scene, and correspondingly, the object detection result of the original image is the location information of different object categories such as pedestrians, buildings, and streets in the scene image and the corresponding object categories. Of course, the application fields of this application embodiment are not limited to these.

[0063] In some embodiments, a computer device can acquire an original image, invoke an object detection model to perform object detection on the original image, and obtain object detection results. The object detection model can be a deep learning model implemented based on an object detection algorithm or an instance segmentation algorithm.

[0064] In some embodiments, an image classification method is performed on a computer device. The computer device may also obtain an original image and object detection results for that original image from another computer device. After performing object detection on the original image and obtaining object detection results, the other computer device sends the object detection results to the computer device performing the image classification.

[0065] Step 204: Based on the object detection results, obtain the mask map corresponding to each object category.

[0066] In this context, a mask, also known as a code mask, is used to specify a region in an image. In this application, a mask image is used to specify a region of the original image so that specific operations can be performed on this selected region. The mask image can be a binary image or a Boolean image of the same size as the original image. Optionally, the pixel values ​​of pixels in the selected region can be labeled with specified pixel values, while the pixel values ​​of pixels in the remaining regions can be labeled with background pixel values, such as 0 or False.

[0067] Specifically, based on the positional information and corresponding object categories of each object in the original image indicated by the object detection results, the computer device can process the original image to obtain a mask map corresponding to each object category. The mask map is used to indicate the position of each category of object in the original image. In the embodiments of this application, the mask map can represent the number of object categories in the original image, and can also represent information such as the position and area of ​​objects of different object categories.

[0068] In some embodiments, obtaining a mask image corresponding to each object category based on the object detection results includes: determining the total number of object categories based on the corresponding object categories of each object; setting multiple different pixel values ​​based on the total number of object categories; for a target object category, determining object pixels belonging to the target object category and other pixels not belonging to the target object category in the original image based on location information, setting the pixel value of the object pixels belonging to the target object category to one of multiple different pixel values, and setting the pixel values ​​of other pixels to background pixel values, thereby obtaining a mask image corresponding to the target object category, wherein the target object category is any object category in the object detection results, and the pixel values ​​of object pixels belonging to different object categories are different.

[0069] For example, based on the object detection results of the original image, there are three object categories A, B, and C in the original image. Three different pixel values, pixel1, pixel2, and pixel3, can be set, all of which are not equal to 0. With the location information and corresponding object category of each object, a mask map for each of the three object categories can be obtained. If, based on the object detection results, the first object in the original image belongs to object category A, the second and third objects belong to object category B, and the fourth object belongs to object category C, then the pixel values ​​of the object pixels in the region where the first object is located can be set as pixel1, and the pixel values ​​of all other pixels in the original image except for the object pixels in the region where the first object is located can be set as background pixel values. Similarly, the pixel values ​​of the object pixels in the regions where the second and third objects are located can be set as pixel2, and the pixel values ​​of all other pixels in the original image except for the object pixels in the regions where the second and third objects are located can be set as background pixel values. The pixel values ​​of the object pixels in the region where the fourth object is located can be set as pixel3, and the pixel values ​​of all other pixels in the original image except for the object pixels in the region where the fourth object is located can be set as background pixel values. The background pixel values ​​can be, for example, 0 or 1, and they are different from the pixel values ​​of the object pixels.

[0070] like Figure 3 The image shown is a schematic diagram of the mask images corresponding to various object categories obtained in one embodiment. (Refer to...) Figure 3 The diagram shows three object categories and their corresponding mask images. The areas where objects are located in the three mask images are represented by different colors, indicating different object categories.

[0071] Step 206: The mask images corresponding to each object category are mixed to obtain a mixed mask image, which is used to represent the distribution information of objects in the original image.

[0072] Specifically, after obtaining the masks corresponding to each object category, the computer device performs blending on these masks to obtain a blending mask. This blending process generates a mask that integrates information from multiple dimensions, i.e., the blending mask. The blending mask represents the distribution information of objects in the original image, including additional information such as the area, quantity, location, category, and overlap of objects. Specifically, the blending mask implicitly includes the quantity of objects in the original image and explicitly includes the area, size, location, category, and overlap between objects of different categories. Therefore, subsequent image classification based on the blending mask can improve the accuracy of image classification. Figure 4 The image shown is a schematic diagram of a hybrid mask pattern in one embodiment.

[0073] In some embodiments, when a computer device performs blending processing on the mask images corresponding to various object categories, it can directly add the pixel values ​​of the corresponding pixels of the mask images corresponding to various object categories to obtain a blended mask image.

[0074] In some embodiments, when a computer device blends the mask images corresponding to different object categories, it can obtain the category weights of each object category and perform a weighted average of the mask images corresponding to each object category according to the category weights to obtain a blended mask image. Optionally, the category weights of each object category can be the same, and the sum of the category weights of each object category is 1. Optionally, the category weights of each object category can be different, and the sum of the category weights of each object category is 1.

[0075] In some embodiments, when a computer device blends the mask images corresponding to various object categories, it may also use other blending methods that can preserve the object category information features and represent the overlapping relationships when there are overlapping parts between object categories.

[0076] Step 208: Obtain the foreground and background segmentation map corresponding to the original image based on the object detection results.

[0077] Image segmentation is the process of dividing different parts of an image into pixels, while foreground / background segmentation is the process of separating the foreground and background portions of the image. In the original image, the object is considered the foreground, and everything else is considered the background. Based on the object detection results of the original image, the location information of each object can be obtained. This location information allows us to determine the regions where the objects and background are located in the original image, thus obtaining the corresponding foreground / background segmentation map.

[0078] In some embodiments, obtaining a foreground / background segmentation map corresponding to the original image based on object detection results includes: determining the region where the object is located in the original image based on the location information of each object; setting the pixel values ​​of the pixels in the region where the object is located in the original image as foreground pixel values; setting the pixel values ​​of the pixels in the original image other than the region where the object is located as background pixel values, thereby obtaining a foreground / background segmentation map corresponding to the original image.

[0079] Specifically, the computer device can determine the region where each object is located in the original image based on the location information of each object. By taking the union of the regions where each object is located, the region where all objects are located in the original image can be obtained. The pixel values ​​of the pixels in the region where the objects are located in the original image are set as foreground pixel values, such as 0, and the pixel values ​​of the pixels in the original image other than the region where the objects are located are set as background pixel values, such as 1, so as to obtain the foreground and background segmentation map corresponding to the original image.

[0080] like Figure 5 The image shown is a schematic diagram of the foreground and background segmentation map corresponding to the original image in one embodiment.

[0081] Step 210: Obtain the enhanced feature map of the original image based on the original image, the embedding representation of the hybrid mask map, and the foreground / background segmentation map. The enhanced feature map is used to classify the original image to obtain the image category of the original image.

[0082] Feature extraction is a prerequisite for image classification. In this embodiment, during feature extraction, in addition to combining the original image itself, the embedding representation of a hybrid mask and a foreground / background segmentation map are also incorporated. The hybrid mask provides information such as the category, location, quantity, area, and overlap of each object in the original image. The foreground / background segmentation map increases the weight of the region where the object is located in the original image, allowing for greater focus on feature extraction of the region containing the object during the feature extraction process. Thus, by obtaining an enhanced feature map of the original image based on the original image, the embedding representation of the hybrid mask, and the foreground / background segmentation map, information such as the category, location, quantity, area, and overlap of each object in the original image can be encoded into the features. Therefore, based on this enhanced feature map, the original image can be classified to obtain its image category. Compared to image classification based solely on the image features of the original image itself, this method can improve the accuracy of image classification.

[0083] In some embodiments, a computer device can extract an embedded representation of a hybrid mask image using an embedding network. The embedding network converts the image pixels of the hybrid mask image into vector representations in a high-dimensional vector space, which capture the semantic and visual features of the hybrid mask image. For example... Figure 6 The diagram shown is a schematic representation of the embedded representation of a hybrid mask image extracted using an embedded network in one embodiment. (Refer to...) Figure 6Each image pixel in the blending mask is converted into a high-dimensional vector. A matrix can be obtained based on the high-dimensional vector corresponding to each image pixel, which serves as the embedding representation of the blending mask.

[0084] In some embodiments, the computer device can classify the original image based on the enhanced feature map of the original image to obtain the image category of the original image, where the image category is one of a plurality of preset image classification categories.

[0085] In the image classification method described above, since the object detection results provide the location information of various objects in the original image, a hybrid mask can be obtained based on the location information of various objects. The hybrid mask provides additional information about the area, quantity, location, category, and overlap of objects in the original image. The foreground and background segmentation map provides the location information of objects in the original image. By combining the embedding representation of the original image, the hybrid mask, and the foreground and background segmentation map, the original image and the above-mentioned additional information can be encoded into the enhancement feature map. The enhancement feature map is obtained by combining the original image and the above-mentioned additional information, so that the image category obtained by classifying the original image based on the enhancement feature map is more accurate.

[0086] In one exemplary embodiment, such as Figure 7 As shown, the enhanced feature map of the original image is obtained based on the original image, the embedding representation of the hybrid mask map, and the foreground / background segmentation map, including:

[0087] Step 702: Extract features from the original image to obtain the feature map of the original image.

[0088] In some embodiments, a computer device can extract features from the original image using a convolutional neural network (CNN) to obtain a feature map of the original image.

[0089] In some embodiments, a computer device may extract features from the original image using a recurrent neural network (RNN) or an autoencoder to obtain a feature map of the original image.

[0090] In some embodiments, the computer device can extract features from the original image through other deep learning-based network architectures. This application embodiment does not limit this. Optionally, feature extraction from the original image can be based on the image's texture, color, and shape.

[0091] Step 704: Embedding processing is performed on the hybrid mask to obtain the embedded representation of the hybrid mask. The hybrid mask is used to represent the distribution information of objects in the original image.

[0092] In some embodiments, a computer device can extract an embedded representation of a hybrid mask image using an embedded network. Distribution information includes additional information such as the area, number, location, category, and overlap of objects.

[0093] Step 706: The feature map and the embedded representation of the hybrid mask map are concatenated to obtain the concatenated features. The foreground and background segmentation maps are fused with the concatenated features to obtain the enhanced feature map of the original image.

[0094] In this embodiment, the feature map of the original image represents the image texture, color, shape, and other information of the original image itself. The embedding representation of the hybrid mask represents the distribution information of each object in the original image, including additional information such as the area, quantity, position, category, and overlap of the objects. The foreground and background segmentation map represents the location information of the objects in the original image. By concatenating the feature map with the embedding representation of the hybrid mask, a concatenated feature is obtained. By fusing the foreground and background segmentation map with the concatenated feature, an enhanced feature map of the original image is obtained. In this way, during the feature extraction process, the image information represented by the original image, the embedding representation of the hybrid mask, and the foreground and background segmentation map can be encoded into the image features to obtain the enhanced feature map. This makes the classification of the original image based on the enhanced feature map more accurate.

[0095] In some embodiments, the feature map and the embedded representation of the hybrid mask map are concatenated to obtain concatenated features. The foreground / background segmentation map and the concatenated features are then fused to obtain an enhanced feature map of the original image. This includes: dividing the feature map, the foreground / background segmentation map, and the embedded representation of the hybrid mask map into sub-blocks to obtain feature sub-blocks, foreground / background sub-blocks, and embedded representation sub-blocks corresponding to each local region in the original image. The embedded representation sub-blocks are used to indicate the distribution information of objects in the corresponding local regions. The feature sub-blocks and embedded representation sub-blocks corresponding to the same local region are concatenated to obtain concatenated sub-blocks. The object coverage of each local region is calculated based on the foreground / background sub-blocks corresponding to each local region. Based on a self-attention mechanism, the enhanced feature map of the original image is obtained based on the object coverage and the concatenated sub-blocks.

[0096] In this embodiment, by segmenting the feature map, foreground / background segmentation map, and hybrid mask map into sub-blocks, feature sub-blocks, foreground / background sub-blocks, and embedding representation sub-blocks corresponding to each local region in the original image are obtained. This allows attention to each local region during feature extraction, facilitating the extraction of features from each local region in the original image. The feature sub-blocks corresponding to each local region represent information such as image texture, color, and shape in each local region of the original image. The embedding representation sub-blocks represent the distribution information of objects in each local region of the original image, and the foreground / background sub-blocks represent the location information of objects in each local region of the original image. The sub-blocks have a consistent size, for example, H×W. The computer device can concatenate the feature sub-blocks and embedding representation sub-blocks corresponding to the same local region to obtain concatenated sub-blocks. Based on the foreground / background sub-blocks corresponding to each local region, the object coverage of each local region is calculated. Object coverage is the proportion of the area where the object is located in each local region, i.e., the foreground region, to the entire local region. Object coverage can be represented by the ratio of the number of object pixels in the local region to the total number of pixels in that local region. Computer devices can obtain enhanced feature maps of the original image based on object coverage and stitched sub-blocks using a self-attention mechanism. In some embodiments, the computer device may also use different variants of the self-attention mechanism, such as relative position encoding self-attention or graph convolutional networks, etc.

[0097] In some embodiments, based on a self-attention mechanism, an enhanced feature map of the original image is obtained according to object coverage and stitched sub-blocks, including: for each stitched sub-block, converting the stitched sub-block into a query vector, a key vector, and a value vector respectively based on three different linear transformation matrices in the self-attention network; determining the mask weight of each local region according to the object coverage of each local region; obtaining the attention score of each local region according to the query vector, key vector, and mask weight of each local region; and weighting and summing the value vectors of each local region according to the attention scores to obtain the enhanced feature map of the original image.

[0098] In this self-attention network, three different linear transformation matrices can be used: the first weight matrix Wq, the second weight matrix Wk, and the third weight matrix Wv. For the concatenated sub-block E(i,j) located in the i-th row and j-th column after segmentation and concatenation, (i,j) represents the sub-block index. Based on these three different linear transformation matrices, the concatenated sub-block E(i,j) can be converted into the query vector q(i,j), the key vector k(i,j), and the value vector v(i,j), respectively, as expressed by the following formula:

[0099] q(i,j)= Wq* E (i,j); k(i,j)= Wk* E (i,j); v(i,j)= Wv* E (i,j).

[0100] In some embodiments, the mask weight of each local region is determined based on the object coverage of each local region. This can be achieved by: when the object coverage of a local region is greater than or equal to a set threshold τ, determining the mask weight w of that local region to be a set value β; and when the object coverage of a local region is less than the set threshold τ, determining the mask weight w of that local region to be 1. The set value β is greater than 1. That is, the larger the object coverage of a local region, the larger its mask weight; and the smaller the object coverage of a local region, the smaller its mask weight. Thus, the attention score of each local region is obtained based on its query vector, key vector, and mask weight. The attention score reflects the influence (or contribution) of other local regions on the current local region. The value vectors of each local region are weighted and summed according to their attention scores to obtain the enhanced features of the current local region. Based on the enhanced features of each local region, the enhanced feature map of the original image can be obtained. This process of extracting the enhanced feature map pays more attention to the image information of the area where the object is located, ensuring that the extracted enhanced feature map accurately reflects the distribution information of the object in the original image.

[0101] In some embodiments, an attention score for each local region is obtained based on the query vector, key vector, and mask weights of each local region. The value vectors of each local region are then weighted and summed according to the attention scores to obtain the enhanced feature map, which can be represented by the following formula:

[0102] ;

[0103] ;

[0104] Where w represents the mask weights of the local region, q is the query vector, k is the key vector, v is the value vector, and attention represents the attention score of the local region. An enhanced feature map representing a local region.

[0105] like Figure 8 The diagram shown illustrates how, in a specific embodiment, an enhanced feature map of the original image is obtained based on the original image, the embedded representation of the hybrid mask map, and the foreground / background segmentation map. (Refer to...) Figure 8 The original image is I, the blending mask is B, and the foreground / background segmentation image is M. The feature extraction process includes the following steps:

[0106] 1. Receive the original image I, the mixed mask image B, and the foreground / background segmentation image M.

[0107] 2. Extract the feature map F of the original image I using an image feature extraction network (e.g., CNN).

[0108] 3. Extract the embedding representation EmbB of the hybrid mask image B through an embedding network.

[0109] 4. Divide the feature map F and the embedding representation EmbB into the same number and size of sub-blocks, namely the feature sub-block Fij and the embedding representation sub-block EmbBij.

[0110] 5. Divide the foreground and background image into M and also divide it into the same number and size of foreground and background sub-blocks Mij.

[0111] 6. Concatenate the feature sub-block Fij corresponding to the same local region with the embedded representation sub-block EmbBij to obtain the concatenated sub-block Eij.

[0112] After obtaining the spliced ​​sub-block Eij, it can be transformed by another CNN to obtain the transformed spliced ​​sub-block Eij.

[0113] 7. Based on the self-attention mechanism, the spliced ​​sub-block Eij is projected to obtain three projection vectors, namely the query vector q(i,j), the key vector k(i,j), and the value vector v(i,j).

[0114] 8. Calculate the object coverage avr_Mij of the local region based on the foreground and background sub-blocks Mij.

[0115] 9. Determine the mask weight Wij for each local region based on the object coverage of each local region, apr_Mij.

[0116] 10. Based on the self-attention mechanism, the attention score is calculated according to the query vector q(i,j), key vector k(i,j), and value vector v(i,j) of each local region, as well as the mask weights Wij.

[0117] 11. Based on the self-attention mechanism, the enhanced feature map of the original image is obtained by weighted summing of the value vectors v(i,j) of each local region according to the attention score vector v(i,j). .

[0118] In some embodiments, obtaining an enhanced feature map of the original image based on the original image, the embedding representation of the hybrid mask, and the foreground / background segmentation map includes: inputting the original image, the embedding representation of the hybrid mask, and the foreground / background segmentation map into a trained image classification model; obtaining an enhanced feature map of the original image based on the original image, the embedding representation of the hybrid mask, and the foreground / background segmentation map using the trained image classification model; and obtaining an image classification result of the original image based on the enhanced feature map of the original image.

[0119] In this embodiment, the above image classification steps can be implemented based on an image classification model. (Refer to...) Figure 9 , Figure 9This diagram illustrates the input and output of an image classification model in one embodiment. The input to the image classification model consists of the original image, the embedding representation of the hybrid mask, and the foreground / background segmentation map; the output is the image category. The computer device invokes the trained image classification model to perform image classification based on the input data. Specifically, the image classification model obtains an enhanced feature map of the original image based on the original image, the embedding representation of the hybrid mask, and the foreground / background segmentation map. The image classification result of the original image is then obtained based on this enhanced feature map.

[0120] In some embodiments, the computer device extracts features from the original image using a feature extraction network of an image classification model to obtain a feature map of the original image; it then performs embedding processing on the hybrid mask to obtain an embedded representation of the hybrid mask; the feature map, foreground / background segmentation map, and embedded representation of the hybrid mask are segmented into sub-blocks to obtain feature sub-blocks, foreground / background sub-blocks, and embedded representation sub-blocks corresponding to each local region in the original image; the feature sub-blocks and embedded representation sub-blocks corresponding to the same local region are concatenated to obtain concatenated sub-blocks; and the object coverage of each local region is calculated based on the foreground / background sub-blocks corresponding to each local region. For each stitched sub-block, based on three different linear transformation matrices in the self-attention network, the stitched sub-block is converted into a query vector, a key vector, and a value vector, respectively. The mask weight of each local region is determined according to the object coverage of each local region. The attention score of each local region is obtained according to the query vector, key vector, and mask weight of each local region. The value vectors of each local region are weighted and summed according to the attention scores to obtain the enhanced feature map of the original image. The original image is classified according to the enhanced feature map of the original image to obtain the image category of the original image. The image category is one of several preset image classification categories.

[0121] In some embodiments, the image classification model is trained based on sample images.

[0122] In some embodiments, the method further includes: acquiring a sample image, object detection results of the sample image, and labeled image categories; acquiring a blending mask and a foreground / background segmentation map of the sample image based on the object detection results; acquiring an initial image classification model; inputting the sample image, the embedding representation of the blending mask of the sample image, and the foreground / background segmentation map into the initial image classification model; outputting the image category prediction probability of the sample image through the initial image classification model; adjusting the model parameters of the initial image classification model based on the image category prediction probability of the sample image and the labeled image categories, until a trained image classification model is obtained when the training stopping condition is met.

[0123] Specifically, computer equipment can build an initial image classification model based on deep learning. The model parameters of the initial image classification model are initialized. Then, the initial image classification model is trained using sample images. The sample images are used to train the image classification model. The object detection results of the sample images include the location information of each object in the sample image and the corresponding object category of each object. Based on the object detection results, a blending mask map and a foreground / background segmentation map of the sample images can be generated. Then, the sample images, the embedding representation of the blending mask map of the sample images, and the foreground / background segmentation map are input into the initial image classification model. Based on the image category prediction probability and labeled image category of the sample images output by the model, a loss function can be constructed. The model parameters of the initial image classification model are adjusted with the goal of minimizing the loss function value. This process is iterated continuously. After training with a large number of samples, a well-trained image classification model is obtained when the training stopping condition is met.

[0124] In one specific embodiment, an image classification method is provided, which can be executed by a computer device, and includes the following steps:

[0125] 1. Obtain the original image and input it into the object detection model. The object detection model performs object detection on the original image to obtain the object detection results. The object detection results include the location information of each object in the original image and the corresponding object category of each object.

[0126] 2. Determine the total number of object categories based on the corresponding object categories of each object;

[0127] 3. Set multiple different pixel values ​​based on the total number of object categories;

[0128] 4. For the target object category, determine the object pixels belonging to the target object category and other pixels not belonging to the target object category in the original image based on the location information. Set the pixel value of the object pixels belonging to the target object category to one of a number of different pixel values, and set the pixel values ​​of other pixels to background pixel values ​​to obtain the mask map corresponding to the target object category. Here, the target object category is any object category in the object detection result, and the pixel values ​​of object pixels belonging to different object categories are different.

[0129] 5. Obtain the category weights of each object category, and perform a weighted average of the mask images corresponding to each object category according to the category weights to obtain the blended mask image;

[0130] 6. Based on the location information of each object, determine the region where the object is located in the original image, set the pixel value of the pixel point in the region where the object is located in the original image as the foreground pixel value, and set the pixel value of the pixel point in the original image other than the region where the object is located as the background pixel value, to obtain the foreground and background segmentation map corresponding to the original image.

[0131] 7. Extract features from the original image to obtain the feature map of the original image;

[0132] 8. Embedding processing is performed on the hybrid mask to obtain the embedded representation of the hybrid mask. The hybrid mask is used to represent the distribution information of objects in the original image.

[0133] 9. Divide the embedding representations of the feature map, foreground / background segmentation map, and hybrid mask map into sub-blocks to obtain the feature sub-blocks, foreground / background sub-blocks, and embedding representation sub-blocks corresponding to each local region in the original image. The embedding representation sub-blocks are used to indicate the distribution information of objects in the corresponding local regions.

[0134] 10. Concatenate the feature sub-blocks corresponding to the same local region with the embedded representation sub-blocks to obtain concatenated sub-blocks;

[0135] 11. Calculate the object coverage of each local region based on the foreground and background sub-blocks corresponding to each local region;

[0136] 12. For each splicing sub-block, based on three different linear transformation matrices in the self-attention network, the splicing sub-block is converted into a query vector, a key vector, and a value vector, respectively.

[0137] 13. Determine the mask weight for each local region based on the object coverage of each local region;

[0138] 14. Obtain the attention score for each local region based on the query vector, key vector, and mask weights. Then, sum the value vectors of each local region according to the attention scores to obtain the enhanced feature map of the original image.

[0139] 15. Classify the original image based on the enhanced feature map of the original image to obtain the image category of the original image. The image category is one of several preset image classification categories.

[0140] In one exemplary embodiment, such as Figure 10 As shown, an image classification method is provided, which can be applied to... Figure 1 The following steps are used as an example of computer equipment (server or terminal), including steps 1002 to 1010. Wherein:

[0141] Step 1002: Obtain the defect detection results of the part image. The defect detection results include the location information of each defect in the part image and the corresponding defect category of each defect.

[0142] Step 1004: Based on the defect detection results, obtain the mask image corresponding to each defect category;

[0143] Step 1006: The mask images corresponding to each defect category are mixed to obtain a mixed mask image, which is used to characterize the distribution information of defects in the part image;

[0144] Step 1008: Obtain the foreground and background segmentation map corresponding to the part image based on the defect detection results;

[0145] Step 1010: Obtain an enhanced feature map of the part image based on the embedded representation of the part image, the hybrid mask image, and the foreground / background segmentation map. The enhanced feature map is used to classify the part image to obtain the image category of the part image. The image category is one of scrapped part image, part image to be repaired, and part image not to be repaired.

[0146] Since the defect detection results provide location information of various defects in the part image, a hybrid mask can be obtained based on the location information of various defects. The hybrid mask provides additional information such as the area, number, location, category, and overlap of defects in the part image. The foreground and background segmentation map provides the location information of defects in the part image. By combining the embedding representation of the part image, the hybrid mask, and the foreground and background segmentation map, the part image and the above-mentioned additional information can be encoded into the enhanced feature map. The enhanced feature map is obtained by combining the part image and the above-mentioned additional information, so that the image category obtained by classifying the part image based on the enhanced feature map is more accurate.

[0147] For details on the specific implementation methods of each step in this embodiment, please refer to the preceding text.

[0148] As shown in Table 1 below, the table contains test data from an industrial quality inspection project.

[0149] Table 1

[0150]

[0151] Here, RGB represents classification using only the image itself. RGB+blending mask refers to the image classification method provided in this application. As can be seen, with the addition of a blending mask, the inference speed remains unchanged, but the accuracy for each category is significantly improved, as shown in Table 1. The distribution of the data used in the test is shown in Table 2.

[0152] Table 2

[0153]

[0154] The image classification method provided in this embodiment can comprehensively utilize information other than the image itself to better classify images. This method combines image features with other information, such as the area and number of defects, and performs classification by integrating features from multiple dimensions. By integrating features from multiple dimensions, this method can improve the accuracy of image classification and play an important role in fields such as industrial quality inspection.

[0155] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0156] Based on the same inventive concept, this application also provides an image classification apparatus for implementing the image classification method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more image classification apparatus embodiments provided below can be found in the limitations of the image classification method described above, and will not be repeated here.

[0157] In one exemplary embodiment, such as Figure 11 As shown, an image classification device 1100 is provided, including: a detection result acquisition module 1102, a detection result encoding module 1104, and an image classification module 1110, wherein:

[0158] The detection result acquisition module 1102 is used to acquire the object detection results of the original image. The object detection results include the location information of each object in the original image and the corresponding object category of each object.

[0159] The detection result encoding module 1104 is used to obtain the mask image corresponding to each object category based on the object detection result;

[0160] The detection result encoding module 1104 is also used to perform a blending process on the mask images corresponding to each object category to obtain a blended mask image, which is used to represent the distribution information of objects in the original image.

[0161] The detection result encoding module 1104 is also used to obtain the foreground and background segmentation map corresponding to the original image based on the object detection result;

[0162] The image classification module 1106 is used to obtain the enhanced feature map of the original image based on the original image, the embedding representation of the hybrid mask map and the foreground and background segmentation map. The enhanced feature map is used to classify the original image to obtain the image category of the original image.

[0163] In some embodiments, the detection result encoding module 1104 is further configured to determine the total number of object categories based on the corresponding object categories of each object; set multiple different pixel values ​​based on the total number of object categories; for a target object category, determine the object pixels belonging to the target object category and other pixels not belonging to the target object category in the original image based on the location information, set the pixel value of the object pixels belonging to the target object category to one of multiple different pixel values, and set the pixel values ​​of other pixels to background pixel values ​​to obtain a mask map corresponding to the target object category, wherein the target object category is any object category in the object detection result, and the pixel values ​​of object pixels belonging to different object categories are different.

[0164] In some embodiments, the detection result encoding module 1104 is further configured to obtain the category weights of each object category; and to perform a weighted average of the mask images corresponding to each object category according to the category weights to obtain a hybrid mask image.

[0165] In some embodiments, the detection result encoding module 1104 is further configured to determine the region where the object is located in the original image based on the location information of each object; set the pixel value of the pixel point in the region where the object is located in the original image as the foreground pixel value; set the pixel value of the pixel point in the original image other than the region where the object is located as the background pixel value, and obtain the foreground and background segmentation map corresponding to the original image.

[0166] In some embodiments, the image classification module 1106 is further configured to extract features from the original image to obtain a feature map of the original image; perform embedding processing on the hybrid mask to obtain an embedded representation of the hybrid mask, wherein the hybrid mask is used to represent the distribution information of objects in the original image; concatenate the feature map with the embedded representation of the hybrid mask to obtain a concatenated feature; and fuse the foreground / background segmentation map with the concatenated feature to obtain an enhanced feature map of the original image.

[0167] In some embodiments, the image classification module 1106 is further configured to segment the embedding representations of the feature map, the foreground / background segmentation map, and the hybrid mask map into sub-blocks respectively, to obtain feature sub-blocks, foreground / background sub-blocks, and embedding representation sub-blocks corresponding to each local region in the original image. The embedding representation sub-blocks are used to indicate the distribution information of objects in the corresponding local region. The feature sub-blocks and embedding representation sub-blocks corresponding to the same local region are concatenated to obtain concatenated sub-blocks. The object coverage of each local region is calculated based on the foreground / background sub-blocks corresponding to each local region. Based on a self-attention mechanism, the enhanced feature map of the original image is obtained based on the object coverage and the concatenated sub-blocks.

[0168] In some embodiments, the image classification module 1106 is further configured to, for each stitched sub-block, convert the stitched sub-block into a query vector, a key vector, and a value vector respectively based on three different linear transformation matrices in the self-attention network; determine the mask weight of each local region according to the object coverage of each local region; obtain the attention score of each local region according to the query vector, key vector, and mask weight of each local region; and weight and sum the value vectors of each local region according to the attention scores to obtain the enhanced feature map of the original image.

[0169] In some embodiments, the detection result acquisition module 1102 is further configured to acquire the original image, input the original image into the object detection model, and perform object detection on the original image through the object detection model to obtain the object detection result.

[0170] In some embodiments, the image classification module 1106 is further configured to input the original image, the embedding representation of the hybrid mask, and the foreground / background segmentation map into the trained image classification model; obtain the enhanced feature map of the original image based on the original image, the embedding representation of the hybrid mask, and the foreground / background segmentation map through the trained image classification model; and obtain the image classification result of the original image based on the enhanced feature map of the original image.

[0171] In some embodiments, the image classification device 1100 further includes:

[0172] The training module is used to acquire sample images, object detection results of sample images, and labeled image categories; based on the object detection results, it obtains the blending mask and foreground / background segmentation map of the sample images; it acquires an initial image classification model; it inputs the sample images, the embedding representation of the blending mask, and the foreground / background segmentation map into the initial image classification model, outputs the image category prediction probability of the sample images through the initial image classification model, and adjusts the model parameters of the initial image classification model based on the image category prediction probability and labeled image categories, until the training stopping condition is met to obtain a trained image classification model.

[0173] The image classification device 1100 described above acquires object detection results of the original image, including the location information of each object in the original image and the corresponding object category of each object; obtains a mask image corresponding to each object category based on the object detection results; performs a mixing process on the mask images corresponding to each object category to obtain a mixed mask image; obtains a foreground and background segmentation image corresponding to the original image based on the object detection results; and obtains an enhanced feature map of the original image based on the embedding representation of the original image, the mixed mask image, and the foreground and background segmentation image. The enhanced feature map is used to classify the original image to obtain the image category of the original image. Since the object detection results provide the location information of various objects in the original image, a hybrid mask can be obtained based on the location information of various objects. The hybrid mask provides additional information about the area, quantity, location, category, and overlap of objects in the original image. The foreground and background segmentation map provides the location information of objects in the original image. By combining the embedding representation of the original image, the hybrid mask, and the foreground and background segmentation map, the original image and the above-mentioned additional information can be encoded into the enhancement feature map. The enhancement feature map is obtained by combining the original image and the above-mentioned additional information, so that the image category obtained by classifying the original image based on the enhancement feature map is more accurate.

[0174] In one exemplary embodiment, an image classification apparatus is provided, comprising: a detection result acquisition module, a detection result encoding module, and an image classification module, wherein:

[0175] The detection result acquisition module is used to acquire the defect detection results of the part image. The defect detection results include the location information of each defect in the part image and the corresponding defect category of each defect.

[0176] The detection result encoding module is used to obtain the mask image corresponding to each defect category based on the defect detection results;

[0177] The detection result encoding module is also used to mix the mask images corresponding to each defect category to obtain a mixed mask image, which is used to characterize the distribution information of defects in the part image;

[0178] The detection result encoding module is also used to obtain the foreground and background segmentation map corresponding to the part image based on the defect detection results;

[0179] The image classification module is used to obtain the enhanced feature map of the part image based on the part image, the embedded representation of the hybrid mask image and the foreground and background segmentation map. The enhanced feature map is used to classify the part image to obtain the image category of the part image. The image category is one of scrapped part image, part image to be repaired, and part image not to be repaired.

[0180] For the embodiments of each module in this example, please refer to the corresponding descriptions above.

[0181] Each module in the aforementioned image classification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0182] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows: Figure 12 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an image classification method.

[0183] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0184] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the image classification method provided in the embodiments of this application.

[0185] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the image classification method provided in the embodiments of this application.

[0186] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the image classification method provided in the embodiments of this application.

[0187] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0188] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0189] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0190] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image classification method, characterized in that, The method includes: Obtain object detection results from the original image, the object detection results including the location information of each object in the original image and the corresponding object category of each object; Based on the object detection results, a mask image corresponding to each object category is obtained; The mask images corresponding to each object category are blended to obtain a blended mask image, which is used to represent the distribution information of objects in the original image. Based on the object detection results, a foreground / background segmentation map corresponding to the original image is obtained; An enhanced feature map of the original image is obtained based on the original image, the embedded representation of the hybrid mask map, and the foreground / background segmentation map. The enhanced feature map is used to classify the original image to obtain the image category of the original image.

2. The method according to claim 1, characterized in that, The step of obtaining the mask image corresponding to each object category based on the object detection results includes: Determine the total number of object categories based on the corresponding object categories of each object; Set multiple different pixel values ​​based on the total number of the object categories; For a target object category, based on location information, determine the object pixels belonging to the target object category and other pixels not belonging to the target object category in the original image. Set the pixel value of the object pixels belonging to the target object category to one of the multiple different pixel values, and set the pixel values ​​of the other pixels to background pixel values ​​to obtain a mask image corresponding to the target object category. The target object category is any object category in the object detection result, and the pixel values ​​of object pixels belonging to different object categories are different.

3. The method according to claim 1, characterized in that, The process of blending the mask images corresponding to each object category to obtain a blended mask image includes: Obtain the category weights of each object category; The mask images corresponding to each object category are weighted and averaged according to their category weights to obtain a hybrid mask image.

4. The method according to claim 1, characterized in that, The step of obtaining the foreground and background segmentation map corresponding to the original image based on the object detection result includes: Based on the location information of each object, the region where the object is located in the original image is determined; Set the pixel values ​​of the pixels in the region where the object is located in the original image to the foreground pixel values; The pixel values ​​of pixels in the original image other than the region where the object is located are set to background pixel values ​​to obtain the foreground and background segmentation map corresponding to the original image.

5. The method according to claim 1, characterized in that, The step of obtaining the enhanced feature map of the original image based on the original image, the embedding representation of the hybrid mask map, and the foreground / background segmentation map includes: Feature extraction is performed on the original image to obtain the feature map of the original image; The hybrid mask is embedded to obtain an embedded representation of the hybrid mask, which is used to represent the distribution information of objects in the original image; The feature map is concatenated with the embedded representation of the hybrid mask map to obtain the concatenated feature map. The foreground and background segmentation map is then fused with the concatenated feature map to obtain the enhanced feature map of the original image.

6. The method according to claim 5, characterized in that, The step of concatenating the feature map with the embedded representation of the hybrid mask map to obtain the concatenated feature map, and fusing the foreground / background segmentation map with the concatenated feature map to obtain the enhanced feature map of the original image, includes: The feature map, the foreground / background segmentation map, and the embedding representation of the hybrid mask map are respectively divided into sub-blocks to obtain feature sub-blocks, foreground / background sub-blocks, and embedding representation sub-blocks corresponding to each local region in the original image. The embedding representation sub-blocks are used to indicate the distribution information of objects in the corresponding local regions. The feature sub-blocks corresponding to the same local region are concatenated with the embedded representation sub-blocks to obtain the concatenated sub-blocks; Calculate the object coverage of each local region based on the foreground and background sub-blocks corresponding to each local region; Based on the self-attention mechanism, an enhanced feature map of the original image is obtained according to the object coverage and the stitched sub-blocks.

7. The method according to claim 6, characterized in that, The method of obtaining the enhanced feature map of the original image based on the self-attention mechanism, according to the object coverage and the stitched sub-blocks, includes: For each concatenated sub-block, based on three different linear transformation matrices in the self-attention network, the concatenated sub-block is converted into a query vector, a key vector, and a value vector, respectively; The mask weight of each local region is determined based on the object coverage of each local region. The attention score for each local region is obtained based on the query vector, key vector, and mask weights of each local region. The value vectors of each local region are then weighted and summed according to the attention scores to obtain the enhanced feature map of the original image.

8. The method according to claim 1, characterized in that, The object detection results obtained from the original image include: Acquire the original image and input the original image into the object detection model; The object detection model is used to perform object detection on the original image to obtain the object detection result.

9. The method according to claim 1, characterized in that, The method further includes: The original image is classified based on the enhanced feature map of the original image to obtain the image category of the original image, which is one of a plurality of preset image classification categories.

10. The method according to claim 1, characterized in that, The step of obtaining the enhanced feature map of the original image based on the original image, the embedding representation of the hybrid mask map, and the foreground / background segmentation map includes: The original image, the embedded representation of the hybrid mask image, and the foreground / background segmentation image are input into the trained image classification model; The trained image classification model obtains an enhanced feature map of the original image based on the original image, the embedding representation of the hybrid mask map, and the foreground / background segmentation map, and obtains the image classification result of the original image based on the enhanced feature map of the original image.

11. The method according to claim 10, characterized in that, The method further includes: Acquire sample images, object detection results of the sample images, and labeled image categories; Based on the object detection results, a blending mask and a foreground / background segmentation map of the sample image are obtained; Obtain the initial image classification model; The sample image, the embedded representation of the mixed mask image of the sample image, and the foreground / background segmentation image are input into the initial image classification model. The initial image classification model outputs the image category prediction probability of the sample image. The model parameters of the initial image classification model are adjusted according to the image category prediction probability of the sample image and the labeled image category until the training stopping condition is met to obtain the trained image classification model.

12. An image classification method, characterized in that, The method includes: Obtain the defect detection results of the part image, the defect detection results including the location information of each defect in the part image and the corresponding defect category of each defect; Based on the defect detection results, mask images corresponding to each defect category are obtained; The mask images corresponding to each defect category are mixed to obtain a mixed mask image, which is used to characterize the distribution information of defects in the part image; Based on the defect detection results, a foreground and background segmentation map corresponding to the part image is obtained; An enhanced feature map of the part image is obtained based on the part image, the embedded representation of the hybrid mask image, and the foreground / background segmentation map. The enhanced feature map is used to classify the part image to obtain an image category of the part image, which is one of scrapped part image, part image to be repaired, and part image not to be repaired.

13. An image classification device, characterized in that, The device includes: The detection result acquisition module is used to acquire the object detection results of the original image. The object detection results include the location information of each object in the original image and the corresponding object category of each object. The detection result encoding module is used to obtain the mask image corresponding to each object category based on the object detection result; The detection result encoding module is also used to perform a mixing process on the mask images corresponding to each object category to obtain a mixed mask image, which is used to characterize the distribution information of objects in the original image; The detection result encoding module is also used to obtain the foreground and background segmentation map corresponding to the original image based on the object detection result; An image classification module is used to obtain an enhanced feature map of the original image based on the original image, the embedding representation of the hybrid mask map, and the foreground / background segmentation map. The enhanced feature map is used to classify the original image to obtain the image category of the original image.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

Images