Image processing method and device, intelligent device, storage medium and product

Abstract

Embodiments of the present application provide an image processing method and device, intelligent equipment, storage medium and product, which can be applied to various fields such as cloud technology, block chain, Internet of Vehicles, intelligent transportation, smart home and the like. The method comprises: obtaining a global image feature of a to-be-processed image, and determining a first positioning area containing a target object in the to-be-processed image; extracting a regional image feature of the first positioning area from the global image feature of the to-be-processed image; the regional image feature of the first positioning area comprises pixel-level features of each pixel point in the first positioning area; performing classification processing on the pixel-level features of each pixel point to obtain a target classification result of each pixel point; performing mask processing on the to-be-processed image according to the target classification result of each pixel point to obtain a mask image of the target object; the mask image is used to segment an area occupied by the target object in the to-be-processed image. By using the present application, the accuracy of image segmentation can be improved.

Description

Technical Field

[0001] This application relates to the field of computer application technology, and in particular to an image processing method, apparatus, intelligent device, storage medium and product. Background Technology

[0002] In traditional industrial manufacturing, quality inspection is a crucial link in the production process, and the inspection of surface defects is of paramount importance. Producers need to inspect the surface condition of products to determine if defects exist and take appropriate action based on the results. For example, in the coating quality inspection of lithium battery production, it is necessary to check whether the slurry is evenly coated on the lithium battery electrodes. If the electrodes exhibit severe powder shedding, poor bending resistance, or white bubbles, they must be screened through inspection and stopped from proceeding to the next stage of production.

[0003] Currently, product images can be captured, and image segmentation techniques can be used to segment the area occupied by defects from the product image. Based on the size of this area, it can be determined whether the product meets the defect standard. However, current semantic segmentation-based image segmentation techniques, such as the Pyramid Scene Parseing Network (PSPNet), struggle to achieve accurate segmentation of defect regions due to multi-class aggregation problems. For example, when two defects of different categories appear close to each other, they are often grouped into the same category and segmented as a single defect. Summary of the Invention

[0004] This application provides an image processing method, apparatus, smart device, storage medium, and product that can improve the accuracy of image segmentation and achieve precise identification of defective regions.

[0005] On one hand, embodiments of this application provide an image processing method, the method comprising:

[0006] Obtain the global image features of the image to be processed, and determine the first localization region in the image to be processed that contains the target object;

[0007] Extract the regional image features of the first positioning region from the global image features of the image to be processed; the regional image features of the first positioning region include the pixel-level features of each pixel within the first positioning region;

[0008] The pixel-level features of each pixel are classified to obtain the target classification result of each pixel; the target classification result is used to indicate whether each pixel belongs to the target region occupied by the target object in the image to be processed.

[0009] Based on the target classification results of each pixel, the first localization region is masked to obtain a mask image of the target object; the mask image is used to segment the target region in the image to be processed.

[0010] On one hand, embodiments of this application provide an image processing apparatus, the apparatus comprising:

[0011] The acquisition unit is used to acquire the global image features of the image to be processed;

[0012] The processing unit is configured to determine a first location region containing a target object in the image to be processed, and extract the regional image features of the first location region from the global image features of the image to be processed; the regional image features of the first location region include the pixel-level features of each pixel point within the first location region.

[0013] The processing unit is further configured to classify the pixel-level features of each pixel to obtain the target classification result of each pixel; the target classification result is used to indicate whether each pixel belongs to the target area occupied by the target object in the image to be processed.

[0014] The processing unit is further configured to perform masking processing on the first positioning region based on the target classification results of each pixel to obtain a mask image of the target object; the mask image is used to segment the target region in the image to be processed.

[0015] On one hand, embodiments of this application provide an intelligent device, which includes a processor, a communication interface, and a memory. The processor, communication interface, and memory are interconnected. The memory stores a computer program, and the processor is used to call the computer program to execute the image processing method of any of the above possible implementations.

[0016] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image processing method of any possible implementation.

[0017] On the one hand, embodiments of this application also provide a computer program product, which includes a computer program or computer instructions, and the computer program or computer instructions are executed by a processor to implement the steps of the image processing method provided in embodiments of this application.

[0018] On the one hand, embodiments of this application also provide a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of a smart device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to implement the image processing method provided in embodiments of this application.

[0019] In this embodiment, starting from the first positioning region in the image to be processed, pixel-level features of each pixel in the first positioning region can be extracted to achieve accurate prediction of image segmentation at the pixel level. Then, a mask image of the target object can be generated based on the prediction results of each pixel. The mask image can indicate the position and shape of the target object in the image to be processed. Therefore, the area occupied by the target object in the image to be processed can be segmented using the mask image. By improving the accuracy of image segmentation, accurate identification of defective areas can be achieved. Attached Figure Description

[0020] To more clearly illustrate the technical methods of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the architecture of an image processing system provided in an embodiment of this application;

[0022] Figure 2 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0023] Figure 3 A flowchart illustrating another image processing method provided in an embodiment of this application;

[0024] Figure 4 A schematic diagram illustrating the extraction effect of a mask proxy feature provided in an embodiment of this application;

[0025] Figure 5 A flowchart illustrating another image processing method provided in an embodiment of this application;

[0026] Figure 6 A schematic flowchart illustrating a model training method provided in an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0028] Figure 8This is a schematic diagram of the structure of a smart device provided in an embodiment of this application. Detailed Implementation

[0029] The technical methods in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0030] This application can be applied to the detection of surface defects in products, such as scratches, spots, and dents. In the specific image processing method, the target object can refer to a defect. First, a smart device (which can be a terminal device or a server) can extract a first localization region containing the target object from the image to be processed. Then, starting from the first localization region, the next stage of segmentation is performed. In the segmentation stage, the regional image features of the first localization region can be extracted from the global image features of the image to be processed. Based on these regional image features, the pixel-level features of each pixel in the first localization region are determined. By classifying the pixel-level features of each pixel, it can be determined whether each pixel belongs to the target area occupied by the target object in the image to be processed (i.e., the target classification result), achieving accurate pixel-level prediction. Then, a mask image of the target object can be generated based on the target classification results of each pixel. This mask image indicates the position and shape of the target object in the image to be processed. Therefore, the area occupied by the target object in the image to be processed can be segmented using this mask image, achieving accurate segmentation of the target object.

[0031] The image processing methods mentioned in the embodiments of this application involve artificial intelligence (AI) technology. AI is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making functions. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0032] With the research and advancement of artificial intelligence (AI) technology, it is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, smart customer service, vehicle networking, and intelligent transportation. It is believed that with further technological development, AI will be applied in even more fields and play an increasingly important role. This application's embodiments mainly involve natural language processing and machine learning / deep learning technologies within AI software. For example, the global image features and the first localization region in this application can be obtained by detecting and processing the image to be processed using a neural network model.

[0033] The image processing methods described above can be implemented using a processing system. Figure 1 An architecture diagram of an image processing system is shown. The image processing system includes a database 101, a server 102, and a terminal device 103. Communication between the database 101, server 102, and terminal device 103 can be established via wired or wireless means. This application does not limit the number of servers 102 and terminal devices 103.

[0034] Database 101 can be a cloud database or a local database, or a private database (i.e., a database in a private environment) or a public database (i.e., a database in a public environment). Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, but is not limited to these. Terminal device 103 can be a smartphone, tablet, smart wearable device, smart voice interaction device, smart home appliance, personal computer, in-vehicle terminal, etc., without restriction.

[0035] Terminal device 103 may have a built-in camera or an external camera connected to it. Both can be used to acquire images to be processed. In one scenario, the terminal device locally stores the image to be processed and directly processes it according to the process described in the aforementioned image processing method, outputting a mask image of the target object. In another scenario, terminal device 103 does not perform local detection but uploads the acquired image to server 102. Server 102 processes the image according to the process described in the aforementioned image processing method, outputs a mask image of the target object, and can also send this mask image back to the terminal device for display.

[0036] Cloud databases can store large amounts of images to be processed, along with their corresponding mask images. Cloud databases utilize cloud technology to achieve storage and other functions. Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology applied in the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0037] Cloud databases involve cloud storage, a new concept that extends and develops from cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to aggregate a large number of various types of storage devices (also called storage nodes) across a network through application software or application interfaces to collaboratively provide data storage and business access functions. Currently, the storage method of a storage system is as follows: Logical volumes are created. When creating a logical volume, physical storage space is allocated to each logical volume. This physical storage space may consist of the disks of one or several storage devices. Clients store data on a logical volume, which means storing the data on the file system. The file system divides the data into many parts, each part being an object. Each object contains not only data but also additional information such as data identifiers. The file system writes each object to the physical storage space of the logical volume and records the storage location information of each object. Therefore, when a client requests data access, the file system can allow the client to access the data based on the storage location information of each object.

[0038] Based on the above description, the image processing method in this application will be described in detail below. Please refer to [link / reference]. Figure 2 , Figure 2 This is a flowchart illustrating an image processing method provided in an embodiment of this application. The method can be executed by a smart device, for example, through the methods described above. Figure 1 The method, executed by server 102, includes the following steps.

[0039] S201. Obtain the global image features of the image to be processed, and determine the first localization region in the image to be processed that contains the target object.

[0040] The image to be processed can be a product image of the product whose surface is to be inspected for defects. For example, it can be a coating image formed after a slurry is applied to a lithium battery electrode, an image of the weld seam of a metal plate, or a product image of the surface of other workpieces. The intelligent device can acquire the image to be processed through a connected image acquisition device, or the image to be processed can be input into the intelligent device by relevant technicians, or the image to be processed can be obtained from a database; no specific limitation is made here.

[0041] Next, global image features can be extracted from the image to be processed. Global image features refer to features that can represent the entire image, and can be one or more of the image's semantic features and visual features (such as color, texture, etc.). Specifically, deep learning algorithms (such as deep neural network algorithms) can be used to extract global image features. The detailed implementation process will not be described in detail in this embodiment, and the method of extracting global image features is not limited to deep neural network algorithms.

[0042] When this application is applied to the detection of appearance defects on the surface of a product, the target object refers to the defect. This application needs to determine a first localization region containing the target object from the image to be processed. This first localization region can be a two-dimensional bounding box, used to indicate the approximate location of the target object in the image to be processed. This first localization region can be obtained through an object detection model, such as Faster R-CNN (a fast region algorithm based on convolutional neural network features), R-FCN (a region algorithm based on convolutional neural network features), etc. Typically, some candidate regions are given first, then features are extracted from them, and then these regions are classified based on these features. Essentially, this transforms object detection into an image classification problem.

[0043] It should be noted that there may be one or more target objects in the image to be processed. For example, there may be multiple spots and multiple pits in the image to be processed. Each target object has its corresponding first positioning region.

[0044] S202. Extract the regional image features of the first positioning region from the global image features of the image to be processed; the regional image features of the first positioning region include the pixel-level features of each pixel within the first positioning region.

[0045] The first localization region is a part (or local) region in the image to be processed. The regional image features of the first localization region are relative to the global image features of the image to be processed, and can reflect the local characteristics of the first localization region in the image to be processed.

[0046] In one embodiment, the global image features of the image to be processed include a first feature map corresponding to one or more feature channels, and the global image features can be represented as follows: Where c represents the dimension (i.e., number, a positive integer) of the feature channels, H represents the height of the first feature map, and W represents the width of the first feature map. It is evident that the global image features include c first feature maps, and the feature size of each first feature map is H×W. This application uses the example where the image size of the image to be processed matches (or is the same as) the feature size of the first feature maps, i.e., the image size of the image to be processed is also H×W, to illustrate the subsequent steps. Understandably, if the feature sizes of the first feature maps included in the global image features are the same as the image size of the image to be processed, then the region layout data of the first positioning region in the image to be processed can be obtained first. This region layout data can be used to draw the first positioning region in the image to be processed; for example, the region layout data is the coordinates of the upper left corner of the bounding box, its length, and its width. Then, based on this region layout data, a second feature map with the same shape and position as the first positioning region is directly extracted from the first feature maps included in the global image features.

[0047] By concatenating the feature values ​​at the same position in the obtained second feature maps, for example, concatenating the feature values ​​at position (m,n) in the various second feature maps, multiple feature vectors corresponding to the positions can be obtained, where m and n are positive integers.

[0048] It should be noted that since the feature size of each extracted second feature map is the same as the image size of the first positioning region (for example, both the feature size of the second feature map and the image size of the first positioning region can be h×w), the position (m,n) in each second feature map can correspond to the position (m,n) in the first positioning region. The feature vector corresponding to this position (m,n) can be used as the pixel-level feature of the pixel corresponding to that position (m,n) in the first positioning region. Since the corresponding pixel can be determined within the first positioning region through each position in the second feature map, the feature vectors corresponding to multiple positions can be used as the pixel-level features of each pixel within the first positioning region. This pixel-level feature describes the characteristics of each pixel within the first positioning region and reflects the properties of each pixel within the first positioning region. Furthermore, since this pixel-level feature is obtained based on global image features at the entire image level, it also preserves spatial information in the image to a certain extent. This application uses this pixel-level feature to identify whether each pixel within the first positioning region belongs to the target region.

[0049] Finally, the regional image features of the first positioning region are determined based on the pixel-level features of each pixel in the first positioning region. The regional image features are feature maps with the same size as the second feature map, carrying the pixel-level features of the first positioning region.

[0050] In one specific embodiment, the regional image features of the first positioning region can be represented as follows: Where c represents the dimension of the feature channels (i.e., the number of second feature maps), h represents the height of the second feature map, and w represents the width of the second feature map. In other words, the regional image features of the first localization region can be constructed based on the obtained second feature maps, and then the feature values ​​at the same position in the various second feature maps included in the regional image features can be concatenated to obtain the pixel-level features of each pixel in the first localization region.

[0051] In feasible embodiments, the feature sizes of each first feature map in the global image features may not match the image size of the image to be processed. Therefore, the scaling ratio of the feature size of the first feature map relative to the image size of the image to be processed can be determined. After scaling the first positioning region proportionally, second feature maps are extracted from each first feature map according to the regional layout data of the scaled first positioning region. Understandably, the feature sizes of the extracted second feature maps also do not match the image size of the first positioning region. Next, each obtained second feature map can be mapped to the image size of the first positioning region to obtain mapped second feature maps. Then, feature values ​​at the same position in the mapped second feature maps are concatenated to obtain the pixel-level features of each pixel within the first positioning region.

[0052] S203. Classify the pixel-level features of each pixel to obtain the target classification result of each pixel; the target classification result is used to indicate whether each pixel belongs to the target area occupied by the target object in the image to be processed.

[0053] The target object occupies the target region in the image to be processed. This target region is usually contained in the first positioning region. Therefore, this application mainly performs pixel-level feature classification processing on each pixel in the first positioning region. This classification processing refers to the process of determining whether each pixel belongs to the target region. Finally, the classification processing can obtain the target classification result of each pixel. The target classification result is used to indicate whether each pixel belongs to the target region, thereby enabling accurate prediction at the pixel level.

[0054] S204. Based on the target classification results of each pixel, perform masking processing on the first localization region to obtain the mask image of the target object; the mask image is used to segment the target region in the image to be processed.

[0055] In one embodiment, one or more reference pixels indicating that the target classification result belongs to the target region can be determined from each pixel in the first positioning region. The pixel values ​​of each reference pixel in the first positioning region are set as a first mask value, and the pixel values ​​of all pixels in the first positioning region other than the reference pixels are set as a second mask value to obtain a mask image of the target object. The first mask value can be 1, and the second mask value can be 0. The region corresponding to the first mask value in the mask image of the target object represents the target region occupied by the target object in the image to be processed, and the region corresponding to the second mask value represents the background region outside the target region in the image to be processed.

[0056] Understandably, the mask image of the target object has the same shape as the first localization region in the image to be processed. Therefore, the target region in the image to be processed can be directly segmented using the region corresponding to the first mask value in the mask image of the target object. This segmentation may include: determining the target region from the image to be processed using the region corresponding to the first mask value in the mask image of the target object, and adding a color marker to the target region in the image to be processed, for example, marking the target region in the image to be processed as red. If the image to be processed includes multiple target objects, the target regions occupied by different target objects can be marked with different colors.

[0057] In feasible embodiments, the feature sizes of each first feature map in the global image features may not match the image size of the image to be processed. Therefore, the feature sizes of the extracted second feature maps may also not match the image size of the first localization region. A reference region with the same image size as the feature sizes of the second feature maps can be determined. The feature vectors corresponding to multiple positions obtained from each second feature map are then used as the pixel-level features of each pixel within the reference region. These pixel-level features are then classified (to determine whether each pixel belongs to the target region). Based on the classification results of each pixel, the reference region is masked to obtain a reference mask image. Similarly, the region corresponding to the first mask value in the reference mask image represents the target region occupied by the target object in the image to be processed, and the region corresponding to the second mask value represents the background region outside the target region in the image to be processed. The reference mask image can then be mapped onto the feature scale of the first localization region, effectively obtaining the mask image of the target object, thereby segmenting the target region in the image to be processed.

[0058] In this embodiment, starting from the first positioning region in the image to be processed, pixel-level features of each pixel in the first positioning region can be extracted to achieve accurate prediction of image segmentation at the pixel level. Then, a mask image of the target object can be generated based on the prediction results of each pixel. The mask image can indicate the position and shape of the target object in the image to be processed. Therefore, the area occupied by the target object in the image to be processed can be segmented using the mask image. By improving the accuracy of image segmentation, accurate identification of defective areas can be achieved.

[0059] Please see Figure 3 , Figure 3 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. This method can be executed by a smart device, for example, through the methods described above. Figure 1 The method, executed by server 102, includes the following steps.

[0060] S301. Obtain the global image features of the image to be processed, and determine the first location region in the image to be processed that contains the target object.

[0061] In one embodiment, the first localization region is obtained by calling a target detection model to detect and process the image to be processed. The target detection model includes a feature extraction module and a recognition processing module. The feature extraction module is used to extract global image features of the image to be processed, and the recognition processing module is used to recognize and process the global image features of the image to be processed to obtain the first localization region and the category to which the target object in the image to be processed belongs. The category can be set manually, such as including scratches, spots and pits.

[0062] In a specific implementation, the feature extraction module may include a backbone network f θ and neck network g θ Backbone network f θ Typically, a convolutional neural network (CNN) is used to extract image features. In practice, the backbone network f... θ Typically, these are CNNs trained on the ImageNet dataset, such as pre-trained ResNet (a deep convolutional neural network) and Swin-Transformer (a transformer model with a sliding window operation and hierarchical design). (Neck network g) θ Typically, a small convolutional neural network is used to identify regions of interest and determine if those regions are defective. Bounding box regression is performed on candidate regions predicted to be defective, such as a Feature Pyramid Network (FPN). The recognition processing module (detection head h) θThis can be a region algorithm based on convolutional neural network features (R-CNN), used to integrate the neck network g θ The defective candidate regions obtained are then subjected to further bounding box regression and defect category prediction.

[0063] Smart devices can first input the image to be processed into the backbone network f θ Then the backbone network f θ The output of the input to the neck network g θ The global image features of the image to be processed are obtained, and the calculation expression is: P = g θ (f θ (x)), where x represents the image to be processed. Next, the global image features of the image to be processed can be input into the recognition processing module to obtain the first localization region y containing the target object in the image to be processed. b and the category y to which the target object in the image to be processed belongs. c The calculation expression is: {y c ,y b}=h θ (g θ (f θ (x))). It should be noted that the object detection model can also be other existing deep learning-based object detection models, and this application does not limit them.

[0064] The target detection model used in this application can first extract the region of interest through the feature extraction module and perform preliminary position regression. Then, the recognition and processing module performs defect classification and further position regression on the region of interest that has undergone preliminary regression, which can effectively improve the positioning accuracy of the first positioning region, so as to facilitate the execution of subsequent segmentation tasks.

[0065] S302. Extract the regional image features of the first positioning region from the global image features of the image to be processed; the regional image features of the first positioning region include the pixel-level features of each pixel within the first positioning region.

[0066] For a detailed explanation of the implementation process of S302, please refer to the description in S202, which will not be repeated here.

[0067] S303. Call the linear classifier to classify the pixel-level features of each pixel and obtain the first classification result of each pixel.

[0068] The first classification result includes a first probability that each pixel belongs to the target region and a second probability that it does not belong to the target region. A linear classifier (LNClassifier) ​​is used to classify and identify each pixel; it can be a linear classifier such as a logistic regression classifier, and this application does not limit this. In one embodiment, the linear classifier used in this application is a convolutional layer with a kernel size of 1×1 and an output dimension of 2. The pixel-level features of each pixel can be input into this linear classifier to determine the probability that each pixel belongs to the foreground category and the background category. Pixels belonging to the foreground category belong to the target region occupied by the target object in the image to be processed. Therefore, the probability that each pixel belongs to the foreground category can be used as the first probability of each pixel, and the probability that each pixel belongs to the background category can be used as the second probability of each pixel. Since pixel-level features are a low-dimensional vector representation (dimension size 1×c), this application can achieve good classification results using a simple linear classifier, which is simple and efficient. It should be noted that the linear classifier is obtained by adjusting the parameters of the initial linear classifier using the difference data between the mask annotation data of the training image and the extracted mask prediction data. For detailed training procedures, please refer to [link to training documentation]. Figure 6 The example embodiments are shown.

[0069] S304. Call the proxy box classifier to classify the pixel-level features of each pixel and obtain the second classification result of each pixel.

[0070] In one embodiment, the Proxy Box Classifier (BPClassifier) ​​is configured with a proxy box feature set, which includes N proxy box features. The i-th proxy box feature is obtained by global average pooling of the region image features of the i-th location region. The region image features of the i-th location region are extracted from the global image features of the sample image. The i-th location region is either a region in the sample image that contains a target object belonging to the i-th category, or a region in the sample image that does not contain a target object. i is a positive integer less than or equal to N.

[0071] Specifically, N-1 categories are predefined for the target object. The i-th category is the i-th of the N-1 categories. When i belongs to [1, N-1], a sample image containing the target object belonging to the i-th category (e.g., a pit) can be obtained (referred to as the sample image corresponding to the i-th category). The target detection model in S301 is used to detect the sample image corresponding to the i-th category, obtaining the i-th localization region containing the target object in the sample image corresponding to the i-th category, as well as the global image features of the sample image corresponding to the i-th category. Alternatively, the i-th localization region can also be obtained through manual annotation. Then, the regional image features of the i-th localization region are extracted from the global image features, and then global average pooling is performed on the regional image features of the i-th localization region to obtain the proxy feature of the i-th bounding box. The formula is expressed as: in Let P represent the i-th localization region, and P represent the global image features. The BAP algorithm represents extracting the region image features of the i-th localization region from the global image features P. Global average pooling is then applied to process the image feature B of this region. Global average pooling involves concatenating feature values ​​at the same position in the feature maps of the region's image features to obtain multiple feature vectors corresponding to those positions, and then averaging these feature vectors. This surrogate feature of the i-th bounding box can describe the characteristics of the region to which the target object belonging to the i-th category belongs.

[0072] When i is N, the i-th localization region that does not contain the target object can be extracted from the sample image. Similarly, the region image features of the i-th localization region can be extracted from the global image features of the sample image. By performing global average pooling on the region image features of the i-th localization region, the i-th bounding box surrogate feature is obtained. This i-th bounding box surrogate feature can describe the characteristics of the region that does not contain the target object.

[0073] In a feasible approach, multiple sample images containing target objects belonging to the i-th category can be acquired. The acquired sample images are processed to obtain multiple initial i-th bounding box proxy features. The average of these multiple initial i-th bounding box proxy features can be used to obtain the i-th bounding box proxy features included in the final bounding box proxy feature set.

[0074] The box proxy feature set can be represented as Where N represents the total number of box proxy features, This indicates a bounding box proxy feature generated based on a location region that does not contain the target object. In This represents the bounding box proxy feature generated based on the location region containing the target object belonging to the i-th category.

[0075] For any target pixel within the first positioning region, in one embodiment, the similarity between the pixel-level features of the target pixel and each bounding box proxy feature in the bounding box proxy feature set can be calculated. Then, based on the similarity between the pixel-level features of the target pixel and the first bounding box proxy features, a third probability is determined that the target pixel belongs to the target region. The first bounding box proxy feature is the bounding box proxy feature obtained from the N bounding box proxy features based on the positioning region containing the target object, i.e. The proxy features of each bounding box in the table are all proxy features of the first bounding box. Specifically, this can be determined... The maximum similarity between the proxy features of each first bounding box and the pixel-level features of the target pixel is used to determine the third probability that the target pixel belongs to the target region. The similarity can be measured by Euclidean distance, Manhattan distance, cosine similarity, etc. In one implementation, the value range of the obtained similarity is [0, 1].

[0076] Next, based on the similarity between the pixel-level features of the target pixel and the second bounding box proxy features, a fourth probability is determined that the target pixel does not belong to the target region. The second bounding box proxy features are the bounding box proxy features obtained from the N bounding box proxy features based on the location regions that do not contain the target object. Specifically, the second box proxy feature can be... The similarity between the pixel-level features of the target pixel and the target pixel is used to determine the fourth probability that the target pixel does not belong to the target region. The second classification result includes the third and fourth probabilities.

[0077] As can be seen, the surrogate box classifier is a metric learning-based classifier. It can describe the characteristics of regions containing different types of defects and the characteristics of regions without defects through the various surrogate box features in the surrogate box feature set. Then, by measuring the similarity between the pixel-level features of the target pixel and each surrogate box feature, it determines which type of region the characteristics reflected by the target pixel are most similar to, thereby determining whether the target pixel tends to belong to a region containing defects or not, and finally obtaining the probability that the target pixel belongs to the region occupied by the target object. Understandably, both the surrogate box features and pixel-level features are low-dimensional vector representations. This application can achieve pixel classification and recognition by calculating the similarity between vectors, achieving both high recognition accuracy and fast recognition speed. When applied to appearance defect detection, it can effectively improve the efficiency of defect detection.

[0078] S305. Call the proxy mask classifier to classify the pixel-level features of each pixel and obtain the third classification result of each pixel.

[0079] In one embodiment, the proxy mask classifier (MPClassjfjer) is configured with a mask proxy feature set, which includes S mask proxy features. The j-th mask proxy feature is obtained by global average pooling of the region image features of the j-th mask region. The region image features of the j-th mask region are extracted from the global image features of the sample image. The j-th mask region is the region occupied by the target object belonging to the j-th category in the sample image, or it is the region in the sample image that does not contain the target object. j is a positive integer less than or equal to S.

[0080] Specifically, S-1 categories (S can equal N) are predefined for the target object, and the j-th category is the j-th type among the S-1 categories. When j belongs to [1, S-1], a sample image containing the target object belonging to the j-th category (e.g., a pit) can be obtained (referred to as the sample image corresponding to the j-th category). Next, the global image features of the sample image corresponding to the j-th category are extracted, and the j-th mask region in the sample image corresponding to the j-th category is determined through manual annotation. Then, the regional image features of the j-th mask region are extracted from the global image features, and then global average pooling is performed on the regional image features of the j-th mask region to obtain the j-th mask proxy feature. The formula is expressed as: in Let P represent the j-th mask region, and P represent the global image features. The MAP algorithm extracts the region image features corresponding to the j-th mask region from the global image features P. Then, global average pooling is used to process the image features of this region. For example... Figure 4 The diagram illustrates the processing of mask proxy features extracted using the MAP algorithm. As can be seen, the MAP algorithm processes the region image features of areas occupied by different object categories to extract their corresponding mask proxy features. The j-th mask proxy feature describes the characteristics of the region occupied by the target object belonging to the j-th category.

[0081] When j is S, the j-th mask region that does not contain the target object can be extracted from the sample image. Similarly, the region image features of the j-th mask region can be extracted from the global image features of the sample image. By performing global average pooling on the region image features of the j-th mask region, the j-th mask proxy features are obtained. The j-th mask region can describe the characteristics of the region that does not contain the target object.

[0082] In a feasible approach, multiple sample images containing target objects belonging to the j-th category can be acquired. By processing the acquired sample images, multiple initial j-th mask proxy features can be obtained. The average of these multiple initial j-th mask proxy features can be used to obtain the j-th mask proxy features included in the final mask proxy feature set.

[0083] The mask proxy feature set can be represented as Where S represents the total number of mask proxy features, This indicates a mask proxy feature generated based on a mask region that does not contain the target object. In This represents the mask proxy feature generated based on the mask region containing the target object belonging to the j-th category.

[0084] For any target pixel within the first positioning region, in one embodiment, the similarity between the pixel-level features of the target pixel and each mask proxy feature in the mask proxy feature set can be calculated. Based on the similarity between the pixel-level features of the target pixel and the first mask proxy feature, the fifth probability that the target pixel belongs to the target region is determined. The first mask proxy feature is the bounding box proxy feature obtained from the S mask proxy features based on the mask region containing the target object, i.e. Each mask proxy feature in the dataset is a first mask proxy feature. Specifically, it can be determined that... The maximum similarity between each first mask proxy feature and the pixel-level feature of the target pixel is used to determine the fifth probability that the target pixel belongs to the target region.

[0085] Next, based on the similarity between the pixel-level features of the target pixel and the second mask proxy feature, the sixth probability that the target pixel does not belong to the target region is determined. The second mask proxy feature is the mask proxy feature obtained from the S mask proxy features based on the mask region that does not contain the target object, i.e. Specifically, the second mask proxy feature can be used. The similarity between the pixel-level features of the target pixel and the pixel-level features of the target pixel is used to determine the sixth probability that the target pixel does not belong to the target region. The third classification result includes the fifth and sixth probabilities.

[0086] As can be seen, the proxy mask classifier is also a metric learning-based classifier. It can describe the characteristics of regions occupied by different types of defects and the characteristics of regions without defects through the various proxy mask features in the proxy mask feature set. Then, by measuring the similarity between the pixel-level features of the target pixel and each proxy mask feature, it determines which type of region the characteristics reflected by the target pixel are more similar to, thereby determining whether the target pixel contains a defective region or not, and finally obtaining the probability that the target pixel belongs to the region occupied by the target object. Understandably, both the proxy mask features and the pixel-level features are low-dimensional vector representations. This application can achieve pixel classification and recognition by calculating the similarity between vectors, achieving both high recognition accuracy and fast recognition speed. When applied to appearance defect detection, it can effectively improve the efficiency of defect detection.

[0087] Although mask proxy features can more accurately describe the characteristics of defective regions compared to bounding box proxy features, this application combines a proxy bounding box classifier and a proxy mask classifier (as well as a linear classifier) ​​to achieve classification and recognition of pixels from multiple dimensions, which is beneficial to further improve the accuracy of classification and recognition.

[0088] S306. Based on one or more of the first classification result, the second classification result, and the third classification result, determine the target classification result for each pixel. The target classification result is used to indicate whether each pixel belongs to the target area occupied by the target object in the image to be processed.

[0089] In one embodiment, one of the first classification result, second classification result, and third classification result of each pixel can be determined as the target classification result of each pixel. Taking the determination of the target classification result of each pixel based on the first classification result as an example, the first probability that the target pixel belongs to the target region can be used as the first predicted probability that the target pixel belongs to the target region in the target classification result of the target pixel; and the second probability that the target pixel belongs to the target region can be used as the second predicted probability that the target pixel does not belong to the target region in the target classification result of the target pixel.

[0090] In another embodiment, the target classification result of each pixel can be determined based on any two of the first, second, and third classification results. Taking the determination of the target classification result of each pixel based on the first and second classification results as an example, the first probability and the third probability of the target pixel belonging to the target region can be added together to obtain the first predicted probability that the target pixel belongs to the target region in the target classification result; and the second probability and the fourth probability of the target pixel belonging to the target region can be added together to obtain the second predicted probability that the target pixel does not belong to the target region in the target classification result.

[0091] In a feasible embodiment, the target classification result of each pixel can be determined based on the first classification result, the second classification result, and the third classification result of each pixel. Specifically, the first probability, the third probability, and the fifth probability that the target pixel belongs to the target region can be added together to obtain the first predicted probability that the target pixel belongs to the target region in the target classification result of the target pixel; and the second probability, the fourth probability, and the sixth probability that the target pixel belongs to the target region can be added together to obtain the second predicted probability that the target pixel does not belong to the target region in the target classification result of the target pixel.

[0092] If the first predicted probability of the target pixel is greater than the second predicted probability, the target classification result of the target pixel indicates that the target pixel belongs to the target region. If the first predicted probability of the target pixel is less than or equal to the second predicted probability, the target classification result of the target pixel indicates that the target pixel does not belong to the target region.

[0093] S307. Based on the target classification results of each pixel, perform masking processing on the first localization region to obtain a mask image of the target object; the mask image is used to segment the target region in the image to be processed.

[0094] For details on the implementation of S307, please refer to the description in S204, which will not be repeated here.

[0095] In summary, Figure 5 This application provides a defect detection model for appearance defect detection, which includes a target detection model and a target segmentation model. The target detection model includes a backbone network f... θ Neck network g θ and detection head h θ The target segmentation model includes a linear classifier, a surrogate box classifier, and a surrogate mask classifier. The image to be processed can be input into the target detection model, which extracts global image features, the first localization region (i.e., bounding box) containing the target object, and the category of the target object. Then, using the first localization region, regional image features of the first localization region are extracted from the global image features of the image to be processed. These regional image features are used as input to the target segmentation model, and the final output is a mask image of the target object. Figure 5 The image shows the mask images of the target object extracted using a linear classifier, a surrogate box classifier, and a surrogate mask classifier, respectively. It can be seen that this application can add the mask images of the target object extracted by the linear classifier, the surrogate box classifier, and the surrogate mask classifier separately to obtain a better mask image. The formula is expressed as:

[0096] Where LNClassifier(B) is the mask image output by the linear classifier, and BPClassifier(B,P) is the mask image output by the linear classifier. B MPClassifier(B, P) is the mask image output by the proxy box classifier. M ) is the mask image output by the proxy mask classifier.

[0097] It should be noted that the linear classifier in the object segmentation model is obtained by adjusting the parameters of the initial linear classifier using the difference data between the mask annotation data of the training image and the extracted mask prediction data. For detailed training procedures, please refer to [link to training documentation]. Figure 6 The illustrated embodiment. Since this application uses a linear classifier, and the surrogate box classifier and surrogate mask classifier do not require parameter tuning during training, only a small number of training images are needed to ensure good accuracy of the object segmentation model. The object detection model can be obtained through fully supervised training using training images and their bounding box annotations.

[0098] In this embodiment, a first localization region containing the target object can be extracted from the image to be processed. Then, a linear classifier, a surrogate box classifier, and a surrogate mask classifier can be used to classify the pixel-level features of each pixel in the first localization region to determine whether each pixel belongs to the target region occupied by the target object in the image to be processed (i.e., the target classification result). By combining the classification results output by these three classifiers, accurate pixel-level prediction can be achieved. Then, a mask image of the target object can be generated based on the target classification result of each pixel, which can achieve accurate segmentation of defective regions and improve the accuracy of image segmentation.

[0099] Please see Figure 6 , Figure 6 This is a flowchart illustrating a model training method provided in an embodiment of this application. The method can be executed by a smart device, for example, through the aforementioned... Figure 1 The method, executed by server 102, includes the following steps.

[0100] S601. Obtain the training image and the mask annotation data of the training image.

[0101] Mask annotation data refers to the information used to annotate the region occupied by the target object in the training image. Specifically, pixel-level annotation can be used, which can identify whether each pixel in the training image belongs to the region occupied by the target object. When it belongs to the region, the annotation label for each pixel is 1; when it does not belong to the region, the annotation label for each pixel is 0.

[0102] S602. Extract global image features from the training image and determine the second localization region containing the target object in the training image. Extract the regional image features of the second localization region from the global image features of the training image.

[0103] In one embodiment, the target detection model described in S301 above can be used to extract global image features from the training image and determine a second localization region containing the target object. The feature size of each feature map in the global image features can be matched with the image size of the training image. Then, the regional image features of the second localization region are extracted from the global image features of the training image based on the regional layout data of the second localization region.

[0104] S603. Based on the initial segmentation model, the regional image features of the second localization region are segmented to obtain mask prediction data; the initial segmentation model includes an initial linear classifier, a surrogate box classifier, and a surrogate mask classifier.

[0105] The initial linear classifier can be a 1×1 convolutional layer to be trained. The region image features of the second localization region include the pixel-level features of each pixel within the second localization region. For any pixel among these pixels, in one embodiment, the pixel-level features of any pixel can be classified using the initial linear classifier, the surrogate box classifier, and the surrogate mask classifier. The processing logic in S306, which determines the target classification result of each pixel based on the first, second, and third classification results of each pixel, is used to determine the third predicted probability that any pixel belongs to the region occupied by the target object in the training image and the fourth predicted probability that it does not belong to that region.

[0106] S604. Based on the difference between the mask annotation data and the mask prediction data, adjust the parameters of the initial linear classifier to obtain a linear classifier.

[0107] In one embodiment, mask annotation data for a second localization region can be obtained from the mask annotation data of the training image, i.e., the annotation labels of each pixel within the second localization region. Based on the annotation labels of each pixel within the second localization region and the classification results (including third and fourth prediction probabilities) of each pixel within the second localization region, a cross-entropy loss is determined, and this cross-entropy loss is then used as the difference data between the mask annotation data and the mask prediction data. The parameters of the initial linear classifier are adjusted in the direction of reducing the difference data. It is understood that multiple training images can be used to adjust the parameters of the initial linear classifier multiple times. During this continuous adjustment process, the actual difference data will continuously decrease, thereby enabling the parameter-adjusted initial linear classifier to predict the probability that a pixel belongs to the region occupied by the target object more and more accurately. When the parameter-adjusted initial linear classifier meets the convergence condition, the latest adjusted initial linear classifier can be used as the linear classifier. The convergence condition can be that the number of parameter adjustments reaches a preset adjustment threshold, or that the actual loss value is less than the loss threshold. This loss threshold is the minimum loss value, used to indicate that the stability of the linear classifier has reached the desired stability.

[0108] In this embodiment, an initial segmentation model can be used to predict the probability that each pixel in the second localization region of the training image belongs to the region occupied by the target object. Based on this probability and the mask annotation data of the training image, the difference data can be determined, thereby training the initial linear classifier to obtain a linear classifier. Since the linear classifier is a simple convolutional layer, and the surrogate box classifier and surrogate mask classifier do not require parameter tuning through training, only a small number of training images are needed to ensure that the target segmentation model including the linear classifier, surrogate box classifier, and surrogate mask classifier has good accuracy.

[0109] It is understood that in the specific implementation of this application, data such as images to be processed are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0110] The methods of the embodiments of this application have been described in detail above. To facilitate better implementation of the methods of the embodiments of this application, the apparatus of the embodiments of this application is provided below. Please refer to... Figure 7 , Figure 7 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. The image processing apparatus 70 may include:

[0111] The acquisition unit 701 is used to acquire the global image features of the image to be processed;

[0112] The processing unit 702 is configured to determine a first location region containing a target object in the image to be processed, and extract the regional image features of the first location region from the global image features of the image to be processed; the regional image features of the first location region include the pixel-level features of each pixel point in the first location region.

[0113] The processing unit 702 is further configured to perform pixel-level feature classification processing on each pixel to obtain a target classification result for each pixel; the target classification result is used to indicate whether each pixel belongs to the target area occupied by the target object in the image to be processed.

[0114] The processing unit 702 is further configured to perform masking processing on the first positioning region according to the target classification results of each pixel to obtain a mask image of the target object; the mask image is used to segment the target region in the image to be processed.

[0115] In one embodiment, the processing unit 702 is specifically configured to: invoke a linear classifier to classify the pixel-level features of each pixel to obtain a first classification result for each pixel, wherein the first classification result includes a first probability that each pixel belongs to the target region and a second probability that it does not belong to the target region; invoke a surrogate box classifier to classify the pixel-level features of each pixel to obtain a second classification result for each pixel; invoke a surrogate mask classifier to classify the pixel-level features of each pixel to obtain a third classification result for each pixel; and determine a target classification result for each pixel based on one or more of the first classification result, the second classification result, and the third classification result.

[0116] In one embodiment, the proxy box classifier is configured with a proxy box feature set, which includes N proxy box features. The i-th proxy box feature is obtained by global average pooling of the region image features of the i-th location region. The region image features of the i-th location region are extracted from the global image features of the sample image. The i-th location region is either a region in the sample image that contains a target object belonging to the i-th category, or a region in the sample image that does not contain a target object. i is a positive integer less than or equal to N. The second classification result includes a third probability and a fourth probability.

[0117] The processing unit 702 is specifically used to: determine the third probability that the target pixel belongs to the target region based on the similarity between the pixel-level features of the target pixel and the first box proxy feature, wherein the first box proxy feature is a box proxy feature obtained from the N box proxy features based on the positioning region containing the target object, and the target pixel is any one of the pixels; and determine the fourth probability that the target pixel does not belong to the target region based on the similarity between the pixel-level features of the target pixel and the second box proxy feature, wherein the second box proxy feature is a box proxy feature obtained from the N box proxy features based on the positioning region not containing the target object.

[0118] In one embodiment, the proxy mask classifier is configured with a proxy mask feature set, which includes S proxy mask features. The j-th proxy mask feature is obtained by global average pooling of the region image features of the j-th mask region. The region image features of the j-th mask region are extracted from the global image features of the sample image. The j-th mask region is the region occupied by the target object belonging to the j-th category in the sample image, or it is a region in the sample image that does not contain the target object. j is a positive integer less than or equal to S. The third classification result includes a fifth probability and a sixth probability.

[0119] The processing unit 702 is specifically used to: determine the fifth probability that the target pixel belongs to the target region based on the similarity between the pixel-level features of the target pixel and the first mask proxy feature, wherein the first mask proxy feature is a bounding box proxy feature obtained from the S mask proxy features based on the mask region containing the target object, and the target pixel is any one of the pixels; and determine the sixth probability that the target pixel does not belong to the target region based on the similarity between the pixel-level features of the target pixel and the second mask proxy feature, wherein the second mask proxy feature is a mask proxy feature obtained from the S mask proxy features based on the mask region not containing the target object.

[0120] In one embodiment, the acquisition unit 701 is specifically used to: acquire training images and mask annotation data of the training images;

[0121] The processing unit 702 is specifically used for: extracting global image features of the training image and determining a second localization region containing a target object in the training image; extracting regional image features of the second localization region from the global image features of the training image; segmenting the regional image features of the second localization region according to an initial segmentation model to obtain mask prediction data; the initial segmentation model includes an initial linear classifier, the surrogate box classifier, and the surrogate mask classifier; and adjusting the parameters of the initial linear classifier according to the difference data between the mask annotation data and the mask prediction data to obtain the linear classifier.

[0122] In one embodiment, the global image features of the image to be processed include one or more first feature maps corresponding to feature channels, and the feature size of each first feature map matches the image size of the image to be processed; the acquisition unit 701 is specifically used to: acquire the region layout data of the first positioning region in the image to be processed;

[0123] The processing unit 702 is specifically used to: extract second feature maps from each of the first feature maps according to the regional layout data; splice the feature values ​​at the same position in each of the obtained second feature maps to obtain the pixel-level features of each pixel in the first positioning region; and determine the regional image features of the first positioning region according to the pixel-level features of each pixel in the first positioning region.

[0124] In one embodiment, the processing unit 702 is specifically configured to: determine one or more reference pixels from the pixels indicating that the target classification result belongs to the target region; set the pixel value of each reference pixel in the first positioning region as a first mask value, and set the pixel value of the pixels in the first positioning region other than the reference pixels as a second mask value, thereby obtaining a mask image of the target object.

[0125] In one embodiment, the first localization region is obtained by calling a target detection model to detect and process the image to be processed. The target detection model includes a feature extraction module and a recognition processing module. The feature extraction module is used to extract global image features of the image to be processed, and the recognition processing module is used to perform recognition processing on the global image features of the image to be processed to obtain the first localization region and the category to which the target object in the image to be processed belongs.

[0126] It is understood that the functions of each functional unit of the image processing apparatus described in the embodiments of this application can be specifically implemented according to the methods in the above method embodiments, and the specific implementation process can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0127] like Figure 8 As shown, Figure 8 This is a schematic diagram of the structure of a smart device provided in an embodiment of this application. The internal structure of the smart device 80 is as follows: Figure 8 As shown, it includes: one or more processors 801, memory 802, and communication interface 803. The processor 801, memory 802, and communication interface 803 can be connected via bus 804 or other means. This embodiment of the application takes the connection via bus 804 as an example.

[0128] The processor 801 (or CPU, Central Processing Unit) is the computing and control core of the smart device 80. It can parse various instructions within the smart device 80 and process various data. For example, the CPU can parse power-on / off commands sent to the smart device 80 and control the smart device 80 to perform power-on / off operations; it can also transmit various interactive data between internal structures of the smart device 80, and so on. The communication interface 803 may optionally include standard wired interfaces or wireless interfaces (such as Wi-Fi, mobile communication interfaces, etc.), and is controlled by the processor 801 for sending and receiving data. The memory 802 is the storage device in the smart device 80, used to store computer programs and data. It is understood that the memory 802 here can include the built-in memory of the smart device 80, or it can include extended memory supported by the smart device 80. The memory 802 provides storage space, which stores the operating system of the smart device 80, including but not limited to: Windows systems, Linux systems, Android systems, iOS systems, etc., which are not limited in this application. In one embodiment, processor 801 performs the following operations by running a computer program stored in memory 802:

[0129] Obtain the global image features of the image to be processed, and determine the first localization region in the image to be processed that contains the target object;

[0130] Extract the regional image features of the first positioning region from the global image features of the image to be processed; the regional image features of the first positioning region include the pixel-level features of each pixel within the first positioning region;

[0131] The pixel-level features of each pixel are classified to obtain the target classification result of each pixel; the target classification result is used to indicate whether each pixel belongs to the target region occupied by the target object in the image to be processed.

[0132] Based on the target classification results of each pixel, the first localization region is masked to obtain a mask image of the target object; the mask image is used to segment the target region in the image to be processed.

[0133] In one embodiment, the processor 801 is specifically configured to: invoke a linear classifier to classify the pixel-level features of each pixel to obtain a first classification result for each pixel, the first classification result including a first probability that each pixel belongs to the target region and a second probability that it does not belong to the target region; invoke a surrogate box classifier to classify the pixel-level features of each pixel to obtain a second classification result for each pixel; invoke a surrogate mask classifier to classify the pixel-level features of each pixel to obtain a third classification result for each pixel; and determine a target classification result for each pixel based on one or more of the first classification result, the second classification result, and the third classification result.

[0134] In one embodiment, the proxy box classifier is configured with a proxy box feature set, which includes N proxy box features. The i-th proxy box feature is obtained by global average pooling of the region image features of the i-th location region. The region image features of the i-th location region are extracted from the global image features of the sample image. The i-th location region is either a region in the sample image that contains a target object belonging to the i-th category, or a region in the sample image that does not contain a target object. i is a positive integer less than or equal to N. The second classification result includes a third probability and a fourth probability.

[0135] The processor 801 is specifically configured to: determine a third probability that the target pixel belongs to the target region based on the similarity between the pixel-level features of the target pixel and the first box proxy feature, wherein the first box proxy feature is a box proxy feature obtained from the N box proxy features based on a positioning region containing the target object, and the target pixel is any one of the pixels; and determine a fourth probability that the target pixel does not belong to the target region based on the similarity between the pixel-level features of the target pixel and the second box proxy feature, wherein the second box proxy feature is a box proxy feature obtained from the N box proxy features based on a positioning region that does not contain the target object.

[0136] In one embodiment, the proxy mask classifier is configured with a proxy mask feature set, which includes S proxy mask features. The j-th proxy mask feature is obtained by global average pooling of the region image features of the j-th mask region. The region image features of the j-th mask region are extracted from the global image features of the sample image. The j-th mask region is the region occupied by the target object belonging to the j-th category in the sample image, or it is a region in the sample image that does not contain the target object. j is a positive integer less than or equal to S. The third classification result includes a fifth probability and a sixth probability.

[0137] The processor 801 is specifically configured to: determine the fifth probability that the target pixel belongs to the target region based on the similarity between the pixel-level features of the target pixel and the first mask proxy feature, wherein the first mask proxy feature is a bounding box proxy feature obtained from the S mask proxy features based on a mask region containing the target object, and the target pixel is any one of the pixels; and determine the sixth probability that the target pixel does not belong to the target region based on the similarity between the pixel-level features of the target pixel and the second mask proxy feature, wherein the second mask proxy feature is a mask proxy feature obtained from the S mask proxy features based on a mask region not containing the target object.

[0138] In one embodiment, the processor 801 is specifically configured to: acquire a training image and mask annotation data of the training image; extract global image features of the training image and determine a second localization region containing a target object in the training image; extract regional image features of the second localization region from the global image features of the training image; perform segmentation processing on the regional image features of the second localization region according to an initial segmentation model to obtain mask prediction data; the initial segmentation model includes an initial linear classifier, the surrogate box classifier, and the surrogate mask classifier; adjust the parameters of the initial linear classifier according to the difference data between the mask annotation data and the mask prediction data to obtain the linear classifier.

[0139] In one embodiment, the global image features of the image to be processed include one or more first feature maps corresponding to feature channels, and the feature size of each first feature map matches the image size of the image to be processed; the processor 801 is specifically configured to: acquire the region layout data of the first positioning region in the image to be processed, and extract second feature maps from each of the first feature maps according to the region layout data; concatenate the feature values ​​at the same position in each of the obtained second feature maps to obtain the pixel-level features of each pixel in the first positioning region; and determine the region image features of the first positioning region according to the pixel-level features of each pixel in the first positioning region.

[0140] In one embodiment, the processor 801 is specifically configured to: determine one or more reference pixels from the pixels indicating that the target classification result belongs to the target region; set the pixel values ​​of each reference pixel in the first positioning region as a first mask value, and set the pixel values ​​of pixels in the first positioning region other than the reference pixels as second mask values, thereby obtaining a mask image of the target object.

[0141] In one embodiment, the first localization region is obtained by calling a target detection model to detect and process the image to be processed. The target detection model includes a feature extraction module and a recognition processing module. The feature extraction module is used to extract global image features of the image to be processed, and the recognition processing module is used to perform recognition processing on the global image features of the image to be processed to obtain the first localization region and the category to which the target object in the image to be processed belongs.

[0142] The processor 801, memory 802, and communication interface 803 described in this application embodiment can execute the implementation method described in the image processing method provided in this application embodiment, or the implementation method described in the image processing apparatus provided in this application embodiment, and will not be repeated here.

[0143] In this embodiment, starting from the first positioning region in the image to be processed, pixel-level features of each pixel in the first positioning region can be extracted to achieve accurate prediction of image segmentation at the pixel level. Then, a mask image of the target object can be generated based on the prediction results of each pixel. The mask image can indicate the position and shape of the target object in the image to be processed. Therefore, the area occupied by the target object in the image to be processed can be segmented using the mask image. By improving the accuracy of image segmentation, accurate identification of defective areas can be achieved.

[0144] This application also provides a computer-readable storage medium storing a computer program that, when run on a smart device, causes the smart device to execute the image processing method described in any of the possible implementations above. Specific implementation details are provided above and will not be repeated here.

[0145] This application also provides a computer program product, which includes a computer program or computer instructions. When executed by a processor, the computer program or computer instructions implement the steps of the image processing method provided in this application. The specific implementation method can be found in the foregoing description and will not be repeated here.

[0146] This application also provides a computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a smart device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the smart device to perform the image processing method provided in this application. The specific implementation can be found in the foregoing description and will not be repeated here.

[0147] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0148] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0149] The above-disclosed embodiments are only some of the embodiments of this application, and should not be construed as limiting the scope of this application. Therefore, any equivalent changes made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the global image features of the image to be processed, and determine the first localization region in the image to be processed that contains the target object; Extract the regional image features of the first positioning region from the global image features of the image to be processed; the regional image features of the first positioning region include the pixel-level features of each pixel within the first positioning region; Based on one or more of the first classification result, the second classification result, and the third classification result of each pixel, a target classification result is determined for each pixel. The target classification result indicates whether each pixel belongs to the target region occupied by the target object in the image to be processed. The first classification result is obtained by calling a linear classifier to classify the pixel-level features of each pixel. The second classification result is obtained by calling a surrogate box classifier to classify the pixel-level features of each pixel. The third classification result is obtained by calling a surrogate mask classifier to classify the pixel-level features of each pixel. The first classification result, the second classification result, and the third classification result all include the probability that each pixel belongs to the target region and the probability that it does not belong to the target region. Based on the target classification results of each pixel, the first localization region is masked to obtain a mask image of the target object; the mask image is used to segment the target region in the image to be processed.

2. The method according to claim 1, characterized in that, The proxy box classifier is configured with a proxy box feature set, which includes N proxy box features. The i-th proxy box feature is obtained by global average pooling of the region image features of the i-th localization region. The region image features of the i-th localization region are extracted from the global image features of the sample image. The i-th localization region is either a region in the sample image that contains a target object belonging to the i-th category, or a region in the sample image that does not contain a target object. i is a positive integer less than or equal to N. The second classification result includes a third probability and a fourth probability. The proxy box classifier is invoked to classify the pixel-level features of each pixel, resulting in a second classification result for each pixel, including: Based on the similarity between the pixel-level features of the target pixel and the first bounding box proxy features, the third probability that the target pixel belongs to the target region is determined. The first bounding box proxy features are the bounding box proxy features obtained from the N bounding box proxy features based on the positioning region containing the target object. The target pixel is any one of the pixels. Based on the similarity between the pixel-level features of the target pixel and the second box proxy features, the fourth probability that the target pixel does not belong to the target region is determined. The second box proxy features are box proxy features obtained from the N box proxy features based on the positioning region that does not contain the target object.

3. The method according to claim 1, characterized in that, The proxy mask classifier is configured with a proxy mask feature set, which includes S proxy mask features. The j-th proxy mask feature is obtained by global average pooling of the regional image features of the j-th mask region. The regional image features of the j-th mask region are extracted from the global image features of the sample image. The j-th mask region is the area occupied by the target object belonging to the j-th category in the sample image, or it is the area in the sample image that does not contain the target object. j is a positive integer less than or equal to S. The third classification result includes a fifth probability and a sixth probability. The proxy mask classifier is invoked to classify the pixel-level features of each pixel, resulting in a third classification result for each pixel, including: The fifth probability that the target pixel belongs to the target region is determined based on the similarity between the pixel-level features of the target pixel and the first mask proxy feature. The first mask proxy feature is a box proxy feature obtained from the S mask proxy features based on the mask region containing the target object. The target pixel is any one of the pixels. Based on the similarity between the pixel-level features of the target pixel and the second mask proxy feature, the sixth probability that the target pixel does not belong to the target region is determined. The second mask proxy feature is the mask proxy feature obtained from the S mask proxy features based on the mask region that does not contain the target object.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: Acquire training images and their mask annotation data; Extract global image features from the training image, determine a second localization region containing the target object in the training image, and extract regional image features of the second localization region from the global image features of the training image; The region image features of the second localization region are segmented according to the initial segmentation model to obtain mask prediction data; the initial segmentation model includes an initial linear classifier, the surrogate box classifier, and the surrogate mask classifier; Based on the difference between the mask annotation data and the mask prediction data, the parameters of the initial linear classifier are adjusted to obtain the linear classifier.

5. The method according to any one of claims 1-3, characterized in that, The global image features of the image to be processed include one or more first feature maps corresponding to feature channels, and the feature size of each first feature map matches the image size of the image to be processed. Extracting the regional image features of the first localization region from the global image features of the image to be processed includes: Obtain the region layout data of the first positioning region in the image to be processed, and extract the second feature map from each of the first feature maps based on the region layout data; The feature values ​​at the same position in the obtained second feature maps are concatenated to obtain the pixel-level features of each pixel in the first positioning region. The regional image features of the first positioning region are determined based on the pixel-level features of each pixel in the first positioning region.

6. The method according to claim 1, characterized in that, The step of performing masking processing on the first positioning region based on the classification results of each pixel to obtain a mask image of the target object includes: From each of the pixels, determine one or more reference pixels that indicate the target region as belonging to the target classification result; The pixel values ​​of each reference pixel in the first positioning region are set as the first mask value, and the pixel values ​​of all pixels in the first positioning region other than the reference pixels are set as the second mask value, to obtain the mask image of the target object.

7. The method according to claim 1, characterized in that, The first localization region is obtained by calling a target detection model to detect and process the image to be processed. The target detection model includes a feature extraction module and a recognition processing module. The feature extraction module is used to extract global image features of the image to be processed, and the recognition processing module is used to recognize and process the global image features of the image to be processed to obtain the first localization region and the category to which the target object in the image to be processed belongs.

8. An image processing apparatus, characterized in that, The device includes: The acquisition unit is used to acquire the global image features of the image to be processed; The processing unit is configured to determine a first location region containing a target object in the image to be processed, and extract the regional image features of the first location region from the global image features of the image to be processed; the regional image features of the first location region include the pixel-level features of each pixel point within the first location region. The processing unit is further configured to determine the target classification result of each pixel based on one or more of the first classification result, the second classification result, and the third classification result of each pixel; the target classification result is used to indicate whether each pixel belongs to the target region occupied by the target object in the image to be processed; the first classification result of each pixel is obtained by calling a linear classifier to classify the pixel-level features of each pixel, the second classification result of each pixel is obtained by calling a surrogate box classifier to classify the pixel-level features of each pixel, and the third classification result of each pixel is obtained by calling a surrogate mask classifier to classify the pixel-level features of each pixel; the first classification result, the second classification result, and the third classification result all include the probability that each pixel belongs to the target region and the probability that it does not belong to the target region. The processing unit is further configured to perform masking processing on the first positioning region based on the target classification results of each pixel to obtain a mask image of the target object; the mask image is used to segment the target region in the image to be processed.

9. A smart device, characterized in that, The intelligent device includes a memory, a communication interface, and a processor, wherein the memory, the communication interface, and the processor are interconnected; the memory stores a computer program, and the processor calls the computer program stored in the memory to implement the image processing method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the image processing method as described in any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, implement the image processing method as described in any one of claims 1-7.

Images