Target object identification method and its device, equipment, medium, and product

Abstract

The present application discloses a method for identifying a target item and its apparatus, device, medium, and product. The method comprises: obtaining an item image to be identified as containing a target item; performing encoding and decoding on the item image at multiple scales to obtain multiple image feature information capturing the item's contour features; extracting an item segmentation map from the item image based on the multiple image feature information; classifying each image feature information to obtain a classification probability that the image feature information contains the target item, and calculating the average probability of the multiple classification probabilities obtained by classifying all image feature information; performing image recognition on the item segmentation map to obtain a recognition probability that the item segmentation map contains the target item; fusing the average probability with the recognition probability to obtain a fused probability for result judgment; and determining that the item image contains the target item when the fused probability is greater than a preset threshold. The present application can more accurately identify target items from item images.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a target object recognition method and its corresponding device, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Image recognition using artificial neural network models has become a mainstream technology. For applications like e-commerce platforms, massive quantities of product and other images are generated daily. For various applications, including automated product classification, compliance testing, and targeted product search, it is generally necessary to detect and identify the target objects in user-uploaded product images. Further processing is then performed based on the specific target objects and needs.

[0003] In application scenarios like e-commerce platforms, if merchants upload product images for display that include items not for sale, such as knives and swords, existing neural network models sometimes struggle to accurately identify these items from product images due to their small size, reflective nature, and diverse appearance. This poses significant challenges for e-commerce platforms, and relying solely on manual screening is unrealistic.

[0004] Therefore, how to accurately and efficiently identify the target object, that is, the target image, from various types of object pictures containing object images so that the recognition results are more reliable has become a technical problem that needs to be solved in this field. Summary of the Invention

[0005] The primary purpose of the present application is to solve at least one of the above problems and to provide a target object identification method and its corresponding device, computer equipment, computer-readable storage medium, and computer program product.

[0006] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0007] A method for identifying a target object provided for one of the purposes of this application includes the following steps:

[0008] Get the image of the object to be identified to see if it contains the target object;

[0009] Calling a pre-trained image segmentation model to perform encoding and decoding on the object image at multiple scales to obtain multiple image feature information capturing contour features of the object, and extracting an object segmentation map from the object image based on the multiple image feature information;

[0010] Classifying each of the image feature information in the image segmentation model to obtain a classification probability that the image feature information includes the target object, and calculating an average probability of multiple classification probabilities obtained by classifying all the image feature information;

[0011] Calling a pre-trained image recognition model to perform image recognition on the object segmentation map to obtain a recognition probability that the object segmentation map contains the target object;

[0012] The average probability and the recognition probability are fused to obtain a fusion probability for result judgment. When the fusion probability is greater than a preset threshold, it is determined that the object image contains the target object, otherwise it does not contain the target object.

[0013] In a further embodiment, a pre-trained image segmentation model is called to perform encoding and decoding on the object image at multiple scales to obtain multiple image feature information capturing the contour features of the object, and an object segmentation map is extracted from the object image based on the multiple image feature information, including the following steps:

[0014] The pre-trained image segmentation model performs multi-level encoding to reduce the original size of the object image step by step and generates intermediate feature information corresponding to each scale. The intermediate feature information is used to represent the outline features of the object in the object image;

[0015] Multi-level decoding of the pre-trained image segmentation model, based on the image feature information produced by the smallest-scale intermediate feature information, uses the intermediate feature information produced by the encoding of the same level as reference information to decode the corresponding higher-scale image feature information. The image feature information is used to represent the contour features of the object in the object image in the form of a mask;

[0016] The image segmentation model performs image segmentation on the standard original image of the object image based on the mask image data formed by fusing all image feature information, so as to extract the object segmentation map corresponding to the object in the object image.

[0017] In a specific embodiment, an image segmentation model performs image segmentation on the standard original image of the item image based on mask image data formed by fusing all image feature information, so as to extract an item segmentation map corresponding to the item in the item image, including the following steps:

[0018] All image feature information is fully connected by a fully connected layer in the image segmentation model to fuse and generate mask image data, wherein the mask image data inherits the contour features of the object in the image feature information;

[0019] The image of the specification original image of the object picture is extracted according to the mask image data, and the image within the outline corresponding to the outline feature of the object is obtained to form the object segmentation map.

[0020] In a further embodiment, the average probability and the recognition probability are combined to obtain a fusion probability for result judgment, including the following steps:

[0021] Obtaining the average probability and the recognition probability;

[0022] fusing the average probability and the recognition probability to calculate a fused probability, wherein the average probability and the recognition probability are associated with the same hyperparameter for smoothing;

[0023] The fusion probability is compared with a preset threshold value. When the fusion probability is greater than the preset threshold value, it is determined that the object image contains the target object; otherwise, it is determined that the object image does not contain the target object.

[0024] In a specific embodiment, the image feature information corresponding to multiple scales generated by the image segmentation model is output to a binary classifier corresponding to each scale for binary classification decision to obtain the classification probability.

[0025] In a preferred embodiment, the image segmentation model is U 2 net model, the basic network architecture of the image segmentation model is U 2 net model, the target item is a knife or a sword.

[0026] A target object identification device provided to meet one of the objectives of the present application includes an image acquisition module, an image segmentation module, a contour classification module, an object classification module, and a fusion judgment module. The image acquisition module is configured to acquire an object image to be identified as containing a target object. The image segmentation module is configured to invoke a pretrained image segmentation model to perform encoding and decoding on the object image at multiple scales to obtain multiple image feature information capturing the object's contour features, and to extract an object segmentation map from the object image based on the multiple image feature information. The contour classification module is configured to classify each image feature information in the image segmentation model to obtain a classification probability that the image feature information contains the target object, and to calculate an average probability of the multiple classification probabilities obtained by classifying all image feature information. The object classification module is configured to invoke a pretrained image recognition model to perform image recognition on the object segmentation map to obtain a recognition probability that the object segmentation map contains the target object. The fusion judgment module is configured to fuse the average probability with the recognition probability to obtain a fused probability for result judgment. When the fused probability is greater than a preset threshold, the object image is determined to contain the target object; otherwise, the object image is determined to contain the target object.

[0027] In a further embodiment, the image segmentation module includes: an encoding path unit, which is configured to perform multi-level encoding of a pre-trained image segmentation model, gradually reduce the scale of the original specification image of the object image, and correspondingly produce intermediate feature information corresponding to each scale, wherein the intermediate feature information is used to represent the contour features of the object in the object image; a decoding path unit, which is configured to perform multi-level decoding of a pre-trained image segmentation model, based on the image feature information produced by the intermediate feature information of the smallest scale, and gradually use the intermediate feature information produced by the encoding of the same level as reference information to correspondingly decode higher-scale image feature information, wherein the image feature information is used to represent the contour features of the object in the object image in the form of a mask; a fusion segmentation unit, which is used to perform image segmentation on the original specification image of the object image based on the mask image data fused by the image segmentation model according to all the image feature information, so as to extract the object segmentation map corresponding to the object in the object image.

[0028] In a specific embodiment, the fusion and segmentation unit includes: a fusion subunit, which is used to fully connect all image feature information by the fully connected layer in the image segmentation model to fuse and generate mask image data, and the mask image data inherits the contour features of the object in the image feature information; a segmentation subunit, which is used to extract the image of the specification original image of the object image according to the mask image data, and obtain the image within the contour corresponding to the contour features of the object to form the object segmentation map.

[0029] In a further embodiment, the fusion judgment module includes: a probability acquisition sub-block, used to obtain the average probability and the recognition probability; a mean calculation sub-block, used to fuse the average probability and the recognition probability to calculate the fusion probability, wherein the average probability and the recognition probability are associated with the same hyperparameter for smoothing; a result judgment sub-module, used to compare the fusion probability with a preset threshold, when the fusion probability is greater than the preset threshold, it is determined that the object image contains the target object, otherwise it is determined that the target object is not contained.

[0030] In a specific embodiment, the image feature information corresponding to multiple scales generated by the image segmentation model is output to a binary classifier corresponding to each scale for binary classification decision to obtain the classification probability.

[0031] In a preferred embodiment, the image segmentation model is U 2 net model, the basic network architecture of the image segmentation model is U 2 net model, the target item is a knife or a sword.

[0032] A computer device provided to meet one of the purposes of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the target object identification method described in the present application.

[0033] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the target object identification method in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the method are executed.

[0034] A computer program product provided to meet another purpose of the present application includes a computer program / instruction, which, when executed by a processor, implements the steps of the method described in any embodiment of the present application.

[0035] Compared with the prior art, the advantages of this application are as follows:

[0036] This application first uses an image segmentation model to segment an image of an object requiring target object recognition, obtaining image feature information at multiple scales representing the outline features of the object in the image. Based on this information, the application then proceeds in two steps: one in which the image feature information corresponding to each scale is individually classified, obtaining the classification probability that each image feature information indicates that the image contains the target object, and calculating the average probability of the multiple classification probabilities; the other in which the image feature information corresponding to each scale is fused to form mask image data representing the outline features of the object in the image. Based on this mask image data, an object segmentation map is segmented from the image. The image recognition model is then used to identify this object segmentation map as the target object, obtaining the recognition probability that the object segmentation map contains the target object. Based on the obtained average probability and recognition probability, the two probabilities are further fused into a single fused probability, which is then compared with a preset threshold. If the fused probability exceeds the preset threshold, the object image is determined to contain the target object.

[0037] The present application realizes the reuse of the image feature information produced by the image segmentation model. By using this image feature information to calculate the classification probability, a decision factor as to whether the object picture contains the target object is added, which can provide reference information for the recognition result of the image recognition model. For the image recognition model, since it only needs to perform image recognition on the object segmentation map, the background image in the object picture no longer interferes with the recognition of the object, so it can also identify the target object from the object segmentation map in a more efficient and accurate manner and obtain its corresponding recognition probability.

[0038] Finally, based on the fusion probability calculated by fusing the average probability calculated by the classification probability and the recognition probability, with the help of the empirical threshold, a comprehensive judgment is made on whether the target item is contained in the item image. This judgment result is more accurate and reliable. Therefore, the judgment result made by this application on whether the target item is contained in the item image has a higher confidence level and can be highly trusted. It is suitable for application scenarios such as e-commerce platforms to detect whether there is a target image corresponding to the target item in the product image. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0040] Figure 1 This is a flow chart of a typical embodiment of the target object identification method of the present application;

[0041] Figure 2 A schematic diagram of a logical framework of a network for implementing the target object identification method of the present application;

[0042] Figure 3 The improved U-based 2 The structural principle diagram of the image segmentation model of net;

[0043] Figure 4 Schematic diagram of the working process of the image segmentation model in the embodiment of the present application;

[0044] Figure 5 This is a flow chart of determining whether an object image contains a target object based on two probabilities in an embodiment of the present application;

[0045] Figure 6 This is a principle block diagram of the target object identification device of the present application;

[0046] Figure 7 This is a schematic diagram of the structure of a computer device used in this application. DETAILED DESCRIPTION

[0047] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0048] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0049] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0050] It will be understood by those skilled in the art that the terms "client," "terminal," and "terminal device" as used herein include both devices that are wireless signal receivers, i.e., devices that only have wireless signal receivers without transmission capabilities, and devices that have receiving and transmitting hardware capable of two-way communication over a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers and tablet computers, which have single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service), which may combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant), which may include a radio frequency receiver, a pager, Internet / Intranet access, a web browser, a notepad, a calendar, and / or a GPS (Global Positioning System) receiver; and conventional laptop and / or palmtop computers or other devices, which have and / or include a radio frequency receiver. As used herein, the terms "client," "terminal," or "terminal device" may be portable, transportable, or installed in a vehicle (air, sea, and / or land), or may be adapted and / or configured to operate locally and / or in a distributed manner at any other location on Earth and / or in space. As used herein, the terms "client," "terminal," or "terminal device" may also refer to a communication terminal, an Internet terminal, or a music / video playback terminal, such as a PDA, an MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or may include a smart TV, a set-top box, or other device.

[0051] The hardware referred to by names such as "server", "client", and "service node" in this application is essentially an electronic device with capabilities equivalent to those of a personal computer. It is a hardware device that has the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. Computer programs are stored in its memory, and the central processing unit loads the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0052] It should be noted that the concept of "server" referred to in this application can also be extended to server clusters. Based on the network deployment principles understood by those skilled in the art, the servers described should be logically divided. In physical space, these servers can be independent of each other but callable through interfaces, or integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method of this application.

[0053] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for implementation.

[0054] Unless expressly specified otherwise, the neural network models referenced or may be referenced in this application may be deployed on a remote server and remotely called on the client, or may be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence may be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0055] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as they are suitable for being called by the technical solution of this application.

[0056] Those skilled in the art should be aware that although the various methods of this application are described based on the same concept and thus exhibit commonality, unless otherwise specified, these methods can be independently executed. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept. Therefore, concepts with the same expression, as well as concepts that are appropriately transformed for convenience despite different expression, should be understood as equivalent.

[0057] Unless expressly stated to be mutually exclusive, the various embodiments disclosed in this application may be cross-combined with the relevant technical features of the various embodiments to flexibly construct new embodiments, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or resolve certain deficiencies in the prior art. Those skilled in the art should be aware of such flexibility.

[0058] A target object identification method of the present application can be programmed as a computer program product and deployed to run on a client or server for implementation. The method can be executed by accessing an interface opened after the computer program product is running and conducting human-computer interaction with the process of the computer program product through a graphical user interface.

[0059] See also Figure 1 In a typical embodiment, the target object identification method of the present application includes the following steps:

[0060] Step S1100: Obtain an image of an item to be identified to determine whether it contains the target item:

[0061] The item picture generally refers to a picture containing an item image and used to show the appearance and shape of the item. In an exemplary application scenario used to assist in the explanation of this application, the item picture can be a product picture on an e-commerce platform. The items displayed in the product picture are generally sellable goods. The implementation of this application is to identify non-sellable goods from the product picture, such as knives, swords, etc. Therefore, it is necessary to identify the corresponding target item in the product picture, that is, its target image.

[0062] In this application scenario, the target image is usually determined by an item customized by the e-commerce platform, that is, a non-saleable commodity. Those skilled in the art will understand that by pre-training the image recognition model, it can learn the ability to recognize the target image from the item picture. Therefore, a large number of product pictures of various appearances and shapes containing the non-saleable goods can be used to train the image recognition model, so that such an image recognition model can learn the ability to recognize non-saleable goods from the product pictures. In other application scenarios, image recognition models for recognizing target images from item pictures can also be prepared based on this principle. However, if Figure 2 As shown, in this application, the image segmentation model will be used to extract features from the object image before the image recognition model recognizes the object image. This will be described in detail below and will not be discussed here for the time being.

[0063] Since image recognition models are a relatively mature technology, there are a variety of image recognition models available in the existing technology that can be used as reference. They are generally based on CNN convolutional neural networks, including but not limited to ViT, Resnet, HTC, etc. Different image recognition models based on neural networks may have different convolutional structures, resulting in different recognition methods. However, in terms of function and purpose, as long as the recognition model has been pre-trained to a convergence state for the target object and put into production, it can be used to identify the target object from the object image. Of course, the recognition capabilities of various models vary depending on the model structure and training samples, resulting in different recognition accuracy rates, which is reflected in the different confidence levels of the recognition results of each recognition model.

[0064] In the application scenario of the e-commerce platform, if it is necessary to obtain item images, one implementation method is to receive input from e-commerce platform users, especially input from warehouse management users of merchant instances when configuring product information, and use the product images in the product information as the item images. Of course, it is also possible to identify images posted by consumers in the comment area and use the images as item images; in another way, the server of the e-commerce platform can batch process the product images in the e-commerce platform database in the background, and use these product images as the item images to identify the target image.

[0065] Step S1200: Call a pre-trained image segmentation model to perform encoding and decoding on the object image at multiple scales to obtain multiple image feature information capturing the contour features of the object, and extract an object segmentation map from the object image based on the multiple image feature information:

[0066] See also Figure 3 This application adopts the U 2 net basic network architecture, which is trained to convergence and used in this application to pre-process the image of the object to be identified in the image recognition model.

[0067] U 2 The net model is based on the residual convolution principle and has an encoding path and a decoding path. The encoding path and the decoding path are respectively provided with the same number of multi-stage decoders. As for the number of stages of encoders and decoders, those skilled in the art can flexibly determine it based on prior knowledge such as experiments and experience. Figure 3 In the example, six levels of encoders and decoders are given. The encoder and decoder at the bottom of the figure are given as a single block diagram. This is mainly because they directly transform the intermediate feature information obtained by the previous level of encoding into image feature information after 1*1 convolution kernel transformation. Therefore, they are generally illustrated as a single block diagram.

[0068] The encoding path, in this embodiment, Figure 3 The six encoders (En_1 to En_5 and En_De) in the side branch path convert the item image into the corresponding specification original image, that is, the image adapted to U 2 The net crops the object image to the specified size based on the input image's specifications, and then encodes the original image of this size step by step. The first-level encoder at the top level extracts intermediate feature information corresponding to the first scale from the original image of this size, and then passes it to the encoder at the next level to extract intermediate feature information corresponding to the second scale. And so on, the scale of the original image of this size is gradually reduced to extract the corresponding intermediate feature information. Therefore, after the encoder encodes the original image of this size step by step, the six intermediate feature information corresponding to the original image of this size can be obtained.

[0069] It can be understood that the intermediate feature information of each scale is a representation obtained after deep semantic understanding of the standard image at the corresponding scale, and is information extracted from the contour features of the object in the object image. 2 This capability of the net model is well known to those skilled in the art. As long as it is trained to convergence with a sufficient number of training samples, its encoding path can have the deep semantic understanding ability to capture the contour features of objects in the object image.

[0070] The decoding path, in this embodiment, Figure 3 Among the six decoders (De_1 to De_5 and En_De) in the right branch path, starting from the bottom layer, the first decoder En_De performs a 1*1 convolution transformation on the intermediate feature information output by the encoder En_5 to decode and obtain the image feature information of the corresponding scale. In addition, each other higher-level decoder is based on the image feature information produced by the next level, and refers to the image feature information produced by the encoder at the same level to restore the image feature information of the corresponding scale at its level. And so on, to the last decoder at the top layer, image feature information of the same scale as the original image of the specification can be obtained.

[0071] It can also be understood that the intermediate feature information at each scale provides contextual information with its intermediate feature information at the same level, filling in the image feature information at the lower scale, thereby obtaining larger-scale image feature information output (Mask1 to Mask6). This image feature information also includes a representation of the outline features of the object within the object image captured from the object image. Unlike the intermediate feature information, this image feature information is decoded and restored, and processed into a mask representing the outline features, which is essentially mask image data. Therefore, it can be understood that after decoding the decoding path step by step, mask image data of six different scales corresponding to the number of levels will be obtained.

[0072] So far, according to the disclosure of the structure and principle of the image segmentation model herein, it can be understood that the image of the object of the present application is encoded and decoded step by step by the image segmentation model to obtain multiple image feature information. 2 net model principle, these image feature information can be further integrated to obtain mask image data (Mask7).

[0073] In the U2net model, a fully connected layer is used to fully connect the multiple image feature information obtained from its decoding path, processing it into a single mask image data (Mask7). This mask image data is essentially a binary grayscale image. Figuratively speaking, when the object image does not contain any objects, the grayscale image is a pure white image; when the object image does contain an object, the grayscale image is a black area defined by the object's outline in the foreground and a pure white background. Therefore, it is not difficult to understand that after the full connection, a mask image commonly known as "cutout" is obtained, which can be used for image segmentation.

[0074] Based on this, the U2net model uses the mask image data (Mask7) and, based on the contour information represented by the mask image data, segments the image defined by the corresponding contour from the original standard image to form an object segmentation map. This object segmentation map removes the background image information from the original object image, leaving only the object image information, which is commonly known as "cutting out" the image.

[0075] It is not difficult to understand that this object segmentation map can be provided to the image recognition model in this application for further recognition, so as to reflect the role of the image segmentation model in preprocessing the object image.

[0076] Step S1300: Classify each of the image feature information in the image segmentation model to obtain the classification probability that the image feature information contains the target object, and calculate the average probability of the multiple classification probabilities obtained by classifying all the image feature information:

[0077] Please continue reading Figure 3 , the U used in the image segmentation model in this application 2 The basic structure of the net model is improved, and a classifier is connected to each decoder (De_1 to De_5 and En_De). This classifier can be a multi-classifier, and preferably, a binary classifier can be directly used. Each classifier is used to perform classification processing based on the image feature information produced by its corresponding decoder. The classifier also participates in the training phase of the image segmentation model, so that it can judge whether the corresponding image feature information contains the target object based on the image feature information of each corresponding scale, and produce the corresponding classification probability, especially the classification probability corresponding to the target object. Accordingly, the image feature information of each scale produced by the image segmentation model will be classified and judged by its corresponding classifier, and finally the classification probability of the image feature information indicating the inclusion of the target object will be produced.

[0078] The classification probabilities (Class 1 to Class 6) of each image feature information correspond to the appearance of the object image at different scales. To make the classification probabilities more valuable and effective, all classification probabilities obtained based on the image feature information can be combined, for example, by taking their mean to obtain a corresponding average probability for later use. In an alternative embodiment, considering that larger scales are closer to the original image, the average probability can be obtained by weighting multiple classification probabilities. Specifically, each classification probability can be weighted from largest to smallest according to its corresponding scale, and the weighted results are summed and averaged.

[0079] It is not difficult to understand that here, by modifying the image segmentation model and connecting it to multiple classifiers, the function of the image segmentation model is expanded, the reuse of each image feature information produced by it is realized, and its image feature information is used for classification judgment, which provides richer decision reference information for judging whether the target object exists in the object picture. These decision reference information are obtained by classifying the image feature information of different scale features obtained based on deep semantic understanding and calculating the classification probability. Its importance to subsequent comprehensive classification judgment is self-evident.

[0080] Step S1400: Call a pre-trained image recognition model to perform image recognition on the object segmentation map to obtain the recognition probability that the object segmentation map contains the target object:

[0081] As mentioned above, after “cutting out”, the object segmentation map can be obtained, and the object segmentation map can be provided to the image recognition model for further target object recognition.

[0082] As mentioned above, the image recognition model can be selected from a variety of relatively excellent image recognition models in the existing technology, including but not limited to: HTC (Hybrid Task Cascade for Instance Segmentation), mask-rcnn, Resnet, ViT and its upgraded and evolved versions, etc., which are all mature image recognition models. As long as they are trained to convergence with sufficient corresponding training samples, they can be used as the image recognition model of this application. In this application, ViT is recommended because of its relatively excellent performance.

[0083] When the image recognition model is trained to convergence, the target object can be identified based on the object segmentation map. The deep semantic feature information of the object segmentation map is classified and mapped with the help of a classifier to obtain the probability of mapping it to the corresponding classification containing or not containing the target object, and thus outputting the recognition probability corresponding to the inclusion of the target object in the object segmentation map.

[0084] It is understood that in this application, the image recognition model does not directly recognize the original image of the item. Instead, it uses the item segmentation map generated by the image segmentation model as input for recognition. The item segmentation map is pre-processed by the image segmentation model, removing background information from the item image. Therefore, the item image in the item segmentation map represents the complete item information. The image recognition model uses this information to perform image recognition and classification, which is significantly more accurate and efficient.

[0085] Step S1500: The average probability and the recognition probability are combined to obtain a fusion probability for result judgment. When the fusion probability is greater than a preset threshold, it is determined that the object image contains the target object; otherwise, it does not contain the target object.

[0086] After the above processing process, this application obtains two probability values ​​for an object picture. One is the average probability calculated by classifying the image feature information produced by the image segmentation model, and the other is the recognition probability calculated by identifying and classifying the object segmentation map produced by the image recognition model. Both probability values ​​can indicate the probability that the object picture contains the target object, providing probability information corresponding to the two decision reference dimensions. Therefore, the two need to be fused to achieve the final judgment.

[0087] To this end, in this application, a variety of methods can be used to combine the average probability and recognition probability into one, including but not limited to adding the two, averaging, weighted sum, smooth sum, and any other methods. As long as the applied method can reflect the contribution of the two probability values ​​in the decision-making, it can be adopted. Based on this, a single probability value will be obtained, which can be called a fusion probability.

[0088] The fusion probability integrates the classification information corresponding to the different scale features of the object picture and the classification information corresponding to the object image therein. Therefore, a comprehensive decision can be made based on this fusion probability.

[0089] To achieve a comprehensive judgment, a threshold can be preset based on prior knowledge such as experimentation and / or experience, and the fusion probability can be compared with it. When the fusion probability exceeds the threshold, the item image is determined to contain the target item; otherwise, it is determined that the item image does not contain the target item. Thus, the preset threshold serves to adjust the confidence level of the fusion probability, helping to enhance the credibility of the judgment result.

[0090] This application first uses an image segmentation model to segment an image of an object requiring target object recognition, obtaining image feature information at multiple scales representing the outline features of the object in the image. Based on this information, the application then proceeds in two steps: one in which the image feature information corresponding to each scale is individually classified, obtaining the classification probability that each image feature information indicates that the image contains the target object, and calculating the average probability of the multiple classification probabilities; the other in which the image feature information corresponding to each scale is fused to form mask image data representing the outline features of the object in the image. Based on this mask image data, an object segmentation map is segmented from the image. The image recognition model is then used to identify this object segmentation map as the target object, obtaining the recognition probability that the object segmentation map contains the target object. Based on the obtained average probability and recognition probability, the two probabilities are further fused into a single fused probability, which is then compared with a preset threshold. If the fused probability exceeds the preset threshold, the object image is determined to contain the target object.

[0091] The present application realizes the reuse of the image feature information produced by the image segmentation model. By using this image feature information to calculate the classification probability, a decision factor as to whether the object picture contains the target object is added, which can provide reference information for the recognition result of the image recognition model. For the image recognition model, since it only needs to perform image recognition on the object segmentation map, the background image in the object picture no longer interferes with the recognition of the object, so it can also identify the target object from the object segmentation map in a more efficient and accurate manner and obtain its corresponding recognition probability.

[0092] Finally, based on the fusion probability calculated by fusing the average probability calculated by the classification probability and the recognition probability, with the help of the empirical threshold, a comprehensive judgment is made on whether the target item is contained in the item image. This judgment result is more accurate and reliable. Therefore, the judgment result made by this application on whether the target item is contained in the item image has a higher confidence level and can be highly trusted. It is suitable for application scenarios such as e-commerce platforms to detect whether there is a target image corresponding to the target item in the product image.

[0093] See also Figure 4 In a further embodiment, step S1200, calling a pre-trained image segmentation model to perform encoding and decoding on the object image at multiple scales to obtain multiple image feature information capturing contour features of the object, and extracting an object segmentation map from the object image based on the multiple image feature information, includes the following steps:

[0094] Step S1210: The pre-trained image segmentation model performs multi-level encoding to reduce the original size of the object image step by step, and generates intermediate feature information corresponding to each scale. The intermediate feature information is used to represent the outline features of the object in the object image:

[0095] Please review again Figure 3 In conjunction with the foregoing, in the encoding branch path of the image segmentation model, the original size of the object image is scaled down one by one, and at each scale, intermediate feature information is extracted using a corresponding encoder. The larger-scale intermediate feature information in the backward direction is downsampled to obtain smaller-scale intermediate feature information, which is then passed to the forward encoder. Simultaneously, this smaller-scale intermediate feature information is also passed to a decoder at the same level as the encoder as input to the decoder. The current encoder performs higher-level semantic feature extraction on the intermediate feature information produced by its backward encoder, and produces intermediate feature information of its own corresponding scale. This is pre-trained, so that each intermediate feature information extracts and represents the contour features of the object in the object image.

[0096] Step S1220: Multi-level decoding of the pre-trained image segmentation model is performed. Based on the image feature information generated by the smallest-scale intermediate feature information, the intermediate feature information generated by the encoding at the same level is used as reference information to decode higher-scale image feature information. The image feature information is used to represent the contour features of the object in the object image in the form of a mask.

[0097] In the decoding branch path of the image segmentation model, the intermediate feature information produced by the last-level encoder of the encoding branch path is transformed into image feature information through a 1*1 convolution kernel. The image feature information is divided into three output paths, one for full connection, one for providing its forward higher-level decoder for scale restoration, and the other for accessing a binary classifier for classification to obtain the classification probability when the corresponding image feature information contains the target object.

[0098] The other decoders at each level of the decoding branch path take two data inputs: one is the image feature information provided by its backward decoder, and the other is the intermediate feature information produced by the encoder at the same level. The latter provides a reference for the context information of the former, enabling the current decoder to restore image feature information of a higher scale than that of its backward decoder. Similarly, it is divided into three outputs, and so on, until the last decoder produces image feature information of the same size as the original image of the object.

[0099] Since the image feature information is restored based on the intermediate feature information, it can be understood that the image feature information also represents the contour features in the object image in the form of a mask.

[0100] Step S1230: The image segmentation model performs image segmentation on the standard original image of the item image based on the mask image data formed by fusing all image feature information, so as to extract the item segmentation image corresponding to the item in the item image.

[0101] In a specific embodiment, step S1230 can be implemented by the following specific steps:

[0102] Step S1231: All image feature information is fully connected by the fully connected layer in the image segmentation model to fuse and generate mask image data. The mask image data inherits the contour features of the object in the image feature information:

[0103] The image feature information from each decoder in the decoding branch path is input into the fully connected layer of the image segmentation model. This fully connected layer fully connects all image feature information to integrate contour features at different scales, thereby achieving the fusion of multiple image feature information, thereby obtaining the fused mask image data (mask7). It can be understood that the mask image data inherits the contour features of the object represented in the image feature information. Combined with the above description, it can be understood that the mask image data is essentially a mask image (mask7), which represents the image area occupied by the object in the object image by combining a black foreground with a white background, acting as an image mask and can be used for the so-called "cutout".

[0104] Step S1232: performing image extraction on the original image of the item according to the mask image data, obtaining an image within the outline corresponding to the outline features of the item to form the item segmentation map.

[0105] After obtaining the corresponding mask image data to form the mask required for image segmentation, the mask data can be used to extract the image of the area occupied by the object from the original standard image of the object. In other words, the image corresponding to the area covered by the black background of the mask is used as the object segmentation map. To achieve this, the following formula can be used:

[0106] Output Img =mask*Img Org +(1-mask)*255

[0107] Among them, Output Img is the object segmentation map, and mask refers to the mask image output after the image segmentation model is fully connected ( Figure 3 (mask7), Img Org This is the original specification image of the item.

[0108] Thus, the object segmentation map required by the image recognition model can be obtained. Of course, those skilled in the art will appreciate that, in some alternative embodiments, the image recognition model can also pre-process the object image, such as scaling it to a fixed size and performing image normalization, to make it suitable for the input requirements of the image recognition model. As long as the input conditions of the image recognition model are met, the image recognition model can determine the probability that the object in the image belongs to the target object. This scenario should also be considered an equivalent replacement for the relevant technical means of this application.

[0109] This embodiment further demonstrates the detailed process of the image segmentation model processing the object image, as well as various alternative embodiments thereof. It can be seen that the image segmentation model of the present application is suitable for various variants based on Unet, and is not limited to Unet. 2 net version, and can be extended to be compatible with other models with equivalent graph segmentation capabilities.

[0110] See also Figure 5 In a modified embodiment, a smoothing function is used to determine the fusion probability. Therefore, step S1500 further includes the following steps:

[0111] Step S1510: Obtain the average probability and recognition probability:

[0112] The average probability has been generated in the process disclosed above, and the recognition probability has been output by the image recognition model, so it can be directly called.

[0113] The average probability can be calculated using the following formula:

[0114] Output Cls =(cls_1+cls_2+……+cls_n) / n

[0115] Among them, Output Cls The average probability of the classification corresponding to the image feature information at each scale of the image segmentation model containing the contour features of the target feature is obtained. cls_n is the nth classification probability, where n corresponds to the number of image feature information. This method achieves simple averaging and high computational efficiency.

[0116] Step S1520: Fusing the average probability and the recognition probability to calculate a fusion probability, wherein the average probability and the recognition probability are associated with the same hyperparameter for smoothing:

[0117] The following formula is used to calculate the fusion probability:

[0118] Output Prob =W Vit *Output Vit +(1-W Vit )*Output Cls

[0119] Among them, Output Prob is the fusion probability, Output Vit Output is the recognition probability of the target item in the segmentation map representing the object output by the image recognition model. Cls is the average probability obtained by averaging the probabilities of each classification in the image segmentation model, W Vit It is a hyperparameter used to achieve smoothing. Those skilled in the art can flexibly adjust the value of the hyperparameter according to actual conditions or experimental data.

[0120] By using this formula to calculate the fusion probability, the organic unification of the average probability and the recognition probability is achieved, so that the classification information obtained based on the image segmentation model and the classification information obtained based on the image recognition model can refer to each other, and it can more accurately determine whether the object picture contains the target image corresponding to the target object.

[0121] Step S1530: compare the fusion probability with a preset threshold. When the fusion probability is greater than the preset threshold, determine that the object image contains the target object; otherwise, determine that the object image does not contain the target object. This can be understood in conjunction with the typical embodiments of the present application.

[0122] This embodiment further applies a smoothing function to fuse the average probability and the recognition probability, so that those skilled in the art can flexibly adjust the role of two types of data, image feature information of different scales and object segmentation map, in the process of identifying target objects in object images based on the principles of this application, thereby more flexibly utilizing the technical solution of this application to serve practical needs, making the technical solution of this application more universal in serving the needs of identifying target objects from object images.

[0123] See also Figure 6, a target object recognition device provided to meet one of the purposes of the present application is a functional embodiment of the target object recognition method of the present application, the device includes: an image acquisition module 1100, an image segmentation module 1200, a contour classification module 1300, an object classification module 1400 and a fusion judgment module 1500, wherein the image acquisition module 1100 is used to obtain an object image to be identified as to whether it contains a target object; the image segmentation module 1200 is used to call a pre-trained image segmentation model to perform encoding and decoding on the object image at multiple scales, obtain multiple image feature information that captures the contour features of the object, and extract the object classification from the object image based on the multiple image feature information. The contour classification module 1300 is used to classify each image feature information in the image segmentation model, obtain the classification probability that the image feature information contains the target item, and calculate the average probability of multiple classification probabilities obtained by classifying all image feature information; the item classification module 1400 is used to call the pre-trained image recognition model to perform image recognition on the item segmentation map, and obtain the recognition probability that the item segmentation map contains the target item; the fusion judgment module 1500 is used to fuse the average probability and the recognition probability to obtain a fusion probability for result judgment. When the fusion probability is greater than a preset threshold, it is determined that the item image contains the target item, otherwise it does not contain the target item.

[0124] In a further embodiment, the image segmentation module 1200 includes: an encoding path unit, which is configured to perform multi-level encoding of a pre-trained image segmentation model, gradually reduce the scale of the original specification image of the item image, and correspondingly generate intermediate feature information corresponding to each scale, wherein the intermediate feature information is used to represent the contour features of the item in the item image; a decoding path unit, which is configured to perform multi-level decoding of a pre-trained image segmentation model, based on the image feature information generated by the intermediate feature information of the smallest scale, and step by step using the intermediate feature information generated by the encoding of the same level as reference information to decode the image feature information of a higher scale, wherein the image feature information is used to represent the contour features of the item in the item image in the form of a mask; a fusion segmentation unit, which is used to perform image segmentation on the original specification image of the item image based on the mask image data fused by the image segmentation model according to all the image feature information, so as to extract the item segmentation map corresponding to the item in the item image.

[0125] In a specific embodiment, the fusion and segmentation unit includes: a fusion subunit, which is used to fully connect all image feature information by the fully connected layer in the image segmentation model to fuse and generate mask image data, and the mask image data inherits the contour features of the object in the image feature information; a segmentation subunit, which is used to extract the image of the specification original image of the object image according to the mask image data, and obtain the image within the contour corresponding to the contour features of the object to form the object segmentation map.

[0126] In a further embodiment, the fusion judgment module 1500 includes: a probability acquisition sub-block, used to obtain the average probability and the recognition probability; a mean calculation sub-block, used to fuse the average probability and the recognition probability to calculate the fusion probability, wherein the average probability and the recognition probability are associated with the same hyperparameter for smoothing; a result judgment sub-module, used to compare the fusion probability with a preset threshold, when the fusion probability is greater than the preset threshold, it is determined that the object image contains the target object, otherwise it is determined that the target object is not contained.

[0127] In a specific embodiment, the image feature information corresponding to multiple scales generated by the image segmentation model is output to a binary classifier corresponding to each scale for binary classification decision to obtain the classification probability.

[0128] In a preferred embodiment, the image segmentation model is U 2 net model, the basic network architecture of the image segmentation model is U 2 net model, the target item is a knife or a sword.

[0129] In order to solve the above technical problems, the embodiment of the present application also provides a computer device. Figure 7 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor may implement a target object identification method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor may execute the target object identification method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0130] In this embodiment, the processor is used to execute Figure 6 The memory stores the program code and various data required to execute the modules and submodules in the target object identification device. The network interface is used to transmit data between user terminals and servers. The memory in this embodiment stores the program code and data required to execute all modules and submodules in the target object identification device of this application. The server can call the server's program code and data to execute the functions of all submodules.

[0131] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the target object identification method of any embodiment of the present application.

[0132] The present application also provides a computer program product, comprising a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.

[0133] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of the method. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0134] In summary, the present application uses an image segmentation model to extract an item segmentation map from an item image to be identified as to whether it contains a target item, uses an image recognition model to identify the item segmentation map to obtain the recognition probability that the item image contains the target item, and uses the image feature information of multiple scales extracted by the image segmentation model to classify to obtain the average probability that the item image contains the target item, and then compares the fusion probability of the average probability and the recognition probability with a preset threshold to obtain the final judgment result of whether the item image contains the target item corresponding to the target image. Since the basis for the judgment of whether the item image contains the target item integrates multiple sources of feature information, the judgment of whether the item image contains the target item is more accurate, and a more reliable target item recognition judgment result can be obtained, thereby reducing the cost of manual labeling. Applying the technical solution of the present application to identify product images in e-commerce platform scenarios can improve the recognition efficiency of e-commerce platforms in processing massive product images.

[0135] Those skilled in the art will appreciate that the steps, measures, and schemes in the various operations, methods, and processes discussed in this application may be interchanged, modified, combined, or deleted. Furthermore, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be interchanged, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be interchanged, modified, rearranged, decomposed, combined, or deleted.

[0136] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A target object identification method, characterized in that: The steps include: Obtaining an item picture to be identified as containing a target item, wherein the item picture is a picture containing an image of the item and used to show the appearance and shape of the item; Calling a pre-trained image segmentation model to perform encoding and decoding on the object image at multiple scales to obtain multiple image feature information capturing contour features of the object, and extracting an object segmentation map from the object image based on the multiple image feature information; Classifying each of the image feature information in the image segmentation model to obtain a classification probability that the image feature information includes the target object, and calculating an average probability of multiple classification probabilities obtained by classifying all the image feature information; Calling a pre-trained image recognition model to perform image recognition on the object segmentation map to obtain a recognition probability that the object segmentation map contains the target object; The average probability and the recognition probability are fused to obtain a fusion probability for result judgment. When the fusion probability is greater than a preset threshold, it is determined that the object image contains the target object, otherwise it does not contain the target object.

2. The target object identification method according to claim 1, characterized in that: Calling a pre-trained image segmentation model to perform encoding and decoding on the object image at multiple scales to obtain multiple image feature information capturing the contour features of the object, and extracting an object segmentation map from the object image based on the multiple image feature information, including the following steps: The pre-trained image segmentation model performs multi-level encoding to reduce the original size of the object image step by step and generates intermediate feature information corresponding to each scale. The intermediate feature information is used to represent the outline features of the object in the object image; Multi-level decoding of the pre-trained image segmentation model, based on the image feature information produced by the smallest-scale intermediate feature information, uses the intermediate feature information produced by the encoding of the same level as reference information to decode the corresponding higher-scale image feature information. The image feature information is used to represent the contour features of the object in the object image in the form of a mask; The image segmentation model performs image segmentation on the standard original image of the object image based on the mask image data formed by fusing all image feature information, so as to extract the object segmentation map corresponding to the object in the object image.

3. The target object identification method according to claim 2, characterized in that: The image segmentation model performs image segmentation on the standard original image of the item image based on the mask image data formed by fusing all image feature information, so as to extract the item segmentation map corresponding to the item in the item image, including the following steps: All image feature information is fully connected by a fully connected layer in the image segmentation model to fuse and generate mask image data, wherein the mask image data inherits the contour features of the object in the image feature information; The image of the specification original image of the object picture is extracted according to the mask image data, and the image within the outline corresponding to the outline feature of the object is obtained to form the object segmentation map.

4. The target object recognition method according to any one of claims 1 to 3, characterized in that: The average probability and the recognition probability are combined to obtain a fusion probability for result judgment, including the following steps: Obtaining the average probability and the recognition probability; fusing the average probability and the recognition probability to calculate a fused probability, wherein the average probability and the recognition probability are associated with the same hyperparameter for smoothing; The fusion probability is compared with a preset threshold value. When the fusion probability is greater than the preset threshold value, it is determined that the object image contains the target object; otherwise, it is determined that the object image does not contain the target object.

5. The target object recognition method according to any one of claims 1 to 3, characterized in that: The image feature information corresponding to the multiple scales generated by the image segmentation model is output to the binary classifier corresponding to each scale for binary classification decision to obtain the classification probability.

6. The target object identification method according to claim 5, characterized in that: The basic network architecture of the image segmentation model is U 2 net model, the target item is a knife or a sword.

7. A target object identification device, characterized in that: include: A picture acquisition module is used to acquire a picture of an object to be identified as containing a target object, wherein the picture includes an image of the object and is used to show the appearance and shape of the object; An image segmentation module is configured to call a pre-trained image segmentation model to perform encoding and decoding on the object image at multiple scales, obtain multiple image feature information capturing the contour features of the object, and extract an object segmentation map from the object image based on the multiple image feature information; a contour classification module, configured to classify each of the image feature information in the image segmentation model, obtain a classification probability that the image feature information contains the target object, and calculate an average probability of multiple classification probabilities obtained by classifying all the image feature information; An object classification module is configured to call a pre-trained image recognition model to perform image recognition on the object segmentation map and obtain a recognition probability that the object segmentation map contains the target object; The fusion judgment module is used to fuse the average probability and the recognition probability to obtain a fusion probability for result judgment. When the fusion probability is greater than a preset threshold, it is determined that the object image contains the target object, otherwise it does not contain the target object.

8. A computer device comprising a central processing unit and a memory, characterized in that: The central processing unit is configured to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that It stores a computer program implemented according to the method described in any one of claims 1 to 6 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

Images