Target image screening method, device, electronic device and storage medium

By extracting and filtering the feature of the image classification model and filtering images that only contain the target objects are selected, solving the problem of insufficient accuracy of the image classification model, and achieving efficient image filtering and enhanced model applicability.

CN114612696BActive Publication Date: 2024-12-03BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202011408930.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-04
Publication Date
2024-12-03
Estimated Expiration
2040-12-04

AI Technical Summary

Technical Problem

Existing image classification models may classify different objects into the same category when classifying images, and the classification standards change over time, resulting in insufficient model accuracy. How to obtain images containing target objects has become an urgent problem.

Method used

By calling the image classification model, the reference image and alternative images are processed, the reference classification features and alternative classification features are obtained, and the target image is filtered, so that the filtered image only contains target objects that affect the classification results, realizing automatic image filtering and improve screening efficiency.

Benefits of technology

Ensure that only the target objects are included in the filtered images, which improves the accuracy and efficiency of image filtering, and is suitable for single-class and multi-classification models, enhancing the applicability of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114612696B_ABST
    Figure CN114612696B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a method, device, electronic device and storage medium for screening target images, and relates to the field of computer technology. The method includes: calling a first image classification model to process a reference image, obtaining a reference classification feature generated in the process of the first image classification model processing the reference image, the reference image includes a target object for determining the classification result of the reference image, and does not include other objects that affect the classification result of the reference image; obtaining alternative classification features generated in the process of the first image classification model processing multiple candidate images; and screening out a target image from multiple candidate images, the target image being an alternative image whose alternative classification features match the reference classification features. Since the classification features include features of the target object, images including the target object and other objects that do not affect the classification results can be screened out based on the classification features, thereby improving the efficiency of image screening.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a target image screening method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of computer technology and network technology, more and more images are circulated on the Internet. In order to ensure the security of the Internet environment, it is usually necessary to determine the category of the image. The commonly used method is to call the image classification model, process the image, and obtain the classification result of the image.

[0003] However, the image classification model may classify images that include different objects into one category. For example, pictures that include unicycles, pictures that include bicycles, and pictures that include cars are all classified as vehicle pictures. However, people's classification standards for vehicles may change over time. For example, a unicycle is no longer a vehicle, which leads to insufficient accuracy of the image classification model. The accuracy of the image classification model is closely related to the object in the sample image of the model used to determine the classification result. Therefore, how to obtain images containing the object has become a problem that needs to be solved urgently. Summary of the invention

[0004] The present disclosure provides a target image screening method, device, electronic device and storage medium to improve the efficiency of image screening. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a target image screening method is provided, comprising:

[0006] Calling a first image classification model to process a reference image, obtaining a reference classification feature generated in the process of the first image classification model processing the reference image, wherein the reference image includes a target object used to determine a classification result of the reference image, and does not include other objects that affect the classification result of the reference image, and the reference classification feature is a feature for the first image classification model to determine a classification result of the reference image;

[0007] Calling the first image classification model to process a plurality of candidate images, and obtaining candidate classification features generated in the process of the first image classification model processing the plurality of candidate images, wherein the candidate classification features are features used by the first image classification model to determine classification results of the candidate images;

[0008] A target image is screened out from the multiple candidate images, where the target image is a candidate image whose candidate classification feature matches the reference classification feature.

[0009] The embodiment of the present disclosure provides a method for screening target images. Considering that an image includes one or more objects, some objects will affect the classification result of the image, and some objects will not affect the classification result of the image, and the classification feature of the image is a feature for the image classification model to determine the classification result, then the feature of the object that affects the classification result in the feature is retained, and the features of other objects that do not affect the classification result are discarded. Since only the target object is used to determine the classification result in the reference image, the reference classification feature of the reference image can be considered as the feature presented by the target object in the image classification model, so that it can be ensured that the selected candidate classification features that match the reference classification feature are all the features presented by the target object in the image classification model, then only the target object affects the classification result in the screened target image, thereby realizing automatic screening of images and improving the efficiency of screening images. Moreover, the screened target image may also include other objects other than the target object that do not affect the classification result, or may not include other objects other than the target object, so that the screened target image is more diversified.

[0010] In some embodiments, the first image classification model includes a feature extraction layer, a feature fusion layer, and a classification layer, and calling the first image classification model to process the reference image includes:

[0011] Calling the feature extraction layer to process the reference image to obtain regional features corresponding to multiple image regions in the reference image;

[0012] Calling the feature fusion layer to process the regional features corresponding to the multiple image regions to obtain the image features of the reference image;

[0013] The classification layer is called to process the image features to obtain reference classification features of the reference image, and then the reference classification features are processed to obtain a classification result of the reference image.

[0014] It can be seen from the process of processing the image with the above-mentioned first image classification model that the reference classification feature is different from the image feature. The reference classification feature is obtained by processing the image feature through the classification layer. Since the classification only focuses on the objects that affect the classification results, but not the objects that are not related to the classification results, the features of the objects that affect the classification results in the classification layer are retained, while the features of other objects that do not affect the classification results are discarded or suppressed. Therefore, by obtaining the classification layer to process the image features, it is achieved that only the features in the image that affect the classification results are obtained.

[0015] In some embodiments, the classification result indicates whether the reference image belongs to a target category, and calling the classification layer to process the image features to obtain reference classification features of the reference image includes:

[0016] The classification layer is called to perform a dot multiplication of the image feature and the weight matrix corresponding to the target category to obtain the reference classification feature.

[0017] The image features of an image will include features of multiple dimensions of the image, and features of some dimensions will affect the classification results, while features of some dimensions will not affect the classification results. The weight matrix includes weights corresponding to different dimensions, and the weights can represent the degree of influence of the features of the dimensions on the classification results. Therefore, by performing a dot multiplication of the image features with the weight matrix, the features that affect the classification results can be accurately retained, and the features that do not affect the classification results can be discarded or suppressed, thereby improving the accuracy of the reference classification features. Moreover, the embodiment of the present application obtains the dot multiplication result of the weight matrix corresponding to the image features and the target category, and does not involve the weight matrices corresponding to other categories. Therefore, this method is not only applicable to single-classification models, but can also be applied to multi-classification models, thereby improving the applicability of this solution.

[0018] In some embodiments, obtaining the reference classification features generated by the first image classification model in the process of processing the reference image includes:

[0019] Acquire a plurality of regional features generated in the process of processing the reference image by the first image classification model, wherein the regional features are features corresponding to image regions in the reference image;

[0020] Determining, according to the multiple region features, a weight of each image region in the reference image to the classification result;

[0021] If the image region with the largest weight is the region where the target object is located, a reference classification feature generated in the process of processing the reference image by the first image classification model is obtained.

[0022] When an image region includes an object that affects the classification result, the image region will affect the classification result and the impact is greater. When an image region does not include an object that affects the classification result, the image region will not affect the classification result or will have a smaller impact. Therefore, determining whether the image region that has the greatest impact on the classification result is the region where the target object is located is to determine whether the image includes the target object that affects the classification result but does not include other objects that affect the classification result. The reference classification features of the reference image will be obtained only when the reference image meets the requirement of including the target object that affects the classification result but not including other objects that affect the classification result, thereby improving the accuracy of the reference classification features.

[0023] In some embodiments, the classification result of the reference image is a probability that the reference image belongs to a target category; and determining, based on the multiple region features, a weight of each image region in the reference image for the classification result includes:

[0024] According to the regional features of each image region, the classification probability corresponding to each image region is determined, the classification probability indicates the probability that the object in the image region belongs to the target category, and the image region with the largest weight is the region with the highest similarity between the classification probability and the classification result.

[0025] Since the classification probability of the reference image is jointly determined by the classification probabilities of multiple image regions, comparing the classification probability of an image region with the classification probability of the reference image can accurately determine the image region that has the greatest impact on the classification probability of the reference image.

[0026] In some embodiments, obtaining the reference classification features generated by the first image classification model in the process of processing the reference image includes:

[0027] Obtaining a classification probability corresponding to each image region generated in a process in which the first image classification model processes the reference image;

[0028] Creating a region identification image according to a mapping relationship between classification probabilities and indicator marks and the classification probability corresponding to each image region, wherein the region identification image includes the indicator mark corresponding to each image region;

[0029] Displaying the area identification image;

[0030] In response to a selection operation on the region identification image, a reference classification feature generated in a process in which the first image classification model processes the reference image is obtained.

[0031] By displaying the region identification image to the user, the user confirms whether the reference image includes the target object but does not include other objects that affect the classification result, and after the user confirms, the reference classification feature is obtained, thereby improving the accuracy of the reference classification feature.

[0032] In some embodiments, after displaying the area identification image, the method further includes:

[0033] In response to a discarding operation on the region identification image, the reference image is discarded.

[0034] For a reference image that does not meet the requirements, the user may also discard the reference image and no longer obtain the reference classification features of the reference image, thereby ensuring the accuracy of the obtained reference classification features.

[0035] In some embodiments, there are multiple reference images, and obtaining reference classification features generated when the first image classification model processes the reference images includes:

[0036] Obtain reference classification features for each reference image;

[0037] The obtained multiple reference classification features are fused to obtain fused reference classification features.

[0038] By referring to multiple reference images, the accuracy of the acquired reference classification features is improved, thereby improving the accuracy of image screening.

[0039] In some embodiments, obtaining the reference classification features generated by the first image classification model in the process of processing the reference image includes:

[0040] Acquire a reference classification feature of the reference image, wherein the reference classification feature includes a plurality of feature values;

[0041] The feature values ​​of the reference classification features that are less than the reference threshold are changed to 0 to obtain the changed reference classification features.

[0042] Since some features in the reference image that do not affect the classification results will be suppressed during the classification process, the obtained reference classification features will include the suppressed features. Since the feature values ​​of the suppressed features are small, the present application discards the suppressed features by changing the feature values ​​that are less than the reference threshold to 0, thereby making the reference classification features more accurate.

[0043] In some embodiments, the step of selecting a target image from the plurality of candidate images includes:

[0044] If the distance between any candidate classification feature and the reference classification feature is less than the reference distance, the candidate image corresponding to the candidate classification feature is determined as the target image.

[0045] Based on the distance between the classification features, the candidate images are screened, which can ensure that the classification features of the screened target image have a high similarity with the reference classification features of the reference image.

[0046] In some embodiments, the plurality of candidate images are sample images used to train the first image classification model. After selecting the target image from the plurality of candidate images, the method further includes:

[0047] The screened target image is discarded from the multiple candidate images, and the first image classification model is trained based on the remaining candidate images so that the classification result determined by the trained first image classification model is no longer affected by the target object.

[0048] Since the model is prone to fit the most recent images and ignore previously trained images during the training process, the present application cleans out the images whose classification results are determined by the target object from the sample images, and then uses the remaining sample images to train the image classification model, so that the classification results determined by the trained image classification model are no longer affected by the target object, and there is no need to re-create the sample set to train a new image classification model.

[0049] In some embodiments, the plurality of candidate images are sample images in a sample set used to train a first image classification model, and the sample set also includes a first labeled classification result of each candidate image. After screening out the target image from the plurality of candidate images, the method further includes:

[0050] reallocating a second labeling classification result to the screened target image to obtain an updated sample set, wherein the second labeling classification result of the target image is opposite to the first labeling classification result;

[0051] The first image classification model is trained according to the updated sample set, so that a classification result determined by the trained first image classification model according to the target object is opposite to a classification result determined by the first image classification model before training.

[0052] The image classification model is trained using a sample set, and the image classification model can learn the classification rules implicit in the sample set. The present application updates the sample set by changing the classification results of the target object, that is, updates the classification rules implicit in the sample set. When the image classification model is trained using the updated sample set, the image classification model can learn the new classification rules, thereby realizing the adjustment of the classification rules of the image classification model.

[0053] In some embodiments, after selecting the target image from the plurality of candidate images, the method further includes:

[0054] The second image classification model is trained according to the determined target image, and the second image classification model is used to process according to the target object in any image.

[0055] The target image screening method provided in this application can automatically screen out the sample set required for training a new model without manual screening, thereby improving the efficiency of sample screening and thus increasing the training speed of the model.

[0056] In some embodiments, there are multiple reference images, each of which includes a different target object. After determining the screened candidate image as the target image, the method further includes:

[0057] According to the target image corresponding to each reference image, multiple second image classification models are trained respectively, and different second image classification models are used to process according to different target objects.

[0058] The embodiment of the present application can also train multiple image classification models based on the screened target images containing different target objects, so that different image classification models can classify according to different target objects, which is conducive to the image classification model to refine the classification rules for classifying the target objects.

[0059] In some embodiments, after training the plurality of second image classification models respectively according to the target image corresponding to each reference image, the method further comprises:

[0060] Get at least one target object;

[0061] Selecting a second image classification model that matches the at least one target object from the plurality of second image classification models;

[0062] At least one selected second image classification model is called to process any image to obtain at least one classification result of the image.

[0063] In different scenarios, the classification results of the same object may be different. Therefore, the present application trains an image classification model that determines the classification results based on only one target object, and then selects the corresponding image classification model for classification according to the scenario requirements. There is no need to train a corresponding image classification model for each classification scenario, which improves the applicability of the image classification model and makes the use of the image classification model more flexible.

[0064] According to a second aspect of an embodiment of the present disclosure, there is provided a target image screening device, comprising:

[0065] A first acquisition unit is configured to execute calling a first image classification model to process a reference image, and acquire reference classification features generated in the process of the first image classification model processing the reference image, wherein the reference image includes a target object used to determine a classification result of the reference image, and does not include other objects that affect the classification result of the reference image, and the reference classification features are features for the first image classification model to determine a classification result of the reference image;

[0066] The first acquisition unit is configured to execute calling the first image classification model to process a plurality of candidate images, and acquire candidate classification features generated in the process of the first image classification model processing the plurality of candidate images, wherein the candidate classification features are features used by the first image classification model to determine classification results of the candidate images;

[0067] The screening unit is configured to screen out a target image from the multiple candidate images, where the target image is a candidate image whose candidate classification feature matches the reference classification feature.

[0068] In some embodiments, the image classification model includes a feature extraction layer, a feature fusion layer and a classification layer, and the first acquisition unit includes:

[0069] A feature extraction subunit is configured to execute calling the feature extraction layer, process the reference image, and obtain regional features corresponding to multiple image regions in the reference image;

[0070] A feature fusion subunit is configured to execute calling the feature fusion layer, process the regional features corresponding to the multiple image regions, and obtain the image features of the reference image;

[0071] The classification subunit is configured to execute the call to the classification layer, process the image features to obtain the reference classification features of the reference image, and then process the reference classification features to obtain the classification result of the reference image.

[0072] In some embodiments, the classification result indicates whether the reference image belongs to a target category, and the classification subunit is configured to execute a call to the classification layer, perform a dot multiplication of the image feature with a weight matrix corresponding to the target category, and obtain the reference classification feature.

[0073] In some embodiments, the first acquisition unit includes:

[0074] An acquisition subunit is configured to acquire a plurality of regional features generated in a process in which the first image classification model processes the reference image, wherein the regional features are features corresponding to an image region in the reference image;

[0075] a determination subunit configured to determine a weight of each image region in the reference image to the classification result according to the multiple region features;

[0076] The acquisition subunit is configured to execute, if the image area with the largest weight is the area where the target object is located, obtaining the reference classification features generated in the process of the first image classification model processing the reference image.

[0077] In some embodiments, the classification result of the reference image is the probability that the reference image belongs to the target category; the determination subunit is configured to determine the classification probability corresponding to each image area based on the regional features of each image area, the classification probability indicating the probability that the object in the image area belongs to the target category, and the image area with the largest weight is the area with the highest similarity between the classification probability and the classification result.

[0078] In some embodiments, the acquisition unit further includes:

[0079] An acquisition subunit is configured to execute acquisition of a classification probability corresponding to each image region generated in a process in which the first image classification model processes the reference image;

[0080] A creating subunit is configured to create a region identification image according to a mapping relationship between the classification probability and the indicator and the classification probability corresponding to each image region, wherein the region identification image includes the indicator corresponding to each image region;

[0081] A display subunit, configured to display the area identification image;

[0082] The acquisition subunit is configured to execute, in response to a selection operation on the region identification image, to acquire reference classification features generated in a process in which the first image classification model processes the reference image.

[0083] In some embodiments, the apparatus further comprises:

[0084] The discarding unit is configured to discard the reference image in response to a discarding operation on the region identification image.

[0085] In some embodiments, there are multiple reference images, and the first acquisition unit is configured to acquire reference classification features of each reference image; and fuse the acquired multiple reference classification features to obtain fused reference classification features.

[0086] In some embodiments, the first acquisition unit is configured to execute acquisition of reference classification features of the reference image, where the reference classification features include multiple feature values; and change feature values ​​in the reference classification features that are less than a reference threshold to 0 to obtain a changed reference classification feature.

[0087] In some embodiments, the screening unit is configured to determine the candidate image corresponding to any candidate classification feature as the target image if the distance between any candidate classification feature and the reference classification feature is less than the reference distance.

[0088] In some embodiments, the plurality of candidate images are sample images used to train the first image classification model, and the apparatus further comprises:

[0089] The training unit is configured to discard the screened target image from the multiple candidate images and train the first image classification model based on the remaining candidate images so that the classification result determined by the trained first image classification model is no longer affected by the target object.

[0090] In some embodiments, the plurality of candidate images are sample images in a sample set used to train a first image classification model, the sample set also includes a first labeled classification result of each candidate image, and the apparatus further includes:

[0091] An allocating unit is configured to re-allocate a second labeling classification result for the screened target image to obtain an updated sample set, wherein the second labeling classification result of the target image is opposite to the first labeling classification result;

[0092] The training unit is configured to perform training on the first image classification model according to the updated sample set so that the classification result determined by the trained first image classification model according to the target object is opposite to the classification result determined by the first image classification model before training.

[0093] In some embodiments, the apparatus further comprises:

[0094] The training unit is configured to perform training on a second image classification model according to the determined target image, wherein the second image classification model is used to process according to the target object in any image.

[0095] In some embodiments, the reference image is multiple, each reference image includes a different target object, and the device further includes:

[0096] The training unit is configured to execute training of multiple second image classification models according to the target image corresponding to each reference image, and different second image classification models are used for processing according to different target objects.

[0097] In some embodiments, the apparatus further comprises:

[0098] A second acquisition unit is configured to acquire at least one target object;

[0099] a selection unit configured to select a second image classification model matching the at least one target object from the plurality of second image classification models;

[0100] The processing unit is configured to execute calling the selected at least one second image classification model to process any image and obtain at least one classification result of the image.

[0101] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0102] one or more processors;

[0103] volatile or non-volatile memory for storing the one or more processor-executable instructions;

[0104] Wherein, the one or more processors are configured to execute the target image screening method as described in the first aspect above.

[0105] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the target image screening method as described in the first aspect above.

[0106] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising executable instructions. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to perform the target image screening method as described in the first aspect above.

[0107] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0109] Figure 1 The figure is a schematic diagram of an implementation environment according to an exemplary embodiment.

[0110] Figure 2 The figure is a flow chart of a target image screening method according to an exemplary embodiment.

[0111] Figure 3 The figure is a flow chart of another target image screening method according to an exemplary embodiment.

[0112] Figure 4 is a schematic diagram showing a reference classification feature according to an exemplary embodiment.

[0113] Figure 5is a schematic diagram showing a fused reference classification feature according to an exemplary embodiment.

[0114] Figure 6 is a schematic diagram showing a modified reference classification feature according to an exemplary embodiment.

[0115] Figure 7 The figure is a block diagram of a target image screening device according to an exemplary embodiment.

[0116] Figure 8 is a block diagram of another target image screening device according to an exemplary embodiment.

[0117] Fig. 9 It is a structural block diagram of a terminal according to an exemplary embodiment.

[0118] Fig.10 The figure is a structural block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0119] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0120] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned figure descriptions are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0121] The terms "at least one", "multiple", "each" and "any" used in this application, at least one includes one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple. For example, the multiple alternative images include 3 alternative images, and each refers to each of the 3 alternative images, and any refers to any one of the 3 alternative images, which can be the first, the second or the third.

[0122] It should be noted that the user information involved in the present disclosure (including but not limited to user device information, user personal information, etc.) may be information authorized by the user or fully authorized by all parties.

[0123] The target image screening method provided by the present disclosure is applied to electronic devices. In some embodiments, the electronic device is a terminal such as a mobile phone, a tablet computer, or a laptop computer. In other embodiments, the electronic device is a server, which is a single server, or a server cluster composed of several servers, or a cloud computing service center. In other embodiments, the electronic device includes a terminal and a server.

[0124] Figure 1 is a schematic diagram of an implementation environment according to an exemplary embodiment. Figure 1 As shown, the implementation environment includes at least one terminal 101 ( Figure 1 3 are taken as an example) and a server 102, the terminal 101 and the server 102 are connected via a wireless network or a wired network.

[0125] In some embodiments, the target application provided by the server 102 is installed on the terminal 101, and the terminal 101 can realize functions such as data transmission and message interaction through the target application. In some embodiments, the target application is a target application in the operating system of the terminal 101, or a target application provided by a third party. For example, the target application is a video playback application, an audio playback application, a browser application, etc.

[0126] In some embodiments, the terminal 101 sends an image to the server through the target application, and the server sends the image to other terminals 101 installed with the target application. In some embodiments, the server is also used to use the image sent by the terminal 101 as a candidate image, and filter out the target image from the candidate images according to the reference classification features of the reference image and the candidate classification features of the candidate images.

[0127] In some embodiments, the terminal 101 sends a video to the server through the target application, and the server sends the video to other terminals 101 installed with the target application. In some embodiments, the server is also used to use the image contained in the video sent by the terminal 101 as a candidate image, and filter out the target image from the candidate images according to the reference classification features of the reference image and the candidate classification features of the candidate images.

[0128] In other embodiments, the terminal 101 obtains an image from a server through a target application, uses the image as a candidate image, and filters out a target image from the candidate images based on reference classification features of the reference image and candidate classification features of the candidate images.

[0129] The target image screening method provided by the embodiments of the present disclosure can be applied to the scenario of training the model.

[0130] For example, the scenario of retraining the trained first image classification model:

[0131] The first image classification model is used to classify bicycles, unicycles, trains and cars as vehicles. However, as people's classification of categories becomes more detailed, unicycles are no longer classified as vehicles. At this time, the first image classification model needs to be adjusted accordingly. The target image screening method provided in the disclosed embodiment is used to screen out target images that include unicycles and do not include other vehicles that affect the classification results. The annotation results of the screened target images are set to not belong to vehicles. The first image classification model is trained based on the screened target images and the annotation results, so that the first image classification model no longer classifies unicycles as vehicles.

[0132] The target image screening method provided by the embodiment of the present disclosure can automatically screen out sample images for training the first image classification model without manually acquiring sample images and labeling sample images, thereby improving the efficiency of acquiring sample images and the efficiency of model training.

[0133] For example, the scenario where an existing first image classification model is used to train a new second image classification model:

[0134] The first image classification model is used to classify bicycles, unicycles, trains, and cars as vehicles, but as people's classification of categories becomes more detailed, people want the image classification model to give more detailed categories, such as bicycles, cars, commercial vehicles, etc. Taking determining whether a vehicle is a bicycle as an example, the target image screening method provided by the embodiment of the present disclosure is used to screen out target images that include bicycles and do not include other vehicles that affect the classification result, and the annotation result of the screened target image is set to a bicycle. The second image classification model is trained based on the screened target image and the annotation result, so that the second image classification model can determine whether the image is a bicycle image. Similarly, a third image classification model that determines whether the image is a car image can be trained, and a fourth image classification model that determines whether the image is a sports car image can be trained, etc.

[0135] The target image screening method provided by the embodiment of the present disclosure can automatically screen out sample images for training image classification models without manually acquiring sample images and labeling sample images, thereby improving the efficiency of acquiring sample images and the efficiency of model training.

[0136] It should be noted that the target image screening method provided by the embodiment of the present disclosure can also be applied to other scenarios, and the embodiment of the present disclosure is not limited to this.

[0137] Figure 2 is a flow chart of a target image screening method according to an exemplary embodiment. Figure 2As shown, the target image screening method is used in an electronic device, comprising the following steps:

[0138] 201. Call a first image classification model to process a reference image, and obtain reference classification features generated in the process of the first image classification model processing the reference image. The reference image includes a target object for determining a classification result of the reference image, and does not include other objects that affect the classification result of the reference image. The reference classification features are features for the first image classification model to determine the classification result of the reference image.

[0139] The first image classification model is a trained image classification model, and the first image classification model has a certain accuracy in classification processing. In addition, the first image classification model can determine the classification result not only according to the target object in the image, but also according to other objects in the image that affect the classification result. For example, the classification result is a vehicle image or a non-vehicle image, the target object is a car, and other objects that affect the classification result are bicycles, trains, and subways.

[0140] 202. Call the first image classification model to process multiple candidate images, and obtain candidate classification features generated in the process of the first image classification model processing the multiple candidate images. The candidate classification features are features used by the first image classification model to determine classification results of the candidate images.

[0141] The process of calling the first image classification model to process multiple candidate images is the same as the process of calling the first image classification model to process the reference image.

[0142] 203. Filter out a target image from the multiple candidate images, where the target image is a candidate image whose candidate classification features match the reference classification features.

[0143] The matching of the candidate classification features with the reference classification features means that the candidate classification features have a high similarity with the reference classification features.

[0144] In the target image screening method provided by the embodiment of the present disclosure, considering that the image includes one or more objects, some objects will affect the classification result of the image, and some objects will not affect the classification result of the image, and the classification feature of the image is a feature for the image classification model to determine the classification result, then the feature of the object that affects the classification result in the feature is retained, and the features of other objects that do not affect the classification result are discarded. Since only the target object is used to determine the classification result in the reference image, the reference classification feature of the reference image can be considered as the feature presented by the target object in the image classification model, so that it can be ensured that the selected candidate classification features that match the reference classification features are all the features presented by the target object in the image classification model, then only the target object affects the classification result in the selected candidate images, which realizes the automatic screening of images and improves the efficiency of screening images. In addition, the selected candidate images can also include other objects other than the target object that do not affect the classification result, or do not include other objects other than the target object, so that the selected candidate images are more diverse.

[0145] In some embodiments, the image classification model includes a feature extraction layer, a feature fusion layer, and a classification layer, and calling the first image classification model to process the reference image includes:

[0146] Calling the feature extraction layer to process the reference image to obtain regional features corresponding to multiple image regions in the reference image;

[0147] Call the feature fusion layer to process the regional features corresponding to multiple image regions to obtain the image features of the reference image;

[0148] The classification layer is called to process the image features to obtain the reference classification features of the reference image, and then the reference classification features are processed to obtain the classification results of the reference image.

[0149] In some embodiments, the classification result indicates whether the reference image belongs to the target category, and the classification layer is called to process the image features to obtain reference classification features of the reference image, including:

[0150] Call the classification layer and perform dot multiplication of the image features with the weight matrix corresponding to the target category to obtain the reference classification features.

[0151] In some embodiments, obtaining a reference classification feature generated by the first image classification model in the process of processing the reference image includes:

[0152] Acquire multiple regional features generated in the process of processing the reference image by the first image classification model, where the regional features are features corresponding to the image regions in the reference image;

[0153] Determine, based on the plurality of regional features, a weight of each image region in the reference image on the classification result;

[0154] If the image region with the largest weight is the region where the target object is located, a reference classification feature generated in the process of processing the reference image by the first image classification model is obtained.

[0155] In some embodiments, the classification result of the reference image is a probability that the reference image belongs to a target category; and determining the weight of each image region in the reference image for the classification result according to the plurality of region features includes:

[0156] According to the regional features of each image region, the classification probability corresponding to each image region is determined. The classification probability indicates the probability that the object in the image region belongs to the target category. The image region with the largest weight is the region with the highest similarity between the classification probability and the classification result.

[0157] In some embodiments, obtaining a reference classification feature generated by the first image classification model in the process of processing the reference image includes:

[0158] Obtaining a classification probability corresponding to each image region generated in a process in which the first image classification model processes the reference image;

[0159] Creating a region identification image according to the mapping relationship between the classification probability and the indicator mark and the classification probability corresponding to each image region, wherein the region identification image includes the indicator mark corresponding to each image region;

[0160] Display area identification image;

[0161] In response to a selection operation on the region identification image, a reference classification feature generated in a process in which the first image classification model processes the reference image is obtained.

[0162] In some embodiments, after displaying the region identification image, the method further includes:

[0163] In response to the discarding operation on the region identification image, the reference image is discarded.

[0164] In some embodiments, there are multiple reference images, and obtaining reference classification features generated when the first image classification model processes the reference images includes:

[0165] Obtain reference classification features for each reference image;

[0166] The obtained multiple reference classification features are fused to obtain fused reference classification features.

[0167] In some embodiments, obtaining a reference classification feature generated by the first image classification model in the process of processing the reference image includes:

[0168] Obtaining a reference classification feature of a reference image, the reference classification feature including a plurality of feature values;

[0169] The feature values ​​of the reference classification features that are less than the reference threshold are changed to 0 to obtain the changed reference classification features.

[0170] In some embodiments, selecting a target image from a plurality of candidate images includes:

[0171] If the distance between any candidate classification feature and the reference classification feature is less than the reference distance, the candidate image corresponding to the candidate classification feature is determined as the target image.

[0172] In some embodiments, the plurality of candidate images are sample images used to train the first image classification model. After selecting the target image from the plurality of candidate images, the method further includes:

[0173] From the multiple candidate images, the screened target image is discarded, and the first image classification model is trained based on the remaining candidate images, so that the classification result determined by the trained first image classification model is no longer affected by the target object.

[0174] In some embodiments, the plurality of candidate images are sample images in a sample set used to train the first image classification model, and the sample set also includes a first labeled classification result of each candidate image. After the target image is screened out from the plurality of candidate images, the method further includes:

[0175] Reassigning the second annotation classification result to the screened target image to obtain an updated sample set, wherein the second annotation classification result of the target image is opposite to the first annotation classification result;

[0176] The first image classification model is trained according to the updated sample set, so that the classification result determined by the trained first image classification model according to the target object is opposite to the classification result determined by the first image classification model before training.

[0177] In some embodiments, after selecting the target image from the plurality of candidate images, the method further includes:

[0178] The second image classification model is trained according to the determined target image, and the second image classification model is used to process according to the target object in any image.

[0179] In some embodiments, there are multiple reference images, each of which includes a different target object. After the selected candidate image is determined as the target image, the method further includes:

[0180] According to the target image corresponding to each reference image, multiple second image classification models are trained respectively, and different second image classification models are used to process according to different target objects.

[0181] In some embodiments, after training a plurality of second image classification models respectively according to the target image corresponding to each reference image, the method further comprises:

[0182] Get at least one target object;

[0183] Selecting a second image classification model that matches at least one target object from a plurality of second image classification models;

[0184] At least one selected second image classification model is called to process any image to obtain at least one classification result of the image.

[0185] It should be noted that after training the image classification model, the image classification model will learn the classification rules contained in the sample images, and then process the images input into the image classification model according to the classification rules. However, with the diversification of images on the Internet and changes in people's classification standards, the classification rules need to change accordingly.

[0186] The classification rule refers to the classification rule for a certain object in the image, and the change of the classification rule may include modifying, adding or deleting a certain rule.

[0187] For example, an image classification model needs to classify vehicle images and non-vehicle images, and the classification rules of the image classification model are "if the image includes tires, then the image is determined to be a vehicle image", "if the image includes a vehicle body, then the image is determined to be a vehicle image", and "if the image includes tires and a vehicle body, then the image is determined to be a vehicle image". However, there is a possibility that the image only includes tires but not a vehicle body. Obviously, it is unreasonable to classify the image as a vehicle image. Therefore, the classification rule for tires in the image classification model needs to be modified to "if the image only includes tires but not a vehicle body, then the image is determined to be a non-vehicle image".

[0188] In order to modify the classification rules in the image classification model, it is necessary to reacquire sample images and train the image classification model. For example, obtain pictures that include tires but not the vehicle body and train the image classification model based on the pictures. Alternatively, discard the pictures that include tires but not the vehicle body in the sample images of the image classification model and retrain the image classification model so that the retrained image classification model complies with the modified classification rules.

[0189] Since the classification rules in the image classification model are all aimed at the objects in the image, whether the classification rules are modified, added or deleted, the image classification model can be trained by acquiring sample images including the corresponding objects to change the classification rules in the image classification model.

[0190] The following Figure 3 The illustrated embodiment provides a target image screening method, which can obtain multiple images that contain the target object targeted by the classification rule for determining the classification result, and do not contain other objects that affect the classification result.

[0191] Figure 3 is a flow chart of a target image screening method according to an exemplary embodiment. Figure 3 As shown, the target image screening method is applied to an electronic device, comprising the following steps:

[0192] 301. Call a first image classification model to process a reference image to obtain a classification result of the image.

[0193] The first classification model is a model for processing images. The embodiment of the present disclosure does not limit the category determined by the first image classification model. The first image classification model is a model that classifies images into any category. For example, the first image classification model is a person image classification model, which is used to determine whether a picture is a person picture. For another example, the first image classification model is a sensitive image classification model, which is used to determine whether an image includes sensitive information.

[0194] It should be noted that the first image classification model is a trained model with certain classification capabilities. In some embodiments, the first image classification model is a single classification model, that is, the first image classification model is used to determine whether the image belongs to a certain category. In other embodiments, the first image classification model is a multi-classification model, that is, the first image classification model is not only used to determine whether the image belongs to a certain category, but also used to determine whether the image belongs to other categories.

[0195] The reference image in step 301 is any image. In some embodiments, the reference image refers to a sample image in a sample set of a first image classification model. In other embodiments, the reference image refers to an image manually screened. In other embodiments, the reference image is any image circulated on the Internet. The embodiments of the present disclosure do not limit the reference image in step 301. In some embodiments, the reference image in step 301 includes a target pair for determining a classification result of the reference image, and does not include other objects that affect the classification result of the reference image; or, the reference image in step 301 includes a target object for determining a classification result of the reference image, and also includes other objects that affect the classification result of the reference image. In some embodiments, the reference image also includes other objects that do not affect the classification result of the reference image.

[0196] In some embodiments, the reference image is an image in any video.

[0197] In some embodiments, the classification result indicates whether the image belongs to the target category, that is, the classification result includes two cases of "yes" and "no". Among them, "yes" indicates that the image belongs to the target category, and "no" indicates that the image does not belong to the target category, and the target category is a reference category.

[0198] In other embodiments, the classification result is the probability that the image belongs to the target category, and the classification result is a reference image in the range of 0 to 1. If the classification result is greater than 0.5, the image is considered to belong to the target category. If the classification result is not greater than 0.5, the image is considered not to belong to the target category.

[0199] In some embodiments, the first image classification model includes multiple layers, and the multiple layers process the image in sequence to obtain the classification result of the image. In some embodiments, the first image classification model includes a feature extraction layer, a feature fusion layer, and a classification layer, and calling the first image classification model to process the reference image to obtain the classification result of the reference image includes: calling the feature extraction layer to process the reference image to obtain regional features corresponding to multiple image regions in the reference image; calling the feature fusion layer to process regional features corresponding to multiple image regions to obtain image features of the reference image, and calling the image classification model to process the image features to obtain the classification result of the reference image.

[0200] Among them, calling the classification layer, processing the image features, and obtaining the classification result of the reference image includes: calling the classification layer, processing the image features, obtaining the classification features of the reference image, and then processing the classification features to obtain the classification result of the reference image.

[0201] It can be seen from the process of processing the image with the above-mentioned first image classification model that the reference classification feature is different from the image feature. The reference classification feature is obtained by processing the image feature through the classification layer. Since the classification only focuses on the objects that affect the classification results, but not the objects that are not related to the classification results, the features of the objects that affect the classification results in the classification layer are retained, while the features of other objects that do not affect the classification results are discarded or suppressed. Therefore, by obtaining the classification layer to process the image features, it is achieved that only the features in the image that affect the classification results are obtained.

[0202] The image region is a region in a reference image. For example, if the reference image is a picture, the image region is a picture region in the picture.

[0203] In some embodiments, the reference image is a reference image among images obtained by manual screening, and the first image classification model includes a feature extraction layer, a feature fusion layer and a classification layer. The first image classification model is called to process the reference image to obtain a classification result of the reference image, including: calling the feature extraction layer to process the reference image to obtain regional features corresponding to multiple image regions in the reference image; calling the feature fusion layer to process regional features corresponding to multiple image regions to obtain image features of the reference image; calling the image classification model to process the image features to obtain the classification result of the reference image.

[0204] Among them, calling the image classification model, processing the image features, and obtaining the classification result of the reference image includes: calling the image classification model, processing the image features, obtaining the reference classification features of the reference image, and then processing the reference classification features to obtain the classification result of the reference image.

[0205] Among them, when performing feature extraction, the feature extraction layer will determine the image area for feature extraction according to the size of the convolution kernel in the feature extraction layer, perform feature extraction on each image area, and obtain regional features corresponding to multiple image areas. The feature fusion layer will fuse multiple regional features into one image feature. In some embodiments, fusing image features means: superimposing multiple regional features; or superimposing multiple regional features and then averaging them, etc. The disclosed embodiments do not limit the fusion method.

[0206] In some embodiments, the fully connected layer includes a weight matrix, which is a matrix formed by the proportion of features of different dimensions in the image features in the classification. The fully connected layer is called to perform statistical processing on the image features and the weight matrix to obtain the classification result of the image. In some embodiments, the fully connected layer is called to perform dot multiplication of the image features and the weight matrix and then add them to obtain a numerical value, which is the probability that the image belongs to the target category, or the numerical value is processed in other ways to obtain the probability that the image belongs to the target category.

[0207] In some embodiments, taking an image as a reference image as an example, calling the classification layer, processing the image features, and obtaining the reference classification features of the reference image, including: calling the classification layer, performing a dot multiplication of the image features with the weight matrix corresponding to the target category, and obtaining the reference classification features.

[0208] Since the weight matrix includes features of different dimensions in the image features, which have different proportions in the classification, the image features are dot-multiplied with the weight matrix corresponding to the target category. The reference classification features obtained are features of multiple dimensions that affect the classification results, that is, the features presented by the target object in the image classification model.

[0209] For example, after the feature extraction layer extracts features from the image, it obtains regional features of 49 image regions, each of which is a 2048-dimensional vector. The feature fusion layer fuses the 49 2048-dimensional vectors into a 2048-dimensional vector, which is input into the classification layer. The 2048-dimensional vector is point-multiplied with the weight matrix of the target category to obtain a 2048-dimensional vector as the classification feature.

[0210] The embodiment of the present application obtains the dot product result of the weight matrix corresponding to the image feature and the target category, and does not involve the weight matrix corresponding to other categories. Therefore, this method is not only applicable to single-classification models, but can also be applied to multi-classification models, which improves the applicability of this solution. In some embodiments, the first image classification model is a single-classification model, or a multi-classification model. If the first image classification model is a single-classification model, the weight matrix corresponding to the above-mentioned target category is: the weight matrix of the classification layer of the first image classification model. If the first image classification model is a multi-classification model, the weight matrix corresponding to the above-mentioned target category is: the weight matrix corresponding to the target category in the classification layer of the first image classification model.

[0211] For example, the first image classification model is an N-classification model, and the weight matrix of the classification layer in the first image classification model is an M*N matrix, where N is a reference integer greater than or equal to 2, and M is the number of dimensions of image features, which is a reference integer greater than or equal to 1. The first column of the weight matrix is ​​the weight matrix corresponding to the first category, the second column of the weight matrix is ​​the weight matrix corresponding to the second category, the third column of the weight matrix is ​​the weight matrix corresponding to the third category, and so on.

[0212] It should be noted that the feature extraction layer includes multiple layers, and the multiple feature extraction layers extract features of the image in sequence, and the number of image regions will decrease successively with the feature extraction process until the number of image regions is 1. At this time, the regional features obtained are equivalent to the image features. Therefore, in some embodiments, the first image classification model includes a feature extraction layer and a classification layer. The feature extraction layer is called to process the reference image to obtain the image features of the reference image, and the classification layer is called to process the reference image features to obtain the classification results of the reference image.

[0213] In some embodiments, the feature extraction layer is a convolutional layer, the feature fusion layer is a pooling layer, the classification layer is a fully connected layer, etc. The embodiment of the present disclosure only exemplifies the structure of the first image classification model with the feature extraction layer, the feature fusion layer and the classification layer, and does not limit the model structure of the first image classification model.

[0214] 302. Acquire multiple region features corresponding to the reference image, where the multiple region features are generated when the first image classification model processes the reference image, and the region features are features corresponding to the image region in the reference image.

[0215] According to the above step 301, the first image classification model generates regional features corresponding to multiple image regions in the reference image during the process of processing the reference image. Therefore, in step 302, multiple regional features corresponding to the reference image can be directly obtained. That is, obtaining multiple regional features corresponding to the reference image means: obtaining multiple regional features generated by the first image classification model during the process of processing the reference image.

[0216] In some embodiments, the regional features are features extracted by the feature extraction layer in the first image classification model, and obtaining multiple regional features corresponding to the reference image includes: obtaining multiple regional features corresponding to the reference image from the feature extraction layer in the first image classification model, and the regional features are features obtained after the feature extraction layer extracts features from the reference image.

[0217] In some embodiments, the feature extraction layer of the first image classification model includes multiple layers, and multiple regional features corresponding to the reference image are obtained from the feature extraction layer in the first image classification model, including: obtaining multiple regional features corresponding to the reference image from the last layer of the multiple feature extraction layers, and the regional features are features obtained after the last feature extraction layer extracts features from the reference image.

[0218] In some embodiments, an output interface is provided in the feature extraction layer of the first image classification model, and after the feature extraction layer generates regional features, the generated regional features are output through the output interface. In other embodiments, other programs are used to obtain from the first image classification model a plurality of regional features generated by the first image classification model in the process of processing the reference image.

[0219] 303. Determine the weight of each image region in the reference image for the classification result according to the multiple region features.

[0220] When the first image classification model is used to process the image, if the first image classification model detects that there is an object belonging to the target category in the reference image, the image is classified as an image belonging to the target category. For example, the first image classification model is a model for determining whether a picture is a vehicle picture. If the first image classification model detects that there is a car in the picture, the picture is classified as a vehicle picture. Although the picture also includes objects such as roads and pedestrians, these objects do not affect the classification result of the picture. It can be seen that the car in the picture has a greater weight on the classification result and has a greater impact on the classification result, while other objects have a smaller weight on the classification result and have a smaller impact on the classification result, or the weight is 0 and does not affect the classification result. Among them, the weight on the classification result indicates the degree of influence on the classification result.

[0221] Since the target image screening method provided by the embodiment of the present disclosure is used to screen images containing the target object for determining the classification result, and the target object for determining the classification result has a greater weight on the classification result, therefore, by determining the degree of influence of multiple image areas in the image on the classification result, as well as the objects in each image area, it is possible to determine whether the reference image includes the target object for determining the classification result of the reference image, but does not include other objects that affect the classification result of the reference image, that is, to determine whether the reference image is the image that the target image screening method wants to screen out. Only when the reference image is the image that the target image screening method wants to screen out, can the desired target image be screened with reference to the reference image.

[0222] Since different objects in the reference image may be distributed in different image areas of the reference image, different image areas have different influences on the classification results. If a certain image area includes a target object used to determine the classification result, then the image area has a higher degree of influence on the classification result. If a certain image area only includes other objects that are not related to the classification result, then the image area has a lower degree of influence on the classification result.

[0223] Each image region has a corresponding regional feature, and according to the regional feature, it can be determined whether the image region belongs to the target category, and according to whether the image region belongs to the target category, it can be determined how much the image region affects the classification result. In some embodiments, the classification result of the image is whether it belongs to the target category; according to the regional feature of each image region, it is determined whether each image region belongs to the target category, if the category to which the image region belongs is the same as the category to which the image belongs, it is considered that the image region has a greater degree of influence on the classification result; if the category to which the image region belongs is different from the category to which the image belongs, it is considered that the image region has a smaller degree of influence on the classification result.

[0224] Among them, according to the regional features of each image area, determining whether each image area belongs to the target category includes: calling the classification layer of the first image classification model, processing the regional features of each image area in turn, and obtaining the classification result corresponding to each image area, and the classification result indicates whether the image area belongs to the target category.

[0225] In other embodiments, the classification result of the image is the probability that the image belongs to the target category. The classification probability corresponding to each image area is determined based on the regional features of the image area. The classification probability indicates the probability that the object in the image area belongs to the target category. The image area with the largest weight is the area with the highest similarity between the classification probability and the classification result.

[0226] Among them, according to the regional features of each image area, the classification probability corresponding to each image area is determined, including: calling the classification layer of the first image classification model, processing the regional features of each image area in turn, and obtaining the classification result corresponding to each image area, and the classification result indicates the probability that the image area belongs to the target category.

[0227] The regional features of any image region are processed to obtain the classification result corresponding to the image region, including: multiplying the image region with the weight matrix of the target category, and adding the bias term to obtain the probability that the image region belongs to the target category.

[0228] In some embodiments, after determining the classification probability corresponding to each image region, the method further includes: according to the similarity between the classification probability of the image region and the classification result, determining the similarity as the weight of the classification result.

[0229] The disclosed embodiment is only illustrative of the image region with the largest weight as the region with the highest classification probability and the highest similarity to the classification result. Since whether an image belongs to the target category is determined by whether the object in the image belongs to the target category, if the classification probability corresponding to the image region with the highest classification probability exceeds 0.5, the image region belongs to the target category and the image also belongs to the target category. If the classification probability corresponding to the image region with the highest classification probability does not exceed 0.5, the image does not belong to the target category.

[0230] 304. If the image region with the largest weight is the region where the target object is located, obtain reference classification features generated in the process of processing the reference image by the first image classification model.

[0231] Among them, the image area with the largest weight is the area where the target object is located, which means that: the image area with the largest weight includes the target object or part of the target object, but does not include other objects. Therefore, the classification result of the image is determined by the target object, not other objects, that is, the reference image includes the target object used to determine the classification result of the reference image, but does not include other objects that affect the classification result of the reference image. The reference classification features of the reference image will be obtained only when the reference image meets the requirements of including the target object that affects the classification result, but not including other objects that affect the classification result, thereby improving the accuracy of the reference classification features, so that other target images whose classification results are determined by the target object can be screened out from the candidate images based on the reference image.

[0232] It should be noted that the region where the target object is located may be multiple image regions, and therefore, the image region with the largest weight may also be multiple regions. The embodiment of the present disclosure does not limit the image region with the largest weight.

[0233] It should be noted that the embodiment of the present disclosure is only described by checking the image area with the largest weight to determine whether the reference image includes the target object used to determine the classification result of the reference image, but does not include other objects that affect the classification result of the reference image. In another embodiment, the area where the target object is located may be multiple image areas, and the weights of the multiple image areas to the classification result are not exactly the same. Therefore, if the image area whose weight reaches the target weight threshold is the area where the target object is located, the reference image includes the target object used to determine the classification result of the reference image, but does not include other objects that affect the classification result of the reference image, and obtains the reference classification feature of the reference image. In some embodiments, the weight reaches the target weight threshold means that the similarity between the classification probability of the image area and the classification result reaches the reference similarity threshold. In other embodiments, the target weight threshold is 0.5 or any value greater than 0.5 and less than 1. Since the classification probability of the reference image is jointly determined by the classification probabilities of multiple image areas, the classification probability of the image area is compared with the classification probability of the reference image, so that the image area with the greatest influence on the classification probability of the reference image can be accurately determined.

[0234] It should be noted that the disclosed embodiments are only described by taking the example of an electronic device processing a reference image through a first image classification model to determine whether the reference image includes a target object used to determine the classification result of the reference image and does not include other objects that affect the classification result of the reference image, while in other embodiments, the user determines whether the reference image includes a target object used to determine the classification result of the reference image and does not include other objects that affect the classification result of the reference image. In order to enable the user to make a more accurate judgment, the electronic device also provides a region identification image, which includes an indicator mark corresponding to each image region, and the indicator mark is used to indicate the classification probability corresponding to the image region.

[0235] In some embodiments, a classification probability corresponding to each image region in a reference image is obtained, and the classification probability is obtained based on regional features generated in the process of processing the reference image by a first image classification model; a region identification image is created based on a mapping relationship between the classification probability and the indicator identifier and the classification probability corresponding to each image region, and the region identification image includes an indicator identifier corresponding to each image region; the region identification image is displayed; and in response to a selection operation on the region identification image, a reference classification feature generated in the process of processing the reference image by the first image classification model is obtained.

[0236] By displaying the region identification image to the user, the user confirms whether the reference image includes the target object but does not include other objects that affect the classification result, and after the user confirms, the reference classification feature is obtained, thereby improving the accuracy of the reference classification feature.

[0237] The manner of determining the classification probability of the image region according to the regional features of the image region has been shown in step 304 and will not be described in detail here.

[0238] The indicator is any form of indicator, for example, the indicator is a color, the smaller the classification probability, the colder the corresponding color; the larger the classification probability, the warmer the corresponding color, and the obtained area identification image is a heat map. For another example, the indicator is an indicator of different shapes, and the indicator shapes corresponding to different classification probability intervals are different. The size of the classification probability can be determined according to the shape of the indicator in the area identification map.

[0239] In some embodiments, the region identification image is a CAM (Class Activation Map) image.

[0240] In some embodiments, when creating a region identification image, the region identification image is created according to the size of each image region and the indicator mark corresponding to each image region, so that each indicator mark is displayed in the corresponding image region. In other embodiments, when creating a region identification image, multiple image regions are proportionally enlarged according to the size of a reference image, so that the sum of the sizes of the multiple image regions is the same as the size of the reference image, and the region identification image is created according to the size of the enlarged image region and the indicator mark corresponding to each image region, so that each indicator mark is displayed in the corresponding and enlarged image region.

[0241] In some embodiments, the created region identification image is directly displayed; or the created region identification image is superimposed on the reference image to enable the user to directly determine which regions in the reference image have a greater impact on the classification result. Alternatively, in some embodiments, when creating the region identification image, an indicator corresponding to each image region is added to the original reference image.

[0242] After the region identification image is displayed, the user can operate the region identification image to process the corresponding reference image. In some embodiments, the selection operation on the region identification image refers to: selecting the region where the region identification image is located, or selecting the option corresponding to the region identification image. The disclosed embodiment does not limit the method for triggering the selection operation on the region identification image.

[0243] In some embodiments, after displaying the region identification image, the method further includes: in response to a discard operation on the region identification image, discarding the reference image. If the user finds that the classification result of the reference image is not determined by the target object in the image, or the classification result of the image is not determined only by the target object in the image, the reference image does not meet the requirements, and therefore, the reference image is discarded.

[0244] It should be noted that the embodiment of the present disclosure only selects or discards the reference image by performing a selection operation or a discarding operation on the region identification image, while in another embodiment, the electronic device only displays the region identification image to the user, and the user manually discards the corresponding reference image based on the region identification image to obtain the reference classification features of the remaining reference images, thereby ensuring the accuracy of the obtained reference classification features.

[0245] It should be noted that the above steps 302 to 304 are optional steps. In some embodiments, after executing step 301, steps 302 to 304 are executed to determine whether the reference image meets the requirements. In other embodiments, the reference image in step 301 is any reference image, and then after executing step 301, step 305 is directly executed. It should be noted that even if the reference image in step 301 is a reference image selected by the user, steps 302 to 304 can still be executed to verify the reference image. In other words, steps 302 to 304 are selectively executed according to actual needs.

[0246] 305. Obtain reference classification features generated when the first image classification model processes the reference image.

[0247] In step 301, the first image classification model is called to process the reference image to obtain the classification result of the reference image, and the reference classification features have been generated. Therefore, in step 305, the reference classification features generated in the process of processing the reference image by the first image classification model can be directly obtained.

[0248] In some embodiments, there are multiple reference images, and obtaining reference classification features generated in the process of the first image classification model processing the reference images includes: obtaining reference classification features of each reference image; fusing the obtained multiple reference classification features to obtain fused reference classification features. By referring to multiple reference images, the accuracy of the obtained reference classification features is improved, thereby improving the accuracy of image screening.

[0249] In some embodiments, fusing the obtained multiple reference classification features to obtain the fused reference classification features includes: summing and averaging the obtained multiple reference classification features to obtain the averaged reference classification features. By summing and averaging the reference classification features of multiple reference images, the reference classification features are made more accurate.

[0250] In some embodiments, obtaining a reference classification feature generated in the process of processing a reference image by a first image classification model includes: obtaining a reference classification feature of the reference image, the reference classification feature including a plurality of feature values; changing a feature value in the reference classification feature that is less than a reference threshold to 0, to obtain a changed reference classification feature. Since in addition to the target object, there may be other objects in the reference image that have similar features to the target object, which may slightly interfere with the classification result. For example, the reference image is a picture of a car, and the picture includes a poster of a tire, so the obtained reference classification feature includes not only the features of the car, but also the features of the tire. Therefore, changing the feature value in the reference classification feature that is less than the reference threshold to 0 can cause the features of other objects in the reference classification feature to be discarded, and the target image subsequently screened according to the reference classification feature is more accurate. In some embodiments, a feature value in the reference classification feature that is less than the reference threshold can be considered to be a feature value close to 0.

[0251] For example, the reference classification features of 10 reference images are obtained, such as Figure 4 As shown, the reference classification features of the 10 reference images are summed and averaged to obtain the averaged reference classification features, as shown in Figure 5 As shown, the feature values ​​less than the reference threshold in the averaged reference classification feature are changed to 0 to obtain the changed reference classification feature, as shown in Figure 6 shown.

[0252] 306. Filter out a target image from the multiple candidate images, where the target image is a candidate image whose candidate classification features match the reference classification features.

[0253] The candidate image is any image. In some embodiments, the candidate image is a sample image in a sample set used to train the first image classification model; in some embodiments, the candidate image is any image circulated on the Internet. The embodiments of the present disclosure do not limit the candidate images.

[0254] It should be noted that the method for obtaining the candidate classification features is similar to the method for obtaining the reference classification features, and the embodiments of the present disclosure will not be described in detail here.

[0255] In some embodiments, selecting a target image from a plurality of candidate images includes: if a distance between any candidate classification feature and a reference classification feature is less than a reference distance, determining the candidate image corresponding to the candidate classification feature as the target image.

[0256] Among them, the smaller the distance between the candidate classification feature and the reference classification feature, the more similar the corresponding candidate image is to the reference image. Based on the distance between the classification features, the candidate images are screened to ensure that the classification features of the screened target image have a high similarity with the reference classification features of the reference image. Here, the more similar the candidate image is to the reference image does not mean that all objects in the candidate image are similar to all objects in the reference image, but that the candidate image, like the reference image, contains the target object used to determine the classification result of the image, but does not include other objects that affect the classification result of the image. In addition, since the classification features do not include features of other objects that do not affect the classification results, the screened target images can include a variety of other objects that do not affect the classification results, making the screened target images more diverse.

[0257] In some embodiments, the distance between the candidate classification feature and the reference classification feature is cosine distance, Euclidean distance, Manhattan distance, Chebyshev distance, etc. The embodiments of the present disclosure do not limit the distance between the candidate classification feature and the reference classification feature, and the method for determining the distance.

[0258] In some embodiments, after executing step 306, execute step 307; or, execute step 308 and step 309; or, execute step 310; or, execute step 307 and step 310; or, execute steps 308 to 310, etc.

[0259] 307. Discard the selected target image from the multiple candidate images, and train the first image classification model based on the remaining candidate images.

[0260] Among them, multiple candidates are sample images used to train the first image classification model. The remaining sample images are obtained by cleaning the sample images, and the first image classification model is trained based on the remaining sample images.

[0261] In some embodiments, the target object is an object that once belonged to a target category but no longer belongs to the target category. Therefore, the image whose classification result is determined by the target object is cleaned out of the sample image, and then the sample image is used to train the first image classification model so that the classification result determined by the trained first image classification model is no longer affected by the target object.

[0262] For example, the first image classification model is a model that classifies bicycles, unicycles, cars and trains as vehicles. However, as people refine the classification rules, they believe that unicycles are no longer vehicles. Therefore, from the sample images of the first image classification model, images that include unicycles and whose classification results are determined by unicycles are screened out and discarded, and the remaining sample images are used to train the first image classification model.

[0263] Since the model is prone to fit the most recent images and ignore previously trained images during the training process, the present application cleans out the images whose classification results are determined by the target object from the sample images, and then uses the remaining sample images to train the image classification model, so that the classification results determined by the trained image classification model are no longer affected by the target object, and there is no need to re-create the sample set to train a new image classification model.

[0264] In some embodiments, the remaining sample images are directly used to continue training the first image classification model; in other embodiments, in order to ensure the accuracy of the model, the first image classification model is initialized, and the remaining sample images are used to train the initialized first image classification model, so that the classification results determined by the trained first image classification model are no longer affected by the target object.

[0265] 308. Reassign a second labeling classification result to the screened target image to obtain an updated sample set, wherein the second labeling classification result of the target image is opposite to the first labeling classification result.

[0266] The plurality of candidate images are sample images in a sample set used to train the first image classification model, and the sample set also includes a first labeled classification result of each candidate image. In some embodiments, the target object is an object that once belonged to the target category but no longer belongs to the target category. Therefore, the image whose classification result is determined by the target object is screened out from the sample images, and the labeling result corresponding to the image is changed so that the changed labeling result is opposite to the previous labeling result.

[0267] For example, the sample set includes bicycle pictures, unicycle pictures, car pictures and train pictures, and the annotation results of each picture are all vehicle pictures. However, as people refine the classification rules, they believe that unicycles are no longer vehicles. Therefore, the unicycle pictures in the sample set are screened out, and the classification results of the unicycle pictures are reclassified and annotated. The unicycle pictures are marked as not belonging to vehicle pictures, and the updated sample set is obtained.

[0268] 309. Train the first image classification model according to the updated sample set, so that the classification result determined by the trained first image classification model according to the target object is opposite to the classification result determined by the first image classification model before training.

[0269] The image classification model is trained using a sample set, and the image classification model can learn the classification rules implicit in the sample set. The present application updates the sample set by changing the classification results of the target object, that is, updates the classification rules implicit in the sample set. When the image classification model is trained using the updated sample set, the image classification model can learn the new classification rules, thereby realizing the adjustment of the classification rules of the image classification model.

[0270] For example, the updated sample set is used to train the first image classification model. After the training, the first image classification model no longer classifies the unicycle image as a vehicle image, but as a non-vehicle image.

[0271] 310. Train a second image classification model according to the screened target image, and the second image classification model is used to process according to the target object in any image.

[0272] In some embodiments, the second image classification model and the first image classification model are the same image classification model. In another embodiment, the second image classification model and the first image classification model are not the same image classification model.

[0273] Taking the second image classification model and the first image classification model as the same image classification model as an example, the following description is made: In some embodiments, the target object is a newly added object belonging to the target category. Since the target object is a newly added object belonging to the target category, the target object has a certain similarity with other objects that previously belonged to the target category. The first image classification model can simply process the target object, but the classification result is not accurate enough. Therefore, it is necessary to obtain multiple images containing the target object and the classification result is determined by the target object, and train the first image classification model so that the first image classification model can accurately classify according to the target object in any image.

[0274] For example, if it is currently necessary to classify the high-speed rail as a vehicle, a picture containing the high-speed rail is obtained, and the first image classification model is trained using the picture so that the first image classification model can classify the picture containing the high-speed rail as a vehicle picture.

[0275] At least one high-speed rail picture is manually screened, and the at least one high-speed rail picture is input into the first image classification model. The first image classification model processes the high-speed rail picture. Since the high-speed rail has similar features to other vehicles such as trains, the first image classification model can process the high-speed rail picture, but the classification result may not be accurate. After the first image classification model processes at least one high-speed rail picture, the classification features generated in the classification process are obtained, and the reference classification features are determined according to the classification features of at least one high-speed rail picture. The first image classification model is used to process the candidate images, and the high-speed rail picture is screened out from the candidate images according to the candidate classification features and the reference classification features. Since the first image classification model does not accurately classify the high-speed rail picture, the screened pictures may also include other pictures. Therefore, the screened pictures are manually annotated, and the first image classification model is trained according to the screened pictures and the corresponding annotations, so that the first image classification model can accurately process the high-speed rail picture.

[0276] Due to reasons such as the inability of the first image classification model to accurately identify the target object, the screened target images also include images that do not contain the target object but contain other objects similar to the target object. In order to make the trained first image classification model more accurate, in some embodiments, from multiple target images, target images that do not contain the target object but contain other objects similar to the target object are discarded, and the remaining target images are used to continue training the first image classification model; or, in some embodiments, the screened target images are manually annotated, and target images that do not contain the target object but contain other objects similar to the target object are used as negative samples, and the trained first image classification model is continued to be trained using the screened target images and the annotated classification results corresponding to the target images.

[0277] Taking the example that the second image classification model is not the same as the first image classification model, the following is explained: In some embodiments, the target object is any object belonging to the target category. However, as people's requirements for classification gradually increase, the target object can also be divided into more detailed categories. For example, the target object is a bicycle, and the target category is a vehicle. Since the classification of the first image classification model is relatively rough, it is necessary to train a second image classification model with more detailed classification, and the second image classification model is used to classify a bicycle as a bicycle, or the second image classification model is used to classify a bicycle as a mountain bike, etc.

[0278] Therefore, the second image classification model is a model for determining whether an image belongs to a first target category, and the first image classification model is a model for determining whether an image belongs to a second target category, wherein the first target category is a refinement or branch of the second target category, etc.

[0279] In some embodiments, training a second image classification model based on the screened target images includes: assigning a labeled classification result to the screened target images, the labeled classification result indicating that the target images belong to a first target category; training the second image classification model based on the screened target images and the labeled classification result corresponding to each target image, so that the second image classification model determines whether the image belongs to the first target category based on the target object in the image.

[0280] However, the screened target images may include images that are similar to the target object and interfere with the classification results. For example, if the target object is a person's legs, the screened target images may not include human legs but include cat's legs, or long socks, or dog's legs, etc. Therefore, in order to make the trained second image classification model more accurate, in some embodiments, the screened target images are manually annotated, and target images that do not contain the target object but contain other objects similar to the target object are used as negative samples. The screened target images and the annotated classification results corresponding to the target images are used to continue training the second image classification model.

[0281] It should be noted that there are multiple reference images, each of which includes a different target object. According to the alternative images corresponding to each reference image, multiple second image classification models are trained respectively, and different second image classification models are used to process according to different target objects. Among them, different target objects are multiple objects belonging to the same target category. The embodiment of the present application trains multiple image classification models by screening out target images containing different target objects, so that different image classification models can be classified according to different target objects, which is conducive to the image classification model to refine the classification rules for classifying target objects.

[0282] For example, the first image classification model is a model for determining whether an image is a vehicle picture, and multiple second image classification models are models for determining whether an image is a bicycle picture, models for determining whether an image is a train picture, models for determining whether an image is a car picture, and so on.

[0283] In some embodiments, after training multiple second image classification models respectively, the method further includes: obtaining at least one target object, selecting a second image classification model that matches the at least one target object from the multiple second image classification models, calling the selected at least one second image classification model, processing any image, and obtaining at least one classification result of the image. In different scenarios, there may be some differences in the classification results of the same object. Therefore, the present application trains an image classification model that determines the classification result based on only one target object, and then selects the corresponding image classification model for classification according to the scenario requirements. There is no need to train a corresponding image classification model for each classification scenario, which improves the applicability of the image classification model and makes the use of the image classification model more flexible.

[0284] In some embodiments, obtaining at least one target object includes: obtaining at least one specified target object, where the target object is a user-specified target object, where the user inputs the at least one target object in an image classification model management interface, and based on the correspondence between the target object and the second image classification model, selecting a second image classification model that matches the at least one target object from multiple second image classification models.

[0285] For example, the first image classification model determines whether a picture is a sensitive picture based on the exposure degree of the sensitive part. The target object is the first sensitive part, or the first sensitive part with a higher exposure degree. It should be noted that the target object is any object, or the target object is the state of any object, that is, the target object is an object in a certain state. For example, the target object is the first sensitive part with a higher exposure degree, or the target object is the first sensitive part with a lower exposure degree. The disclosed embodiments do not limit the target object.

[0286] For example, the first image classification model is used to screen out pictures including the first sensitive part, and the pictures are manually labeled according to the exposure degree of the first sensitive part. In some embodiments, the pictures are labeled according to different standards. One is: if the exposure degree of the first sensitive part in the picture is high, it is labeled as a sensitive picture; if the exposure degree of the first sensitive part in the picture is low or not exposed, it is labeled as a non-sensitive picture. The labeled pictures and the labeled classification results are used to train the second image classification model.

[0287] Another method is: if the exposure level of the first sensitive part in the picture is low or high, it is marked as a sensitive picture; if the first sensitive part in the picture is not exposed, it is marked as a non-sensitive picture. The marked pictures and the marked classification results are used to train another second image classification model.

[0288] Therefore, the two trained second image classification models are both used to determine the classification results according to the target object, but the classification standards are different. Therefore, in practical applications, different second image classification models can be used for processing according to people's requirements for classification standards.

[0289] In some embodiments, since different second image classification models are processed according to different target objects, the classification reason can be displayed to the user according to the classification results of different second image classification models. For example, the classification reason is generated according to the classification result of the second image classification model and the classification object that the second image classification model focuses on. For example, if the classification result of the second image classification model is that it belongs to a sensitive picture, and the target object is a first sensitive part with a high degree of exposure, the generated classification reason is: since the first sensitive part has a high degree of exposure, the image belongs to a sensitive picture.

[0290] In the target image screening method provided by the embodiment of the present disclosure, considering that the image includes one or more objects, some objects will affect the classification result of the image, and some objects will not affect the classification result of the image, and the classification feature of the image is a feature for the image classification model to determine the classification result, then the feature of the object that affects the classification result in the feature is retained, and the features of other objects that do not affect the classification result are discarded. Since only the target object is used to determine the classification result in the reference image, the reference classification feature of the reference image can be considered as the feature presented by the target object in the image classification model, so that it can be ensured that the selected alternative classification features that match the reference classification feature are all the features presented by the target object in the image classification model, then only the target object affects the classification result in the screened target image, realizing automatic screening of the image and improving the efficiency of screening the image. Moreover, the screened target image may also include other objects other than the target object that do not affect the classification result, or may not include other objects other than the target object, so that the screened target image is more diversified.

[0291] Furthermore, the reference image is automatically screened by using an electronic device to identify whether the image area that affects the classification result is the area where the target object is located, so that the screened reference image includes the target object that affects the classification result, but does not include other objects that affect the classification result. Therefore, the interference features in the reference classification features obtained from the reference image are reduced, making the image screening more accurate.

[0292] Furthermore, by showing the user the area identification image of the image, the user can understand whether the image includes the target object that affects the classification result, as well as other objects that affect the classification result, and the user determines whether the image can be used as a reference image. Therefore, the interference features in the reference classification features obtained by the reference image are reduced, making image screening more accurate.

[0293] In addition, the disclosed embodiment also obtains reference classification features of multiple reference images and performs fusion processing on the reference classification features, so that the fused reference classification features are more accurate.

[0294] Furthermore, the disclosed embodiment will also change the feature values ​​in the reference classification features that are less than the reference threshold to 0, so that the interference features in the reference classification features are suppressed, thereby making the screened target image more accurate.

[0295] Figure 7 is a block diagram of a device for screening target images according to an exemplary embodiment. Figure 7 , the device includes a first acquisition unit 701 and a screening unit 702.

[0296] The first acquisition unit 701 is configured to execute calling a first image classification model to process a reference image, and acquire reference classification features generated in the process of the first image classification model processing the reference image, wherein the reference image includes a target object used to determine a classification result of the reference image, and does not include other objects that affect the classification result of the reference image, and the reference classification features are features for the first image classification model to determine a classification result of the reference image;

[0297] The first acquisition unit 701 is configured to execute calling the first image classification model to process a plurality of candidate images, and acquire candidate classification features generated in the process of the first image classification model processing the plurality of candidate images, wherein the candidate classification features are features used by the first image classification model to determine classification results of the candidate images;

[0298] The screening unit 702 is configured to screen out a target image from the multiple candidate images, where the target image is a candidate image whose candidate classification feature matches the reference classification feature.

[0299] like Figure 8 As shown, in some embodiments, the first acquisition unit 701 includes:

[0300] The feature extraction subunit 7011 is configured to execute calling the feature extraction layer, process the reference image, and obtain regional features corresponding to multiple image regions in the reference image;

[0301] The feature fusion subunit 7012 is configured to execute calling the feature fusion layer, process the regional features corresponding to the multiple image regions, and obtain the image features of the reference image;

[0302] The classification subunit 7013 is configured to execute the call to the classification layer, process the image features to obtain the reference classification features of the reference image, and then process the reference classification features to obtain the classification result of the reference image.

[0303] In some embodiments, the classification result indicates whether the reference image belongs to a target category, and the classification subunit 7013 is configured to execute a call to the classification layer, perform a dot multiplication of the image feature with the weight matrix corresponding to the target category, and obtain the reference classification feature.

[0304] In some embodiments, the first acquiring unit 701 includes:

[0305] An acquisition subunit 7014 is configured to acquire a plurality of regional features generated in the process of processing the reference image by the first image classification model, wherein the regional features are features corresponding to an image region in the reference image;

[0306] A determination subunit 7015 is configured to determine a weight of each image region in the reference image to the classification result according to the multiple region features;

[0307] The acquisition subunit 7014 is configured to execute, if the image area with the largest weight is the area where the target object is located, obtaining the reference classification features generated in the process of the first image classification model processing the reference image.

[0308] In some embodiments, the classification result of the reference image is the probability that the reference image belongs to the target category; the determination subunit 7015 is configured to determine the classification probability corresponding to each image area based on the regional features of each image area, the classification probability indicating the probability that the object in the image area belongs to the target category, and the image area with the largest weight is the area with the highest similarity between the classification probability and the classification result.

[0309] In some embodiments, the first acquiring unit 701 further includes:

[0310] An acquisition subunit 7014 is configured to execute acquisition of a classification probability corresponding to each image region generated in a process in which the first image classification model processes the reference image;

[0311] A creating subunit 7016 is configured to create a region identification image according to a mapping relationship between the classification probability and the indicator and the classification probability corresponding to each image region, wherein the region identification image includes the indicator corresponding to each image region;

[0312] The display subunit 7017 is configured to display the area identification image;

[0313] The acquisition subunit 7014 is configured to execute, in response to a selection operation on the region identification image, to acquire reference classification features generated in the process of the first image classification model processing the reference image.

[0314] In some embodiments, the apparatus further comprises:

[0315] The discarding unit 703 is configured to execute a discarding operation in response to the region identification image to discard the data.

[0316] In some embodiments, there are multiple reference images, and the first acquisition unit 701 is configured to acquire reference classification features of each reference image; and fuse the acquired multiple reference classification features to obtain fused reference classification features.

[0317] In some embodiments, the first acquisition unit 701 is configured to execute acquisition of reference classification features of the reference image, where the reference classification features include multiple feature values; and change feature values ​​in the reference classification features that are less than a reference threshold to 0 to obtain the changed reference classification features.

[0318] In some embodiments, the screening unit 702 is configured to determine the candidate image corresponding to any candidate classification feature as the target image if the distance between any candidate classification feature and the reference classification feature is less than the reference distance.

[0319] In some embodiments, the plurality of candidate images are sample images used to train the first image classification model, and the apparatus further comprises:

[0320] The training unit 704 is configured to discard the screened target image from the multiple candidate images, and train the first image classification model based on the remaining candidate images so that the classification result determined by the trained first image classification model is no longer affected by the target object.

[0321] In some embodiments, the plurality of candidate images are sample images in a sample set used to train a first image classification model, the sample set also includes a first labeled classification result of each candidate image, and the apparatus further includes:

[0322] An allocating unit 705 is configured to re-allocate a second labeling classification result for the screened target image to obtain an updated sample set, wherein the second labeling classification result of the target image is opposite to the first labeling classification result;

[0323] The training unit 704 is configured to perform training on the first image classification model according to the updated sample set so that the classification result determined by the trained first image classification model according to the target object is opposite to the classification result determined by the first image classification model before training.

[0324] In some embodiments, the apparatus further comprises:

[0325] The training unit 704 is configured to perform training on a second image classification model according to the determined target image, where the second image classification model is used to process according to the target object in any image.

[0326] In some embodiments, there are multiple reference images, each of which includes a different target object, and the device further includes:

[0327] The training unit 704 is configured to execute training of multiple second image classification models according to the target image corresponding to each reference image, and different second image classification models are used for processing according to different target objects.

[0328] In some embodiments, the apparatus further comprises:

[0329] A second acquisition unit 706 is configured to acquire at least one target object;

[0330] A selection unit 707 is configured to select a second image classification model that matches at least one target object from a plurality of second image classification models;

[0331] The processing unit 708 is configured to execute calling the selected at least one second image classification model to process any image and obtain at least one classification result of the data.

[0332] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0333] In an exemplary embodiment, an electronic device is also provided, which includes: one or more processors; a volatile or non-volatile memory for storing one or more processor executable instructions; wherein the one or more processors are configured to execute the target image screening method provided by the embodiment of the present disclosure.

[0334] Optionally, the electronic device is provided as a terminal. Fig. 91 is a block diagram of a terminal according to an exemplary embodiment. The terminal 900 is used to execute the steps executed by the terminal in the above embodiment, and can be a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop or a desktop computer. The terminal 900 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0335] The terminal 900 includes: a processor 901 and a memory 902 .

[0336] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0337] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one program code, which is used to be executed by the processor 901 to implement the target image screening method provided in the method embodiment of the present application.

[0338] In some embodiments, the terminal 900 may also optionally include: a peripheral device interface 903 and at least one peripheral device. The processor 901, the memory 902 and the peripheral device interface 903 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 903 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 904, a touch display screen 905, a camera assembly 906, an audio circuit 907, a positioning assembly 908 and a power supply 909.

[0339] The peripheral device interface 903 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0340] The radio frequency circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 904 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 904 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 904 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0341] The display screen 905 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen 905 is a touch display screen, the display screen 905 also has the ability to collect touch signals on the surface or above the surface of the display screen 905. The touch signal can be input to the processor 901 as a control signal for processing. At this time, the display screen 905 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 905 can be one, and the front panel of the terminal 900 is set; in other embodiments, the display screen 905 can be at least two, which are respectively set on different surfaces of the terminal 900 or are folded; in some other embodiments, the display screen 905 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 900. Even, the display screen 905 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 905 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0342] The camera assembly 906 is used to capture images or videos. Optionally, the camera assembly 906 includes a front camera and a rear camera. In some embodiments, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. The dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0343] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 901 for processing, or input them into the radio frequency circuit 904 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 900. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0344] The positioning component 908 is used to locate the current geographical location of the terminal 900 to implement navigation or LBS (Location Based Service). The positioning component 908 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas system, or the European Union's Galileo system.

[0345] The power supply 909 is used to power various components in the terminal 900. The power supply 909 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 909 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0346] In some embodiments, the terminal 900 further includes one or more sensors 910 , including but not limited to: an acceleration sensor 911 , a gyroscope sensor 912 , a pressure sensor 913 , a fingerprint sensor 914 , an optical sensor 915 , and a proximity sensor 916 .

[0347] The acceleration sensor 911 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 900. For example, the acceleration sensor 911 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 901 can control the touch display screen 905 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 911. The acceleration sensor 911 can also be used for collecting game or user motion data.

[0348] The gyro sensor 912 can detect the body direction and rotation angle of the terminal 900, and the gyro sensor 912 can cooperate with the acceleration sensor 911 to collect the user's 3D actions on the terminal 900. The processor 901 can implement the following functions based on the data collected by the gyro sensor 912: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0349] The optical sensor 915 is used to collect the ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the touch display screen 905 according to the ambient light intensity collected by the optical sensor 915. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 905 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 905 is reduced. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera component 906 according to the ambient light intensity collected by the optical sensor 915.

[0350] Those skilled in the art will understand that Fig. 9 The structure shown in the figure does not constitute a limitation on the terminal 900, and the terminal 900 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0351] Optionally, the electronic device is provided as a server. Fig.10 It is a structural block diagram of a server 1000 according to an exemplary embodiment. The server 1000 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 1001 and one or more memories 1002, wherein the memory 1002 stores at least one program code, and the at least one program code is loaded and executed by the processor 1001 to implement the methods provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the server may also include other components for implementing device functions, which will not be repeated here.

[0352] The server 1000 can be used to execute the steps executed by the server in the above target image screening method.

[0353] In an exemplary embodiment, a storage medium is also provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the target image screening method provided by the embodiment of the present disclosure.

[0354] In an exemplary embodiment, a computer program product is also provided, including executable instructions. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute the target image screening method provided by the embodiment of the present disclosure.

[0355] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the schemes disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0356] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A target image screening method, characterized in that: The method comprises: Calling a first image classification model to process a reference image, obtaining a reference classification feature generated in the process of the first image classification model processing the reference image, wherein the reference image includes a target object used to determine a classification result of the reference image, and does not include other objects that affect the classification result of the reference image, and the reference classification feature is a feature for the first image classification model to determine a classification result of the reference image; Calling the first image classification model to process a plurality of candidate images, obtaining candidate classification features generated in the process of the first image classification model processing the plurality of candidate images, wherein the candidate classification features are features for the first image classification model to determine classification results of the candidate images, the plurality of candidate images are sample images in a sample set used to train the first image classification model, and the sample set also includes a first annotated classification result of each candidate image; Filtering out a target image from the multiple candidate images, the target image being a candidate image whose candidate classification feature matches the reference classification feature; reallocating a second labeling classification result to the screened target image to obtain an updated sample set, wherein the second labeling classification result of the target image is opposite to the first labeling classification result; The first image classification model is trained according to the updated sample set, so that a classification result determined by the trained first image classification model according to the target object is opposite to a classification result determined by the first image classification model before training.

2. The method according to claim 1, characterized in that The first image classification model includes a feature extraction layer, a feature fusion layer and a classification layer, and calling the first image classification model to process the reference image includes: Calling the feature extraction layer to process the reference image to obtain regional features corresponding to multiple image regions in the reference image; Calling the feature fusion layer to process the regional features corresponding to the multiple image regions to obtain the image features of the reference image; The classification layer is called to process the image features to obtain reference classification features of the reference image, and then the reference classification features are processed to obtain a classification result of the reference image.

3. The method according to claim 2, characterized in that The classification result indicates whether the reference image belongs to a target category, and the calling of the classification layer to process the image features to obtain reference classification features of the reference image includes: The classification layer is called to perform a dot multiplication of the image feature and the weight matrix corresponding to the target category to obtain the reference classification feature.

4. The method according to claim 1, characterized in that: The obtaining of reference classification features generated in the process of processing the reference image by the first image classification model includes: Acquire a plurality of regional features generated in the process of processing the reference image by the first image classification model, wherein the regional features are features corresponding to image regions in the reference image; Determining, according to the multiple region features, a weight of each image region in the reference image to the classification result; If the image region with the largest weight is the region where the target object is located, a reference classification feature generated in the process of processing the reference image by the first image classification model is obtained.

5. The method according to claim 4, characterized in that The classification result of the reference image is the probability that the reference image belongs to the target category; The step of determining the weight of each image region in the reference image to the classification result according to the multiple region features includes: According to the regional features of each image region, the classification probability corresponding to each image region is determined, the classification probability indicates the probability that the object in the image region belongs to the target category, and the image region with the largest weight is the region with the highest similarity between the classification probability and the classification result.

6. The method according to claim 1, characterized in that The obtaining of reference classification features generated in the process of processing the reference image by the first image classification model includes: Obtaining a classification probability corresponding to each image region generated in a process in which the first image classification model processes the reference image; Creating a region identification image according to a mapping relationship between classification probabilities and indicator marks and the classification probability corresponding to each image region, wherein the region identification image includes the indicator mark corresponding to each image region; Displaying the area identification image; In response to a selection operation on the region identification image, a reference classification feature generated in a process in which the first image classification model processes the reference image is obtained.

7. The method according to claim 6, characterized in that After displaying the area identification image, the method further includes: In response to a discarding operation on the region identification image, the reference image is discarded.

8. The method according to claim 1, characterized in that There are multiple reference images, and obtaining reference classification features generated in the process of processing the reference images by the first image classification model includes: Obtain reference classification features for each reference image; The obtained multiple reference classification features are fused to obtain fused reference classification features.

9. The method according to claim 1, characterized in that: The obtaining of reference classification features generated in the process of processing the reference image by the first image classification model includes: Acquire a reference classification feature of the reference image, wherein the reference classification feature includes a plurality of feature values; The feature values ​​of the reference classification features that are less than the reference threshold are changed to 0 to obtain the changed reference classification features.

10. The method according to claim 1, characterized in that The step of selecting a target image from the plurality of candidate images comprises: If the distance between any candidate classification feature and the reference classification feature is less than the reference distance, the candidate image corresponding to the candidate classification feature is determined as the target image.

11. The method according to claim 1, characterized in that: The multiple candidate images are sample images used to train the first image classification model. After the target image is screened out from the multiple candidate images, the method further includes: The target image selected out from the multiple candidate images is discarded, and the first image classification model is trained based on the remaining candidate images, so that the classification result determined by the trained first image classification model is no longer affected by the target object.

12. The method according to claim 1, characterized in that After selecting the target image from the plurality of candidate images, the method further includes: A second image classification model is trained according to the determined target image, and the second image classification model is used to process according to the target object in any image.

13. The method according to claim 1, characterized in that There are multiple reference images, each of which includes a different target object. After the selected candidate image is determined as the target image, the method further includes: According to the target image corresponding to each reference image, multiple second image classification models are trained respectively, and different second image classification models are used to process according to different target objects.

14. The method according to claim 13, characterized in that After training the plurality of second image classification models respectively according to the target image corresponding to each reference image, the method further comprises: Get at least one target object; Selecting a second image classification model that matches the at least one target object from the plurality of second image classification models; At least one selected second image classification model is called to process any image to obtain at least one classification result of the image.

15. A target image screening device, characterized in that: The device comprises: A first acquisition unit is configured to execute calling a first image classification model to process a reference image, and acquire reference classification features generated in the process of the first image classification model processing the reference image, wherein the reference image includes a target object used to determine a classification result of the reference image, and does not include other objects that affect the classification result of the reference image, and the reference classification features are features for the first image classification model to determine a classification result of the reference image; The first acquisition unit is configured to execute calling the first image classification model to process a plurality of candidate images, and acquire candidate classification features generated in the process of the first image classification model processing the plurality of candidate images, wherein the candidate classification features are features used by the first image classification model to determine classification results of the candidate images, the plurality of candidate images are sample images in a sample set used to train the first image classification model, and the sample set also includes a first annotated classification result of each candidate image; a screening unit configured to screen out a target image from the plurality of candidate images, the target image being a candidate image whose candidate classification feature matches the reference classification feature; An allocating unit is configured to re-allocate a second labeling classification result for the screened target image to obtain an updated sample set, wherein the second labeling classification result of the target image is opposite to the first labeling classification result; The training unit is configured to perform training on the first image classification model according to the updated sample set so that the classification result determined by the trained first image classification model according to the target object is opposite to the classification result determined by the first image classification model before training.

16. The device according to claim 15, characterized in that The first image classification model includes a feature extraction layer, a feature fusion layer and a classification layer, and the first acquisition unit includes: A feature extraction subunit is configured to execute calling the feature extraction layer, process the reference image, and obtain regional features corresponding to multiple image regions in the reference image; A feature fusion subunit is configured to execute calling the feature fusion layer, process the regional features corresponding to the multiple image regions, and obtain the image features of the reference image; The classification subunit is configured to execute the call to the classification layer, process the image features to obtain the reference classification features of the reference image, and then process the reference classification features to obtain the classification result of the reference image.

17. The device according to claim 16, characterized in that The classification result indicates whether the reference image belongs to a target category. The classification subunit is configured to execute the call to the classification layer, perform a dot multiplication of the image feature with a weight matrix corresponding to the target category, and obtain the reference classification feature.

18. The device according to claim 15, characterized in that The first acquisition unit includes: An acquisition subunit is configured to acquire a plurality of regional features generated in a process in which the first image classification model processes the reference image, wherein the regional features are features corresponding to an image region in the reference image; a determination subunit configured to determine a weight of each image region in the reference image to the classification result according to the multiple region features; The acquisition subunit is configured to execute, if the image area with the largest weight is the area where the target object is located, obtaining the reference classification features generated in the process of the first image classification model processing the reference image.

19. The device according to claim 18, characterized in that The classification result of the reference image is the probability that the reference image belongs to the target category; the determination subunit is configured to determine the classification probability corresponding to each image area based on the regional features of each image area, the classification probability indicating the probability that the object in the image area belongs to the target category, and the image area with the largest weight is the area with the highest similarity between the classification probability and the classification result.

20. The device according to claim 15, characterized in that The first acquisition unit further includes: An acquisition subunit is configured to execute acquisition of a classification probability corresponding to each image region generated in a process in which the first image classification model processes the reference image; A creating subunit is configured to create a region identification image according to a mapping relationship between the classification probability and the indicator and the classification probability corresponding to each image region, wherein the region identification image includes the indicator corresponding to each image region; A display subunit, configured to display the area identification image; The acquisition subunit is configured to execute, in response to a selection operation on the region identification image, to acquire reference classification features generated in a process in which the first image classification model processes the reference image.

21. The device according to claim 20, characterized in that The device also includes: The discarding unit is configured to discard the reference image in response to a discarding operation on the region identification image.

22. The device according to claim 15, characterized in that There are multiple reference images, and the first acquisition unit is configured to acquire reference classification features of each reference image; and fuse the acquired multiple reference classification features to obtain fused reference classification features.

23. The device according to claim 15, characterized in that The first acquisition unit is configured to acquire reference classification features of the reference image, where the reference classification features include multiple feature values; and change feature values ​​in the reference classification features that are less than a reference threshold to 0 to obtain changed reference classification features.

24. The device according to claim 15, characterized in that The screening unit is configured to determine the candidate image corresponding to any candidate classification feature as the target image if the distance between any candidate classification feature and the reference classification feature is less than the reference distance.

25. The device according to claim 15, characterized in that The multiple candidate images are sample images used to train the first image classification model. The training unit is also configured to discard the target image screened out from the multiple candidate images and train the first image classification model based on the remaining candidate images so that the classification result determined by the trained first image classification model is no longer affected by the target object.

26. The device according to claim 15, characterized in that The training unit is further configured to perform training on a second image classification model according to the determined target image, wherein the second image classification model is used to perform processing according to the target object in any image.

27. The device according to claim 15, characterized in that There are multiple reference images, each of which includes a different target object. The training unit is also configured to execute training on multiple second image classification models respectively according to the target image corresponding to each reference image, and different second image classification models are used for processing according to different target objects.

28. The device according to claim 27, characterized in that The device also includes: A second acquisition unit is configured to acquire at least one target object; a selection unit configured to select a second image classification model matching the at least one target object from the plurality of second image classification models; The processing unit is configured to execute calling the selected at least one second image classification model to process any image and obtain at least one classification result of the image.

29. An electronic device, characterized in that: The electronic device comprises: one or more processors; volatile or non-volatile memory for storing the one or more processor-executable instructions; Wherein, the one or more processors are configured to execute the target image screening method as described in any one of claims 1-14.

30. A storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the target image screening method as described in any one of claims 1 to 14.

31. A computer program product comprising executable instructions, characterized in that: When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to perform the target image screening method as described in any one of claims 1 to 14.

Citation Information

Patent Citations

  • Remote sensing image designated building area detection method in hierarchical local structure constraint

    CN107622239A

  • Vehicle detection method and classification recognition model training method

    CN111126224A

  • KR20190021095A