Target detection model construction, image classification method, and electronic device

Abstract

This application discloses a method and apparatus, a computer-readable storage medium, and an electronic device for object detection model construction, image classification, image publishing, image search, and product object publishing. The method includes: obtaining an external image dataset as training samples, including sample images and the categories of sample objects contained in the sample images; obtaining an initial model for image classification, including a candidate region network and a classification network; inputting the sample images as input to the initial model to obtain multiple candidate region boxes determined by the candidate region network from the sample images; calculating the foreground score associated with each of the multiple candidate region boxes, including a first score determined by the candidate region network and a second score representing the degree of matching between the candidate region box and the categories included in the external image dataset; determining the region of interest from the multiple candidate region boxes based on the foreground score and inputting it into the classification network for image classification training to obtain an object detection model. This helps improve the model's generalization ability.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and in particular to a method and apparatus for constructing a target detection model, an image classification method and apparatus, an image publishing method and apparatus, an image search method and apparatus, a product object publishing method and apparatus, a computer-readable storage medium, and an electronic device. Background Technology

[0002] Object detection is a fundamental task in the field of computer vision. It uses machine learning techniques to identify and locate all objects of interest (or target objects) in an image and determine their category and location.

[0003] Typically, before training an object detection model, a target detection dataset associated with the recognition task needs to be defined. For example, a target detection dataset associated with a task to detect knife categories might include manually labeled data such as images related to knives, the location of the knife in the images, and its corresponding category. However, object detection models trained in this way have the following two problems:

[0004] 1. Object detection models lack the ability to recognize objects outside the training dataset. For example, the model trained above to identify knife categories cannot identify the category of an image that includes a flag.

[0005] 2. Object detection models are highly dependent on manually labeled data. When it is necessary to add or modify detection categories, the training dataset must be relabeled, and the model must be retrained using the relabeled dataset, resulting in high model training costs.

[0006] Open-Vocabulary Detection (OVD) can incorporate large-scale external image datasets, such as the Image-Caption dataset, to increase the model's ability to detect new categories (or unknown categories) by mining more semantic information about object nouns, without manually expanding the amount of labeled data. Summary of the Invention

[0007] This application provides a method and apparatus for constructing an object detection model, an image classification method and apparatus, an image publishing method and apparatus, an image search method and apparatus, a product object publishing method and apparatus, a computer-readable storage medium, and an electronic device, which help improve the generalization ability of the object detection model and thus improve the model's recognition accuracy for unknown categories.

[0008] This application provides the following solution:

[0009] A method for constructing an object detection model, comprising:

[0010] An external image dataset is obtained as training samples, the external image dataset including sample images and the categories of sample objects contained in the sample images;

[0011] An initial model for image classification is obtained, the initial model comprising at least a candidate region network and a classification network;

[0012] The sample image is used as input to the initial model to obtain multiple candidate region boxes determined by the candidate region network from the sample image;

[0013] Calculate the foreground score associated with each of the plurality of candidate region boxes, wherein the foreground score includes a first score determined by the candidate region network and a second score representing the degree of matching between the candidate region box and the categories included in the external image dataset;

[0014] Based on the foreground score, the region of interest is determined from the multiple candidate region boxes, and input into the classification network for image classification training to obtain the object detection model.

[0015] The second score is obtained in the following manner:

[0016] Obtain the visual features of the candidate region bounding boxes, and the text features of the categories included in the external image dataset;

[0017] The similarity between the visual features and the text features is calculated to obtain the second score.

[0018] The step of calculating the similarity between the visual features and the text features to obtain the second score includes:

[0019] The categories included in the external image dataset are clustered to obtain multiple cluster centers;

[0020] The similarity between the visual features and the text features of the multiple cluster centers is calculated to obtain the second score.

[0021] The external image dataset includes a first dataset and a second dataset; the categories of sample objects contained in the sample images in the first dataset include: the location of the sample object and the category label of the sample object; the categories of sample objects contained in the sample images in the second dataset include: the text description of the sample image, and the text description includes category phrases related to the category of the sample object;

[0022] The classification network is trained to classify images and obtain an object detection model, including: taking sample images from the first dataset and sample images from the second dataset as inputs to the initial model, and training the model alternately to obtain the object detection model.

[0023] Specifically, the object detection model is trained by using sample images from the first dataset as input to the initial model, including:

[0024] The known categories associated with the initial model are determined, and the reconstructed text features associated with each of the different known categories are obtained. The reconstructed text features are generated from the text features of the known categories and the text features of the positive sample categories associated with the known categories. The positive sample categories associated with the known categories belong to the category phrases included in the second dataset and are similar words to the associated known categories.

[0025] The visual features of the region of interest are matched with the reconstructed text features associated with the known categories to determine the predicted category associated with the region of interest from the known categories;

[0026] Based on the predicted categories associated with different regions of interest in the sample images, as well as the location and category labels of the sample objects contained in the sample images, the classification loss function and regression loss function are calculated, and the model is adjusted to obtain the object detection model.

[0027] Specifically, the object detection model is trained by using sample images from the second dataset as input to the initial model, including:

[0028] Obtain text features of available categories, wherein the available categories include the category phrase and the negative sample category associated with the category phrase, the negative sample category associated with the category phrase belongs to the categories included in the external image dataset, and the associated category phrase is a non-approximate word;

[0029] The visual features of the region of interest are matched with the text features of the available categories to determine the predicted category associated with the region of interest from the available categories;

[0030] The target detection model is obtained by adjusting the model through the calculation of a contrastive loss function.

[0031] The negative sample category associated with the category phrase is determined in the following manner:

[0032] Clustering is performed on the categories included in the external image dataset to obtain multiple classes;

[0033] Determine the negative sample category associated with the category phrase from other classes besides the target class to which the category phrase belongs.

[0034] An image classification method, comprising:

[0035] The image to be classified and the target category are obtained as input to the target detection model. The target detection model includes at least a candidate region network and a classification network. The candidate region network is used to determine multiple candidate region boxes from the image to be classified, and to determine the region of interest based on the foreground scores associated with each of the multiple candidate region boxes. The foreground score includes a first score determined by the candidate region network and a second score used to indicate the degree of matching between the candidate region box and the categories included in the external image dataset.

[0036] The classification result output by the target detection model is obtained. The classification result is used to indicate whether the image to be classified includes a target object with the target category. The classification result is determined by the classification network based on the matching result between the visual features of the region of interest and the text features of the target category.

[0037] The text features of the target category are reconstructed text features, which are obtained in the following manner:

[0038] A candidate category is determined from the categories included in the external image dataset, wherein the candidate category and the target category are similar words;

[0039] The text features of the target category are reconstructed using the text features of the category to be used, thereby obtaining the reconstructed text features.

[0040] An image classification method, comprising:

[0041] A conversation is established between the target detection model and the intelligent dialogue client, providing a conversation interface. The target detection model includes at least a candidate region network and a classification network.

[0042] The candidate region network obtains the image to be classified and the target category submitted in a dialog-based manner through the conversation interface, so that the candidate region network can determine multiple candidate region boxes from the image to be classified, and determine the region of interest based on the foreground score associated with each of the multiple candidate region boxes; the foreground score includes a first score determined by the candidate region network and a second score used to indicate the degree of matching between the candidate region box and the categories included in the external image dataset;

[0043] The classification result returned by the target detection model is obtained and displayed in the session interface. The classification result is used to indicate whether the image to be classified includes a target object with the target category. The classification result is determined by the classification network based on the matching result of the visual features of the region of interest and the text features of the target category.

[0044] A method for publishing images, including:

[0045] Obtain the image to be published submitted by the first client in the target system, and call the target detection model, which includes at least a candidate region network and a classification network;

[0046] The risk categories that need to be security detected are obtained. The image to be published and the risk categories are used as input to the target detection model so that the candidate region network can determine multiple candidate region boxes from the image to be published, and determine the region of interest based on the foreground scores associated with each of the multiple candidate region boxes. The foreground scores include a first score determined by the candidate region network and a second score used to indicate the degree of matching between the candidate region box and the categories included in the external image dataset.

[0047] The classification result output by the target detection model is obtained. When the classification result indicates that the image to be published does not contain a target object with the risk category, the image to be published is published. The classification result is determined by the classification network based on the matching result between the visual features of the region of interest and the text features of the risk category.

[0048] An image search method includes:

[0049] Obtain user viewing requests submitted by the second client;

[0050] From the published images associated with the target system, a target image matching the user's viewing needs is searched. The published image is one that, according to the target detection model, does not include target objects with a risk category. The candidate region network in the target detection model is used to determine multiple candidate region boxes from the published image, and to determine the region of interest based on the foreground scores associated with each of the multiple candidate region boxes. The foreground score includes a first score determined by the candidate region network and a second score representing the degree of matching between the candidate region box and the categories included in the external image dataset. The classification network in the target detection model is used to determine that the published image does not include target objects with the risk category based on the matching result between the visual features of the region of interest and the text features of the risk category.

[0051] A method for publishing a product object, comprising:

[0052] The system obtains a publishing request submitted by a third client in the product object service system, and calls the object detection model. The publishing request includes product description information and product images of the product object to be published. The object detection model includes at least a candidate region network and a classification network.

[0053] The target category of the risky product object is obtained, and the product image and the target category are used as input to the object detection model so that the candidate region network determines multiple candidate region boxes from the product image, and determines the region of interest based on the foreground score associated with each of the multiple candidate region boxes; the foreground score includes a first score determined by the candidate region network and a second score used to indicate the degree of matching between the candidate region box and the categories included in the external image dataset;

[0054] The classification result output by the target detection model is obtained. When the classification result indicates that the product image does not contain a target object with the target category, an online display page for the product object to be published is generated. The classification result is determined by the classification network based on the matching result between the visual features of the region of interest and the text features of the target category.

[0055] An apparatus for constructing a target detection model, comprising:

[0056] The training sample acquisition unit is used to acquire an external image dataset as training samples, wherein the external image dataset includes sample images and the categories of sample objects contained in the sample images;

[0057] An initial model acquisition unit is used to obtain an initial model for image classification, wherein the initial model includes at least a candidate region network and a classification network;

[0058] The candidate region box acquisition unit is used to take the sample image as input to the initial model and obtain multiple candidate region boxes determined by the candidate region network from the sample image.

[0059] A foreground score calculation unit is used to calculate the foreground score associated with each of the plurality of candidate region boxes. The foreground score includes a first score determined by the candidate region network and a second score used to represent the degree of matching between the candidate region box and the categories included in the external image dataset.

[0060] The image classification training unit is used to determine the region of interest from the multiple candidate region boxes based on the foreground score, input the region into the classification network for image classification training, and obtain the target detection model.

[0061] An image classification device, comprising:

[0062] The model input acquisition unit is used to obtain the image to be classified and the target category as input to the object detection model. The object detection model includes at least a candidate region network and a classification network. The candidate region network is used to determine multiple candidate region boxes from the image to be classified and to determine the region of interest based on the foreground scores associated with each of the multiple candidate region boxes. The foreground score includes a first score determined by the candidate region network and a second score representing the degree of matching between the candidate region box and the categories included in the external image dataset.

[0063] The classification result acquisition unit is used to obtain the classification result output by the target detection model. The classification result is used to indicate whether the image to be classified includes a target object with the target category. The classification result is determined by the classification network based on the matching result between the visual features of the region of interest and the text features of the target category.

[0064] An image classification device, applied to an intelligent dialogue client, the device comprising:

[0065] A session interface providing unit is used to establish a session with the object detection model and provide a session interface, wherein the object detection model includes at least a candidate region network and a classification network;

[0066] The image and category acquisition unit is used to obtain the image to be classified and the target category submitted in a dialog manner through the conversation interface, so that the candidate region network can determine multiple candidate region boxes from the image to be classified, and determine the region of interest based on the foreground score associated with each of the multiple candidate region boxes; the foreground score includes a first score determined by the candidate region network and a second score used to indicate the degree of matching between the candidate region box and the categories included in the external image dataset;

[0067] The classification result acquisition unit is used to obtain the classification result returned by the target detection model and display it in the session interface. The classification result is used to indicate whether the image to be classified includes a target object with the target category. The classification result is determined by the classification network based on the matching result of the visual features of the region of interest and the text features of the target category.

[0068] An image publishing device, comprising:

[0069] The model invocation unit is used to obtain the image to be published submitted by the first client in the target system and invoke the target detection model, which includes at least a candidate region network and a classification network.

[0070] A risk category acquisition unit is used to acquire risk categories that require security detection. The image to be published and the risk categories are used as inputs to the target detection model, so that the candidate region network determines multiple candidate region boxes from the image to be published, and determines the region of interest based on the foreground scores associated with each of the multiple candidate region boxes. The foreground scores include a first score determined by the candidate region network and a second score used to represent the degree of matching between the candidate region boxes and the categories included in the external image dataset.

[0071] The classification result acquisition unit is used to obtain the classification result output by the target detection model. When the classification result indicates that the image to be published does not contain a target object with the risk category, the image to be published is published. The classification result is determined by the classification network based on the matching result between the visual features of the region of interest and the text features of the risk category.

[0072] An image search device, comprising:

[0073] The User View Request Acquisition Unit is used to obtain user view requests submitted by the second client.

[0074] The target image search unit is used to search for target images that match the user's viewing needs from the published images associated with the target system. The published images are those that, according to the target detection model, do not include target objects with a risk category. The candidate region network in the target detection model is used to determine multiple candidate region boxes from the published images and to determine regions of interest based on the foreground scores associated with each of the multiple candidate region boxes. The foreground scores include a first score determined by the candidate region network and a second score representing the degree of matching between the candidate region boxes and the categories included in the external image dataset. The classification network in the target detection model is used to determine that the published images do not include target objects with the risk category based on the matching results between the visual features of the regions of interest and the text features of the risk category.

[0075] A product object publishing device, comprising:

[0076] The model invocation unit is used to obtain the publishing request submitted by the third client in the commodity object service system, and invoke the target detection model. The publishing request includes the commodity description information and commodity image of the commodity object to be published. The target detection model includes at least a candidate region network and a classification network.

[0077] The target category acquisition unit is used to obtain the target category of the risky product object. It takes the product image and the target category as input to the target detection model, so that the candidate region network can determine multiple candidate region boxes from the product image and determine the region of interest based on the foreground scores associated with each of the multiple candidate region boxes. The foreground score includes a first score determined by the candidate region network and a second score used to indicate the degree of matching between the candidate region box and the categories included in the external image dataset.

[0078] The classification result acquisition unit is used to obtain the classification result output by the target detection model. When the classification result indicates that the product image does not include a target object with the target category, the unit generates an online display page for the product object to be published. The classification result is determined by the classification network based on the matching result between the visual features of the region of interest and the text features of the target category.

[0079] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the preceding methods.

[0080] An electronic device, comprising:

[0081] One or more processors; and

[0082] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of any of the preceding methods.

[0083] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0084] In this embodiment, it was found that the candidate region network of the object detection model suffers from overfitting when determining the region of interest, which weakens the model's generalization ability for unknown categories. Therefore, a targeted approach is proposed to match the candidate region box with more categories included in the external image dataset and obtain a second score representing the degree of matching. The first score determined by the candidate region network is then combined with the first score to serve as the foreground score of the candidate region box, thereby determining the region of interest. This approach can solve the problem of the candidate region network overfitting to known categories, help improve the generalization ability of the object detection model, and thus improve the model's recognition accuracy for unknown categories.

[0085] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0086] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0087] Figure 1 This is a flowchart of the target detection model construction method provided in the embodiments of this application;

[0088] Figure 2 This is a schematic diagram of the target detection model provided in the embodiments of this application;

[0089] Figure 3 This is a flowchart of the image classification method provided in the embodiments of this application;

[0090] Figure 4 This is a flowchart of the image publishing method provided in the embodiments of this application;

[0091] Figure 5 This is a flowchart of the product object publishing method provided in the embodiments of this application;

[0092] Figure 6 This is a schematic diagram of the target detection model construction device provided in the embodiments of this application;

[0093] Figure 7 This is a schematic diagram of the image classification device provided in the embodiments of this application;

[0094] Figure 8 This is a schematic diagram of the image publishing device provided in the embodiments of this application;

[0095] Figure 9 This is a schematic diagram of the product object publishing device provided in the embodiments of this application;

[0096] Figure 10 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0097] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0098] The core idea of ​​open vocabulary object detection models is to train on visible (base class) data to achieve the recognition and detection of unseen / target data.

[0099] In related technologies, external image datasets can be introduced, and labeled data for sample images contained in the external image datasets can be automatically generated. Based on this, an open-vocabulary object detection model can be trained to optimize the model's generalization ability. This automatically generated labeled data can be called pseudo-labels.

[0100] As an example, pseudo-labels can be generated using a VLP (Vision-Language Pre-training) model. Taking the Image-Caption dataset as an example, sample images and their corresponding text descriptions from the dataset can be input into the VLP model. The model outputs automatically generated regions of interest (ROIs, representing the location information of target objects) and category labels (representing the category of target objects located within ROIs), which serve as pseudo-label data for the sample images and are used for model training.

[0101] While the approach of optimizing model generalization ability through pseudo-labeled data can enable the model to identify unknown categories, the actual experimental results show that the model's accuracy in identifying unknown categories is still relatively low.

[0102] The inventors discovered through analysis that this is mainly because the Region Proposal Network (RPN) in the object detection model is trained on a detection dataset with known class labels. This causes the RPN to overfit to known classes when determining regions of interest. In other words, the RPN easily classifies candidate region boxes of known classes as foreground (i.e., regions of interest) and candidate region boxes of unknown classes as background, thus weakening the model's ability to generalize to unknown classes.

[0103] Correspondingly, this application provides the following target detection model construction scheme, which helps to further improve the generalization ability of open vocabulary target detection models, thereby improving the model's recognition accuracy for unknown categories.

[0104] As an example, the target detection model construction method of this application embodiment can be as follows: Figure 1 As shown, it includes:

[0105] S101: Obtain an external image dataset as training samples, wherein the external image dataset includes sample images and the categories of sample objects contained in the sample images.

[0106] As an example, an external image dataset can be represented as at least one of the following two types of datasets:

[0107] 1. An image description dataset, which may include sample images and corresponding text descriptions. The text descriptions may include category phrases related to the categories of the sample objects contained in the sample images. In other words, the categories of the sample objects contained in the sample images can be specifically represented by category phrases in the text descriptions.

[0108] 2. The detection dataset, which may include sample images and their corresponding label data. The label data may include the location and category labels of the sample objects contained in the sample images. In other words, the category of the sample objects contained in the sample images can be specifically represented by the label data corresponding to those objects.

[0109] S102: Obtain an initial model for image classification, the initial model including at least a candidate region network and a classification network.

[0110] In practical applications, object detection models may include an image encoder for extracting visual features from an image and a text encoder for extracting text features of a category. This application's embodiments primarily focus on optimizing the network of the image encoder portion; therefore, the encoding process of the text encoder is briefly described, with specific implementation details available in related technologies.

[0111] For the image encoder part, such as Figure 2 As shown, it can include at least a backbone network, candidate region networks, pooled fully connected layers, and classification networks.

[0112] S103: Using the sample image as input to the initial model, obtain multiple candidate region boxes determined by the candidate region network from the sample image.

[0113] The implementation process of RPN in generating candidate region boxes can be found in related technologies, and this application does not limit it.

[0114] S104: Calculate the foreground score associated with each of the plurality of candidate region boxes, wherein the foreground score includes a first score determined by the candidate region network and a second score representing the degree of matching between the candidate region box and the categories included in the external image dataset.

[0115] In this embodiment, to address the overfitting problem in RPN's determination of the region of interest, a foreground score for each candidate region box can be calculated by combining a second score O2, which represents the degree of matching between the candidate region box and more categories included in the external image dataset, with the first score O1 determined by the RPN for each candidate region box. The implementation process for the RPN to determine the first score can be found in related technologies, and this embodiment does not limit it.

[0116] Specifically, the second score of the candidate region can be calculated as follows: obtain the visual features of the candidate region box and the text features of the categories included in the external image dataset; calculate the similarity between the visual features and the text features to obtain the second score.

[0117] As an example, the visual features of the candidate region box can be represented in the form of vision embedding, and the text features of the categories included in the external image dataset can be represented in the form of text embedding. By performing cross-modal alignment between the visual features of the candidate region box and the text features of each category, the similarity between the candidate region box and each category is calculated, which serves as a second score representing the degree of matching between the two.

[0118] For example, an external image dataset includes P categories, and the text features of each category can be represented as V1, V2, ..., V... P For candidate region boxes determined from sample images, the similarity between their visual features and text features can be represented as S1, S2, ..., S... P In one implementation, the second score of the candidate region box can be represented as the average of P similarities:

[0119]

[0120] Methods for calculating the similarity between visual features and text features can refer to related technologies, and the embodiments of this application do not limit this to any particular method. For example, S i =V i *f T , where f T It is a transpose of visual features.

[0121] Furthermore, it should be noted that to optimize the model's generalization ability, a large-scale external image dataset is typically chosen, meaning the external image dataset includes a large number of categories. To reduce computational complexity, the P categories included in the external image dataset can be clustered first to obtain multiple cluster centers, for example, K cluster centers. Then, the similarity between the visual features of the candidate region box and the text features of the K cluster centers is calculated to obtain the second score corresponding to the candidate region box. Here, K < P.

[0122] Clustering P categories can be understood as clustering based on the text features of the P categories. The implementation process of clustering can be found in related technologies. This application does not limit the clustering algorithm or the number of clusters, but can be determined based on actual usage requirements.

[0123] In this implementation, the second score of the candidate region box can be represented as the average of K similarities:

[0124]

[0125] In this example, S i Specifically, this can be reflected in the similarity between the visual features of the candidate region box and the text features of the i-th cluster center.

[0126] As an example, for the M candidate region boxes extracted from sample image 1, the foreground score of the j-th candidate region box can be expressed as: O j =O j1 +O j2 Among them, O j1 O is the first score determined by RPN for the j-th candidate region box. j2 The second score is the score of the j-th candidate region box determined according to the embodiment of this application.

[0127] Understandably, in practical applications, the weights of the two scores can be set according to usage requirements, and the foreground score can be calculated by weighted summation. This application embodiment does not limit the specific implementation method of calculating the foreground score based on the first score and the second score.

[0128] S105: Based on the foreground score, determine the region of interest from the multiple candidate region boxes, input it into the classification network for image classification training, and obtain the target detection model.

[0129] Taking the M candidate region boxes extracted from sample image 1 as an example, the foreground scores associated with each candidate region box can be compared with a preset threshold. Candidate region boxes with foreground scores not less than the preset threshold are identified as regions of interest. The region features of the region of interest are then extracted by a pooling fully connected layer and then input into a classification network for image classification training to obtain the object detection model.

[0130] In summary, compared to existing related technologies that only divide the foreground and background based on the score determined by the RPN (i.e., the first score in this embodiment), which leads to the problem of classifying candidate region boxes of unknown categories as background, this embodiment can match candidate region boxes with more external categories, rather than being limited to the known categories used during RPN training. This solves the problem of RPN overfitting to known categories and helps to further improve the generalization ability of the object detection model to unknown categories.

[0131] Furthermore, considering that the classification network is also trained on a detection dataset with known category labels, the inventors' experimental analysis revealed an overfitting problem in the image classification process. This is mainly manifested in the classification and recognition results overfitting to the detection dataset with known category labels. In other words, the classification network has a high accuracy rate in classifying and recognizing target objects in the training detection dataset, but a low accuracy rate, or even no recognition ability, for target objects in other datasets.

[0132] Correspondingly, embodiments of this application can also provide the following preferred solution: Approximate words of known categories can be found among more categories included in the external image dataset, serving as positive sample categories associated with the known category; then, the text features of the positive sample categories are used to correct the text features of the known categories, so that the corrected reconstructed text features can contain richer semantic information. In this way, when the classification network uses the reconstructed text features for image-text matching, it helps to improve the matching probability between the visual features of the region of interest and the reconstructed text features of the known categories, thereby solving the problem of overfitting to the training detection dataset.

[0133] As an example, considering that the categories included in the detection dataset are more fixed and clear, this embodiment of the application preferably uses the category labels in the detection dataset as the positive sample categories corresponding to the known categories for model training.

[0134] The following example illustrates the process of training a model based on a detection dataset.

[0135] First, the known categories associated with the initial model can be determined, and positive sample category expansion can be performed.

[0136] The known categories associated with the initial model are some or all of the category labels included in the detection dataset used during the initial model training, which can be determined based on the actual training situation.

[0137] After identifying the known categories, corresponding approximate words can be selected from the category phrases included in the second dataset, i.e., the image description dataset, as the associated positive sample categories for each known category. For example, the similarity between the text features of the known category and the text features of the category phrases can be calculated, and the category phrases with the highest similarity can be determined as the positive sample categories associated with that known category; or, the category phrases with a similarity not lower than a preset similarity threshold can be determined as the positive sample categories associated with that known category. This application embodiment does not limit the method of determining positive sample categories, the number of positive sample categories, etc., and can be determined according to actual usage requirements.

[0138] In one implementation, the text features of a known category and the text features of positive sample categories associated with the known category can be weighted and calculated to obtain the reconstructed features of the known category.

[0139] Figure 2 In the diagram shown, the text features of known categories can be represented as C1, C2, ..., C C The reconstructed text features after feature reconstruction can be represented as C1′, C2′, ..., C1′. C Compared to existing related technologies that rely solely on the matching degree between text features of known categories and visual features of regions of interest (ROIs) for image classification, the reconstructed features of known categories in this application can express richer semantic information, which helps to improve the matching probability with the visual features of ROIs, thereby identifying the category of ROIs, solving the problem of model overfitting to the training detection dataset, and improving the model's generalization ability.

[0140] Secondly, the visual features of the region of interest are matched with the reconstructed text features associated with known categories, and the predicted category associated with the region of interest is determined from the known categories.

[0141] Specifically, the similarity between the visual features of the region of interest and the reconstructed text features of each known category can be calculated as the classification score of the region of interest under that known category, and then the known category with the highest classification score can be determined as the predicted category of the region of interest.

[0142] For example, if N regions of interest are extracted from sample image 1, the corresponding visual features can be represented as R1, R2, ..., R... N The classification scores of different regions of interest under different known categories can be as follows: Figure 2 As shown, S gh ′ represents the similarity between the visual features of the g-th region of interest and the reconstructed text features of the h-th known category.

[0143] Finally, the loss function can be calculated, and the model can be adjusted based on the loss function to obtain the object detection model.

[0144] After obtaining the predicted categories from the model output, the loss function can be calculated, and the model can be adjusted accordingly. This involves adjusting the network parameters of the image encoder part of the model to ultimately obtain a target detection function that meets the usage requirements.

[0145] As an example, the classification loss function Cls_loss (to represent class loss) and the regression loss function Reg_loss (to represent location loss) can be calculated based on the predicted categories associated with different regions of interest in the sample image, as well as the location and category labels of the sample objects contained in the sample image.

[0146] In summary, the object detection model trained in this way can solve the problem of the model overfitting to the training detection dataset, and help improve the model's classification and recognition performance on different datasets, that is, improve the model's recognition accuracy on other datasets.

[0147] Optionally, embodiments of this application may also utilize two types of datasets—detection datasets and image description datasets—for alternating training to further optimize model performance and improve the model's recognition accuracy.

[0148] For example, when training a model using a detection dataset, sample images from that dataset can be used as input to the initial model, according to... Figure 1 The method shown identifies the region of interest from the sample image and inputs it into the classification network. As described above, the classification network can reconstruct text features based on known categories, identify the preset category of the region of interest, and adjust the model based on the classification loss function and regression loss function to optimize model performance.

[0149] When training a model using the image description dataset, namely the second dataset in this embodiment, sample images from this dataset can be obtained as input to the initial model. Similarly, the model can be trained according to... Figure 1 The method shown identifies regions of interest from sample images and inputs them into a classification network.

[0150] Correspondingly, the classification network can be trained as follows:

[0151] First, the category phrases included in the image description dataset can be determined, and negative sample category expansion can be performed.

[0152] Once the category phrases are identified, corresponding non-approximate words can be selected from the categories included in the external image dataset for each category phrase as its associated negative sample category. The external image dataset used for selecting negative sample categories can be a second dataset, i.e., selecting negative sample categories from the category phrases included in the image description dataset; or it can be a second dataset combined with other datasets. Taking another dataset as the first dataset as an example, negative sample categories can be selected from the category phrases included in the image description dataset and the category labels included in the detection dataset.

[0153] As an example, by calculating the text feature similarity, a preset number of categories with the lowest similarity, or categories with similarity below a preset similarity threshold, can be identified as negative sample categories.

[0154] As another example, negative sample categories can be filtered through clustering. Specifically, the categories included in the external image dataset can be clustered to obtain multiple classes; the negative sample categories associated with the category phrase can be determined from other classes besides the target class to which the category phrase belongs. In practical applications, the cluster centers of other classes can be used to determine the negative sample categories, or a subset of the categories included in other classes can be selected as negative sample categories. This application does not limit the number of negative sample categories.

[0155] In this example, the category phrase and the negative sample category associated with the category phrase can be used as the available categories to obtain the text features of the available categories for image-text matching. Figure 2 In the diagram shown, the text features of the category phrases included in the image description dataset can be represented as W1, W2, W3, ..., and the specific number can be determined according to the image description dataset used.

[0156] Next, the visual features of the region of interest are matched with the text features of the available categories to determine the predicted category associated with the region of interest from the available categories.

[0157] Specifically, the similarity between the visual features of the region of interest and the text features of each available category can be calculated as the classification score of the region of interest under that available category, and then the available category with the highest classification score can be determined as the predicted category of the region of interest.

[0158] Finally, the loss function can be calculated, and the model can be adjusted based on the loss function to obtain the object detection model.

[0159] After obtaining the predicted categories from the model output, the loss function can be calculated, and the model can be adjusted accordingly. This involves adjusting the network parameters of the image encoder part of the model to ultimately obtain a target detection function that meets the usage requirements.

[0160] As an example, by calculating the contrastive loss function, matching image-text pairs can be brought closer together, while unmatched pairs can be pushed further away, resulting in a higher-performing object detection model. Training the model with more negative sample classes and experiments demonstrate that contrastive loss can improve the model's recognition performance.

[0161] In summary, object detection models can perceive more positive and negative sample categories in the feature space, thereby obtaining a more reasonable optimization direction.

[0162] As an example, embodiments of this application may also provide an image classification method, see [link to relevant documentation]. Figure 3 The flowchart shown may include:

[0163] S301: Obtain the image to be classified and the target category as input to the target detection model. The target detection model includes at least a candidate region network and a classification network. The candidate region network is used to determine multiple candidate region boxes from the image to be classified and to determine the region of interest based on the foreground scores associated with each of the multiple candidate region boxes. The foreground scores include a first score determined by the candidate region network and a second score used to indicate the degree of matching between the candidate region box and the categories included in the external image dataset.

[0164] S302: Obtain the classification result output by the target detection model. The classification result is used to indicate whether the image to be classified includes a target object with the target category. The classification result is determined by the classification network based on the matching result between the visual features of the region of interest and the text features of the target category.

[0165] In practical applications, the image to be classified and the target category can be input into the target detection model. The backbone network of the target detection model can extract the feature map of the image to be classified and input it into the RPN. The RPN determines multiple candidate region boxes and the first score of each candidate region box from the image to be classified. Then, combined with the second score of each candidate region box calculated according to the embodiment of this application, the foreground score associated with each of the multiple candidate region boxes is obtained. The foreground score is compared with a preset threshold to determine the region of interest. The region of interest is then input into the pooling fully connected layer to extract the regional features of the region of interest, i.e., the visual features of the region of interest, and input into the classification network.

[0166] In addition, the text encoder of the object detection model can encode the object category, and the text features of the object category are input into the classification network.

[0167] In this way, the classification network can calculate the similarity between the visual features of the region of interest and the text features of the target category. If the similarity is not lower than the preset score, it can be determined that the region of interest includes the target object with the target category.

[0168] As an example, the classification results output by an object detection model can include whether the image to be classified contains a target object with the target category. If the image to be classified contains a target object with the target category, the classification results can further include: the location of the target object (i.e., the coordinate information of the region of interest), and the category label of the target object (especially when there are multiple target categories, the category label corresponding to each target object can be marked in the classification results).

[0169] Understandably, the text features of the target category used for similarity calculation can be either the text features of the target category itself or the reconstructed text features of the target category. That is, from the categories included in the external image dataset, candidate categories that can serve as approximate words for the target category can be selected. The text features of the candidate categories are then used to reconstruct the text features of the target category itself, thus obtaining the reconstructed text features of the target category. For example, the reconstructed features of the target category can be obtained by weighting the text features of the target category itself and the text features of the candidate categories.

[0170] As an example, the above image classification function can be implemented through intelligent dialogue. Specifically, a conversation can be established between an intelligent dialogue client and an object detection model, providing a conversational interface. The object detection model includes at least a candidate region network and a classification network. Through the conversational interface, the user submits the image to be classified and the target category in a conversational manner. The candidate region network then determines multiple candidate region boxes from the image to be classified and determines the region of interest based on the foreground scores associated with each of the multiple candidate region boxes. The foreground score includes a first score determined by the candidate region network and a second score representing the degree of matching between the candidate region box and the categories included in the external image dataset. The classification result returned by the object detection model is obtained and displayed in the conversational interface. The classification result indicates whether the image to be classified includes a target object with the target category. The classification result is determined by the classification network based on the matching result between the visual features of the region of interest and the text features of the target category.

[0171] In other words, users can invoke the object detection model through conversation in the chat interface and instruct the object detection model to perform image classification processing on the image to be classified, and determine whether it includes target objects with the target category, which helps to improve the user's interactive experience.

[0172] For example, when a user wants to filter images from multiple pictures to find those belonging to a target category, they can enter "Please find photos containing the sun from album 1" in the chat interface. The object detection model can identify all images in album 1 as images to be classified, identify the sun as the target category, perform image-text matching, determine the classification result for each image, and return a response to the user, indicating whether there are photos containing the sun in album 1. If such photos exist, the model can further display the photo IDs for the user to view in the chat interface. This improves the user experience and also enhances the efficiency and accuracy of image classification.

[0173] As an example, the target detection model provided in this application embodiment can be applied to the field of information security to realize the review and interception of inappropriate images.

[0174] With the continuous development of information technology, a massive amount of user-generated content has emerged online, including some inappropriate images. If these inappropriate images are not identified in a timely manner, they may lead to a poor user experience and may even result in the account being shut down if they violate relevant laws and regulations.

[0175] To ensure user experience and prevent malicious users from posting potentially harmful images on the target system, thereby harming its users, the target system's security measures can be implemented to detect and identify inappropriate images posted within the system. Specifically, the system's security measures can invoke target detection models as needed to perform targeted detection on images to be posted, determining whether they contain risky objects, thus serving as a defense against malicious attacks.

[0176] See Figure 4 The flowchart shown illustrates the image publishing method provided in this application embodiment, which may include:

[0177] S401: Obtain the image to be published submitted by the first client in the target system, and call the object detection model, which includes at least a candidate region network and a classification network.

[0178] S402: Obtain the risk category that needs to be security detected, and use the image to be published and the risk category as input to the target detection model, so that the candidate region network determines multiple candidate region boxes from the image to be published, and determines the region of interest based on the foreground score associated with each of the multiple candidate region boxes; the foreground score includes a first score determined by the candidate region network and a second score used to indicate the degree of matching between the candidate region box and the categories included in the external image dataset.

[0179] S403: Obtain the classification result output by the target detection model. When the classification result indicates that the image to be published does not contain a target object with the risk category, publish the image to be published. The classification result is determined by the classification network based on the matching result between the visual features of the region of interest and the text features of the risk category.

[0180] In this example, the first client can be deployed on the terminal device associated with the image publisher, either as a webpage or as a standalone application.

[0181] The first client provides a publishing page for submitting images to be published, where image publishers can upload and publish images. Specifically, when an image publisher clicks the publish button, the currently uploaded image is identified as the image to be published, and an object detection model is invoked to perform a security check on the image to be published. Simultaneously, the risk category associated with the target system can be determined. The image to be published and the risk category are used as input to the object detection model, which identifies whether the image to be published contains a target object with a risk category. If the image to be published does not contain a target object with a risk category, it can be determined that the image to be published has passed the security check and can be published on the target system.

[0182] In this way, images published to the target system can be specifically inspected, which helps ensure that the images published by the image publisher meet the compliance requirements of the target system and avoids the publication of inappropriate images that may harm the users of the target system.

[0183] Understandably, the risk category associated with the target system can be a known category used in model training or a newly added unknown category. For unknown categories, text features of the unknown category can be obtained by encoding or reconstructing text features through the text encoder of the object detection model, and then aligned with the visual features of the identified region of interest in the image to be published to determine whether the image to be published includes a target object with that unknown category.

[0184] Correspondingly, embodiments of this application may also provide an image search method, which may include: obtaining a user viewing request submitted by a second client; searching for a target image matching the user viewing request from published images associated with the target system, wherein the published image is an image that, as determined by an object detection model, does not include target objects with a risk category; a candidate region network in the object detection model is used to determine multiple candidate region boxes from the published image, and to determine a region of interest based on the foreground scores associated with each of the multiple candidate region boxes; the foreground score includes a first score determined by the candidate region network and a second score used to represent the degree of matching between the candidate region box and the categories included in the external image dataset; a classification network in the object detection model is used to determine that the published image does not include target objects with the risk category based on the matching result between the visual features of the region of interest and the text features of the risk category.

[0185] The second client can be deployed on the terminal device associated with the image viewer, either as a webpage or a standalone application. This ensures that users of the target system are viewing images that have undergone security checks, helping to mitigate the negative impact of inappropriate images on users and improving their user experience on the target system.

[0186] The target system performs security detection based on the target detection model provided in this application embodiment, which can improve the accuracy and efficiency of identifying malicious images, thereby enhancing the defense capabilities of the target system's prevention and control chain. Especially when a new risk category appears, the solution in this application embodiment can directly encode the new risk category into text, enabling image-text matching to identify whether the image to be published contains a target object with the new risk type. This eliminates the need for the traditional approach of first determining and manually labeling samples corresponding to the new risk category, and then retraining the model using the labeled samples to enable the model to identify the new risk category. Retraining the model not only involves a long risk window period, during which malicious images related to the new risk category may be published to the target system, negatively impacting users, but also may result in a small number of samples corresponding to the new risk category, making it impossible to guarantee the model's recognition accuracy even after retraining.

[0187] As an example, the target detection model provided in this application embodiment can be applied to a commodity object service system to review and intercept the online publication of risky commodity objects.

[0188] See Figure 5 The flowchart shown illustrates the product object publishing method provided in this application embodiment, which may include:

[0189] S501: Obtain the publishing request submitted by the third client in the product object service system, and call the object detection model. The publishing request includes product description information and product images of the product object to be published. The object detection model includes at least a candidate region network and a classification network.

[0190] S502: Obtain the target category of the risky product object, and use the product image and the target category as input to the object detection model, so that the candidate region network determines multiple candidate region boxes from the product image, and determines the region of interest based on the foreground score associated with each of the multiple candidate region boxes; the foreground score includes a first score determined by the candidate region network and a second score used to represent the degree of matching between the candidate region box and the categories included in the external image dataset.

[0191] S503: Obtain the classification result output by the target detection model. When the classification result indicates that the product image does not include a target object with the target category, generate an online display page for the product object to be published. The classification result is determined by the classification network based on the matching result between the visual features of the region of interest and the text features of the target category.

[0192] In this example, the third client can be deployed on the terminal device associated with the product object publisher, existing as a webpage or a standalone application. As an example, the third client can be a client of the product object service system.

[0193] The third client can provide a product publishing page where publishers can submit information related to the product to be published, such as product details and images. Specifically, when the publisher clicks the publish button, the uploaded product image on the page can be identified as the image to be published, and an object detection model can be invoked to perform security checks on the image to be published.

[0194] Furthermore, the target categories of risky product objects associated with the product object service system can be determined. The target categories of risky product objects requiring security testing can be pre-determined. The product image and the target category are used as input to the target detection model, which identifies whether the product image includes a target object with the target category. If the product image does not contain a target object with the target category, the product image passes the security test. If all other information about the product object passes the security test, it can be published online in the product object service system for consumers to shop online.

[0195] Understandably, when a new category of risky goods appears (this could be due to the appearance of a new risky goods object leading to a new category, or it could be that the risky goods object has not changed but a new category has appeared), the text features of the new category can be obtained by encoding the text encoder of the object detection model or the text features can be reconstructed for image-text matching. For the specific implementation process, please refer to the above introduction, and no further examples will be given here.

[0196] In this way, targeted testing can be performed on the products published online by the product object service system, which helps to ensure that the published products meet the system's compliance requirements, thereby guaranteeing the shopping experience for consumers.

[0197] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0198] Corresponding to the foregoing method embodiments, this application also provides a target detection model construction apparatus, see [link to relevant documentation]. Figure 6 The device may include:

[0199] The training sample acquisition unit 601 is used to acquire an external image dataset as training samples, wherein the external image dataset includes sample images and the categories of sample objects contained in the sample images;

[0200] The initial model acquisition unit 602 is used to obtain an initial model for image classification, the initial model including at least a candidate region network and a classification network;

[0201] The candidate region box obtaining unit 603 is used to take the sample image as input to the initial model and obtain multiple candidate region boxes determined by the candidate region network from the sample image.

[0202] The foreground score calculation unit 604 is used to calculate the foreground score associated with each of the plurality of candidate region boxes. The foreground score includes a first score determined by the candidate region network and a second score used to represent the degree of matching between the candidate region box and the categories included in the external image dataset.

[0203] The image classification training unit 605 is used to determine the region of interest from the plurality of candidate region boxes based on the foreground score, input the region of interest into the classification network for image classification training, and obtain a target detection model.

[0204] Specifically, the foreground score calculation unit can be used to: obtain the visual features of the candidate region box and the text features of the categories included in the external image dataset; calculate the similarity between the visual features and the text features to obtain the second score.

[0205] Specifically, the foreground score calculation unit can be used to: perform clustering processing on the categories included in the external image dataset to obtain multiple cluster centers; calculate the similarity between the visual features and the text features of the multiple cluster centers to obtain the second score.

[0206] The external image dataset includes a first dataset and a second dataset; the categories of sample objects contained in the sample images in the first dataset include: the location of the sample object and the category label of the sample object; the categories of sample objects contained in the sample images in the second dataset include: the text description of the sample image, and the text description includes category phrases related to the category of the sample object;

[0207] The image classification training unit can be specifically used to: take sample images from the first dataset and sample images from the second dataset as inputs to the initial model, and alternately train the model to obtain the object detection model.

[0208] Specifically, the image classification training unit uses sample images from the first dataset as input to the initial model for training.

[0209] The known categories associated with the initial model are determined, and the reconstructed text features associated with each of the different known categories are obtained. The reconstructed text features are generated from the text features of the known categories and the text features of the positive sample categories associated with the known categories. The positive sample categories associated with the known categories belong to the category phrases included in the second dataset and are similar words to the associated known categories.

[0210] The visual features of the region of interest are matched with the reconstructed text features associated with the known categories to determine the predicted category associated with the region of interest from the known categories;

[0211] Based on the predicted categories associated with different regions of interest in the sample images, as well as the location and category labels of the sample objects contained in the sample images, the classification loss function and regression loss function are calculated, and the model is adjusted to obtain the object detection model.

[0212] Specifically, the sample images from the second dataset are used as input to the initial model for model training. The image classification training unit can be used for:

[0213] Obtain text features of available categories, wherein the available categories include the category phrase and the negative sample category associated with the category phrase, the negative sample category associated with the category phrase belongs to the categories included in the external image dataset, and the associated category phrase is a non-approximate word;

[0214] The visual features of the region of interest are matched with the text features of the available categories to determine the predicted category associated with the region of interest from the available categories;

[0215] The target detection model is obtained by adjusting the model through the calculation of a contrastive loss function.

[0216] Specifically, the image classification training unit can be used to determine the negative sample category associated with the category phrase in the following manner:

[0217] Clustering is performed on the categories included in the external image dataset to obtain multiple classes;

[0218] Determine the negative sample category associated with the category phrase from other classes besides the target class to which the category phrase belongs.

[0219] Corresponding to the foregoing method embodiments, this application also provides an image classification device, see [link to relevant documentation]. Figure 7 The device may include:

[0220] The model input acquisition unit 701 is used to acquire the image to be classified and the target category as input to the object detection model. The object detection model includes at least a candidate region network and a classification network. The candidate region network is used to determine multiple candidate region boxes from the image to be classified and to determine the region of interest based on the foreground scores associated with each of the multiple candidate region boxes. The foreground score includes a first score determined by the candidate region network and a second score representing the degree of matching between the candidate region box and the categories included in the external image dataset.

[0221] The classification result acquisition unit 702 is used to obtain the classification result output by the target detection model. The classification result is used to indicate whether the image to be classified includes a target object with the target category. The classification result is determined by the classification network based on the matching result between the visual features of the region of interest and the text features of the target category.

[0222] The text features of the target category are reconstructed text features, which are obtained in the following manner:

[0223] A candidate category is determined from the categories included in the external image dataset, wherein the candidate category and the target category are similar words;

[0224] The text features of the target category are reconstructed using the text features of the category to be used, thereby obtaining the reconstructed text features.

[0225] Corresponding to the foregoing method embodiments, this application also provides an image classification device applied to an intelligent dialogue client, which may include:

[0226] A session interface providing unit is used to establish a session with the object detection model and provide a session interface, wherein the object detection model includes at least a candidate region network and a classification network;

[0227] The image and category acquisition unit is used to obtain the image to be classified and the target category submitted in a dialog manner through the conversation interface, so that the candidate region network can determine multiple candidate region boxes from the image to be classified, and determine the region of interest based on the foreground score associated with each of the multiple candidate region boxes; the foreground score includes a first score determined by the candidate region network and a second score used to indicate the degree of matching between the candidate region box and the categories included in the external image dataset;

[0228] The classification result acquisition unit is used to obtain the classification result returned by the target detection model and display it in the session interface. The classification result is used to indicate whether the image to be classified includes a target object with the target category. The classification result is determined by the classification network based on the matching result of the visual features of the region of interest and the text features of the target category.

[0229] Corresponding to the foregoing method embodiments, this application also provides an image publishing device, see below. Figure 8 The device may include:

[0230] The model invocation unit 801 is used to obtain the image to be published submitted by the first client in the target system and invoke the target detection model, wherein the target detection model includes at least a candidate region network and a classification network;

[0231] The risk category acquisition unit 802 is used to acquire the risk category that needs to be detected for security purposes. The image to be published and the risk category are used as input to the target detection model so that the candidate region network can determine multiple candidate region boxes from the image to be published and determine the region of interest based on the foreground score associated with each of the multiple candidate region boxes. The foreground score includes a first score determined by the candidate region network and a second score used to indicate the degree of matching between the candidate region box and the categories included in the external image dataset.

[0232] The classification result acquisition unit 803 is used to obtain the classification result output by the target detection model. When the classification result indicates that the image to be published does not contain a target object with the risk category, the image to be published is published. The classification result is determined by the classification network based on the matching result between the visual features of the region of interest and the text features of the risk category.

[0233] Corresponding to the foregoing method embodiments, this application also provides an image search device, which may include:

[0234] The User View Request Acquisition Unit is used to obtain user view requests submitted by the second client.

[0235] The target image search unit is used to search for target images that match the user's viewing needs from the published images associated with the target system. The published images are those that, according to the target detection model, do not include target objects with a risk category. The candidate region network in the target detection model is used to determine multiple candidate region boxes from the published images and to determine regions of interest based on the foreground scores associated with each of the multiple candidate region boxes. The foreground scores include a first score determined by the candidate region network and a second score representing the degree of matching between the candidate region boxes and the categories included in the external image dataset. The classification network in the target detection model is used to determine that the published images do not include target objects with the risk category based on the matching results between the visual features of the regions of interest and the text features of the risk category.

[0236] Corresponding to the foregoing method embodiments, this application also provides a product object publishing device, see [link to relevant documentation]. Figure 9 The device may include:

[0237] The model invocation unit 901 is used to obtain the publishing request submitted by the third client in the commodity object service system, and invoke the target detection model. The publishing request includes the commodity description information and commodity image of the commodity object to be published. The target detection model includes at least a candidate region network and a classification network.

[0238] The target category acquisition unit 902 is used to obtain the target category of the risky product object. The product image and the target category are used as input to the target detection model so that the candidate region network can determine multiple candidate region boxes from the product image and determine the region of interest based on the foreground scores associated with each of the multiple candidate region boxes. The foreground scores include a first score determined by the candidate region network and a second score used to indicate the degree of matching between the candidate region box and the categories included in the external image dataset.

[0239] The classification result acquisition unit 903 is used to obtain the classification result output by the target detection model. When the classification result indicates that the product image does not include a target object with the target category, an online display page for the product object to be published is generated. The classification result is determined by the classification network based on the matching result between the visual features of the region of interest and the text features of the target category.

[0240] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0241] And an electronic device, comprising:

[0242] One or more processors; and

[0243] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0244] in, Figure 10 The architecture of an electronic device is illustrated by example. For instance, device 1000 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, aircraft, etc.

[0245] Reference Figure 10 The device 1000 may include one or more of the following components: processing component 1002, memory 1004, power supply component 1006, multimedia component 1008, audio component 1010, input / output (I / O) interface 1012, sensor component 1014, and communication component 1016.

[0246] Processing component 1002 typically controls the overall operation of device 1000, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1002 may include one or more processors 1020 to execute instructions to perform all or part of the steps of the methods provided in this disclosure. Furthermore, processing component 1002 may include one or more modules to facilitate interaction between processing component 1002 and other components. For example, processing component 1002 may include a multimedia module to facilitate interaction between multimedia component 1008 and processing component 1002.

[0247] Memory 1004 is configured to store various types of data to support the operation of device 1000. Examples of this data include instructions for any application or method operating on device 1000, contact data, phonebook data, messages, pictures, videos, etc. Memory 1004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0248] Power supply component 1006 provides power to various components of device 1000. Power supply component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 1000.

[0249] Multimedia component 1008 includes a screen that provides an output interface between device 1000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1008 includes a front-facing camera and / or a rear-facing camera. When device 1000 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0250] Audio component 1010 is configured to output and / or input audio signals. For example, audio component 1010 includes a microphone (MIC) configured to receive external audio signals when device 1000 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1004 or transmitted via communication component 1016. In some embodiments, audio component 1010 also includes a speaker for outputting audio signals.

[0251] Input / output (I / O) interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0252] Sensor assembly 1014 includes one or more sensors for providing state assessments of various aspects of device 1000. For example, sensor assembly 1014 may detect the on / off state of device 1000, the relative positioning of components such as the display and keypad of device 1000, changes in the position of device 1000 or a component of device 1000, the presence or absence of user contact with device 1000, the orientation or acceleration / deceleration of device 1000, and temperature changes of device 1000. Sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1014 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1014 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0253] Communication component 1016 is configured to facilitate wired or wireless communication between device 1000 and other devices. Device 1000 can access wireless networks based on communication standards, such as WiFi, or mobile communication networks such as 2G, 3G, 4G / LTE, and 5G. In one exemplary embodiment, communication component 1016 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1016 further includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0254] In an exemplary embodiment, device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0255] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, which can be executed by a processor 1020 of device 1000 to perform the method provided by the present disclosure. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0256] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0257] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0258] The above provides a detailed description of the target detection model-related schemes provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for constructing a target detection model, characterized in that, include: An external image dataset is obtained as training samples, the external image dataset including sample images and the categories of sample objects contained in the sample images; An initial model for image classification is obtained, the initial model comprising at least a candidate region network and a classification network; The sample image is used as input to the initial model to obtain multiple candidate region boxes determined by the candidate region network from the sample image; Calculate the foreground score associated with each of the plurality of candidate region boxes, wherein the foreground score includes a first score determined by the candidate region network and a second score representing the degree of matching between the candidate region box and the categories included in the external image dataset; Based on the foreground score, a region of interest is determined from the multiple candidate region boxes, and input into the classification network for image classification training to obtain a target detection model; The second score is obtained in the following manner: Obtain the visual features of the candidate region bounding boxes, and the text features of the categories included in the external image dataset; The similarity between the visual features and the text features is calculated to obtain the second score.

2. The method according to claim 1, characterized in that, The step of calculating the similarity between the visual features and the text features to obtain the second score includes: The categories included in the external image dataset are clustered to obtain multiple cluster centers; The similarity between the visual features and the text features of the multiple cluster centers is calculated to obtain the second score.

3. The method according to claim 1 or 2, characterized in that, The external image dataset includes a first dataset and a second dataset; the categories of sample objects contained in the sample images in the first dataset include: the location of the sample object and the category label of the sample object; the categories of sample objects contained in the sample images in the second dataset include: the text description of the sample image, and the text description includes category phrases related to the category of the sample object; The classification network is trained to classify images and obtain an object detection model, including: taking sample images from the first dataset and sample images from the second dataset as inputs to the initial model, and training the model alternately to obtain the object detection model.

4. The method according to claim 3, characterized in that, The object detection model is obtained by using sample images from the first dataset as input to the initial model for training, including: The known categories associated with the initial model are determined, and the reconstructed text features associated with each of the different known categories are obtained. The reconstructed text features are generated from the text features of the known categories and the text features of the positive sample categories associated with the known categories. The positive sample categories associated with the known categories belong to the category phrases included in the second dataset and are similar words to the associated known categories. The visual features of the region of interest are matched with the reconstructed text features associated with the known categories to determine the predicted category associated with the region of interest from the known categories; Based on the predicted categories associated with different regions of interest in the sample images, as well as the location and category labels of the sample objects contained in the sample images, the classification loss function and regression loss function are calculated, and the model is adjusted to obtain the object detection model.

5. The method according to claim 3, characterized in that, The object detection model is obtained by using sample images from the second dataset as input to the initial model for training, including: Obtain text features of available categories, wherein the available categories include the category phrase and the negative sample category associated with the category phrase, the negative sample category associated with the category phrase belongs to the categories included in the external image dataset, and the associated category phrase is a non-approximate word; The visual features of the region of interest are matched with the text features of the available categories to determine the predicted category associated with the region of interest from the available categories; The target detection model is obtained by adjusting the model through the calculation of a contrastive loss function.

6. The method according to claim 5, characterized in that, The negative sample category associated with the category phrase is determined as follows: Clustering is performed on the categories included in the external image dataset to obtain multiple classes; Determine the negative sample category associated with the category phrase from other classes besides the target class to which the category phrase belongs.

7. An image classification method, characterized in that, include: The image to be classified and the target category are obtained as input to the object detection model. The object detection model includes at least a candidate region network and a classification network. The candidate region network is used to determine multiple candidate region boxes from the image to be classified, and to determine the region of interest based on the foreground scores associated with each of the multiple candidate region boxes. The foreground score includes a first score determined by the candidate region network and a second score used to indicate the degree of matching between the candidate region box and the categories included in the external image dataset. The classification result output by the target detection model is obtained. The classification result is used to indicate whether the image to be classified includes a target object with the target category. The classification result is determined by the classification network based on the matching result between the visual features of the region of interest and the text features of the target category. The second score is obtained in the following manner: Obtain the visual features of the candidate region bounding boxes, and the text features of the categories included in the external image dataset; The similarity between the visual features and the text features is calculated to obtain the second score.

8. The method according to claim 7, characterized in that, The text features of the target category are reconstructed text features, which are obtained in the following manner: A candidate category is determined from the categories included in the external image dataset, wherein the candidate category and the target category are similar words; The text features of the target category are reconstructed using the text features of the category to be used, thereby obtaining the reconstructed text features.

9. An image classification method, characterized in that, include: A conversation is established between the target detection model and the intelligent dialogue client, providing a conversation interface. The target detection model includes at least a candidate region network and a classification network. The candidate region network obtains the image to be classified and the target category submitted in a dialog-based manner through the dialogue interface, so that the candidate region network can determine multiple candidate region boxes from the image to be classified, and determine the region of interest based on the foreground score associated with each of the multiple candidate region boxes; the foreground score includes a first score determined by the candidate region network and a second score used to indicate the degree of matching between the candidate region box and the categories included in the external image dataset; The classification result returned by the target detection model is obtained and displayed in the session interface. The classification result is used to indicate whether the image to be classified includes a target object with the target category. The classification result is determined by the classification network based on the matching result of the visual features of the region of interest and the text features of the target category. The second score is obtained in the following manner: Obtain the visual features of the candidate region bounding boxes, and the text features of the categories included in the external image dataset; The similarity between the visual features and the text features is calculated to obtain the second score.

10. A method for publishing images, characterized in that, include: Obtain the image to be published submitted by the first client in the target system, and call the target detection model, which includes at least a candidate region network and a classification network; The risk categories that need to be security detected are obtained. The image to be published and the risk categories are used as input to the target detection model so that the candidate region network can determine multiple candidate region boxes from the image to be published, and determine the region of interest based on the foreground scores associated with each of the multiple candidate region boxes. The foreground scores include a first score determined by the candidate region network and a second score used to indicate the degree of matching between the candidate region box and the categories included in the external image dataset. The classification result output by the target detection model is obtained. When the classification result indicates that the image to be published does not contain a target object with the risk category, the image to be published is published. The classification result is determined by the classification network based on the matching result between the visual features of the region of interest and the text features of the risk category. The second score is obtained in the following manner: Obtain the visual features of the candidate region bounding boxes, and the text features of the categories included in the external image dataset; The similarity between the visual features and the text features is calculated to obtain the second score.

11. An image search method, characterized in that, include: Obtain user viewing requests submitted by the second client; From the published images associated with the target system, find the target image that matches the user's viewing needs. The published image is an image that is determined by the target detection model not to include target objects with risk categories. The candidate region network in the object detection model is used to determine multiple candidate region boxes from the published image, and to determine the region of interest based on the foreground scores associated with each of the multiple candidate region boxes; the foreground scores include a first score determined by the candidate region network and a second score used to indicate the degree of matching between the candidate region box and the categories included in the external image dataset; The classification network in the target detection model is used to determine that the published image does not contain a target object with the risk category based on the matching result between the visual features of the region of interest and the text features of the risk category; The second score is obtained in the following manner: Obtain the visual features of the candidate region bounding boxes, and the text features of the categories included in the external image dataset; The similarity between the visual features and the text features is calculated to obtain the second score.

12. A method for publishing product objects, characterized in that, include: The system obtains a publishing request submitted by a third client in the product object service system, and calls the object detection model. The publishing request includes product description information and product images of the product object to be published. The object detection model includes at least a candidate region network and a classification network. The target category of the risky product object is obtained, and the product image and the target category are used as input to the object detection model so that the candidate region network determines multiple candidate region boxes from the product image, and determines the region of interest based on the foreground score associated with each of the multiple candidate region boxes; the foreground score includes a first score determined by the candidate region network and a second score used to indicate the degree of matching between the candidate region box and the categories included in the external image dataset; The classification result output by the target detection model is obtained. When the classification result indicates that the product image does not contain a target object with the target category, an online display page for the product object to be published is generated. The classification result is determined by the classification network based on the matching result between the visual features of the region of interest and the text features of the target category. The second score is obtained in the following manner: Obtain the visual features of the candidate region bounding boxes, and the text features of the categories included in the external image dataset; The similarity between the visual features and the text features is calculated to obtain the second score.

13. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 12.

Images