Category recognition method, apparatus, device, storage medium, and program product

Abstract

Embodiments of the present application provide a category identification method, device, equipment, storage medium and program product, the method comprises: obtaining a picture sample set and a text sample set, the picture samples in the picture sample set and the text samples in the text sample set have different association relationships; training an identification model set based on the picture sample set and the text sample set; inputting a to-be-identified picture and set category information into the trained identification model to obtain a matching picture of the category information in the to-be-identified picture. The present scheme can greatly save time cost and labor cost, and improve the category identification efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a category identification method, apparatus, device, storage medium, and program product. Background Technology

[0002] Currently, the mainstream implementation of visual recognition systems used in large-scale image review technology is based on deep learning. This method requires accumulating a large amount of data to ensure the model has good generalization ability and practical application value. It requires obtaining the desired positive samples from a large amount of data; when applied to large-scale review data, this often requires hundreds of thousands or even tens of thousands of accumulated positive samples. For category recognition in some training data that is not clearly defined, related technologies alleviate data shortages by manually collecting more data of the desired category, while simultaneously training a separate visual recognition model. This approach enables effective recognition of new categories.

[0003] The above approach, which involves retraining the recognition model, requires significant investment of human and time resources, such as a large amount of time spent on labeling. This is especially true when the proportion of positive samples is extremely small (less than one in a million), requiring the labeling of millions or even tens of millions of data points to obtain a sufficient number of positive samples. In practical applications, this approach is very costly in terms of both time and manpower and needs improvement. Summary of the Invention

[0004] This application provides a category identification method, apparatus, device, storage medium, and program product, which can greatly save time and labor costs and improve category identification efficiency.

[0005] In a first aspect, embodiments of this application provide a category identification method, the method comprising:

[0006] Obtain an image sample set and a text sample set, wherein the image samples in the image sample set and the text samples in the text sample set have different association relationships;

[0007] The established recognition model is trained based on the image sample set and the text sample set;

[0008] The image to be identified and the set category information are input into the trained recognition model to obtain the matching image of the category information in the image to be identified.

[0009] Secondly, embodiments of this application also provide a category identification device, including:

[0010] The acquisition module is configured to acquire an image sample set and a text sample set, wherein the image samples in the image sample set and the text samples in the text sample set have different association relationships.

[0011] The training module is configured to train the set recognition model based on the image sample set and the text sample set.

[0012] The recognition module is configured to input the image to be recognized and the set category information into the trained recognition model to obtain the matching image of the category information in the image to be recognized.

[0013] Thirdly, embodiments of this application also provide a category identification device, the device comprising:

[0014] One or more processors;

[0015] Storage device for storing one or more programs.

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the category recognition method described in the embodiments of this application.

[0017] Fourthly, embodiments of this application also provide a non-volatile storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the category identification method described in embodiments of this application.

[0018] Fifthly, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium, wherein at least one processor of the device reads from the computer-readable storage medium and executes the computer program, causing the device to perform the category identification method described in embodiments of this application.

[0019] In this embodiment, an image sample set and a text sample set are obtained, wherein the image samples in the image sample set and the text samples in the text sample set have different association relationships. A set recognition model is trained based on the image sample set and the text sample set. The image to be recognized and the set category information are input into the trained recognition model to obtain the matching image of the category information in the image to be recognized. In the above-described category recognition method, the recognition model trained using image and text training data determines the matching image corresponding to the set category method, eliminating the need for separate model training for specific categories. Furthermore, this solution eliminates the need for sample annotation during the training process of the recognition model, saving significant time and manpower costs. The model also boasts high recognition accuracy, is more closely aligned with actual usage scenarios, and has greater versatility. Attached Figure Description

[0020] Figure 1 A flowchart illustrating a category identification method provided in this application embodiment;

[0021] Figure 2 A flowchart illustrating a recognition model training method provided in this application embodiment;

[0022] Figure 3 This is a schematic diagram of a network structure in a recognition model provided in an embodiment of this application;

[0023] Figure 4 A flowchart illustrating a method for model training based on generated associations, provided in an embodiment of this application;

[0024] Figure 5 A flowchart illustrating another category identification method provided in this application embodiment;

[0025] Figure 6 A structural block diagram of a category identification device provided in an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the structure of a category identification device provided in an embodiment of this application. Detailed Implementation

[0027] The embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this application and are not intended to limit the scope of the embodiments. Furthermore, it should be noted that, for ease of description, only the parts relevant to the embodiments of this application are shown in the accompanying drawings, not the entire structure.

[0028] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0029] The category recognition method provided in this application can be executed by a computing device, such as a laptop, desktop computer, or smart terminal device. This category recognition method can be applied to the review process of videos and images in the live streaming industry, providing matching images for newly added categories.

[0030] Figure 1 A flowchart of a category identification method provided in an embodiment of this application is shown below. Figure 1 As shown, the specific steps include the following:

[0031] Step S101: Obtain an image sample set and a text sample set, wherein the image samples in the image sample set and the text samples in the text sample set have different association relationships.

[0032] In one embodiment, before training the recognition model, a stored image sample set and a text sample set are acquired. The image sample set includes multiple images, and the text sample set includes multiple texts. The text in the text sample set can be natural language descriptive text composed of words and sentences. The image samples in the image sample set and the text samples in the text sample set have different association relationships. Optionally, this association relationship can be a binary classification of associated or unassociated relationships.

[0033] In one embodiment, before storing the image sample set and text sample set, the process further includes acquiring images and text via the network and generating associations between the images and text. Optionally, this could involve: acquiring images and text descriptions from website information, generating image sample sets and text sample sets based on the images and text descriptions, and establishing associations between image samples in the image sample set and text samples in the text sample set. The website information could be information from a live streaming platform's online forum, which includes images and corresponding text. Specifically, images from the forum are collected online and stored as image samples in the image sample set, while text is stored as text samples in the text sample set. When generating associations between image and text samples, associations can be automatically generated based on the specific source of the images and text. For example, images and text from the same source are associated, while images and text from different sources are not associated. The criteria for determining the same or different sources can be set by the developers. For example, in the case of online forum information, text and images in the same post can be identified as having the same source, while text and images in different posts can be identified as having different sources. The image and text sample sets generated using the methods described above can be generated automatically without the need for sample annotation.

[0034] The sample generation method described above starts with a batch of initial data. Each item in this data contains at least an image and brief text information. Although this data lacks explicit labels, it is massive in quantity and very common, such as posts in live streams and on social media platforms, all of which contain images and brief text descriptions. Using this as samples for subsequent model training significantly improves the efficiency of obtaining samples, and it eliminates the need for labeling, enabling efficient information recognition even under data scarcity. It also addresses the shortcomings of related technologies that rely on pre-defining possible labels or entity sets. The main drawback of these technologies is that while pre-defined label attribute information can achieve recognition of zero-shot samples to some extent, this approach often assumes that different labels or categories have similar underlying attribute information. For example, different birds have different colors, head, and belly attributes. This assumption is too strict in practical recognition systems and requires experts to pre-define the attribute vectors corresponding to different category labels, making the design process very time-consuming. Furthermore, collecting a certain number of samples requires significant time and manpower, thus greatly reducing its practicality.

[0035] Step S102: Train the set recognition model based on the image sample set and the text sample set.

[0036] In one embodiment, after obtaining an image sample set and a text sample set, the configured recognition model is trained based on these sets. Optionally, the configured network model includes an image encoding network and a text encoding network. Optional training methods include... Figure 2 As shown, Figure 2 A flowchart of a recognition model training method provided in this application embodiment includes:

[0037] Step S1021: Perform image standardization processing on the image samples in the image sample set to obtain standard images, and perform text standardization processing on the text in the text sample set to obtain standard text.

[0038] In one embodiment, before inputting images and text into the image encoding network and text encoding network respectively, image samples are normalized to obtain standard images, and text samples are normalized to obtain standard text. The resulting standard images can be tensor matrices of a preset size, and the standard text can be matrices of a preset dimension. For example, a standard image can be a 224*224*3 tensor matrix, and standard text can be a 76*768 dimension matrix. The specific methods for image and text normalization can be implemented using corresponding function calls or programming language code, such as using the `imread(filename)` function for image normalization, or using Python to write code for TXT text normalization. Optionally, during image normalization, for RGB images, the image can be converted into a 2D matrix by subtracting the mean and variance of the three channels, or the image can be converted into a 1D vector.

[0039] Step S1022: Input the standard image and the standard text into the set image encoding network and text encoding network respectively to obtain the image vector corresponding to the standard image and the text vector corresponding to the standard text.

[0040] After obtaining the standard image and standard text, the standard image is input into an image encoding network being trained, and the standard text is input into a text encoding network being trained, to obtain image vectors and text vectors respectively. For example, a 512-dimensional image vector E can be obtained. I (I)∈R 512 And the 512-dimensional text vector E T (T)∈R 512 This standardization process aligns the massive amounts of images and text acquired for subsequent model training. This enables multimodal model training and subsequent recognition even under data-scarce conditions.

[0041] The image encoding network and text encoding network can be Transformer networks, or they can adopt visual network structures based on RNN or CNN.

[0042] Optionally, the image encoding network and text encoding network in this solution are specifically configured to use a Transformer network architecture, including a self-attention module, a residual neural network module, and a feedforward network module. This network structure can be a 12-layer self-attention and residual neural network, such as... Figure 3 As shown, Figure 3This diagram illustrates a network structure in a recognition model provided in this application embodiment. It includes a multi-head self-attention module and a short-cut residual link normalization module for information extraction, and also includes a learnable feedforward network. These components form an Encoder-Block. For example, the input and output of each Encoder-Block are fixed as a 768-dimensional matrix, maintaining the same dimensionality. The Transformer network structure contains 12 cascaded Encoder-Blocks as described above. After extracting high-level semantic information from images and text, this cascaded module connects to a fully connected layer and outputs image vectors / text vectors.

[0043] Step S1023: Calculate the similarity between the image vector and the text vector, and train the image encoding network and the text encoding network based on the association between the image samples and the text samples to obtain the trained recognition model.

[0044] In one embodiment, the similarity between image vectors and text vectors can be calculated either by Euclidean distance or by cosine distance; this approach is not limited to either method. Optionally, during the training of the recognition model, such as... Figure 4 As shown, Figure 4 A flowchart illustrating a method for model training based on generated associations, provided in this application embodiment, includes:

[0045] Step S10231: Calculate the similarity between the image vector and the text vector using the set similarity calculation formula.

[0046] Step S10232: Calculate the loss value based on the similarity and the association between the image sample and the text sample.

[0047] Taking the aforementioned association relationship, which includes both associated and unassociated relationships, as an example, the sample label value corresponding to association is 1, and that of unassociated relationships is 0. Specifically, for image vector E... I (I), and text vector E T For (T), the similarity calculation result between the two labels y is denoted as Sim(I,T), and the loss value can be calculated as follows:

[0048] L(Sim(I,T),y)=-ylog(Sim(I,T))

[0049] Step S10233: Adjust the network parameters of the image encoding network and the text encoding network based on the loss value to obtain the trained recognition model.

[0050] By using different image and text inputs and calculating the loss value, the network parameters of the image encoding network and the text encoding network are continuously adjusted. After convergence and obtaining the optimal network parameters, the weights are no longer updated, and the encoding network with fixed network parameters is used as the feature extraction model for subsequent category recognition.

[0051] Step S103: Input the image to be identified and the set category information into the trained recognition model to obtain the matching image of the category information in the image to be identified.

[0052] The image to be identified can be, for example, a screenshot from a live stream generated during the live broadcast for review purposes, or it can be any other set of images used to find matching images within a category. The category information can be a newly set category for which a matching image needs to be found. For example, if the previously entered category was volleyball, the newly set category could be beach volleyball or beach soccer. The recognition model trained as described above can then use images with high matching scores corresponding to this category information as the matching images.

[0053] As described above, by acquiring image and text sample sets, where the image samples in the image sample set and the text samples in the text sample set have different correlations, the established recognition model is trained based on these sets. The image to be recognized and the set category information are then input into the trained recognition model to obtain the matching image for the category information within the image to be recognized. In this category recognition method, the recognition model trained using image and text training data determines the matching image corresponding to the set category, eliminating the need for separate model training for specific categories. Furthermore, this solution eliminates the need for sample annotation during the training process, saving significant time and manpower costs. The model also boasts high recognition accuracy, is more closely aligned with real-world applications, and has greater versatility.

[0054] Figure 5 A flowchart of another category identification method provided in the embodiments of this application is shown below. Figure 5 As shown, it includes:

[0055] Step S201: Obtain an image sample set and a text sample set, wherein the image samples in the image sample set and the text samples in the text sample set have different association relationships.

[0056] Step S202: Train the set recognition model based on the image sample set and the text sample set.

[0057] Step S203: Input the image to be identified and the set category information into the trained recognition model to obtain the similarity value between the category information and each image in the image to be identified, and determine the image corresponding to the similarity value that meets the set similarity conditions as the matching image of the category information.

[0058] In one embodiment, the category information can be a brief descriptive information that meets the category requirements, without strictly limiting the text to the specifications. This category information can be a category that has never been explicitly defined before. By inputting the image to be identified and the set category information into the trained recognition model, the image vector of each image in the image to be identified and the text vector corresponding to the category information can be obtained. The similarity between the category information and each image is obtained by calculating the similarity between the text vector and each image vector. Images with similarity values ​​greater than a set threshold are considered the matching images corresponding to the category information, i.e., directly used as the recognition result.

[0059] As described above, by acquiring image and text sample sets, where the image samples in the image sample set and the text samples in the text sample set have different correlations, the established recognition model is trained based on these sets. The image to be recognized and the set category information are then input into the trained recognition model to obtain the matching image for the category information within the image to be recognized. In this category recognition method, the recognition model trained using image and text training data determines the matching image corresponding to the set category, eliminating the need for separate model training for specific categories. Furthermore, this solution eliminates the need for sample annotation during the training process, saving significant time and manpower costs. The model also boasts high recognition accuracy, is more closely aligned with real-world applications, and has greater versatility.

[0060] In the aforementioned category recognition scheme, within the scenario of a multimodal visual recognition system under data scarcity, a large number of target category images can be obtained by defining text labels or text descriptions of target data, i.e., category information. This significantly reduces the time and manpower costs of data collection and annotation. In this data-scarce multimodal visual recognition system, by utilizing massive amounts of image-text pairs from business scenarios, a multimodal model semantically aligned with text and image information can be obtained for subsequent efficient recognition. Simultaneously, matching recognition results can be obtained by describing the test sample set, avoiding the inefficient practice of developing additional algorithm models for recognition under limited categories. Text descriptions can be based on natural language or words, greatly reducing the time cost of refining category annotations, making it more relevant to user scenarios, more universal, and capable of quickly responding to dynamic needs such as standard changes in auditing operations.

[0061] Figure 6A structural block diagram of a category identification device provided in an embodiment of this application is shown below. Figure 6 As shown, this device is used to execute the category recognition method provided in the above embodiments, and has the corresponding functional modules and beneficial effects for executing the method. For example... Figure 6 As shown, the device specifically includes: an acquisition module 101, a training module 102, and a recognition module 103, wherein,

[0062] The acquisition module 101 is configured to acquire an image sample set and a text sample set, wherein the image samples in the image sample set and the text samples in the text sample set have different association relationships.

[0063] Training module 102 is configured to train the set recognition model based on the image sample set and the text sample set;

[0064] The recognition module 103 is configured to input the image to be recognized and the set category information into the trained recognition model to obtain the matching image of the category information in the image to be recognized.

[0065] As described above, by acquiring image and text sample sets, where the image samples in the image sample set and the text samples in the text sample set have different correlations, the established recognition model is trained based on these sets. The image to be recognized and the set category information are then input into the trained recognition model to obtain the matching image for the category information within the image to be recognized. In this category recognition method, the recognition model trained using image and text training data determines the matching image corresponding to the set category, eliminating the need for separate model training for specific categories. Furthermore, this solution eliminates the need for sample annotation during the training process, saving significant time and manpower costs. The model also boasts high recognition accuracy, is more closely aligned with real-world applications, and has greater versatility.

[0066] In one possible embodiment, the device further includes a sample generation module configured to:

[0067] Before obtaining the image sample set and text sample set, obtain the images and text description information from the website information;

[0068] Based on the images and the text description information, generate an image sample set and a text sample set, as well as the association relationship between the image samples in the image sample set and the text samples in the text sample set.

[0069] In one possible embodiment, the training module 102 is configured as follows:

[0070] Standard images are obtained by performing image standardization on the image samples in the image sample set, and standard text is obtained by performing text standardization on the text samples in the text sample set.

[0071] The standard image and the standard text are respectively input into the set image encoding network and text encoding network to obtain the image vector corresponding to the standard image and the text vector corresponding to the standard text;

[0072] The similarity between the image vector and the text vector is calculated, and the image encoding network and the text encoding network are trained based on the association between the image samples and the text samples to obtain the trained recognition model.

[0073] In one possible embodiment, the standard image comprises a tensor matrix of a preset size, and the standard text comprises a matrix of a preset dimension.

[0074] In one possible embodiment, the training module 102 is configured as follows:

[0075] The similarity between the image vector and the text vector is calculated using the set similarity calculation formula;

[0076] The loss value is obtained by calculating the loss based on the similarity and the association between the image sample and the text sample;

[0077] The network parameters of the image encoding network and the text encoding network are adjusted based on the loss value to obtain the trained recognition model.

[0078] In one possible embodiment, the image encoding network and the text encoding network include a self-attention module, a residual neural network module, and a feedforward network module.

[0079] In one possible embodiment, the identification module 103 is configured as follows:

[0080] The image to be identified and the set category information are input into the trained recognition model to obtain the similarity value between the category information and each image in the image to be identified;

[0081] Images whose similarity values ​​meet the set similarity conditions are identified as matching images for the category information.

[0082] Figure 7 This is a schematic diagram of the structure of a category identification device provided in an embodiment of this application, as shown below. Figure 7 As shown, the device includes a processor 201, a memory 202, an input device 203, and an output device 204; the number of processors 201 in the device can be one or more. Figure 7Taking a processor 201 as an example; the processor 201, memory 202, input device 203, and output device 204 in the device can be connected via a bus or other means. Figure 7 Taking a bus connection as an example, the memory 202, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the category recognition method in this embodiment. The processor 201 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 202, thereby implementing the aforementioned category recognition method. The input device 703 can be used to receive input digital or character information and generate key signal inputs related to user settings and function control of the device. The output device 204 may include a display device such as a screen.

[0083] This application also provides a non-volatile storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a category recognition method described in the above embodiments, wherein the method includes:

[0084] Obtain an image sample set and a text sample set, wherein the image samples in the image sample set and the text samples in the text sample set have different association relationships;

[0085] The established recognition model is trained based on the image sample set and the text sample set;

[0086] The image to be identified and the set category information are input into the trained recognition model to obtain the matching image of the category information in the image to be identified.

[0087] It is worth noting that in the embodiments of the above-mentioned category identification device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this application.

[0088] In some possible implementations, various aspects of the methods provided in this application can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of the methods according to the various exemplary embodiments of this application described above. For example, the computer device can perform the category identification method described in the embodiments of this application. The program product can be implemented using any combination of one or more readable media.

Claims

1. A category recognition method, characterized in that, include: Obtain an image sample set and a text sample set, wherein the image samples in the image sample set and the text samples in the text sample set have different association relationships, including association or non-association relationships determined by a dichotomy method. The recognition model is trained based on the image sample set and the text sample set. The recognition model includes an image encoding network and a text encoding network. The image encoding network and the text encoding network include a self-attention module, a residual neural network module, and a feedforward network module. The image to be identified and the set category information are input into the trained recognition model to obtain the matching image of the category information in the image to be identified.

2. The category identification method according to claim 1, characterized in that, Before obtaining the image sample set and text sample set, the following is also included: Retrieve images and text descriptions from website information; Based on the images and the text description information, generate an image sample set and a text sample set, as well as the association relationship between the image samples in the image sample set and the text samples in the text sample set.

3. The category identification method according to claim 1, characterized in that, The training of the recognition model based on the image sample set and the text sample set includes: Standard images are obtained by performing image standardization on the image samples in the image sample set, and standard text is obtained by performing text standardization on the text samples in the text sample set. The standard image and the standard text are respectively input into the set image encoding network and text encoding network to obtain the image vector corresponding to the standard image and the text vector corresponding to the standard text; The similarity between the image vector and the text vector is calculated, and the image encoding network and the text encoding network are trained based on the association between the image samples and the text samples to obtain the trained recognition model.

4. The category identification method according to claim 3, characterized in that, The standard image includes a tensor matrix of a preset size, and the standard text includes a matrix of a preset dimension.

5. The category identification method according to claim 3, characterized in that, The step of calculating the similarity between the image vector and the text vector, and training the image encoding network and the text encoding network based on the association between the image samples and the text samples to obtain the trained recognition model includes: The similarity between the image vector and the text vector is calculated using the set similarity calculation formula; The loss value is obtained by calculating the loss based on the similarity and the association between the image sample and the text sample; The network parameters of the image encoding network and the text encoding network are adjusted based on the loss value to obtain the trained recognition model.

6. The category identification method according to any one of claims 1-5, characterized in that, The step of inputting the image to be identified and the set category information into the trained recognition model to obtain the matching image of the category information in the image to be identified includes: The image to be identified and the set category information are input into the trained recognition model to obtain the similarity value between the category information and each image in the image to be identified; Images whose similarity values ​​meet the set similarity conditions are identified as matching images for the category information.

7. A category identification device, characterized in that, include: The acquisition module is configured to acquire an image sample set and a text sample set, wherein the image samples in the image sample set and the text samples in the text sample set have different association relationships, including association or non-association relationships determined by a dichotomy method. The training module is configured to train the set recognition model based on the image sample set and the text sample set. The recognition model includes an image encoding network and a text encoding network. The image encoding network and the text encoding network include a self-attention module, a residual neural network module, and a feedforward network module. The recognition module is configured to input the image to be recognized and the set category information into the trained recognition model to obtain the matching image of the category information in the image to be recognized.

8. A category recognition device, the device comprising: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the category recognition method according to any one of claims 1-6.

9. A non-volatile storage medium storing computer-executable instructions, which, when executed by a computer processor, are used to perform the category identification method according to any one of claims 1-6.

Images