Category identification method, apparatus, and device, storage medium, and program product

By using multimodal data of pictures and text to train the recognition model, the problem of high data and labor costs in large-scale image audits in the prior art is solved, and efficient and accurate category recognition is achieved without sample annotation.

WO2025051016A1PCT designated stage expired Publication Date: 2025-03-13BIGO TECH PTE LTD +1

Patent Information

Application Number
PCT/CN2024/114545
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-09-08
Filing Date
2024-08-26
Publication Date
2025-03-13

AI Technical Summary

Technical Problem

Existing deep learning methods require a lot of data and labor costs in large-scale image audits, especially when identifying undefined categories, the time and labor costs are very high.

Method used

By obtaining the image sample set and the text sample set, where the image sample has different associations with the text sample, the identification model is trained based on these data, without the need for model training for each category alone, and sample annotation is not required during the training process.

Benefits of technology

This method can significantly save time and labor costs, improve category recognition efficiency, and the model recognition accuracy is high, closer to the usage scenario, and more versatile.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024114545_13032025_PF_FP_ABST
    Figure CN2024114545_13032025_PF_FP_ABST
Patent Text Reader

Abstract

Embodiments of the present application provide a category identification method, apparatus, and device, a storage medium, and a program product. The method comprises: acquiring a picture sample set and a text sample set, picture samples in the picture sample set and text samples in the text sample set having different association relationships; on the basis of the picture sample set and the text sample set, training a configured identification model; and inputting pictures to be identified and configured category information into the trained identification model to obtain a matching picture of the category information among the pictures to be identified. According to the present solution, time costs and labor costs can be greatly saved, and category identification efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Category identification method, device, equipment, storage medium and program product

[0001] This application claims priority to the Chinese patent application No. 2023111600766 filed with the China Patent Office on September 8, 2023, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of computer technology, and in particular to a category identification method, apparatus, device, storage medium, and program product. Background Art

[0003] Currently, the mainstream implementation of visual recognition systems for large-scale image review technology is based on deep learning. This approach requires the accumulation of a large amount of data to ensure the model's good generalization ability and practical application value. It requires obtaining positive samples of interest from a large amount of data. When applied to large-scale review data, such positive samples often require hundreds of thousands or tens of thousands of accumulations. For the recognition of categories that are not clearly defined in the training data, related technologies alleviate data shortages by manually collecting more data of the relevant category, while further training a separate visual recognition model, thereby achieving effective recognition of new categories.

[0004] In the above solution, the method of retraining the recognition model requires a lot of manpower and time costs, such as a lot of labeling time costs. Especially when the proportion of positive samples of interest is extremely small (such as less than one in a million), obtaining a sufficient number of positive samples requires labeling millions or even tens of millions of data. In practical applications, the time and manpower costs are very high and need to be improved.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a category identification method, apparatus, device, storage medium, and program product, which can greatly save time and manpower costs and improve category identification efficiency.

[0007] In a first aspect, an embodiment of the present application provides a category identification method, the method comprising:

[0008] Acquire a picture sample set and a text sample set, wherein the picture samples in the picture sample set and the text samples in the text sample set have different association relationships;

[0009] Training the set recognition model based on the image sample set and the text sample set;

[0010] The image to be identified and the set category information are input into the trained recognition model to obtain a matching image of the category information in the image to be identified.

[0011] In a second aspect, an embodiment of the present application further provides a category identification device, including:

[0012] an acquisition module configured to acquire a picture sample set and a text sample set, wherein the picture samples in the picture sample set and the text samples in the text sample set have different association relationships;

[0013] A training module, configured to train a set recognition model based on the image sample set and the text sample set;

[0014] The recognition module is configured to input the image to be recognized and the set category information into the trained recognition model to obtain a matching image of the category information in the image to be recognized.

[0015] In a third aspect, an embodiment of the present application further provides a category identification device, the device comprising:

[0016] one or more processors;

[0017] a storage device for storing one or more programs,

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the category identification method described in the embodiment of the present application.

[0019] In a fourth aspect, an embodiment of the present application further provides a non-volatile storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are used to execute the category identification method described in the embodiment of the present application.

[0020] In a fifth aspect, an embodiment of the present application further provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads and executes the computer program from the computer-readable storage medium, so that the device performs the category identification method described in the embodiment of the present application.

[0021] In an embodiment of the present application, a picture sample set and a text sample set are obtained, wherein the picture samples in the picture sample set and the text samples in the text sample set have different associations, a set recognition model is trained based on the picture sample set and the text sample set, and the picture to be identified and the set category information are input into the trained recognition model to obtain a matching picture of the category information in the picture to be identified. In the above-mentioned category recognition method, the recognition model obtained by training based on the training data of pictures and texts is used to determine the matching picture corresponding to the set category method, without the need for separate model training for specific categories. At the same time, this solution does not require sample labeling during the training process of the recognition model, which can save a lot of time and labor costs, and the model has high recognition accuracy, is closer to the use scenario, and has stronger versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1 is a flow chart of a category identification method provided in an embodiment of the present application;

[0023] FIG2 is a flow chart of a recognition model training method provided in an embodiment of the present application;

[0024] FIG3 is a schematic diagram of a network structure in an identification model provided in an embodiment of the present application;

[0025] FIG4 is a flow chart of a method for performing model training based on generated association relationships provided in an embodiment of the present application;

[0026] FIG5 is a flow chart of another category identification method provided in an embodiment of the present application;

[0027] FIG6 is a structural block diagram of a category identification device provided in an embodiment of the present application;

[0028] FIG7 is a schematic structural diagram of a category identification device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following is a further detailed description of the embodiments of the present application in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the embodiments of the present application, and are not intended to limit the embodiments of the present application. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions of the embodiments of the present application, rather than all structures.

[0030] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0031] The category identification method provided in the embodiment of the present application can be executed by a computing device, such as a laptop computer, a desktop computer, and a smart terminal device. The category identification method can be applied to the review process of videos and pictures in the live broadcast industry, and can provide application scenarios for matching pictures for newly added categories.

[0032] FIG1 is a flow chart of a category identification method provided in an embodiment of the present application, as shown in FIG1 , which specifically includes the following steps:

[0033] Step S101: Acquire a picture sample set and a text sample set, wherein the picture samples in the picture sample set and the text samples in the text sample set have different association relationships.

[0034] In one embodiment, before training a recognition model, a stored image sample set and text sample set are obtained, wherein the image sample set includes multiple images, and the text sample set includes multiple texts. The text included in the text sample set can be natural language descriptive text composed of words and sentences. The image samples in the image sample set and the text samples in the text sample set have different associations. Optionally, the association can be a binary association or a non-association relationship.

[0035] In one embodiment, before storing the image sample set and text sample set, the process also includes obtaining images and text via the network and generating an association relationship between the images and text. Optionally, the process may include obtaining images and text description information from website information, generating an image sample set and a text sample set based on the images and text description information, and generating an association relationship between the image samples in the image sample set and the text samples in the text sample set. The website information may be Tieba information from a live streaming platform, which includes images and corresponding text. Specifically, by collecting information online, images from the Tieba are stored as image samples in the image sample set, and text samples are stored as text samples in the text sample set. When generating the association relationship between the image samples and text samples, the association relationship may be automatically generated based on the specific sources of the images and text. For example, an association relationship may be generated for images and text from the same source, while an association relationship may be generated for images and text from different sources. The criteria for determining the same source and different sources can be set by the developer. For example, in Tieba information from the website information, text and images in the same post may be determined to have the same source, while text and images in different posts may be determined to have different sources. By using the above-mentioned method for generating the image sample set and the text sample set, the sample set can be automatically generated without the need for sample labeling.

[0036] The above-mentioned sample generation method begins with a batch of initial data, each of which contains at least an image and a brief text message. Although this data is not clearly labeled, it is very large and common. For example, posts in live broadcast rooms and on social media platforms often contain images and brief text descriptions. Using this data as samples for subsequent model training significantly improves sample generation efficiency and eliminates the need for labeling, enabling efficient information recognition even in data-poor conditions. This method also addresses the shortcomings of related technologies that predefine possible labels or entity sets. While the attribute information of predefined labels can achieve a certain degree of recognition for zero-shot samples, this technique often assumes that different labels or categories share similar underlying attribute information. For example, different birds have different colors, heads, and abdomens. This assumption is too strict in practical recognition systems and requires experts to predefine the attribute vectors corresponding to different category labels. The design process is very time-consuming, and collecting a sufficient number of samples also requires significant time and labor costs, significantly reducing its practicality.

[0037] Step S102: training the set recognition model based on the image sample set and the text sample set.

[0038] In one embodiment, after obtaining a set of image sample sets and a set of text sample sets, a set recognition model is trained based on the image sample sets and the text sample sets. Optionally, the set network model includes an image encoding network and a text encoding network. An optional training method is shown in FIG2 , which is a flow chart of a recognition model training method provided in an embodiment of the present application, which includes:

[0039] Step S1021 : performing image standardization processing on the image samples in the image sample set to obtain standard images, and performing text standardization processing on the texts in the text sample set to obtain standard texts.

[0040] In one embodiment, before inputting the picture and text into the picture encoding network and the text encoding network respectively, the picture samples are subjected to picture standardization processing to obtain standard pictures, and the texts in the text sample set are subjected to text standardization processing to obtain standard texts. The processed standard pictures can be tensor matrices of preset sizes, and the standard texts can be matrices of preset dimensions. For example, the standard pictures can be tensor matrices of 224*224*3, and the standard texts can be matrices of 76*768 dimensions. The specific methods of picture standardization and text standardization can be to use the corresponding function functions or programming language codes to perform processing, such as the imread(filename) function to perform picture standardization processing, and the python programming language is used to write the txt text standardization processing code to implement text standardization processing. Optionally, when performing image standardization processing, for RGB images, the image can be converted into a 2-dimensional matrix by subtracting the mean and variance of the three channels of the image from the RGB image information, or the image can be converted into a 1-dimensional vector.

[0041] Step S1022: input the standard picture and the standard text into the set picture encoding network and text encoding network respectively to obtain the picture vector corresponding to the standard picture and the text vector corresponding to the standard text.

[0042] After obtaining the standard image and standard text, the standard image is input into the image encoding network being trained, and the standard text is input into the text encoding network being trained to obtain the image vector and text vector respectively. For example, a 512-dimensional image vector E can be obtained. I (I)∈R 512 , and the 512-dimensional text vector E T (T)∈R 512 Through this normalization process, the massive amount of images and text acquired can be aligned for subsequent model training, thereby achieving multimodal model training and subsequent recognition under data-deficient conditions.

[0043] Among them, the image encoding network and the text encoding network can be Transformer networks, or they can adopt visual network structures based on RNN or CNN.

[0044] Optionally, the specially set image encoding network and text encoding network of this solution adopt the Transformer network architecture, including a self-attention module, a residual neural network module and a forward network module. Among them, the network structure can be a 12-layer self-attention and residual neural network, as shown in Figure 3. Figure 3 is a schematic diagram of the network structure in a recognition model provided by an embodiment of the present application, wherein a multi-head self-attention module and a short-cut residual link normalization module are included for information extraction, and a learnable forward network is also included. The above components constitute an Encoder-Block, and each Encoder-Block information input and information output are exemplarily fixed to a matrix output of 768 dimensions, keeping the dimension unchanged. The Transformer network structure includes the 12 Encoder-Blocks arranged in cascade as described above. After the cascade module extracts high-level semantic information from images and texts, it is connected to the fully connected layer and outputs the image vector / text vector.

[0045] Step S1023: Calculate the similarity between the image vector and the text vector, and train the image encoding network and the text encoding network based on the association relationship between the image sample and the text sample to obtain a trained recognition model.

[0046] In one embodiment, the method for calculating the similarity between the image vector and the text vector can be to calculate the Euclidean distance or the cosine distance between the two to obtain the similarity, which is not limited in this solution. Optionally, when training the recognition model, as shown in Figure 4, Figure 4 is a flow chart of a method for model training based on a generated association relationship provided in an embodiment of the present application, which includes:

[0047] Step S10231: Calculate the similarity between the image vector and the text vector using a set similarity calculation formula.

[0048] Step S10232: performing loss calculation based on the similarity and the association relationship between the image sample and the text sample to obtain a loss value.

[0049] Among them, taking the aforementioned association relationship including association and non-association as an example, the sample label value corresponding to association is 1, and non-association is 0. Among them, for the image vector E I (I), and the text vector E T(T), label value y, the similarity calculation result between the two is recorded as Sim(I,T), and the loss value can be calculated as: L(Sim(I,T),y)=-ylog(Sim(I,T))

[0050] Step S10233: Adjust the network parameters of the image encoding network and the text encoding network based on the loss value to obtain a trained recognition model.

[0051] Through the input of different images and texts as well as the calculation feedback of loss values, the network parameters of the image encoding network and the text encoding network are continuously adjusted. After the optimal network parameters are obtained after convergence, the weights are no longer updated, and the encoding network under fixed network parameters is used as the feature extraction model for subsequent category recognition.

[0052] Step S103: input the picture to be identified and the set category information into the trained recognition model to obtain a matching picture of the category information in the picture to be identified.

[0053] The image to be identified can be, for example, a live screenshot or other image generated during a live broadcast for review, or a collection of other images used to find images with matching categories. The category information can be a newly set category for which matching images need to be found. For example, if the previously entered category was volleyball, the newly set category can be beach volleyball or beach soccer, etc. Using the recognition model trained as described above, images with a high degree of matching corresponding to the category information can be obtained as matching images.

[0054] As can be seen from the above, by obtaining a picture sample set and a text sample set, wherein the picture samples in the picture sample set and the text samples in the text sample set have different associations, the set recognition model is trained based on the picture sample set and the text sample set, and the picture to be identified and the set category information are input into the trained recognition model to obtain a matching picture of the category information in the picture to be identified. In the above-mentioned category recognition method, the recognition model obtained by training based on the training data of pictures and texts is used to determine the matching picture corresponding to the set category method, without the need for separate model training for specific categories. At the same time, this solution does not require sample labeling during the training process of the recognition model, which can save a lot of time and labor costs, and the model has high recognition accuracy, is closer to the use scenario, and is more versatile.

[0055] FIG5 is a flow chart of another category identification method provided in an embodiment of the present application, as shown in FIG5 , including:

[0056] Step S201: Acquire a picture sample set and a text sample set, wherein the picture samples in the picture sample set and the text samples in the text sample set have different association relationships.

[0057] Step S202: training the set recognition model based on the image sample set and the text sample set.

[0058] Step S203: Input the image to be identified and the set category information into the trained recognition model, obtain the similarity value between the category information and each image in the image to be identified, and determine the image corresponding to the similarity value that meets the set similarity condition as the matching image of the category information.

[0059] In one embodiment, the set category information can be a brief description information that meets the category requirements, without the need to strictly limit the text required by the specification. The category information can be a category that has never been clearly defined before for identification. The picture to be identified and the set category information are input into the trained recognition model, and the picture vector of each picture in the picture to be identified and the text vector corresponding to the category information can be obtained. By calculating the similarity between the text vector and each picture vector, the similarity between the category information and each picture is obtained, and the picture with a similarity value greater than the set threshold is used as the matching picture corresponding to the category information, that is, directly used as the recognition result.

[0060] As can be seen from the above, by obtaining a picture sample set and a text sample set, wherein the picture samples in the picture sample set and the text samples in the text sample set have different associations, the set recognition model is trained based on the picture sample set and the text sample set, and the picture to be identified and the set category information are input into the trained recognition model to obtain a matching picture of the category information in the picture to be identified. In the above-mentioned category recognition method, the recognition model obtained by training based on the training data of pictures and texts is used to determine the matching picture corresponding to the set category method, without the need for separate model training for specific categories. At the same time, this solution does not require sample labeling during the training process of the recognition model, which can save a lot of time and labor costs, and the model has high recognition accuracy, is closer to the use scenario, and is more versatile.

[0061] In the above-mentioned category recognition scheme, in the scenario of a multimodal visual recognition system based on data-deficient conditions, a large number of images of target categories can be obtained by defining text labels or text descriptions of target data, namely category information, which greatly reduces the time and labor costs of data collection and annotation. In a multimodal visual recognition system based on data-deficient conditions, by utilizing massive amounts of image-text data in business scenarios, a multimodal model with semantic alignment of text and image information can be obtained for subsequent efficient recognition. At the same time, by describing the test sample set, matching recognition results can be obtained, avoiding the inefficient behavior of requiring additional development of algorithm models for recognition under limited categories. Text descriptions can be based on natural language or words, which greatly reduces the time cost of refining category annotations, is closer to user usage scenarios, is more versatile, and can quickly respond to dynamic needs such as changes in standards in the audit business.

[0062] Figure 6 is a structural block diagram of a category recognition device provided by an embodiment of the present application. As shown in Figure 6, the device is used to execute the category recognition method provided by the above embodiment, and has the corresponding functional modules and beneficial effects of the execution method. As shown in Figure 6, the device specifically includes: an acquisition module 101, a training module 102 and an identification module 103, wherein,

[0063] An acquisition module 101 is configured to acquire a picture sample set and a text sample set, wherein the picture samples in the picture sample set and the text samples in the text sample set have different association relationships;

[0064] A training module 102 is configured to train a set recognition model based on the image sample set and the text sample set;

[0065] The recognition module 103 is configured to input the image to be recognized and the set category information into the trained recognition model to obtain a matching image of the category information in the image to be recognized.

[0066] As can be seen from the above, by obtaining a picture sample set and a text sample set, wherein the picture samples in the picture sample set and the text samples in the text sample set have different associations, the set recognition model is trained based on the picture sample set and the text sample set, and the picture to be identified and the set category information are input into the trained recognition model to obtain a matching picture of the category information in the picture to be identified. In the above-mentioned category recognition method, the recognition model obtained by training based on the training data of pictures and texts is used to determine the matching picture corresponding to the set category method, without the need for separate model training for specific categories. At the same time, this solution does not require sample labeling during the training process of the recognition model, which can save a lot of time and labor costs, and the model has high recognition accuracy, is closer to the use scenario, and is more versatile.

[0067] In a possible embodiment, the apparatus further includes a sample generation module configured to:

[0068] Before obtaining the image sample set and the text sample set, obtaining images and text description information from the website information;

[0069] A picture sample set and a text sample set are generated based on the picture and the text description information, and an association relationship between the picture samples in the picture sample set and the text samples in the text sample set is generated.

[0070] In a possible embodiment, the training module 102 is configured to:

[0071] performing image standardization processing on the image samples in the image sample set to obtain standard images, and performing text standardization processing on the texts in the text sample set to obtain standard texts;

[0072] Inputting the standard picture and the standard text into a set picture encoding network and a text encoding network respectively to obtain a picture vector corresponding to the standard picture and a text vector corresponding to the standard text;

[0073] The similarity between the image vector and the text vector is calculated, and the image encoding network and the text encoding network are trained based on the association relationship between the image sample and the text sample to obtain a trained recognition model.

[0074] In a possible embodiment, the standard image includes a tensor matrix of a preset size, and the standard text includes a matrix of a preset dimension.

[0075] In a possible embodiment, the training module 102 is configured to:

[0076] Calculate the similarity between the image vector and the text vector using a set similarity calculation formula;

[0077] Performing loss calculation based on the similarity and the association relationship between the image sample and the text sample to obtain a loss value;

[0078] Based on the loss value, the network parameters of the image encoding network and the text encoding network are adjusted to obtain a trained recognition model.

[0079] In a possible embodiment, the image encoding network and the text encoding network include a self-attention module, a residual neural network module, and a forward network module.

[0080] In a possible embodiment, the identification module 103 is configured to:

[0081] Input the image to be identified and the set category information into the trained recognition model to obtain a similarity value between the category information and each image in the image to be identified;

[0082] The pictures corresponding to the similarity values ​​that meet the set similarity conditions are determined as matching pictures of the category information.

[0083] Figure 7 is a schematic diagram of the structure of a category recognition device provided in an embodiment of the present application. As shown in Figure 7, the device includes a processor 201, a memory 202, an input device 203, and an output device 204. The device may contain one or more processors 201, with one processor 201 being used as an example in Figure 7. The processor 201, memory 202, input device 203, and output device 204 in the device may be connected via a bus or other means, with bus connection being used as an example in Figure 7. Memory 202, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the category recognition method in the embodiment of the present application. Processor 201 executes the software programs, instructions, and modules stored in memory 202 to execute various functional applications and data processing of the device, thereby implementing the aforementioned category recognition method. Input device 203 can be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. Output device 204 may include a display device such as a display screen.

[0084] An embodiment of the present application further provides a non-volatile storage medium containing computer-executable instructions. When the computer-executable instructions are executed by a computer processor, the method for classifying a category described in the above embodiment is used to perform the following steps:

[0085] Acquire a picture sample set and a text sample set, wherein the picture samples in the picture sample set and the text samples in the text sample set have different association relationships;

[0086] Training the set recognition model based on the image sample set and the text sample set;

[0087] The image to be identified and the set category information are input into the trained recognition model to obtain a matching image of the category information in the image to be identified.

[0088] It is worth noting that in the embodiment of the above-mentioned category identification device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the embodiments of this application.

[0089] In some possible implementations, various aspects of the methods provided herein may also be implemented in the form of a program product, which includes program code. When the program product is executed on a computer device, the program code is used to cause the computer device to perform the steps of the methods according to the various exemplary embodiments of the present application described above in this specification. For example, the computer device may perform the category identification method described in the embodiments of the present application. The program product may be implemented using any combination of one or more readable media.

Claims

1. Category identification method, where include: Acquire a picture sample set and a text sample set, wherein the picture samples in the picture sample set and the text samples in the text sample set have different association relationships; Training the set recognition model based on the image sample set and the text sample set; The image to be identified and the set category information are input into the trained recognition model to obtain a matching image of the category information in the image to be identified.

2. The category identification method according to claim 1, wherein: Before obtaining the image sample set and the text sample set, the method further includes: Get pictures and text description information from website information; A picture sample set and a text sample set are generated based on the picture and the text description information, and an association relationship between the picture samples in the picture sample set and the text samples in the text sample set is generated.

3. The category identification method according to claim 1 or 2, wherein: The training of the set recognition model based on the image sample set and the text sample set includes: Performing image standardization processing on the image samples in the image sample set to obtain standard images, and performing text standardization processing on the texts in the text sample set to obtain standard texts; Inputting the standard picture and the standard text into a set picture encoding network and a set text encoding network respectively, to obtain a picture vector corresponding to the standard picture and a text vector corresponding to the standard text; The similarity between the image vector and the text vector is calculated, and based on the association relationship between the image sample and the text sample, the image encoding network and the text encoding network are trained to obtain a trained recognition model.

4. The category identification method according to claim 3, wherein: The standard image includes a tensor matrix of a preset size, and the standard text includes a matrix of a preset dimension.

5. The category identification method according to claim 3, wherein: The calculating the similarity between the image vector and the text vector, and training the image encoding network and the text encoding network based on the association relationship between the image sample and the text sample to obtain a trained recognition model includes: Calculate the similarity between the image vector and the text vector using a set similarity calculation formula; Performing loss calculation based on the similarity and the association relationship between the image sample and the text sample to obtain a loss value; Based on the loss value, the network parameters of the image encoding network and the text encoding network are adjusted to obtain a trained recognition model.

6. The category identification method according to claim 3, wherein: The picture encoding network and the text encoding network include a self-attention module, a residual neural network module and a forward network module.

7. The category identification method according to any one of claims 1 to 6, wherein: The step of inputting the image to be identified and the set category information into the trained recognition model to obtain a matching image of the category information in the image to be identified includes: Input the image to be identified and the set category information into the trained recognition model to obtain the similarity value between the category information and each image in the image to be identified; The pictures corresponding to the similarity values ​​that meet the set similarity conditions are determined as matching pictures of the category information.

8. A category identification device, wherein: include: An acquisition module, configured to acquire a picture sample set and a text sample set, wherein the picture samples in the picture sample set and the text samples in the text sample set have different association relationships; A training module, configured to train a set recognition model based on the image sample set and the text sample set; The recognition module is configured to input the image to be recognized and the set category information into the trained recognition model to obtain a matching image of the category information in the image to be recognized.

9. A category identification device, the device comprising: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the category identification method described in any one of claims 1-7.

10. A non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are used to perform the category identification method according to any one of claims 1 to 7 when executed by a computer processor.

11. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the category identification method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Image-text retrieval method and system based on image-text semantic embedding

    CN114741487A

  • Cross-modal retrieval model training method, cross-modal retrieval method, equipment and medium

    CN114841243A

  • Training method, image recognition method and device, equipment and readable storage medium

    CN116092101A

  • Image fine granularity identification method and device, storage medium and computer equipment

    CN116664857A

  • Class identification method and device, equipment, storage medium and program product

    CN117173511A

Cited By

  • Congeneric commodity identification method, device and equipment, storage medium and product

    CN120544219A