Image generation and sample image expansion method, device and computer storage medium

Abstract

The present application provides an image generation and sample image expansion method, device and computer storage medium, including determining multiple word nodes associated with the target word in the knowledge graph according to the word features of the target word, and obtaining the extended features of the target word according to the word features of each word node associated with the target word; using a mapping prediction model, performing mapping conversion prediction according to the extended features of the target word, and generating a target image of the target word. In this way, the present application can realize the automatic expansion of image sample categories to increase the number of recognition categories of the model and improve the accuracy of the model recognition results.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to an image generation and sample image expansion method, device and computer storage medium. Background Art

[0002] In many practical application scenarios, if a model is required to recognize a certain type of target object, a large amount of image data containing such target objects must be provided and manually annotated during the training phase to serve as training data for the model. However, large-scale image data annotation tasks require a lot of time and manpower costs, resulting in insufficient training data, so that the trained model can only recognize the same categories as the training data, and cannot correctly recognize categories not included in the training data.

[0003] In view of this, how to expand the recognition categories of the model to improve the accuracy of the model recognition results without increasing the training cost is the technical issue to be solved in this application. Summary of the invention

[0004] In view of the above problems, the present application provides an image generation and sample image expansion method, device and computer storage medium, which can at least partially solve the above problems in the prior art.

[0005] The first aspect of the present application provides an image generation method, comprising: determining multiple word nodes associated with the target word in a knowledge graph according to the word features of the target word, and obtaining extended features of the target word according to the word features of each word node associated with the target word; using a mapping prediction model, performing mapping conversion prediction according to the extended features of the target word to generate a target image of the target word.

[0006] The second aspect of the present application provides a sample image expansion method, comprising: obtaining a word to be recognized; using the image generation method described in the first aspect above to generate a word image of the word to be recognized; using the word to be recognized and the word image of the word to be recognized as a new sample word and a sample image of the new sample word, respectively, to expand them into a sample database.

[0007] The third aspect of the present application provides an image generating device, comprising: a feature expansion module, used to determine multiple word nodes associated with the target word in the knowledge graph according to the word features of the target word, and obtain the extended features of the target word according to the word features of each word node associated with the target word; a mapping prediction model, used to perform mapping conversion prediction according to the extended features of the target word to generate a target image of the target word.

[0008] The fourth aspect of the present application provides a sample image expansion device, comprising: an acquisition module, used to acquire a word to be recognized; a conversion module, used in the image generation device described in the third aspect above, to generate a word image of the word to be recognized; an expansion module, used to use the word to be recognized and the word image of the word to be recognized as a new sample word and a sample image of the new sample word, respectively, to expand them into a sample database.

[0009] The fifth aspect of the present application provides a computer storage medium, in which instructions for executing each step in the method described in the first aspect above, or instructions for executing each step in the method described in the second aspect above are stored.

[0010] To summarize, the present application can determine the extended features of the target word based on the knowledge graph and the word features of the target word, and then perform mapping conversion prediction based on the extended features to generate a target image corresponding to the target word. Accordingly, the present application provides a zero-shot learning method based on the knowledge graph. By learning the mapping conversion processing from the semantic space to the feature space through the knowledge graph, the sample image categories that the model has not been trained on can be expanded without additional manual sample labeling, so as to increase the number of recognition categories of the model, thereby improving the accuracy of the model recognition results, and also improving the overall generalization performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0012] Figure 1 This is a processing flow chart of an image generating method according to an exemplary embodiment of the present application.

[0013] Figure 2 This is a schematic diagram of a knowledge graph implementation example of the present application.

[0014] Figure 3 This is a processing flow chart of an image generating method according to another exemplary embodiment of the present application.

[0015] Figure 4 This is a processing flow chart of a sample image expansion method according to an exemplary embodiment of the present application.

[0016] Figure 5 This is a processing flow chart of a sample image expansion method according to another embodiment of the present application.

[0017] Figure 6It is a structural block diagram of an image generating device according to an exemplary embodiment of the present application.

[0018] Figure 7 It is a structural block diagram of a sample image expansion device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0020] In the current model training process, since the training data labeling task requires a lot of time and manpower costs, the training data is not sufficient, so the trained model can only recognize the same categories as the training data, but cannot correctly recognize the categories not included in the training data. In view of this, the present application provides a zero-shot learning method based on a knowledge graph, which can expand the training data categories of the model without the need for additional manual sample labeling, so as to improve the accuracy of the model recognition results.

[0021] The embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

[0022] Figure 1 This is a processing flow chart of the image generation method of the exemplary embodiment of the present application. As shown in the figure, this embodiment mainly includes the following processing steps:

[0023] Step S102, based on the word features of the target word, determine multiple word nodes associated with the target word in the knowledge graph, and obtain the extended features of the target word based on the word features of each word node associated with the target word.

[0024] Optionally, a natural language processing (NLP) model may be used to perform coarse-grained encoding on the target word to obtain word features of the target word.

[0025] In this embodiment, the knowledge graph is used to explain the relationship between people, things, and objects, and to establish a knowledge database of entity relationships by describing the textual semantics of words.

[0026] Optionally, the knowledge graph can be constructed in the following way: according to the word features corresponding to each word, the association relationship between the words is determined, and according to the association relationship between the words, a knowledge graph containing a relationship tree structure of each word node is constructed (refer to Figure 2 ).

[0027] Optionally, based on the word features of the target word, the word node corresponding to the target word in the knowledge graph can be determined as the source node, and based on the determined source node, the parent nodes and / or child nodes of all levels associated with the source node in the knowledge graph can be determined as associated nodes, and based on the word features of the determined source node and each associated node, the extended features of the target word can be determined.

[0028] For example, assuming that based on the word features of the target word, the corresponding word node in the knowledge graph is determined to be word node b, then word node b is determined as the source node. According to the relationship tree structure shown in the knowledge graph, it can be seen that word node b has a parent node (that is, word node a) and three child nodes (that is, word nodes c, d, and e), then word nodes c, d, and e can all be determined as associated nodes.

[0029] Optionally, all child nodes and / or parent nodes associated with the source node in the knowledge graph may be determined as associated nodes of the source node.

[0030] Optionally, according to a preset tracing level, each child node and / or parent node in the knowledge graph that is associated with the source node and satisfies the preset tracing level may be determined as an associated node of the source node.

[0031] For example, assuming that the preset tracing level is set to level 2, the parent node and ancestor node of the source node (i.e., the parent node of the parent node) and the child node and descendant node of the source node (i.e., the child node of the child node) can be determined as the associated nodes of the source node.

[0032] It should be noted that the number of preset traceability levels is not limited to the above two levels, and the preset traceability level of the parent node and the preset traceability level of the child node can be set to be the same or different, and this application does not impose any restrictions on this.

[0033] Optionally, the extended feature of the target word may be obtained according to a weighted sum or weighted average of the word feature of the source node and the word feature of each associated node.

[0034] For example, a weighted sum operation or a weighted average operation may be performed on the word features of each of the word nodes a, c, d, and e to obtain the extended features of the target word.

[0035] Step S104, using the mapping prediction model, performs mapping conversion prediction according to the extended features of the target word to generate a target image of the target word.

[0036] In this embodiment, the mapping prediction model may include a multilayer perceptron (MLP) model.

[0037] Specifically, the mapping prediction model can be used to map the word feature vector in the natural language space to the image feature vector in the image space according to the extended features of the target word, so as to generate a target image corresponding to the target word.

[0038] To summarize, the image generation method provided in the embodiment of the present application obtains the extended features of the target word based on the knowledge graph, and performs a mapping transformation from the natural language space to the image space according to the extended features to generate a target image of the target word, thereby realizing automatic expansion of image resources without human intervention, thereby saving labor costs.

[0039] Figure 3 This is a processing flow chart of an image generation method according to another embodiment of the present application. This embodiment mainly shows a training scheme of a mapping prediction model. As shown in the figure, this embodiment mainly includes the following steps:

[0040] Step S302: taking the extended features of the sample word as input for the mapping prediction model to perform image mapping prediction and generate a predicted image of the sample word.

[0041] Optionally, encoding processing may be performed on the sample words to obtain word features of the sample words, and based on the knowledge graph and the word features of the sample words, extended features of the sample words may be obtained.

[0042] The specific scheme for obtaining the extended features of the sample words mentioned above can refer to the description of the target words in the aforementioned step S102, and this step will not be repeated here.

[0043] Step S304, performing comparison based on the real image and the predicted image of the sample word to obtain the loss function of the mapping prediction model.

[0044] Optionally, the similarity between the real image and the predicted image of the sample word may be compared (for example, by calculating the Euclidean distance between the two), and the loss function of the mapping prediction model may be obtained according to the comparison result.

[0045] Optionally, a discriminator may be used to perform true-false adversarial learning based on real images and predicted images of sample words to determine a loss function of the mapping prediction model.

[0046] Step S306, determine whether the loss function meets the convergence condition. If so, end this step to complete the training of the mapping prediction model. If not, return to step S302 to repeat the training of the mapping prediction model based on the loss function.

[0047] Optionally, when the similarity value between the real image and the predicted image of the sample word meets a preset similarity threshold (for example, when the Euclidean distance between the real image and the predicted image is less than a preset distance threshold), a judgment result that the loss function meets the convergence condition can be obtained.

[0048] Optionally, when the discriminator cannot distinguish the authenticity between the real image and the predicted image, a judgment result that the loss function satisfies the convergence condition can be obtained.

[0049] Figure 4 The processing flow chart of the image generation method of another embodiment of the present application is shown. This embodiment can continue to execute the above step S106. As shown in the figure, this embodiment mainly includes the following processing steps:

[0050] Step S402, obtaining the word to be recognized.

[0051] In this embodiment, the acquired word to be recognized may be a sample data category not included in the sample database.

[0052] Step S404: Generate a word image of the word to be recognized by using an image generation method.

[0053] In this embodiment, the above Figure 1 or Figure 3 The image generation method shown generates a word image of a word to be recognized.

[0054] Specifically, based on the knowledge graph and the word features of the word to be recognized, the extended features of the word to be recognized can be determined, and a mapping conversion can be performed according to the extended features to generate a word image of the word to be recognized.

[0055] Step S406: Use the word to be recognized and the word image of the word to be recognized as a new sample word and a sample image of the new sample word respectively, so as to expand them into the sample database.

[0056] To sum up, the embodiment of the present application utilizes the above-mentioned image generation method to realize automatic expansion of data categories in the sample database without manual sample labeling, which not only reduces the cost of producing training data, but also increases the number of recognition categories of the model to improve the accuracy of the model recognition results.

[0057] Figure 5 This is another exemplary embodiment of the sample image expansion method of the present application. As shown in the figure, this embodiment mainly includes the following steps:

[0058] Step S502: perform a similarity comparison between the image to be identified and each sample image in the sample database to obtain a similarity value of each sample image.

[0059] Optionally, similarity comparison may be performed based on image features of the image to be identified and image features of each sample image (eg, calculating the Euclidean distance between the image to be identified and each sample image) to obtain a similarity value of the image to be identified relative to each sample image.

[0060] Step S504: determining the category label of the image to be identified according to the sample words of the sample image with the highest similarity value.

[0061] Optionally, a sample image with the highest similarity value may be determined according to the similarity value of each sample image, and the sample words of the determined sample image may be used as the category label of the image to be identified.

[0062] Step S506: Use the image to be identified and the category label of the image to be identified as a new sample word and the category label of the new sample word respectively, so as to expand them into the sample database.

[0063] To sum up, this embodiment can expand the number of recognition categories of the model without using the model to perform any training on sample images corresponding to the new categories, so as to achieve the technical effect of zero-time object recognition, which can effectively reduce the model training cost and improve the accuracy of the model recognition results.

[0064] Figure 6 The structure block diagram of the image generation device of the exemplary embodiment of the present application is shown in the figure. As shown in the figure, the image generation device 600 of the present embodiment mainly includes: a feature expansion module 602 and a mapping prediction model 604.

[0065] The feature expansion module 602 is used to determine multiple word nodes associated with the target word in the knowledge graph based on the word features of the target word, and obtain the extended features of the target word based on the word features of each word node associated with the target word.

[0066] The mapping prediction model 604 is used to perform mapping conversion prediction according to the extended features of the target word to generate a target image of the target word.

[0067] Optionally, the image generating device 600 further includes a graph construction module, which is used to determine the association relationship between the words according to the word features corresponding to the words; and to construct the knowledge graph having a relationship tree structure including nodes of each word according to the association relationship between the words.

[0068] Optionally, the feature expansion module 602 is also used to: determine the word node corresponding to the target word in the knowledge graph as the source node based on the word feature of the target word; and determine the parent nodes and / or child nodes of all levels associated with the source node in the knowledge graph as associated nodes based on the source node.

[0069] Optionally, the feature expansion module 602 is further used to obtain the extended feature of the target word according to a weighted sum or weighted average of the word feature of the source node and the word feature of each associated node.

[0070] Optionally, the image generating device 600 also includes a training module for training the mapping prediction model, which includes: executing an image prediction step, taking the extended features of the sample word as input for the mapping prediction model to perform image mapping prediction and generate a predicted image of the sample word; performing a comparison between the real image of the sample word and the predicted image to obtain a loss function of the mapping prediction model; and repeatedly executing the image prediction step based on the loss function until the loss function meets a convergence condition to complete the training of the mapping prediction model.

[0071] Figure 7 The structure block diagram of the sample image expansion device of the exemplary embodiment of the present application is shown. As shown in the figure, the sample image expansion device 700 of the present embodiment mainly includes: an acquisition module 702, a conversion module 704, and an expansion module 706.

[0072] The acquisition module 702 is used to acquire the word to be recognized.

[0073] The conversion module 704 is used to generate a word image of the word to be recognized by using the image generation device 600.

[0074] The expansion module 706 is used to use the to-be-recognized word and the word image of the to-be-recognized word as a new sample word and the sample image of the new sample word, respectively, to expand them into a sample database.

[0075] Optionally, the expansion module 706 is also used to: perform a similarity comparison between the image to be identified and each sample image in the sample database to obtain a similarity value for each sample image; determine the category label of the image to be identified based on the sample words of the sample image with the highest similarity value; and use the image to be identified and the category label of the image to be identified as a new sample word and the category label of the new sample word, respectively, to expand them into the sample database.

[0076] The exemplary embodiment of the present application further provides a computer storage medium, characterized in that the computer storage medium stores a program for executing the above Figure 1 or Figure 3 Each instruction of each step in the embodiment of the image generation method, or for executing the above Figure 4 or Figure 5 The instructions of each step in the sample image expansion method embodiment.

[0077] In summary, the graphics generation and sample image expansion methods, devices and computer storage media provided in the embodiments of the present application can automatically expand the sample categories of training samples without manually labeling samples, so that the model does not need to perform any training on sample images corresponding to new categories (i.e., zero-time object recognition), thereby increasing the number of recognition categories of the model, reducing the model training cost, and improving the accuracy of the model recognition results.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating an image, comprising: According to the word features of the target word, multiple word nodes associated with the target word in the knowledge graph are determined, and according to the word features of each word node associated with the target word, an extended feature of the target word is obtained; Using a mapping prediction model, performing mapping conversion prediction according to the extended features of the target word to generate a target image of the target word; The mapping prediction model is trained in the following way: Performing an image prediction step, taking the extended features of the sample word as input, so that the mapping prediction model performs image mapping prediction and generates a predicted image of the sample word; Performing comparison according to the real image of the sample word and the predicted image to obtain a loss function of the mapping prediction model; The image prediction step is repeatedly performed based on the loss function until the loss function meets a convergence condition to complete the training of the mapping prediction model.

2. The method according to claim 1, wherein: The knowledge graph is constructed in the following way: Determine the association between the words according to the features of each word corresponding to each word; According to the association relationship between each word, the knowledge graph containing the relationship tree structure of each word node is constructed.

3. The method according to claim 1 or 2, wherein: The step of determining a plurality of word nodes associated with the target word in the knowledge graph according to the word features of the target word comprises: According to the word features of the target word, determining the word node corresponding to the target word in the knowledge graph as the source node; According to the source node, parent nodes at all levels and / or child nodes at all levels associated with the source node in the knowledge graph are determined as associated nodes.

4. The method according to claim 3, wherein: The step of obtaining the extended features of the target word according to the word features of each word node associated with the target word comprises: The extended features of the target word are obtained according to a weighted sum or weighted average of the word features of the source node and each associated node.

5. A sample image expansion method, comprising: Get the word to be recognized; Generate a word image of the word to be recognized by using the image generation method according to any one of claims 1 to 4; The word to be recognized and the word image of the word to be recognized are respectively used as a new sample word and a sample image of the new sample word to be expanded into a sample database.

6. The method according to claim 5, wherein: The method further comprises: Performing a similarity comparison between the image to be identified and each sample image in the sample database to obtain a similarity value for each sample image; Determining the category label of the image to be identified according to the sample words of the sample image with the highest similarity value; The image to be identified and the category label of the image to be identified are respectively used as a new sample word and the category label of the new sample word to expand the sample database.

7. An image generating device, comprising: A feature expansion module, used to determine a plurality of word nodes associated with the target word in the knowledge graph according to the word feature of the target word, and obtain an extended feature of the target word according to the word feature of each word node associated with the target word; A mapping prediction model, used to perform mapping conversion prediction according to the extended features of the target word to generate a target image of the target word; A training module, used for training the mapping prediction model, comprising: performing an image prediction step, taking the extended features of the sample word as input, so that the mapping prediction model performs image mapping prediction, and generates a predicted image of the sample word; Comparison is performed based on the real image of the sample word and the predicted image to obtain a loss function of the mapping prediction model; based on the loss function, the image prediction step is repeatedly performed until the loss function meets a convergence condition to complete the training of the mapping prediction model.

8. A sample image expansion device, comprising: An acquisition module, used to acquire words to be recognized; a conversion module, configured to generate a word image of the word to be recognized by using the image generation device according to claim 7; The expansion module is used to use the word to be recognized and the word image of the word to be recognized as a new sample word and the sample image of the new sample word respectively, so as to expand them into a sample database.

9. A computer storage medium, characterized in that: The computer storage medium stores instructions for executing each step of the method according to any one of claims 1 to 4, or instructions for executing each step of the method according to any one of claims 5 to 6.

Images