Image generation method and system, data processing method

Abstract

The present application discloses an image generation method and system, and a data processing method. The method includes: generating scene data based on received text information, wherein the scene data includes: the semantic subject contained in the text information and the relationship between any two of the semantic subjects; predicting the image information of the image corresponding to the text information based on the scene sequence used to represent the scene data, wherein the image information includes the layout parameters of the semantic subject in the image; generating the image corresponding to the text information based on the image information. The present application solves the technical problem in the prior art that the scene graph is used as the input of the neural network model, resulting in inaccurate results of the semantic composition.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and more specifically, to an image generation method and system, and a data processing method. Background Art

[0002] Multi-object image generation and image retrieval based on semantic information has always been one of the core problems that need to be solved in the field of computer vision. In recent years, researchers have gradually realized the importance of semantic composition and regarded semantic composition as an intermediate expression form connecting semantic information and images. Therefore, semantic composition derived from semantic information has gradually developed into a core independent sub-problem.

[0003] The researchers further refined the semantic information and transitioned from unstructured semantic information such as natural language forms to structured scene graph expressions, splitting the entire problem into two parts: 1) predicting structured scene graphs from unstructured semantic information, which belongs to the category of natural language processing; 2) predicting semantic composition from structured scene graphs, which belongs to the category of computer vision.

[0004] In order to solve the problem mentioned in part 2), the prior art uses a graph convolutional network to complete the end-to-end generation of the entire scene graph to the entire semantic composition. However, the prior art method of using the entire scene graph as the input of the neural network model greatly increases the changes in the scene graph, resulting in almost infinite different scene graphs, making it difficult to effectively express and generalize the data, and ultimately leading to inaccurate learning results of the semantic composition.

[0005] Currently, no effective solution has been proposed to the problem that the scene graph is used as the input of the neural network model in the above-mentioned prior art, resulting in inaccurate semantic composition results. Summary of the invention

[0006] The embodiments of the present application provide an image generation method and system, and a data processing method, so as to at least solve the technical problem in the prior art that the scene graph is used as the input of the neural network model, resulting in inaccurate semantic composition results.

[0007] According to one aspect of an embodiment of the present application, a method for generating an image is provided, comprising: generating scene data based on received text information, wherein the scene data comprises: a semantic subject contained in the text information and a relationship between any two of the semantic subjects; predicting image information of an image corresponding to the text information based on a scene sequence for representing the scene data, wherein the image information comprises layout parameters of the semantic subject in the image; and generating an image corresponding to the text information based on the image information.

[0008] According to another aspect of an embodiment of the present application, a method for generating an image is also provided, including: receiving text information; performing semantic analysis on the above text information to obtain scene data corresponding to the above text information, wherein the above scene data includes: semantic subjects contained in the above text information and the relationship between the above semantic subjects; determining an image corresponding to the above text information based on the above scene data.

[0009] According to another aspect of an embodiment of the present application, a method for generating an image is also provided, including: obtaining a scene sequence corresponding to a structured scene graph, wherein the structured scene graph includes: a node for representing a semantic subject and a directed connection relationship between any two nodes; predicting a semantic composition sequence corresponding to the structured scene graph based on the scene sequence, wherein the semantic composition sequence includes: layout parameters of the semantic subject in the semantic composition corresponding to the structured scene graph; generating the semantic composition based on the semantic composition sequence.

[0010] According to another aspect of an embodiment of the present application, a method for generating an image is also provided, including: receiving text information, wherein the above text information includes; a semantic subject and a relationship between any two of the above semantic subjects; determining layout parameters of the above semantic subjects in the image according to a scene sequence corresponding to the above text information; searching for image elements corresponding to the above semantic subjects from an image database, and restoring the above image elements to corresponding positions according to the layout parameters of the above semantic subjects, to obtain an image corresponding to the above semantic subjects.

[0011] According to another aspect of an embodiment of the present application, there is also provided an image generation system, comprising: a processor; and a memory, connected to the processor, for providing the processor with instructions for processing the following processing steps: generating scene data based on received text information, wherein the scene data comprises: a semantic subject contained in the text information and a relationship between any two of the semantic subjects; predicting image information of an image corresponding to the text information based on a scene sequence for representing the scene data, wherein the image information comprises layout parameters of the semantic subject in the image; and generating an image corresponding to the text information based on the image information.

[0012] According to another aspect of an embodiment of the present application, a storage medium is further provided, wherein the storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute any one of the above-mentioned methods for generating an image.

[0013] According to another aspect of an embodiment of the present application, a processor is further provided, and the processor is used to run a program, wherein the program executes any one of the above-mentioned methods for generating an image when running.

[0014] According to another aspect of an embodiment of the present application, a data processing method is also provided, including: receiving text information, wherein the text information includes: a semantic subject and a relationship between at least two semantic subjects; generating a scene sequence based on the text information; generating image information corresponding to the text information based on the scene sequence, wherein the image information includes layout parameters of the semantic subject in the image.

[0015] In an embodiment of the present application, a method of learning the generation process of semantic composition rather than directly learning its generation result is adopted, and scene data is generated according to the received text information, wherein the above-mentioned scene data includes: the semantic subject contained in the above-mentioned text information and the relationship between any two of the above-mentioned semantic subjects; based on the scene sequence used to represent the above-mentioned scene data, image information of the image corresponding to the above-mentioned text information is predicted, wherein the above-mentioned image information includes the layout parameters of the above-mentioned semantic subject in the above-mentioned image; and the image corresponding to the above-mentioned text information is generated according to the above-mentioned image information.

[0016] Based on the embodiments of the present application, by adopting a method of learning the generation process of semantic composition rather than directly learning its generation results, the purpose of improving the accuracy of the results of predicting semantic composition based on scene graphs is achieved, thereby achieving a better technical effect of completing the visual tasks of generating images or retrieving images, and further solving the technical problem in the prior art of using scene graphs as inputs to neural network models, resulting in inaccurate results of semantic composition. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image generation method according to an embodiment of the present application;

[0019] Figure 2 is a flow chart of a method for generating an image according to an embodiment of the present application;

[0020] Figure 3 is a flowchart of an optional method for generating an image according to an embodiment of the present application;

[0021] Figure 4 is a scene schematic diagram of an optional method for generating an image according to an embodiment of the present application;

[0022] Figure 5 is a flow chart of another method for generating an image according to an embodiment of the present application;

[0023] Figure 6 is a flowchart of another method for generating an image according to an embodiment of the present application;

[0024] Figure 7 is a flowchart of another method for generating an image according to an embodiment of the present application;

[0025] Figure 8 is a schematic diagram of an image generating device according to an embodiment of the present application;

[0026] Fig. 9 is a schematic diagram of another image generating device according to an embodiment of the present application;

[0027] Fig.10 is a schematic diagram of another image generating device according to an embodiment of the present application;

[0028] Fig.11 is a schematic diagram of another image generating device according to an embodiment of the present application;

[0029] Fig.12 is a structural block diagram of a computer terminal according to an embodiment of the present application;

[0030] Fig.13 is a flow chart of a data processing method according to an embodiment of the present application; and

[0031] Fig.14 is a schematic diagram of a data processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0035] Sequence-to-sequence learning: also known as sequence-to-sequence neural network, is a technique for training neural networks that convert sequences in one field (such as English) into sequences in another field (such as French).

[0036] Scene graph: It is a "graph" data structure used to organize scenes. It usually divides data according to logical relationships (such as whole-part), but can also divide data according to other relationships (such as spatial relationships, status, etc.). Its characteristic is a hierarchical structure. Since the scale of data at a certain level is greatly reduced compared to the overall data, the scale can be reduced and efficiency can be improved.

[0037] Semantic layout: refers to a hierarchical text image synthesized based on the layout parameters of the semantic subject, which is a set of all semantic bounding boxes.

[0038] Semantic bounding box (sBBox): used in semantic composition to mark the position and size of the semantic subject.

[0039] Graph convolution network: refers to a method that can perform deep learning on graph data (Graph). Its research object is graph data, and its research model is convolutional neural network. Among them, convolutional neural network is a type of feedforward neural network that includes convolution calculations and has a deep structure.

[0040] Example 1

[0041] According to an embodiment of the present application, an embodiment of a method for generating an image is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] The method embodiment provided in Embodiment 1 of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image generation method. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0043] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0044] The memory 104 may be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for generating an image in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the method for generating an image described above is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0045] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0046] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0047] Under the above operating environment, this application provides Figure 2 The method for generating the image shown in FIG. Figure 2 is a flow chart of a method for generating an image according to an embodiment of the present application, such as Figure 2 As shown, the method for generating the above image includes the following steps:

[0048] Step S202: Generate scene data according to the received text information, wherein the scene data includes: semantic subjects included in the text information and the relationship between any two of the semantic subjects.

[0049] The steps of the above-mentioned embodiments of the present application can be executed by a smart terminal, and in particular, can be a smart home device, such as a smart speaker.

[0050] Specifically, the text information may be text information received after the device enters a specified function, for example, text information received after the device enters a text search function. The scene data is data converted from the text information, and the data may be stored in the form of scene images or other forms.

[0051] Figure 3 is a schematic diagram of generating an image according to Embodiment 1 of the present application. In an optional embodiment, in combination with Figure 3 As shown, the left side records the scene data in the form of scene images. The scene image includes multiple nodes, and there is a directed connection relationship between the nodes, wherein each node is used to represent a semantic subject, and the connection relationship between the nodes is used to represent the relationship between any two semantic subjects.

[0052] As an optional embodiment, the above-mentioned text information can be unstructured semantic information, such as natural language text, for example, a sentence spoken by a child, for example, "A girl is feeding an elephant"; after receiving the voice information, the voice information is subjected to voice recognition to obtain text information, and then the scene data corresponding to the text information can be generated according to the semantic subjects "girl" and "elephant" contained in the text information, and the relationship "feed" between the two semantic subjects "girl" and "elephant"; wherein the above-mentioned semantic subjects can include but are not limited to: a first semantic subject "girl" used to represent the subject, a second semantic subject "feed" used to represent the predicate and a third semantic subject "elephant" used to represent the object.

[0053] Step S204: predicting image information of an image corresponding to the text information according to a scene sequence used to represent the scene data, wherein the image information includes layout parameters of the semantic subject in the image.

[0054] Specifically, the scene sequence is a representation of scene data, which is used to abstract the relationship between various semantic subjects in the scene data in a sequence. The above layout parameters may include parameters such as the position and size of the semantic subject in the image.

[0055] In an optional embodiment, the above-mentioned image information can be predicted according to the scene sequence using a preset sequence-to-sequence neural network model, so that the obtained image information is also in the form of a sequence.

[0056] Step S206: Generate an image corresponding to the text information according to the image information.

[0057] Since the semantic body and its layout parameters are determined based on the above step S204, the corresponding elements can be found according to the semantic body, and then according to the layout parameters of the semantic body, the elements corresponding to the semantic body are pasted into the preset background image and the image corresponding to the above text information can be generated at the position corresponding to the layout parameters.

[0058] In an optional embodiment, still combined with Figure 3 As shown, the right part represents the generated image, which has the same relationship between the various semantic subjects expressed in the scene image on the left.

[0059] like Figure 3 As shown, the image includes sub-image A, sub-image B, and sub-image C, which respectively correspond to the relationship between the various semantic subjects expressed in the scene image on the left. Sub-image A shows a man taking a photo, sub-image B shows a girl feeding an elephant with a banana, and sub-image C shows an elephant stretching its head to grab a banana. There is an intersection between sub-image B and sub-image C, and the intersection is the girl holding the banana and the elephant's trunk. The man in sub-image A stands behind the girl in sub-image B. In addition, according to the semantic subject "girl", the corresponding elements "smile", "wear" and "T-shirt" are found, and it can also be determined that the girl is wearing a T-shirt and has a smiling expression.

[0060] It should be noted that the image generation method proposed in the embodiment of the present application can be understood as a sequence-to-sequence neural network learning method (sequence-to-sequence learning method) for generating images (i.e., semantic composition) from scene graphs. Specifically, the neural network can be a graph convolutional neural network; the method can be but is not limited to application in children's education fields such as early childhood education and children's intellectual development. For example, scene data can be generated based on the text information received from the child's output, and image information of the image corresponding to the text information can be predicted based on the scene sequence used to represent the scene data, and then the image corresponding to the text information can be generated based on the image information.

[0061] As an optional embodiment, the scene data corresponding to the above text information can be obtained by performing semantic analysis on the above text information, and the above scene data can also include: background information and a bounding box of the image, wherein the bounding box of the image is set according to the semantic subject in the image information of the above image and the layout parameters of the semantic subject (for example, position and size); the background information of the above image can be random, for example, it can be an image element in a preset material library, and the image element can be fixed at the bottom layer of the image as background information and tiled to the entire image.

[0062] For example, the text information is a sentence said by a child: "A girl is feeding an elephant, and behind her there is a man taking pictures." The "man" taking pictures behind the "girl" can be used as the background information of the image, fixed at the bottom layer of the image, and spread over the entire image.

[0063] In an embodiment of the present application, scene data can be first generated based on the received text information, and then based on the scene sequence of the scene data, image information of the image corresponding to the above text information can be predicted, and an image corresponding to the text information can be generated based on the image information, wherein the image finally generated corresponding to the text information can be but is not limited to a map.

[0064] The embodiment of the present application learns the generation process of semantic composition rather than directly learning its generation result. In addition, the embodiment of the present application uses unstructured text information as the input of the sequence-to-sequence neural network model to realize the prediction of semantic composition, and then complete the visual task of generating images or retrieving images, thereby achieving better image generation effect or image retrieval effect.

[0065] It is still necessary to explain that, since a semantic composition is mainly determined by all the semantic subjects that are related to each other, the embodiment of the present application further subdivides the input of the neural network model into the most basic unit that constitutes the semantic composition: the relationship between the semantic subjects, and learns how to drag the semantic subjects within the relationship to the semantic composition. For example, each relationship in a scene graph is learned, and each time a relationship between any two semantic subjects is obtained, the positions of the two bounding boxes of the semantic subjects (for example, a first semantic subject corresponding to a subject, and a third semantic subject corresponding to an object) are dragged on the semantic composition "canvas" and the sizes of the two bounding boxes are adjusted; and the process of generating the semantic composition is modeled and abstracted into several basic action code segments BACS as shown in Table 1. By learning the commonalities of the common basic action code segments, the input changes of the neural network model are reduced to achieve the purpose of better learning the final semantic composition.

[0066] Table 1

[0067]

[0068]

[0069] In the above Table 1, the prefix c indicates setting the index number (class index) of the semantic class to which the bounding box belongs. For example, if there are 100 classes of objects in total, and "house" is the first class of objects, the index number is 0. In Table 1, if the prefix c indicates setting the class index number of the semantic bounding box sBBox to 0, the corresponding code can be exemplified as c00.

[0070] The bounding box sBBox in Table 1 above also includes: x: the minimum x of the bounding box on the image, y: the minimum y of the bounding box on the image, the coordinates (x, y) define the coordinates of the upper left corner of the bounding box; w: the width of the bounding box, h: the height of the bounding box. In Table 1 above, xp means setting the minimum value of x in the semantic bounding box sBBox to a positive value, which corresponds to the index number of the image grid along the x direction. If the image has 1000 pixels along the x direction and is quantized into 40 image grids, then all pixels with coordinates x of 0-24 will fall into the image grid with x index number 0. Similar to xp, xn, yp, and yn in Table 1 have similar meanings.

[0071] In Table 1 above, ixp, ixn, iyp, iyn (index numbers 6-9 in Table 1) are the x and y coordinates corresponding to the object semantic bounding box sBBox; unlike the subject code, the subject semantic bounding box sBBox and the object semantic bounding box sBBox constitute the two ends corresponding to a semantic relationship triple, and the relationship is learned by relative position, and the object ix, iy corresponds to this relative position, which means that relative to the x and y coordinates of the subject semantic bounding box sBBox, the x and y coordinates of the object semantic bounding box sBBox are set to right (ixp), left (ixn), up (iyn), and down (iyp) several image grid coordinates. All semantic relations are encoded in this relative coordinate. Because the relationship does not affect the size of the object, the w and h of the object subject and object use the same code.

[0072] It should be noted that the prefix imgar in Table 1 above is mainly used to indicate the aspect ratio of the canvas and is an optional output item, not required.

[0073] In an embodiment of the present application, a method of learning the generation process of semantic composition rather than directly learning its generation result is adopted, and scene data is generated according to the received text information, wherein the above-mentioned scene data includes: the semantic subject contained in the above-mentioned text information and the relationship between any two of the above-mentioned semantic subjects; based on the scene sequence used to represent the above-mentioned scene data, image information of the image corresponding to the above-mentioned text information is predicted, wherein the above-mentioned image information includes the layout parameters of the above-mentioned semantic subject in the above-mentioned image; and the image corresponding to the above-mentioned text information is generated according to the above-mentioned image information.

[0074] Based on the embodiments of the present application, by adopting a method of learning the generation process of semantic composition rather than directly learning its generation results, the purpose of improving the accuracy of the results of predicting semantic composition based on scene graphs is achieved, thereby achieving a better technical effect of completing the visual tasks of generating images or retrieving images, and further solving the technical problem in the prior art of using scene graphs as inputs to neural network models, resulting in inaccurate results of semantic composition.

[0075] In an optional embodiment, predicting the image information of the image corresponding to the text information according to the scene sequence used to represent the scene data includes:

[0076] Step S302, obtaining a scene sequence corresponding to the scene data;

[0077] Step S304: Based on the scene sequence, an image information sequence is predicted using a sequence-to-sequence neural network, wherein the image information sequence includes a basic action code segment for representing the image information.

[0078] Specifically, the input and output of the above sequence-to-sequence neural network are both data represented in a sequence manner, and the sequence-to-sequence neural network learns the actions dragged by the designer during the training process.

[0079] In an optional embodiment, a large number of image elements are obtained (in the initial training, the image elements can be drawn by the designer. As the number of samples increases, the designer does not need to draw new image elements, but can directly use them). The designer drags the image elements corresponding to the semantic subject to the predetermined position according to the scene data. The sequence corresponding to the scene data and the action or position dragged by the designer are used as a sample set, and the sequence-to-sequence neural network can be used for learning.

[0080] In an optional embodiment, the above-mentioned image information sequence includes two groups of basic action code segments, wherein the first group of action code segments is used to represent the layout parameters of the first semantic subject (i.e., the first semantic subject used to represent the subject), and the second group of action code segments is used to represent the layout parameters of the third semantic subject (i.e., the third semantic subject used to represent the object).

[0081] In another optional embodiment, the above-mentioned scene sequence may be a scene sequence composed of relation triplets in the scene data. Based on the scene sequence corresponding to the acquired scene data, an image information sequence can be predicted according to a sequence-to-sequence neural network. Since the image information sequence includes a basic action code segment for representing the above-mentioned image information, the image information of the image (i.e., semantic composition) corresponding to the text information can be predicted.

[0082] In an optional embodiment, a transformer (for example, a 6-layer transformer Transformer) can be used as a sequence-to-sequence neural network composed of an encoder and decoder, and a scene sequence SGSeq is used as the input of the sequence-to-sequence neural network model to predict an image information sequence SLSeq, wherein the image information sequence SLSeq may include two groups of basic action code segments (brick-action code segments, BACS), for example, a first group of action code segments for representing layout parameters of a first semantic subject, and a second group of action code segments for representing layout parameters of a third semantic subject.

[0083] It should be noted that each group of basic action code segments BACS corresponds one-to-one to a relation triple in the scene sequence SGSeq and contains the following information: how to set the layout parameters of the first semantic subject in the image (that is, the position and size of the bounding box of the first semantic subject in the image) and the layout parameters of the third semantic subject (that is, the position and size of the bounding box of the third semantic subject in the image).

[0084] As an optional embodiment, the type and function of the basic action code segment BACS may be as shown in Table 1 above. For example, each basic action code segment BACS may be composed of 10 words, corresponding to the relation triples of the corresponding scene sequence; the prefixes of the above 10 words may be: c, xp(n), yp(n), w, h, c, ixp(n), iyp(n), w, h; among the above 10 words, the first 5 words are used to indicate the type of the bounding box sBBox corresponding to the subject and its absolute position and size in the semantic composition, and the last 5 words are used to indicate the type of the bounding box sBBox corresponding to the object and its position and size relative to the subject in the semantic composition.

[0085] By learning the absolute position of the subject in the semantic composition and the relative position of the object in the semantic composition, the relationship between the subject and the object in the scene data can be learned. By learning the historical data of multiple relations that appear in the semantic composition at the same time, the image information of the semantic composition between multiple relations can be learned.

[0086] In an optional embodiment, obtaining a scene sequence corresponding to the scene data includes:

[0087] Step S402, extracting a relation triple in the scene data, wherein the relation triple includes: a first semantic subject for representing a subject, a second semantic subject for representing a predicate, and a third semantic subject for representing an object;

[0088] Step S404: construct the scene sequence according to the relationship triples.

[0089] In the embodiment of the present application, the above scene data can be displayed in the form of a scene graph, such as Figure 4 As shown, scene data is generated according to the received text information, and the scene data at least includes: the semantic subject contained in the above text information and the relationship between any two of the above semantic subjects, and by extracting the relationship triples in the scene data, a scene graph is converted into a scene sequence SGSeq composed of the relationship triples in the scene data (a first semantic subject for representing a subject - a second semantic subject for representing a predicate - a third semantic subject for representing an object), wherein, as Figure 4 As shown, the process of forming a scene sequence according to the relationship triples is scene serialization. As an optional embodiment, the scene graph can be a structured scene graph, for example, a directed graph.

[0090] In an optional embodiment, generating an image corresponding to the text information according to the image information includes:

[0091] Step S502, setting the bounding box according to the semantic body in the image information and the layout parameters of the semantic body;

[0092] Step S504: construct the image based on the bounding box.

[0093] In an optional embodiment, the layout parameters of the semantic body include: the position and size of the bounding box of the semantic body in the image.

[0094] Optionally, the above-mentioned bounding box, namely the semantic bounding box, can be applied to the semantic composition to mark the position and size of the semantic subject.

[0095] In an optional embodiment of the present application, since the image information includes the layout parameters of the semantic subject in the image, that is, the position and size of the bounding box of the semantic subject in the above image. After forming a scene sequence according to the relationship triples in the scene data, the image information of the image corresponding to the text information is predicted according to the scene sequence corresponding to the scene data, and then the bounding box of the semantic subject is set according to the semantic subject in the image information and the layout parameters of the semantic subject; and the above image is formed based on the bounding box.

[0096] As an optional embodiment, the image elements corresponding to the semantic subject may be searched from an image database, and the image elements may be restored to corresponding positions according to the layout parameters of the semantic subject to obtain the image corresponding to the semantic subject.

[0097] In an optional embodiment, constructing the image based on the bounding box includes:

[0098] Step S602, selecting a semantic subject image corresponding to the semantic subject in the above image information from a preset material library;

[0099] Step S604: determining the position and size of the semantic subject image according to the bounding box to obtain an image corresponding to the text information.

[0100] As an optional embodiment, the semantic subject image can be pre-stored in a preset material library. Since there is a one-to-one correspondence between the semantic subject in the image information and the semantic subject image, when generating an image corresponding to the text information, the semantic subject image corresponding to the semantic subject in the image information can be selected from the preset material library, and the position and size of the semantic subject image can be determined according to the bounding box of the semantic subject to obtain the image corresponding to the text information.

[0101] In an optional embodiment, before generating the image corresponding to the text information according to the image information, the method further includes: step S702, acquiring an identification information sequence.

[0102] Since converting a scene graph into a scene sequence SGSeq consisting of relationship triplets may result in the problem of losing the corresponding relationship between semantic entities, in the embodiment of the present application, before generating the image corresponding to the text information according to the image information, an identification information sequence nSeq (such as Figure 4 The node sequence shown in FIG. 1 is used as an auxiliary to generate an image corresponding to the text information according to the image information.

[0103] In the above step S702, the step of obtaining the identification information sequence includes:

[0104] Step S7021, extracting a relation triple in the scene data, wherein the relation triple includes: a first semantic subject for representing a subject, a second semantic subject for representing a predicate, and a third semantic subject for representing an object;

[0105] Step S7023, obtaining the identifier corresponding to each semantic subject in the above relationship triple;

[0106] Step S7025: construct the identification information sequence according to the identification corresponding to each semantic subject in the relationship triple.

[0107] In the above optional embodiment, after generating scene data based on the received text information, the relationship triples in the scene data can be extracted, and the identifier corresponding to each semantic subject in the relationship triples can be obtained, and then an identification information sequence is constructed based on the identifier corresponding to each semantic subject in the relationship triples.

[0108] In an optional embodiment, generating an image corresponding to the text information according to the image information includes: step S802, generating an image corresponding to the text information according to the image information and the identification information sequence.

[0109] In order to avoid the problem of losing the correspondence between semantic entities in the process of converting a scene graph into a scene sequence SGSeq composed of relationship triples, in an optional embodiment of the present application, the identification information sequence can be used as an auxiliary to generate an image corresponding to the text information according to the image information. Figure 4 As shown, after obtaining the identification information sequence nSeq, according to the image information sequence SLSeq (i.e. Figure 4 The scene composition sequence shown in FIG. 1 ) and the identification information sequence nSeq (i.e., Figure 4 The scene composition is obtained by using the node sequence shown in FIG. 1 , and an image corresponding to the text information is generated based on the scene composition.

[0110] In step S802, the step of generating an image corresponding to the text information according to the image information and the identification information sequence includes:

[0111] Step S8021, determining a bounding box in the image information for representing the same semantic subject according to the identification information sequence;

[0112] Step S8023, merging bounding boxes used to represent the same semantic subject;

[0113] Step S8025, setting the bounding box according to the semantic body in the image information and the layout parameters of the semantic body;

[0114] Step S8027, constructing the above image based on the above bounding box.

[0115] In an embodiment of the present application, an image corresponding to the received text information can be generated based on the basic action code segment BACS of the image information included in the predicted image information sequence SLSeq. Using the identifier (e.g., node ID) corresponding to each semantic subject in the relationship triple in the identification information sequence nSeq, it can be determined which semantic bounding boxes in the predicted basic action code segment BACS correspond to the same semantic subject, and then the bounding boxes used to represent the same semantic subject can be merged, and finally an image corresponding to the text information is generated based on the above bounding boxes.

[0116] It should be noted that similar methods of using sequence-to-sequence neural network learning to learn the relative positions of semantic entities within a relationship to represent the characteristics of this relationship and solving such problems should be included in the scope of protection claimed by this application.

[0117] Example 2

[0118] According to an embodiment of the present application, an embodiment of another method for generating an image is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0119] Under the above operating environment, this application provides Figure 5 The method for generating the image shown in FIG. Figure 5 is a flowchart of another method for generating an image according to an embodiment of the present application, such as Figure 5 As shown, the method for generating the above image includes the following steps:

[0120] Step S902, receiving text information;

[0121] Step S904, performing semantic analysis on the text information to obtain scene data corresponding to the text information, wherein the scene data includes: semantic subjects included in the text information and the relationship between the semantic subjects;

[0122] Step S906: determining the image corresponding to the text information according to the scene data.

[0123] The steps of the above-mentioned embodiments of the present application can be executed by a smart terminal, and in particular, can be a smart home device, such as a smart speaker.

[0124] Specifically, the above text information may be text information received after the device enters a specified function, for example, text information received after the device enters a text search image function, or text information obtained after the device enters a text search image function and performs voice recognition on received voice information. The above scene data is data converted from text information, and the data may be stored in the form of scene images or other forms.

[0125] In an optional embodiment, still combined with Figure 3 As shown, the left side records the scene data in the form of scene images. The scene image includes multiple nodes, and there is a directed connection relationship between the nodes, wherein each node is used to represent a semantic subject, and the connection relationship between the nodes is used to represent the relationship between any two semantic subjects.

[0126] In an embodiment of the present application, scene data can be first generated based on the received text information, and then based on the scene sequence of the scene data, image information of the image corresponding to the above text information can be predicted, and an image corresponding to the text information can be generated based on the image information, wherein the finally generated image corresponding to the text information can be a map.

[0127] Specifically, the scene sequence is a representation of scene data, which is used to abstract the relationship between various semantic subjects in the scene data in a sequence. The above layout parameters may include parameters such as the position and size of the semantic subject in the image.

[0128] In an optional embodiment, the above-mentioned image information can be predicted according to the scene sequence using a preset sequence-to-sequence neural network model, so that the obtained image information is also in the form of a sequence.

[0129] In the above optional embodiment of the present application, the semantic subject and its layout parameters are determined, so the corresponding element can be found according to the semantic subject, and then according to the layout parameters of the semantic subject, the element corresponding to the semantic subject is pasted to the preset background image at the position corresponding to the layout parameters, and the generated image can be obtained. Figure 3 As shown, the right part represents the generated image, which has the same relationship between the various semantic subjects expressed in the image and the scene image on the left.

[0130] It should be noted that the image generation method proposed in the embodiment of the present application can be understood as a sequence-to-sequence neural network learning method for generating images (i.e., semantic composition) from scene graphs, which can be but is not limited to application in the field of children's education. For example, scene data can be generated based on the received text information of the child, and image information of the image corresponding to the child's text information can be predicted based on the scene sequence used to represent the scene data, and then an image corresponding to the child's text information can be generated based on the image information.

[0131] Optionally, the above-mentioned text information can be unstructured semantic information, such as natural language text, for example, a sentence spoken by a child, for example, "A girl is feeding an elephant"; based on the semantic subjects "girl" and "elephant" contained in the text information, and the relationship "feed" between the two semantic subjects "girl" and "elephant", scene data corresponding to the text information can be generated; wherein the above-mentioned semantic subjects can include but are not limited to: a first semantic subject "girl" used to represent the subject, a second semantic subject "feed" used to represent the predicate and a third semantic subject "elephant" used to represent the object.

[0132] As an optional embodiment, the scene data corresponding to the above text information can be obtained by performing semantic analysis on the above text information, and the above scene data can also include: background information and a bounding box of the image, wherein the bounding box of the image is set according to the semantic subject in the image information of the above image and the layout parameters of the semantic subject (for example, position and size); the background information of the above image can be random, for example, it can be an image element in a preset material library, and the image element can be fixed at the bottom layer of the image as background information and tiled to the entire image.

[0133] Taking the above text information as an example, which is "a girl is feeding an elephant, and behind her there is a man taking pictures" said by a child, the "man" taking pictures behind the "girl" can be used as the background information of the image, fixed at the bottom layer of the image, and tiled to the entire image.

[0134] The embodiment of the present application learns the generation process of semantic composition rather than directly learning its generation result. In addition, the embodiment of the present application uses unstructured text information as the input of the sequence-to-sequence neural network model to achieve the prediction of semantic composition, and then complete the visual task of generating images or retrieving images, thereby achieving better image generation effect or image retrieval effect.

[0135] It is still necessary to explain that, since a semantic composition is mainly determined by all the mutually related semantic subjects therein, the embodiment of the present application further subdivides the input of the neural network model into the most basic units that constitute the semantic composition: the relationship between the semantic subjects, and learns how to drag the semantic subjects within the relationship to the semantic composition. For example, each relationship in a scene graph is learned, and each time a relationship between any two semantic subjects is obtained, the positions of the two bounding boxes of the semantic subjects (for example, a first semantic subject corresponding to a subject, and a third semantic subject corresponding to an object) are dragged on the semantic composition "canvas" and the sizes of the two bounding boxes are adjusted; and the process of generating the semantic composition is modeled and abstracted into several basic action code segments BACS. By learning the commonalities of the common basic action code segments, the input changes of the neural network model are reduced, so as to achieve the purpose of better learning the final semantic composition.

[0136] In an embodiment of the present application, a method of learning the generation process of semantic composition rather than directly learning its generation result is adopted, by receiving text information; performing semantic analysis on the above text information to obtain scene data corresponding to the above text information, wherein the above scene data includes: the semantic subject contained in the above text information and the relationship between the above semantic subjects; and determining the image corresponding to the above text information based on the above scene data.

[0137] Based on the embodiments of the present application, by adopting a method of learning the generation process of semantic composition rather than directly learning its generation results, the purpose of improving the accuracy of the results of predicting semantic composition based on scene graphs is achieved, thereby achieving a better technical effect of completing the visual tasks of generating images or retrieving images, and further solving the technical problem in the prior art of using scene graphs as inputs to neural network models, resulting in inaccurate results of semantic composition.

[0138] It should be noted that the optional or preferred implementation of this embodiment can refer to the relevant description in Example 1, which will not be repeated here.

[0139] Example 3

[0140] According to an embodiment of the present application, another embodiment of a method for generating an image is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0141] Under the above operating environment, this application provides Figure 6 The method for generating the image shown in FIG. Figure 6 is a flowchart of another method for generating an image according to an embodiment of the present application, such as Figure 6 As shown, the method for generating the above image includes the following steps:

[0142] Step S1002: Obtain a scene sequence corresponding to a structured scene graph, wherein the structured scene graph includes: nodes for representing semantic entities and directed connection relationships between any two nodes.

[0143] The steps of the above-mentioned embodiments of the present application can be executed by a smart terminal, and in particular, can be a smart home device, such as a smart speaker.

[0144] Specifically, the structured scene graph may be a scene image for recording scene data; more specifically, it may be a scene image for recording scene data converted from text information. The nodes included therein are semantic subjects in the text information, and the connection relationship between the nodes may be an association relationship between the semantic subjects in the text information. The association relationship may be a connection relationship between a subject and an object.

[0145] Step S1004: predicting a semantic composition sequence corresponding to the structured scene graph according to the scene sequence, wherein the semantic composition sequence includes: layout parameters of the semantic subject in the semantic composition corresponding to the structured scene graph.

[0146] Step S1006: generating the above semantic composition according to the above semantic composition sequence.

[0147] Specifically, the above semantic composition refers to a set of semantic borders. Figure 3 As shown, Figure 3 A collection of medium rectangular borders.

[0148] It should be noted that the image generation method proposed in the embodiment of the present application can be understood as a sequence-to-sequence neural network learning method for generating images (i.e., semantic composition) from scene graphs. Specifically, the neural network can be a graph convolutional neural network; the method can be but is not limited to application in children's education fields such as early childhood education and children's intelligence development.

[0149] Optionally, the structured scene graph may be a directed graph, including nodes for representing semantic entities and directed connection relationships between any two nodes.

[0150] Specifically, Figure 4 The scene sequence shown is a representation of scene data, which is used to abstract the relationship between various semantic subjects in the scene data in a sequence. The above layout parameters may include parameters such as the position and size of the semantic subject in the image.

[0151] In an optional embodiment of the present application, the above-mentioned scene sequence can be used to represent the scene data of the above-mentioned structured scene graph. Based on the above-mentioned scene sequence, according to a sequence-to-sequence neural network, a semantic composition sequence is predicted, that is, the layout parameters of the semantic subject in the semantic composition corresponding to the above-mentioned structured scene graph; then, a bounding box of the semantic subject is set according to the semantic subject in the semantic composition and the layout parameters of the semantic subject; and the semantic composition is generated based on the above-mentioned bounding box.

[0152] In the above optional embodiment of the present application, the semantic subject and its layout parameters are determined, so the corresponding elements can be found according to the semantic subject, and then according to the layout parameters of the semantic subject, the elements corresponding to the semantic subject are pasted to the preset background image at the position corresponding to the layout parameters, and the generated semantic composition can be obtained. Figure 3 As shown, the right part represents the generated semantic composition, which has the same relationship between the various semantic subjects expressed in the scene image on the left.

[0153] Optionally, the above-mentioned semantic composition sequence can be an image information sequence, which can include two groups of basic action code segments, wherein the first group of action code segments is used to represent the layout parameters of the above-mentioned first semantic body, and the second group of action code segments is used to represent the layout parameters of the above-mentioned third semantic body.

[0154] In an optional embodiment, the layout parameters of the semantic subject include: the position and size of the bounding box of the semantic subject in the image. Optionally, the bounding box is a semantic bounding box, which can be applied to the semantic composition to mark the position and size of the semantic subject.

[0155] It is still necessary to explain that, since a semantic composition is mainly determined by all the mutually related semantic subjects therein, the embodiment of the present application further subdivides the input of the neural network model into the most basic units that constitute the semantic composition: the relationship between the semantic subjects, and learns how to drag the semantic subjects within the relationship to the semantic composition. For example, each relationship in a scene graph is learned, and each time a relationship between any two semantic subjects is obtained, the positions of the two bounding boxes of the semantic subjects (for example, a first semantic subject corresponding to a subject, and a third semantic subject corresponding to an object) are dragged on the semantic composition "canvas" and the sizes of the two bounding boxes are adjusted; and the process of generating the semantic composition is modeled and abstracted into several basic action code segments BACS. By learning the commonalities of the common basic action code segments, the input changes of the neural network model are reduced, so as to achieve the purpose of better learning the final semantic composition.

[0156] In an embodiment of the present application, a method of learning the generation process of semantic composition rather than directly learning its generation result is adopted, by obtaining a scene sequence corresponding to a structured scene graph, wherein the above-mentioned structured scene graph includes: a node for representing a semantic subject and a directed connection relationship between any two nodes; predicting a semantic composition sequence corresponding to the above-mentioned structured scene graph based on the above-mentioned scene sequence, wherein the above-mentioned semantic composition sequence includes: layout parameters of the above-mentioned semantic subject in the semantic composition corresponding to the above-mentioned structured scene graph; and generating the above-mentioned semantic composition based on the above-mentioned semantic composition sequence.

[0157] Based on the embodiments of the present application, by adopting a method of learning the generation process of semantic composition rather than directly learning its generation results, the purpose of improving the accuracy of the results of predicting semantic composition based on scene graphs is achieved, thereby achieving a better technical effect of completing the visual tasks of generating images or retrieving images, and further solving the technical problem in the prior art of using scene graphs as inputs to neural network models, resulting in inaccurate results of semantic composition.

[0158] It should be noted that the optional or preferred implementation of this embodiment can refer to the relevant description in Example 1, which will not be repeated here.

[0159] Example 4

[0160] According to an embodiment of the present application, an embodiment of another method for generating an image is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0161] Under the above operating environment, this application provides Figure 7 The method for generating the image shown in FIG. Figure 7 is a flowchart of another method for generating an image according to an embodiment of the present application, such as Figure 7 As shown, the method for generating the above image includes the following steps:

[0162] Step S1102, receiving text information, wherein the text information includes: a semantic subject and a relationship between any two of the semantic subjects;

[0163] Step S1104, determining the layout parameters of the semantic subject in the image according to the scene sequence corresponding to the text information;

[0164] Step S1106, searching the image element corresponding to the semantic subject from the image database, and restoring the image element to the corresponding position according to the layout parameter of the semantic subject, so as to obtain the image corresponding to the semantic subject.

[0165] The steps of the above-mentioned embodiments of the present application can be executed by a smart terminal, and in particular, can be a smart home device, such as a smart speaker.

[0166] Specifically, the text information may be text information received after the device enters a specified function, for example, text information received after the device enters a text search function. The scene data is data converted from the text information, and the data may be stored in the form of scene images or other forms.

[0167] In an optional embodiment, still combined with Figure 3 As shown, the left side records the scene data in the form of scene images. The scene image includes multiple nodes, and there is a directed connection relationship between the nodes, wherein each node is used to represent a semantic subject, and the connection relationship between the nodes is used to represent the relationship between any two semantic subjects.

[0168] In the above optional embodiment of the present application, the semantic subject and its layout parameters are determined, so the corresponding element can be found according to the semantic subject, and then according to the layout parameters of the semantic subject, the element corresponding to the semantic subject is pasted to the preset background image at the position corresponding to the layout parameters, and the generated image can be obtained. Figure 3As shown, the right part represents the generated image, which has the same relationship between the various semantic subjects expressed in the image and the scene image on the left.

[0169] In the embodiment of the present application, the layout parameters of the semantic subject in the image can be determined according to the scene sequence corresponding to the received text information, the image elements corresponding to the semantic subject can be found from the image database, and the image elements can be restored to the corresponding positions according to the layout parameters of the semantic subject to obtain the image corresponding to the semantic subject. The image corresponding to the semantic subject can be a map.

[0170] It should be noted that the image generation method proposed in the embodiment of the present application can be understood as a sequence-to-sequence neural network learning method for generating images (i.e., semantic composition) from scene graphs, which can be but is not limited to being applied in the field of children's education. For example, scene data can be generated based on the received text information of the children, and the layout parameters of the semantic subject in the image of the children's text information can be predicted based on the scene sequence used to represent the scene data, and then the image elements corresponding to the above semantic subjects can be searched from the image database, and the above image elements can be restored to the corresponding positions according to the layout parameters of the above semantic subjects to obtain the image corresponding to the above semantic subjects.

[0171] Optionally, the above-mentioned text information can be unstructured semantic information, such as natural language text, for example, a sentence spoken by a child, for example, "A girl is feeding an elephant"; based on the semantic subjects "girl" and "elephant" contained in the text information, and the relationship "feed" between the two semantic subjects "girl" and "elephant", scene data corresponding to the text information can be generated; wherein the above-mentioned semantic subjects can include but are not limited to: a first semantic subject "girl" used to represent the subject, a second semantic subject "feed" used to represent the predicate and a third semantic subject "elephant" used to represent the object.

[0172] As an optional embodiment, the scene data corresponding to the above text information can be obtained by performing semantic analysis on the above text information, and the above scene data can also include: background information and a bounding box of the image, wherein the bounding box of the image is set according to the semantic subject in the image information of the above image and the layout parameters of the semantic subject (for example, position and size); the background information of the above image can be random, for example, it can be an image element in a preset material library, and the image element can be fixed at the bottom layer of the image as background information and tiled to the entire image.

[0173] Taking the above text information as an example, which is "a girl is feeding an elephant, and behind her there is a man taking pictures" said by a child, the "man" taking pictures behind the "girl" can be used as the background information of the image, fixed at the bottom layer of the image, and tiled to the entire image.

[0174] In an optional embodiment, the layout parameters of the semantic subject include: the position and size of the bounding box of the semantic subject in the image. Optionally, the bounding box is a semantic bounding box, which can be applied to the semantic composition to mark the position and size of the semantic subject.

[0175] In an embodiment of the present application, since the layout parameters of the above-mentioned semantic subject in the image, that is, the position and size of the bounding box of the semantic subject in the above-mentioned image, can be determined according to the scene sequence corresponding to the received text information; then, the bounding box of the semantic subject can be set according to the semantic subject and the layout parameters of the semantic subject, and the image elements corresponding to the semantic subject can be restored to the corresponding positions based on the above-mentioned bounding box, so as to obtain the image corresponding to the semantic subject.

[0176] The embodiment of the present application learns the generation process of semantic composition rather than directly learning its generation result. In addition, the embodiment of the present application uses unstructured text information as the input of the sequence-to-sequence neural network model to achieve the prediction of semantic composition, and then complete the visual task of generating images or retrieving images, thereby achieving better image generation effect or image retrieval effect.

[0177] It is still necessary to explain that, since a semantic composition is mainly determined by all the mutually related semantic subjects therein, the embodiment of the present application further subdivides the input of the neural network model into the most basic units that constitute the semantic composition: the relationship between the semantic subjects, and learns how to drag the semantic subjects within the relationship to the semantic composition. For example, each relationship in a scene graph is learned, and each time a relationship between any two semantic subjects is obtained, the positions of the two bounding boxes of the semantic subjects (for example, a first semantic subject corresponding to a subject, and a third semantic subject corresponding to an object) are dragged on the semantic composition "canvas" and the sizes of the two bounding boxes are adjusted; and the process of generating the semantic composition is modeled and abstracted into several basic action code segments BACS. By learning the commonalities of the common basic action code segments, the input changes of the neural network model are reduced, so as to achieve the purpose of better learning the final semantic composition.

[0178] In an embodiment of the present application, a method of learning the generation process of semantic composition rather than directly learning its generation result is adopted, by receiving text information, wherein the above-mentioned text information includes: a semantic subject and a relationship between any two of the above-mentioned semantic subjects; determining the layout parameters of the above-mentioned semantic subjects in the image according to the scene sequence corresponding to the above-mentioned text information; searching for image elements corresponding to the above-mentioned semantic subjects from an image database, and restoring the above-mentioned image elements to corresponding positions according to the layout parameters of the above-mentioned semantic subjects, so as to obtain an image corresponding to the above-mentioned semantic subject.

[0179] Based on the embodiments of the present application, by adopting a method of learning the generation process of semantic composition rather than directly learning its generation results, the purpose of improving the accuracy of the results of predicting semantic composition based on scene graphs is achieved, thereby achieving a better technical effect of completing the visual tasks of generating images or retrieving images, and further solving the technical problem in the prior art of using scene graphs as inputs to neural network models, resulting in inaccurate results of semantic composition.

[0180] It should be noted that the optional or preferred implementation of this embodiment can refer to the relevant description in Example 1, which will not be repeated here.

[0181] It is still necessary to explain that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0182] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the above methods of each embodiment of the present application.

[0183] Example 5

[0184] According to an embodiment of the present application, an embodiment of an image generation system for implementing the above-mentioned image generation method is also provided, and the image generation system includes:

[0185] A processor; and a memory connected to the processor, for providing the processor with instructions for processing the following processing steps: generating scene data based on received text information, wherein the scene data includes: a semantic subject contained in the text information and a relationship between any two of the semantic subjects; predicting image information of an image corresponding to the text information based on a scene sequence used to represent the scene data, wherein the image information includes layout parameters of the semantic subject in the image; and generating an image corresponding to the text information based on the image information.

[0186] Example 6

[0187] According to an embodiment of the present application, a device embodiment for implementing the above-mentioned image generation method is also provided. Figure 8 is a schematic diagram of an image generating device according to an embodiment of the present application, such as Figure 8 As shown, the image generation device 800 includes: a first generation module 801, a prediction module 803 and a second generation module 805, wherein:

[0188] The first generation module 801 is used to generate scene data based on the received text information, wherein the above scene data includes: the semantic subject contained in the above text information and the relationship between any two of the above semantic subjects; the prediction module 803 is used to predict the image information of the image corresponding to the above text information based on the scene sequence used to represent the above scene data, wherein the above image information includes the layout parameters of the above semantic subject in the above image; the second generation module 805 is used to generate the image corresponding to the above text information based on the above image information.

[0189] It should be noted that the first generation module 801, the prediction module 803 and the second generation module 805 correspond to steps S202 to S206 in Example 1, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0190] It should be noted that the optional or preferred implementations of this embodiment can refer to the relevant descriptions in Embodiments 1 to 4, which will not be repeated here.

[0191] Example 7

[0192] According to an embodiment of the present application, another device embodiment for implementing the above-mentioned method for generating an image is also provided. Fig. 9 is a schematic diagram of another image generating device according to an embodiment of the present application, such as Fig. 9As shown, the image generation device 900 includes: a first receiving module 901, an analysis module 903 and a determination module 905, wherein:

[0193] The first receiving module 901 is used to receive text information; the analyzing module 903 is used to perform semantic analysis on the above text information to obtain scene data corresponding to the above text information, wherein the above scene data includes: the semantic subject contained in the above text information and the relationship between the above semantic subjects; the determining module 905 is used to determine the image corresponding to the above text information according to the above scene data.

[0194] It should be noted that the first receiving module 901, the analyzing module 903 and the determining module 905 correspond to steps S902 to S906 in Example 2, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 2. It should be noted that the above-mentioned modules as part of the device can be run in the computer terminal 10 provided in Example 1.

[0195] It should be noted that the optional or preferred implementations of this embodiment can refer to the relevant descriptions in Embodiments 1 to 4, which will not be repeated here.

[0196] Example 8

[0197] According to an embodiment of the present application, another device embodiment for implementing the above-mentioned method for generating an image is also provided. Fig.10 is a schematic diagram of another image generating device according to an embodiment of the present application, such as Fig.10 As shown, the image generation device 1000 includes: an acquisition unit 1001, a prediction unit 1003 and a first generation unit 1005, wherein:

[0198] An acquisition unit 1001 is used to acquire a scene sequence corresponding to a structured scene graph, wherein the structured scene graph includes: a node for representing a semantic subject and a directed connection relationship between any two nodes; a prediction unit 1003 is used to predict a semantic composition sequence corresponding to the structured scene graph based on the scene sequence, wherein the semantic composition sequence includes: layout parameters of the semantic subject in the semantic composition corresponding to the structured scene graph; a first generation unit 1005 is used to generate the semantic composition based on the semantic composition sequence.

[0199] It should be noted that the acquisition unit 1001, the prediction unit 1003 and the first generation unit 1005 correspond to steps S1002 to S1006 in Example 3, and the three units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 3. It should be noted that the above-mentioned modules as part of the device can be run in the computer terminal 10 provided in Example 1.

[0200] It should be noted that the optional or preferred implementations of this embodiment can refer to the relevant descriptions in Embodiments 1 to 4, which will not be repeated here.

[0201] Example 9

[0202] According to an embodiment of the present application, another device embodiment for implementing the above-mentioned method for generating an image is provided. Fig.11 is a schematic diagram of another image generating device according to an embodiment of the present application, such as Fig.11 As shown, the image generation device 1100 includes: a receiving unit 1101, a determining unit 1103 and a second generating unit 1105, wherein:

[0203] A receiving unit 1101 is used to receive text information, wherein the above text information includes: a semantic subject and the relationship between any two of the above semantic subjects; a determining unit 1103 is used to determine the layout parameters of the above semantic subjects in the image according to the scene sequence corresponding to the above text information; a second generating unit 1105 is used to search for image elements corresponding to the above semantic subjects from an image database, and restore the above image elements to corresponding positions according to the layout parameters of the above semantic subjects, so as to obtain the image corresponding to the above semantic subjects.

[0204] It should be noted that the receiving unit 1101, the determining unit 1103 and the second generating unit 1005 correspond to steps S1102 to S1104 in Example 4, and the three units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 4. It should be noted that the above-mentioned modules as part of the device can be run in the computer terminal 10 provided in Example 1.

[0205] It should be noted that the optional or preferred implementations of this embodiment can refer to the relevant descriptions in Embodiments 1 to 4, which will not be repeated here.

[0206] Example 10

[0207] According to an embodiment of the present application, an embodiment of a computer terminal is also provided, and the computer terminal can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the above-mentioned computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0208] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0209] In this embodiment, the computer terminal can execute the following steps of the program code in the method for generating an image of an application: generating scene data based on the received text information, wherein the scene data includes: the semantic subject contained in the text information and the relationship between any two of the semantic subjects; predicting the image information of the image corresponding to the text information based on the scene sequence used to represent the scene data, wherein the image information includes the layout parameters of the semantic subject in the image; and generating the image corresponding to the text information based on the image information.

[0210] Optionally, Fig.12 is a structural block diagram of a computer terminal according to an embodiment of the present application, such as Fig.12 As shown, the computer terminal 1200 may include: one or more (only one is shown in the figure) processors 1202 , a memory 1204 , and a peripheral interface 1206 .

[0211] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the image generation method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned image generation method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 1200 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0212] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: generate scene data based on the received text information, wherein the above-mentioned scene data includes: the semantic subject contained in the above-mentioned text information and the relationship between any two of the above-mentioned semantic subjects; predict the image information of the image corresponding to the above-mentioned text information based on the scene sequence used to represent the above-mentioned scene data, wherein the above-mentioned image information includes the layout parameters of the above-mentioned semantic subject in the above-mentioned image; generate the image corresponding to the above-mentioned text information based on the above-mentioned image information.

[0213] Optionally, the processor may also execute program code of the following steps: obtaining a scene sequence corresponding to the scene data; based on the scene sequence, predicting an image information sequence according to a sequence-to-sequence neural network, wherein the image information sequence includes a basic action code segment for representing the image information.

[0214] Optionally, the processor may also execute the program code of the following steps: extracting relation triples from the scene data, wherein the relation triples include: a first semantic subject for representing a subject, a second semantic subject for representing a predicate, and a third semantic subject for representing an object; and constructing the scene sequence according to the relation triples.

[0215] Optionally, the processor may further execute program codes of the following steps: setting the bounding box according to the semantic body in the image information and the layout parameters of the semantic body; and constructing the image based on the bounding box.

[0216] Optionally, the processor may also execute program code of the following steps: selecting a semantic subject image corresponding to the semantic subject in the image information from a preset material library; determining the position and size of the semantic subject image according to the bounding box to obtain an image corresponding to the text information.

[0217] Optionally, the processor may also execute the program code of the following steps: obtaining an identification information sequence, wherein the step of obtaining the identification information sequence includes: extracting the relationship triples in the scene data, wherein the relationship triples include: a first semantic subject for representing a subject, a second semantic subject for representing a predicate, and a third semantic subject for representing an object; obtaining an identification corresponding to each semantic subject in the relationship triples; and constructing the identification information sequence according to the identification corresponding to each semantic subject in the relationship triples.

[0218] Optionally, the processor may also execute program code for the following steps: generating an image corresponding to the text information according to the image information and the identification information sequence; the step of generating an image corresponding to the text information according to the image information and the identification information sequence includes: determining a bounding box in the image information for representing the same semantic subject according to the identification information sequence; fusing the bounding boxes for representing the same semantic subject; setting the bounding box according to the semantic subject in the image information and the layout parameters of the semantic subject; and constructing the image based on the bounding box.

[0219] The processor can also call the information and application stored in the memory through the transmission device to execute the following steps: receiving text information; performing semantic analysis on the above text information to obtain scene data corresponding to the above text information, wherein the above scene data includes: the semantic subject contained in the above text information and the relationship between the above semantic subjects; determining the image corresponding to the above text information based on the above scene data.

[0220] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: obtain a scene sequence corresponding to the structured scene graph, wherein the structured scene graph includes: a node for representing a semantic subject and a directed connection relationship between any two nodes; predict a semantic composition sequence corresponding to the structured scene graph based on the scene sequence, wherein the semantic composition sequence includes: layout parameters of the semantic subject in the semantic composition corresponding to the structured scene graph; generate the semantic composition based on the semantic composition sequence.

[0221] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: receiving text information, wherein the above text information includes: a semantic subject and the relationship between any two of the above semantic subjects; determining the layout parameters of the above semantic subjects in the image according to the scene sequence corresponding to the above text information; searching for image elements corresponding to the above semantic subjects from an image database, and restoring the above image elements to corresponding positions according to the layout parameters of the above semantic subjects, so as to obtain the image corresponding to the above semantic subjects.

[0222] According to an embodiment of the present application, a scheme for generating an image is provided. Scene data is generated according to received text information, wherein the scene data includes: a semantic subject contained in the text information and a relationship between any two of the semantic subjects; image information of an image corresponding to the text information is predicted according to a scene sequence used to represent the scene data, wherein the image information includes layout parameters of the semantic subject in the image; and an image corresponding to the text information is generated according to the image information.

[0223] Based on the embodiments of the present application, by adopting a method of learning the generation process of semantic composition rather than directly learning its generation results, the purpose of improving the accuracy of the results of predicting semantic composition based on scene graphs is achieved, thereby achieving a better technical effect of completing the visual tasks of generating images or retrieving images, and further solving the technical problem in the prior art of using scene graphs as inputs to neural network models, resulting in inaccurate results of semantic composition.

[0224] It can be understood by those skilled in the art that Fig.12The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.12 The structure of the electronic device is not limited. For example, the computer terminal 1200 may also include Fig.12 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.12 Different configurations are shown.

[0225] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0226] Embodiment 11

[0227] According to an embodiment of the present application, an embodiment of a storage medium is also provided. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image generation method provided in the above embodiments 1 to 4.

[0228] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0229] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: generating scene data based on the received text information, wherein the above-mentioned scene data includes: semantic subjects contained in the above-mentioned text information and the relationship between any two of the above-mentioned semantic subjects; predicting image information of an image corresponding to the above-mentioned text information based on a scene sequence used to represent the above-mentioned scene data, wherein the above-mentioned image information includes layout parameters of the above-mentioned semantic subjects in the above-mentioned image; and generating an image corresponding to the above-mentioned text information based on the above-mentioned image information.

[0230] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: obtaining a scene sequence corresponding to the above-mentioned scene data; based on the above-mentioned scene sequence, predicting an image information sequence according to a sequence-to-sequence neural network, wherein the above-mentioned image information sequence includes a basic action code segment for representing the above-mentioned image information.

[0231] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: extracting relation triples from the above-mentioned scene data, wherein the above-mentioned relation triples include: a first semantic subject for representing a subject, a second semantic subject for representing a predicate and a third semantic subject for representing an object; and constructing the above-mentioned scene sequence according to the above-mentioned relation triples.

[0232] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: setting the bounding box according to the semantic body in the image information and the layout parameters of the semantic body; and constructing the image based on the bounding box.

[0233] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: selecting a semantic subject image corresponding to the semantic subject in the above-mentioned image information from a preset material library; determining the position and size of the above-mentioned semantic subject image according to the above-mentioned bounding box, and obtaining an image corresponding to the above-mentioned text information.

[0234] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining an identification information sequence, wherein the step of obtaining the identification information sequence includes: extracting relationship triples in the above-mentioned scene data, wherein the above-mentioned relationship triples include: a first semantic subject for representing a subject, a second semantic subject for representing a predicate and a third semantic subject for representing an object; obtaining an identification corresponding to each semantic subject in the above-mentioned relationship triples; and constructing the above-mentioned identification information sequence according to the identification corresponding to each of the above-mentioned semantic subjects in the above-mentioned relationship triples.

[0235] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: generating an image corresponding to the text information according to the image information and the identification information sequence; the steps of generating an image corresponding to the text information according to the image information and the identification information sequence include: determining a bounding box in the image information for representing the same semantic subject according to the identification information sequence; fusing the bounding boxes for representing the same semantic subject; setting the bounding box according to the semantic subject in the image information and the layout parameters of the semantic subject; and constructing the image based on the bounding box.

[0236] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: receiving text information; performing semantic analysis on the above text information to obtain scene data corresponding to the above text information, wherein the above scene data includes: the semantic subject contained in the above text information and the relationship between the above semantic subjects; determining the image corresponding to the above text information based on the above scene data.

[0237] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining a scene sequence corresponding to a structured scene graph, wherein the structured scene graph includes: a node for representing a semantic subject and a directed connection relationship between any two nodes; predicting a semantic composition sequence corresponding to the structured scene graph based on the scene sequence, wherein the semantic composition sequence includes: layout parameters of the semantic subject in the semantic composition corresponding to the structured scene graph; generating the semantic composition based on the semantic composition sequence.

[0238] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: receiving text information, wherein the above text information includes: a semantic subject and a relationship between any two of the above semantic subjects; determining layout parameters of the above semantic subjects in an image according to a scene sequence corresponding to the above text information; searching for image elements corresponding to the above semantic subjects from an image database, and restoring the above image elements to corresponding positions according to the layout parameters of the above semantic subjects, to obtain an image corresponding to the above semantic subject.

[0239] Example 12

[0240] According to an embodiment of the present application, an embodiment of a data processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0241] Under the above operating environment, this application provides Fig.13 The data processing method shown, Fig.13 is a flow chart of a data processing method according to an embodiment of the present application, such as Fig.13 As shown, the above data processing method includes the following steps:

[0242] Step 1302, receiving text information, wherein the text information includes: a semantic subject and a relationship between at least two semantic subjects;

[0243] Step 1304, generating a scene sequence according to the above text information;

[0244] Step 1306: Generate image information corresponding to the text information according to the scene sequence, wherein the image information includes layout parameters of the semantic subject in the image.

[0245] The steps of the above-mentioned embodiments of the present application can be executed by a smart terminal, and in particular, can be a smart home device, such as a smart speaker.

[0246] Specifically, the above text information may be text information received after the device enters a specified function, for example, text information received after the device enters a text-image search function.

[0247] Still combined Figure 3 As shown, the left side records the scene data in the form of scene images. The scene image includes multiple nodes, and there is a directed connection relationship between the nodes, wherein each node is used to represent a semantic subject, and the connection relationship between the nodes is used to represent the relationship between any two semantic subjects.

[0248] As an optional embodiment, the above-mentioned text information can be unstructured semantic information, such as natural language text, for example, a sentence spoken by a child, for example, "A girl is feeding an elephant"; after receiving the voice information, the voice information is subjected to voice recognition to obtain text information, and then the scene data corresponding to the text information can be generated according to the semantic subjects "girl" and "elephant" contained in the text information, and the relationship "feed" between the two semantic subjects "girl" and "elephant"; wherein the above-mentioned semantic subjects can include but are not limited to: a first semantic subject "girl" used to represent the subject, a second semantic subject "feed" used to represent the predicate and a third semantic subject "elephant" used to represent the object.

[0249] Specifically, the scene sequence is a representation of scene data, which is used to abstract the relationship between various semantic subjects in the scene data in a sequence manner. The above scene data is data converted from text information, and the data can be stored in the form of scene images or other forms. The above layout parameters may include parameters such as the position and size of the semantic subject in the image.

[0250] In an optional embodiment, the above-mentioned image information can be predicted according to the scene sequence using a preset sequence-to-sequence neural network model, so that the obtained image information is also in the form of a sequence.

[0251] In an optional embodiment, generating image information corresponding to the text information according to the scene sequence includes:

[0252] Step S13062: Based on the scene sequence and using a sequence-to-sequence neural network, an image information sequence for representing the image information is predicted, wherein the image information sequence includes a basic action code segment for representing the image information.

[0253] Specifically, the input and output of the above sequence-to-sequence neural network are both data represented in a sequence manner, and the sequence-to-sequence neural network learns the actions dragged by the designer during the training process.

[0254] In an optional embodiment, a large number of image elements are obtained (in the initial training, the image elements can be drawn by the designer. As the number of samples increases, the designer does not need to draw new image elements, but can directly use them). The designer drags the image elements corresponding to the semantic subject to the predetermined position according to the scene data. The sequence corresponding to the scene data and the action or position dragged by the designer are used as a sample set, and the sequence-to-sequence neural network can be used for learning.

[0255] In an optional embodiment, the above-mentioned image information sequence includes two groups of basic action code segments, wherein the first group of action code segments is used to represent the layout parameters of the first semantic subject (i.e., the first semantic subject used to represent the subject), and the second group of action code segments is used to represent the layout parameters of the third semantic subject (i.e., the third semantic subject used to represent the object).

[0256] In another optional embodiment, the above-mentioned scene sequence may be a scene sequence composed of relation triplets in the scene data. Based on the scene sequence corresponding to the acquired scene data, an image information sequence can be predicted according to a sequence-to-sequence neural network. Since the image information sequence includes a basic action code segment for representing the above-mentioned image information, the image information of the image (i.e., semantic composition) corresponding to the text information can be predicted.

[0257] As an optional embodiment, after generating the image information corresponding to the text information according to the scene sequence, the method further includes: step S1308, generating an image corresponding to the text information according to the image information.

[0258] Since the semantic body and its layout parameters have been determined in the embodiment of the present application, the corresponding elements can be found according to the semantic body, and then according to the layout parameters of the semantic body, the elements corresponding to the semantic body can be pasted to the positions corresponding to the layout parameters in the preset background image, thereby generating an image corresponding to the above-mentioned text information.

[0259] In an optional embodiment, still combined with Figure 3 As shown, the right part represents the generated image, which has the same relationship between the various semantic subjects expressed in the scene image on the left.

[0260] It should be noted that the image generation method proposed in the embodiment of the present application can be understood as a sequence-to-sequence neural network learning method (sequence-to-sequence learning method) for generating images (i.e., semantic composition) from scene graphs. Specifically, the neural network can be a graph convolutional neural network; the method can be but is not limited to application in children's education fields such as early childhood education and children's intellectual development. For example, scene data can be generated based on the text information received from the child's output, and image information of the image corresponding to the text information can be predicted based on the scene sequence used to represent the scene data, and then the image corresponding to the text information can be generated based on the image information.

[0261] As an optional embodiment, the scene data corresponding to the above text information can be obtained by performing semantic analysis on the above text information, and the above scene data can also include: background information and a bounding box of the image, wherein the bounding box of the image is set according to the semantic subject in the image information of the above image and the layout parameters of the semantic subject (for example, position and size); the background information of the above image can be random, for example, it can be an image element in a preset material library, and the image element can be fixed at the bottom layer of the image as background information and tiled to the entire image.

[0262] In the data processing method provided in the embodiment of the present application, scene data can be first generated based on the received text information, and then based on the scene sequence of the scene data, image information of the image corresponding to the above text information can be predicted, and an image corresponding to the text information is generated based on the image information, wherein the image finally generated corresponding to the text information can be but is not limited to a map.

[0263] The embodiment of the present application learns the generation process of semantic composition rather than directly learning its generation result. In addition, the embodiment of the present application uses unstructured text information as the input of the sequence-to-sequence neural network model to achieve the prediction of semantic composition, and then complete the visual task of generating images or retrieving images, thereby achieving better image generation effect or image retrieval effect.

[0264] In an embodiment of the present application, a method of learning the generation process of semantic composition rather than directly learning its generation result is adopted, by receiving text information, wherein the above-mentioned text information includes: a semantic subject and a relationship between at least two semantic subjects; generating a scene sequence based on the above-mentioned text information; generating image information corresponding to the above-mentioned text information based on the above-mentioned scene sequence, wherein the above-mentioned image information includes layout parameters of the above-mentioned semantic subject in the above-mentioned image.

[0265] Based on the embodiments of the present application, by adopting a method of learning the generation process of semantic composition rather than directly learning its generation results, the purpose of improving the data processing accuracy of generating image information based on the scene sequence corresponding to text information is achieved, thereby achieving a better technical effect of completing the visual task of generating or retrieving images, and further solving the technical problem in the prior art of using scene graphs as inputs of neural network models, resulting in inaccurate semantic composition results.

[0266] It is still necessary to explain that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0267] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the above methods of each embodiment of the present application.

[0268] Embodiment 13

[0269] According to an embodiment of the present application, a device embodiment for implementing the above data processing method is also provided. Fig.14 is a schematic diagram of a data processing device according to an embodiment of the present application, such as Fig.14 As shown, the data processing device 1400 includes: a second receiving module 1402, a third generating module 1404 and a fourth generating module 1406, wherein:

[0270] The second receiving module 1402 receives text information, wherein the text information includes: a semantic subject and a relationship between at least two semantic subjects; the third generating module 1404 is used to generate a scene sequence based on the text information; the fourth generating module 1406 is used to generate image information corresponding to the text information based on the scene sequence, wherein the image information includes layout parameters of the semantic subject in the image.

[0271] It should be noted that the second receiving module 1402, the third generating module 1404 and the fourth generating module 1406 correspond to steps S1302 to S1306 in Example 12, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 12. It should be noted that the above-mentioned modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0272] It should be noted that the optional or preferred implementation of this embodiment can refer to the relevant description in Example 12 and will not be repeated here.

[0273] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0274] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0275] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0276] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0277] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0278] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0279] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating an image, characterized in that: include: Generate scene data according to the received text information, wherein the scene data includes: a semantic subject included in the text information and a relationship between any two of the semantic subjects; Predicting image information of an image corresponding to the text information according to a scene sequence used to represent the scene data, wherein the image information includes layout parameters of the semantic subject in the image; generating an image corresponding to the text information according to the image information; Among them, predicting the image information of the image corresponding to the text information according to the scene sequence used to represent the scene data includes: based on the scene sequence, predicting the image information according to a sequence-to-sequence neural network.

2. The method according to claim 1, characterized in that Based on the scene sequence, the image information is predicted according to a sequence-to-sequence neural network, including: Acquire a scene sequence corresponding to the scene data; Based on the scene sequence, an image information sequence is predicted according to a sequence-to-sequence neural network, wherein the image information sequence includes a basic action code segment for representing the image information.

3. The method according to claim 2, characterized in that Acquiring a scene sequence corresponding to the scene data, including: Extracting a relation triplet from the scene data, wherein the relation triplet includes: a first semantic subject for representing a subject, a second semantic subject for representing a predicate, and a third semantic subject for representing an object; The scene sequence is constructed according to the relationship triples.

4. The method according to claim 3, characterized in that The image information sequence includes two groups of basic action code segments, wherein the first group of action code segments is used to represent the layout parameters of the first semantic body, and the second group of action code segments is used to represent the layout parameters of the third semantic body.

5. The method according to claim 1, characterized in that The layout parameters of the semantic body include: the position and size of the bounding box of the semantic body in the image.

6. The method according to claim 5, characterized in that Generating an image corresponding to the text information according to the image information includes: Setting the bounding box according to the semantic body in the image information and the layout parameters of the semantic body; The image is composed based on the bounding box.

7. The method according to claim 6, characterized in that Composing the image based on the bounding box includes: Selecting a semantic subject image corresponding to the semantic subject in the image information from a preset material library; The position and size of the semantic main image are determined according to the bounding box to obtain an image corresponding to the text information.

8. The method according to claim 5, characterized in that Before generating the image corresponding to the text information according to the image information, the method further includes: acquiring an identification information sequence, wherein the step of acquiring the identification information sequence includes: Extracting a relation triplet from the scene data, wherein the relation triplet includes: a first semantic subject for representing a subject, a second semantic subject for representing a predicate, and a third semantic subject for representing an object; Obtaining an identifier corresponding to each semantic subject in the relationship triple; The identification information sequence is formed according to the identification corresponding to each semantic subject in the relationship triple.

9. The method according to claim 8, characterized in that Generating an image corresponding to the text information according to the image information, including: generating an image corresponding to the text information according to the image information and the identification information sequence; The step of generating an image corresponding to the text information according to the image information and the identification information sequence comprises: Determining, according to the identification information sequence, a bounding box in the image information used to represent a same semantic subject; Merge bounding boxes used to represent the same semantic subject; Setting the bounding box according to the semantic body in the image information and the layout parameters of the semantic body; The image is composed based on the bounding box.

10. A method for generating an image, characterized in that: include: Receive text messages; Performing semantic analysis on the text information to obtain scene data corresponding to the text information, wherein the scene data includes: semantic subjects contained in the text information and the relationship between the semantic subjects; Determine the image corresponding to the text information according to the scene data; Wherein, determining the image corresponding to the text information according to the scene data includes: predicting the scene sequence used to represent the scene data according to a sequence-to-sequence neural network to obtain image information of the image corresponding to the text information, wherein the image information includes layout parameters of the semantic subject in the image; and generating the image corresponding to the text information according to the image information.

11. A method for generating an image, characterized in that: include: Acquire a scene sequence corresponding to a structured scene graph, wherein the structured scene graph includes: a node for representing a semantic subject and a directed connection relationship between any two nodes; Predicting a semantic composition sequence corresponding to the structured scene graph according to the scene sequence, wherein the semantic composition sequence includes: layout parameters of the semantic subject in the semantic composition corresponding to the structured scene graph; generating the semantic composition according to the semantic composition sequence; Among them, predicting the semantic composition sequence corresponding to the structured scene graph according to the scene sequence includes: based on the scene sequence, predicting the semantic composition sequence according to a sequence-to-sequence neural network.

12. A method for generating an image, characterized in that: include: Receiving text information, wherein the text information includes: a semantic subject and a relationship between any two of the semantic subjects; Determining layout parameters of the semantic subject in the image according to the scene sequence corresponding to the text information; Searching for image elements corresponding to the semantic subject in an image database, and restoring the image elements to corresponding positions according to layout parameters of the semantic subject, to obtain an image corresponding to the semantic subject; Wherein, determining the layout parameters of the semantic subject in the image according to the scene sequence corresponding to the text information includes: based on the scene sequence, predicting the layout parameters according to a sequence-to-sequence neural network.

13. A system for generating an image, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Generate scene data according to the received text information, wherein the scene data includes: a semantic subject included in the text information and a relationship between any two of the semantic subjects; Predicting image information of an image corresponding to the text information according to a scene sequence used to represent the scene data, wherein the image information includes layout parameters of the semantic subject in the image; generating an image corresponding to the text information according to the image information; Among them, predicting the image information of the image corresponding to the text information according to the scene sequence used to represent the scene data includes: based on the scene sequence, predicting the image information according to a sequence-to-sequence neural network.

14. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the image generation method according to any one of claims 1 to 12.

15. A processor, characterized in that: The processor is used to run a program, wherein the program executes the image generation method described in any one of claims 1 to 12 when running.

16. A data processing method, characterized in that: include: Receiving text information, wherein the text information includes: a semantic subject and a relationship between at least two semantic subjects; Generate a scene sequence according to the text information; Generate image information corresponding to the text information according to the scene sequence, wherein the image information includes layout parameters of the semantic subject in the image; Among them, generating image information corresponding to the text information according to the scene sequence includes: based on the scene sequence, predicting the image information according to a sequence-to-sequence neural network.

17. The method according to claim 16, characterized in that Generating image information corresponding to the text information according to the scene sequence, including: Based on the scene sequence, predicting an image information sequence for representing the image information according to a sequence-to-sequence neural network, wherein the image information sequence includes a basic action code segment for representing the image information; After generating the image information corresponding to the text information according to the scene sequence, the method further includes: generating the image corresponding to the text information according to the image information.

Images