Image generation method for maintaining position relationship of text content

Abstract

The present application relates to a kind of image generation method for keeping the position relationship of text content, the keeping ability of generation model for target position relationship is improved from two aspects of training data and loss function, and the generation result of model is verified by constructing quantitative evaluation standard.By the position relationship constraint added in different stages of training, the expression ability of generation model for the semantic information of position relationship in text content is enhanced, so as to help model understand the relationship between text content and image features from higher semantic level, improve the content layout rationality and semantic consistency of generated image.

Description

Technical Field

[0001] This invention belongs to the fields of local security, computer vision, artificial intelligence, and multimodal learning, and specifically relates to an image generation method that preserves the semantic meaning of the positional relationship of text content. Background Technology

[0002] Images, as an important medium for information dissemination, are a crucial way for humans to understand the world. Computer vision applications in engineering primarily involve constructing acquisition devices and analysis algorithms to endow computers with the ability to perceive their environment, showing broad application prospects in on-site security fields, including intelligent monitoring and underwater security. However, establishing a comprehensive intelligent security system requires a large amount of real image data. Limited by acquisition costs, it is difficult to obtain large amounts of image data for specific scenarios. Therefore, some researchers have proposed using small-scale raw image data to apply generation techniques for data augmentation. However, current unconditional generation algorithms can only produce results similar to the characteristics of the original image data, unable to further control the content of the generated images according to human will. How to achieve conditional and precise control over the content of generated images is a key problem that urgently needs to be solved. Compared to images, text is an easily accessible information carrier. Generating images containing specific content based on textual descriptions can alleviate the problem of uncontrolled generated image content to some extent. However, a semantic gap exists between text and images. How to establish a correspondence between text and image content, so that the generated images possess both realism and semantic consistency, are two challenges currently faced in research.

[0003] To reduce the difficulty of text-to-image generation, the paper "T.Xu,P.Zhang,Q.Huang,H.Zhang,Z.Gan,X.Huang,andX.He,AttnGAN:Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks,IEEE Conference on ComputerVision and Pattern Recognition,1316-1324,2018" breaks down the process of generating images from text content into two parts. First, a deep attention mechanism multimodal similarity model is used to understand and learn the correspondence between text semantics and image features. Second, the learned semantic information is input into a multi-stage adversarial generative network to progressively improve the resolution of the generated images. Compared with a single-stage generative model, this method effectively enhances the quality of the generated images and improves the stability of the model training process.

[0004] To enhance the semantic consistency between the generated image and the input text, the paper "W.Li, P.Zhang, L.Zhang, Q.Huang, X.He, S.Lyu, and J.Gao, Object-Driven Text-to-Image Synthesis via Adversarial Training, IEEE Conference on Computer Vision and Pattern Recognition, 12174-12182, 2019" breaks down the image generation process into different dimensions. First, a bounding box generator and a shape generator are used to obtain a semantic layout map containing object category and location information based on the given text information. Then, the details of the semantic layout map are supplemented based on global text features and local word features. This method achieves good generation results when there are labeled object bounding boxes and shape masks. However, in practice, generating images using only text as input still cannot fully express the visual representation of the object types and positional relationships mentioned in the text.

[0005] While both methods described above achieve the task of generating images from text, they both have certain limitations. Specifically, the semantic expression of the text description remains at a relatively shallow level. The models can only reflect superficial semantic information mentioned in the text, such as color, texture, and type, in the generated images. The generated images are more like simple combinations of different sub-regions, and higher-level semantic information such as quantity and positional relationships cannot be reflected in the generated image content. The lack of high-level semantic information in the conversion process also greatly limits the practical application scenarios of text-to-image generation. Promoting the reasonable expression of high-level semantic information in generated images is a major challenge currently facing researchers. Summary of the Invention

[0006] Technical problems to be solved

[0007] To overcome the problem of insufficient expression of high-level semantic information contained in text in existing text-to-image generation practices, this invention proposes an image generation method that preserves the positional relationships of text content.

[0008] Technical solution

[0009] A method for generating images while preserving the positional relationships of text content, characterized by the following steps:

[0010] Step 1: Reorganize the training data

[0011] 1a: Align the object position information in the training data with the text content;

[0012] 1b: Distillation is performed on training data that does not contain textual descriptions of positional relationships;

[0013] 1c: Model the object position information contained in the semantic graph. The descriptive information of objects in the coarse semantic graph layout consists of two parts: image coordinate information that determines the placement position and category information that reflects the group attributes of the objects; as shown in formula (1), through a four-dimensional vector b t Let b represent the position of the object with index t on the image, where b t,x ,b t,y These represent the horizontal and vertical pixel coordinates of the object on the 2D image, with the top-left corner as the origin. t,w ,b t,h These represent the pixel length and width of the location box on the two-dimensional image, respectively; as shown in formula (2), l t It is an L+1 dimensional vector using one-hot encoding, whose elements consist of 0 or 1, with only one digit being 1 and the rest being 0. The position of 1 represents the category represented by the vector, L represents the number of object categories in the dataset, and the L+1 class code is set as a terminator as the termination symbol of the category sequence of a single image; as shown in formula (3), the constraint information B composed of the two vectors t The position of the object with index t on the semantic layout diagram can be uniquely determined;

[0014] b t =[b t,x ,b t,y ,b t,w ,b t,h (1)

[0015] l t ∈{0,1} L+1 (2)

[0016] B t =(b t ,l t (3)

[0017] 1d: The order of objects in each image input into the prediction model is rearranged according to certain rules; the arrangement rules are as follows: calculate the center point coordinates of the objects according to the marked position box information, and arrange them incrementally from left to right according to the two-dimensional plane coordinates of the image to obtain the position information of objects with positional proximity dependence.

[0018] Step 2: Train the object layout generation module that preserves positional semantics

[0019] 2a: Construct an object layout generation module that preserves positional semantics. The function of this module is to generate a list of object position information that reflects positional semantics based on the input text constraints. The processing of the module is shown in formula (4). The input is a 256-dimensional text feature vector s, which is extracted from the input natural language description by a pre-trained long short-term memory network. box This represents a bidirectional long short-term memory network; assuming each image contains objects numbered 1-T, the object layout generation module needs to generate positional constraint information with numbers 1-T based on text features. As shown in formula (5), based on the properties of LSTM, the output of each unit depends on the output state of the previous unit and the input of the current unit, and the position constraint information B of the object with index t. t The generation depends on the position constraint information B generated in the previous t-1 generation. 1:t-1 And text features s; positional constraint information B of objects numbered 1-T 1:T The conditional probability distribution p(B) for text feature s 1:T |s) can be decomposed into a series of product of the probability distributions generated in the first T-1 iterations.

[0020]

[0021]

[0022] 2b: Vector l representing the object type information with index T t The vector b representing the location information constraint t It has a strong dependency relationship, and the type information of the object to be generated must be determined first during the generation process; as shown in formula (6), in each generation process, a category label l is first sampled from the input text constraints. t Then, based on the category label, predict its corresponding location information b. t The generation of positional relationship information is divided into two parts: the generation of category labels and the generation of corresponding object position descriptions. Both steps use LSTM structure. The generation result of the current stage depends on the output of the previous stage. · represents the generation results of all previous steps. The results are normalized using the Softmax function.

[0023] p(B t |·)=p(b t ,l t ∣·)=p(l t ∣·)p(b t |l t ,·). (6)

[0024] 2c: Design a position-sensitive loss function, as shown in formula (7), B t This represents the positional constraint information in the training data. This represents the position constraint information output by the layout generation module. The image coordinates of the center point of the object are calculated based on the position constraint information, where c t,x c is the horizontal coordinate. t,y For the vertical coordinates; as shown in formula (8), the horizontal cosine value between the two points is used to describe their planar positional relationship, where This represents the coordinates of the center point of the object with index i. This represents the coordinates of the center point of the object with index j; as shown in formula (9). The value of the loss function when generating position constraint information for object t is based on the center point coordinates calculated from the dataset annotation information for object i. The average of the horizontal cosine differences between the object with index t and the preceding t-1 objects is calculated pairwise.

[0025]

[0026]

[0027]

[0028] Step 3: Input the training data obtained in Step 1 into the object layout generation module that preserves positional relationship semantics in Step 2 to obtain a semantic layout map. Input the semantic layout map into the subsequent generation module to complete the expression of positional relationship semantics in the final generated image content.

[0029] Step 1a specifically involves extracting nouns from the descriptive text to obtain a list of generated objects contained therein, and then finding the intersection of this list with the list of objects labeled in the original data.

[0030] Step 1b specifically involves: establishing a positional relationship dictionary and searching and filtering the text descriptions.

[0031] A computer system is characterized by comprising: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method described above.

[0032] A computer-readable storage medium is characterized by storing computer-executable instructions, which, when executed, are used to implement the above-described method.

[0033] Beneficial effects

[0034] This invention proposes an image generation method that preserves the positional relationships within text content. It enhances the generative model's ability to maintain target positional relationships by addressing both training data and the loss function, and verifies the model's generation results through the construction of quantitative evaluation criteria. By incorporating positional relationship constraints at different training stages, the generative model's ability to express semantic information about positional relationships within text content is strengthened. This helps the model understand the relationship between text content and image features at a higher semantic level, improving the rationality of the generated image's content layout and semantic consistency. Attached Figure Description

[0035] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0036] Figure 1 This is a flowchart illustrating how the method of the present invention reorganizes training data based on positional relationships.

[0037] Figure 2 This is a flowchart of the spatially sensitive loss function calculation in the method of this invention.

[0038] Figure 3 This is the overall framework diagram of the semantic image generation model that preserves the positional relationship of text content according to the method of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0040] This invention provides an image generation method that preserves the semantic meaning of positional relationships within text content. Based on generative adversarial networks (GANs), this method introduces positional relationship constraints at multiple steps in the generation process to ensure that the positional relationships mentioned in the text are fully expressed in the generated image content. 1) Regarding data preparation, unlike existing models that randomly sample the objects to be generated, this method performs two operations on the original training data: semantic annotation distillation and positional relationship organization. The aim is to enhance the positional relationship semantic information contained in the training data. 2) During the training phase, in addition to probabilistic modeling of the training data and the generated results to standardize model training, a positional relationship-sensitive loss function is introduced to further supervise the expression of positional information during the generation process. 3) In terms of validating the model's effectiveness, a pre-trained object detection model is used to quantitatively evaluate the positional relationships of objects expressed in the generated image, thereby reflecting the effect of the added positional relationship constraints on the high-level semantic expression of the image.

[0041] The technical solution adopted by this invention to solve its technical problem is: an image generation method that preserves the semantic meaning of the positional relationship of text content, comprising the following parts:

[0042] a) Training Data Organization Stage: Compared to images, text has inherent ambiguity and semantic vagueness. Therefore, in the process of converting text information into image information, we need to extract key content from the text step by step and supervise and correct semantic information at each stage of image generation. This method reorganizes the text descriptions and object segmentation annotations in the original MS COCO dataset in two dimensions. First, data distillation is performed on all matching data pairs to enhance the model's understanding of location information and reduce interference from irrelevant information during model convergence. Specifically, text-image matching pairs that do not contain location relationship descriptions or have vague descriptions are removed to prevent the model from learning incorrect location relationships during training and reducing training difficulty. Second, the generation task is split into two parts: generating a semantic layout map and generating realistic images. The former is the foundation for ensuring the correct propagation of location relationship information in subsequent stages. To generate a correct semantic layout map, this method rearranges the model's input based on the annotation information to assign proximity dependencies to the location relationships of different objects.

[0043] b) Model Training Phase: This method breaks down the text-to-image generation task into two phases. First, a semantic layout map containing positional relationship information is generated based on the input text description. Second, the positional relationships and object content of the semantic layout map are supplemented and refined to generate a realistic image. By replicating the generative models of other researchers, it was found that when using the semantic layout information from the original training data for image generation, clear content and reasonable object arrangement can often be obtained, demonstrating the significant impact of obtaining correct semantic layout information on the final generation effect. To improve the rationality of the semantic expression of text positional relationships in the layout map, this invention introduces a spatial relationship-sensitive loss function in the model training phase. This function maintains the expression of positional relationships during the transformation process by constraining the positional relationships between the generated result and different targets in the training data. Specifically, the difference in the cosine angle between the two targets in the generated result and the training data is minimized to help the model learn to reflect spatial association semantics in the text content. Then, the semantic relationship map enhanced with positional relationships is input into the image refinement generation module to complete the expression of text semantics in the real image content.

[0044] c) Model Evaluation Phase: A pre-trained object detection model is used to process the generated images to obtain the types of objects contained in the images and the positional relationships between them. The extracted results are compared with the labeled information in the dataset, and the proportion of successfully expressed positional relationship semantic information is calculated, thus completing a quantitative evaluation of the model's ability to transform positional relationship semantic information.

[0045] To enable those skilled in the art to better understand the present invention, the present invention will be described in detail below with reference to specific embodiments.

[0046] Reference Figure 1 The process for reorganizing training data is as follows:

[0047] Step 0: Align the object location information in the training data with the text content. The purpose is to remove the location information of objects not mentioned in the text description and enhance the consistency between the annotation information and the text semantics. Specifically, this involves extracting nouns from the descriptive text to obtain a list of generated objects, and then finding the intersection of this list with the list of annotated objects in the original data.

[0048] Step 1: Distill the training data that does not contain textual descriptions of positional relationships to enhance the model's ability to learn positional relationships. Specifically, this involves creating a dictionary of commonly used positional relationships and searching and filtering the textual descriptions.

[0049] Step 2: Model the object position information contained in the semantic graph. The descriptive information of objects in the coarse semantic graph layout mainly consists of two parts: image coordinate information that determines the placement position and category information that reflects the group attributes of the objects. As shown in formula (1), taking the object with index t as an example, we use a four-dimensional vector b t To represent its position on the image, where b t,x ,b t,y These represent the horizontal and vertical pixel coordinates of the object on the 2D image, with the top-left corner as the origin. t,w ,b t,h These represent the pixel length and width of the bounding box on the two-dimensional image, respectively. As shown in formula (2), l t It is an L+1 dimensional vector using one-hot encoding, whose elements consist of 0 or 1, with only one digit being 1 and the rest being 0. The position of 1 represents the category represented by the vector, L represents the number of object categories in the dataset, and the L+1 class code is set as a terminator as the termination symbol for the category sequence of a single image. As shown in formula (3), the constraint information B composed of the two vectors t The position of the object with index t on the semantic layout map can be uniquely determined.

[0050] b t =[b t,x ,b t,y ,b t,w ,b t,h (1)

[0051] l t ∈{0,1} L+1 (2)

[0052] B t =(b t ,l t (3)

[0053] Step 3: The original location information is randomly distributed and lacks dependencies between objects, causing the generated results to ignore the arrangement of object positions. To assign proximity dependencies to different target positions, this method rearranges the order of objects input into the prediction model in each image according to certain rules. The arrangement rules are as follows: Calculate the center point coordinates of the objects based on the labeled bounding box information, and arrange them incrementally from left to right according to the two-dimensional plane coordinates of the image, thus obtaining the object position information with proximity dependencies.

[0054] Reference Figure 2 The training process of the object layout generation module that preserves positional semantics is as follows:

[0055] Step 0: Construct an object layout generation module that preserves positional semantics. This module generates a list of object position information that reflects positional semantics based on the input text constraints. The module's processing is shown in formula (4). The input is a 256-dimensional text feature vector s, which is extracted from the input natural language description by a pre-trained Long Short-Term Memory (LSTM) network. box This represents a Bi-directional Long Short-Term Memory (Bi-LSTM) network. Assuming each image contains objects numbered 1-T, the object layout generation module needs to generate positional constraint information with numbers 1-T based on text features. As shown in formula (5), based on the properties of LSTM, the output of each unit depends on the output state of the previous unit and the input of the current unit, and the position constraint information B of the object with index t. t The generation depends on the position constraint information B generated in the previous t-1 generation. 1:t-1 And text features s. Position constraint information B of object number 1-T 1:T The conditional probability distribution p(B) for text feature s 1:T |s) can be decomposed into a series of product of the probability distributions generated in the first T-1 iterations.

[0056]

[0057]

[0058] Step 1: Vector l representing the object type information with index T t The vector b representing the location information constraint t Due to strong dependencies, the type of object to be generated must be determined first during the generation process. As shown in formula (6), in each generation process, a category label l is first sampled from the input text constraints. t Then, based on the category label, predict its corresponding location information b. t The generation of positional relationship information is divided into two parts: the generation of category labels and the generation of corresponding object position descriptions. Both steps use an LSTM structure. The generation result of the current stage depends on the output of the previous stage. · represents the generation results of all previous steps. The results are normalized using the Softmax function.

[0059] p(B t |·)=p(b t ,l t ∣·)=p(l t∣·)p(b t |l t ,·). (6)

[0060] Step 2: To ensure that the object layout generation module not only conforms to the conditional probability distribution of the training data but also reflects the positional relationship semantics in the input text description, this method adds a positional relationship-sensitive loss function to the original objective function. As shown in formula (7), taking the object with index t as an example, B t This represents the positional constraint information in the training data. This represents the position constraint information output by the layout generation module. The image coordinates of the center point of the object are calculated based on the position constraint information, where c t,x c is the horizontal coordinate. t,y is the vertical coordinate. As shown in formula (8), this method uses the horizontal cosine value between two points to describe their planar positional relationship, where This represents the coordinates of the center point of the object with index i. This represents the coordinates of the center point of the object with index j. As shown in formula (9), The value of the loss function when generating position constraint information for object t is based on the center point coordinates calculated from the dataset annotation information for object i. It is the average of the horizontal cosine differences between the object with index t and the preceding t-1 objects, calculated in pairs.

[0061]

[0062]

[0063]

[0064] Reference Figure 3 The subsequent generation module refines the semantic layout map into real images, and then quantitatively evaluates the model's ability to express the semantics of positional relationships.

[0065] Step 0: After obtaining the target list containing location information, the positional relationships between target objects in the desired generated image can be obtained, and a corresponding semantic layout map can be constructed. This semantic layout map is then input into the subsequent generation module to complete the expression of positional relationship semantics in the final generated image content. The subsequent generation model structure is constructed based on the literature "S. Hong, D. Yang, J. Choi, and H. Lee, Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis, in IEEE Conference on Computer Vision and Pattern Recognition, pp. 7986-7994, 2018." Finally, the generated semantic layout map is supplemented and refined to obtain the real image.

[0066] Step 1: Based on the evaluation method for object types in generated images proposed in the literature "T.Hinz, S.Heinrich, and S.Wermter, Semantic Object Accuracy for Generative Text-to-Image Synthesis, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.44, no.3, pp.1552-1565, 2022," a quantitative evaluation method for positional relationships in descriptive text is constructed. The pre-trained object detection model YOLOv3 is used to detect targets in the generated images, and elements in the target list are paired to determine if their positional relationships are the same as those in the training data. Finally, the accuracy of positional relationship expression (SLRA, Semantic Location Relationship Accuracy) is calculated. The calculation process is shown in formulas (10) and (11), where N represents the number of targets detected by the object detection model. R represents the positional relationship between object i and object j in the generated result. i,j This represents the positional relationship between objects i and j in the dataset. If their positional relationships are the same, then... It is 1 if it is true, otherwise it is 0.

[0067]

[0068]

[0069] To illustrate the specific implementation of this invention in detail, the MS COCO image dataset will be used as an example. This dataset is applied to multiple application scenarios such as object detection, segmentation, and image description. It contains over 100,000 labeled images, with 118,287 images in the training set, 40,670 images in the test set, and 5,000 images in the validation set. The dataset contains 80 object categories. The annotation information for each image includes object-level instance segmentation and category labels, as well as five text descriptions from different perspectives. In this invention, the 118,287 images and their corresponding annotation information from the original data segmentation are used as training data, and then 5,000 images and their annotation information are used to test and verify the model's performance.

[0070] Step 0: Align the original labeled data according to the list of nouns extracted from the text content, preserving the bounding box information of the objects mentioned in the text. The original dataset contains 410,622 data sets, which are then aligned to obtain 209,554 training data sets;

[0071] Step 1: Distill the raw data based on positional relationships. After segmenting the descriptive text, extract locative words. Candidate locative words include above, below, left, right, on, over, under, between, in front of, next to, near, etc. After distillation, 161,697 sets of training data containing locative words describing positional relationships are obtained.

[0072] Step 2: Train the semantic layout information generation model based on the matching pairs of text description and object position description information, and introduce a position-sensitive position relationship loss function to help the model converge.

[0073] Step 3: Based on the generative model proposed in the literature "S.Hong, D.Yang, J.Choi, and H.Lee, Inferring SemanticLayout for Hierarchical Text-to-Image Synthesis, in IEEE Conference on Computer Vision and Pattern Recognition, pp.7986-7994, 2018," the semantic layout graph is finally transformed into a real image.

[0074] Step 4: After the model training phase is complete, the model's generation performance is tested using descriptive text from the test set. The model's generation results are evaluated in two aspects: the quality of the generated image and the semantic consistency between the generated image and the descriptive text. The quality evaluation of the generated image is mainly achieved by calculating its IS index, which examines the class distinguishability of targets in the generated image. The evaluation of the semantic consistency of the generated image is mainly achieved by calculating the accuracy of positional relationship expression proposed in this invention. In practice, since the current model's ability to express high-level semantics is still relatively poor, some positional relationships are difficult to judge in two-dimensional images. Therefore, four relatively simple positional relationships—above, below, left, and right—are used when constructing the evaluation system. Finally, the above two evaluation indicators are combined to verify the effectiveness of the proposed method in improving the quality of text-to-image generation and the semantic expression of positional relationships.

[0075] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the scope of the technology disclosed in the present invention, and such modifications or substitutions should all be covered within the scope of protection of the present invention.

Claims

1. An image generation method that preserves the positional relationships of text content, characterized in that... The steps are as follows: Step 1: Reorganize the training data 1a: Align the object position information in the training data with the text content; 1b: Distillation is performed on training data that does not contain textual descriptions of positional relationships; 1c: Model the object position information contained in the semantic graph. The descriptive information of objects in the coarse semantic graph layout consists of two parts: image coordinate information that determines the placement position and category information that reflects the group attributes of the objects; as shown in formula (1), through a four-dimensional vector To represent the serial number t The position of the object in the image, where , These represent the horizontal and vertical pixel coordinates of the object on the 2D image, with the top-left corner as the origin. , These represent the pixel length and width of the location box on the two-dimensional image, respectively; As shown in formula (2), It is a one-hot encoded algorithm. L A +1 dimensional vector whose elements consist of either 0 or 1, with only one digit being 1 and the rest being 0. The position of the 1 represents the category that the vector represents. L This represents the number of object types in the dataset. L +1 class coding is set as a terminator as the termination symbol for a single image class sequence; as shown in formula (3), the constraint information consists of two vectors. The serial number can be uniquely determined as t The position of the object on the semantic layout graph; (1) (2) (3) 1d: The order of objects in each image input into the prediction model is rearranged according to certain rules; the arrangement rules are as follows: calculate the center point coordinates of the objects according to the marked position box information, and arrange them incrementally from left to right according to the two-dimensional plane coordinates of the image to obtain the position information of objects with positional proximity dependence. Step 2: Train the object layout generation module that preserves positional semantics 2a: Construct an object layout generation module that preserves positional semantics. The function of this module is to generate a list of object position information that reflects positional semantics based on the input text constraints. The processing of the module is shown in formula (4). The input is a 256-dimensional text feature vector s, which is extracted from the input natural language description by a pre-trained long short-term memory network. This represents a bidirectional long short-term memory network; assuming each image contains an index 1- T The object layout generation module needs to generate an object with a sequence number of 1- based on the text features. T Position constraint information As shown in formula (5), based on the properties of LSTM, the output of each unit depends on the output state of the previous unit and the input of the current unit. t Position constraint information of the object The generation depends on the previous t -1 generated position constraint information and text features ; Serial number 1- T Object position constraint information Targeting text features conditional probability distribution It can be broken down into the first T -1 product of generation probability distributions ; (4) (5) 2b: indicates that the serial number is T Vector of object type information and the vector representing positional information constraints It has strong dependencies, and the type of object to be generated must be determined first during the generation process; As shown in formula (6), in each generation process, a category label is first sampled from the input text constraints. Then, based on the category label, predict its corresponding location information. The generation of location relationship information is divided into two parts: the generation of category labels and the generation of corresponding object location descriptions. Both steps use an LSTM structure, and the generation result of the current stage depends on the output of the previous stage. This represents the result generated from all previous steps, and the result was normalized using the Softmax function. (6) 2c: Design a position-relationship-sensitive loss function, as shown in formula (7). This represents the positional constraint information in the training data. This represents the position constraint information output by the layout generation module. The coordinates of the center point of the object are calculated based on the position constraint information, where The coordinates are in the horizontal direction. For the vertical coordinates; as shown in formula (8), the horizontal cosine value between the two points is used to describe their planar positional relationship, where The representative serial number is i The coordinates of the center point of the object The representative serial number is j The coordinates of the center point of the object; as shown in formula (9), The representative serial number is i The center point coordinates of the object are calculated based on the annotation information in the dataset, and the generated sequence number is... t When considering object position constraint information, the value of the loss function For serial number t The object and the previously generated t -The average of the horizontal cosine differences calculated by pairwise pairing of 1 object: (7) (8) (9) Step 3: Input the training data obtained in Step 1 into the object layout generation module that preserves positional relationship semantics in Step 2 to obtain a semantic layout map. Input the semantic layout map into the subsequent generation module to complete the expression of positional relationship semantics in the final generated image content.

2. The image generation method for maintaining the positional relationship of text content according to claim 1, characterized in that: Step 1a specifically involves extracting nouns from the descriptive text to obtain a list of generated objects contained therein, and then finding the intersection of this list with the list of objects labeled in the original data.

3. The image generation method for maintaining the positional relationship of text content according to claim 1, characterized in that: Step 1b specifically involves: establishing a positional relationship dictionary and searching and filtering the text descriptions.

4. A computer system, characterized in that... include: One or more processors, a computer-readable storage medium for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of claim 1.

5. A computer-readable storage medium, characterized in that... The device stores computer-executable instructions, which, when executed, are used to implement the method of claim 1.

Images