Training set determination method and device, image generation method and device, equipment and medium

Abstract

The invention provides a training set determination method and device, an image generation method and device, equipment and a medium, and the method comprises the steps: extracting plane position parameters and depth position parameters of a plurality of first targets included in a first image sample, and determining a depth relation description text based on the depth position parameters of the plurality of first targets, and determining a plane relation description text based on the plane position parameters of the plurality of first targets, and obtaining a first text description sample based on the depth relation description text, the plane relation description text and the initial text description of the first image sample. The method not only can reduce the marking cost of the image generation model training set, but also can improve the understanding ability of the image generation model to the spatial relationship, and ensures that the generated image accurately reflects the spatial relationship to be expressed by the input text.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a training set determination method, an image generation method, a device, a equipment and a medium. Background Art

[0002] In the related art, the spatial relationship understanding ability of the image generation model based on text description is relatively limited, which makes it easier for errors to occur in the spatial relationship between different objects in the images generated by the image generation model. Summary of the invention

[0003] According to one aspect of the present disclosure, a method for determining a training set is provided, wherein the image generation model training set includes at least a first image sample and a first text description sample matching the first image sample, and the method includes:

[0004] Extracting a plurality of plane position parameters of the first targets and a plurality of depth position parameters of the first targets included in the first image sample;

[0005] Determine a depth relationship description text based on a plurality of depth position parameters of the first targets;

[0006] Determine the plane relationship description text based on the plane position parameters of the plurality of first targets;

[0007] The first text description sample is obtained based on the depth relationship description text, the plane relationship description text and the initial text description of the first image sample.

[0008] According to another aspect of the present disclosure, there is provided an image generating method, comprising:

[0009] Get target text information;

[0010] Inputting target text information into an image generation model to obtain a target image matching the text information;

[0011] The training set of the image generation model in the training phase is generated and determined by the method described in the exemplary embodiment of the present disclosure.

[0012] According to another aspect of the present disclosure, a training set determination device is provided, wherein the training set is a training set of an image generation model, and the training set at least includes a first image sample and a first text description sample matching the first image sample, and the device includes:

[0013] An extraction module, used for extracting a plurality of plane position parameters of the first targets and a plurality of depth position parameters of the first targets included in the first image sample;

[0014] A determination module, configured to determine a depth relationship description text based on a plurality of depth position parameters of the first targets, and to determine a plane relationship description text based on a plurality of plane position parameters of the first targets;

[0015] An acquisition module is used to obtain the first text description sample based on the depth relationship description text, the plane relationship description text and the initial text description of the first image sample.

[0016] According to another aspect of the present disclosure, there is provided an image generating device, comprising:

[0017] An acquisition module is used to acquire target text information;

[0018] An inference module, used for inputting target text information into an image generation model to obtain a target image matching the text information;

[0019] The training set of the image generation model in the training phase is determined by the method described in the exemplary embodiment of the present disclosure.

[0020] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0021] processor; and,

[0022] A memory for storing programs;

[0023] The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to the exemplary embodiment of the present disclosure.

[0024] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium is used to store computer instructions, and the computer instructions are used to cause the computer to execute the method according to the exemplary embodiments of the present disclosure.

[0025] One or more technical solutions provided in the exemplary embodiments of the present disclosure can extract the plane position parameters of multiple first targets and the depth position parameters of multiple first targets included in the first image sample, and then determine the depth relationship description text based on the depth position parameters of the multiple first targets, determine the plane relationship description text based on the plane position parameters of the multiple first targets, and finally obtain the first text description sample based on the depth relationship description text, the plane relationship description text and the initial text description of the image sample. It can be seen that the exemplary embodiments of the present disclosure can introduce the three-dimensional spatial relationship contained in the first image sample into the initial text description corresponding to the first image sample in the form of depth relationship description text and plane relationship description text, so that the obtained first text description sample can reflect both the initial text description corresponding to the first image sample and the spatial relationship of multiple first targets included in the first image sample.

[0026] It can be seen that the method of the exemplary embodiment of the present disclosure can extract and process information from the first image sample, and obtain a training set of an image generation model constructed by an image-text formed by the first image sample and the first text description sample. Therefore, the image generation model of the exemplary embodiment of the present disclosure does not need to be annotated in large quantities, which can save annotation costs and manpower investment costs. Since the first text description sample can reflect both the initial text description corresponding to the first image sample and the spatial relationship of multiple first targets included in the first image sample, when the image generation model is trained or adjusted with the training set of the image generation model, the first image sample can be used as supervision information and the first text description sample can be used as input to improve the image generation model's ability to understand spatial relationships, thereby ensuring that the trained image generation model outputs an image that can more accurately reflect the spatial relationship of each target represented by the text description content when the spatial information of the input text description content is fuzzy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0028] Figure 1 A schematic diagram showing a flow chart of a training set determination method according to an exemplary embodiment of the present disclosure;

[0029] Figure 2 A schematic diagram of a process of determining a depth relationship description text according to an exemplary embodiment of the present disclosure is shown;

[0030] Figure 3 A schematic diagram showing the depth relationship of different targets of an exemplary embodiment of the present disclosure is shown;

[0031] Figure 4 A schematic diagram of a determination process of a plane relationship description text according to an exemplary embodiment of the present disclosure is shown;

[0032] Figure 5 A schematic diagram showing the distribution of deflection parameter intervals of an exemplary embodiment of the present disclosure is shown;

[0033] Figure 6 A schematic diagram showing a visualization representation of two plane position parameters of the second target according to an exemplary embodiment of the present disclosure is shown;

[0034] Figure 7 A schematic flow chart of an image generating method provided by an exemplary embodiment of the present disclosure is shown;

[0035] Figure 8 A schematic block diagram of functional modules of a training set determination device according to an exemplary embodiment of the present disclosure is shown;

[0036] Fig. 9 A schematic block diagram of functional modules of an image generating device according to an exemplary embodiment of the present disclosure is shown;

[0037] Fig.10 A schematic block diagram of a chip according to an exemplary embodiment of the present disclosure is shown;

[0038] Fig.11 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0039] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0040] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0041] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0042] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0043] An exemplary embodiment of the present disclosure provides a method for determining a training set, wherein the determined training set may be a training set of an image generation model. Using the training set of the image generation model to train or adjust the image generation model can not only save the annotation cost of the image generation model, but also improve the image generation model's ability to understand spatial relationships, thereby ensuring that the trained image generation model can output an image that reflects the real spatial relationship of each target contained in the text description content when the spatial information of the input text description content is fuzzy.

[0044] In practical applications, training or adjusting the image generation model with the training set of the image generation model of the exemplary embodiment of the present disclosure can improve the image generation model's ability to understand spatial relationships. By fine-tuning the trained text generation model with the image generation model training set, the training speed of the image generation model with better spatial relationship understanding ability can be shortened, thereby improving training efficiency.

[0045] Exemplarily, the architecture of the image generation model of the exemplary embodiment of the present disclosure can be various models for generating images based on text, such as: generative adversarial networks, variational autoencoders, diffusion models, cross-modal models, etc., but are not limited to these, and the required model architecture can be selected according to actual conditions.

[0046] Exemplarily, the training set of the exemplary embodiment of the present disclosure includes at least a first image sample and a first text description sample matching the first image sample. The first text description sample can describe the first image sample that it matches, which can reflect both the initial text description corresponding to the first image sample and the multiple first target spatial relationships included in the first image sample. Therefore, the image generation model can be trained or adjusted using the first image sample as supervisory information and the first text description sample as input to the image generation model.

[0047] The method for generating an image generation model training set provided by the exemplary embodiment of the present disclosure can be executed by an electronic device or a chip applied to an electronic device, and the electronic device can be an electronic device with a display function or a computer device without a display function.

[0048] When an electronic device has a display function, the electronic device may include a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and a wearable device based on augmented reality (AR) and / or virtual reality (VR) technology.

[0049] Exemplarily, when the electronic device is a wearable device, the wearable device can also be a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not just hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are fully functional, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0050] The following is an introduction to the training set determination process of the exemplary embodiment of the present disclosure in conjunction with the accompanying drawings. It should be understood that the following training set determination process is only used for explanation and is not intended to limit the present disclosure.

[0051] Figure 1 FIG. 2 is a flow chart showing a method for determining a training set according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the method for generating a training set of an exemplary embodiment of the present disclosure may include:

[0052] Step 101: extracting plane position parameters of a plurality of first objects and depth position parameters of a plurality of first objects included in a first image sample.

[0053] The first image sample of the exemplary embodiment of the present disclosure may include multiple first targets, and the target detector may be used to detect the multiple first targets included in the first image sample, thereby obtaining the categories and plane position parameters of the multiple first targets. For example, the target detector may perform target detection on the first image sample based on a convolutional neural network to obtain the category of each first target and the position information of the first target.

[0054] Exemplarily, for the same first target, the location information of various target frames may be detected, and each target frame corresponds to a certain detection confidence. The target frame with a detection confidence greater than a preset confidence or the target frame with the highest detection confidence can be retained, and the location information of the target frame can be used as the location information of the first target.

[0055] The plane position parameters of each first target may be formed as the position information of the target frame, and the position information of the target frame may be determined by the position of the diagonal point of the target frame, the position of the upper left corner of the target frame, and the size parameters of the target frame, or the center position of the target frame and the size parameters of the target frame. The size parameters of the target frame here may be the height and width of the target frame.

[0056] In the exemplary embodiment of the present disclosure, when extracting the depth position parameters of the multiple first targets included in the first image sample, the depth of field estimator can be used to perform depth estimation on the first image sample. Here, the depth of field estimator can perform semantic segmentation on the first image sample based on the speech segmentation algorithm to obtain multiple depth position parameters, and then obtain the depth position parameter corresponding to the first target from the multiple depth position parameters based on the plane position parameter of each first target.

[0057] Exemplarily, when the above-mentioned multiple depth position parameters can exist in the form of a first image sample depth map, the center position of the first target is determined based on the plane position parameters of the first target, and then the depth position parameters corresponding to the center position of the first target are obtained from the first image sample depth map as the depth position parameters of the first target.

[0058] For example, the first image sample is detected using an object detector, and the bounding boxes whose detection confidence is greater than a preset confidence are retained. Based on the bounding boxes, the category C of each first object can be obtained. i , the center point coordinates of the first target (X i ,Y i ) and the width W of the first target i and height H i ; Then use the depth estimator to perform depth estimation on the first image sample to obtain the first image sample depth map, and obtain the coordinates of the center point (X) of the first target from the first image sample depth map i ,Y i ), which is defined as the depth D of the first target i On this basis, the plane position parameters and depth position of each target can constitute the spatial information of the target. i =(C i ,X i ,Y i ,W i ,H i ,D i ), and the set S consisting of the spatial information of all targets included in the first image sample can be expressed as S = {O A ,O B ,...}, O A is the spatial information of target A, O B The space for target B.

[0059] Step 102: Determine the depth relationship description text based on the depth position parameters of the multiple first targets. The target depth relationship may be determined based on the depth position parameters of the multiple first targets, and then the depth relationship description text corresponding to the target depth relationship may be determined based on the mapping relationship between the depth relationship and the description text.

[0060] The depth position parameter of the first target may be the depth of the first target. The smaller the depth of the first target, the closer the first target is to the front of the first image sample, and the larger the depth of the first target, the closer the first target is to the back of the first image sample. Based on this, the exemplary embodiment of the present disclosure may determine the depth relationship of different first targets according to the depth size relationship determined by the depth position parameters of each first target, and generate a depth relationship description text based on the depth relationship of different first targets.

[0061] In the mapping relationship between the depth relationship and the description text of the exemplary embodiment of the present disclosure, each depth relationship can map one or more semantically identical description texts. Of course, multiple semantically identical depth relationship description texts corresponding to the target depth relationship can also be generated based on various corpus generation models.

[0062] Step 103: Determine the plane relationship description text based on the plane position parameters of the multiple first targets. Here, the target plane relationship can be determined based on the plane position parameters of the multiple first targets, and then the plane relationship description text corresponding to the target plane relationship is determined based on the mapping relationship between the plane relationship and the description text. It should be understood that step 102 and step 103 can be performed synchronously or asynchronously.

[0063] The plane position parameter of the above-mentioned first target may be the center position of the first target. Based on the center positions of different first targets, the orientation parameters of different first targets may be determined. Based on the orientation parameters of different first targets, the target plane relationship of different first targets may be determined.

[0064] In the mapping relationship between the plane relationship and the description text of the exemplary embodiment of the present disclosure, each plane relationship can map one or more semantically identical multiple description texts. Of course, multiple semantically identical target relationship description texts corresponding to the target plane relationship can also be generated based on various corpus generation models.

[0065] Step 104: Obtain a first text description sample based on the depth relationship description text, the plane relationship description text and the initial text description of the first image sample.

[0066] The initial text description of the first image sample may be a text that simply describes the various first targets included in the first image sample, which does not contain the spatial relationship of the various first targets, or the spatial relationship of the various first targets is unclear, or the spatial relationship of the various first targets is not complete. Based on this, the exemplary embodiment of the present disclosure can introduce the depth relationship description text and the plane relationship description text as the three-dimensional spatial relationship of the first image sample into the initial text description corresponding to the first image sample, so that the obtained first text description sample can reflect both the initial text description corresponding to the first image sample and the spatial relationship of multiple first targets included in the first image sample.

[0067] The method of the exemplary embodiment of the present disclosure can extract and process information of the first image sample, and obtain a training set constructed by an image-text formed by the first image sample and the first text description sample. Therefore, the image generation model of the exemplary embodiment of the present disclosure does not require a large amount of annotation, which can save annotation costs and manpower investment costs.

[0068] Since the first text description sample can reflect both the initial text description corresponding to the first image sample and the multiple first target spatial relationships included in the first image sample, when training or adjusting the image generation model with the training set determined by the method of the exemplary embodiment of the present disclosure, the first image sample can be used as supervision information and the first text description sample as input to improve the image generation model's ability to understand spatial relationships, thereby ensuring that the trained image generation model can output an image that more accurately reflects the spatial relationships of the various targets represented by the text description content when the spatial information of the input text description content is fuzzy.

[0069] In an alternative approach, Figure 2 FIG. 2 shows a schematic diagram of a process for determining a depth relationship description text according to an exemplary embodiment of the present disclosure. Figure 2 As shown, the exemplary embodiment of the present disclosure determines the depth relationship description text based on the depth position parameters of the plurality of first targets, including:

[0070] Step 201: grouping the depth position parameters of a plurality of first targets to obtain a plurality of groups of depth position parameters, wherein the difference between different groups of depth position parameters is greater than or equal to a first preset difference. At this time, the depths determined by different groups of depth position parameters have relatively large differences.

[0071] When the exemplary embodiment of the present disclosure groups the depth position parameters of multiple first targets, the depth position parameters of the multiple first targets can be clustered to obtain multiple clustering clusters, and then the corresponding group depth position parameters are determined based on the depth position parameters of the first targets included in each clustering cluster.

[0072] Exemplarily, the silhouette coefficient corresponding to the clustering of the exemplary embodiment of the present disclosure is greater than the preset silhouette coefficient. For example: given multiple cluster quantity parameters, the depth position parameters of multiple first targets are clustered using each cluster quantity parameter to obtain multiple cluster clusters corresponding to the cluster quantity parameter, and then the clustering effect corresponding to each cluster quantity parameter is evaluated by the silhouette coefficient. If the silhouette coefficient corresponding to the cluster quantity parameter is the maximum value among the silhouette coefficients corresponding to the multiple cluster quantity parameters, then it can be determined that the clustering effect is relatively good, and the number of clusters determined by the cluster quantity parameter can be used as the number of groups for grouping the depth position parameters of the multiple first targets.

[0073] Step 202: If different groups of depth position parameters meet the target depth relationship confirmation condition, determine the depth relationship description text based on the target depth relationship confirmation condition.

[0074] In practical applications, the depth position parameters of any first target in different groups of first target depth position parameters may be compared to determine the comparison result of the different groups of first target depth parameters, or the comparison result of the different groups of first target depth parameters may be determined by other means.

[0075] For example, if a cluster includes a depth position parameter of a first target, the depth position parameter corresponding to the cluster is equal to the depth position parameter of a first target included in the corresponding cluster; if the cluster includes depth position parameters of multiple first targets, the depth position parameter corresponding to the cluster is equal to the average value of the depth positions of multiple first targets included in the corresponding cluster.

[0076] Exemplarily, the above-mentioned target depth relationship confirmation condition may include that the comparison results of different groups of depth position parameters satisfy the preset depth comparison result. Here, there may be a mapping relationship between the depth comparison result and the depth relationship, that is, different depth comparison results may have a mapping relationship with different depth relationships. Therefore, when determining the depth relationship description text based on the target depth relationship confirmation condition, the depth relationship description text corresponding to the preset depth comparison result may be determined based on the mapping relationship between the depth comparison result and the depth relationship.

[0077] When the depth position parameter is expressed in depth, the greater the depth, the further back it is, and the smaller the depth, the further forward it is. For example, if the depth D of target A is A <Depth D of target B B , then the depth relationship between target A and target B is classified as target A is in front of target B, which can be represented by Front(A,B); For example, the depth D of target A A >Depth D of target B B , then the depth relationship between target A and target B is classified as target A is behind target B, which can be represented by Behind(A,B).

[0078] In order to increase the diversity of text descriptions and improve the generalization ability of the image generation model, taking the Front(A,B) deep relationship as an example, we can generate "target A is in front of target B", "target B is in front of target A", "target B is behind target A", and "target A is behind target B", etc., to increase the diversity of text descriptions of the same deep relationship. On this basis, the deep relationship description text is introduced into the initial text description of the first image sample and used as a training set to train or adjust the image generation model, so as to ensure that the image generation model is compatible with image generation tasks with text descriptions that have the same semantics but different text expressions.

[0079] Exemplarily, in the exemplary embodiments of the present disclosure, the depth position parameters of the multiple first targets included in each cluster are similar. For example, if the cluster includes the depth position parameters of multiple first targets, the difference between the depth position parameters of every two first targets included in the same cluster is less than or equal to the second preset difference. That is to say, in this case, the depth position parameters of the first targets in the same cluster are relatively close, and because the first preset difference is less than the second preset difference, the depth position parameters of the first targets in different clusters are relatively different.

[0080] It can be seen that the exemplary embodiment of the present disclosure can approximately consider multiple first targets corresponding to the same cluster as being in the same depth plane, and multiple first targets corresponding to different clusters as being in different depth planes. Therefore, the depth position parameters of the multiple first targets are clustered, and the number of clusters obtained is actually the number of depth planes.

[0081] Considering that multiple first targets corresponding to the same clustering cluster are approximately considered to be in the same depth plane, and multiple first targets corresponding to different clustering clusters are approximately considered to be in different depth planes, therefore, when the comparison results of different groups of depth position parameters are equal to the preset depth comparison results, the depth relationship description text corresponding to the preset depth comparison results can describe the depth relationship between the first targets corresponding to different clustering clusters, without the need to compare the depth position parameters of the first targets corresponding to different clustering clusters one by one.

[0082] Assuming that the cluster number parameter K with the largest silhouette coefficient is the optimal cluster number parameter, the cluster number parameter is used to cluster the depth position parameters of different first targets, and K clusters can be obtained, corresponding to K depth planes, which are P 1 , P 2 ,……,P K .P 1 represents the first depth plane, P 2 represents the second depth plane, P KIt represents the Kth depth plane. Considering that the depth position parameters of multiple first targets corresponding to the same cluster can be approximately considered the same, it is only necessary to compare the depth position parameters corresponding to different clusters.

[0083] Figure 3 FIG. 2 shows a schematic diagram of the depth relationship of different targets of an exemplary embodiment of the present disclosure. Figure 3 As shown, the first image sample 300 includes a target 1 301, a target 2 302 and a target 303. The target 1 301 and the target 2 302 are located in a first depth plane. The target space information set P of the first depth plane is 1 = {O 1 ,O 2},O 1 Represents the spatial information of target one, O 2 Represents the spatial information of target 2, the target spatial information set P of the second depth plane 2 = {O 3},O 3 Indicates the spatial information of target 3. For each target's spatial information, the depth position parameter included therein may be depth.

[0084] In one embodiment, the exemplary embodiment of the present disclosure can compare the depth of target 1 301, the depth of target 2 302, and the depth of target 3 303. Figure 3 It can be seen that the depth of target one 301 and the depth of target two 302 are obviously smaller than the depth of target three 303. Therefore, target one 301 is located in front of target three 303, and target two 302 is located in front of target three 303. The depth relationship between target one 301 and target two 302 can be determined according to the numerical values ​​of their depths.

[0085] In another embodiment, if Figure 3 As shown, in the exemplary embodiment of the present disclosure, the depth of target 1 301 corresponding to the first depth plane is relatively close to the depth of target 2 302, so it can be determined that target 1 301 and target 2 302 are in the same row or the same depth plane. The front-to-back position relationship between target 1 301 and target 3 303, and the front-to-back position relationship between target 2 302 and target 3 303 are described in two ways below.

[0086] For example: Figure 3 As shown, the depth of target one 301 corresponding to the first depth plane and the depth of target three 303 corresponding to the second depth plane can be compared to obtain the front-to-back relationship between target one 301 and target three 303. Since target one 301 and target two 302 are identified to be in the same row or the same depth plane, the front-to-back relationship between target two 302 and target three 303 is the same as the front-to-back relationship between target one 301 and target 303.

[0087] Another example: Figure 3 As shown, the average value of the depth of target one 301 and the depth of target two 302 corresponding to the first depth plane can be determined, and the average value is determined as the depth of the first depth plane, and the depth of target three 303 is defined as the depth of the second depth plane, and then the depth of the first depth plane and the depth of the second depth plane are compared to obtain the front-back relationship between the first depth plane and the second depth plane. Since the first depth plane corresponds to target one 301 and target two 302, and the second depth plane corresponds to target three 303, the front-back relationship between target one 301 and target 303, and the front-back relationship between target two 302 and target three 303 can all be represented by the front-back relationship between the first depth plane and the second depth plane.

[0088] In an optional manner, the plane relationship description text of the exemplary embodiment of the present disclosure includes the plane relationship description text between the two first targets. Figure 4 FIG. 2 shows a schematic diagram of a process for determining a plane relationship description text according to an exemplary embodiment of the present disclosure. Figure 4 As shown, the exemplary embodiment of the present disclosure determines the plane relationship description text based on the plane position parameters of the plurality of first targets, which may include:

[0089] Step 401: Determine the relative deflection parameters of the two first targets based on the plane position parameters of the two first targets. The plane position parameters of the two first targets here may refer to the plane position parameters of the two first targets located in the same depth plane, or may refer to the plane position parameters of the two first targets located in different depth planes, and the depth plane here refers to the definition above.

[0090] Step 402: If the relative deflection parameters of the two first targets are within the target deflection parameter interval, based on the mapping relationship between the deflection parameter interval and the plane relationship, determine the plane relationship description text corresponding to the target deflection parameter interval. Considering that the deflection parameters are different, the relative orientations of the two first targets will be different. Based on this, the exemplary embodiment of the present disclosure can set that when there is a mapping relationship between the deflection parameter interval and the plane relationship, different deflection parameter intervals correspond to different plane relationships, and determine the plane relationship of the two first targets according to the difference in the deflection parameter intervals where the relative deflection parameters of the two first targets are located.

[0091] Considering that the sizes of different first targets are quite different, the sizes of different first targets can be scaled uniformly. The following takes the plane position parameters of two first targets on the same depth plane as an example to describe the scale uniformity method.

[0092] The method of the exemplary embodiment of the present disclosure may also include: acquiring plane position parameters of multiple first targets located in the same depth plane, determining the center position and size parameters of the corresponding first target based on the plane position parameters of each first target, and then determining the half-height of the corresponding target based on the center position parameter and size parameter of each of the first targets, and unifying the scale of the target center position parameters of each first target based on the minimum half-height and size parameters of the half-heights of the multiple first targets.

[0093] Exemplarily, the clustering cluster can be determined by the aforementioned clustering method to include depth position parameters of multiple first targets, and these first targets can be considered to be in the same depth plane. Considering that there are various ways to represent the plane position parameters of the first target, if the center position of the first target is not included, the center position and height of the corresponding first target can be calculated by the plane position parameters of the first target.

[0094] When the plane position parameters of the first target include the positions of the diagonal points of the frame of the first target, the height and center point coordinates of the first target can be determined by the diagonal point position coordinates. After the center point coordinates and height of each first target are obtained, the half height of each first target can be determined by the center point coordinates and height of each first target, and then the minimum half height of the half heights of each first target is obtained, and the center point coordinates of each first target are unified by the minimum half height.

[0095] Assuming that the coordinates of the center point of the first target are (X, Y), the distance from the center point coordinates of the first target to the bottom of the frame of the first target can be defined as the half height H of the first target. Therefore, the vertical coordinate of the bottom of the frame of the first target is B = YH. Then, the half heights of all the first targets in the same depth plane are selected, and the minimum half height is selected as h. Then, the scale of each first target is unified using the minimum half height h. After the center point coordinates of the first target are unified, they become (X', Y'), and X' = X, Y' = B + h / H.

[0096] For example, let the coordinates of target A be (X' A ,Y' A ), the coordinates of target B are (X' B ,Y' B ),exist Figure 5 The deflection parameter is expressed as a deflection angle. Here, the deflection angle α of target A and target B is determined by the following formula:

[0097]

[0098] Figure 5 FIG. 2 shows a schematic diagram of the distribution of deflection parameter intervals of an exemplary embodiment of the present disclosure. Figure 5As shown, with target A as the center, the circumference is divided into eight deflection angle intervals, and there is a one-to-one mapping relationship between the deflection angle intervals and the plane relationship.

[0099] If 0≤α<22.5° or 337.5°<α≤360°, the plane relationship between target A and target B is classified as target A is on the right of target B, which can be represented by Right(A,B); if 22.5°≤α<67.5°, the plane relationship between target A and target B is classified as target A is on the upper right of target B, which can be represented by TopRight(A,B); if 67.5°≤α<112.5°, the plane relationship between target A and target B is classified as target A is on the upper side of target B, which can be represented by Top(A,B); if 112.5°≤α<157.5°, the plane relationship between target A and target B is classified as target A is on the upper left of target B, which can be represented by TopLeft(A,B); If 157.5°≤α<202.5°, the plane relationship between target A and target B is classified as target A is to the left of target B, which can be represented by Left(A,B); if 202.5°≤α<247.5°, the plane relationship between target A and target B is classified as target A is to the lower left of target B, which can be represented by BottomLeft(A,B); if 247.5°≤α<292.5°, the plane relationship between target A and target B is classified as target A is below target B, which can be represented by Bottom(A,B); if 292.5°≤α<337.5°, the plane relationship between target A and target B is classified as target A is to the lower right of target B, which can be represented by BottomRight(A,B).

[0100] In order to increase the diversity of text descriptions and improve the generalization ability of the image generation model, taking the Right(A,B) plane relationship as an example, we can generate "target A is on the right of target B", "target B is on the left of target A", "target B is on the left of target A", and "target A is on the right of target B", etc., to increase the diversity of text descriptions of the same plane relationship. On this basis, the plane relationship description text is introduced into the initial text description of the first image sample, and the image generation model is trained and adjusted together with the first image sample as a training set, which can ensure that the image generation model is compatible with image generation tasks with text descriptions of the same semantics but different text expressions.

[0101] In an optional manner, the exemplary embodiment of the present disclosure obtains the first text description sample based on the depth relationship description text, the plane relationship description text and the initial text description of the first image sample, which essentially involves splicing the depth relationship description text, the plane relationship description text and the initial text description of the first image sample.

[0102] For example, the initial text description of the first image sample is: "An elephant and a motorcycle appeared on a road in the wilderness." The depth relationship description text and the plane relationship description text can be obtained from the first image sample through the above-mentioned related methods. The depth relationship description text includes: "The big tree is behind the motorcycle, and there is a big tree behind the elephant." The plane relationship description text includes "The elephant is on the left side of the motorcycle, and there is an elephant on the right side of the motorcycle." The depth relationship description text, the plane relationship description text, and the initial text description of the first image sample are spliced ​​together to form the first text description sample of "An elephant and a motorcycle appeared on a road in the wilderness, the elephant is on the left side of the motorcycle, there is an elephant on the right side of the motorcycle, the big tree is behind the motorcycle, and there is a big tree behind the elephant."

[0103] In an optional manner, in order to ensure that the image generation model can fully learn each spatial relationship and improve the text graph model's ability to understand various spatial relationships, the amount of data of various spatial relationships determined by the training set determination method of the exemplary embodiment of the present disclosure is as balanced as possible. Based on this, when the plane relationship description text includes multiple plane relationship description texts, the exemplary embodiment of the present disclosure can expand the spatial relationship with less data in the training set, so that the training set also includes a second image sample and a second text description sample matching the second image sample.

[0104] When expanding spatial relationships with less data in the training set, the method of the exemplary embodiment of the present disclosure may also include: if the number of plane relationship description texts meets the sample expansion conditions, determining a second image sample and a second text description sample based on the attributes of the plane relationship description text and the attributes of the two second targets.

[0105] The above sample expansion conditions may include that the number of plane relationship description texts is less than a preset number. Here, the preset number can be regarded as the minimum number requirement of the plane relationship description texts. If the number of plane relationship description texts is less than the preset number, it means that the data volume of the plane relationship description text is too small and needs to be expanded.

[0106] Exemplarily, the maximum value among the quantities of multiple plane relationship description texts may be defined as the maximum quantity, and the difference between the preset quantity and the maximum quantity is less than or equal to the preset difference. The preset difference may be 0 or a value greater than 0. When the preset difference is equal to 0, the preset quantity is substantially equal to the maximum quantity, that is, as long as the plane relationship description text is less than the maximum quantity, it needs to be expanded.

[0107] Exemplarily, when the second image sample and the second text description sample are determined based on the attributes of the plane relationship description text and the attributes of the two second targets, the exemplary embodiment of the present disclosure can determine the second text description sample based on the attributes of the plane relationship description text, determine the plane position parameters of the two second targets based on the second text description sample, and determine the second image sample based on the second text description sample, the plane position parameters of the two second targets and the attribute information of the two second targets.

[0108] When determining the second text description sample based on the attributes of the plane relationship description text, two types of second targets can be first obtained, and the second text description sample can be generated according to the attributes of the plane relationship description text, so that the second text description sample is used to describe the plane relationship attributes of the two second targets. For example, two types of second targets can be randomly obtained from the MSCOCO dataset.

[0109] When determining the plane position parameters of two second targets based on the second text description sample, the plane position parameters of the second target may include the size parameters of the second target and the center position parameters of the second target. The plane position parameters of the second target may be expressed in the form of a border, and the size parameters of the second target and the center position parameters of the second target may both be determined based on the category of the second target.

[0110] Figure 6 FIG. 2 shows a schematic diagram of a visualization representation of the planar position parameters of two second targets according to an exemplary embodiment of the present disclosure. Figure 6 As shown, assuming the planar relationship of the left and right relationship is in the visualization representation interface 600, a first border 601 and a second border 602 are displayed, and the position parameters of the first border 601 can represent the planar position parameters of the teddy bear, and the position parameters of the second border 602 can represent the planar position parameters of the parrot.

[0111] When generating a second image sample, the second text description sample, the plane position parameters of the two second targets, and the attribute information of the two second targets can be input into the spatial relationship guided image generation model to generate a second image sample, so that the second image sample and the second text description sample can constitute new data to expand the corresponding plane relationship description text data volume.

[0112] For example, when the amount of text data describing the plane relationship of the left-right relationship is less than a preset amount, two types of targets, a teddy bear and a parrot, can be selected, and then a second text description sample of "the parrot is on the right of the teddy bear" can be generated. Then, two types of borders distributed on the left and right are generated using the second text description sample, one of which corresponds to the teddy bear and the other corresponds to the parrot. Then, the plane position parameters of the two borders, the category information of the teddy bear, the category information of the parrot, and "the parrot is on the right of the teddy bear" are input into the image generation model guided by the spatial relationship, thereby obtaining the second image sample and expanding the plane relationship data of the left-right relationship. The number of operations for expanding the data can be set according to the actual situation. For example, when the amount of text data describing the plane relationship of the left-right relationship is greater than or equal to the preset amount, the data expansion of the text describing the plane relationship of the left-right relationship can be stopped.

[0113] Figure 7 FIG. 2 shows a flow chart of an image generation method provided by an exemplary embodiment of the present disclosure. Figure 7 As shown, the Wensheng diagram method of the exemplary embodiment of the present disclosure may include:

[0114] Step 701: Obtain target text information. The target text information here may be target text information obtained in response to a text input operation, or may be target text information obtained based on an audio recognition technology, which is not limited here.

[0115] Step 702: Input the target text information into the image generation model to obtain the target image matching the text information. The training set of the image generation model in the training stage is determined by the method of the exemplary embodiment of the present disclosure. The training set used by the image generation model in the training stage includes various text-image pairs, where the image is used as supervisory information and the text is used as input, so that the image generation model to be trained can generate a predicted image based on the input text, and then use the predicted image and the image as supervisory information to calculate the loss, and determine whether the image generation model is trained according to the size of the loss value.

[0116] One or more technical solutions provided in the exemplary embodiments of the present disclosure can extract the plane position parameters of multiple first targets and the depth position parameters of multiple first targets included in the first image sample, and then determine the depth relationship description text based on the depth position parameters of the multiple first targets, determine the plane relationship description text based on the plane position parameters of the multiple first targets, and finally obtain the first text description sample based on the depth relationship description text, the plane relationship description text and the initial text description of the image sample. It can be seen that the exemplary embodiments of the present disclosure can introduce the three-dimensional spatial relationship contained in the first image sample into the initial text description corresponding to the first image sample in the form of depth relationship description text and plane relationship description text, so that the obtained first text description sample can reflect both the initial text description corresponding to the first image sample and the spatial relationship of multiple first targets included in the first image sample.

[0117] It can be seen that the method of the exemplary embodiment of the present disclosure can extract and process information from the first image sample, and obtain a training set of an image generation model constructed by an image-text formed by the first image sample and the first text description sample. Therefore, the image generation model of the exemplary embodiment of the present disclosure does not need to be annotated in large quantities, which can save annotation costs and manpower investment costs. Since the first text description sample can reflect both the initial text description corresponding to the first image sample and the spatial relationship of multiple first targets included in the first image sample, when the image generation model is trained or adjusted with the image generation model training set, the first image sample can be used as supervision information and the first text description sample can be used as input to improve the image generation model's ability to understand spatial relationships, thereby ensuring that the trained image generation model can output an image that more accurately reflects the spatial relationship of each target represented by the text description content when the spatial information of the input text description content is vague.

[0118] The above mainly introduces the solution provided by the embodiment of the present disclosure from the perspective of an electronic device. It is understandable that, in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.

[0119] The disclosed embodiment can divide the electronic device into functional units according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the disclosed embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0120] In the case of dividing each functional module according to each function, an exemplary embodiment of the present disclosure provides a training set determination device, which may be an electronic device or a chip applied to the electronic device. Figure 8 FIG. 1 shows a schematic block diagram of functional modules of a training set determination device according to an exemplary embodiment of the present disclosure. Figure 8 As shown, a training set determination device 800 of an exemplary embodiment of the present disclosure, wherein the training set is a training set of an image generation model, wherein the training set includes at least a first image sample and a first text description sample matching the first image sample, and the device includes:

[0121] An extraction module 801 is used to extract the plane position parameters of the plurality of first targets and the depth position parameters of the plurality of first targets included in the first image sample;

[0122] A determination module 802, configured to determine a depth relationship description text based on a plurality of depth position parameters of the first targets, and to determine a plane relationship description text based on a plurality of plane position parameters of the first targets;

[0123] The acquisition module 803 is used to obtain the first text description sample based on the depth relationship description text, the plane relationship description text and the initial text description of the first image sample.

[0124] In one possible implementation, the determination module 802 is used to group the depth position parameters of the multiple first targets to obtain multiple groups of depth position parameters. If the depth position parameters of different groups are equal to a preset depth comparison result, based on the mapping relationship between the depth comparison result and the depth relationship, the depth relationship description text corresponding to the preset depth comparison result is determined, and the depth relationship description text includes the depth relationship description text of the first target corresponding to the depth position parameters of different groups, and the difference between the depth position parameters of different groups is greater than or equal to the first preset difference.

[0125] In a possible implementation, the determination module 802 is used to cluster the depth position parameters of the plurality of first targets to obtain a plurality of clusters, and determine a corresponding group of the depth position parameters based on the depth position parameters of the first targets included in each cluster.

[0126] In a possible implementation, the silhouette coefficient corresponding to the clustering is greater than a preset silhouette coefficient;

[0127] If the cluster includes depth position parameters of multiple first targets, the difference between the depth position parameters of every two first targets included in the same cluster is less than or equal to a second preset difference, and the first preset difference is less than the second preset difference.

[0128] In a possible implementation, if the cluster includes depth position parameters of a plurality of first targets, the depth position parameter corresponding to the cluster is equal to an average value of the depth position parameters of the plurality of first targets included in the cluster.

[0129] In a possible implementation, the plane relationship description text includes a plane relationship description text between the two first targets, and the determination module 802 is used to determine the relative deflection parameters of the two first targets based on the plane position parameters of the two first targets. If the relative deflection parameters of the two first targets are within the target deflection parameter interval, the plane relationship description text corresponding to the target deflection parameter interval is determined based on the mapping relationship between the deflection parameter interval and the plane relationship.

[0130] In a possible implementation, the determination module 802 is also used to obtain the plane position parameters of multiple first targets located in the same depth plane, determine the half-height of the corresponding target based on the center position parameter and size parameter of each first target, and scale the target center position parameters of each first target based on the minimum half-height and size parameter of the half-heights of multiple first targets.

[0131] In a possible implementation, the training set also includes a second image sample and a second text description sample matching the second image sample, the plane relationship description text includes multiple plane relationship description texts, and the acquisition module 803 is also used to determine the second image sample and the second text description sample based on the attributes of the plane relationship description text and the attributes of the two second targets if the number of the plane relationship description texts meets the sample expansion condition.

[0132] In a possible implementation, the sample expansion condition is that the number of plane relationship description texts is less than a preset number, the difference between the preset number and the maximum number is less than or equal to the preset difference, and the maximum number is the maximum value among the multiple numbers of the plane relationship description texts.

[0133] In one possible implementation, the acquisition module 803 is used to include: determining a second text description sample based on the attributes of the plane relationship description text, the second text description sample is used to describe the plane relationship attributes of the two second targets, determining the plane position parameters of the two second targets based on the second text description sample, and determining the second image sample based on the second text description sample, the plane position parameters of the two second targets and the attribute information of the two second targets.

[0134] In the case of dividing each functional module according to each function, an exemplary embodiment of the present disclosure provides an image generating device, which may be an electronic device or a chip applied to an electronic device. Fig. 9 FIG. 1 shows a schematic block diagram of functional modules of an image generating device according to an exemplary embodiment of the present disclosure. Fig. 9 As shown, the image generating device 900 of the exemplary embodiment of the present disclosure includes:

[0135] An acquisition module 901 is used to acquire target text information;

[0136] An inference module 902 is used to input the target text information into the image generation model to obtain a target image matching the text information;

[0137] The training set of the image generation model in the training phase is determined by the method described in the exemplary embodiment of the present disclosure.

[0138] Fig.10 Schematic block diagram of a chip according to an exemplary embodiment of the present disclosure is shown. Fig.10 As shown, the chip 1000 includes one or more (including two) processors 1001 and a communication interface 1002. The communication interface 1002 can support the electronic device to perform the data sending and receiving steps in the above method, and the processor 1001 can support the electronic device to perform the data processing steps in the above method.

[0139] Optional, such as Fig.10 As shown, the chip 1000 also includes a memory 1003, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM).

[0140] In some embodiments, Fig.10 As shown, the processor 1001 performs corresponding operations by calling the operation instructions stored in the memory (the operation instructions may be stored in the operating system). The processor 1001 controls the processing operations of any one of the terminal devices, and the processor may also be called a central processing unit (CPU). The memory 1003 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1001. A portion of the memory 1003 may also include NVRAM. For example, in an application, the memory, the communication interface, and the memory are coupled together through a bus system, wherein the bus system may include a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, in Fig.10 Various buses are labeled as bus system 1004 .

[0141] The method disclosed in the above-mentioned embodiment of the present disclosure can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above-mentioned processor may be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an ASIC, a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiment of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware.

[0142] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.

[0143] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.

[0144] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.

[0145] refer to Fig.11 , a block diagram of an electronic device 1100 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0146] like Fig.11 As shown, the electronic device 1110 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0147] A plurality of components in the electronic device 1100 are connected to the I / O interface 1105, including: an input unit 1106, an output unit 1107, a storage unit 1108, and a communication unit 1109. The input unit 1106 may be any type of device capable of inputting information to the electronic device 1100, and the input unit 1106 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 1107 may be any type of device capable of presenting information, and may include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1108 may include but is not limited to a disk, an optical disk. The communication unit 1109 allows the electronic device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0148] like Fig.11 As shown, the computing unit 1101 can be various general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above. For example, in some embodiments, the method of the exemplary embodiment of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1100 via the ROM 1102 and / or the communication unit 1109. In some embodiments, the computing unit 1101 can be configured to perform the method of the exemplary embodiment of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0149] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0150] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0151] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0152] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0153] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0154] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0155] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0156] Although the present disclosure has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present disclosure. Accordingly, this specification and the drawings are merely exemplary illustrations of the present disclosure as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present disclosure. Obviously, those skilled in the art may make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A method for determining a training set, characterized in that: The training set is a training set of an image generation model, and the training set includes at least a first image sample and a first text description sample matching the first image sample. The method includes: Extracting a plurality of plane position parameters of the first targets and a plurality of depth position parameters of the first targets included in the first image sample; Determine a depth relationship description text based on a plurality of depth position parameters of the first targets; Determine the plane relationship description text based on the plane position parameters of the plurality of first targets; The first text description sample is obtained based on the depth relationship description text, the plane relationship description text and the initial text description of the first image sample.

2. The method according to claim 1, characterized in that The determining of the depth relationship description text based on the depth position parameters of the plurality of first targets comprises: Grouping the depth position parameters of the plurality of first targets to obtain a plurality of groups of depth position parameters, wherein the difference between the depth position parameters of different groups is greater than or equal to a first preset difference; If the depth position parameters of different groups are equal to the preset depth comparison results, based on the mapping relationship between the depth comparison results and the depth relationship, the depth relationship description text corresponding to the preset depth comparison results is determined, and the depth relationship description text includes the depth relationship description text of the first target corresponding to the depth position parameters of different groups.

3. The method according to claim 2, characterized in that The step of grouping the depth position parameters of the plurality of first targets to obtain a plurality of groups of depth position parameters includes: Clustering the depth position parameters of the plurality of first targets to obtain a plurality of clustering clusters; Based on the depth position parameters of the first object included in each of the clusters, a corresponding group of the depth position parameters is determined.

4. The method according to claim 3, characterized in that The silhouette coefficient corresponding to the clustering is greater than a preset silhouette coefficient; If the cluster includes depth position parameters of multiple first targets, the difference between the depth position parameters of every two first targets included in the same cluster is less than or equal to a second preset difference, and the first preset difference is less than the second preset difference.

5. The method according to claim 3, characterized in that: If the cluster includes depth position parameters of a plurality of first objects, the depth position parameter corresponding to the cluster is equal to an average value of the depth position parameters of the plurality of first objects included in the cluster.

6. The method according to claim 1, characterized in that The plane relationship description text includes a plane relationship description text between two of the first targets, and the determining of the plane relationship description text based on the plane position parameters of the plurality of the first targets includes: Determining relative deflection parameters of the two first targets based on the planar position parameters of the two first targets; If the relative deflection parameters of the two first targets are within a target deflection parameter interval, the plane relationship description text corresponding to the target deflection parameter interval is determined based on a mapping relationship between the deflection parameter interval and the plane relationship.

7. The method according to claim 6, characterized in that The method further comprises: Acquire plane position parameters of a plurality of the first targets located on the same depth plane; Based on the center position parameter and size parameter of each of the first targets, determining the half height of the corresponding target; The target center position parameters of each of the first targets are scaled based on the minimum half-height and size parameters of the half-heights of the plurality of first targets.

8. The method according to any one of claims 1 to 7, characterized in that: The training set also includes a second image sample and a second text description sample matching the second image sample, the plane relationship description text includes a plurality of plane relationship description texts, and the method further includes: If the number of the plane relationship description texts meets the sample expansion condition, the second image sample and the second text description sample are determined based on the attributes of the plane relationship description texts and the attributes of the two second objects.

9. The method according to claim 8, characterized in that The sample expansion condition is that the number of the plane relationship description texts is less than a preset number, the difference between the preset number and the maximum number is less than or equal to the preset difference, and the maximum number is the maximum value among the numbers of multiple plane relationship description texts.

10. The method according to claim 8, characterized in that The determining the second image sample and the second text description sample based on the attribute of the plane relationship description text and the attributes of the two second targets includes: Determining a second text description sample based on the attributes of the plane relationship description text, wherein the second text description sample is used to describe the plane relationship attributes of the two second objects; Determining two plane position parameters of the second targets based on the second text description sample; The second image sample is determined based on the second text description sample, two plane position parameters of the second objects, and two attribute information of the second objects.

11. An image generation method, characterized in that: include: Get target text information; Inputting target text information into an image generation model to obtain a target image matching the text information; Wherein, the training set of the image generation model in the training stage is determined by the method described in any one of claims 1 to 10.

12. A training set determination device, characterized in that: The training set is a training set of an image generation model, and the training set includes at least a first image sample and a first text description sample matching the first image sample. The device includes: An extraction module, used for extracting a plurality of plane position parameters of the first targets and a plurality of depth position parameters of the first targets included in the first image sample; A determination module, configured to determine a depth relationship description text based on a plurality of depth position parameters of the first targets, and to determine a plane relationship description text based on a plurality of plane position parameters of the first targets; An acquisition module is used to obtain the first text description sample based on the depth relationship description text, the plane relationship description text and the initial text description of the first image sample.

13. An image generating device, characterized in that: include: An acquisition module is used to acquire target text information; An inference module, used for inputting target text information into an image generation model to obtain a target image matching the text information; Wherein, the training set of the image generation model in the training stage is determined by the method described in any one of claims 1 to 10.

14. An electronic device, characterized in that: include: processor; as well as, A memory for storing programs; The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 11.

15. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium is used to store computer instructions, and the computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 11.

Images