Image generation method, device and equipment
By receiving image generation requests and obtaining 3D scene description information, and generating images in combination with object attribute information, the problem of low image generation stability and certainty in the prior art is solved, and higher image generation quality is achieved.
Patent Information
- Application Number
- CN202311375668.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-19
- Publication Date
- 2025-04-22
AI Technical Summary
In the existing image generation technology, the quality of text prompt words has a greater impact on the stability and certainty of the generated image, resulting in low stability and certainty of the generated image.
By receiving an image generation request, prompt information carrying pre-built 3D scenes and object preset attribute information is obtained, description information of 3D scenes is obtained, description information is used to describe the orientation relationship and distance relationship between objects, and finally the image is generated based on the description information and object attribute information.
By utilizing rich 3D scene description information and object attribute information, the stability and certainty of image generation are improved and the dependence on the quality of text prompt words is reduced.
Smart Images
Figure CN119863559A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image generation method, device and equipment. Background Art
[0002] In recent years, with the rapid development of deep learning technology and the increase in computing resources, image generation technology based on deep learning has gradually emerged.
[0003] At present, image generation models can generate images related to input text prompts by learning and pattern recognition of existing data. However, in current image generation technology, the quality of text prompts has a great influence on the image generation results, and the stability and certainty of generated images are low. Summary of the invention
[0004] In view of this, the present application provides an image generation method, apparatus and device to improve the stability and certainty of generated images.
[0005] Specifically, the present application is implemented through the following technical solutions:
[0006] A first aspect of the present application provides an image generation method, the method comprising:
[0007] Receive an image generation request; wherein the image generation request carries prompt information; the prompt information includes a pre-built 3D scene, and the 3D scene includes multiple objects and preset attribute information of each of the multiple objects;
[0008] Acquire description information of the 3D scene; wherein the description information is used to describe the orientation relationship and / or distance relationship between various objects in the 3D scene;
[0009] An image is generated based on the description information and the preset attribute information of each object.
[0010] A second aspect of the present application provides an image generation device, the device comprising: a receiving module, a determining module and a generating module, wherein:
[0011] The receiving module is used to receive an image generation request; wherein the image generation request carries prompt information; the prompt information includes a pre-built 3D scene, and the 3D scene includes multiple objects and preset attribute information of each of the multiple objects;
[0012] The determination module is used to obtain description information of the 3D scene; wherein the description information is used to describe the orientation relationship and / or distance relationship between various objects in the 3D scene;
[0013] The generating module is used to generate an image based on the description information and the preset attribute information of each object.
[0014] The third aspect of the present application provides an image generating device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the methods provided in the first aspect of the present application are implemented.
[0015] The image generation method, device and equipment provided by the present application receive an image generation request, wherein the image generation request carries prompt information; the prompt information includes a pre-built 3D scene, the 3D scene includes multiple objects and preset attribute information of each of the multiple objects, and then obtains the description information of the 3D scene used to describe the orientation relationship and / or distance relationship between the various objects in the 3D scene, and finally generates an image based on the description information and the preset attribute information of each object. In this way, by pre-building a 3D scene, obtaining the description information of the 3D scene, and finally combining the description information of the 3D scene to generate an image, because when generating an image, not only the preset attribute information of the object itself is used, but also the description information of the richer scene is used, so that the stability and certainty of image generation can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 A flowchart of the first embodiment of the image generation method provided by the present application;
[0017] Figure 2 A schematic diagram of a constructed 3D scene is shown as an exemplary embodiment of the present application;
[0018] Figure 3 A flowchart of the second embodiment of the image generation method provided by the present application;
[0019] Figure 4 This is a flowchart of the third embodiment of the image generation method provided by the present application;
[0020] Figure 5 A flowchart of a fourth embodiment of the image generation method provided in this application;
[0021] Figure 6 This is a rendering of a 3D scene according to an exemplary embodiment of the present application;
[0022] Figure 7 This is an effect diagram of saliency detection shown in an exemplary embodiment of the present application;
[0023] Figure 8 A flowchart of a fifth embodiment of the image generation method provided in this application;
[0024] Fig. 9 A flowchart of a sixth embodiment of the image generation method provided in this application;
[0025] Fig.10 This is a schematic diagram showing an implementation principle of calculating position relationship information according to an exemplary embodiment of the present application;
[0026] Fig.11 A flowchart of the seventh embodiment of the image generation method provided by the present application;
[0027] Fig.12 This is a schematic diagram showing an implementation principle of calculating position relationship information according to an exemplary embodiment of the present application;
[0028] Fig.13 A schematic diagram of a generated image shown as an exemplary embodiment of the present application;
[0029] Fig.14 A hardware structure diagram of an image generating device where the image generating device provided by the present application is located;
[0030] Fig.15 This is a structural diagram of the first embodiment of the image generation device provided in this application. DETAILED DESCRIPTION
[0031] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0032] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0033] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0034] The present application provides an image generation method, apparatus and device to improve the stability and certainty of generated images.
[0035] The image generation method, device and equipment provided by the present application receive an image generation request, wherein the image generation request carries prompt information; the prompt information includes a pre-built 3D scene, the 3D scene includes multiple objects and preset attribute information of each of the multiple objects, and then obtains description information of the 3D scene for describing the orientation relationship and / or distance relationship between the various objects in the 3D scene, and finally generates an image based on the description information and the preset attribute information of each object. In this way, by pre-building a 3D scene, obtaining the description information of the 3D scene, and finally combining the description information of the 3D scene to generate an image, since the information used is richer, the stability and certainty of image generation can be improved.
[0036] Specific embodiments are given below to introduce the technical solution of the present application in detail.
[0037] Figure 1 This is a flow chart of the first embodiment of the image generation method provided by this application. Please refer to Figure 1 , the method provided in this embodiment includes:
[0038] S101. Receive an image generation request; wherein the image generation request carries prompt information; the prompt information includes a pre-built 3D scene, and the 3D scene includes multiple objects and preset attribute information of each of the multiple objects.
[0039] Specifically, the image generation request is used to instruct the image generation device to generate a corresponding image according to the prompt information. It should be noted that when the user wants to generate an image, the prompt information can be pre-constructed, and then based on the pre-constructed prompt information, the corresponding image generation request is triggered to instruct the image generation device to generate the corresponding image. Specifically, for example, in one embodiment, the prompt information may include a pre-built 3D scene, and the 3D scene includes multiple objects and preset attribute information of each of the multiple objects. For another example, in one embodiment, the prompt information may include a pre-built 3D scene, and the 3D scene includes multiple objects and preset attribute information of each of the multiple objects, as well as preset visual expressions, etc.
[0040] Specifically, the expression of a 3D scene refers to a method of presenting objects in a three-dimensional space on a two-dimensional screen. In specific implementation, users can use 3D modeling software to build a 3D scene (for example, unreal engine, Blender, Maya, 3ds Max can be used to build a 3D scene). It should be noted that the built 3D scene can be a geometric scene, that is, objects in the 3D scene are represented by basic geometric bodies. In addition, when building a 3D scene, users can use advanced modeling techniques, such as surface modeling, subdivision surfaces, Boolean operations, etc., to create more complex shapes or structures, and use more complex shapes or structures to represent objects in the 3D scene.
[0041] It should be noted that, in a possible implementation, objects within a 3D scene are divided into primary objects and secondary objects. When a user creates an object when building a 3D scene, the user will specify the object as a primary object or a secondary object. In addition, when building a 3D scene, the user can also specify the importance of each object.
[0042] The pre-built 3D scene is built according to actual needs, and is not limited in this embodiment. For example, in one embodiment, the pre-built 3D scene is as follows: Figure 2 As shown ( Figure 2 This is a schematic diagram of a constructed 3D scene shown as an exemplary embodiment of the present application). Please refer to Figure 2 ,exist Figure 2 In the example shown, the 3D scene includes object 1, object 2, object 3, and so on.
[0043] The preset attribute information of each object in the plurality of objects is set according to actual needs and is not limited in this embodiment. For example, in one embodiment, the preset attribute information may include at least one of the following information: level information (used to indicate whether it is a primary object or a secondary object, and the importance of the object), category information, color information, and shape information.
[0044] Combining the above example, for example, Figure 2 For each object in the 3D scene shown, the preset attribute information set by the user is as follows: the preset attribute information of object 1 is: main object, girl, short hair, white hair color, blue eyes, wearing a hoodie; the preset attribute information of object 2 is: secondary object, cute white puppy; the preset attribute information of object 3 is: secondary object, tree.
[0045] Furthermore, the preset visual expression is set according to actual needs, and in this embodiment, the specific content of the preset visual expression is not limited. For example, in one embodiment, the preset visual expression may include a preset painting style and a preset painting quality.
[0046] S102: Acquire description information of the 3D scene; wherein the description information is used to describe the orientation relationship and / or distance relationship between various objects in the 3D scene.
[0047] Specifically, in a possible implementation, a pre-trained description information acquisition model may be used to acquire description information of the 3D scene. The training process and internal implementation principle of the description information acquisition model are not specifically introduced here.
[0048] Furthermore, the description information is used to describe the orientation relationship and / or distance relationship between various objects in the 3D scene.
[0049] For example, in one embodiment, the description information may include the azimuth angles of each secondary object in the 3D scene relative to the primary object, which is used to describe the azimuth relationship of the secondary object relative to the primary object; for another example, the description information may also include the distances of each secondary object in the 3D scene relative to the primary object, which is used to describe the distance relationship of the secondary object relative to the primary object.
[0050] S103: Generate an image based on the description information and the preset attribute information of each object.
[0051] In a specific implementation, the description information and the preset attribute information of each object may be input into a pre-trained image generation model so that the image generation model can generate an image.
[0052] It should be noted that, referring to the previous introduction, if the prompt information includes a preset visual representation, in this step, the preset visual representation is input into the pre-trained image generation model.
[0053] Furthermore, the image generation model is a machine learning model that can generate corresponding images based on input information. It should be noted that the image generation model is pre-trained. In one embodiment, the pre-trained image generation model is stable diffusion. In other embodiments, the image generation model can also be other models that generate images based on input information, such as Midjourney. The training process and internal implementation principles of the image generation model can be referred to the description in the relevant technology, which will not be repeated here.
[0054] The image generation method provided in this embodiment receives an image generation request, wherein the image generation request carries prompt information; the prompt information includes a pre-built 3D scene, the 3D scene includes multiple objects and preset attribute information of each of the multiple objects, and then obtains description information of the 3D scene for describing the orientation relationship and / or distance relationship between the various objects in the 3D scene, and finally generates an image based on the description information and the preset attribute information of each object. In this way, by pre-building a 3D scene, obtaining the description information of the 3D scene, and finally combining the description information of the 3D scene to generate an image, since the information used is richer, the stability and certainty of image generation can be improved.
[0055] Figure 3 This is a flow chart of the second embodiment of the image generation method provided by this application. Please refer to Figure 3 The method provided in this embodiment, based on the above embodiment, the step of obtaining the description information of the 3D scene includes:
[0056] S301. For each secondary object in the 3D scene, determine the orientation relationship information of the secondary object relative to the primary object and / or the distance relationship information of the secondary object relative to the primary object according to the first coordinates of the primary object and the second coordinates of the secondary object in the 3D scene.
[0057] Specifically, referring to the above introduction, the preset attribute information of the object includes level information, and the primary object and the secondary object in the 3D scene can be determined through the level information of the object. It should be noted that the coordinates of the object can be the coordinates of its center point.
[0058] Further, for example, in a possible implementation manner, the process of obtaining the position relationship information may include:
[0059] (1) Determine the azimuth angle of the secondary object relative to the primary object according to the first coordinates of the primary object and the second coordinates of the secondary object in the 3D scene.
[0060] (2) Determine the azimuth angle as the azimuth relationship information of the secondary object relative to the primary object.
[0061] It should be noted that the specific implementation process and implementation principle of calculating the azimuth angle can be referred to the description in the relevant technology, which will not be repeated here.
[0062] Combined with the previous example, for example, object 1 is the main object, the first coordinate of object 1 is (x1, y1, z1), object 2 is the secondary object, the second coordinate of object 2 is (x2, y2, z2), the calculated azimuth angle of object 2 relative to object 1 is α. At this time, the azimuth relationship information of object 2 relative to object 1 is determined to be α.
[0063] For another example, in another possible implementation, the specific implementation process of obtaining the distance relationship information may include:
[0064] (1) determining a distance between the secondary object and the primary object according to a first coordinate of the primary object and a second coordinate of the secondary object in the 3D scene;
[0065] (2) Determine the distance as distance relationship information of the secondary object relative to the primary object.
[0066] It should be noted that the specific implementation process and implementation principle of calculating the distance can be referred to the description in the relevant technology, which will not be repeated here.
[0067] Combined with the previous example, for example, the first coordinate of object 1 is (x1, y1, z1), the second coordinate of object 2 is (x2, y2, z2), and the distance between object 2 and object 1 is d, where, At this time, the distance relationship information of object 2 relative to object 1 is determined to be d.
[0068] S302: Determine the orientation relationship information and / or the distance relationship information as the description information.
[0069] In combination with the above example, for example, the orientation relationship information of object 2 relative to object 1 is α, and the distance relationship information of object 2 relative to object 1 is d. At this time, the description information is: the orientation relationship information of object 2 relative to object 1 is α, and the distance relationship information is d.
[0070] The image generation method provided in this embodiment determines the orientation relationship information of the secondary object relative to the primary object and / or the distance relationship information of the secondary object relative to the primary object according to the first coordinate of the primary object and the second coordinate of the secondary object in the 3D scene for each secondary object in the 3D scene, and then determines the orientation relationship information and / or the distance relationship information as the description information. In this way, the description information can be accurately determined based on the coordinates of the object, and then the final image can be obtained based on the description information and attribute information, which can improve the stability and certainty of image generation.
[0071] Figure 4This is a flowchart of the third embodiment of the image generation method provided by this application. Please refer to Figure 4 The method provided in this embodiment is based on the above embodiment, wherein the preset attribute information of each object includes a plurality of preset attribute information; and the generating of an image based on the description information and the preset attribute information of each object includes:
[0072] S401. Determine a weight of each object; wherein the weight of each object is positively correlated with a prominence of each object in the 3D scene.
[0073] Specifically, the weight of each object is positively correlated with the saliency of each object in the 3D scene, that is, a higher weight generally corresponds to a higher saliency, and a lower weight generally corresponds to a lower saliency.
[0074] In one possible implementation, referring to the previous explanation, the preset attribute information of the object may include level information for characterizing whether the object is a primary object or a secondary object, and the importance of the object. In this step, the weight of each object may be determined based on the preset attribute information of each object.
[0075] For example, in one embodiment, there are objects 1, 2, 3, and 4 in scene A. Based on the preset attribute information of each object, the importance of each object is determined as follows: the importance of object 1> the importance of object 2> the importance of object 3> the importance of object 4. In this step, weights are assigned to each object according to its importance. For example, the weight assigned to object 1 is 0.4, the weight assigned to object 2 is 0.3, the weight assigned to object 3 is 0.2, and the weight assigned to object 4 is 0.1.
[0076] S402: Determine the expected number of prompt words corresponding to each object according to the weight of each object and the preset number of prompt words.
[0077] Specifically, the number of preset prompt words is set according to actual needs, and in this embodiment, the specific value of the number of preset prompt words is not limited. For example, in one embodiment, the number of preset prompt words is 10.
[0078] In a possible implementation, the expected number of prompt words corresponding to each object can be obtained by multiplying the weight of each object by the preset number of prompt words.
[0079] For example, combining the above example, the weight of object 1 is 0.4, the preset number of prompt words is 10, and the expected number of prompt words corresponding to object 1 is 4.
[0080] S403: For each object, select the expected number of preset attribute information from multiple preset attribute information of each object, and use the selected preset attribute information as the prompt word corresponding to each object.
[0081] In a possible implementation, for each object, an expected number of preset attribute information may be randomly selected from multiple preset attribute information of the object.
[0082] Combined with the above example, for example, the number of preset prompt words is 10, there is object 1 in scene A, the weight of object 1 is 0.4, and the preset attribute information of object 1 includes preset attribute information 1, preset attribute information 2, preset attribute information 3, preset attribute information 4, and preset attribute information 5. Preset attribute information 2, preset attribute information 4, preset attribute information 1, and preset attribute information 3 are randomly selected as the prompt words corresponding to object 1.
[0083] In another possible implementation, each of the multiple preset attribute information of each object has a different priority. In this step, the expected number of preset attribute information can be selected from the multiple preset attribute information of each object in order of priority from high to low.
[0084] Specifically, the higher the priority, the greater the possibility that the preset attribute information is selected.
[0085] Combined with the above example, for example, the number of preset prompt words is 10, there is object 2 in scene A, the weight of object 2 is 0.3, and the preset attribute information of object 3 includes preset attribute information 1, preset attribute information 2, preset attribute information 3, preset attribute information 4, and preset attribute information 5. Among them, the priority of preset attribute information 1> the priority of preset attribute information 4> the priority of preset attribute information 3> the priority of preset attribute information 2> preset attribute information 5. In this step, preset attribute information 1, preset attribute information 4, and preset attribute information 5 are selected as the prompt words corresponding to object 2 in order of priority from high to low.
[0086] It should be noted that, for each object, if the number of preset attribute information of the object is less than or equal to the expected number of prompt words corresponding to the object, in this step, all the preset attribute information of the object is directly used as the prompt words corresponding to the object; if the number of preset attribute information of the object is greater than the expected number of prompt words corresponding to the object, in this step, the corresponding preset attribute information is selected as the prompt words according to the above method.
[0087] S404: Generate an image according to the description information and the prompt word corresponding to each object.
[0088] Specifically, the description information and the prompt word corresponding to each object are input into a pre-trained image generation model to generate an image.
[0089] It should be noted that the image generation model is pre-trained. The training process and internal implementation principle of the image generation model can be referred to the description in the relevant technology, which will not be repeated here.
[0090] The image generation method provided in this embodiment, when the number of prompt words is limited, determines the weight of each object; wherein the weight of each object is positively correlated with the degree of significance of each object in the 3D scene, and determines the expected number of prompt words corresponding to each object according to the weight of each object and the number of preset prompt words, and then selects the expected number of preset attribute information from multiple preset attribute information of each object for each object, and uses the selected preset attribute information as the prompt words corresponding to each object, so as to generate an image according to the description information and the prompt words corresponding to each object. In this way, the number of prompt words corresponding to each object is related to the weight of the object, and under the condition that the number of prompt words is significant, the number of prompt words for each object can be reasonably allocated, so that objects with high significance are allocated more prompt words, so that even if the number of prompt words is limited, it will not affect the effect of the generated image.
[0091] Figure 5 This is a flowchart of the fourth embodiment of the image generation method provided by this application. Please refer to Figure 5 In the method provided in this embodiment, the step of determining the weight of each object includes:
[0092] S501. Obtain a rendering of the 3D scene; wherein the rendering includes a color image and a segmentation image.
[0093] Specifically, Figure 6 This is a rendering of a 3D scene shown in an exemplary embodiment of the present application. Please refer to Figure 6 , the rendering of the 3D scene includes color images ( Figure 6 Figure a) and segmentation diagram ( Figure 6 A color map is a realistic color image generated by the rendering process, which shows the color, texture, and lighting effects of objects in the scene. A segmentation map is an image in which objects in the rendering image are segmented and labeled according to different categories, and each pixel is assigned a specific label to indicate which object it belongs to.
[0094] S502: Perform saliency detection on the color image to obtain a saliency value of each pixel in the color image.
[0095] Specifically, Figure 7 This is a diagram showing the effect of saliency detection according to an exemplary embodiment of the present application. Figure 7 , saliency detection is used to determine the saliency value of each pixel in the color image. The saliency value of each pixel represents the saliency or importance of the pixel in the color image. The higher the saliency value, the higher the saliency of the pixel in the color image.
[0096] For the specific implementation principle of saliency detection, please refer to the description in the related art, which will not be repeated here.
[0097] S503: Determine the saliency value of each object according to the saliency value of each pixel in the segmentation map and the color map.
[0098] Specifically, the significance value of each object indicates the significance or importance of the object in the color map. The higher the significance value, the higher the significance of the object in the color map.
[0099] In specific implementation, for example, each object can be identified based on the segmentation map. Further, for each object, the significance value of each pixel in the multiple pixel points corresponding to the object is found from the significance value of each pixel in the color map, and then the average value of the significance values found is determined as the significance value of the object.
[0100] S504: Allocate a weight to each object according to the significance value of each object.
[0101] Specifically, the significance value of each object is positively correlated with the weight, that is, the greater the significance value of each object, the greater the weight of the object.
[0102] In specific implementation, a weight may be assigned to each object based on a preset weight assignment rule and the significance value of each object. The preset weight assignment rule is set according to actual needs. In this embodiment, it is not limited to this. When setting the preset weight assignment rule, it can be set according to the principle of "the greater the significance value, the greater the weight". For example, in one embodiment, when assigning weights to objects, weights may be assigned to each object in proportion according to the significance value of each object. For another example, in another embodiment, when assigning weights to objects, three objects are first selected from all objects in order of significance value from high to low, and then for these three objects, weight ratios of 4, 3, and 2 are respectively given, and for other objects, weight ratios of 1 are respectively given, and further, the weights of each object are finally determined according to the weight ratio.
[0103] For example, in combination with the above introduction, in one embodiment, the 3D scene includes object 1, object 2, and object 3. Further, the significance value of object 1 is 0.2, the significance value of object 2 is 0.5, and the significance value of object 3 is 0.3. In this step, a weight of 0.2 is assigned to object 1, a weight of 0.5 is assigned to object 2, and a weight of 0.3 is assigned to object 3.
[0104] The image generation method provided in this embodiment is to obtain the 3D scene rendering image; wherein the rendering image includes a color image and a segmentation image, and perform a saliency test on the color image to obtain the saliency value of each pixel in the color image, and then determine the saliency value of each object based on the saliency value of each pixel in the segmentation image and the color image, so as to assign a weight to each object according to the saliency value of each object. In this way, the saliency value of each object can be obtained based on the saliency value of each pixel in the 3D scene rendering image, and then the weight of each object can be obtained based on the saliency value of each object, and then a suitable number of prompt words can be assigned to each object according to the weight, and the number of prompt words for each object can be reasonably assigned, so that objects with high saliency are assigned more prompt words, so that even if the number of prompt words is limited, it will not affect the effect of the generated image.
[0105] Figure 8 This is a flowchart of the fifth embodiment of the image generation method provided by this application. Please refer to Figure 8 The method provided in this embodiment, based on the above embodiment, determines the distance relationship information of the secondary object relative to the primary object according to the first coordinate of the primary object and the second coordinate of the secondary object in the 3D scene, including:
[0106] S801. Calculate the distance between the secondary object and the primary object according to the first coordinate and the second coordinate.
[0107] For the specific implementation method of calculating the distance, please refer to the description in the above embodiment, which will not be repeated here.
[0108] S802: When the distance is greater than a preset distance, determine that the distance relationship information of the secondary object relative to the primary object is that the secondary object is located far from the primary object.
[0109] Specifically, the preset distance is set according to actual needs, and in this embodiment, the specific value of the preset distance is not limited. For example, in one embodiment, the preset distance is 20 cm.
[0110] In combination with the previous example, for example, for object 3, the distance between object 3 and object 1 is calculated to be 30 cm. At this time, the distance is greater than the preset distance, and the distance relationship information of object 3 relative to object 1 is determined to be located far away from object 1.
[0111] S803: When the distance is less than a preset distance, determine the distance relationship information of the secondary object relative to the primary object as being located near the primary object.
[0112] In combination with the previous example, for example, for object 2, the distance between object 2 and object 1 is calculated to be 10 cm. At this time, the distance is less than the preset distance, and the distance relationship information of object 2 relative to object 1 is determined to be located close to object 1.
[0113] The image generation method provided by the present application calculates the distance between the secondary object and the primary object according to the first coordinate and the second coordinate, and then when the distance is greater than a preset distance, determines the distance relationship information of the secondary object relative to the primary object as being located far from the primary object, and when the distance is less than a preset distance, determines the distance relationship information of the secondary object relative to the primary object as being located near the primary object. In this way, the distance relationship information between the primary object and the secondary object can be determined based on the coordinates of the primary object and the secondary object, and then the description information is determined based on the distance relationship information, so that the final image is obtained based on the description information and the attribute information, which can improve the stability and certainty of image generation.
[0114] Fig. 9 This is a flowchart of the sixth embodiment of the image generation method provided by this application. Please refer to Fig. 9 The method provided in this embodiment is based on the above embodiment, wherein the orientation relationship information includes horizontal orientation relationship information; and the determining the orientation relationship information of the secondary object relative to the primary object includes:
[0115] S901. Determine a first straight line equation corresponding to a first straight line; wherein the first straight line is a straight line formed by a first point indicated by the first coordinate and a third point indicated by a third coordinate; and the third coordinate is the coordinate of a camera in the 3D scene.
[0116] It should be noted that the horizontal orientation relationship information is used to characterize the horizontal orientation relationship between the secondary object and the primary object, wherein the horizontal orientation relationship refers to the positional relationship between the secondary object and the primary object in the direction indicated by the X-axis of the 3D scene coordinate system. Figure 2 ,exist Figure 2 In the example shown, the horizontal orientation relationship refers to the positional relationship between the secondary object and the primary object in the left-right direction of the paper.
[0117] It should be noted that the specific implementation process and implementation principle of calculating the first straight line equation according to the first point and the third point can be referred to the description in the relevant technology, which will not be repeated here.
[0118] Combined with the above example, for example, the coordinates of the first point are (x1, y1), the coordinates of the third point are (x2, y2), and the linear equation of the first line is L1. Figure 2 Each object in is replaced by its center point, and the first straight line is Fig.10 As shown ( Fig.10 A schematic diagram of an implementation principle of calculating orientation relationship information shown as an exemplary embodiment of the present application).
[0119] S902. Searching for a first target point on the first straight line; wherein the longitudinal coordinate of the first target point is the same as the longitudinal coordinate of the secondary object.
[0120] Specifically, the first target point refers to a point on the first straight line that has the same ordinate as the second coordinate of the secondary object. Fig.10 In the example shown, for object 2, the first target point corresponding to object 2 is found to be Fig.10 Further, for object 3, the first target point corresponding to object 3 is found to be Fig.10 Point B in .
[0121] S903: Determine the horizontal coordinate of the first target point according to the first straight line equation.
[0122] Specifically, the ordinate in the second coordinate of each secondary object is substituted into the first straight line equation to obtain the corresponding abscissa, which is the abscissa of the first target point corresponding to the secondary object.
[0123] S904: When the horizontal coordinate of the secondary object is smaller than the horizontal coordinate of the first target point, determine the horizontal position relationship information of the secondary object to be located on the left side of the primary object; otherwise, determine the horizontal position relationship information of the secondary object to be located on the right side of the primary object.
[0124] In combination with the above example, for example, the horizontal coordinate of object 2 is greater than the horizontal coordinate of point A, and the horizontal position relationship information of object 2 is determined to be located on the right side of object 1. Further, the horizontal coordinate of object 3 is less than the horizontal coordinate of point B, and the horizontal position relationship information of object 3 is determined to be located on the left side of object 1.
[0125] The image generation method provided in this embodiment provides a method for calculating horizontal orientation relationship information based on coordinates. Through this method, the horizontal orientation relationship information can be accurately determined.
[0126] Fig.11 This is a flowchart of the seventh embodiment of the image generation method provided by this application. Please refer to Fig.11The method provided in this embodiment is based on the above embodiment, wherein the orientation relationship information includes vertical orientation relationship information; and the determining the orientation relationship information of the secondary object relative to the primary object includes:
[0127] S1101. Determine, based on the first straight line equation, a second straight line equation that is perpendicular to the first straight line; wherein the second straight line passes through the first point.
[0128] It should be noted that the vertical orientation relationship information is used to characterize the vertical orientation relationship between the secondary object and the primary object, wherein the vertical orientation relationship refers to the positional relationship between the secondary object and the primary object in the direction indicated by the Y axis of the 3D scene coordinate system. Figure 2 ,exist Figure 2 In the example shown, the vertical orientation relationship refers to the positional relationship between the secondary object and the primary object in the up-and-down direction on the paper.
[0129] It should be noted that the specific implementation process and implementation principle of calculating the second straight line equation according to the first straight line equation can be referred to the description in the relevant technology, which will not be repeated here.
[0130] Combining the above example, for example, the linear equation of the second straight line is L2. Further, Figure 2 Each object in is replaced by its center point, and the second straight line is Fig.12 As shown ( Fig.12 A schematic diagram of an implementation principle of calculating orientation relationship information shown as an exemplary embodiment of the present application).
[0131] S1102. Search for a second target point on the second straight line; wherein the horizontal coordinate of the second target point is the same as the horizontal coordinate of the secondary object.
[0132] Specifically, the second target point refers to a point on the second straight line that has the same horizontal coordinate as the second coordinate of the secondary object. Fig.12 In the example shown, for object 2, the second target point corresponding to object 2 is found to be Fig.12 Point C in ; further, for object 3, the second target point corresponding to object 3 is found to be Fig.12 Point D in .
[0133] S1103. Determine the longitudinal coordinate of the second target point according to the second straight line equation.
[0134] Specifically, the horizontal coordinate of the second coordinate of each secondary object is substituted into the second straight line equation to obtain the corresponding vertical coordinate, which is the vertical coordinate of the second target point corresponding to the secondary object.
[0135] S1104. When the longitudinal coordinate of the secondary object is smaller than the longitudinal coordinate of the second target point, determine the vertical orientation relationship information of the secondary object to be located in front of the primary object; otherwise, determine the vertical orientation relationship information of the secondary object to be located behind the primary object.
[0136] In combination with the above example, for example, the ordinate of object 2 is less than the ordinate of point C, and the vertical position relationship information of object 2 is determined to be located in front of object 1. Further, the ordinate of object 3 is greater than the ordinate of point D, and the vertical position relationship information of object 3 is determined to be located behind object 1.
[0137] The image generation method provided in this embodiment provides a method for calculating vertical orientation relationship information based on coordinates. Through this method, the vertical orientation relationship information can be accurately determined.
[0138] Specifically, combining the above example, for example, in Figure 2 In the example shown, the attribute information of each object is: Object 1: main object, girl, short hair, white hair color, blue eyes, wearing a hoodie, Object 2: secondary object, cute white puppy, Object 3: secondary object, tree.
[0139] The description information obtained is: 1. Object 2 is located in front of the right side of object 1 and is near object 1; 2. Object 3 is located behind the left side of object 1 and is far away from object 1.
[0140] At this time, the generated image is Fig.13 As shown ( Fig.13 is a schematic diagram of a generated image shown in an exemplary embodiment of the present application). Fig.13 The method provided in this application can improve the stability and accuracy of image generation.
[0141] It should be noted that, in a possible implementation, the training samples of the image generation model are obtained by the following method, including:
[0142] (1) Obtaining a sample set of images; the label of each image in the sample set of images includes attribute information of each object contained in the image, and orientation relationship information and / or distance relationship information between the objects.
[0143] (2) Using the image sample set to train a pre-built image generation model.
[0144] For the specific implementation principle of the sample set-based training model, please refer to the description in the relevant technology, which will not be repeated here.
[0145] The image generation method provided in this embodiment, by adding attribute information and azimuth distance information to the label of the training sample and then using the training sample to train the image generation model, can provide the accuracy of the image generation model and further improve the stability and certainty of image generation.
[0146] Corresponding to the above-mentioned embodiment of the image generating method, the present application also provides an embodiment of the image generating device.
[0147] The embodiments of the image generation apparatus provided in the present application can be applied to image generation devices. The apparatus embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logically sensed apparatus, the processor of the image generation device in which it is located reads the corresponding computer program instructions in the non-volatile memory into the memory and runs them. From the hardware level, if Fig.14 As shown, it is a hardware structure diagram of the image generating device provided by the present application where the image generating device is located, except Fig.14 In addition to the processor, memory, network interface, and non-volatile memory shown, the image generating device where the apparatus in the embodiment is located may also include other hardware according to the actual function of the image generating device, which will not be described in detail.
[0148] Fig.15 This is a structural diagram of the first embodiment of the image generation device provided by this application. Please refer to Fig.15 The device provided in this embodiment includes: a receiving module 1510, a determining module 1520 and a generating module 1530, wherein:
[0149] The receiving module 1510 is configured to receive an image generation request; wherein the image generation request carries prompt information; the prompt information includes a pre-built 3D scene, and the 3D scene includes multiple objects and preset attribute information of each of the multiple objects;
[0150] The determination module 1520 is used to obtain description information of the 3D scene; wherein the description information is used to describe the orientation relationship and / or distance relationship between various objects in the 3D scene;
[0151] The generating module 1530 is used to generate an image based on the description information and the preset attribute information of each object.
[0152] The device provided in this embodiment can be used to perform Figure 1 The technical method of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.
[0153] Optionally, the determination module 1520 is specifically used to determine, for each secondary object in the 3D scene, the orientation relationship information of the secondary object relative to the primary object and / or the distance relationship information of the secondary object relative to the primary object based on the first coordinate of the primary object in the 3D scene and the second coordinate of the secondary object, and determine the orientation relationship information and / or the distance relationship information as the description information.
[0154] Optionally, the generating module 1530 is specifically configured to:
[0155] Determining a weight of each object; wherein the weight of each object is positively correlated with a degree of prominence of each object in the 3D scene;
[0156] Determining the expected number of prompt words corresponding to each object according to the weight of each object and the preset number of prompt words;
[0157] For each of the objects, selecting the expected number of preset attribute information from a plurality of preset attribute information of each of the objects, and using the selected preset attribute information as a prompt word corresponding to each of the objects;
[0158] An image is generated according to the description information and the prompt word corresponding to each object.
[0159] Optionally, the generating module 1530 is specifically configured to:
[0160] Obtaining a rendering of the 3D scene; wherein the rendering includes a color image and a segmentation image;
[0161] Performing saliency detection on the color image to obtain a saliency value of each pixel in the color image;
[0162] Determine the saliency value of each object according to the saliency value of each pixel in the segmentation map and the color map;
[0163] A weight is assigned to each object according to the significance value of each object.
[0164] Optionally, the generating module 1530 is specifically configured to: select the expected number of preset attribute information from the multiple preset attribute information of each object in order of priority from high to low.
[0165] Optionally, the determining module 1520 is specifically configured to:
[0166] Calculating the distance between the secondary object and the primary object according to the first coordinate and the second coordinate;
[0167] When the distance is greater than a preset distance, determining the distance relationship information of the secondary object relative to the primary object as being located far from the primary object;
[0168] When the distance is less than a preset distance, it is determined that the distance relationship information of the secondary object relative to the primary object is located near the primary object.
[0169] Optionally, the determination module 1520 is specifically used to: determine a first straight line equation corresponding to a first straight line; wherein the first straight line is a straight line formed by a first point indicated by the first coordinate and a third point indicated by a third coordinate; and the third coordinate is a coordinate of a camera in the 3D scene;
[0170] Finding a first target point on the first straight line; wherein the ordinate of the first target point is the same as the ordinate of the secondary object;
[0171] Determining the horizontal coordinate of the first target point according to the first straight line equation;
[0172] When the horizontal coordinate of the secondary object is smaller than the horizontal coordinate of the first target point, the horizontal position relationship information of the secondary object is determined to be located on the left side of the primary object; otherwise, the horizontal position relationship information of the secondary object is determined to be located on the right side of the primary object.
[0173] Optionally, the determining module 1520 is specifically configured to:
[0174] Determine, based on the first straight line equation, a second straight line equation of a second straight line perpendicular to the first straight line; wherein the second straight line passes through the first point;
[0175] Finding a second target point on the second straight line; wherein the horizontal coordinate of the second target point is the same as the horizontal coordinate of the secondary object;
[0176] Determining the ordinate of the second target point according to the second straight line equation;
[0177] When the longitudinal coordinate of the secondary object is smaller than the longitudinal coordinate of the second target point, the vertical position relationship information of the secondary object is determined to be located in front of the primary object; otherwise, the vertical position relationship information of the secondary object is determined to be located behind the primary object.
[0178] Please continue to refer to Fig.14 The present application also provides an image generating device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the methods provided in the first aspect of the present application are implemented.
[0179] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.
[0180] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.
[0181] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.
Claims
1. An image generation method, characterized in that: The method comprises: Receive an image generation request; wherein the image generation request carries prompt information; the prompt information includes a pre-built 3D scene, and the 3D scene includes multiple objects and preset attribute information of each of the multiple objects; Acquire description information of the 3D scene; wherein the description information is used to describe the orientation relationship and / or distance relationship between various objects in the 3D scene; An image is generated based on the description information and the preset attribute information of each object.
2. The method according to claim 1, characterized in that Obtaining description information of the 3D scene, including: For each secondary object in the 3D scene, determining, according to the first coordinate of the primary object in the 3D scene and the second coordinate of the secondary object, the orientation relationship information of the secondary object relative to the primary object and / or the distance relationship information of the secondary object relative to the primary object; The orientation relationship information and / or the distance relationship information are determined as the description information.
3. The method according to claim 1, characterized in that The preset attribute information of each object includes a plurality of preset attribute information; and the generating an image based on the description information and the preset attribute information of each object includes: Determining a weight of each object; wherein the weight of each object is positively correlated with a degree of prominence of each object in the 3D scene; Determining the expected number of prompt words corresponding to each object according to the weight of each object and the preset number of prompt words; For each of the objects, selecting the expected number of preset attribute information from a plurality of preset attribute information of each of the objects, and using the selected preset attribute information as a prompt word corresponding to each of the objects; An image is generated according to the description information and the prompt word corresponding to each object.
4. The method according to claim 3, characterized in that Determining the weight of each object includes: Obtaining a rendering of the 3D scene; wherein the rendering includes a color image and a segmentation image; Performing saliency detection on the color image to obtain a saliency value of each pixel in the color image; Determine the saliency value of each object according to the saliency value of each pixel in the segmentation map and the color map; A weight is assigned to each object according to the significance value of each object.
5. The method according to claim 3, characterized in that: The priorities of the respective preset attribute information among the plurality of preset attribute information of each object are different; The selecting the expected number of preset attribute information from the plurality of preset attribute information of each object includes: The expected number of preset attribute information are selected from the plurality of preset attribute information of each object in order of priority from high to low.
6. The method according to claim 2, characterized in that The determining, according to the first coordinate of the main object in the 3D scene and the second coordinate of the secondary object, the distance relationship information of the secondary object relative to the main object comprises: Calculating the distance between the secondary object and the primary object according to the first coordinate and the second coordinate; When the distance is greater than a preset distance, determining the distance relationship information of the secondary object relative to the primary object as being located far from the primary object; When the distance is less than a preset distance, it is determined that the distance relationship information of the secondary object relative to the primary object is located near the primary object.
7. The method according to claim 2 or 6, characterized in that: The orientation relationship information includes horizontal orientation relationship information; The determining the position relationship information of the secondary object relative to the primary object includes: Determine a first straight line equation corresponding to a first straight line; wherein the first straight line is a straight line formed by a first point indicated by the first coordinate and a third point indicated by a third coordinate; and the third coordinate is a coordinate of a camera in the 3D scene; Finding a first target point on the first straight line; wherein the ordinate of the first target point is the same as the ordinate of the secondary object; Determining the horizontal coordinate of the first target point according to the first straight line equation; When the horizontal coordinate of the secondary object is smaller than the horizontal coordinate of the first target point, the horizontal position relationship information of the secondary object is determined to be located on the left side of the primary object; otherwise, the horizontal position relationship information of the secondary object is determined to be located on the right side of the primary object.
8. The method according to claim 7, characterized in that The orientation relationship information includes vertical orientation relationship information; The determining the position relationship information of the secondary object relative to the primary object includes: Determine, based on the first straight line equation, a second straight line equation of a second straight line perpendicular to the first straight line; wherein the second straight line passes through the first point; Finding a second target point on the second straight line; wherein the horizontal coordinate of the second target point is the same as the horizontal coordinate of the secondary object; Determining the ordinate of the second target point according to the second straight line equation; When the longitudinal coordinate of the secondary object is smaller than the longitudinal coordinate of the second target point, the vertical position relationship information of the secondary object is determined to be located in front of the primary object; otherwise, the vertical position relationship information of the secondary object is determined to be located behind the primary object.
9. An image generating device, characterized in that: The device comprises: a receiving module, a determining module and a generating module, wherein: The receiving module is used to receive an image generation request; wherein the image generation request carries prompt information; the prompt information includes a pre-built 3D scene, and the 3D scene includes multiple objects and preset attribute information of each of the multiple objects; The determination module is used to obtain description information of the 3D scene; wherein the description information is used to describe the orientation relationship and / or distance relationship between various objects in the 3D scene; The generating module is used to generate an image based on the description information and the preset attribute information of each object.
10. An image generating device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 8 are implemented.