Image generation methods, apparatus, devices, media, and program products

By using a method to generate images from text information, and leveraging the correspondence between field content and word vectors, as well as attention weights, this approach solves the problem of low efficiency in manually collecting sample images for smart home device recognition, and achieves efficient and diverse image generation.

CN123118809APending Publication Date: 2026-11-17CHINA MOBILEHANGZHOUINFORMATION TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510638143.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-16
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies for identifying smart home devices suffer from low efficiency in manually collecting sample images, making it difficult to meet the demand for diverse and efficient image generation. Furthermore, generative adversarial networks lack scalability.

Method used

By acquiring text information, utilizing the pre-defined correspondence between field content and word vectors, and attention weights, a text matrix is ​​generated and converted into an image, reducing manual intervention and improving the accuracy and quality of image generation.

Benefits of technology

It enables the automatic generation of installation images of smart home devices that match semantic features based on text information, reducing manual intervention, improving the accuracy and quality of image generation, and meeting the diverse and efficient image generation needs in the smart home field.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118809A_ABST
    Figure CN123118809A_ABST
Patent Text Reader

Abstract

This application discloses an image generation method, apparatus, device, medium, and program product. The method, in response to an image generation request, acquires content information of multiple first fields and generates content information of the second field according to preset content generation rules. It then acquires target word vectors corresponding to the content information of the first field and the content information of the second field from a preset target correspondence between field content and word vectors. Based on multiple target word vectors, it generates a text matrix. Finally, it acquires target weights corresponding to the target word vectors from a preset target correspondence between fields and attention weights. Based on multiple target weights, it generates an image corresponding to the text matrix. According to embodiments of this application, it can automatically generate installation images of smart home devices that fit semantic features, improving the accuracy and quality of image generation and meeting the urgent need for diversified and efficient image generation in the smart home field.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to an image generation method, apparatus, device, medium and program product. Background Technology

[0002] With the rapid development of the smart home industry, the functions of smart home devices are constantly being innovated, and the types of devices are becoming increasingly complex with significant differences between manufacturers and models. This poses a huge challenge to device management and identification. As smart home devices become more and more widespread, how to quickly and accurately locate and identify these devices to improve device connection speed and stability has become a critical issue that urgently needs to be addressed.

[0003] Currently, while deep learning-based object detection and optical character recognition technologies are applied to device recognition to some extent, when incorporating images into new business scenarios, significant manpower and resources are required to collect and manually classify and label sample images. This results in high upfront costs, difficulty in guaranteeing the effectiveness of small-sample training, and a heavy reliance on constantly updated real training images for model optimization. Furthermore, manual image collection is inefficient in existing technologies, and while generative adversarial networks can generate similar images, their scalability is insufficient and manual labeling is still necessary.

[0004] Therefore, existing technologies suffer from the problem that manual collection of sample images is inefficient and cannot meet the diverse and efficient image generation needs of the smart home field. Summary of the Invention

[0005] This application provides an image generation method, apparatus, device, medium, and program product that can automatically generate images of smart home devices that fit semantic features based on text information. This reduces manual intervention, improves the accuracy and quality of image generation, and meets the urgent need for diversified and efficient image generation in the smart home field.

[0006] In a first aspect, embodiments of this application provide a method for generating an image, comprising:

[0007] In response to an image generation request, the text information in the image generation request is obtained, and the text information includes the content information of multiple first fields;

[0008] Based on the content information of multiple first fields, the content information of the second field is generated according to the preset content generation rules of the second field;

[0009] In the preset target correspondence between field content and word vector, obtain the target word vector corresponding to the content information of the first field and the target word vector corresponding to the content information of the second field;

[0010] A text matrix is ​​generated based on multiple target word vectors, and the text matrix includes multiple target word vectors arranged in a preset target order;

[0011] In the pre-defined target correspondence between fields and attention weights, obtain the target weight corresponding to the target word vector;

[0012] Based on multiple target weights, an image corresponding to the text matrix is ​​generated.

[0013] In one possible embodiment of the first aspect, the first field includes at least one of the following: device type field, device color field, device style field, device connection method field, text information field on the device nameplate image, and QR code content information field on the device nameplate image.

[0014] In one possible embodiment of the first aspect, the preset content generation rules for the second field include field combination rules and / or semantic expansion rules.

[0015] In one possible embodiment of the first aspect, before obtaining the text information in the image generation request in response to the image generation request, the method further includes:

[0016] Retrieve historical text information, which includes historical content information from multiple first fields;

[0017] Based on the historical content information of multiple first fields, the historical content information of the second field is generated according to the preset initial generation rules of the second field content.

[0018] Based on the preset initial correspondence between field content and word vectors, the preset initial order, and the preset initial correspondence between fields and attention weights, an initial historical image corresponding to the historical content information of multiple fields is generated. The historical content information of the fields includes the historical content information of the first field and the historical content information of the second field.

[0019] Identify image content information in the initial historical images and match the image content information with historical content information in multiple fields to obtain the matching results;

[0020] If the matching result is a failure, the preset initial generation rules for the second field content, the preset initial correspondence between the content and word vectors, the preset initial order, and the preset initial correspondence between the field and attention weights are iteratively updated to obtain the preset initial generation rules for the second field content, the preset target correspondence between the content and word vectors, the preset target order, and the preset target correspondence between the field and attention weights.

[0021] In one possible embodiment of the first aspect, an initial historical image corresponding to the historical content information of multiple fields is generated based on a preset initial correspondence between field content and word vectors, a preset initial order, and a preset initial correspondence between fields and attention weights, including:

[0022] In the preset initial correspondence between field content and word vectors, obtain the initial historical word vectors corresponding to the historical content information of the field;

[0023] An initial historical text matrix is ​​generated based on multiple initial historical word vectors. The initial historical text matrix includes multiple initial historical word vectors arranged in a preset initial order.

[0024] In the preset initial correspondence between fields and attention weights, obtain the initial historical weights corresponding to the initial historical word vectors;

[0025] Based on multiple initial historical weights, an initial historical image corresponding to the initial historical text matrix is ​​generated.

[0026] In one possible embodiment of the first aspect, image content information in an initial historical image is identified, and the image content information is matched with historical content information in multiple fields to obtain a matching result, including:

[0027] Using a pre-defined target detection model, information on the categories of historical objects in the initial historical images is identified;

[0028] Using a pre-defined text recognition model, historical text information in the initial historical images is identified;

[0029] The image content information in the initial historical image is matched with the historical content information in multiple fields to obtain the matching results. The image content information in the initial historical image includes information on the historical object category and historical text information.

[0030] In one possible embodiment of the first aspect, before generating the content information of the second field based on the content information of multiple first fields according to a preset content generation rule for the second field, the method further includes:

[0031] In the image generation request, the content information of the target field in the text information is obtained, and the target field is the device type field;

[0032] Based on the preset correspondence between device type and content generation rules of the second field, obtain the content generation rules of the second field corresponding to the content information of the target field.

[0033] In one possible embodiment of the first aspect, generating an image corresponding to a text matrix based on multiple target weights includes:

[0034] Based on multiple target weights, the text matrix is ​​fused with a preset noise matrix and then denoised to obtain the image corresponding to the text matrix.

[0035] Based on the same inventive concept, in a second aspect, embodiments of this application also provide an image generation apparatus, comprising:

[0036] The acquisition module is used to acquire text information from the image generation request in response to the image generation request. The text information includes the content information of multiple first fields.

[0037] The generation module is used to generate the content information of the second field based on the content information of multiple first fields, according to the preset content generation rules of the second field.

[0038] The acquisition module is also used to acquire the target word vector corresponding to the content information of the first field and the target word vector corresponding to the content information of the second field from the preset target correspondence between field content and word vector;

[0039] The generation module is also used to generate a text matrix based on multiple target word vectors, the text matrix including multiple target word vectors arranged in a preset target order;

[0040] The acquisition module is also used to obtain the target weight corresponding to the target word vector in the preset field-attention weight target correspondence relationship;

[0041] The generation module is also used to generate images corresponding to text matrices based on multiple target weights.

[0042] Based on the same inventive concept, in a third aspect, embodiments of this application also provide an image generation apparatus, the apparatus including a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the image generation method of the first aspect or any embodiment of the first aspect.

[0043] Based on the same inventive concept, in a fourth aspect, embodiments of this application also provide a computer storage medium, on which computer program instructions are stored, which, when executed by a processor, implement the image generation method of the first aspect or any embodiment of the first aspect.

[0044] Based on the same inventive concept, in a fifth aspect, embodiments of this application also provide a computer program product, wherein instructions in the computer program product, when executed by a processor of a device, enable the device to perform the image generation method of the first aspect or any embodiment of the first aspect.

[0045] This application discloses an image generation method, apparatus, device, medium, and program product. The method, in response to an image generation request, acquires text information from the request. This text information includes content information from multiple fields, which meticulously describe the desired image in text form, providing key semantic guidance for subsequent image generation. Then, based on the content information of multiple first fields, the method generates the content information of the second field according to a preset content generation rule (this rule can be flexibly set as a field combination rule and / or semantic expansion rule, etc.), thereby expanding the text information, enriching the text expression for different scenarios and types, and laying the foundation for generating diverse images. Next, in a preset target correspondence between field content and word vectors, the method acquires the target word vectors corresponding to the content information of the first field and the second field. These target word vectors are then arranged in a preset target order to generate a text matrix, which forms the overall semantic representation of the text, enabling the text semantics to be presented digitally and structurally. Finally, in a preset target correspondence between fields and attention weights, the method acquires the target weights corresponding to the target word vectors. These target weights reflect the importance of different fields in the image generation process, helping the model to pay more attention to the content information of important fields when generating images. Next, images corresponding to the text matrix are generated based on multiple target weights. This ensures that the generated images closely match the semantic features of the text information, improving the accuracy and quality of image generation. It can automatically generate images of home smart devices that match semantic features based on the text information input by the user, reducing manual intervention and improving the accuracy and quality of image generation. This can meet the urgent need for diversified and efficient image generation in the smart home field. Attached Figure Description

[0046] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings, in which the same or similar reference numerals denote the same or similar features, and the drawings are not drawn to scale.

[0047] Figure 1 This is a schematic flowchart of an image generation method provided in an embodiment of this application;

[0048] Figure 2 This is a schematic flowchart of an image generation method provided in an embodiment of this application;

[0049] Figure 3 This is a schematic flowchart of an image generation method provided in an embodiment of this application;

[0050] Figure 4 This is a schematic flowchart of an image generation method provided in an embodiment of this application;

[0051] Figure 5 This is a schematic flowchart of an image generation method provided in an embodiment of this application;

[0052] Figure 6 This is a schematic flowchart of an image generation method provided in an embodiment of this application;

[0053] Figure 7 This is a schematic diagram of an image generated by the image generation method provided in the embodiments of this application;

[0054] Figure 8 This is a schematic diagram of images collected manually in related technologies;

[0055] Figure 9 This is a schematic diagram of the structure of an image generation apparatus provided in an embodiment of this application;

[0056] Figure 10 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this application. Detailed Implementation

[0057] The features and exemplary embodiments of various aspects of this application will now be described in detail. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only configured to explain this application and are not configured to limit this application. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples of this application.

[0058] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0059] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0060] Various modifications and variations can be made to this application without departing from its spirit or scope, which will be apparent to those skilled in the art. Therefore, this application is intended to cover modifications and variations falling within the scope of the corresponding claims (the claimed technical solutions) and their equivalents. It should be noted that the embodiments provided in this application can be combined with each other without contradiction.

[0061] Before describing the technical solutions provided in the embodiments of this application, in order to facilitate understanding of the embodiments of this application, this application first specifically explains the problems existing in the related technologies:

[0062] With the rapid development of the smart home industry, the functions of smart home devices are constantly being innovated, and the types of devices are becoming increasingly complex with significant differences between manufacturers and models. This poses a huge challenge to device management and identification. As smart home devices become more and more widespread, how to quickly and accurately locate and identify these devices to improve device connection speed and stability has become a critical issue that urgently needs to be addressed.

[0063] While related deep learning-based object detection and optical character recognition technologies have been applied to device recognition to some extent, when incorporating images into new business scenarios, they require significant manpower and resources to collect and manually classify sample images. This results in high upfront costs, difficulty in guaranteeing the effectiveness of small-sample training, and a heavy reliance on constantly updated real training images for model optimization. Furthermore, manually collecting sample images is inefficient, and while generative adversarial networks can generate similar images, they lack scalability and still require manual annotation.

[0064] Therefore, the relevant technologies suffer from the problem that manual collection of sample images is inefficient and cannot meet the diverse and efficient image generation needs of the smart home field.

[0065] Based on this, embodiments of this application provide an image generation method, apparatus, device, medium, and program product that can automatically generate installation images of smart home devices that fit semantic features based on text information. This can reduce manual intervention, improve the accuracy and quality of image generation, and meet the urgent need for diversified and efficient image generation in the smart home field.

[0066] The image generation method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0067] Figure 1 This is a schematic flowchart of an image generation method provided in an embodiment of this application, such as... Figure 1 As shown, the method may include steps S110 to S160.

[0068] S110, in response to the image generation request, obtain the text information in the image generation request, the text information including the content information of multiple first fields.

[0069] The text information entered by the user is structured data, including content information from multiple fields.

[0070] For example, the text information entered by the user is device description information, and the first field may include at least one of the following: device type field, device color field, device style field, device connection method field, text information field on the device nameplate image, and QR code content information field on the device nameplate image.

[0071] Specifically, when an image generation request is received from a user, the text information contained in the request can be extracted. This text information includes various aspects such as device type, device color, device style, device connection method, and text information and QR code content on the device nameplate image, providing basic data for generating images that meet the requirements based on this information.

[0072] S120: Based on the content information of multiple first fields, generate the content information of the second field according to the preset content generation rules of the second field.

[0073] The first field can be a basic description of the device's attributes.

[0074] The second field can be a field that is further processed or combined based on the basic attributes of the first field.

[0075] In one example, the pre-defined rules for generating the content of the second field may include field combination rules and / or semantic expansion rules, etc.

[0076] Specifically, the content information of the second field can be generated based on the content information contained in multiple first fields, according to pre-defined second field content generation rules. These second field content generation rules are diverse; they can be field combination rules, which combine and integrate the content of different first fields according to a specific order and format; or they can be semantic expansion rules, such as enriching and expanding the semantics by adding relevant descriptions, explanations, or related information based on the first field content, thereby generating the second field content information to meet different business needs or information presentation requirements.

[0077] S130, in the preset target correspondence between field content and word vector, obtain the target word vector corresponding to the content information of the first field and the target word vector corresponding to the content information of the second field.

[0078] The word vectors in the preset correspondence between field content and word vector target are obtained through pre-training. These word vectors correspond to specific field content and can express the semantic information contained in the corresponding field content more accurately and effectively.

[0079] Specifically, based on the pre-trained correspondence between field content and word vectors, the target word vectors corresponding to the content information of the first field and the second field can be obtained respectively.

[0080] S140: Generate a text matrix based on multiple target word vectors. The text matrix includes multiple target word vectors arranged in a preset target order.

[0081] The target order refers to the pre-trained sequential arrangement of the content of each field. Multiple target word vectors are arranged according to this target order to form an ordered structured matrix, which can be called a text matrix and represents the overall text information.

[0082] Specifically, according to the pre-set order of the various fields, multiple target word vectors can be arranged sequentially to form an ordered structured matrix, or text matrix, which represents the overall expression of the text information.

[0083] S150: Obtain the target weight corresponding to the target word vector from the preset field-attention weight target correspondence relationship.

[0084] The attention weights in the preset field-attention weight target correspondence are obtained through pre-training.

[0085] Specifically, in the pre-constructed target correspondence between fields and attention weights, the target weights corresponding to the target word vectors can be obtained. The target weights reflect the importance of different fields in the image generation process, which helps the model pay more attention to the content information of important fields when generating images.

[0086] S160 generates an image corresponding to the text matrix based on multiple target weights.

[0087] Specifically, based on the target weights corresponding to each target word vector in the text matrix, an image corresponding to the text matrix can be generated.

[0088] For example, target weight values ​​can be mapped to visual attributes such as brightness, color saturation, or transparency of image pixels. At the same time, combined with the structural information of the text matrix (such as the order of word vectors), the target weights and text matrix structure can be transformed into the pixel layout and color distribution of the image. Through this series of transformations and combinations, an image that can intuitively reflect the semantic features of the text matrix can be generated.

[0089] According to the image generation method provided in this application embodiment, in response to an image generation request, text information from the image generation request is obtained. This text information includes content information from multiple fields, which meticulously describe the content of the desired generated image in text form, providing key semantic guidance for subsequent image generation. Then, based on the content information of multiple first fields, the content information of the second field is generated according to a preset content generation rule for the second field (this rule can be flexibly set as a field combination rule and / or semantic expansion rule, etc.), thereby expanding the text information, enriching the text expression for different scenarios and types, and laying the foundation for generating diverse images. Next, in the preset target correspondence between field content and word vectors, the target word vectors corresponding to the content information of the first field and the content information of the second field are obtained. Then, multiple target word vectors are arranged according to a preset target order to generate a text matrix. This matrix forms the overall semantic representation of the text, enabling the text semantics to be presented digitally and in a structured manner. Finally, in the preset target correspondence between fields and attention weights, the target weights corresponding to the target word vectors are obtained. These target weights reflect the importance of different fields in the image generation process, helping the model to pay more attention to the content information of important fields when generating images. Next, images corresponding to the text matrix are generated based on multiple target weights. This ensures that the generated images closely match the semantic features of the text information, improving the accuracy and quality of image generation. It can automatically generate images of home smart devices that match semantic features based on the text information input by the user, reducing manual intervention and improving the accuracy and quality of image generation. This can meet the urgent need for diversified and efficient image generation in the smart home field.

[0090] The following is combined Figures 2-4 This paper describes the process of continuously updating the preset initial generation rules for the content of the second field, the preset initial correspondence between the content and word vectors, the preset initial order, and the preset initial correspondence between the field and attention weight in the image generation method provided in the embodiments of this application, so as to obtain the preset target correspondence between the content and word vectors, the preset target order, and the preset target correspondence between the field and attention weight.

[0091] Figure 2 This is another schematic flowchart of the image generation method provided in the embodiments of this application.

[0092] In some embodiments, such as Figure 2 As shown, before step S110, which responds to the image generation request and obtains the text information in the image generation request, the method may further include steps S171 to S175.

[0093] S171, Obtain historical text information, which includes historical content information of multiple first fields.

[0094] Specifically, historical content information of each first field is obtained to train the target correspondence between the preset content and word vectors, the preset target order, and the target correspondence between the preset fields and attention weights.

[0095] S172, based on the historical content information of multiple first fields, generate the historical content information of the second field according to the preset initial generation rules of the second field content.

[0096] Specifically, after obtaining the historical content information corresponding to multiple first fields as basic data, the historical content information of the first fields can be combined and semantically expanded according to the pre-set initial generation rules of the second field content to generate the historical content information of the second field.

[0097] S173. Based on the preset initial correspondence between field content and word vector, the preset initial order, and the preset initial correspondence between fields and attention weights, generate initial historical images corresponding to the historical content information of multiple fields. The historical content information of the fields includes the historical content information of the first field and the historical content information of the second field.

[0098] Specifically, based on the pre-defined initial correspondence between field content and word vectors, the pre-determined initial order, and the pre-defined initial correspondence between fields and attention weights, an image corresponding to the historical content information of multiple fields (the historical content information of the first field and the historical content information of the second field) can be generated. This image can be called the initial historical image.

[0099] S174, identify the image content information in the initial historical image, and match the image content information with the historical content information of multiple fields to obtain the matching result.

[0100] Specifically, by analyzing and processing the generated initial historical image, the image content information contained in the image can be identified. Then, the identified image content information is compared with the historical content information of multiple fields to finally obtain a matching result.

[0101] For example, the reward value for a field that matches successfully is 1, and the reward value for a field that fails to match is 0. The reward values ​​for multiple fields are multiplied by their respective weights and then added together to obtain the total reward value. The total reward value is then compared with a preset threshold. If the total reward value is greater than the preset threshold, the matching result is a successful match; if the total reward value is less than or equal to the preset threshold, the matching result is a failed match.

[0102] S175, if the matching result is a failure, the preset initial generation rules for the second field content, the preset initial correspondence between the content and the word vector, the preset initial order, and the preset initial correspondence between the field and the attention weight are iteratively updated to obtain the preset initial generation rules for the second field content, the preset target correspondence between the content and the word vector, the preset target order, and the preset target correspondence between the field and the attention weight.

[0103] Specifically, if the matching result is a failure, for example, if the total reward value corresponding to multiple fields is less than a preset threshold, it indicates a failure. In this case, the preset initial generation rules for the content of the second field, the preset initial correspondence between the content and word vectors, the preset initial order, and the preset initial correspondence between the fields and attention weights can be continuously updated and adjusted according to the reward value corresponding to each field, until the preset content generation rules for the second field, the preset target correspondence between the content and word vectors, the preset target order, and the preset target correspondence between the fields and attention weights are obtained. Among them, the matching result obtained according to the "preset target correspondence between the content and word vectors, the preset target order, and the preset target correspondence between the fields and attention weights" is a successful match (e.g., the total reward value corresponding to multiple fields is greater than the preset threshold).

[0104] This application embodiment can continuously optimize a series of preset relationships (preset initial generation rules for the content of the second field, preset initial correspondence between content and word vectors, preset initial order, and preset initial correspondence between fields and attention weights) until the preset content generation rules for the second field, the preset target correspondence between content and word vectors, the preset target order, and the preset target correspondence between fields and attention weights are obtained. Specifically, the initial historical image corresponding to the historical text information is first generated using the preset initial generation rules for the content of the second field, the preset initial correspondence between content and word vectors, the preset initial order, and the preset initial correspondence between fields and attention weights. Then, the historical text information is matched with the image content information in the initial historical image to obtain a matching result. Based on the matching result, the series of preset relationships (preset initial generation rules for the content of the second field, preset initial correspondence between content and word vectors, preset initial order, and preset initial correspondence between fields and attention weights) are continuously optimized until the preset content generation rules for the second field, the preset target correspondence between content and word vectors, the preset target order, and the preset target correspondence between fields and attention weights are obtained. This ensures that the generated image can highly fit the semantic features of the text information, improving the accuracy and quality of image generation.

[0105] For example, the generated image is input into the image quality inspection model to identify and detect smart devices in the image. The detection results (image content) are compared with the set quality inspection results (text information), and the model is further iterated and optimized (by iteratively updating the preset initial generation rules for the second field content, the preset initial correspondence between the content and word vectors, the preset initial order, and the preset initial correspondence between the field and attention weights). This can eliminate the need for manual annotation, and the image quality inspection model can directly participate in the training of the large model, thereby optimizing and iterating the image quality inspection model to improve the accuracy of image recognition.

[0106] Figure 3 This is another schematic flowchart of the image generation method provided in the embodiments of this application.

[0107] In some embodiments, such as Figure 3 As shown, step S173 generates an initial historical image corresponding to the historical content information of multiple fields based on the preset initial correspondence between field content and word vectors, the preset initial order, and the preset initial correspondence between fields and attention weights. This may include steps S1731 to S1734.

[0108] S1731, in the preset initial correspondence between field content and word vector, obtain the initial historical word vector corresponding to the historical content information of the field.

[0109] S1732, Based on multiple initial historical word vectors, generate an initial historical text matrix, which includes multiple initial historical word vectors arranged in a preset initial order.

[0110] The text matrix is ​​a holistic representation of textual information.

[0111] S1733, in the preset initial correspondence between fields and attention weights, obtain the initial historical weights corresponding to the initial historical word vectors.

[0112] S1734, based on multiple initial historical weights, generates the initial historical image corresponding to the initial historical text matrix.

[0113] For example, based on multiple initial historical weights, the initial historical text matrix can be fused with a preset noise matrix to obtain an initial historical image.

[0114] Specifically, the process of generating initial historical images corresponding to the historical content information of multiple fields based on the preset initial correspondence between field content and word vectors, the preset initial order, and the preset initial correspondence between fields and attention weights is as follows: First, obtain the initial historical word vectors corresponding to the historical content information of the fields from the preset initial correspondence between field content and word vectors. Then, arrange these initial historical word vectors according to the preset initial order to generate an initial historical text matrix. Next, obtain the initial historical weights corresponding to the initial historical word vectors from the preset initial correspondence between fields and attention weights. Finally, based on multiple initial historical weights, generate the initial historical image corresponding to the initial historical text matrix. For example, the initial historical text matrix can be fused with a preset noise matrix based on multiple initial historical weights, and then denoised to obtain the initial historical image.

[0115] This application embodiment can gradually generate initial historical images corresponding to the historical content information of multiple fields based on the preset initial correspondence between field content and word vectors, the preset initial order, and the preset initial correspondence between fields and attention weights. That is, first, the initial historical word vectors are obtained, then the initial historical text matrix is ​​generated, then the initial historical weights are obtained, and finally the initial historical image is generated. This application embodiment transforms discrete text into vector representation in a continuous semantic space through the preset mapping between field content and word vectors, effectively preserving the semantic features of the text; by introducing an attention weight mechanism, the model can automatically focus on key field information, enhancing the semantic relevance of image generation; strict order constraints ensure the orderly arrangement of multiple field information in the matrix, providing clear and accurate semantic expression; modular design allows key links such as word vector mapping, matrix construction, and weight allocation to be optimized independently, significantly improving the scalability and training efficiency of the model. This end-to-end design from semantic parsing to spatial modeling not only reduces manual intervention but also achieves intelligent filtering and visualization of complex text information through the attention mechanism.

[0116] Figure 4 This is another schematic flowchart of the image generation method provided in the embodiments of this application.

[0117] In some embodiments, such as Figure 4 As shown, step S174 identifies the image content information in the initial historical image and matches the image content information with the historical content information of multiple fields to obtain the matching result, which may include steps S1741 to S1743.

[0118] S1741, using a preset target detection model, identifies information about the categories of historical objects in the initial historical image.

[0119] S1742 uses a preset text recognition model to identify historical text information in the initial historical image.

[0120] S1743, Match the image content information in the initial historical image with the historical content information of multiple fields to obtain the matching result. The image content information in the initial historical image includes information on the historical object category and historical text information.

[0121] The embodiments of this application employ a target detection model to locate objects and object categories in an image, and use a text recognition model to extract text information from the image. Subsequently, these visual recognition results are compared and verified with structured field data in historical text information, which enhances the accuracy and reliability of image generation. Furthermore, it can automatically detect inconsistencies between field descriptions and visual presentations, providing a clear direction for subsequent image optimization and realizing a closed loop of "generation-verification-optimization".

[0122] Figure 5 This is another schematic flowchart of the image generation method provided in the embodiments of this application.

[0123] In some embodiments, such as Figure 5 As shown, before generating the content information of the second field based on the content information of multiple first fields in step S120 according to the preset content generation rules of the second field, the image generation method may further include steps S181 and S182.

[0124] S181, in the image generation request, obtain the content information of the target field in the text information, where the target field is the device type field.

[0125] S182, in the preset correspondence between device type and content generation rules of the second field, obtain the content generation rule of the second field corresponding to the content information of the target field.

[0126] Specifically, the field data information of the device type field can be extracted from the image generation request, and then the content generation rule of the second field corresponding to the field data information of the device type field can be matched in the preset rule base, so that the exclusive text expansion rule can be automatically called according to different device types.

[0127] In one example, different feature attributes (the content generation rules of the second field) can be set for each smart device. Text information can be expanded based on these feature attributes, enriching the text information for different scenarios and types. Setting different feature attributes for each smart device enriches the expression of text information, making the generated images more realistic and diverse.

[0128] Before generating an image, this embodiment first extracts the device type field data from the image generation request. Then, it automatically matches the content generation rule of the second field specific to that device in the rule base. Therefore, different device types can automatically trigger corresponding text expansion logic, ensuring the accuracy of the generated image and enabling flexible configuration of diverse images. In other words, based on text semantics and parameter settings (the content generation rule of the second field), the corresponding text data can be automatically expanded. Then, a text encoder converts the text description into a high-dimensional vector (target word vector) that the image generation model can understand, which is used to control the generation of the corresponding image. Therefore, only the scene type of the image to be generated and the specific text description of the result to be recognized (the content information of the first field) need to be defined to generate the corresponding type of image, and the model has extremely high scalability.

[0129] Figure 6 This is another schematic flowchart of the image generation method provided in the embodiments of this application.

[0130] In some embodiments, such as Figure 6As shown, step S160 generates an image corresponding to the text matrix based on multiple target weights, which may include step S161.

[0131] S161, based on multiple target weights, the text matrix is ​​fused with a preset noise matrix and then denoised to obtain the image corresponding to the text matrix.

[0132] In this embodiment, the noise matrix is ​​used as the starting point for generation to introduce initial randomness. After being fused with the text matrix, it is gradually denoised, which increases the diversity and richness of the image and achieves a dynamic balance between semantic constraints and image generation diversity.

[0133] In one embodiment, the method for generating the image may include:

[0134] 1) The training obtains the preset content generation rules for the second field, the preset target correspondence between the content and the word vector, the preset target order, and the preset target correspondence between the field and the attention weight.

[0135] 1.1) Obtain historical text information, which includes historical content information of multiple first fields.

[0136] 1.2) Based on the historical content information of multiple first fields, generate the historical content information of the second field according to the preset initial generation rules of the second field content.

[0137] 1.3) Based on the preset initial correspondence between field content and word vectors, the preset initial order, and the preset initial correspondence between fields and attention weights, generate initial historical images corresponding to the historical content information of multiple fields. The historical content information of the fields includes the historical content information of the first field and the historical content information of the second field. Identify the image content information in the initial historical images and match the image content information with the historical content information of multiple fields to obtain the matching results.

[0138] For example, algorithms for recognizing image content information in initial historical images employ neural network models (such as FasterRegion-based Convolutional Neural Network, FasterRCNN) to perform global target localization in the image and detect target objects using the Non-Maximum Suppression (NMS) algorithm. The text recognition model consists of a two-stage algorithm: a Connectionist Text Proposal Network (CTPN) and a Convolutional Recurrent Neural Network (CRNN). CTPN is used to locate text regions in the image, while CRNN decodes these text regions, converting them into readable text symbols.

[0139] 1.4) In the case of a failed match, the preset initial generation rules for the second field content, the preset initial correspondence between the content and word vectors, the preset initial order, and the preset initial correspondence between the field and attention weights are iteratively updated to obtain the preset initial generation rules for the second field content, the preset target correspondence between the content and word vectors, the preset target order, and the preset target correspondence between the field and attention weights. Among these, the matching result corresponding to the image generated based on the preset initial generation rules for the second field content, the preset target correspondence between the content and word vectors, the preset target order, and the preset target correspondence between the field and attention weights is considered a successful match.

[0140] 2) Generate images based on text information using preset content generation rules for the second field, preset target correspondence between content and word vectors, preset target order, and preset target correspondence between fields and attention weights.

[0141] 2.1) In response to the image generation request, obtain device description information (text information) about the image from the image generation request. The text information includes the content information of multiple first fields. For example, device type field, device color field, device style field, device connection method field, text information field on the device nameplate image, and QR code content information field on the device nameplate image.

[0142] 2.2) In the image generation request, obtain the content information of the target field in the text information. The target field is the device type field.

[0143] 2.3) In the preset correspondence between device type and content generation rules of the second field, obtain the content generation rules of the second field corresponding to the content information of the target field (such as field combination rules and / or semantic expansion rules).

[0144] 2.4) Based on the content information of multiple first fields, generate the content information of the second field according to the preset content generation rules of the second field. In the preset target correspondence between field content and word vectors, obtain the target word vectors corresponding to the content information of the first field and the target word vectors corresponding to the content information of the second field.

[0145] 2.5) Based on multiple target word vectors, a text matrix is ​​generated. The text matrix (a holistic representation of the text information) includes multiple target word vectors arranged in a preset target order. The target weights corresponding to the target word vectors are obtained from a preset field-attention weight correspondence. Based on multiple target weights, the text matrix is ​​fused with a preset noise matrix and then denoised to obtain the image corresponding to the text matrix.

[0146] 3) Calculate the matching results (the matching results of the image content and the first field content in the text information) based on the image generated using the preset second field content generation rules, the preset content and word vector target correspondence, the preset target order, and the preset field and attention weight target correspondence. Continuously optimize the preset second field content generation rules, the preset content and word vector target correspondence, the preset target order, and the preset field and attention weight target correspondence based on the matching results.

[0147] This application embodiment acquires historical text information, generates the content of the second field according to initial rules, and then generates an initial image based on the initial correspondence and performs recognition and matching. If the matching fails, the initial correspondence is iteratively updated until successful. When generating the image, it responds to the request to acquire text information, generates the content of the second field according to the corresponding rules of the target field, acquires the target word vector to generate a text matrix, combines the target weight and noise matrix to denoise and obtain the image, and finally calculates the matching result and optimizes the correspondence (preset second field content generation rules, preset target correspondence between content and word vectors, preset target order, and preset target correspondence between fields and attention weights). It can automatically generate high-quality images that fit the semantics of the text, improve the accuracy and efficiency of image generation, and achieve a high degree of matching between the image and text information.

[0148] In one embodiment, a Stable Diffusion (SD) model is used to generate images corresponding to text information, automatically generating image data and solving the problem of low efficiency in manual image collection. After image collection, an image recognition module (object detection module and text recognition module) is used to perform further object detection, classification, and text recognition operations on the output images to obtain the content information in the images. Then, the image content is matched with the text information input by the user to obtain the matching results, and the model is iteratively updated based on the matching results.

[0149] For example, the main modules of the Stable Diffusion (SD) model are as follows: A CLIP Text Encoder is used as a pre-module to encode the input text information, generating a text embedding feature matrix, i.e., a text matrix. This text matrix and a random Gaussian noise matrix are then fed into the U-Net neural network architecture. A scheduler algorithm is used for iterative denoising, generating denoised latent features after N iterations. The denoised latent features are then fed into the decoder module of a Variational Autoencoder (VAE) to reconstruct the pixel-level image.

[0150] For example, the Stable Diffusion (SD) model also includes a text augmentation module, which can automatically augment the corresponding text data based on the input text information and parameter settings (the content generation rules of the second field). It can also dynamically adjust the content generation rules of the second field based on the image quality inspection results (matching results) fed back by the image recognition module, thereby improving the semantic consistency between the text description and the generated image and consolidating the potential correspondence between local regions of the image and the semantic labels of the text.

[0151] In another embodiment, the stable diffusion SD model and image recognition module are modeled based on deep reinforcement learning. The basic model of reinforcement learning can be represented by a Markov Decision Process (MDP), and a Markov Decision Process (MDP) can be represented by a quadruple.<S,A,R,T> express.

[0152] For example, define a quadruple.<S,A,R,T> They are represented as follows:

[0153] S stands for State Space. Within the system, the system can be viewed as an agent. For example, the user inputs textual descriptions of a router device, which are expanded by a text expansion module. The expanded text module outputs the expanded text information about the router device (the content of the first and second fields). This expanded text is then processed by a CLIP Text Encoder model using a Stable Diffusion (SD) model to extract text embeddings. The Stable Diffusion (SD) model iterates through a U-Net neural network architecture to generate denoised latent features. The text embeddings feature matrix and the latent features are combined to form the feature attribute state value S. In other words, S represents the state of the preset initial generation rules for the second field content, the preset initial correspondence between the content and word vectors, the preset initial order, and the preset initial correspondence between the fields and attention weights. Whether S needs updating depends on the matching result.

[0154] A represents the Action Space. The CrossAttention module in the Stable Diffusion (SD) model is crucial for controlling image content generation using input text information. The input text information, encoded by the CLIP Text Encoder model, forms text embeddings. The CrossAttention module couples the semantic information corresponding to these text embeddings with the corresponding semantic information in the image, outputting a new latent feature representing the feature correspondence between the text and the image. The core idea of ​​the Attention mechanism is to allow the model to focus on the most relevant information when processing the input sequence. Different values ​​(attention weights) can be set for different feature attributes (keys, such as the first field and the second field), resulting in different attention values ​​when the query sequence is coupled with key-value pairs. In reinforcement learning frameworks, the values ​​in the Attention mechanism can be automatically explored and learned based on the reward function R (matching result), which means continuously updating the pre-defined correspondence between fields and attention weights. Different values ​​(attention weights) represent the level of attention given to the corresponding semantic information in the image. The higher the value (attention weight), the higher the weight of the corresponding feature attribute, and the greater the weight of the corresponding semantic information in image generation. Therefore, the set of values ​​(attention weights) can be used as the action set A.

[0155] T is the state transition function. Once the state space S and action space A are determined, T is also randomly determined. T(s, a, s') represents the probability of performing action a in state s and transitioning to state s'. In other words, i is the iterative update of the preset initial generation rules for the second field content, the preset initial correspondence between the content and word vectors, the preset initial order, and the preset initial correspondence between the fields and attention weights.

[0156] R is the reward function, defined as the matching results of each attribute in the text information. The denoised latent features generated after iteration of the U-Net neural network architecture are reconstructed into pixel-level images by the decoder module of the Variational Autoencoder (VAE). The generated images are input into the image recognition module for detection and target recognition. The recognition results are compared with each feature attribute in the text information. If the match is consistent, the reward value for that feature attribute is 1; otherwise, it is 0 (for example, the reward value for the first field 1 that matches is 1, and the reward value for the first field 2 that does not match is 0). If the initial input text information contains N feature attributes (N first fields), the corresponding reward function R∈[0,N], and the N reward values ​​together constitute the reward function (that is, the N reward values ​​together generate the matching result).

[0157] After defining the basic parameters of the reinforcement learning model, it is necessary to continuously iterate and update the basic model parameters (preset initial generation rules for the second field content, preset initial correspondence between content and word vectors, preset initial order, and preset initial correspondence between fields and attention weights) to obtain the preset second field content generation rules, preset target correspondence between content and word vectors, preset target order, and preset target correspondence between fields and attention weights. Among them, the matching result of the image generated based on the preset second field content generation rules, preset target correspondence between content and word vectors, preset target order, and preset target correspondence between fields and attention weights is considered a successful match.

[0158] In one example, the Q-learning algorithm is used to iteratively update the basic model parameters (such as the preset initial generation rules for the second field content, the preset initial correspondence between the content and word vectors, the preset initial order, and the preset initial correspondence between the field and attention weights) to obtain the preset content generation rules for the second field, the preset target correspondence between the content and word vectors, the preset target order, and the preset target correspondence between the field and attention weights.

[0159] It should be noted that since the goal of reinforcement learning is composed of the reward function passed from the environment, the goal of learning is to find a strategy that maximizes the long-term cumulative reward, which means maximizing the long-term cumulative reward rather than the immediate reward value.

[0160] The basic idea behind Q-learning training models is to use tuples (s) consisting of (state, behavior, reward, next state) t a t r t+1s t+1 ) is used as the training sample for training, where s t Given the current state, a t For the Action executed in the current state, r t+1 The reward obtained after performing the Action, s t+1 For the next state, the feature is (s t a t The learning objective is r. t+1 +γ·max a Q(s t+1 a) This learning objective is the reward obtained from the current action plus the maximum expected value that can be obtained in the next step. Here, γ is the discount factor, which determines the importance of future rewards in learning. It is generally set to a value slightly smaller than 1. Thus, the entire Q-learning process can be written as the following formula:

[0161]

[0162] The learning rate determines the ratio of newly acquired sample information to previously acquired information. It is usually set to a relatively small value to ensure the stability of the learning process and to ensure final convergence.

[0163] It should be noted that in this case, the learned action-value function Q directly approximates the optimal action-value function, independent of the adopted policy. As can be seen from the above formula, the objective of single-step Q-learning is r. t+1 +γmaxQ(s t+1 a) , each step uses the maximum Q(s) t+1 a) value — equivalent to using Q(s) t+1 a) The action with the largest value, and the action selected according to the current policy. tThe strategy used is independent. This simplifies the algorithm to some extent and facilitates early convergence. However, if the agent chooses the action with the highest expected value for each Q-learning step, i.e., choosing the action with the highest target Q-value at every step, it cannot try more actions and is easily trapped in local optima. To avoid this problem, a strategy method that randomly generates an action can be used, also known as exploration, which explores what kind of reward an unknown action will produce. To obtain the globally optimal policy behavior, a balance needs to be struck between exploration and exploitation. Then, based on the adaptive learning capability of deep reinforcement learning, the parameters representing the correspondence between image semantic information and text semantic information in the Stable Diffusion (SD) model are dynamically adjusted according to the matching degree between image recognition results and text feature information, i.e., the reward value R, thereby directly affecting the image output by the Stable Diffusion (SD) model. The generated image is input into the image recognition module. Based on the results, it can be determined which feature attributes of the generated image do not match the semantic information of the text. The selected action value 'a' is then adjusted, as are the weights of the parameters representing the correspondence between the image and the text semantic information. The output latent feature is input into the U-Net neural network architecture for further iterative optimization, making the generated image more closely match the text semantic information. This achieves continuous iterative updates to the basic model parameters (preset initial generation rules for the second field content, preset initial correspondence between content and word vectors, preset initial order, and preset initial correspondence between fields and attention weights), resulting in the preset second field content generation rules, the preset target correspondence between content and word vectors, the preset target order, and the preset target correspondence between fields and attention weights.

[0164] This application's embodiments generate text-corresponding images based on a stable diffusion (SD) model and process them using an image recognition module. Deep reinforcement learning is then used to model and optimize the model, enabling automatic image data generation and solving the problem of low efficiency in manual collection. The output images are further processed to obtain content information, which is then matched with the text, and the model is iteratively updated. By defining basic reinforcement learning model parameters (such as state space and action space), the Q-learning algorithm is used to continuously update these parameters. By balancing exploration and utilization, the parameters representing the correspondence between image and text semantic information are dynamically adjusted based on the reward value, ultimately obtaining the target correspondence. This makes the generated images more closely match the text semantic information, effectively improving model performance and the quality of the generated images.

[0165] It should be noted that the embodiments of this application are designed based on the idea of ​​reinforcement learning, giving the system independent learning capabilities. Throughout the process, there is no need to set sample labels, manually review whether the image recognition results match the text information, or manually adjust model parameters. Only the state s and action 'a', and the reward 'r' value for performing action 'a' in state s, need to be set. The system automatically and autonomously learns and adjusts the SD model parameters based on the matching degree between the image recognition results and text feature information (i.e., the reward value 'r'). In other words, based on the reward value 'r', it identifies which feature attributes are not matched, and then adjusts the weight values ​​of the text feature information to image semantic information mapping in the SD model's attention mechanism, giving more attention to the corresponding feature attributes and making the images generated by the SD model more closely match the text feature information. This process eliminates manual detection of matching results and manual intervention in parameter adjustment, making the system more intelligent and reducing the impact of human intervention.

[0166] In one example, the device is a home indoor camera device. The first field includes a device type field, a device color field, a device style field, a device connection method field, a text information field on the device nameplate image, and a QR code content information field on the device nameplate image.

[0167] First, acquire historical text information (with a capacity of H), which is the text description information of the camera device images. For example, historical text information includes device type, color, style, device connection method (fiber optic, Ethernet cable, wireless), information on the device nameplate (MAC address, SN, SSID, product name, and model), and QR code information on the device nameplate (MAC address and SSID). The text expansion module can combine and semantically expand the historical text information, arranging it in a certain order to obtain text records: A connection diagram of a white smart router from a certain manufacturer, with fiber optic connection method. The QR code, MAC address, SSID, SN, product name, and model information on the router's nameplate are respectively xxx, resulting in a text information sample set.

[0168] Next, a text record is selected from the historical text information sample set, and the text embeddings feature matrix (77×768) is extracted by the text encoder (CLIP TextEncoder). At the same time, a random Gaussian noise matrix (dimension 64×64×4, corresponding to a 512×512 resolution image) is initialized. The text embeddings feature matrix and the random Gaussian noise matrix are fed into the neural network architecture U-Net through the cross-attention module for iterative denoising. After N iterations, the denoised latent features are generated as the initial state of the sample, which is s0 by default.

[0169] The denoised latent features are processed by the decoder module of a Variational Autoencoder (VAE) to reconstruct the camera image (512x512 resolution). The image content and corresponding text information are then input into the image recognition module. The image recognition module detects and discriminates the target, comparing the identified content with the feature attributes in the text information. If they match, the corresponding feature attribute receives a reward of 1; otherwise, it receives 0. Since the text information has 10 feature attributes (n=10), the reward function R has a dimension of 1x10.

[0170] Next, initialize the action a0 = {a1, a2, ... a n}, where n is the number of text information feature attributes, initialized a1 = a2 = ... a n =0.5, where a n ∈[0,1], each step is quantized by 0.2, so the size of the action set A is 5. n Based on state S and reward function R, randomly select an action a to perform, obtain reward r and the next state s′, and iterate until state s converges. The learned Q(s) t ,a t The values ​​are updated in the Q-table. The model parameters θ and ω are iteratively trained until the experience pool contains W training data points, where W = 100,000. After training, n records are selected from the sample set for validation. Based on the learned Q-table, the optimal solution, i.e., the best matching scheme, is quickly obtained according to the current state. This achieves iterative updates to the preset initial generation rules for the second field, the preset initial correspondence between the content and word vectors, the preset initial order, and the preset initial correspondence between the field and attention weights, until the preset second field content generation rules, the preset target correspondence between the content and word vectors, the preset target order, and the preset target correspondence between the field and attention weights are obtained.

[0171] The final training-generated sample image set (see [link]) Figure 7 Compared to manually collected sample sets (see [reference]), Figure 8 The images are higher resolution, the targets are clearer, the backgrounds are cleaner and tidier, and there are no messy lines, which is more conducive to the model's target detection and improves the overall recognition accuracy.

[0172] It should be noted that within the reinforcement learning framework, as the entire system autonomously learns and iterates, the closer the images generated by the SD model are to the semantic information of the text and the higher the matching degree with the text's feature attributes, the higher the quality of the final training images generated. With high-quality training images, the image object recognition and OCR models show significant iterative optimization, resulting in a 10% improvement in model recognition accuracy.

[0173] This application proposes a reinforcement learning-based approach that integrates a text augmentation module, a text-to-image generation module, and an image recognition module into a single system. This system enables autonomous learning throughout the process, automatically adjusting the model parameters based on predefined states. It autonomously learns the optimal model parameters (predefined content generation rules for the second field, preset target correspondences between content and word vectors, preset target order, and preset target correspondences between fields and attention weights) based on the input text information, thereby generating images that most closely resemble the semantics of the text.

[0174] It should be noted that the embodiments of this application can be applied to the home smart device installation and identification business process. For newly accessed business scenario images that need to be detected, even in the absence of training samples, a large number of high-quality images with different detection results can be generated based on the input text information. The generation of high-quality training images greatly reduces the manpower and time costs of manually collecting and labeling images. The time for collecting and labeling training images for each scenario is reduced from 2 person-months to 0.3 person-months. High-quality training data enables the model to iterate quickly, reducing the model training time and shortening the development version iteration by 2 cycles. The image detection capability for new business scenarios can reach the accuracy required by the group as soon as it goes online, and the model training iteration is shortened by an average of 2 cycles, saving 1 person-month of R&D manpower costs. The entire system is autonomous learning, requiring no manual intervention or manual adjustment of model parameters. It can be extended to different business scenarios, with high scalability and extensibility. The final output results and images of the model are based on the initially set text feature information, so the training direction and results of the entire model are controllable and observable.

[0175] It should be noted that the embodiments in this application are text-to-image generation models based on the Stable Diffusion (SD) model. These models can generate corresponding high-quality images based on input text information. Compared to manually collecting images, this method is time-saving and labor-saving. Compared to GAN networks, it can be parallelized during training, potentially resulting in lower training costs on large-scale datasets. Furthermore, the SD model can accept text descriptions as input and directly generate images that match the descriptions. The specific content and style of the generated images can be controlled through text prompts, conditional inputs, or adjustments to model parameters, offering greater controllability.

[0176] It should also be noted that there are many types of smart home devices, and even within the same type, different manufacturers and models vary. To address this, this application's embodiments include a text expansion module (the generation rules for the second field). This module allows for the setting of different feature attributes for each smart device, enriching the text information for different scenarios and types based on these attributes. This customized and diversified feature attribute model enables the SD model to generate more realistic images from text that better align with the text's semantics. Furthermore, based on the home device installation scenario, specific installation environment and installation method features can be assigned to the text, generating sample images corresponding to the home smart device installation and maintenance scenario, such as connection methods, connection locations, and indicator light flashing on the router.

[0177] Based on the same inventive concept, embodiments of this application also provide an image generation apparatus, such as... Figure 9 As shown, the device 900 may include an acquisition module 910 and a generation module 920:

[0178] The acquisition module 910 is used to acquire text information in the image generation request in response to the image generation request. The text information includes the content information of multiple first fields.

[0179] The generation module 920 is used to generate the content information of the second field based on the content information of multiple first fields, according to the preset content generation rules of the second field.

[0180] The acquisition module 910 is also used to acquire the target word vector corresponding to the content information of the first field and the target word vector corresponding to the content information of the second field from the preset target correspondence between field content and word vector;

[0181] The generation module 920 is also used to generate a text matrix based on multiple target word vectors, wherein the text matrix includes multiple target word vectors arranged in a preset target order;

[0182] The acquisition module 910 is also used to obtain the target weight corresponding to the target word vector in the preset target correspondence between fields and attention weights;

[0183] The generation module 920 is also used to generate an image corresponding to the text matrix based on multiple target weights.

[0184] The image generation apparatus according to embodiments of this application, in response to an image generation request, acquires text information from the request. This text information includes content information from multiple fields, which meticulously describe the desired appearance of the image in text form, providing key semantic guidance for subsequent image generation. Then, based on the content information of multiple first fields, the apparatus generates content information for the second field according to a preset content generation rule (this rule can be flexibly set as a field combination rule and / or semantic expansion rule, etc.), thereby expanding the text information, enriching the text expression for different scenarios and types, and laying the foundation for generating diverse images. Next, in a preset target correspondence between field content and word vectors, the apparatus acquires the target word vectors corresponding to the content information of the first field and the second field. These target word vectors are then arranged in a preset target order to generate a text matrix. This matrix forms a complete semantic representation of the text, enabling the text semantics to be presented digitally and in a structured manner. Finally, in a preset target correspondence between fields and attention weights, the apparatus acquires the target weights corresponding to the target word vectors. These target weights reflect the importance of different fields in the image generation process, helping the model to pay more attention to the content information of important fields when generating images. Next, images corresponding to the text matrix are generated based on multiple target weights. This ensures that the generated images closely match the semantic features of the text information, improving the accuracy and quality of image generation. It can automatically generate images of home smart devices that match semantic features based on the text information input by the user, reducing manual intervention and improving the accuracy and quality of image generation. This can meet the urgent need for diversified and efficient image generation in the smart home field.

[0185] In some embodiments, before the acquisition module acquires the text information in the image generation request in response to the image generation request, the apparatus further includes a matching module and an updating module:

[0186] The acquisition module is also used to acquire historical text information, which includes historical content information of multiple first fields;

[0187] The generation module is also used to generate historical content information of the second field based on the historical content information of multiple first fields, according to the preset initial generation rules of the second field content;

[0188] The generation module is also used to generate initial historical images corresponding to the historical content information of multiple fields based on the preset initial correspondence between field content and word vectors, the preset initial order, and the preset initial correspondence between fields and attention weights. The historical content information of the fields includes the historical content information of the first field and the historical content information of the second field.

[0189] The matching module is used to identify image content information in the initial historical images and match the image content information with historical content information in multiple fields to obtain the matching results;

[0190] The update module is used to iteratively update the preset initial generation rules for the second field content, the preset initial correspondence between the content and word vectors, the preset initial order, and the preset initial correspondence between the field and attention weights when the matching result is a failure. This results in the preset initial generation rules for the second field content, the preset target correspondence between the content and word vectors, the preset target order, and the preset target correspondence between the field and attention weights.

[0191] In some embodiments, the generation module is used to generate an initial historical image corresponding to the historical content information of multiple fields based on a preset initial correspondence between field content and word vectors, a preset initial order, and a preset initial correspondence between fields and attention weights. Specifically, it can be used for:

[0192] In the preset initial correspondence between field content and word vectors, obtain the initial historical word vectors corresponding to the historical content information of the field;

[0193] An initial historical text matrix is ​​generated based on multiple initial historical word vectors. The initial historical text matrix includes multiple initial historical word vectors arranged in a preset initial order.

[0194] In the preset initial correspondence between fields and attention weights, obtain the initial historical weights corresponding to the initial historical word vectors;

[0195] Based on multiple initial historical weights, an initial historical image corresponding to the initial historical text matrix is ​​generated.

[0196] In some embodiments, the matching module is used to identify image content information in the initial historical image and match the image content information with historical content information in multiple fields to obtain a matching result, which can be specifically used for:

[0197] Using a pre-defined target detection model, information on the categories of historical objects in the initial historical images is identified;

[0198] Using a pre-defined text recognition model, historical text information in the initial historical images is identified;

[0199] The image content information in the initial historical image is matched with the historical content information in multiple fields to obtain the matching results. The image content information in the initial historical image includes information on the historical object category and historical text information.

[0200] In some embodiments, before the generation module generates the content information of the second field based on the content information of multiple first fields and according to a preset content generation rule for the second field, the apparatus further includes:

[0201] The acquisition module is also used to acquire the content information of the target field in the text information in the image generation request, where the target field is the device type field;

[0202] The acquisition module is also used to acquire the content generation rule of the second field corresponding to the content information of the target field, based on the preset correspondence between device type and content generation rule of the second field.

[0203] In some embodiments, the generation module is used to generate an image corresponding to the text matrix based on multiple target weights, specifically for:

[0204] Based on multiple target weights, the text matrix is ​​fused with a preset noise matrix and then denoised to obtain the image corresponding to the text matrix.

[0205] The various modules in the image generation apparatus provided in this application embodiment can achieve... Figures 1 to 6 The functions of each step in the provided image generation method, and the corresponding technical effects they achieve, will not be elaborated upon here for the sake of brevity.

[0206] Figure 10 A schematic diagram of the hardware structure of the image generation device provided in an embodiment of this application is shown.

[0207] The image generating device may include a processor 1001 and a memory 1002 storing computer program instructions.

[0208] Specifically, the processor 1001 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0209] Memory 1002 may include mass storage for data or instructions. For example, and not limitingly, memory 1002 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where suitable, memory 1002 may include removable or non-removable (or fixed) media. Where suitable, memory 1002 may be internal or external to an image-generating device. In a particular embodiment, memory 1002 is a non-volatile solid-state memory.

[0210] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0211] The processor 1001 reads and executes computer program instructions stored in the memory 1002 to implement any of the image generation methods in the above embodiments.

[0212] In one example, the image generating device may further include a communication interface 1003 and a bus 1004. Wherein, as Figure 10 As shown, the processor 1001, memory 1002, and communication interface 1003 are connected through bus 1004 and complete communication with each other.

[0213] The communication interface 1003 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0214] Bus 1004 includes hardware, software, or both, that couples components of an image-generating device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Linear Predictive Coding (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (Peripheral Component Interconnect-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VESA Local Bus, VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 1004 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0215] This device can execute the image generation method of the embodiments of this application based on the various units / components in the image generation apparatus, thereby achieving a combination Figures 1 to 6 The method for generating the described image.

[0216] Furthermore, in conjunction with the image generation methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the image generation methods in the above embodiments.

[0217] This application also provides a computer program product, wherein the instructions in the computer program product, when executed by a processor of an electronic device, cause the electronic device to perform various processes implementing any of the above-described image generation method embodiments.

[0218] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0219] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0220] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0221] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0222] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for generating an image, characterized in that, include: In response to an image generation request, text information in the image generation request is obtained, wherein the text information includes content information of multiple first fields; Based on the content information of multiple first fields, the content information of the second field is generated according to the preset content generation rules of the second field; In the preset target correspondence between field content and word vector, obtain the target word vector corresponding to the content information of the first field and the target word vector corresponding to the content information of the second field; A text matrix is ​​generated based on the multiple target word vectors, the text matrix including the multiple target word vectors arranged in a preset target order; In the preset correspondence between fields and attention weights, the target weights corresponding to the target word vectors are obtained; Based on the multiple target weights, an image corresponding to the text matrix is ​​generated.

2. The method according to claim 1, characterized in that, The first field includes at least one of the following: device type field, device color field, device style field, device connection method field, text information field on the device nameplate image, and QR code content information field on the device nameplate image.

3. The method according to claim 1, characterized in that, The preset content generation rules for the second field include field combination rules and / or semantic expansion rules.

4. The method according to claim 1, characterized in that, Before obtaining the text information in the image generation request in response to the image generation request, the method further includes: Obtain historical text information, which includes historical content information of multiple first fields; Based on the historical content information of multiple first fields, the historical content information of the second field is generated according to the preset initial generation rules of the second field content; Based on the preset initial correspondence between field content and word vectors, the preset initial order, and the preset initial correspondence between fields and attention weights, an initial historical image corresponding to the historical content information of multiple fields is generated. The historical content information of the fields includes the historical content information of the first field and the historical content information of the second field. Identify the image content information in the initial historical image, and match the image content information with the historical content information of multiple fields to obtain a matching result; If the matching result is a failure, the preset initial generation rule for the content of the second field, the preset initial correspondence between the content and the word vector, the preset initial order, and the preset initial correspondence between the field and the attention weight are iteratively updated to obtain the preset initial generation rule for the content of the second field, the preset target correspondence between the content and the word vector, the preset target order, and the preset target correspondence between the field and the attention weight.

5. The method according to claim 4, characterized in that, The step of generating an initial historical image corresponding to the historical content information of multiple fields based on the preset initial correspondence between field content and word vectors, the preset initial order, and the preset initial correspondence between fields and attention weights includes: In the preset initial correspondence between field content and word vectors, the initial historical word vectors corresponding to the historical content information of the field are obtained; An initial historical text matrix is ​​generated based on the multiple initial historical word vectors, wherein the initial historical text matrix includes the multiple initial historical word vectors arranged in a preset initial order; In the preset initial correspondence between fields and attention weights, obtain the initial historical weights corresponding to the initial historical word vectors; Based on the multiple initial historical weights, an initial historical image corresponding to the initial historical text matrix is ​​generated.

6. The method according to claim 4, characterized in that, The process of identifying image content information in the initial historical image and matching the image content information with historical content information in multiple fields to obtain a matching result includes: Using a preset target detection model, information on the categories of historical objects in the initial historical image is identified; Using a preset text recognition model, historical text information in the initial historical image is identified; The image content information in the initial historical image is matched with the historical content information of the multiple fields to obtain a matching result. The image content information in the initial historical image includes the information of the historical object category and the historical text information.

7. The method according to claim 1, characterized in that, Before generating the content information of the second field based on the content information of multiple first fields according to a preset content generation rule for the second field, the method further includes: In the image generation request, the content information of the target field in the text information is obtained, and the target field is the device type field; Based on the preset correspondence between device type and content generation rules of the second field, obtain the content generation rules of the second field corresponding to the content information of the target field.

8. The method according to any one of claims 1 to 7, characterized in that, The step of generating the image corresponding to the text matrix based on multiple target weights includes: Based on multiple target weights, the text matrix is ​​fused with a preset noise matrix and then denoised to obtain the image corresponding to the text matrix.

9. An image generation apparatus, characterized in that, include: The acquisition module is used to acquire text information in the image generation request in response to the image generation request, wherein the text information includes the content information of multiple first fields; The generation module is used to generate the content information of the second field based on the content information of multiple first fields, according to a preset content generation rule for the second field. The acquisition module is also used to acquire the target word vector corresponding to the content information of the first field and the target word vector corresponding to the content information of the second field from the preset target correspondence between field content and word vector; The generation module is also used to generate a text matrix based on the multiple target word vectors, wherein the text matrix includes the multiple target word vectors arranged in a preset target order; The acquisition module is also used to acquire the target weight corresponding to the target word vector in the preset target correspondence between fields and attention weights; The generation module is also used to generate an image corresponding to the text matrix based on multiple target weights.

10. An image generation apparatus, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the image generation method as described in any one of claims 1 to 8.