Image generation method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to an image generation method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring an initial image and a target description text; performing feature extraction on the target description text to obtain overall description features and object description features; feature extraction is carried out on the initial image to obtain a plurality of image feature layers, and the plurality of image feature layers comprise a semantic feature layer and a relation feature layer; sequentially adjusting the plurality of image feature layers according to the object description features and the overall description features to obtain a plurality of adjusted image feature layers; the multiple adjusted image feature layers are fused, an adjusted image is generated, the adjusted image is used for generating a training image set, and the training image set is used for training an image recognition model. By adopting the method, the efficiency and the image diversity can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image generation method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] In recent years, computer vision technology has made breakthrough progress in the fields of object recognition and target detection, which has greatly promoted the development of applications such as autonomous driving, intelligent security, and medical imaging diagnosis. In the field of computer vision, large-scale, high-quality training image data is crucial, and building a comprehensive and diverse image data set often requires a lot of manpower, material resources, and time investment.

[0003] In related technologies, image datasets can be enriched through data augmentation techniques such as geometric transformation, color transformation, and noise addition to obtain large-scale training image data. However, the image data obtained by this method is still relatively simple and difficult to meet diverse data needs. Summary of the invention

[0004] Based on this, it is necessary to provide an image generation method, apparatus, computer device, computer-readable storage medium and computer program product that can improve efficiency and image diversity in response to the above technical problems.

[0005] In a first aspect, the present application provides an image generation method, comprising:

[0006] Get the initial image and target description text;

[0007] Performing feature extraction on the target description text to obtain overall description features and object description features; performing feature extraction on the initial image to obtain multiple image feature layers, the multiple image feature layers include a semantic feature layer and a relationship feature layer, the semantic feature layer includes at least one of an object feature layer, an attribute feature layer, a relationship feature layer, and an environment feature layer;

[0008] According to the object description feature and the overall description feature, the plurality of image feature layers are adjusted in sequence to obtain a plurality of adjusted image feature layers;

[0009] The adjusted multiple image feature layers are fused to generate an adjusted image, and the adjusted image is used to generate a training image set, and the training image set is used to train an image recognition model.

[0010] In one embodiment, the extracting features from the initial image to obtain multiple image feature layers includes:

[0011] Processing the initial image using a preset graph convolutional neural network to obtain image relationship features corresponding to the initial image, and generating a relationship feature layer according to the image relationship features;

[0012] The initial image is transformed using a preset visual transformation network to obtain image semantic features corresponding to the initial image, and a semantic feature layer is generated according to the image semantic features.

[0013] In one embodiment, the adjusting the multiple image feature layers in sequence according to the object description feature and the overall description feature to obtain the multiple adjusted image feature layers includes:

[0014] Adjusting the feature vectors corresponding to the plurality of image feature layers according to the object description features to obtain a plurality of initial image feature layers;

[0015] The overall feature vectors in the multiple initial image feature layers are adjusted using the overall description feature to obtain multiple adjusted image feature layers.

[0016] In one embodiment, the step of adjusting the feature vectors corresponding to the plurality of image feature layers according to the object description features to obtain a plurality of initial image feature layers includes:

[0017] Determine a target feature layer in the semantic feature layer that matches the object description feature;

[0018] Adjusting the feature vector of the target feature layer according to the object description feature to obtain an adjusted target feature layer;

[0019] The multiple image feature layers are adjusted according to the adjusted target feature layer to obtain multiple initial image feature layers.

[0020] In one embodiment, the adjusting the multiple image feature layers according to the adjusted target feature layer to obtain multiple initial image feature layers includes:

[0021] Adjusting the relationship feature layer according to the adjusted target feature layer to obtain an adjusted relationship feature layer;

[0022] The semantic feature layer is adjusted based on the adjusted relationship feature layer to obtain multiple initial image feature layers.

[0023] In one embodiment, the feature extraction of the target description text to obtain overall description features and object description features includes:

[0024] Inputting the target description text into a preset text analysis model to obtain initial description features;

[0025] According to the feature type corresponding to the initial description feature, the initial description feature is divided to obtain the overall description feature and the object description feature.

[0026] In a second aspect, the present application further provides an image generating device, comprising:

[0027] An acquisition module is used to acquire an initial image and target description text;

[0028] An extraction module is used to perform feature extraction on the target description text to obtain overall description features and object description features; perform feature extraction on the initial image to obtain multiple image feature layers, the multiple image feature layers include a semantic feature layer and a relationship feature layer, the semantic feature layer includes at least one of an object feature layer, an attribute feature layer, a relationship feature layer, and an environment feature layer;

[0029] An adjustment module, used for adjusting the multiple image feature layers in sequence according to the object description feature and the overall description feature to obtain multiple adjusted image feature layers;

[0030] A fusion module is used to fuse the adjusted multiple image feature layers to generate an adjusted image, and the adjusted image is used to generate a training image set, and the training image set is used to train an image recognition model.

[0031] In one embodiment, the extraction module is further used to:

[0032] Processing the initial image using a preset graph convolutional neural network to obtain image relationship features corresponding to the initial image, and generating a relationship feature layer according to the image relationship features;

[0033] The initial image is transformed using a preset visual transformation network to obtain image semantic features corresponding to the initial image, and a semantic feature layer is generated according to the image semantic features.

[0034] In one embodiment, the adjustment module is further used for:

[0035] Adjusting the feature vectors corresponding to the plurality of image feature layers according to the object description features to obtain a plurality of initial image feature layers;

[0036] The overall feature vectors in the multiple initial image feature layers are adjusted using the overall description feature to obtain multiple adjusted image feature layers.

[0037] In one embodiment, the adjustment module is further used for:

[0038] Determine a target feature layer in the semantic feature layer that matches the object description feature;

[0039] Adjusting the feature vector of the target feature layer according to the object description feature to obtain an adjusted target feature layer;

[0040] The multiple image feature layers are adjusted according to the adjusted target feature layer to obtain multiple initial image feature layers.

[0041] In one embodiment, the adjustment module is further used for:

[0042] Adjusting the relationship feature layer according to the adjusted target feature layer to obtain an adjusted relationship feature layer;

[0043] The semantic feature layer is adjusted based on the adjusted relationship feature layer to obtain multiple initial image feature layers.

[0044] In one embodiment, the extraction module is further used to:

[0045] Inputting the target description text into a preset text analysis model to obtain initial description features;

[0046] According to the feature type corresponding to the initial description feature, the initial description feature is divided to obtain the overall description feature and the object description feature.

[0047] In a third aspect, an embodiment of the present disclosure further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0048] In a fourth aspect, the embodiments of the present disclosure further provide a computer-readable storage medium, wherein a computer program is stored thereon, and when the computer program is executed by a processor, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0049] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0050] The above-mentioned image generation method, device, computer equipment, computer-readable storage medium and computer program product, when generating an image, obtain an initial image and a target description text, and perform feature extraction respectively to obtain the overall description features and object description features corresponding to the target description text and multiple image feature layers corresponding to the initial image, and use the object description features and the overall description features to adjust the multiple image feature layers in turn to obtain the adjusted multiple image feature layers, and fuse them to generate the adjusted image, which can be used to generate a training image set, and the training image set can be used to train the image recognition model, effectively improving the diversity of generating the training image set, ensuring the effect of model training, and being applicable to more application scenarios. Through the initial image and the target description text, the generated image is automatically adjusted, without the need for manual image acquisition of multiple scenes, reducing the manual workload and improving the efficiency of obtaining the training image set; and by processing the target scanned text through feature extraction, and processing the features of the initial image according to the extracted features, the image can be intelligently adjusted and enhanced according to the user's semantic description, thereby realizing the generation and adjustment of images of different scenes, so that the generated image has more visual authenticity and semantic consistency. During the adjustment process, multiple image feature layers of the initial image are adjusted and then fused, which improves the reliability and effectiveness of the adjusted image, avoids the problem of poor model training effect caused by image distortion, and is suitable for more application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 is a schematic flow chart of an image generating method in one embodiment;

[0053] Figure 2 is a schematic flow chart of an image generating method in another embodiment;

[0054] Figure 3 is a schematic flow chart of an image generating method in another embodiment;

[0055] Figure 4 is a schematic flow chart of an image generating method in another embodiment;

[0056] Figure 5 is a structural block diagram of an image generating device in another embodiment;

[0057] Figure 6FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] In one embodiment, Figure 1 As shown, an image generation method is provided. This embodiment takes the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0060] Step S110, obtaining an initial image and target description text;

[0061] Exemplarily, an initial image and a target description text are obtained. In one example, the initial image may be an original image used for image expansion and enhancement, and the target description text may include a task description input by a user. When the user inputs the task description, the target description text may be obtained after conversion through voice, text, etc. For example, the target description text may be "change the scene from day to night and add a white sports car driving on the road", and the initial image may be a daytime scene of a city street.

[0062] Step S120, extracting features from the target description text to obtain overall description features and object description features; extracting features from the initial image to obtain multiple image feature layers, the multiple image feature layers including a semantic feature layer and a relationship feature layer, the semantic feature layer including at least one of an object feature layer, an attribute feature layer, a relationship feature layer, and an environment feature layer;

[0063] Exemplarily, feature extraction is performed on the target description text to obtain overall description features and object description features. In one example, semantic extraction can be performed on the target description text by a semantic parser to obtain overall description features and object description features. Among them, the overall description features can be used to characterize the features used to describe the overall change in the target description text, such as the semantic features corresponding to "day turns to night"; the object description features can be used to characterize the features used to describe local changes in the target description text, such as the semantic features corresponding to "add a white sports car driving on the road", which can be determined according to the actual application scenario.

[0064] The initial image is subjected to feature extraction to obtain multiple image feature layers. In one example, different feature extraction methods may be used on the initial image to obtain features of different dimensions, such as a semantic feature layer corresponding to a semantic dimension and a relational feature layer corresponding to a relational dimension.

[0065] In one possible implementation, feature decomposer can be used to extract features in semantic dimension, and pre-trained visual converter is used as the backbone network. A multi-head attention mechanism is designed so that the model can focus on different semantic features and output a semantic feature layer. The semantic feature layer may include but is not limited to at least one of object feature layer, attribute feature layer, relationship feature layer, and environment feature layer.

[0066] In one possible implementation, a graph convolutional neural network can be used to extract features of the relational dimension of the initial image, and a scene graph can be constructed using the graph convolutional neural network to represent the relationship between elements in the image, and a relational feature layer can be output. The relational feature layer may include but is not limited to scene type, the relationship between main objects, time, weather, lighting, etc., which can be determined based on the actual application scenario.

[0067] Step S130, adjusting the plurality of image feature layers in sequence according to the object description feature and the overall description feature to obtain a plurality of adjusted image feature layers;

[0068] Exemplarily, multiple image feature layers are adjusted in sequence according to the object description features and the overall description features to obtain adjusted image feature layers.

[0069] In one example, the image feature layer may be adjusted first according to the object description feature, and then adjusted according to the overall description feature. The adjustment process of the overall description feature may be implemented through a style transfer network.

[0070] In one example, when adjusting the image feature layer, one or more image feature layers in the image feature layer may be adjusted, which may be determined based on features involved in the object description feature and the overall description feature.

[0071] Optionally, the image features may be adjusted by adjusting the feature vectors in the feature layer.

[0072] Step S140, fusing the adjusted multiple image feature layers to generate an adjusted image, wherein the adjusted image is used to generate a training image set, and the training image set is used to train an image recognition model.

[0073] Exemplarily, after obtaining multiple image feature layers, the adjusted multiple image feature layers can be fused using a preset feature fusion method to generate an adjusted image. In one example, a generative adversarial network can be used as an infrastructure to design a feature fusion module, and the feature fusion module can be used to fuse the adjusted multiple image feature layers to generate an adjusted image.

[0074] In one example, the adjusted image is used to generate a training image set. For example, different adjusted images can be generated according to different target description texts to form a training image set. The training image set can be used to train an image recognition model. Since the training image set is determined according to the target description text, a variety of training image sets can be generated based on semantic guidance to ensure the training effect of the image recognition model.

[0075] In the disclosed embodiment, when generating an image, an initial image and a target description text are obtained, and feature extraction is performed respectively to obtain overall description features and object description features corresponding to the target description text and multiple image feature layers corresponding to the initial image. The multiple image feature layers are adjusted in turn using the object description features and overall description features to obtain multiple adjusted image feature layers, which are fused to generate an adjusted image. The adjusted image can be used to generate a training image set, and the training image set can be used to train an image recognition model, which effectively improves the diversity of generating a training image set, ensures the effect of model training, and is applicable to more application scenarios. The generated image is automatically adjusted through the initial image and the target description text, without the need for manual image acquisition of multiple scenes, which reduces manual workload and improves the efficiency of acquiring the training image set; and the target scanned text is processed through feature extraction, and the features of the initial image are processed according to the extracted features, so that the image can be intelligently adjusted and enhanced according to the user's semantic description, thereby realizing the generation and adjustment of images of different scenes, so that the generated image has more visual authenticity and semantic consistency. During the adjustment process, multiple image feature layers of the initial image are adjusted and then fused, which improves the reliability and effectiveness of the adjusted image, avoids the problem of poor model training effect caused by image distortion, and is suitable for more application scenarios.

[0076] In one embodiment, the extracting features from the initial image to obtain multiple image feature layers includes:

[0077] Processing the initial image using a preset graph convolutional neural network to obtain image relationship features corresponding to the initial image, and generating a relationship feature layer according to the image relationship features;

[0078] The initial image is transformed using a preset visual transformation network to obtain image semantic features corresponding to the initial image, and a semantic feature layer is generated according to the image semantic features.

[0079] Exemplarily, the initial image is processed using a preset graph convolutional neural network and a preset visual transformation network, respectively, to obtain image feature layers of different dimensions.

[0080] A preset graph convolutional neural network is used to construct a scene graph to represent the relationship between the elements in the image, output image relationship features, including but not limited to scene type, relationship between main objects, time, weather, lighting, etc., and generate a relationship feature layer.

[0081] The initial image is transformed using a preset visual transformation network to obtain image semantic features and generate a semantic feature layer. In one example, the preset visual transformation network can be used as a backbone network, and a multi-head attention mechanism can be set up so that the model can focus on different semantic features and generate multiple semantic feature layers.

[0082] In the disclosed embodiment, when extracting features from an initial image, a graph convolutional neural network is used to process the initial image to generate a relational feature layer, and a preset visual transformation network is used to process the initial image to generate a semantic feature layer, thereby obtaining a relational feature layer for characterizing the association relationship between elements in the image and a semantic feature layer for characterizing different semantic features in the image. When adjusting the image, both the semantic features of the image and the association relationship between elements in the image can be taken into account, thereby further ensuring the authenticity and reliability of the adjusted image, improving the accuracy of the training image set, and ensuring the model training effect.

[0083] In one embodiment, Figure 2 As shown, the multiple image feature layers are adjusted in sequence according to the object description feature and the overall description feature to obtain the adjusted multiple image feature layers, including:

[0084] Step S131, adjusting the feature vectors corresponding to the multiple image feature layers according to the object description features to obtain multiple initial image feature layers;

[0085] Step S132: using the overall description feature to adjust the overall feature vectors in the multiple initial image feature layers to obtain multiple adjusted image feature layers.

[0086] Exemplarily, adjustments are first made based on the object description features, and then based on the overall description features.

[0087] In a possible implementation, feature vectors corresponding to multiple image feature layers are adjusted according to the object description feature to obtain multiple initial image feature layers, wherein the feature vectors corresponding to the object description feature may be determined based on the correspondence between the object description feature and the features in the image feature layer. In an example, the feature vector corresponding to the object description feature may be a feature vector corresponding to the image feature layer associated with the object corresponding to the object description feature.

[0088] After adjusting using the object description feature, the overall feature vector in the initial image feature layer is adjusted according to the overall description feature to obtain multiple adjusted image feature layers. In one example, a style transfer network can be used to perform global style transformation, and the overall feature vector is adjusted to obtain multiple adjusted image feature layers. In one example, after adjusting using the object description feature, it can be determined whether to continue adjusting according to the overall description feature. If necessary, the overall feature vector in the initial image feature layer is adjusted using the overall description feature.

[0089] In the disclosed embodiment, when adjusting the image feature layer, the object description feature is used to adjust the feature vector corresponding to the part, and then the overall description feature is used to adjust the overall feature vector. This allows the adjustment to be performed in the order of first the local and then the overall according to different description features, thereby ensuring that the adjusted multiple image feature layers conform to the target description text, and making the generated adjusted image more in line with actual needs.

[0090] In one embodiment, the step of adjusting the feature vectors corresponding to the plurality of image feature layers according to the object description features to obtain a plurality of initial image feature layers includes:

[0091] Determine a target feature layer in the semantic feature layer that matches the object description feature;

[0092] Adjusting the feature vector of the target feature layer according to the object description feature to obtain an adjusted target feature layer;

[0093] The multiple image feature layers are adjusted according to the adjusted target feature layer to obtain multiple initial image feature layers.

[0094] Exemplarily, according to the object description feature, a target feature layer in the semantic feature layer that matches the object description feature is determined. According to the object description feature, a feature vector of the target feature layer is first adjusted to obtain an adjusted target feature layer.

[0095] Since the image feature layers of an image are interrelated, when the feature vector of the target feature layer changes, other image feature layers need to be adjusted accordingly. In a possible implementation, multiple image feature layers are adjusted according to the adjusted target feature layer to obtain multiple initial image feature layers. In one example, multiple image feature layers can be adjusted according to the adjusted target feature layer based on the correlation between the image feature layers.

[0096] In the embodiment of the present disclosure, when making adjustments, the feature vector of the matched target feature layer is first adjusted according to the object description features, and then all the image feature layers are adjusted according to the adjusted target feature layer to obtain multiple initial image feature layers, so that the multiple initial image feature layers can all meet the object description features. At the same time, the accuracy of the target feature layer adjustment is guaranteed, thereby effectively improving the semantic consistency of the generated adjusted image.

[0097] In one embodiment, the adjusting the multiple image feature layers according to the adjusted target feature layer to obtain multiple initial image feature layers includes:

[0098] Adjusting the relationship feature layer according to the adjusted target feature layer to obtain an adjusted relationship feature layer;

[0099] The semantic feature layer is adjusted based on the adjusted relationship feature layer to obtain multiple initial image feature layers.

[0100] Exemplarily, after adjusting the target feature layer, the relationship feature layer is adjusted, and according to the association relationship between the elements and the adjusted target feature layer, the feature vector corresponding to the relationship feature layer is adjusted to obtain the adjusted relationship feature layer.

[0101] The semantic feature layer is adjusted based on the adjusted relationship feature layer to obtain multiple initial image feature layers. In one example, the association relationship between the elements can be re-determined based on the adjusted relationship feature layer, and the semantic feature layer can be adjusted. In one example, the other semantic feature layers can also be adjusted in combination with the adjusted relationship feature layer and the target feature layer.

[0102] In the disclosed embodiment, the relationship feature layer is first adjusted according to the adjusted target feature layer to obtain the adjusted relationship feature layer, and the semantic feature layer is adjusted based on the adjusted relationship feature layer to obtain multiple initial image feature layers, so that the association relationship between the elements in the adjusted image is more consistent with the actual scene and the scene corresponding to the target description text, thereby ensuring the accuracy and reliability of the adjusted image.

[0103] In one embodiment, Figure 3As shown, the feature extraction of the target description text to obtain overall description features and object description features includes:

[0104] Step S121, inputting the target description text into a preset text analysis model to obtain initial description features;

[0105] Step S122: dividing the initial description features according to the feature types corresponding to the initial description features to obtain overall description features and object description features.

[0106] Exemplarily, the target description text may be feature extracted by a preset text analysis model. In one example, when adjusting an image, since the target description text may contain descriptions of different feature types, different feature types may be divided and adjusted separately to ensure the accuracy of the adjusted image. Therefore, the initial description features obtained by the preset text analysis model are feature classified.

[0107] Optionally, the initial description features are divided according to the feature types corresponding to them to obtain overall description features and object description features. In one example, the feature types may include but are not limited to overall features, local features, etc. The initial description features corresponding to the overall features may be determined as overall description features, and the initial description features corresponding to the local features may be determined as object description features.

[0108] In the disclosed embodiment, when extracting features from a target text, the extracted initial description features are divided to obtain overall description features and object description features, so that in the subsequent feature adjustment process, the overall description features and the object description features can be processed separately, thereby ensuring the accuracy and reliability of the adjusted image while effectively reducing the data processing workload, improving image generation efficiency, and being suitable for more application scenarios.

[0109] Figure 4 is a flow chart of an image generation method according to an exemplary embodiment, referring to Figure 4As shown, step 1, receiving the target description text and the initial image input by the user; step 2, using the semantic parser to convert the target description text into the initial description features, including the overall description features and the object description features; step 3, using the feature decomposer to decompose the initial image into multiple semantic feature layers; step 4, analyzing the overall semantic environment of the image through the context analyzer to obtain the relationship feature layer; step 5, adjusting the semantic feature layer and the relationship feature layer according to the object description feature; step 6, based on the overall description feature, using the style transfer network to perform global style conversion; step 7, finally, recombining the modified semantic feature layer and the relationship feature layer through the image synthesizer to generate the adjusted image. Specifically, when using the semantic parser to convert the target description text into the object description features and the overall description features, a pre-trained large-scale language model (such as BERT or GPT) can be used as the basis. When using the feature decomposer to decompose the initial image into multiple semantic feature layers, a pre-trained vision transformer (Vision Transformer) can be used as the backbone network, and a multi-head attention mechanism can be designed so that the model can focus on different semantic features and output semantic feature layers, including but not limited to: object feature layer, attribute feature layer, relationship feature layer, and environment feature layer. When the overall semantic environment of the image is analyzed by the context analyzer and the relational feature layer is obtained, a graph convolutional neural network (GCN) can be used to construct a scene graph to represent the relationship between the elements in the image and output a relational feature layer, including but not limited to: scene type, relationship between main objects, time, weather, lighting, etc. In one example, when adjusting the image feature layer according to the object description features, the weights of each feature layer can be calculated and the feature layer can be adjusted according to the weights. In one example, when performing style transfer, content consistency loss is introduced to ensure that the key object features do not change significantly after the style transfer. The corresponding adjustments to the overall description features include but are not limited to changes in overall lighting conditions, weather conditions, seasonal characteristics, etc. When fusing and generating images, a generative adversarial network (GAN) can be used as the infrastructure to design a feature fusion module to effectively integrate information from different feature layers.

[0110] The disclosed embodiment adopts semantic parsing technology to convert the user's natural language requirements into semantic features to achieve image enhancement and modification. The system can understand complex logical relationships and multi-step descriptions and support the processing of multiple scenes. A feature decomposition technology based on deep learning is proposed to decompose the image into multiple semantic feature layers, including dimensions such as objects, attributes, spatial relationships, and environments, so as to facilitate independent modification and enhancement of each feature layer. The context analyzer is used to understand the overall semantic environment in the image, analyze the relationship between scene types and objects, and obtain the relationship feature layer, which is used as a basis to provide background information support for subsequent operations such as object insertion and replacement. Through semantic parsing, the automatic conversion of natural language to operation instructions is realized, which enhances the intelligence and user experience of the system. Through multi-dimensional feature decomposition and local feature retention mechanism, the system can finely control the enhancement effect of each part of the image to ensure image quality. Through context analysis and style transfer, the system can better handle global and local changes in the image to ensure that the modified image is semantically coherent and visually natural.

[0111] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0112] Based on the same inventive concept, the embodiment of the present application also provides an image generation device for implementing the above-mentioned image generation method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image generation device embodiments provided below can refer to the limitations on the image generation method above, and will not be repeated here.

[0113] In an exemplary embodiment, Figure 5 As shown, an image generating device 500 is provided, comprising:

[0114] An acquisition module 510 is used to acquire an initial image and a target description text;

[0115] Extraction module 520, used to extract features from the target description text to obtain overall description features and object description features; extract features from the initial image to obtain multiple image feature layers, the multiple image feature layers include semantic feature layers and relationship feature layers, the semantic feature layers include at least one of object feature layers, attribute feature layers, relationship feature layers, and environment feature layers;

[0116] An adjustment module 530, configured to adjust the plurality of image feature layers in sequence according to the object description feature and the overall description feature to obtain a plurality of adjusted image feature layers;

[0117] The fusion module 540 is used to fuse the adjusted multiple image feature layers to generate an adjusted image, and the adjusted image is used to generate a training image set, and the training image set is used to train an image recognition model.

[0118] In one embodiment, the extraction module is further used to:

[0119] Processing the initial image using a preset graph convolutional neural network to obtain image relationship features corresponding to the initial image, and generating a relationship feature layer according to the image relationship features;

[0120] The initial image is transformed using a preset visual transformation network to obtain image semantic features corresponding to the initial image, and a semantic feature layer is generated according to the image semantic features.

[0121] In one embodiment, the adjustment module is further used for:

[0122] Adjusting the feature vectors corresponding to the plurality of image feature layers according to the object description features to obtain a plurality of initial image feature layers;

[0123] The overall feature vectors in the multiple initial image feature layers are adjusted using the overall description feature to obtain multiple adjusted image feature layers.

[0124] In one embodiment, the adjustment module is further used for:

[0125] Determine a target feature layer in the semantic feature layer that matches the object description feature;

[0126] Adjusting the feature vector of the target feature layer according to the object description feature to obtain an adjusted target feature layer;

[0127] The multiple image feature layers are adjusted according to the adjusted target feature layer to obtain multiple initial image feature layers.

[0128] In one embodiment, the adjustment module is further used for:

[0129] Adjusting the relationship feature layer according to the adjusted target feature layer to obtain an adjusted relationship feature layer;

[0130] The semantic feature layer is adjusted based on the adjusted relationship feature layer to obtain multiple initial image feature layers.

[0131] In one embodiment, the extraction module is further used to:

[0132] Inputting the target description text into a preset text analysis model to obtain initial description features;

[0133] According to the feature type corresponding to the initial description feature, the initial description feature is divided to obtain the overall description feature and the object description feature.

[0134] Each module in the above-mentioned image generation device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0135] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the data involved in the method described in this embodiment, such as the initial image. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image generation method is implemented.

[0136] Those skilled in the art will understand that Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0137] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0138] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0139] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0140] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0141] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0142] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0143] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image generation method, characterized in that: The method comprises: Get the initial image and target description text; Performing feature extraction on the target description text to obtain overall description features and object description features; performing feature extraction on the initial image to obtain multiple image feature layers, the multiple image feature layers include a semantic feature layer and a relationship feature layer, the semantic feature layer includes at least one of an object feature layer, an attribute feature layer, a relationship feature layer, and an environment feature layer; According to the object description feature and the overall description feature, the plurality of image feature layers are adjusted in sequence to obtain a plurality of adjusted image feature layers; The adjusted multiple image feature layers are fused to generate an adjusted image, and the adjusted image is used to generate a training image set, and the training image set is used to train an image recognition model.

2. The method according to claim 1, characterized in that The extracting features of the initial image to obtain multiple image feature layers includes: Processing the initial image using a preset graph convolutional neural network to obtain image relationship features corresponding to the initial image, and generating a relationship feature layer according to the image relationship features; The initial image is transformed using a preset visual transformation network to obtain image semantic features corresponding to the initial image, and a semantic feature layer is generated according to the image semantic features.

3. The method according to claim 1, characterized in that The step of sequentially adjusting the plurality of image feature layers according to the object description feature and the overall description feature to obtain the plurality of adjusted image feature layers comprises: Adjusting the feature vectors corresponding to the plurality of image feature layers according to the object description features to obtain a plurality of initial image feature layers; The overall feature vectors in the multiple initial image feature layers are adjusted using the overall description feature to obtain multiple adjusted image feature layers.

4. The method according to claim 3, characterized in that The step of adjusting the feature vectors corresponding to the plurality of image feature layers according to the object description features to obtain a plurality of initial image feature layers comprises: Determine a target feature layer in the semantic feature layer that matches the object description feature; Adjusting the feature vector of the target feature layer according to the object description feature to obtain an adjusted target feature layer; The multiple image feature layers are adjusted according to the adjusted target feature layer to obtain multiple initial image feature layers.

5. The method according to claim 4, characterized in that The step of adjusting the multiple image feature layers according to the adjusted target feature layer to obtain multiple initial image feature layers includes: Adjusting the relationship feature layer according to the adjusted target feature layer to obtain an adjusted relationship feature layer; The semantic feature layer is adjusted based on the adjusted relationship feature layer to obtain multiple initial image feature layers.

6. The method according to claim 1, characterized in that The feature extraction of the target description text to obtain overall description features and object description features includes: Inputting the target description text into a preset text analysis model to obtain initial description features; According to the feature type corresponding to the initial description feature, the initial description feature is divided to obtain the overall description feature and the object description feature.

7. An image generating device, characterized in that: The device comprises: An acquisition module is used to acquire an initial image and target description text; An extraction module is used to perform feature extraction on the target description text to obtain overall description features and object description features; perform feature extraction on the initial image to obtain multiple image feature layers, the multiple image feature layers include a semantic feature layer and a relationship feature layer, the semantic feature layer includes at least one of an object feature layer, an attribute feature layer, a relationship feature layer, and an environment feature layer; An adjustment module, used for adjusting the multiple image feature layers in sequence according to the object description feature and the overall description feature to obtain multiple adjusted image feature layers; A fusion module is used to fuse the adjusted multiple image feature layers to generate an adjusted image, and the adjusted image is used to generate a training image set, and the training image set is used to train an image recognition model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images