Image processing method, image generation model training method and device

Abstract

The present application discloses an image processing method, a training method and a device for an image generation model. The present application relates to the field of artificial intelligence. The method comprises: receiving an image generation request of a target object, wherein the image generation request at least includes the category information of the target object in the target image to be generated; using a random number generation function to generate noise data; inputting the category information and the noise data into a target image generation model for processing to obtain a target image, wherein the target image generation model at least includes multiple groups of multi-scale deformable convolutional networks, and the multiple groups of multi-scale deformable convolutional networks are used to determine the image features of the target image. The present application solves the technical problem that the image generation model in the related art has poor scalability for adaptive resolution generation, resulting in poor visual effects of the generated image.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to an image processing method, an image generation model training method and a device. Background Art

[0002] With the continuous development of technology, people can use artificial intelligence models for image generation. At present, the existing image generation model based on Transformer (a deep learning model based on the self-attention mechanism) has poor scalability for adaptive resolution generation, and the generated images have poor visual effects, making it difficult to perform arbitrary resolution image generation tasks. In addition, in high-resolution scenarios, since the attention block in the Transformer has a computational complexity of the square of the token, it leads to low computational efficiency, high inference latency, and low image generation efficiency.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, an image generation model training method and a device, so as to at least solve the technical problem that the image generation model in the related art has poor scalability for adaptive resolution generation, resulting in poor visual effect of the generated image.

[0005] According to one aspect of an embodiment of the present application, an image processing method is provided, comprising: receiving an image generation request of a target object, wherein the image generation request includes at least category information of the target object in a target image to be generated; generating noise data using a random number generation function; inputting the category information and the noise data into a target image generation model for processing to obtain a target image, wherein the target image generation model includes at least a plurality of groups of multi-scale deformable convolutional networks, and the plurality of groups of multi-scale deformable convolutional networks are used to determine image features of the target image.

[0006] Furthermore, the category information and noise data are input into the target image generation model for processing to obtain the target image, including: processing the category information through the target image generation model to obtain category features, processing the noise data through the target image generation model to obtain multiple sets of input features of multi-scale deformable convolutional networks; processing the input features through multiple sets of multi-scale deformable convolutional networks to obtain image features of the target image; injecting the category features into the image features of the target image through a preset multi-layer perceptron of the target image generation model to obtain hidden variables corresponding to the target image; decoding the hidden variables corresponding to the target image into the image RGB space through a preset decoder of the target image generation model to obtain the target image.

[0007] Furthermore, the input features are processed by multiple groups of multi-scale deformable convolutional networks to obtain image features of the target image, including: determining the scale initial values ​​and direction initial values ​​corresponding to the multi-scale deformable convolutional networks in the multiple groups of multi-scale deformable convolutional networks; determining the weight information of the sampling features corresponding to the deformable field and the input features according to the scale initial values ​​and direction initial values ​​corresponding to the multi-scale deformable convolutional networks through the multiple groups of multi-scale deformable convolutional networks; performing feature aggregation according to the weight information of the sampling features corresponding to the deformable field and the input features through the deformable convolution modules of the multiple groups of multi-scale deformable convolutional networks to obtain aggregated features; and projecting the aggregated features through the output prediction modules of the multiple groups of multi-scale deformable convolutional networks to obtain the image features of the target image.

[0008] Furthermore, determining the weight information of the sampling features corresponding to the deformable field and the input features through multiple groups of multi-scale deformable convolutional networks according to the scale initial values ​​and direction initial values ​​corresponding to the multi-scale deformable convolutional networks includes: obtaining the scale coefficient of the multi-scale deformable convolutional network according to the scale initial value prediction through multiple groups of multi-scale deformable convolutional networks' scale prediction modules; obtaining the direction of the multi-scale deformable convolutional network according to the direction initial value prediction through multiple groups of multi-scale deformable convolutional networks' direction; determining the deformable field according to the position information of the input features, the scale coefficient of the multi-scale deformable convolutional network, the direction initial value and the direction of the multi-scale deformable convolutional network through multiple groups of deformable core modules of the multi-scale deformable convolutional networks; and obtaining the weight information of the sampling features corresponding to the input features according to the weight coefficient matrix and the bias vector of the sampling features corresponding to the input features through multiple groups of dynamic weight prediction modules of the multi-scale deformable convolutional networks.

[0009] Furthermore, feature aggregation is performed according to the deformable field and weight information of sampling features corresponding to the input features by deformable convolution modules of multiple groups of multi-scale deformable convolution networks to obtain aggregated features, including: sampling input features according to the deformable field by deformable convolution modules of multiple groups of multi-scale deformable convolution networks to obtain sampling features corresponding to the input features; feature aggregation is performed according to the sampling features corresponding to the input features and the weight information of the sampling features corresponding to the input features to obtain aggregated features.

[0010] Furthermore, the category information is processed by the target image generation model to obtain category features, and the noise data is processed by the target image generation model to obtain multiple sets of input features of the multi-scale deformable convolutional network, including: processing the category information by the text processor of the target image generation model to obtain the semantic vector corresponding to the category information; processing the semantic vector and noise data corresponding to the category information by the preset encoder of the target image generation model to obtain noise latent variables; processing the noise latent variables by the block processing module of the target image generation model to obtain input features; processing the category information by the embedding module of the target image generation model to obtain category features.

[0011] According to another aspect of an embodiment of the present application, a training method for an image generation model is also provided, including: constructing a training sample set based on images in a preset image database; encoding the training samples in the training sample set to obtain hidden variables corresponding to the training samples; constructing noisy hidden variables based on the hidden variables corresponding to the training samples and time and noise randomly sampled from a preset time interval; training the initial image generation model based on the noisy hidden variables and preset training objectives until the loss function of the initial image generation model converges, thereby obtaining any of the above target image generation models.

[0012] According to another aspect of an embodiment of the present application, an image processing method is also provided, including: obtaining an image generation request of a target object uploaded by a client, wherein the image generation request includes at least category information of the target object in the target image to be generated; generating noise data using a random number generation function in a cloud server; inputting the category information and the noise data into a target image generation model for processing to obtain a target image, wherein the target image generation model includes at least multiple groups of multi-scale deformable convolutional networks, and the multiple groups of multi-scale deformable convolutional networks are used to determine the image features of the target image; and feeding back the target image to the client.

[0013] According to another aspect of an embodiment of the present application, an image processing device is also provided, including: a receiving unit, used to receive an image generation request of a target object, wherein the image generation request includes at least category information of the target object in the target image to be generated; a first processing unit, used to generate noise data using a random number generation function; and a second processing unit, used to input the category information and the noise data into a target image generation model for processing to obtain a target image, wherein the target image generation model includes at least multiple groups of multi-scale deformable convolutional networks, and the multiple groups of multi-scale deformable convolutional networks are used to determine the image features of the target image.

[0014] Furthermore, the second processing unit includes: a first processing subunit, used to process the category information through the target image generation model to obtain category features, and process the noise data through the target image generation model to obtain multiple sets of input features of multi-scale deformable convolutional networks; a second processing subunit, used to process the input features through multiple sets of multi-scale deformable convolutional networks to obtain image features of the target image; a third processing subunit, used to inject the category features into the image features of the target image through a preset multi-layer perceptron of the target image generation model to obtain hidden variables corresponding to the target image; a fourth processing subunit, used to decode the hidden variables corresponding to the target image into the image RGB space through a preset decoder of the target image generation model to obtain the target image.

[0015] Furthermore, the second processing subunit includes: a first determination module, used to determine the scale initial value and direction initial value corresponding to the multi-scale deformable convolution network in the multiple groups of multi-scale deformable convolution networks; a second determination module, used to determine the weight information of the sampling features corresponding to the deformable field and the input features according to the scale initial value and direction initial value corresponding to the multi-scale deformable convolution network through the multiple groups of multi-scale deformable convolution networks; a first processing module, used to perform feature aggregation according to the weight information of the sampling features corresponding to the deformable field and the input features through the deformable convolution modules of the multiple groups of multi-scale deformable convolution networks to obtain aggregated features; a second processing module, used to project the aggregated features through the output prediction modules of the multiple groups of multi-scale deformable convolution networks to obtain image features of the target image.

[0016] Furthermore, the second determination module includes: a first determination submodule, used to obtain the scale coefficient of the multi-scale deformable convolutional network according to the scale initial value prediction through multiple groups of multi-scale deformable convolutional networks' scale prediction modules; a second determination submodule, used to obtain the direction of the multi-scale deformable convolutional network according to the direction initial value prediction through multiple groups of multi-scale deformable convolutional networks' direction prediction modules; a third determination submodule, used to determine the deformable field according to the position information of the input feature, the scale coefficient of the multi-scale deformable convolutional network, the direction initial value and the direction of the multi-scale deformable convolutional network through multiple groups of deformable core modules of the multi-scale deformable convolutional network; a fourth determination submodule, used to obtain the weight information of the sampling feature corresponding to the input feature according to the weight coefficient matrix and the bias vector of the sampling feature corresponding to the input feature through multiple groups of dynamic weight prediction modules of the multi-scale deformable convolutional network.

[0017] Furthermore, the first processing module includes: a first processing sub-module, used to sample the input features according to the deformable field through the deformable convolution modules of multiple groups of multi-scale deformable convolution networks, and obtain sampling features corresponding to the input features; a second processing sub-module, used to perform feature aggregation based on the sampling features corresponding to the input features and the weight information of the sampling features corresponding to the input features, and obtain the aggregated features.

[0018] Furthermore, the first processing subunit includes: a third processing module, which is used to process the category information through the text processor of the target image generation model to obtain a semantic vector corresponding to the category information; a fourth processing module, which is used to process the semantic vector and noise data corresponding to the category information through a preset encoder of the target image generation model to obtain a noise latent variable; a fifth processing module, which is used to process the noise latent variable through the block processing module of the target image generation model to obtain an input feature; and a sixth processing module, which is used to process the category information through the embedding module of the target image generation model to obtain a category feature.

[0019] According to another aspect of an embodiment of the present application, a training device for an image generation model is also provided, including: a first construction unit, used to construct a training sample set based on images in a preset image database; a third processing unit, used to encode the training samples in the training sample set to obtain latent variables corresponding to the training samples; a second construction unit, used to construct noisy latent variables based on the latent variables corresponding to the training samples and time and noise randomly sampled from a preset time interval; a fourth processing unit, used to train the initial image generation model based on the noisy latent variables and the preset training objectives, until the loss function of the initial image generation model converges, thereby obtaining a target image generation model of any of the above items.

[0020] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, comprising: a memory storing an executable program; and a processor for running the program, wherein any one of the above-mentioned image processing methods is executed when the program is running.

[0021] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a program, wherein when the program is executed, the device where the storage medium is located is controlled to execute any one of the above-mentioned image processing methods.

[0022] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program or instructions, and when the computer program or instructions are executed by a processor, any one of the above-mentioned image processing methods is implemented.

[0023] In an embodiment of the present application, an image generation request of a target object is received, wherein the image generation request includes at least category information of the target object in the target image to be generated; noise data is generated by using a random number generation function; the category information and the noise data are input into a target image generation model for processing to obtain a target image, wherein the target image generation model includes at least multiple groups of multi-scale deformable convolutional networks, and the multiple groups of multi-scale deformable convolutional networks are used to determine the image features of the target image. Since the multiple groups of multi-scale deformable convolutional networks have linear computational complexity and higher computational efficiency compared with the attention blocks in the Transformer, and have stronger dynamic performance compared with general convolutional networks, the multiple groups of multi-scale deformable convolutional networks based on the target image generation model can generate images with arbitrary resolutions, thereby achieving the purpose of improving the quality of the model's raw images, thereby achieving the technical effect of improving the visual effect of the model-generated images, and thus solving the technical problem that the image generation model in the related art has poor scalability for adaptive resolution generation, resulting in poor visual effects of the generated images. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 is a schematic diagram of a computer terminal provided according to Embodiment 1 of the present application;

[0026] Figure 2 is a flowchart of an image processing method provided according to Embodiment 1 of the present application;

[0027] Figure 3 is a schematic diagram of the structure of an optional target image generation model provided according to Embodiment 1 of the present application;

[0028] Figure 4 It is a schematic diagram of the structure of the optional inter-group multi-scale deformable convolution provided according to the first embodiment of the present application;

[0029] Figure 5 is a flowchart of a method for training an image generation model provided in Example 2 of the present application;

[0030] Figure 6 is a flowchart of an image processing method provided according to Embodiment 3 of the present application;

[0031] Figure 7 is a schematic diagram of an image processing device provided according to Embodiment 4 of the present application;

[0032] Figure 8is a schematic diagram of a training device for an image generation model provided in Embodiment 5 of the present application;

[0033] Fig. 9 This is a structural block diagram of an electronic device provided according to Embodiment 6 of the present application. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0037] Example 1

[0038] According to an embodiment of the present application, an image processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image processing method. Figure 1 As shown, the computer terminal (or mobile device) 10 may include a processor set 102 (the processor set 102 may include but is not limited to a processing device such as a microprocessor (Microcontroller Unit, referred to as MCU) or a programmable logic device (Field Programmable Gate Array, referred to as FPGA), and the processor set 102 may include a processor set, Figure 1 102a, 102b, ..., 102n are used to illustrate), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0040] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuits may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device).

[0041] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the image processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned image processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0042] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0043] The display may be a touch screen type liquid crystal display, which may enable a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0044] With the continuous development of technology, people can use artificial intelligence models for image generation. At present, the existing image generation model based on Transformer (a deep learning model based on the self-attention mechanism) has poor scalability for adaptive resolution generation, and the generated images have poor visual effects, making it difficult to perform arbitrary resolution image generation tasks. In addition, in high-resolution scenarios, since the attention block in the Transformer has a computational complexity of the square of the token, it leads to low computational efficiency, high inference latency, and low image generation efficiency.

[0045] In the above technical background, the present application provides Figure 2 The image processing method shown. Figure 2 : is a flowchart of an image processing method provided according to Embodiment 1 of the present application. The method comprises:

[0046] Step S201 : receiving an image generation request of a target object, wherein the image generation request at least includes category information of the target object in the target image to be generated.

[0047] Step S202: Generate noise data using a random number generation function.

[0048] Optionally, an image generation request of a target object (such as a user) is received through the image processing system. For example, the user wants to generate an image of an animal (such as a cat or a dog) and initiates a corresponding image generation request. The category information of the target object may be information such as a cat or a dog.

[0049] Optionally, after receiving an image generation request of the target object, a random number generation function is used to generate noise data, wherein the noise data is pure noise data, for example, a random number generation function (random function) is used to generate initial pure noise data by random sampling.

[0050] Step S203, inputting the category information and the noise data into a target image generation model for processing to obtain a target image, wherein the target image generation model includes at least a plurality of groups of multi-scale deformable convolutional networks, and the plurality of groups of multi-scale deformable convolutional networks are used to determine the image features of the target image.

[0051] Optionally, multiple groups of multi-scale deformable convolution networks can also be called group-wise multi-scale deformable convolution networks (Group-wise Multi-Scale Deformable Convolution, referred to as Group-wise MSDCN). In this solution, an image generation model (i.e., a target image generation model) with Group-wise MSDCN as the basic network module is proposed. The category information and pure noise data are input into the model for processing to obtain the target image desired by the user. For example, the pure noise data eps is randomly sampled, and the pure noise data eps is integrated from time 0 to 1 through the sampler and predicted velocity field of the target image generation model to obtain the latent variable (latent) corresponding to the image to be generated, and then the decoder in the variational autoencoder is used to decode the latent to the image RGB space to obtain the target image. The following is a detailed introduction in conjunction with a specific embodiment.

[0052] In this scheme, since the multiple groups of multi-scale deformable convolutional networks have linear computational complexity and higher computational efficiency compared with the attention blocks in the Transformer, and have stronger dynamic performance compared with general convolutional networks, the multiple groups of multi-scale deformable convolutional networks based on the target image generation model can generate images with arbitrary resolutions, thereby achieving the purpose of improving the raw image quality of the model, thereby achieving the technical effect of improving the visual effect of the image generated by the model, and further solving the technical problem that the image generation model in the related technology has poor scalability for adaptive resolution generation, resulting in poor visual effect of the generated image.

[0053] In order to obtain a target image, in the image processing method provided in the first embodiment of the present application, category information and noise data are input into a target image generation model for processing, and obtaining the target image includes: processing the category information through the target image generation model to obtain category features, processing the noise data through the target image generation model to obtain multiple groups of input features of multi-scale deformable convolutional networks; processing the input features through multiple groups of multi-scale deformable convolutional networks to obtain image features of the target image; injecting the category features into the image features of the target image through a preset multi-layer perceptron of the target image generation model to obtain hidden variables corresponding to the target image; decoding the hidden variables corresponding to the target image into the image RGB space through a preset decoder of the target image generation model to obtain the target image.

[0054] Optionally, the category information is first processed by the target image generation model to obtain category features, and the pure noise data is processed to obtain the input features of the Group-wise MSDCN.

[0055] In order to obtain category features and input features, in the image processing method provided in Example 1 of the present application, category information is processed by a target image generation model to obtain category features, and noise data is processed by the target image generation model to obtain multiple groups of input features of a multi-scale deformable convolutional network, including: processing the category information by a text processor of the target image generation model to obtain a semantic vector corresponding to the category information; processing the semantic vector and noise data corresponding to the category information by a preset encoder of the target image generation model to obtain noise latent variables; processing the noise latent variables by a block processing module of the target image generation model to obtain input features; processing the category information by an embedding module of the target image generation model to obtain category features.

[0056] Optionally, the category information is first processed by the text processor of the target image generation model to obtain the semantic vector corresponding to the category information, and then the semantic vector corresponding to the category information and the pure noise data are processed by the preset encoder of the target image generation model (such as the encoder in the variational autoencoder) to obtain the noise latent variable (NoisedLatent), and then the Noised Latent is processed by the patch processing module (Patchify module) of the target image generation model to obtain the input features (Input Features), and the category information is processed by the embedding module (Embed module) of the target image generation model to obtain the category features. Among them, the category information is represented by time t and label y, and the category features are represented by condition Condition.

[0057] In an optional embodiment, it is possible to use Figure 3 The schematic diagram shown in FIG. 1 realizes the generation of images with arbitrary resolution. Figure 3 is a schematic diagram of the structure of an optional target image generation model provided in Example 1 of the present application, such as Figure 3As shown in the figure, the target image generation model includes N (N is a positive integer) network modules. The network modules are constructed alternately using Group-wise MSDCN and multi-layer perceptrons (such as SwiGLU), and the category information of the image is injected using Adaptive InstanceNormalization (AdaIN). The noise latent variables are processed by the block processing module and then enter the network module as input features. The time and label are processed by the embedding module and then enter the network module as conditions. The linear layer is used to predict the dynamic weights of scale offset and gating based on the conditions. The input features are processed by Group-wise MSDCN to obtain the image features of the target image.

[0058] It should be noted that the noise latent variables are processed by the block processing module of the target image generation model to obtain the input features, and the category information is processed by the embedding module of the target image generation model to obtain the category features, thereby realizing accurate determination of the input of the network module.

[0059] In order to obtain the image features of the target image, in the image processing method provided in the first embodiment of the present application, the input features are processed by multiple groups of multi-scale deformable convolutional networks to obtain the image features of the target image, including: determining the scale initial value and direction initial value corresponding to the multi-scale deformable convolutional network in the multiple groups of multi-scale deformable convolutional networks; determining the weight information of the sampling features corresponding to the deformable field and the input features according to the scale initial value and direction initial value corresponding to the multi-scale deformable convolutional network through the multiple groups of multi-scale deformable convolutional networks; performing feature aggregation according to the weight information of the sampling features corresponding to the deformable field and the input features through the deformable convolution modules of the multiple groups of multi-scale deformable convolutional networks to obtain the aggregated features; and projecting the aggregated features through the output prediction modules of the multiple groups of multi-scale deformable convolutional networks to obtain the image features of the target image.

[0060] Figure 4 is a schematic diagram of the structure of the optional inter-group multi-scale deformable convolution provided in the first embodiment of the present application, such as Figure 4 As shown in the figure, Group-wise MSDCN includes a multi-scale prior and direction prior module, a deformable kernel module, a dynamic weight prediction module, a scale prediction module, a direction prediction module, a numerical prediction module, a deformable convolution module and an output prediction module.

[0061] Optionally, the scale initialization and direction initialization corresponding to each group of multi-scale deformable convolutional networks are first determined by the multi-scale prior and direction prior modules. For example, given the number of groups of multi-scale deformable convolutional networks (MSDCN) ( is a positive integer), input feature x, first determine the initial value of the direction (i.e. direction prior) , k means that each feature in each group predicts k points. Different scale initial values ​​(i.e. scale prior) are initialized for each group of MSDCN By decoupling the scale, different groups can initialize different , which can achieve multi-scale between groups. The initialization is as follows:

[0062]

[0063] in, Indicates the total number of groups.

[0064] Optionally, through Group-wise MSDCN according to MSDCN corresponding and The weight information (ie, dynamic weight) of the sampled features corresponding to the deformable field p and the input features can be determined.

[0065] In order to accurately determine the dynamic weights of the deformable field and the sampling features, in the image processing method provided in the first embodiment of the present application, the weight information of the sampling features corresponding to the deformable field and the input features is determined according to the scale initial value and the direction initial value corresponding to the multi-scale deformable convolutional network through multiple groups of multi-scale deformable convolutional networks, including: obtaining the scale coefficient of the multi-scale deformable convolutional network according to the scale initial value prediction through multiple groups of multi-scale deformable convolutional networks; obtaining the direction of the multi-scale deformable convolutional network according to the direction initial value prediction through multiple groups of multi-scale deformable convolutional networks; determining the deformable field according to the position information of the input feature, the scale coefficient of the multi-scale deformable convolutional network, the direction initial value and the direction of the multi-scale deformable convolutional network through multiple groups of deformable core modules of the multi-scale deformable convolutional network; and obtaining the weight information of the sampling features corresponding to the input features according to the weight coefficient matrix and the bias vector of the sampling features corresponding to the input features through multiple groups of dynamic weight prediction modules of the multi-scale deformable convolutional networks.

[0066] Optionally, the scale prediction module is used according to the scale prior corresponding to each group of MSDCN Predict the scale coefficients of each group of MSDCN , the formula is as follows:

[0067]

[0068] Among them, S represents the scale, represents the largest scale among the multiple scales between groups, is an activation function, The matrix is ​​the transposed matrix of the coefficient matrix of the input features x.

[0069] Optionally, the direction prediction module is used according to the direction prior corresponding to MSDCN. Predict the direction of each group of MSDCN , and then the deformable kernel module is used according to the position information of the input feature x , scale coefficient , Direction Prior And direction Construct a deformable field p, the formula is as follows:

[0070]

[0071] Optionally, the dynamic weight of the sampling feature corresponding to the input feature is predicted by the dynamic weight prediction module according to the weight coefficient matrix and bias vector of the sampling feature corresponding to the input feature. , the formula is as follows:

[0072]

[0073] in, The matrix is ​​the transposed matrix of the weight coefficient matrix of the sampled features, is the bias vector of the dynamic weight. Each feature predicts G groups of K dynamic weights, totaling G*K dynamic weights.

[0074] Optionally, the deformable convolution module is used to dynamically weight the sampled features corresponding to the deformable field p and the input features. Perform feature aggregation to obtain aggregated features.

[0075] In order to obtain aggregated features, in the image processing method provided in the first embodiment of the present application, feature aggregation is performed based on the deformable field and the weight information of the sampling features corresponding to the input features through multiple groups of deformable convolution modules of multi-scale deformable convolution networks, and the aggregated features are obtained, including: sampling the input features based on the deformable field through multiple groups of deformable convolution modules of multi-scale deformable convolution networks to obtain the sampling features corresponding to the input features; feature aggregation is performed based on the sampling features corresponding to the input features and the weight information of the sampling features corresponding to the input features to obtain the aggregated features.

[0076] Optionally, dynamic weights and deformable fields are used within the group to dynamically aggregate features, that is, the input feature x is sampled according to the deformable field p by the deformable convolution module to obtain the sampled features corresponding to the input feature x (expressed as ), feature aggregation is performed based on the sampling features corresponding to the input features and the dynamic weights of the sampling features corresponding to the input features, and the aggregated features can be obtained. , the formula is as follows:

[0077]

[0078] Among them, K represents the total number of predictions in each group, express The The first The dynamic weights of the sampled features, represents sampling the input feature x with a deformable field p.

[0079] Optionally, the aggregated features are projected through the output prediction module to obtain the features processed by Group-wise MSDCN, that is, the image features of the target image.

[0080] It should be noted that feature aggregation is performed based on the sampling features corresponding to the input features and the weight information of the sampling features corresponding to the input features to obtain aggregated features, which provides a basis for determining the image features of the target image.

[0081] Alternatively, if Figure 3 As shown in the figure, after the image features of the target image are obtained through Group-wise MSDCN processing, the category features are injected into the image features of the target image through a preset multi-layer perceptron (such as SwiGLU) to obtain the hidden variables corresponding to the target image (i.e., the output of the network module), and then processed through the normalization module, linear and reshaping modules to obtain the predicted flow (i.e., the target image). For example, the hidden variables corresponding to the target image are decoded into the image RGB space through the preset decoder of the target image generation model (such as the decoder in the variational autoencoder) to obtain the target image.

[0082] In an embodiment of the present application, an image generation request of a target object is received, wherein the image generation request includes at least category information of the target object in the target image to be generated; noise data is generated by using a random number generation function; the category information and the noise data are input into a target image generation model for processing to obtain a target image, wherein the target image generation model includes at least multiple groups of multi-scale deformable convolutional networks, and the multiple groups of multi-scale deformable convolutional networks are used to determine the image features of the target image. Since the multiple groups of multi-scale deformable convolutional networks have linear computational complexity and higher computational efficiency compared with the attention blocks in the Transformer, and have stronger dynamic performance compared with general convolutional networks, the multiple groups of multi-scale deformable convolutional networks based on the target image generation model can generate images with arbitrary resolutions, thereby achieving the purpose of improving the quality of the model's raw images, thereby achieving the technical effect of improving the visual effect of the model-generated images, and thus solving the technical problem that the image generation model in the related art has poor scalability for adaptive resolution generation, resulting in poor visual effects of the generated images.

[0083] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0084] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0085] Example 2

[0086] According to an embodiment of the present application, a training method for an image generation model is also provided, such as Figure 5 As shown, the method includes:

[0087] Step S501, constructing a training sample set based on images in a preset image database;

[0088] Step S502, encoding the training samples in the training sample set to obtain latent variables corresponding to the training samples;

[0089] Step S503, constructing a latent variable with noise added according to the latent variable corresponding to the training sample and the time and noise randomly sampled from a preset time interval;

[0090] Step S504, training the initial image generation model according to the latent variables after adding noise and the preset training target until the loss function of the initial image generation model converges, thereby obtaining any of the above-mentioned target image generation models.

[0091] Optionally, training is performed through a flow matching generation framework, first constructing a noise input, then constructing a training target, and then training the network so that the output of the network is closer and closer to the target. First, a training sample set is constructed based on images in a preset image database (such as ImageNet). For example, the ImageNet image is subjected to image preprocessing such as center cropping, and the preprocessed image is used as a training sample to obtain a training sample set. The training samples in the training sample set are then encoded to obtain latent variables corresponding to the training samples. For example, image encoding is performed through a pre-trained variational autoencoder to map the training samples into latent variables (for example, represented by z). Then a noise input is constructed, that is, the latent variables after adding noise are constructed based on the latent variables corresponding to the training samples and the time and noise randomly sampled from a preset time interval (such as between 0 and 1). For example, time t and noise are uniformly randomly sampled from 0 to 1. , the noise input is constructed by the following formula :

[0092]

[0093] Then according to the hidden variable after adding noise The initial image generation model is trained with the preset training target until the loss function of the initial image generation model converges to obtain the target image generation model.

[0094] Optionally, the preset training objectives are as follows:

[0095]

[0096] Among them, v represents the predicted velocity field of the model. By training the predicted velocity field, the model can be equipped with the ability to generate images.

[0097] Optionally, the loss function is as follows:

[0098]

[0099] in, Express expectations, Represents model parameters.

[0100] In this scheme, a deformable convolution (DCN)-based image generation model is proposed. Group-wise MSDCN is proposed as the basic module, and training and sampling are performed through the flow matching generation framework. On ImageNet, using only 1 / 5 of the training samples, the FID performance (a metric used to measure the similarity between two image data distributions) and sFID performance (an improved version of the FID, a generative model evaluation metric) of the target image generation model in this scheme can surpass the Transformer-based diffusion generation model, while having fewer parameters and computation.

[0101] Example 3

[0102] According to an embodiment of the present application, an image processing method is also provided, such as Figure 6 As shown, the method includes:

[0103] Step S601, obtaining an image generation request of a target object uploaded by a client, wherein the image generation request at least includes category information of a target object in a target image to be generated;

[0104] Step S602, generating noise data using a random number generation function in the cloud server; inputting the category information and the noise data into a target image generation model for processing to obtain a target image, wherein the target image generation model includes at least a plurality of groups of multi-scale deformable convolutional networks, and the plurality of groups of multi-scale deformable convolutional networks are used to determine image features of the target image;

[0105] Step S603: Feedback the target image to the client.

[0106] Through the above scheme, since the multiple groups of multi-scale deformable convolutional networks have linear computational complexity and higher computational efficiency compared with the attention blocks in the Transformer, and have stronger dynamic performance compared with the general convolutional networks, the multiple groups of multi-scale deformable convolutional networks based on the target image generation model can generate images with arbitrary resolutions, thereby achieving the purpose of improving the raw image quality of the model, thereby achieving the technical effect of improving the visual effect of the image generated by the model, and further solving the technical problem that the image generation model in the related technology has poor scalability for adaptive resolution generation, resulting in poor visual effect of the generated image.

[0107] In the cloud server, the specific method for image processing is the same as that in the first embodiment, and will not be repeated here.

[0108] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0110] Example 4

[0111] According to an embodiment of the present application, an image processing device for implementing the above-mentioned image processing method is also provided. Figure 7 As shown, the device includes: a receiving unit 701, a first processing unit 702, and a second processing unit 703.

[0112] The receiving unit 701 is configured to receive an image generation request of a target object, wherein the image generation request at least includes category information of the target object in the target image to be generated;

[0113] A first processing unit 702 is used to generate noise data using a random number generation function;

[0114] The second processing unit 703 is used to input the category information and noise data into the target image generation model for processing to obtain the target image, wherein the target image generation model includes at least multiple groups of multi-scale deformable convolutional networks, and the multiple groups of multi-scale deformable convolutional networks are used to determine the image features of the target image.

[0115] In the image processing device provided in the fourth embodiment of the present application, an image generation request of a target object is received by a receiving unit 701, wherein the image generation request includes at least the category information of the target object in the target image to be generated; the first processing unit 702 generates noise data using a random number generation function; the second processing unit 703 inputs the category information and the noise data into the target image generation model for processing to obtain the target image, wherein the target image generation model includes at least a plurality of groups of multi-scale deformable convolutional networks, and the plurality of groups of multi-scale deformable convolutional networks are used to determine the image features of the target image. In this solution, since the plurality of groups of multi-scale deformable convolutional networks have linear computational complexity and higher computational efficiency compared with the attention block in the Transformer, and have stronger dynamic performance compared with the general convolutional network, the plurality of groups of multi-scale deformable convolutional networks based on the target image generation model can generate images at any resolution, thereby achieving the purpose of improving the quality of the raw image of the model, thereby achieving the technical effect of improving the visual effect of the model-generated image, and thus solving the technical problem that the image generation model in the related art has poor scalability for adaptive resolution generation, resulting in poor visual effect of the generated image.

[0116] Optionally, in the image processing device provided in Example 4 of the present application, the second processing unit 703 includes: a first processing subunit, used to process the category information through a target image generation model to obtain category features, and to process the noise data through the target image generation model to obtain multiple groups of input features of multi-scale deformable convolutional networks; a second processing subunit, used to process the input features through multiple groups of multi-scale deformable convolutional networks to obtain image features of the target image; a third processing subunit, used to inject the category features into the image features of the target image through a preset multi-layer perceptron of the target image generation model to obtain hidden variables corresponding to the target image; and a fourth processing subunit, used to decode the hidden variables corresponding to the target image into the image RGB space through a preset decoder of the target image generation model to obtain the target image.

[0117] Optionally, in the image processing device provided in Example 4 of the present application, the second processing sub-unit includes: a first determination module, used to determine the scale initial value and direction initial value corresponding to the multi-scale deformable convolution network in the multiple groups of multi-scale deformable convolution networks; a second determination module, used to determine the weight information of the sampling features corresponding to the deformable field and the input features according to the scale initial value and direction initial value corresponding to the multi-scale deformable convolution network through the multiple groups of multi-scale deformable convolution networks; a first processing module, used to perform feature aggregation according to the weight information of the sampling features corresponding to the deformable field and the input features through the deformable convolution modules of the multiple groups of multi-scale deformable convolution networks to obtain aggregated features; a second processing module, used to project the aggregated features through the output prediction modules of the multiple groups of multi-scale deformable convolution networks to obtain image features of the target image.

[0118] Optionally, in the image processing device provided in Embodiment 4 of the present application, the second determination module includes: a first determination submodule, used to obtain the scale coefficient of the multi-scale deformable convolutional network according to the scale initial value prediction through multiple groups of scale prediction modules of the multi-scale deformable convolutional network; a second determination submodule, used to obtain the direction of the multi-scale deformable convolutional network according to the direction initial value prediction through multiple groups of direction prediction modules of the multi-scale deformable convolutional network; a third determination submodule, used to determine the deformable field according to the position information of the input feature, the scale coefficient of the multi-scale deformable convolutional network, the direction initial value and the direction of the multi-scale deformable convolutional network through multiple groups of deformable core modules of the multi-scale deformable convolutional network; a fourth determination submodule, used to obtain the weight information of the sampling feature corresponding to the input feature according to the weight coefficient matrix and the bias vector of the sampling feature corresponding to the input feature through multiple groups of dynamic weight prediction modules of the multi-scale deformable convolutional network.

[0119] Optionally, in the image processing device provided in Example 4 of the present application, the first processing module includes: a first processing sub-module, used to sample input features according to the deformable field through multiple groups of multi-scale deformable convolutional networks, and obtain sampling features corresponding to the input features; a second processing sub-module, used to perform feature aggregation based on the sampling features corresponding to the input features and weight information of the sampling features corresponding to the input features, to obtain aggregated features.

[0120] Optionally, in the image processing device provided in Example 4 of the present application, the first processing subunit includes: a third processing module, used to process the category information through the text processor of the target image generation model to obtain a semantic vector corresponding to the category information; a fourth processing module, used to process the semantic vector and noise data corresponding to the category information through a preset encoder of the target image generation model to obtain a noise latent variable; a fifth processing module, used to process the noise latent variable through the block processing module of the target image generation model to obtain an input feature; and a sixth processing module, used to process the category information through the embedding module of the target image generation model to obtain a category feature.

[0121] It should be noted that the above-mentioned receiving unit 701, first processing unit 702, and second processing unit 703 correspond to steps S201 to S203 in Example 1, and the above-mentioned units and corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules as part of the device can be run in the computer terminal 10 provided in Example 1.

[0122] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0123] Example 5

[0124] According to an embodiment of the present application, a training device for an image generation model for implementing the training method of the above-mentioned image generation model is also provided, such as Figure 8 As shown, the device includes: a first construction unit 801, a third processing unit 802, a second construction unit 803, and a fourth processing unit 804.

[0125] A first constructing unit 801 is used to construct a training sample set according to images in a preset image database;

[0126] The third processing unit 802 is used to encode the training samples in the training sample set to obtain latent variables corresponding to the training samples;

[0127] The second construction unit 803 is used to construct the latent variable after adding noise according to the latent variable corresponding to the training sample and the time and noise randomly sampled from the preset time interval;

[0128] The fourth processing unit 804 is used to train the initial image generation model according to the latent variables after adding noise and the preset training target until the loss function of the initial image generation model converges, thereby obtaining any of the above-mentioned target image generation models.

[0129] It should be noted here that the above-mentioned first construction unit 801, third processing unit 802, second construction unit 803, and fourth processing unit 804 correspond to steps S501 to S504 in Example 2, and the instances and application scenarios implemented by the above-mentioned units and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 2.

[0130] Example 6

[0131] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0132] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0133] In this embodiment, the above-mentioned electronic device can execute the program code of the following steps in the image processing method: receiving an image generation request of a target object, wherein the image generation request includes at least category information of the target object in the target image to be generated; generating noise data using a random number generation function; inputting the category information and the noise data into a target image generation model for processing to obtain a target image, wherein the target image generation model includes at least multiple groups of multi-scale deformable convolutional networks, and the multiple groups of multi-scale deformable convolutional networks are used to determine the image features of the target image.

[0134] The electronic device can also execute the program code of the following steps in the image processing method: process the category information through the target image generation model to obtain the category features, process the noise data through the target image generation model to obtain multiple sets of input features of the multi-scale deformable convolutional network; process the input features through multiple sets of multi-scale deformable convolutional networks to obtain the image features of the target image; inject the category features into the image features of the target image through the preset multi-layer perceptron of the target image generation model to obtain the hidden variables corresponding to the target image; decode the hidden variables corresponding to the target image into the image RGB space through the preset decoder of the target image generation model to obtain the target image.

[0135] The electronic device can also execute the program code of the following steps in the image processing method: determining the scale initial value and direction initial value corresponding to the multi-scale deformable convolution network in the multiple groups of multi-scale deformable convolution networks; determining the weight information of the sampling features corresponding to the deformable field and the input features according to the scale initial value and direction initial value corresponding to the multi-scale deformable convolution network through the multiple groups of multi-scale deformable convolution networks; performing feature aggregation according to the weight information of the sampling features corresponding to the deformable field and the input features through the deformable convolution modules of the multiple groups of multi-scale deformable convolution networks to obtain aggregated features; projecting the aggregated features through the output prediction modules of the multiple groups of multi-scale deformable convolution networks to obtain the image features of the target image.

[0136] The electronic device can also execute the program code of the following steps in the image processing method: obtaining the scale coefficient of the multi-scale deformable convolutional network according to the initial scale value through multiple groups of scale prediction modules of the multi-scale deformable convolutional network; obtaining the direction of the multi-scale deformable convolutional network according to the initial direction value through multiple groups of direction prediction modules of the multi-scale deformable convolutional network; determining the deformable field according to the position information of the input feature, the scale coefficient of the multi-scale deformable convolutional network, the initial direction value and the direction of the multi-scale deformable convolutional network through multiple groups of deformable core modules of the multi-scale deformable convolutional network; and obtaining the weight information of the sampling feature corresponding to the input feature according to the weight coefficient matrix and bias vector of the sampling feature corresponding to the input feature through multiple groups of dynamic weight prediction modules of the multi-scale deformable convolutional network.

[0137] The above-mentioned electronic device can also execute the program code of the following steps in the image processing method: sampling the input features according to the deformable field through the deformable convolution modules of multiple groups of multi-scale deformable convolution networks to obtain sampling features corresponding to the input features; performing feature aggregation according to the sampling features corresponding to the input features and the weight information of the sampling features corresponding to the input features to obtain the aggregated features.

[0138] The above-mentioned electronic device can also execute the program code of the following steps in the image processing method: processing the category information through the text processor of the target image generation model to obtain the semantic vector corresponding to the category information; processing the semantic vector and noise data corresponding to the category information through the preset encoder of the target image generation model to obtain the noise latent variable; processing the noise latent variable through the block processing module of the target image generation model to obtain the input feature; processing the category information through the embedding module of the target image generation model to obtain the category feature.

[0139] Optionally, Fig. 9 is a structural block diagram of an electronic device according to an embodiment of the present application. Fig. 9 As shown, the electronic device 90 may include: one or more ( Fig. 9Only one is shown in the figure) processor 902, memory 904. The electronic device 90 may also include a storage controller, through which the memory 904 is controlled and managed; the electronic device 90 may also include a peripheral interface, through which the radio frequency module, audio module and display screen are connected.

[0140] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the image processing method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned image processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0141] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receiving an image generation request of a target object, wherein the image generation request includes at least category information of the target object in the target image to be generated; using a random number generation function to generate noise data; inputting the category information and the noise data into a target image generation model for processing to obtain a target image, wherein the target image generation model includes at least multiple groups of multi-scale deformable convolutional networks, and the multiple groups of multi-scale deformable convolutional networks are used to determine the image features of the target image.

[0142] Optionally, the processor may also execute the following program code: processing the category information through a target image generation model to obtain category features, processing the noise data through a target image generation model to obtain multiple sets of input features of multi-scale deformable convolutional networks; processing the input features through multiple sets of multi-scale deformable convolutional networks to obtain image features of a target image; injecting the category features into the image features of the target image through a preset multi-layer perceptron of the target image generation model to obtain hidden variables corresponding to the target image; decoding the hidden variables corresponding to the target image into the image RGB space through a preset decoder of the target image generation model to obtain the target image.

[0143] Optionally, the processor may also execute the program code of the following steps: determining the scale initial values ​​and direction initial values ​​corresponding to the multi-scale deformable convolutional networks in the multiple groups of multi-scale deformable convolutional networks; determining the weight information of the sampling features corresponding to the deformable field and the input features according to the scale initial values ​​and direction initial values ​​corresponding to the multi-scale deformable convolutional networks through the multiple groups of multi-scale deformable convolutional networks; performing feature aggregation according to the weight information of the sampling features corresponding to the deformable field and the input features through the deformable convolution modules of the multiple groups of multi-scale deformable convolutional networks to obtain aggregated features; and projecting the aggregated features through the output prediction modules of the multiple groups of multi-scale deformable convolutional networks to obtain the image features of the target image.

[0144] Optionally, the processor may also execute the program code of the following steps: obtaining the scale coefficient of the multiscale deformable convolutional network according to the scale initial value prediction through multiple groups of scale prediction modules of the multiscale deformable convolutional network; obtaining the direction of the multiscale deformable convolutional network according to the direction initial value prediction through multiple groups of direction prediction modules of the multiscale deformable convolutional network; determining the deformable field according to the position information of the input feature, the scale coefficient of the multiscale deformable convolutional network, the direction initial value and the direction of the multiscale deformable convolutional network through multiple groups of deformable core modules of the multiscale deformable convolutional network; obtaining the weight information of the sampling feature corresponding to the input feature according to the weight coefficient matrix and the bias vector of the sampling feature corresponding to the input feature through multiple groups of dynamic weight prediction modules of the multiscale deformable convolutional network.

[0145] Optionally, the processor may also execute the program code of the following steps: sampling the input features according to the deformable field through the deformable convolution modules of multiple groups of multi-scale deformable convolution networks to obtain sampling features corresponding to the input features; performing feature aggregation according to the sampling features corresponding to the input features and the weight information of the sampling features corresponding to the input features to obtain aggregated features.

[0146] Optionally, the processor may also execute the following program code: processing the category information through the text processor of the target image generation model to obtain a semantic vector corresponding to the category information; processing the semantic vector and noise data corresponding to the category information through the preset encoder of the target image generation model to obtain noise latent variables; processing the noise latent variables through the block processing module of the target image generation model to obtain input features; processing the category information through the embedding module of the target image generation model to obtain category features.

[0147] It can be understood by those skilled in the art that Fig. 9The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Fig. 9 The structure of the electronic device is not limited. Fig. 9 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Fig. 9 Different configurations shown.

[0148] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0149] Example 7

[0150] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image processing method provided in the first embodiment.

[0151] Optionally, in this embodiment, the above storage medium may be located in any electronic device in a group of electronic devices in a computer network, or in any mobile terminal in a group of mobile terminals.

[0152] Example 8

[0153] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and when the computer program is executed by a processor, the image processing method provided in the first embodiment is implemented.

[0154] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0155] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0157] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0159] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0160] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: include: Receiving an image generation request of a target object, wherein the image generation request at least includes category information of the target object in the target image to be generated; A random number generation function is used to generate noise data; The category information and the noise data are input into a target image generation model for processing to obtain the target image, wherein the target image generation model includes at least a plurality of groups of multi-scale deformable convolutional networks, and the plurality of groups of multi-scale deformable convolutional networks are used to determine image features of the target image.

2. The method according to claim 1, characterized in that Inputting the category information and the noise data into a target image generation model for processing to obtain the target image includes: Processing the category information through the target image generation model to obtain category features, and processing the noise data through the target image generation model to obtain input features of the multiple groups of multi-scale deformable convolutional networks; Processing the input features through the multiple groups of multi-scale deformable convolutional networks to obtain image features of the target image; Injecting the category features into the image features of the target image through a preset multi-layer perceptron of the target image generation model to obtain latent variables corresponding to the target image; The target image is obtained by decoding the latent variables corresponding to the target image into the image RGB space through a preset decoder of the target image generation model.

3. The method according to claim 2, characterized in that The input features are processed by the multiple groups of multi-scale deformable convolutional networks to obtain image features of the target image, including: Determine initial scale values ​​and initial direction values ​​corresponding to the multi-scale deformable convolutional networks in the plurality of groups of multi-scale deformable convolutional networks; Determining weight information of sampling features corresponding to the deformable field and the input features through the multiple groups of multi-scale deformable convolutional networks according to the scale initial values ​​and the direction initial values ​​corresponding to the multi-scale deformable convolutional networks; Performing feature aggregation according to weight information of the sampled features corresponding to the deformable field and the input features through the deformable convolution modules of the multiple groups of multi-scale deformable convolution networks to obtain aggregated features; The aggregated features are projected through the output prediction modules of the multiple groups of multi-scale deformable convolutional networks to obtain image features of the target image.

4. The method according to claim 3, characterized in that Determining weight information of sampling features corresponding to the deformable field and the input features through the multiple groups of multi-scale deformable convolutional networks according to the scale initial values ​​and the direction initial values ​​corresponding to the multi-scale deformable convolutional networks includes: Obtaining a scale coefficient of the multi-scale deformable convolutional network according to the scale initial value prediction by the scale prediction modules of the multiple groups of multi-scale deformable convolutional networks; Obtaining the direction of the multi-scale deformable convolutional network according to the direction initial value prediction by the direction prediction modules of the multiple groups of multi-scale deformable convolutional networks; Determining the deformable field by using the deformable kernel modules of the multiple groups of multi-scale deformable convolutional networks according to the position information of the input feature, the scale coefficient of the multi-scale deformable convolutional network, the initial direction value, and the direction of the multi-scale deformable convolutional network; The dynamic weight prediction module of the multiple groups of multi-scale deformable convolutional networks predicts the weight information of the sampling features corresponding to the input features based on the weight coefficient matrix and bias vector of the sampling features corresponding to the input features.

5. The method according to claim 3, characterized in that: The deformable convolution modules of the multiple groups of multi-scale deformable convolutional networks perform feature aggregation according to the deformable field and the weight information of the sampling features corresponding to the input features, and the aggregated features obtained include: The input features are sampled according to the deformable field by using the deformable convolution modules of the multiple groups of multi-scale deformable convolution networks to obtain sampling features corresponding to the input features; Feature aggregation is performed according to the sampling feature corresponding to the input feature and the weight information of the sampling feature corresponding to the input feature to obtain the aggregated feature.

6. The method according to claim 2, characterized in that Processing the category information through the target image generation model to obtain category features, and processing the noise data through the target image generation model to obtain the input features of the multiple groups of multi-scale deformable convolutional networks include: Processing the category information through a text processor of the target image generation model to obtain a semantic vector corresponding to the category information; Processing the semantic vector corresponding to the category information and the noise data through a preset encoder of the target image generation model to obtain a noise latent variable; Processing the noise latent variable by a block processing module of the target image generation model to obtain the input feature; The category information is processed by an embedding module of the target image generation model to obtain the category feature.

7. A training method for an image generation model, characterized in that: include: Constructing a training sample set based on images in a preset image database; Encoding the training samples in the training sample set to obtain latent variables corresponding to the training samples; Constructing a noisy latent variable based on the latent variable corresponding to the training sample and the time and noise randomly sampled from a preset time interval; The initial image generation model is trained according to the latent variables after adding noise and the preset training target until the loss function of the initial image generation model converges, thereby obtaining the target image generation model described in any one of claims 1 to 6.

8. An image processing method, characterized in that: include: Obtaining an image generation request of a target object uploaded by a client, wherein the image generation request at least includes category information of a target object in a target image to be generated; In a cloud server, a random number generation function is used to generate noise data; the category information and the noise data are input into a target image generation model for processing to obtain the target image, wherein the target image generation model includes at least a plurality of groups of multi-scale deformable convolutional networks, and the plurality of groups of multi-scale deformable convolutional networks are used to determine image features of the target image; The target image is fed back to the client.

9. An image processing device, characterized in that: include: A receiving unit, configured to receive an image generation request of a target object, wherein the image generation request at least includes category information of the target object in the target image to be generated; A first processing unit, configured to generate noise data using a random number generation function; The second processing unit is used to input the category information and the noise data into a target image generation model for processing to obtain the target image, wherein the target image generation model includes at least a plurality of groups of multi-scale deformable convolutional networks, and the plurality of groups of multi-scale deformable convolutional networks are used to determine the image features of the target image.

10. A training device for an image generation model, characterized in that: include: A first construction unit, used to construct a training sample set based on images in a preset image database; A third processing unit is used to encode the training samples in the training sample set to obtain latent variables corresponding to the training samples; A second construction unit is used to construct a noisy latent variable according to the latent variable corresponding to the training sample and the time and noise randomly sampled from a preset time interval; The fourth processing unit is used to train the initial image generation model according to the latent variables after adding noise and the preset training objectives until the loss function of the initial image generation model converges, thereby obtaining the target image generation model described in any one of claims 1 to 6.

11. An electronic device, characterized in that: include: A memory storing an executable program; A processor is used to run the program, wherein the program executes the image processing method of any one of claims 1 to 6 and 8 when running.

12. A computer program product, characterized in that The method comprises a computer program or instructions, which implement the image processing method according to any one of claims 1 to 6 and 8 when executed by a processor.

Images