Image generation methods, apparatuses, electronic devices, storage media, and program products

Abstract

This application provides an image generation method, apparatus, electronic device, storage medium, and program product. The method includes: acquiring at least one first instance in a first image; extracting features from the first instance to obtain first instance features; determining a noise gradient of the first instance based on the first instance features; acquiring a noise amplitude negatively correlated with the noise gradient; injecting noise positively correlated with the noise amplitude into the first instance to obtain a first noisy image; performing denoising processing on the first noisy image to obtain a first generated image; and fusing the first generated image with the positional information in the first instance to obtain a second image including the first instance. This application improves the quality and efficiency of image generation.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image generation method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the continuous development of computer technology, users have an increasing demand for image generation. Image generation technology can use algorithms or models to recover or create high-fidelity images from noisy, low-resolution or defective data based on text or images, and is applied in many fields such as medical care, artistic creation, and industrial inspection.

[0003] In the image generation process of related technologies, images are uniformly processed by applying random Gaussian noise to the image step by step and then denoising to obtain images of different styles. During the generation process, the local features of the image may be distorted or lose details under the influence of global noise, resulting in unstable image quality. The iterative processing of the overall image affects the efficiency of image generation. Summary of the Invention

[0004] This application provides an image generation method, apparatus, electronic device, storage medium, and program product, which can improve the quality and efficiency of image generation.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides an image generation method, the method comprising:

[0007] Obtain at least one first instance from the first image;

[0008] Feature extraction is performed on the first instance to obtain the features of the first instance;

[0009] The noise gradient of the first instance is determined based on the features of the first instance;

[0010] Obtain the noise amplitude that is negatively correlated with the noise gradient, and inject noise that is positively correlated with the noise amplitude into the first instance to obtain a first noise image;

[0011] The first noisy image is denoised to obtain the first generated image;

[0012] The first generated image and the location information in the first instance are fused to obtain a second image that includes the first instance.

[0013] This application provides an image generation apparatus, including:

[0014] The image segmentation module is used to obtain at least one first instance in the first image;

[0015] The image generation module is used to extract features from the first instance to obtain first instance features; determine the noise gradient of the first instance based on the first instance features; obtain the noise amplitude negatively correlated with the noise gradient, inject noise positively correlated with the noise amplitude into the first instance to obtain a first noise image; perform denoising processing on the first noise image to obtain a first generated image; and perform fusion processing on the first generated image and the position information in the first instance to obtain a second image including the first instance.

[0016] This application provides an electronic device, the electronic device comprising:

[0017] Memory is used to store executable instructions or computer programs.

[0018] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the image generation method provided in the embodiments of this application.

[0019] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the image generation method provided in this application when executed by a processor.

[0020] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the image generation method provided in this application.

[0021] The embodiments of this application have the following beneficial effects:

[0022] At least one first instance is acquired from the first image, and feature extraction is performed on the first instance to obtain its features. Instance acquisition ensures that subsequent processing focuses on independent processing of each first instance. The noise gradient of the first instance is determined based on its features, and the features are quantified. Different noise injection strategies are designed for different instances. These strategies inject noise positively correlated with the noise amplitude into the first instance by acquiring the noise amplitude negatively correlated with the noise gradient. The injected noise is adjusted according to the noise amplitude of each first instance, improving the efficiency of noise injection and preserving the original details of the first instance in the first noisy image. The first noisy image is then denoised to obtain a first generated image, reducing interference during denoising and improving the accuracy and speed of the generated image. The positional information contained in the first instance and the first generated image are fused to restore the blurred boundaries of the first instance in the first generated image, ensuring accurate positioning of the second image and avoiding misalignment, thus improving the quality of the generated second image. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating the application mode of the image generation method provided in the embodiments of this application;

[0024] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0025] Figure 3A This is a first flowchart illustrating the image generation method provided in this application embodiment;

[0026] Figure 3B This is a schematic diagram of the second process of the image generation method provided in the embodiments of this application;

[0027] Figure 3C This is a schematic diagram of the third process of the image generation method provided in the embodiments of this application;

[0028] Figure 3D This is a schematic diagram of the fourth process of the image generation method provided in the embodiments of this application;

[0029] Figure 4 This is a schematic diagram of the structure of the instance segmentation model provided in the embodiments of this application;

[0030] Figure 5 This is a schematic diagram of the instance segmentation results provided in the embodiments of this application;

[0031] Figure 6 This is a schematic diagram illustrating the changes in the noise state of the image;

[0032] Figure 7 This is a schematic diagram of the denoising model provided in the embodiments of this application.

[0033] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0036] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0037] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0038] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0039] In this application embodiment, the collection and processing of relevant data (e.g., image data) should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0040] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0041] 1) Image Generation: refers to the technology of generating high-fidelity, high-resolution visual content (e.g., photos, videos, artworks) from noisy, low-resolution, defective data or abstract descriptions (e.g., text, images) using algorithms or models. By learning the distribution of real data, new visual information is gradually recovered or created. In the embodiments of this application, image generation is to generate higher quality and higher precision images based on the original images.

[0042] 2) Instance Segmentation: This refers to the process of separating multiple independent objects (instances) from the background in a complex image using computer vision algorithms, and accurately labeling the boundaries and location information of each instance. Convolutional neural networks (e.g., Mask R-CNN) are used to extract multi-level features from the image, generate candidate region proposals, and achieve fine-grained separation of pixels-level instances through bounding box regression and mask prediction.

[0043] 3) Diffusion Model: This is a generative deep learning technique based on probability theory and stochastic processes. The diffusion model maps low-dimensional noise (e.g., random noise, low-resolution images) to high-dimensional real data distribution (e.g., natural images) through an iterative process of gradually injecting Gaussian noise and inverse denoising, generating visual content with semantic consistency and detail fidelity.

[0044] 4) Noising: This is the forward process of the diffusion model, referring to the process of gradually injecting Gaussian noise in the early stages of image generation to transform a clear image (or a low-resolution image) into a purely noisy image. Its core is controlling the noise intensity at each time step, causing the image to gradually distort and providing a reversible path for subsequent denoising.

[0045] 5) Denoising: Denoising is the reverse process of the diffusion model, referring to the process of gradually restoring structure from pure noise. It uses a neural network to predict the noise distribution at each step and inversely restores image details.

[0046] With the continuous development of computer technology, users' demand for image generation is increasing. Image generation technology can recover or create high-fidelity images from noisy, low-resolution, or defective data based on text or images using algorithms or models. Related technologies perform uniform image processing; during image generation, all regions (whether complex instances or simple backgrounds) are subjected to the same noise treatment. Local image features may be distorted or lose details under the influence of global noise, resulting in unstable image quality. Image generation requires multiple steps, each processing the entire image and progressively denoising and restoring it in each iteration. This iterative processing of the entire image affects the efficiency of image generation; on high-resolution or large-scale datasets, image generation is time-consuming and computationally resource-intensive.

[0047] This application provides an image generation method, an image generation apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the quality and efficiency of image generation.

[0048] The following describes exemplary applications of the electronic devices provided in the embodiments of this application. These devices can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals, or as servers. Exemplary applications when the device is implemented as a terminal or server will be described below.

[0049] See Figure 1 , Figure 1 This is a schematic diagram illustrating the application mode of the image generation method provided in this application embodiment. It is an example to support an image generation application. Figure 1 The system involves server 200, network 300, terminal device 400 and database 500. Terminal device 400 is connected to server 200 through network 300. Network 300 can be a wide area network or a local area network, or a combination of both.

[0050] In some embodiments, the present application embodiments can be implemented collaboratively by a server and a terminal device. For example, the user can be someone skilled in the art, server 200 is a server for generating model images, terminal device 400 is a terminal operated by the user, and database 500 stores the original images to be generated. Terminal device 400 sends an image generation request to server 200, server 200 receives the image generation request, and through the image generation method provided in the present application embodiments, uses the original image to be generated as a first image, optimizes the accuracy of the first image to obtain a generated second image as an optimized new image, and sends the optimized new image to terminal device 400.

[0051] The image generation method provided in this application can be applied to various scenarios that require image optimization, such as medical imaging, industrial inspection, and security. Examples are given below.

[0052] 1) In the field of medical imaging, for example, when a terminal device receives an image generation request from a medical professional, the server optimizes the original diagnostic images or case images using image generation methods to generate clearer, more accurate images to assist medical professionals in making medical diagnoses.

[0053] 2) In the field of industrial inspection, for example, when terminal equipment receives image generation requests from technicians, the server optimizes the original mechanical part images using image generation methods to generate clearer, higher-precision part images, assisting technicians in quality inspection of part production.

[0054] 3) In the security field, for example, when a terminal device receives an image generation request from a staff member, the server optimizes the original low-light and inclement weather images using image generation methods to generate clearer and more accurate images of faces or license plates, assisting staff in traffic safety control.

[0055] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 2 The server 200 shown includes at least one processor 410, memory 450, and at least one network interface 420. The various components of server 200 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0056] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0057] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0058] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0059] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0060] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0061] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including Bluetooth, WiFi, and Universal Serial Bus (USB).

[0062] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 An image generation apparatus 455 stored in memory 450 is shown. This apparatus can be software in the form of programs and plugins, and includes the following software modules: an image segmentation module 4551, an image generation module 4552, and a model training module 4553. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0063] In some embodiments, the terminal or server can implement the image generation method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs); or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0064] The image generation method provided in this application will be described in conjunction with exemplary applications and implementations of the electronic devices provided in the embodiments of this application.

[0065] The image generation method provided in the embodiments of this application will be described below. As mentioned above, the electronic device implementing the image generation method of the embodiments of this application can be a terminal, a server, or a combination of both. Therefore, the executing entity of each step will not be described again below.

[0066] See Figure 3A , Figure 3A This is a first flowchart illustrating the image generation method provided in this application embodiment, which will be combined with... Figure 3A The steps shown are explained below. Figure 3A The implementing body is Figure 1Server 200.

[0067] In step 301, at least one first instance in the first image is obtained.

[0068] In some embodiments, see Figure 3B , Figure 3B This is a schematic diagram of the second process of the image generation method provided in the embodiments of this application; Figure 3A Step 301 can be executed Figure 3B Steps 3011 to 3015 are implemented, and the details are explained below.

[0069] In step 3011, the global feature map of the first image is obtained.

[0070] In some embodiments, the pre-trained convolutional neural network utilizes network layers of different depths to extract features from the first image. Shallow networks extract local features of the first image, while deep networks extract global semantics. Through layer-by-layer convolution of the convolutional neural network, multi-level features of the first image are obtained. The pre-trained convolutional neural network can be a backbone network. The multi-level features of the first image are aligned at different resolutions to obtain aligned multi-scale features. Then, pooling is performed on the aligned multi-scale features. The resulting global feature map of the first image includes semantic and spatial information of the first image. The semantic information includes the category and shape of objects, and the spatial information includes the position and size of objects. This information provides the foundation for subsequent object detection and instance segmentation.

[0071] In step 3012, at least one candidate region is identified from the global feature map.

[0072] In some embodiments, each pixel in the global feature map is considered as a feature point, and the row and column coordinates (i.e., row and column numbers) of the feature point represent its position in the matrix of the global feature map. In the global feature map, the first image is divided into local regions with fixed step sizes, and each local region is associated with a feature point in the global feature map to determine the mapping relationship. The width and height of the first image are divided by the downsampling rate, and the result is rounded down to obtain the width and height of the global feature map. Based on the mapping relationship, each feature point in the global feature map is converted to the coordinate system of the first image according to the downsampling rate. The geometric center of the local region corresponding to the feature point converted to the first image is used as the reference point of the rectangular anchor frame. Combined with a preset reference scale and aspect ratio, candidate boxes of multiple sizes are generated. The actual width and height of the candidate boxes are calculated according to the aspect ratio using a square root transformation to ensure that the area of ​​the rectangular anchor frame is consistent under different aspect ratios. Rectangular anchor frames covering multiple scales and shapes are generated at the position of each feature point, forming at least one candidate region strictly aligned with the spatial distribution of the global feature map.

[0073] For example, the first image has dimensions of 1280 pixels (width) × 720 pixels (height), a downsampling rate (step size) S = 32, a preset baseline scale of 32 pixels, and aspect ratios of 1:1 (square), 1:2 (vertical rectangle), and 2:1 (horizontal rectangle). Three rectangular anchor boxes corresponding to these aspect ratios are generated for each feature point. The width and height of the first image are divided by the downsampling rate, and the results are rounded down to obtain the width and height (i.e., dimensions) of the global feature map. Based on the global feature map dimensions of 40 × 22... A total of 40×22=880 feature points were determined in the global feature map. Taking feature point (2,3) as an example (located in the 3rd row and 4th column of the feature map), feature point (2,3) was mapped to the region with the upper left corner (32×2, 32×3)=(64,96) and the lower right corner (32×3, 32×4)=(96,128) as the diagonal vertices in the coordinate system of the first image. The region covers 32×32 pixels.

[0074] Each feature point in the global feature map is converted to the coordinate system of the first image according to the downsampling rate. The horizontal coordinate is 32×(2+0.5)=80, and the vertical coordinate is 32×(3+0.5)=112. That is, the center coordinates of the rectangular anchor frame are (80, 112) in the coordinate system of the first image. When the aspect ratio of the rectangular anchor frame is 1:1, the width is... Height is The rectangular anchor frame has a horizontal dimension of [80-16, 80+16] (i.e., 64~96) and a vertical dimension of [112-16, 112+16] (i.e., 96~128), completely coinciding with a local area in the first image. When the aspect ratio of the rectangular anchor frame is 1:2, the width is... Height is The horizontal dimensions of the rectangular anchor frame are 68.69–91.31, and the vertical dimensions are 89.37–134.63. If three rectangular anchor frames with different aspect ratios are generated for each feature point, a total of 40 × 22 × 3 = 2640 rectangular anchor frames are generated, and each rectangular anchor frame corresponds to a candidate region.

[0075] In step 3013, the local feature map corresponding to the candidate region in the global feature map is determined.

[0076] In some embodiments, the different levels contained in the global feature map constitute a multi-scale feature pyramid. Based on the width and height of the candidate region in the first image, the corresponding level of the candidate region in the multi-scale feature pyramid is calculated using a preset formula. By calculating the ratio of the candidate region area to a preset baseline area, and combining this with a logarithmic function, the mapping level is dynamically adjusted. Smaller candidate regions are mapped to higher resolution levels, and larger regions are mapped to lower resolution levels. For example, if the candidate region area is w×h, the mapping level k can be expressed as... Where k0 is the baseline layer (e.g., layer 3) and S0 is the baseline size (e.g., 224 pixels), the calculation result is rounded down, and the target layer is determined based on the rounded result. Smaller candidate regions (e.g., 32×32) are mapped to a high-resolution layer (e.g., layer 2, downsampling rate 16), and larger regions (e.g., 256×256) are mapped to a low-resolution layer (e.g., layer 5, downsampling rate 128).

[0077] Candidate regions are aligned with the corresponding level's global feature map through a candidate region alignment operation. After alignment, the candidate region is projected onto the corresponding level's global feature map. A bounding box is defined on the projected candidate region to define its extent. This bounding box is uniformly divided into a fixed number of grid cells, and the geometric center within each grid cell is determined. Based on bilinear interpolation, the weighted pixel values ​​of the four adjacent feature points on the global feature map are calculated for the geometric center within each grid cell. Bilinear interpolation is a commonly used interpolation method used to find the pixel values ​​corresponding to the bounding box coordinates of the candidate region in the feature map. After aligning the feature grid through bilinear interpolation, a fixed-size local feature map is obtained, ensuring that candidate regions of different sizes can be processed uniformly. The local feature map is a standardized representation of the candidate region in the depth feature space.

[0078] For example, the area of ​​the candidate region is 64×128=8192, the calculated mapping level k=2, the corresponding downsampling rate is 16, the size of the projection of level 2 is 4×6 (64 / 16=4, 128 / 16=6), and the floating-point coordinate area in the feature map of level 2 is 4.0×8.0. The projected area of ​​4.0×8.0 is uniformly divided into target-sized (e.g., 7×7 grid cells). The width of each grid cell is approximately 4.0 / 7≈0.571, and the height is approximately 8.0 / 7≈1.143. Taking the grid cell in the 3rd row and 4th column as an example, the x-coordinate of the geometric center of the grid cell is determined to be 0.571×(3-0.5)=1.428, and the y-coordinate is determined to be 1.143×(4-0.5)=4.000. The adjacent feature points are (1,4), (1,5), (2,4), and (2,5). Based on the bilinear interpolation method, the weighted pixel values ​​of the four adjacent feature points of the geometric center in each grid cell on the global feature map are calculated. The interpolated weighted pixel values ​​are aggregated to obtain the interpolation result F=0.572·F (1,4) +0.428·F(F (2,4) The interpolated value F is a continuous feature representation on the local feature map, which incorporates the spatial context information of the four surrounding points. The interpolation results of all 7×7 grid cells are arranged into a fixed matrix to form a standardized local feature map.

[0079] In step 3014, category prediction is performed based on the local feature map to obtain the confidence level of the candidate region, and regression is performed based on the local feature map to obtain the position offset. The position of the candidate region is corrected based on the position offset to obtain the corrected candidate region.

[0080] In some embodiments, the classification branch of a pre-trained classification and regression model predicts the class of candidate regions based on local feature maps. The fully connected layer and activation function of the classification branch output the class probability distribution of each candidate region, and the highest value in the class probability distribution is used as the confidence score of the candidate region. The regression branch of the pre-trained classification and regression model performs bounding box regression on the candidate regions based on local feature maps. The fully connected layer and linear activation function of the regression branch predict the bounding box offset, obtaining the offset relative to the original candidate region. The coordinates of the original candidate region are corrected based on the offset obtained from the regression, resulting in the corrected candidate region.

[0081] For example: Suppose the class probability distribution of a local feature map is [cat: 0.7, dog: 0.2, background: 0.1]. Then, the highest value of 0.7 in the class probability distribution is taken as the confidence score of the candidate region, corresponding to the class "cat". The original candidate region is set as (x = 50, y = 60, w = 20, h = 30), where x and y are the coordinates of the center point, w represents the width of the candidate region, and h represents the height of the candidate region. The regression prediction offsets are Δx = 0.5, Δy = -0.2, Δw = 0.1, and Δh = 0.3. The position of the candidate region is adjusted according to the offsets. The new x is 60 (50 + 20 * 0.5), the new y is 54 (60 + 30 * (-0.2)), and the new w is approximately 22.1 (20 * e). 0.1 The new h is approximately 40.5 (30*e) 0.3 The corrected candidate region is (x=60, y=54, w≈22.1, h≈40.5).

[0082] In step 3015, a first instance is determined based on the corrected candidate region and the confidence level.

[0083] In some embodiments, based on the corrected candidate regions and their confidence levels, the top N corrected candidate regions in the class probability distribution are retained during initial filtering. N is a preset threshold for the number of candidate regions to exclude low-probability candidate regions. Non-Maximum Suppression (NMS) is then applied to the remaining candidate regions after initial filtering, eliminating overlapping corrected candidate regions based on a preset intersection-over-union (IoU) threshold. NMS is an algorithm in computer vision used to remove overlapping candidate regions and retain the best detection result in object detection tasks. Multiple corrected candidate regions may overlap on the same target; NMS selects the best candidate region from the overlapping corrected candidate regions as the final corrected candidate region. Calculate the intersection-union ratio (CIU) between any two corrected candidate regions. If the CIU exceeds a preset CIU threshold (e.g., 0.5), it is determined that the two corrected candidate regions overlap on the same target. Only the corrected candidate region with higher confidence is retained to achieve redundant detection of the target. Iterate the above process until the determination of all corrected candidate regions is completed. The first instance is determined based on the selected and retained corrected candidate regions and their corresponding confidence levels.

[0084] In some embodiments, see Figure 4 , Figure 4This is a schematic diagram of the instance segmentation model provided in the embodiment of this application. Instance segmentation of the first image is achieved through the instance segmentation model. The first image 401 is input into the instance segmentation model. The region detection module 402 identifies the candidate regions in the first image. The convolutional layer 403 containing multiple convolutions predicts the category of the candidate regions and adjusts the coordinates of the candidate regions to obtain the category probability distribution and the corrected candidate regions. The highest value in the category probability distribution is taken as the classification result 4031, that is, the confidence of the corrected candidate regions. The first instance 404 is determined from the first image based on the corrected candidate regions and the confidence, thus completing the instance segmentation process performed on the first image 401. Each first instance 404 is clearly marked in the first image 401, and the first image 401 is segmented into multiple non-overlapping first instances 404.

[0085] Through the embodiments of this application, a global feature map integrating semantic and spatial information is generated, providing rich contextual support for instance localization. Candidate regions covering targets of different scales are generated. By aligning the feature grid through bilinear interpolation, the candidate regions are mapped to local feature maps of fixed size, ensuring the standardized representation of targets of different sizes in the feature space. Based on the local feature maps, class prediction and bounding box regression are performed simultaneously to determine the confidence of the target candidate regions and perform position correction. Candidate boxes with high confidence and accurate localization are retained as the first instance, effectively balancing detection recall and accuracy.

[0086] In some embodiments, see Figure 3C , Figure 3C This is a schematic diagram of the third process of the image generation method provided in the embodiments of this application; Figure 3B Step 3015 can be executed Figure 3C Steps 30151 to 30156 are implemented, and the details are explained below.

[0087] In step 30151, the corrected candidate regions that are below the preset confidence threshold are removed, and the remaining corrected candidate regions after removal are divided into multiple groups. The confidence of the corrected candidate regions in a group is within the confidence interval corresponding to the group.

[0088] In some embodiments, candidate regions with confidence levels lower than the confidence threshold are filtered out from the corrected candidate regions according to a preset confidence threshold, thereby eliminating the interference of false detections or fuzzy predictions. The remaining corrected candidate regions after removal are divided into multiple groups according to a preset confidence interval, and the corrected candidate regions in a group have the same or similar confidence levels.

[0089] For example: There are 10 corrected candidate regions with confidence levels of 0.95, 0.92, 0.88, 0.85, 0.80, 0.75, 0.70, 0.65, 0.60, and 0.55. The preset confidence threshold is 0.7. Iterate through the confidence levels of all corrected candidate regions and remove those with confidence levels below 0.7. The remaining corrected candidate regions and their confidence levels are: 0.95, 0.92, 0.88, 0.85, 0.80, 0.75, and 0.70. The preset confidence intervals are: Interval 1: [0.90, 1.00], Interval 2: [0.80, 0.89], Interval 3: [0.70, 0.79], and Interval 4: [0.65, 0.69]. The remaining corrected candidate regions are divided into corresponding interval groups according to their confidence levels. The group corresponding to interval 1 includes the corrected candidate regions with confidence levels of 0.95 and 0.92, and the group corresponding to interval 2 includes the corrected candidate regions with confidence levels of 0.88 and 0.85.

[0090] In step 30152, the following processing is performed for each group: determine the intersection-union ratio (IUR) of any two corrected candidate regions in the group; when the IUR exceeds a preset IUR threshold, retain the corrected candidate region with higher confidence; iteratively perform the following processing: determine the IUR of the retained corrected candidate region with any remaining corrected candidate region in the group; when the IUR exceeds a preset IUR threshold, retain the corrected candidate region with higher confidence; when the iteration ends, the finally retained corrected candidate region is taken as the target candidate region in the group.

[0091] In some embodiments, the corrected candidate regions are sorted from highest to lowest confidence level. The current processing index is initialized as the first position of the group. Then, the candidate region corresponding to the current index is selected as the baseline candidate region. All subsequent corrected candidate regions are traversed, and the intersection-over-union (IoU) ratio between the current baseline candidate region and the uncompared corrected candidate regions is calculated. IoU is an indicator that measures the degree of overlap between two regions. When IoU exceeds a preset IoU threshold, the two corrected candidate regions are determined to be highly overlapping and pointing to the same target, and the corrected candidate region with higher confidence is retained. After completing one round of traversal, the current index is shifted one position to the right, and the above iterative process is repeated. The step of determining the IoU ratio between the retained corrected candidate region and any remaining corrected candidate region in the group is executed. When the IoU exceeds the preset IoU threshold, the corrected candidate region with higher confidence is retained until all corrected candidate regions have been processed, and the iteration ends. By successively retaining the candidate regions with higher confidence, it is ensured that only non-overlapping corrected candidate regions with decreasing confidence are retained in the group, and finally, the target candidate region in the candidate region group is determined.

[0092] For example, with an intersection-union (IU) threshold of 0.5, four corrected candidate regions for the same target category (e.g., "cat") are sorted in descending order of confidence: Box1 (confidence 0.9, coordinates [10, 20, 50, 60]), Box2 (confidence 0.8, coordinates [12, 22, 52, 62]), Box3 (confidence 0.7, coordinates [15, 25, 55, 65]), and Box4 (confidence 0.6, coordinates [100, 150, 200, 200]). The current processing index points to Box1, and the baseline candidate region for the first round of traversal is Box1. The IU of Box1 and Box2 is 0.85, which is greater than the IU threshold. Since Box1 has a higher confidence level than Box2, Box2 is removed and Box1 is retained. The IU of Box1 and Box3 is 0.72, which is greater than the IU threshold. Since Box1 has a higher confidence level than Box3, Box3 is removed and Box1 is retained. The cross-union ratio (CUI) of Box1 and Box4 is 0.01, which is less than the CUI threshold. Therefore, Box4 is retained, and Box1 and Box4 are selected as target candidate regions in the candidate region grouping.

[0093] In step 30153, the maximum confidence value of the target candidate region is taken as the confidence value of the first instance.

[0094] In some embodiments, all confidence scores in the target candidate region are traversed, and the maximum confidence score is taken as the confidence score of the first instance. For example, the confidence scores of Box1 and Box4 are traversed, and the confidence score of Box1 is the largest, so the confidence score of Box1 (0.9) is taken as the confidence score of the first instance.

[0095] In step 30154, a first mask of the target candidate region is obtained, the first mask is upsampled, and the upsampled first mask is binarized to obtain a second mask.

[0096] In some embodiments, a first mask corresponding to the target candidate region is obtained through an instance segmentation model. The first mask is a low-resolution probability map with a resolution lower than that of the first image. The shape and size of the first mask correspond to the target candidate region and are used to represent the shape and position of the instances included in the candidate region. The pixel values ​​of the first mask are typically between 0 and 1, representing the probability that each pixel belongs to an instance. The first mask is upsampled. Upsampling is a technique that enlarges a low-resolution image to a high-resolution image. Each pixel of the first mask is mapped to its corresponding position in a high-resolution grid. Interpolation weights are calculated based on the coordinate distances of the four neighboring pixels, and the probability values ​​of each point in the high-resolution grid are obtained by weighted summation, making the size of the first mask consistent with the spatial resolution of the original input image. The upsampled first mask is binarized. Binarization presents the entire image with a clear visual effect of only black and white. A fixed pixel value threshold is selected, and pixel values ​​in the upsampled mask that are greater than or equal to the pixel value threshold are set to 1, and pixel values ​​that are less than the pixel value threshold are set to 0, resulting in a second mask. The second mask has only 0 and 1 pixel values, indicating whether each pixel belongs to the target candidate region.

[0097] For example, the first mask of the target candidate region is 28×28 with a probability value range of [0, 1]. After bilinear interpolation, a high-resolution probability map of 224×224 is generated. The probability of the edge region gradually smooths out. The pixel value threshold is set to 0.5. The pixel values ​​in the first mask with a probability value greater than or equal to the set pixel threshold are set to 1, and the pixel values ​​in the first mask with a probability value less than the set pixel threshold are set to 0. The second mask is 224×224, which is a matrix containing 0 and 1.

[0098] In step 30155, the second mask is mapped onto the image coordinate system of the first image according to the target candidate region to obtain the third mask.

[0099] In some embodiments, the bounding box coordinates of the target candidate region are extracted, the spatial coverage of the bounding box coordinates in the first image is determined, each pixel of the second mask is mapped to the global coordinate system of the first image in a linear relationship, and the second mask is scaled and adjusted according to the size of the target candidate region to ensure that the second mask spatially matches the target candidate region. All pixels of the second mask are traversed, and when a pixel's value is 1, its corresponding coordinate position in the first image is marked. After traversing all pixels, a third mask aligned with the space of the first image is obtained. The third mask is used to represent the precise position of the target candidate region in the first image.

[0100] In step 30156, the first instance is obtained based on the target candidate region, the confidence level of the first instance, and the third mask.

[0101] In some embodiments, the coordinate information of the bounding box is extracted from the target candidate region to serve as the spatial location of the first instance. The third mask is used as the pixel segmentation attribute of the first instance. The spatial location, pixel segmentation attribute, and confidence score of the first instance are bound together by a data structure to form a complete first instance. The bounding box and the third mask are associated through coordinates, and the confidence score is stored independently as a scalar value.

[0102] In some embodiments, see Figure 5 , Figure 5 This is a schematic diagram of the instance segmentation results provided in the embodiments of this application; Figure 5 The image shows the results of instance segmentation of the first image, including multiple segmented first instances. These first instances do not overlap, and their categories can be people, animals, objects, or background. Each first instance includes a bounding box corresponding to a corrected candidate region and a confidence score. The spatial localization of the bounding box of the corrected candidate region corresponding to first instance 5011 reflects the coverage of the target candidate region. The bounding box includes the identified instance region, reflecting the pixel segmentation attribute of the first instance, i.e., the third mask of the first instance. If the information displayed for first instance 5011 is (person, 1.0), it means that the instance category in the bounding box of first instance 5011 is "person," and the confidence score is 1.0. If the information displayed for first instance 5012 is (bottle, 0.99), it means that the instance category in the bounding box of first instance 5012 is "bottle," and the confidence score is 0.99. If the information displayed by the first instance 5021 is (umbrella, 0.98), it means that the category of the instance in the bounding box of the first instance 5021 is umbrella, and the confidence level is 0.98; if the information displayed by the first instance 5022 is (backpack, 1.0), it means that the category of the instance in the bounding box of the first instance 5022 is backpack, and the confidence level is 1.0. If the information displayed by the first instance 5031 is (motorcycle, 1.0), it means that the category of the instance in the bounding box of the first instance 5031 is motorcycle, and the confidence level is 1.0; if the information displayed by the first instance 5032 is (motorcycle, 1.0), it means that the category of the instance in the bounding box of the first instance 5032 is motorcycle, and the confidence level is 1.0. If the information displayed by the first instance 5041 is (elephant, 1.0), it means that the category of the instance in the bounding box of the first instance 5041 is elephant, and the confidence level is 1.0; if the information displayed by the first instance 5042 is (person, 1.0), it means that the category of the instance in the bounding box of the first instance 5042 is person, and the confidence level is 1.0.

[0103] Through the embodiments of this application, low-quality candidate regions are removed by confidence threshold filtering and grouping mechanisms to reduce false detection interference. Redundant boxes are suppressed based on intersection-union ratio iteration to ensure that only the optimal candidate region is retained for the same target, thus solving the localization ambiguity caused by overlapping detection. By using a confidence maximum value allocation strategy, the highest probability prediction of the model's target existence is solidified as an instance attribute, enhancing the credibility of the results. The first mask is upsampled and converted into a high-precision binary mask to achieve pixel-level segmentation detail restoration. Through coordinate mapping and data structure binding, the target candidate region, confidence, and mask are integrated into a spatially aligned and attribute-complete first instance, providing a foundation for subsequent image generation.

[0104] See also Figure 3A In step 302, feature extraction is performed on the first instance to obtain the features of the first instance.

[0105] In some embodiments, feature extraction is performed on the first instance. The features of the first instance include feature information such as texture, edge, and shape of the first instance. Feature extraction can be achieved by convolutional neural networks or other feature extraction models. This application embodiment does not limit this.

[0106] In step 303, the noise gradient of the first instance is determined based on the features of the first instance.

[0107] In some embodiments, step 303 can be implemented by: determining the lateral gradient and longitudinal gradient of the first instance feature; determining the partial derivatives of the lateral gradient and the longitudinal gradient; determining the sum of squares of the partial derivatives of the lateral gradient and the longitudinal gradient; and determining the square root of the sum of squares as the noise gradient of the first instance.

[0108] In some embodiments, the gradient of the first instance feature map is calculated through an automatic differentiation mechanism. The corresponding differentiation function is called, and the noise gradient of the first instance feature map is calculated according to the chain rule. The chain rule is a method used in neural networks for backpropagating gradients. It is implemented by accumulating local gradients layer by layer. The gradient is backpropagated from the loss function to the input. The calculated noise gradient is a tensor with the same dimension as the first instance feature map. Each element of the tensor represents the rate of change of the first instance feature map with respect to the input at the corresponding position. The first instance feature is convolved to extract gradient information in the horizontal and vertical directions. The horizontal gradient and vertical gradient of the first instance feature map are calculated based on the gradient information. The partial derivatives of the horizontal gradient and the vertical gradient are determined. The sum of the squares of the two partial derivatives is calculated, and the square root of the sum of squares is taken as the noise gradient of the first instance feature. This can be achieved by formula (1), which is explained in detail below.

[0109]

[0110] in, It is the noise gradient of the first instance feature. It is the first instance feature. It is the partial derivative of the lateral gradient of the first instance feature in the horizontal direction. It is the partial derivative of the vertical gradient of the first instance feature in the vertical direction. It is the square root of the sum of the squares of the partial derivatives of the lateral gradient and the partial derivatives of the longitudinal gradient.

[0111] Through the embodiments of this application, the lateral and vertical gradients of the feature map are calculated to quantify the changes of the features in the horizontal and vertical directions, reflecting the rate of change of the features in space, and helping to identify important information such as the edges, textures, and shapes of the features. By determining the noise gradient to quantify the degree of change of the first instance features, an important basis is provided for subsequent noise injection and denoising processing.

[0112] In step 304, the noise amplitude negatively correlated with the noise gradient is obtained, and noise positively correlated with the noise amplitude is injected into the first instance to obtain the first noise image.

[0113] In some embodiments, the noise amplitude in step 304 can be obtained by: determining the average value and variance of the noise gradient; determining the sum of the average value and variance, and multiplying the reciprocal of the sum by a preset adjustment parameter to obtain the noise amplitude negatively correlated with the noise gradient.

[0114] In some embodiments, the average value and variance of the noise gradient are calculated, and the average value and variance of the noise gradient are added together to obtain the sum of the average value and variance, so as to determine the complexity corresponding to the noise gradient. The reciprocal of the sum is multiplied by a preset adjustment parameter to obtain the noise amplitude that is negatively correlated with the noise gradient. This can be achieved by formula (2), which is explained in detail below.

[0115]

[0116] Here, γ is a preset adjustment parameter used to adjust the overall scale of the noise amplitude. The selection of the preset adjustment parameter is based on balancing the degree of difference in noise amplitude between different instances. It is the first instance feature, β i It refers to the noise amplitude, which is a parameter that controls the intensity of noise injection. It is the sum of the average and variance of the noise gradient. A lower noise amplitude means that less noise is added to the first instance during the noise injection process, and the image is less disturbed during the denoising process, which is more conducive to preserving the original details of the instance.

[0117] For example, if the noise amplitude of facial features has a large mean and variance, the face is considered a complex instance. When processing complex instances, a lower noise amplitude is used to ensure that the facial texture and details remain clearly discernible after denoising. A higher noise amplitude indicates more added noise, requiring greater restoration and reconstruction during denoising. Conversely, if both the mean and variance of the noise amplitude are small, the image is considered a simple instance. For instances with simple textures (e.g., background, monochromatic areas), a higher noise amplitude can speed up the generation process and increase the diversity of the generated images.

[0118] Through the embodiments of this application, the average value and variance of the noise gradient are calculated to reflect the overall level and degree of variation of the noise gradient, which together describe the complexity of the first instance features. Obtaining the noise amplitude negatively correlated with the noise gradient and adaptively adjusting the intensity of noise injection according to the complexity of the first instance features helps to preserve the detailed information of complex instances during noise injection, accelerates the generation speed of simple instances, and increases the diversity of generated images, thereby improving the overall quality and efficiency of image generation.

[0119] In some embodiments, the first noise image in step 304 can be obtained by the following method: performing the following processing in iteration t: according to the noise ratio at time step t determined by the scheduling algorithm, injecting noise positively correlated with the noise amplitude into the noise image at time step (t-1) to obtain the noise image at time step t, where t is a positive integer variable that increments from 1, and the value of t satisfies 2≤t≤T, where T is a preset time step threshold, and the noise image at time step 1 is the first instance; when the value of t iterates to T, the noise image at time step T is taken as the first noise image.

[0120] In some embodiments, a time step is the smallest unit after dividing continuous time into discrete intervals, and each time step corresponds to a state update. According to the noise ratio of the t-th time step determined by the scheduling algorithm, the scheduling algorithm is a strategy for controlling the noise injection process. The noise ratio is dynamically adjusted according to the time step t. The scheduling algorithm can be linear scheduling, cosine scheduling, and quadratic scheduling. The embodiments of this application are not limited. Noise positively correlated with the noise amplitude is injected into the noise image of the (t-1)-th time step to obtain the noise image of the t-th time step. The noise injection is performed on each pixel of the image by adding the noise value to the original pixel value. The time step t is executed iteratively. t is a positive integer variable that increments from 1. The value of t satisfies 2≤t≤T. T is a preset time step threshold. The noise image of the 1st time step is the first instance. When the value of t is iterated to T, the noise image of the T-th time step is taken as the first noise image. The noise image of the t-th time step can be realized by formula (3), which is explained in detail below.

[0121]

[0122] in, It is the first instance x i The noisy image at time step t, It is the proportion of noise that is positively correlated with the noise amplitude at time step t. It is the first instance x i The noise image at time step (t-1), i.e., the noise image of the previous state at time step t, ∈ i This represents independently sampled Gaussian noise. When time step t reaches a preset time step threshold T, the noise image at time step t is used as the first noise image.

[0123] Through the embodiments of this application, the noise ratio is dynamically adjusted according to the scheduling algorithm at each time step t, and noise that is positively correlated with the noise amplitude is injected into the noise image of the previous time step. The noise injection process is precisely controlled, and the noise intensity is adaptively adjusted according to the feature complexity of the instance, making the entire noise injection process more efficient and accurate.

[0124] In some embodiments, the first noisy image, the first generated image, and the second image are implemented by calling a first model, see [link to relevant documentation]. Figure 3D , Figure 3D This is a schematic diagram of the fourth process of the image generation method provided in the embodiments of this application; during execution Figure 3A Before step 304, it was also executed Figure 3D Steps 3041 to 3044 in the training process are used to train the first model, which will be explained in detail below.

[0125] In step 3041, at least one second instance is obtained from the image sample.

[0126] In some embodiments, the image sample is segmented to obtain at least one second instance. The principle of instance segmentation is the same as that in step 301 above, and will not be repeated here.

[0127] In step 3042, a second generated image including the second instance is obtained through the first model.

[0128] In some embodiments, feature extraction is performed on the second instance using the first model to obtain the features of the second instance. The noise gradient of the second instance is determined based on these features, and the noise amplitude negatively correlated with the noise gradient is obtained. Noise is injected into the second instance based on the noise amplitude to obtain a noise image corresponding to the second instance. The noise image corresponding to the second instance is then denoised to obtain a second generated image including the second instance. The principles for determining the noise amplitude and obtaining the noise image are the same as those in step 304 above, and the denoising process for the second instance is the same as that in step 305 above, and will not be repeated here.

[0129] In step 3043, the second generated image is fused with the location information in the second instance to obtain the third image.

[0130] In some embodiments, the second instance includes location information. The second generated image is fused with the location information in the second instance to obtain a third image. The principle of obtaining the third image is the same as that of obtaining the second image in step 306 above, and will not be repeated here.

[0131] In step 3044, a first loss is determined based on the image sample and the third image, and a second loss is determined based on the second instance and the second generated image. The first model is then optimized using the first loss and the second loss.

[0132] In some embodiments, a first loss is determined based on the image sample and the third image. The first loss is a global loss between image features. A second loss is determined based on the second instance and the second generated image. The second loss is a pixel-level loss specific to the instance. The first model is optimized by backpropagation using the first loss and the second loss. The network parameters in the first model are updated. The above optimization process is iterated. When the iteration meets a preset number of iterations, the optimization of the first model is stopped, resulting in an optimized first model. The optimized first model is used to generate high-quality and high-precision images based on the input image. The global loss and pixel-level loss can achieve diverse optimization of the global and local features of the first model.

[0133] Through the embodiments of this application, the global image features are optimized by using a first loss and the accuracy of local instances is optimized by using a second loss. The first model is trained hierarchically in both global and local aspects, balancing the overall features and local features, improving the realism and accuracy of the images generated by the first model, and enhancing the generalization ability and noise resistance of the first model.

[0134] In some embodiments, the first loss in step 3044 can be achieved by: acquiring image sample features and third image features; determining a first difference between the image sample features and the third image features; and using the mean of the sum of squares of the first difference as the first loss.

[0135] In some embodiments, feature extraction is performed on the image sample to obtain image sample features, and feature extraction is performed on the third image to obtain third image features. The first difference between the image sample features and the third image features is determined. The mean of the sum of squares of the first difference is used as the first loss, which can be achieved by formula (4), as explained in detail below.

[0136]

[0137] in, That's the first loss. This is the third image, where x is the real image sample, φ is the extracted image feature, and φ(x) is the image sample feature. It is the third image feature. It calculates the mean square error between the features of the third image and the features of the sample image.

[0138] In some embodiments, the second loss in step 3044 can be implemented by: determining a second difference between the second instance and the pixels at the same location in the second generated image; calculating the sum of squares of the second differences of all pixels, and using the mean of the sum of squares of the second differences as the second loss.

[0139] In some embodiments, a second difference is determined between pixels at the same position in the second instance and the second generated image. The second difference is a pixel difference. There are multiple instances in the second instance and the second generated image. The sum of squares of the second differences of all pixels is calculated. The mean of the sum of squares of the second differences is used as the second loss, which can be achieved by formula (5), as explained in detail below.

[0140]

[0141] in, This is the second loss. It is the i-th instance in the second generated image, x i It is the real image of the i-th second instance. It is the sum of the mean squared errors between all instances of the second generated image and the real image of the second instance.

[0142] Through the embodiments of this application, by combining global feature alignment and instance-level pixel consistency optimization, the balance between overall structural fidelity and local detail restoration of the generated image is improved. The first loss is based on the mean square error constraint of the feature space to constrain global semantic consistency, ensuring that the generated image and the real sample are highly matched in high-level feature distribution, avoiding overall structural distortion or semantic shift. The second loss is to align the generated instance (generated image) with the local details of the real instance through pixel-level constraints, and enhance the fine reconstruction of key areas.

[0143] In step 305, the first noisy image is denoised to obtain the first generated image.

[0144] In some embodiments, a trained denoising network is used to denoise the first noisy image. The denoising network can be a symmetrical U-shaped convolutional neural network (U-Net), which is not limited in this embodiment. The first noisy image is input into the denoising network, which includes an encoder-decoder architecture. The encoder is responsible for extracting features from the first noisy image, and the decoder gradually reconstructs a clear first generated image based on the features of the first noisy image. The encoder reduces the resolution of the first noisy image by performing convolutional layers and downsampling operations, and extracts features from the first noisy image. In each layer of the decoder, the feature maps of the corresponding layers of the encoder are connected to reintroduce the detailed information extracted by the encoder into the decoder, helping to reconstruct the image more accurately. At the output layer of the decoder, a denoised image is output. The denoising process is a gradual refinement process that requires multiple iterations. In each iteration, a denoised image is output based on the current noisy image. The denoising intensity corresponds to the amplitude of the added noise. When the number of iterations reaches a preset threshold or the loss function converges, the denoising process ends, and the first generated image is obtained.

[0145] In some embodiments, see Figure 6 , Figure 6 This is a schematic diagram of the noise state changes in the image; the first noisy image 601 is a pure noise image obtained after noise iterative injection over time step T, X T The first noisy image 601 can be obtained through step 304 described above. A Markov chain-based reverse process is then performed on the first noisy image 601 to generate the first generated image 603. t-1 Let X0 represent the noise image 602 of the intermediate state at time step t-1, and let X0 represent the first generated image 603. θ (x t-1 |x t ) is the probability distribution of the reverse process, representing the probability of generating the image state at time step t-1 from the image state at time step t. t |x t-1 The probability distribution p in the forward process represents the probability of generating the image state at time step t from the image state at time step t-1 by injecting noise, corresponding to the noise injection process in step 304 above. The reverse process starts from the initial first noisy image 601, and by gradually removing noise, finally generates the first generated image 603, which is determined by a predefined noise distribution p. θ The noise distribution of the controlled and denoised noise corresponds to the noise distribution of the injected noise, and is consistent with the noise injected in step 304 above.

[0146] In some embodiments, see Figure 7 , Figure 7This is a schematic diagram of the denoising model provided in this application embodiment. A first noisy image 701 is input into the denoising network. The encoder 702 performs a series of convolutional layers and downsampling operations, gradually reducing the resolution of the first noisy image 701 and extracting features such as texture, edges, and shape. The innermost layer of the encoder 702 contains the most abstract representation of the first noisy image 701. Each layer of the decoder 703 is connected to the feature map of the corresponding layer in the encoder 702, reintroducing the detailed information extracted by the encoder 702 into the decoder 703 to help reconstruct the image more accurately. The output layer of the decoder 703 outputs a denoised image. The denoising process is a progressive refinement process requiring multiple iterations. In each iteration, a denoised image is output based on the current noisy image. The denoising intensity corresponds to the magnitude of the added noise. When the number of iterations reaches a preset threshold or the loss function converges, the denoising process ends, and the resulting first generated image 704 is the final result after denoising the first noisy image 701.

[0147] In step 306, the first generated image and the position information in the first instance are fused to obtain a second image containing the first instance.

[0148] In some embodiments, the location information of a first instance is obtained, which determines a region of the same size as the input image, where the pixel value of the region containing the first instance is 1, and the pixel value of the background region is 0, i.e., the mask corresponding to the first instance is obtained. An interpolation method is used to adjust the first generated image to the same size as the first instance mask to ensure spatial alignment between the first generated image and the mask. The resized first generated image is then placed pixel by pixel in the second image at the position indicated by the mask. Each pixel value in the mask is traversed, and the pixel values ​​in the first generated image are copied to the same position in the second image. A smoothing algorithm is used to achieve a transition between the first generated image and the surrounding background or adjacent first instances. Smoothing algorithms are mathematical methods used to eliminate local abrupt changes in an image and achieve natural transitions, making the fusion boundary visually continuous and natural while preserving the details of the generated region. By using image fusion technology, while preserving the details of the first generated image, the boundary region is made consistent with the surrounding environment in terms of lighting, color, etc. Image fusion technology is the process of merging multiple images (or image regions) from the same scene or different sources into a high-quality, more information-rich image. Through direct overlay, weighted fusion, Poisson fusion or a combination of above, the first generated image is fused according to the mask to obtain a second image that is optimized and restored in the first instance region.

[0149] The image production method provided by the embodiments of this application has the following beneficial effects:

[0150] Global feature maps integrating semantic and spatial information are extracted and combined with a region proposal network and multi-scale anchors to generate candidate regions covering different targets. Local feature maps are aligned using bilinear interpolation to achieve standardized representations. Candidate region positions are accurately corrected based on class prediction and bounding box regression. Non-maximum suppression is used to select high-confidence first instances, covering targets at different scales and reducing redundant candidate regions, thus improving candidate region recall and localization efficiency, and enhancing localization accuracy. Noise amplitude is calculated based on feature gradients, and the noise injection intensity is dynamically adjusted. Low noise is used to preserve details for complex instances, while high noise is used to accelerate generation for simple regions, reducing unnecessary computation and balancing detail accuracy and processing efficiency. Images are gradually reconstructed through a denoising network, ensuring that the denoising process restores clear structure while preserving edges and textures. A first loss constraint based on global features is used to constrain the high-level semantic consistency between the generated image and the samples. A second loss at the instance level pixel level is used to enhance local detail alignment, simultaneously optimizing overall structure and local accuracy, improving the visual realism of the generated image. The generated image is fused with the original image using a positional mask, and a smoothing algorithm is used to process the transition between instances and surrounding regions, resulting in an optimized high-quality image that ensures consistency in style and structure among different instances in the optimized image.

[0151] The following description continues to illustrate the exemplary structure of the image generation apparatus 455 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules stored in the image generation device 455 of the memory 450 may include: an image segmentation module 4551, used to acquire at least one first instance in a first image; an image generation module 4552, used to extract features from the first instance to obtain first instance features; determine the noise gradient of the first instance based on the first instance features; acquire the noise amplitude negatively correlated with the noise gradient, inject noise positively correlated with the noise amplitude into the first instance to obtain a first noise image; perform denoising processing on the first noise image to obtain a first generated image; and perform fusion processing on the first generated image and the position information in the first instance to obtain a second image including the first instance.

[0152] In some embodiments, the image segmentation module 4551 is further configured to: acquire a global feature map of the first image; identify at least one candidate region from the global feature map; determine a local feature map corresponding to the candidate region in the global feature map; perform category prediction based on the local feature map to obtain the confidence level of the candidate region; perform regression based on the local feature map to obtain a position offset; correct the position of the candidate region based on the position offset to obtain a corrected candidate region; and determine a first instance based on the corrected candidate region and the confidence level.

[0153] In some embodiments, the image segmentation module 4551 is further configured to remove corrected candidate regions that are below a preset confidence threshold, divide the remaining corrected candidate regions after removal into multiple groups, wherein the confidence of the corrected candidate regions in a group is within the confidence interval corresponding to the group; for each group, determine the target candidate region in the group; take the maximum confidence value in the target candidate region as the confidence value of the first instance; obtain a first mask of the target candidate region, perform upsampling processing on the first mask, binarize the upsampled first mask to obtain a second mask; map the second mask onto the image coordinate system of the first image according to the target candidate region to obtain a third mask; and obtain the first instance according to the target candidate region, the confidence value of the first instance, and the third mask.

[0154] In some embodiments, the image segmentation module 4551 is further configured to perform the following processing for each group: determine the intersection-union ratio (IU / R) of any two corrected candidate regions in the group; when the IU / R exceeds a preset IU / R threshold, retain the corrected candidate region with higher confidence; and iteratively perform the following processing: determine the IU / R of the retained corrected candidate region with any remaining corrected candidate region in the group; when the IU / R exceeds a preset IU / R threshold, retain the corrected candidate region with higher confidence; and when the iteration ends, use the finally retained corrected candidate region as the target candidate region in the group.

[0155] In some embodiments, the image generation module 4552 is further configured to determine the average value and variance of the noise gradient; determine the sum of the average value and variance; and multiply the reciprocal of the sum by a preset adjustment parameter to obtain the noise amplitude negatively correlated with the noise gradient.

[0156] In some embodiments, the image generation module 4552 is further configured to iterate t and perform the following processing: according to the noise ratio at time step t determined by the scheduling algorithm, inject noise positively correlated with the noise amplitude into the noise image at time step (t-1) to obtain the noise image at time step t, where t is a positive integer variable that increments from 1, and the value of t satisfies 2≤t≤T, where T is a preset time step threshold, and the noise image at time step 1 is the first instance; when the value of t iterates to T, the noise image at time step T is used as the first noise image.

[0157] In some embodiments, the image generation module 4552 is further configured to determine the lateral gradient and the longitudinal gradient of the first instance features; determine the partial derivatives of the lateral gradient and the longitudinal gradient; determine the sum of squares of the partial derivatives of the lateral gradient and the longitudinal gradient; and determine the square root of the sum of squares as the noise gradient of the first instance.

[0158] In some embodiments, the first noisy image, the first generated image, and the second image are implemented by calling a first model. The model training module 4553 is further configured to obtain at least one second instance in the image samples; obtain a second generated image including the second instance through the first model; fuse the second generated image with the position information in the second instance to obtain a third image; determine a first loss based on the image samples and the third image, and determine a second loss based on the second instance and the second generated image; and optimize the first model through the first loss and the second loss.

[0159] In some embodiments, the model training module 4553 is further configured to acquire image sample features and third image features; determine a first difference between the image sample features and the third image features; use the mean of the sum of squares of the first difference as a first loss; determine a second difference between the second instance and pixels at the same position in the second generated image; calculate the sum of squares of the second differences of all pixels, and use the mean of the sum of squares of the second difference as a second loss.

[0160] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the image generation method described in this application.

[0161] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the image generation method provided in this application. For example, ... Figure 3A The image generation method is shown.

[0162] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0163] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0164] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0165] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0166] In summary, this application embodiment independently processes the first instance in the first image, determines the noise gradient of the first instance based on its features, and designs different noise injection strategies for different instances. It obtains the noise amplitude negatively correlated with the noise gradient and adjusts the noise injection amount according to the noise amplitude of each first instance, improving noise addition efficiency and helping to preserve the original detailed information of the first instance. The denoising process corresponds to the noise amplitude in the denoising process, reducing the interference of global noise during denoising and improving the accuracy and generation speed of the first generated image. It aligns the positional information of the first generated image and the first instance, restoring the blurred boundaries of the first instance in the first generated image, avoiding misalignment in the second image, improving the image generation quality, and ensuring the consistency of style and structure of different instances in the second image.

[0167] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An image generation method, characterized in that, The method includes: Obtain at least one first instance from the first image; Feature extraction is performed on the first instance to obtain the features of the first instance; Determine the lateral and longitudinal gradients of the features of the first instance; Determine the partial derivatives of the lateral gradient and the longitudinal gradient, and determine the sum of the squares of the partial derivatives of the lateral gradient and the longitudinal gradient. Determine the square root of the sum of squares as the noise gradient of the first instance; Determine the average value and variance of the noise gradient; The sum of the average value and the variance is determined, and the reciprocal of the sum is multiplied by a preset adjustment parameter to obtain the noise amplitude that is negatively correlated with the noise gradient. Iteration t performs the following processing: Based on the noise ratio determined by the scheduling algorithm at time step t, noise positively correlated with the noise amplitude is injected into the noise image at time step (t-1), resulting in the noise image at time step t. Here, t is a positive integer variable incrementing from 1, and the value of t satisfies... T is a preset time step threshold, and the noise image at the first time step is the first instance; When the value of t iterates to T, the noise image at time step T is taken as the first noise image; The first noisy image is denoised to obtain the first generated image; The first generated image and the location information in the first instance are fused to obtain a second image that includes the first instance.

2. The method according to claim 1, characterized in that, The acquisition of at least one first instance of the first image includes: Obtain the global feature map of the first image; At least one candidate region is identified from the global feature map; Determine the local feature map corresponding to the candidate region in the global feature map; Based on the local feature map, category prediction is performed to obtain the confidence level of the candidate region, and regression is performed based on the local feature map to obtain the position offset. Based on the position offset, the position of the candidate region is corrected to obtain the corrected candidate region. The first instance is determined based on the corrected candidate region and the confidence level.

3. The method according to claim 2, characterized in that, The step of determining the first instance based on the corrected candidate region and the confidence level includes: Remove the corrected candidate regions that are below a preset confidence threshold, and divide the remaining corrected candidate regions after the removal into multiple groups. The confidence of the corrected candidate regions in a group is within the confidence interval corresponding to that group. For each group, a target candidate region is determined within that group; The maximum confidence level of the target candidate region is taken as the confidence level of the first instance; A first mask of the target candidate region is obtained, the first mask is upsampled, and the upsampled first mask is binarized to obtain a second mask. The second mask is mapped onto the image coordinate system of the first image based on the target candidate region to obtain the third mask; The first instance is obtained based on the target candidate region, the confidence level of the first instance, and the third mask.

4. The method according to claim 3, characterized in that, Determining the target candidate region in the group includes: For each group, the following process is performed: determine the intersection-union ratio (CIU) of any two corrected candidate regions in the group; when the CIU exceeds a preset CIU threshold, retain the corrected candidate region with higher confidence; and iteratively perform the following process: The cross-union ratio (CUN) of the retained modified candidate region with any remaining modified candidate region in the group is determined. When the CUN exceeds the preset CUN threshold, the modified candidate region with higher confidence is retained. When the iteration ends, the finally retained modified candidate region is taken as the target candidate region in the group.

5. The method according to claim 1, characterized in that, The first noisy image, the first generated image, and the second image are implemented by calling a first model. Before the following processing is performed in iteration t, the method further includes: Obtain at least one second instance from the image sample; A second generated image, including the second instance, is obtained through the first model; The second generated image is fused with the location information in the second instance to obtain the third image; A first loss is determined based on the image sample and the third image, and a second loss is determined based on the second instance and the second generated image. The first model is then optimized using the first loss and the second loss.

6. The method according to claim 5, characterized in that, The step of determining the first loss based on the image sample and the third image includes: Obtain image sample features and third image features; Determine the first difference between the image sample features and the third image features; The mean of the sum of squares of the first difference is taken as the first loss; The step of determining the second loss based on the second instance and the second generated image includes: Determine a second difference between the pixels at the same location in the second instance and the second generated image; Calculate the sum of squares of the second difference for all pixels, and use the mean of the sum of squares of the second difference as the second loss.

7. An image generation apparatus, characterized in that, The device includes: The image segmentation module is used to obtain at least one first instance in the first image; An image generation module is used to extract features from the first instance to obtain first instance features; determine the horizontal and vertical gradients of the first instance features; determine the partial derivatives of the horizontal and vertical gradients, and determine the sum of squares of the partial derivatives of the horizontal and vertical gradients; determine the square root of the sum of squares as the noise gradient of the first instance; determine the average and variance of the noise gradient; determine the sum of the average and variance, and multiply the reciprocal of the sum by a preset adjustment parameter to obtain the noise amplitude negatively correlated with the noise gradient; iterate t and perform the following processing: according to the noise ratio at time step t determined by the scheduling algorithm, inject noise positively correlated with the noise amplitude into the noise image at time step (t-1) to obtain the noise image at time step t, where t is a positive integer variable that increments from 1, and the value of t satisfies T is a preset time step threshold, and the noise image at the first time step is the first instance; when the value of t iterates to T, the noise image at the Tth time step is taken as the first noise image; the first noise image is denoised to obtain the first generated image; the first generated image is fused with the position information in the first instance to obtain the second image including the first instance.

8. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, configured to execute computer-executable instructions or computer programs stored in the memory, implements the image generation method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the image generation method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the image generation method according to any one of claims 1 to 6 is implemented.

Images