Image generation method and apparatus, image processing model training method and apparatus, image processing method and apparatus, and electronic device and storage medium

Abstract

Provided in the present disclosure are an image generation method and apparatus, an image processing model training method and apparatus, an image processing method and apparatus, and an electronic device and a storage medium, which can be applied to the fields of computer technologies, deep-learning technologies and image processing technologies. The image generation method comprises: on the basis of an initial noise image and an image category, determining a first noise image; on the basis of the initial noise image and the first noise image, determining a first intermediate image; and on the basis of the first intermediate image, generating a first target image that belongs to the image category.

Description

Image generation, image processing model training, image processing method and device, electronic device and storage medium Technical Field

[0001] The present disclosure relates to the fields of computer technology, deep learning technology, and image processing technology, and more specifically, to image generation, image processing model training, image processing methods and devices, electronic equipment, computer-readable storage media, and computer program products. Background Art

[0002] With the development of computer technology, diffusion models (DMs) have emerged. Diffusion model operations can include a forward process and a reverse process. The forward process can refer to the noise addition process, and the reverse process can refer to the denoising process. In one example, the original image can be repeatedly denoised and denoised based on the diffusion model and a text description (prompt) to generate a sample image. However, because the above image generation process requires a text description, it consumes a lot of resources and time.

[0003] Summary of the Invention

[0004] In view of this, the present disclosure provides an image generation, image processing model training, image processing method and device, electronic device, computer-readable storage medium and computer program product.

[0005] According to one aspect of the present disclosure, an image generation method is provided, comprising: determining a first noise image based on an initial noise image and an image category; determining a first intermediate image based on the initial noise image and the first noise image; and generating a first target image belonging to the image category based on the first intermediate image.

[0006] According to another aspect of the present disclosure, a training method for an image processing model is provided, comprising: inputting at least one of a first target image and a second target image into a second deep learning model to obtain output information; determining a loss function value based on the output information and the image category of at least one of the first target image and the second target image; and adjusting model parameters of the second deep learning model based on the loss function value to obtain an image processing model; wherein at least one of the first target image and the second target image is generated using an image generation method.

[0007] According to another aspect of the present disclosure, an image processing method is provided, comprising: acquiring an image to be processed; and inputting the image to be processed into an image processing model to obtain an image processing result; wherein the image processing model is trained using an image processing model training method.

[0008] According to another aspect of the present disclosure, an image generating device is provided, comprising: a first determination module for determining a first noise image based on an initial noise image and an image category; a second determination module for determining a first intermediate image based on the initial noise image and the first noise image; and a generation module for generating a first target image belonging to the image category based on the first intermediate image.

[0009] According to another aspect of the present disclosure, a training device for an image processing model is provided, comprising: a first input module for inputting at least one of a first target image and a second target image into a deep learning model to obtain output information; a third determination module for determining a loss function value based on the output information and the image category of at least one of the first target image and the second target image; and an adjustment module for adjusting model parameters of the deep learning model based on the loss function value to obtain an image processing model; wherein at least one of the first target image and the second target image is generated using an image generating device.

[0010] According to another aspect of the present disclosure, an image processing device is provided, comprising: an acquisition module for acquiring an image to be processed; and a second input module for inputting the image to be processed into an image processing model to obtain an image processing result; wherein the image processing model is trained using an image processing model training device.

[0011] According to another aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors implement the method described in the present disclosure.

[0012] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which executable instructions are stored. When the executable instructions are executed by a processor, the processor implements the method described in the present disclosure.

[0013] According to another aspect of the present disclosure, a computer program product is provided. The computer program product includes computer-executable instructions. When the computer-executable instructions are executed, they are used to implement the method described in the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0015] FIG1 schematically illustrates a system architecture to which an image generation method, an image processing model training method, and an image processing method according to an embodiment of the present disclosure can be applied;

[0016] FIG2 schematically shows a flow chart of an image generation method according to an embodiment of the present disclosure;

[0017] FIG3 schematically shows an example of a process of obtaining a first noise image according to an embodiment of the present disclosure;

[0018] FIG4 schematically illustrates an example diagram of a training process of a trained deep learning model according to an embodiment of the present disclosure;

[0019] FIG5 schematically shows an example diagram of a process for obtaining an intermediate deep learning model according to an embodiment of the present disclosure;

[0020] FIG6 schematically shows an example diagram of a process for obtaining a sample fusion vector according to an embodiment of the present disclosure;

[0021] FIG7 schematically shows an example diagram of an image generation process according to an embodiment of the present disclosure;

[0022] FIG8 schematically shows an example schematic diagram of obtaining a first target image according to an embodiment of the present disclosure;

[0023] FIG9 schematically shows an example of a process of obtaining a second noise image according to an embodiment of the present disclosure;

[0024] FIG10 schematically shows an example of a process of obtaining a masked noise image according to an embodiment of the present disclosure;

[0025] FIG11 schematically shows an example diagram of an image generation process according to another embodiment of the present disclosure;

[0026] FIG12 schematically shows an example of a process of obtaining a second target image according to an embodiment of the present disclosure;

[0027] FIG13 schematically shows a flow chart of a method for training an image processing model according to an embodiment of the present disclosure;

[0028] FIG14 schematically shows a flow chart of an image processing method according to an embodiment of the present disclosure;

[0029] FIG15 schematically shows a block diagram of an image generating apparatus according to an embodiment of the present disclosure;

[0030] FIG16 schematically shows a block diagram of a training apparatus for an image processing model according to an embodiment of the present disclosure;

[0031] FIG17 schematically shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure; and

[0032] FIG18 schematically shows a block diagram of an electronic device suitable for implementing an image generation method, an image processing model training method, and an image processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0034] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0035] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0036] When expressions such as "at least one of A, B and C, etc." are used, they should generally be interpreted in accordance with the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0037] In the technical solution of the present invention, the user information involved (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0038] In scenarios where personal information is used for automated decision-making, the methods, devices, and systems provided by embodiments of the present invention provide users with corresponding operational portals, allowing them to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The term "automated decision-making" herein refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, or credit status through computer programs and making decisions. The term "expert decision-making" herein refers to the activity of decision-making by individuals who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0039] In one example, an image can be generated based on a diffusion model and a text description (prompt). For example, the text description can be input into the diffusion model to obtain features on the image corresponding to the text description, and then the features can be fed into the diffusion model to generate the image.

[0040] However, since the features corresponding to the text description in the above process are equivalent to conditional guidance in the diffusion model, that is, the use of the diffusion model needs to rely on the text description, and the training of the diffusion model requires more hardware equipment and large-scale data sets with text descriptions and images, and these data sets need to be finely labeled, resulting in high resource and time costs.

[0041] To this end, the present disclosure provides an image generation, image processing model training, image processing method and apparatus, electronic device, and storage medium, which can be applied in the fields of computer technology, deep learning technology, and image processing technology. The image generation method includes: determining a first noise image based on an initial noise image and an image category; determining a first intermediate image based on the initial noise image and the first noise image; and generating a first target image belonging to the image category based on the first intermediate image.

[0042] Figure 1 schematically illustrates a system architecture to which an image generation method, an image processing model training method, and an image processing method according to an embodiment of the present disclosure can be applied. It should be noted that Figure 1 is merely an example of a system architecture to which an embodiment of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiment of the present disclosure cannot be used in other devices, systems, environments, or scenarios.

[0043] As shown in FIG1 , the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0044] The user may use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0045] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0046] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received user requests and other data, and feed back processing results (e.g., web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0047] It should be noted that the training method of the image processing model provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the training device of the image processing model provided in the embodiment of the present disclosure can generally be set in the server 105. The training method of the image processing model provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the training device of the image processing model provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0048] Alternatively, the training method of the image processing model provided in the embodiment of the present disclosure may also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may also be executed by another terminal device different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the training apparatus of the image processing model provided in the embodiment of the present disclosure may also be provided in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may be provided in another terminal device different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0049] It should be noted that the image generation method and image processing method provided in the embodiments of the present disclosure can generally be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or can also be executed by a terminal device other than the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the image generation device and image processing device provided in the embodiments of the present disclosure can also be provided in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or can be provided in a terminal device other than the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0050] Alternatively, the image generation method and image processing method provided in the embodiments of the present disclosure may also be executed by the server 105. Accordingly, the image generation device and image processing device provided in the embodiments of the present disclosure may generally be provided in the server 105. The image generation method and image processing method provided in the embodiments of the present disclosure may also be performed by a server or server cluster that is different from the server 105 and that is capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the image generation device and image processing device provided in the embodiments of the present disclosure may also be provided in a server or server cluster that is different from the server 105 and that is capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0051] It should be understood that the number of first terminal devices, second terminal devices, third terminal devices, networks and servers in Figure 1 is merely illustrative and any number of first terminal devices, second terminal devices, third terminal devices, networks and servers may be provided as required.

[0052] It should be noted that the sequence numbers of the operations in the following method are only used to indicate the operation for the purpose of description, and should not be regarded as indicating the order in which the operations should be performed. Unless explicitly stated, the method does not need to be performed in the order shown.

[0053] FIG2 schematically shows a flow chart of an image generating method according to an embodiment of the present disclosure.

[0054] As shown in FIG. 2 , the image generating method includes operations S210 to S230 .

[0055] In operation S210, a first noise image is determined according to an initial noise image and an image category.

[0056] In operation S220, a first intermediate image is determined based on the initial noise image and the first noise image.

[0057] In operation S230, a first target image belonging to an image category is generated based on the first intermediate image.

[0058] The initial noise image can be determined based on the original image and random noise. The original image can refer to a small sample image of any desired target scene, which can be understood as an image with a small number of similar samples. After obtaining the original image, a random number generator or statistical distribution model can be used to generate random noise of the same size as the original image, and this random noise can be added to the original image. For example, the initial noise image can be generated by adding irregular points or by adding random noise to each pixel value in the original image through an addition operation.

[0059] The type of random noise can be configured based on actual business needs and is not limited here. For example, random noise can include at least one of the following: Gaussian noise and salt and pepper noise. Gaussian noise refers to noise embedded within the original image, whose noise density function follows a Gaussian distribution. The degree of noise can be controlled by adjusting the standard deviation. Salt and pepper noise can refer to noise overlaid on the front of the original image, where "pepper" can refer to black noise and "salt" can refer to white noise. By controlling the noise ratio, the degree of image damage can be set.

[0060] Image categories can be used to characterize the category of the original image, which can be determined based on the target scene and task requirements. Target scenes can include natural scenes, industrial scenes, medical scenes, cultural relics scenes, film and television scenes, transportation scenes, and astronomical scenes. For example, if the target scene is a natural scene, image categories can include mountains, oceans, forests, and lakes. Alternatively, if the target scene is a transportation scene, image categories can include cars, airplanes, trains, and ships.

[0061] In one example, when the target scene is an industrial scene, image categories can be used to characterize different targets in defect detection, that is, image categories can include water droplets, blisters, bulges, scratches and good products, etc. In one example, different targets can be compared and assigned values ​​in a preset manner to facilitate label encoding. The preset method can be configured according to actual business needs and is not limited here. For example, the preset method can be to assign different targets in a "0-n" comparison, that is, the image category corresponding to water droplets can be "0-1", the image category corresponding to blisters can be "0-2", the image category corresponding to bulges can be "0-3", the image category corresponding to scratches can be "0-4", the image category corresponding to good products can be "0-5", and so on.

[0062] After obtaining the initial noise image and image category, the trained deep learning model can be used to process the initial noise image and image category to obtain a first noise image. The first noise image can refer to a noise image predicted based on the initial noise image with random noise added and the desired image category. For example, the trained deep learning model can be used to perform forward processing on the initial noise image and image category to obtain the first noise image.

[0063] After obtaining the first noisy image, the trained deep learning model can be used to process the initial noisy image and the first noisy image to obtain a first intermediate image. The first intermediate image can be an image obtained by denoising the initial noisy image to which random noise has been added and the predicted first noisy image. For example, the trained deep learning model can be used to perform backward processing on the initial noisy image and the first noisy image to obtain the first intermediate image.

[0064] The trained deep learning model can be configured based on actual business needs and is not limited here. For example, the trained deep learning model can include at least one of the following: a heat diffusion model, a nonlinear diffusion model, an isotropic diffusion model, and an anisotropic diffusion model.

[0065] After obtaining the first intermediate image, a post-processing operation can be performed on the first intermediate image to obtain a first target image. The first target image may refer to an image belonging to an image category obtained through the post-processing operation. The post-processing operation can be configured based on actual business needs and is not limited here. For example, the post-processing operation may include at least one of the following: image scaling, image transformation, image normalization, image enhancement, texture analysis, and image segmentation.

[0066] According to the embodiments of the present disclosure, since the first target image is generated based on the first intermediate image, the first intermediate image is determined based on the initial noise image and the first noise image, and the first noise image is determined based on the initial noise image and the image category, it is possible to learn the essential features of the original image on the original image with a small data volume to increase the data volume of the first target image, and it is also possible to ensure that the generated first target image conforms to the specified image category, thereby improving the accuracy and pertinence of the generation of the first target image, thereby helping to improve the implementation effect of subsequent image processing tasks.

[0067] The image generation method 200 according to the embodiment of the present invention will be further described below with reference to FIG. 3 to FIG. 12 .

[0068] In one example, images can be classified according to specific application fields or acquisition sources. The initial noisy images include images of at least one of the following application scenarios: industrial images, medical images, cultural relic images, film and television images, traffic images, and astronomical images.

[0069] For example, industrial images may refer to images related to industrial production, which can be used for automated production and quality inspection. Alternatively, medical images may refer to images used for medical diagnosis and treatment, which can be used to help doctors observe lesions, diagnose diseases, and develop treatment plans. Alternatively, cultural relic images may refer to images used to record, protect, and study cultural relics, which can be used for cultural relic protection or cultural heritage research. Alternatively, film and television images may refer to images in film, television series, animation, and other film and television works, which can refer to images shot from real scenes and synthesized with special effects. Alternatively, traffic images may refer to images of traffic scenes captured by road surveillance cameras, which can refer to images used for traffic management, vehicle tracking, or accident investigation. Alternatively, astronomical images may refer to images used to record and study astrophysical phenomena, which can refer to images used for astronomical science and space exploration.

[0070] According to the embodiments of the present disclosure, since the image category can be set according to the first target image to be generated, the generated first target image can meet the needs of different application fields, thereby improving the applicability of image generation.

[0071] In the embodiment of the present disclosure, the first noise image in the image generation method may also be generated based on the original image, random noise, and image category. The process of obtaining the first noise image will be further described below with reference to FIG. 3 .

[0072] FIG3 schematically shows an example diagram of a process of obtaining a first noise image according to an embodiment of the present disclosure.

[0073] As shown in FIG3 , in 300 , after obtaining an original image 301, the original image 301 can be processed based on random noise 302 to obtain an initial noisy image 303. Based on this, the initial noisy image 303 and the image category 304 can be input into a trained deep learning model 305 to obtain a first noisy image 306.

[0074] In one example, to ensure that the input and output scales are the same, the first deep learning model can be a UNet network. The original image 301, random noise 302, initial noise image 303, and first noise image 306 can have the same dimensions. The dimension of an image can refer to the dimension of the image when it is represented in a computer. For example, a grayscale image may include one channel, which is used to represent the brightness information of the image. Alternatively, a color image may include three channels of red, green, and blue, each channel being used to represent the pixel values ​​of the red, green, and blue channels at the corresponding positions contained therein. For example, if the original image 301 is of (3, 64, 64) dimensions and the random noise 302 is also of (3, 64, 64) dimensions, then the initial noise image 303 and the first noise image 306 are both of (3, 64, 64) dimensions.

[0075] Because the denoising process requires a trained deep learning model to fit the noise, the trained deep learning model must be able to predict the noise added to the image based on the current input image and the real-time step number t, so that the denoised image can be closer to the image of the specified image category. Therefore, in addition to guiding by the real-time step number t, a guidance method based on image category is also introduced. The real-time step number t can be used to guide the time step number and the parameter setting of the first deep learning model.

[0076] According to an embodiment of the present disclosure, an intermediate deep learning model 303 with specified style features is obtained by training a first training sample set 301 with a larger amount of data and the corresponding sample image category, and then a second training sample set 304 with a smaller amount of data is used to fine-tune the intermediate deep learning model 303 with the same style features, and guided by the image category, a final trained deep learning model 305 is obtained. When the trained deep learning model 305 is subsequently used, the image category and the random input of the specified size can be specified to generate a first target image of the same size and belonging to the same image category, thereby achieving unlimited expansion of small sample data, which is beneficial for dealing with scenarios where data is difficult to obtain. In addition, since the required hardware resource configuration is relatively low, the problem of small sample data volume can be solved at a lower cost. For example, image generation model training and reasoning can be achieved with 12G of video memory.

[0077] It is understandable that in order to obtain the first noise image, the first deep learning model can be trained in advance to obtain a trained deep learning model. This will be further explained below with reference to FIG4 .

[0078] FIG4 schematically shows an example diagram of the training process of the first deep learning model according to an embodiment of the present disclosure.

[0079] As shown in FIG4 , in 400 , a first deep learning model 402 can be trained using a first training sample set 401 to obtain an intermediate deep learning model 403. The multiple first training samples included in the first training sample set 401 can each have a sample image category. The above image category can be the same as at least one of the multiple sample image categories. For example, when the target scene is an industrial scene, the image category can be water droplets, and the multiple sample image categories can include water droplets, blisters, bulges, scratches, and good products. The first training sample set 401 can refer to sample data with a large amount of data. The intermediate deep learning model 403 trained using the existing, sufficient sample data can be used as a basic model. The intermediate deep learning model 303 can express relatively rich features of the same application scenario.

[0080] After obtaining the intermediate deep learning model 403, the intermediate deep learning model 403 can be fine-tuned using the second training sample set 404 to obtain a trained deep learning model 405. The multiple second training samples included in the second training sample set 404 can have the same image category as the first target image. The second training sample set 404 can refer to a small amount of sample data, for example, 10 sample images. By loading the trained base model and re-performing the above training steps, a fine-tuned trained deep learning model is obtained, which enables the essential characteristics of the samples to be learned from the small second training sample set. The trained deep learning model can then be used to perform data inference and image generation for the image category.

[0081] For example, using industrial defect data as an example, a first training sample set 401, which contains sufficient defect data, can be used to train a basic intermediate deep learning model 403, enabling it to generate categories based on image categories. Furthermore, a second training sample set 404, with less data, can be used to fine-tune this intermediate deep learning model 403, allowing it to learn the characteristics of this type of data, resulting in a trained deep learning model 405. Subsequently, by inputting features of a specified size into this trained deep learning model 405, countless first target images of the same image type can be generated.

[0082] It can be understood that the above describes how to train the first deep learning model to obtain a trained deep learning model. The following will further describe how to train the intermediate deep learning model of the present disclosure in conjunction with Figure 5.

[0083] FIG5 schematically shows an example diagram of a process for obtaining an intermediate deep learning model according to an embodiment of the present disclosure.

[0084] As shown in FIG. 5 , in 500 , the first training sample set may include a plurality of first training samples 510 , and each first training sample 510 may have a sample image category 501 .

[0085] In one example, the number of sample image categories 501 and the dimension of the iteration step number 503 are represented as a mapping, and feature extraction processing is performed on the sample image category 501 to obtain a sample image category vector 502 with an output dimension of 1*the iteration step number 503. Feature extraction processing is performed on the iteration step number 503 to obtain an iteration step vector 504 with an output dimension of 1*the iteration step number 503. The feature extraction processing method can be configured according to actual business needs and is not limited here. For example, the feature extraction processing method can include at least one of the following: feature extraction based on deep learning, local feature extraction, and feature extraction based on an attention mechanism.

[0086] After obtaining multiple sample image category vectors 502 and iteration step vectors 504, multiple sample fusion vectors 505 can be determined based on the multiple sample image category vectors 502 and iteration step vectors 504. By fusing the sample image category vectors 502 and the iteration step vector 504, and inputting them together with the first training sample 510 into the first deep learning model for processing, that is, different sample image categories 501 enter the first deep learning model for training along with the first training sample 510, guiding the first deep learning model on how to output the most correct sample noise image 511 under certain sample image categories. The vector fusion processing method can be configured according to actual business needs and is not limited here. For example, the vector fusion processing method can include at least one of the following: concatenation, addition, weighted average, dimensionality reduction after concatenation, and neural network fusion, etc.

[0087] According to an embodiment of the present disclosure, by performing feature extraction processing on multiple sample image categories and iteration steps respectively, it is helpful to extract key features that characterize different sample categories and iteration steps, and provide richer information for subsequent model training. By combining multiple sample image category vectors and iteration step vectors into multiple sample fusion vectors, the diversity and comprehensiveness of features can be improved, thereby enhancing the representation and generalization capabilities of the intermediate deep learning model. On this basis, by utilizing multiple first training samples and multiple sample fusion vectors, the first deep learning model is trained, so that the first deep learning model can fully consider the information of different sample image categories and iteration steps during the training process, thereby optimizing the learning process of the first deep learning model and improving the performance of the intermediate deep learning model on specific tasks.

[0088] In one example, after obtaining the plurality of sample fusion vectors 505, the first deep learning model can be trained using the plurality of first training samples 510 and the plurality of sample fusion vectors 505 to obtain an intermediate deep learning model 512. The first deep learning model can include a convolutional layer 506, a plurality of first cascaded levels 507, an intermediate level 508, and a plurality of second cascaded levels 509.

[0089] For example, the plurality of first cascade levels 507 may include a first cascade level 507_1, a first cascade level 507_2, a first cascade level 507_3, and a first cascade level 507_4, each of which may be an encoder. The intermediate level 508 may be a skip connection, for directly transmitting the output feature map of a layer of the encoder to the corresponding layer of the decoder. The plurality of second cascade levels 509 may include a second cascade level 509_1, a second cascade level 509_2, a second cascade level 509_3, and a second cascade level 509_4, each of which may be a decoder.

[0090] For each first training sample 510, the first training sample 510 can be input into a convolutional layer 506 to obtain a first intermediate vector. The first intermediate vector and the sample fusion vector 505 corresponding to the first training sample 510 are processed through multiple first cascade levels 507 to obtain first intermediate feature maps corresponding to each of the multiple first cascade levels 507. After obtaining the first intermediate feature maps corresponding to each of the multiple first cascade levels 507, the first intermediate feature maps corresponding to each of the multiple first cascade levels 507 can be input into an intermediate level 508 to obtain second intermediate feature maps corresponding to each of the multiple first cascade levels 507. Based on this, the second intermediate feature maps corresponding to each of the multiple first cascade levels 507 can be processed through multiple second cascade levels 509 to obtain a sample noise image 511. The multiple sample noise images 511 are used to train a first deep learning model to obtain an intermediate deep learning model 512.

[0091] According to the embodiments of the present disclosure, through the processing of multiple first cascade levels and multiple second cascade levels, layer-by-layer feature extraction and processing of the first training sample and the sample fusion vector are realized, which helps to extract more abstract and high-dimensional feature representations from the original data, thereby providing richer feature representations for subsequent model training, and can further optimize the representation and generalization capabilities of the model, which helps to improve the performance and effects of the model in practical applications.

[0092] It can be understood that the above describes how to train the intermediate deep learning model in the present disclosure. The following will describe some methods of obtaining the sample fusion vector in conjunction with Figure 6.

[0093] FIG6 schematically shows an example diagram of a process for obtaining a sample fusion vector according to an embodiment of the present disclosure.

[0094] As shown in Figure 6, in 600, the sample image category vector and the iteration step vector may have the same number of vector dimensions. A vector dimension may refer to the number of elements contained in a quantity, or may be understood as the length of a vector.

[0095] Taking multiple sample image category vectors including sample image category vector 601 and sample image category vector 602 as an example, for each vector dimension of each sample image category vector 601, the elements corresponding to the vector dimension and the elements corresponding to the same vector dimension of the iteration step vector 603 can be accumulated to obtain a sample fusion vector 604 corresponding to the sample image category vector 601. For each vector dimension of each sample image category vector 602, the elements corresponding to the vector dimension and the elements corresponding to the same vector dimension of the iteration step vector 603 can be accumulated to obtain a sample fusion vector 605 corresponding to the sample image category vector 602.

[0096] For example, the dimensions of sample image category vector 601 and sample image category vector 602 are both N*Dim, where N can be used to represent the number of first training samples. Sample image category vector 601 and sample image category vector 602 can be represented as torch.tensor([[2, 1, 2], [3, 3, 3]]). The dimension of iteration step vector 603 is 1*Dim, and iteration step vector 603 can be represented as torch.tensor([4, 4, 3]). By accumulating the elements corresponding to the vector dimensions, multiple sample fusion vectors can be obtained as torch.tensor([[6, 5, 5], [7, 7, 6]]).

[0097] The combination of multiple sample image category vectors and iteration step vectors is an addition operation, which means adding the corresponding element values ​​of the two while keeping the dimension unchanged. Its practical meaning is to add the different values ​​of the sample image category vectors to the iteration step vector and pass it to the first deep learning model to play a guiding role.

[0098] According to the embodiments of the present disclosure, by performing accumulation operations on the elements of each sample image category vector and the elements of the iteration step vector on the corresponding dimension, the fusion of sample feature information and iteration step information is achieved, which helps to integrate feature information of different dimensions together, so that the model can more comprehensively utilize the information of the data during the training process, thereby improving the richness and diversity of feature representation.

[0099] The following further illustrates some specific implementations of the image generation method disclosed herein with reference to FIG7 . Compared to FIG2 , the image generation method illustrated in FIG7 is based on a trained deep learning model and a denoising function, thereby enabling unlimited expansion of small sample data. However, the present disclosure is not limited to this method; the specific image generation method can be configured based on actual business needs and is not limited here.

[0100] FIG7 schematically shows an example diagram of an image generation process according to an embodiment of the present disclosure.

[0101] As shown in FIG7 , in 700 , the initial noise image 701 and the image category 702 may be input into a trained deep learning model 703 , and the trained deep learning model 703 may gradually predict noise for the initial noise image 701 to obtain a predicted first noise image 704 .

[0102] After obtaining first noisy image 704, initial noisy image 701, first noisy image 704, and the number of iterations can be input into denoising function 705 to gradually denoise first noisy image 704, thereby obtaining a denoised first intermediate image 706. For example, after executing this function a preset number of times, a clear denoised first intermediate image 706 can be obtained. The preset number of times can be 1000. The denoised first intermediate image 706 has the same dimensions as the predicted first noisy image 704 and the input initial noisy image 701.

[0103] The first intermediate image 706 may include at least one first intermediate pixel. Since the first intermediate image 706 is scaled and normalized, the at least one first intermediate pixel may be processed separately to convert the pixel value of each first intermediate pixel from a (-1, 1) distribution to a (0, 1) distribution. For example, for each first intermediate pixel (i.e., image), a linear transformation calculation may be performed on the first intermediate pixel to obtain a fifth intermediate pixel. The linear transformation calculation may be configured according to actual business needs and is not limited here. For example, the linear transformation calculation method may be (image / 2+0.5).

[0104] For each fifth intermediate pixel, the fifth intermediate pixel can be multiplied by a preset value to obtain a second intermediate pixel. Based on this, a first target image 707 can be generated based on at least one second intermediate pixel. The preset value can be configured based on actual business needs and is not limited here. For example, the preset value can be set to 255.

[0105] According to embodiments of the present disclosure, pixel information can be converted by processing first intermediate pixels to obtain second intermediate pixels. Furthermore, by generating a first target image based on at least one second intermediate pixel, conversion and reconstruction from the processed first intermediate pixels to the first target image can be achieved, thereby improving the efficiency and accuracy of generating the first target image.

[0106] It can be understood that some specific implementations of the image generation method disclosed in the present disclosure are described above. Below, an example of obtaining a first target image will be schematically illustrated by taking a given original image as an example.

[0107] FIG8 schematically shows an example schematic diagram of obtaining a first target image according to an embodiment of the present disclosure.

[0108] As shown in FIG8 , in 800, taking the image category A of original image 801 as an example, the image generation method provided by the present disclosure can be performed on the original image 801 to obtain a first target image 802 belonging to the image category A. Taking the image category B of original image 803 as an example, the image generation method provided by the present disclosure can be performed on the original image 803 to obtain a first target image 804 belonging to the image category B.

[0109] In the embodiment of the present disclosure, the second noise image in the image generation method can be generated based on the initial noise image, the region mask image and the image category. The process of obtaining the second noise image will be further described below with reference to FIG. 9 .

[0110] FIG9 schematically shows an example diagram of a process of obtaining a second noise image according to an embodiment of the present disclosure.

[0111] As shown in Fig. 9, in 900, a mask noise image 903 may be determined according to an initial noise image 901 and a region mask image 902. The region mask image 903 may include a target region.

[0112] In one example, after obtaining masked noise image 903, region mask image 902, masked noise image 903, and image category 904 can be input into a trained deep learning model 905 to obtain a second noise image 906. Masked noise image 903 and second noise image 906 can have the same dimensions. The dimensions of the masked noise image and the second noise image are preset multiples of the dimensions of the first noise image. The preset multiple can be configured based on actual business needs and is not limited here. For example, the preset multiple can be 2.

[0113] In one example, the number of channels of the trained deep learning model 905 can be changed to accommodate the region mask image 902. Since the masked noise image 903 is the processed initial noise image 901 and the region mask image 902, for example, the dimensions of the initial noise image 901 and the region mask image 902 are (3, 64, 64), the masked noise image 903 is the dimension obtained by concatenating the initial noise image 901 and the region mask image 902 at the 0th dimension, that is, the dimension of the masked noise image 903 is (6, 64, 64). Correspondingly, the dimension of the second noise image 906 is also (6, 64, 64).

[0114] According to an embodiment of the present disclosure, a masked noise image belonging to an image category is generated at a specified location, and a model is trained using the target location specified by the masked noise image, image category, and regional mask image. The dimension of the second noise image generated after processing by the deep learning model is consistent with the input masked noise image, and satisfies a preset numerical multiple relationship, that is, the model can learn to generate defects of a specified image category at a specified location, thereby solving the problem of insufficient samples in specific application scenarios and helping to improve the accuracy and reliability of subsequent image processing.

[0115] It can be understood that the above describes how to obtain the second noise image in the present disclosure, and some methods of how to obtain the mask noise image will be described below in conjunction with FIG. 10 .

[0116] FIG10 schematically shows an example diagram of a process of obtaining a masked noise image according to an embodiment of the present disclosure.

[0117] As shown in FIG10 , in 1000 , the initial noise image 1001 and the region mask image 1002 may have the same size. The region mask image 1002 may refer to an image with a black background and a white background. The region mask image 1002 may refer to a binary image used to identify a specific region in the image, that is, it may include a target region 1002_1 formed by image segmentation and annotation. The target region 1002_1 may be used to characterize a specified defect generation location. The target region 1002_1 includes at least one first target location, and each pixel at each first target location has a preset pixel value. The preset pixel value can be configured according to actual business needs and is not limited here. For example, the preset pixel value can be set to 0. The pixel value of the region mask image 1002 other than the at least one first target location can be set to 255.

[0118] In one example, the target region 1002_1 may be located in the original image, extracted, and converted into a binary image. The generated binary mask image is then applied to the original image to obtain the region mask image 1002 .

[0119] After obtaining the initial noise image 1001 and the regional mask image 1002, a first pixel value corresponding to each first target location in the initial noise image 1001 can be determined based on the at least one first target location in the regional mask image 1002. For each first target location, the preset pixel value corresponding to the first target location in the regional mask image 1002 is replaced with the first pixel value, while the pixel values ​​of locations other than the at least one first target location remain unchanged, thereby obtaining a masked noise image 1003.

[0120] According to an embodiment of the present disclosure, by determining the first pixel value corresponding to each first target position in the initial noise image, finding the corresponding position in the regional mask image, and then replacing the preset pixel value with the first pixel value, the pixel value of the specific area around the target position can be replaced and updated, thereby obtaining a mask noise image corresponding to the first target position, thereby enabling the generation of a mask noise image of a specified defect position, which is beneficial to improving the accuracy and reliability of subsequent image processing.

[0121] The following further illustrates some specific implementations of the image generation method disclosed herein with reference to Figure 11. Compared to Figure 2, the image generation method illustrated in Figure 11 adds a regional mask image, enabling the model to learn to generate defects of a specified image category at a specified location. However, the present disclosure is not limited to this method; the specific image generation method can be configured based on actual business needs and is not limited here.

[0122] FIG11 schematically shows an example diagram of an image generation process according to another embodiment of the present disclosure.

[0123] As shown in FIG11 , a masked noise image 1103 can be determined based on an initial noise image 1101 and a region mask image 1102. After obtaining the masked noise image 1103, the region mask image 1102, the masked noise image 1103, and the image category 1104 can be input into a trained deep learning model 1105. The trained deep learning model 1105 predicts noise for the initial noise image 1101 step by step, thereby obtaining a predicted second noise image 1106.

[0124] After obtaining second noisy image 1106, initial noisy image 1101, second noisy image 1106, and the number of iterations can be input into denoising function 1107 to denoise second noisy image 1106 step by step, obtaining a denoised second intermediate image 1108. For example, after executing this function a preset number of times, a clear denoised second intermediate image 1108 can be obtained. The preset number of times can be 1000. The dimensions of denoised second intermediate image 1108 and predicted second noisy image 1106 are a preset value multiple of the dimensions of the input initial noisy image 1101.

[0125] Second intermediate image 1108 may include at least one third intermediate pixel. Since second intermediate image 1108 has been scaled, each third intermediate pixel may be multiplied by a preset value to generate a third intermediate image. The preset value can be configured based on actual business needs and is not limited here. For example, the preset value may be set to 255. Based on this, second target image 1109 may be generated based on the third and fourth intermediate images.

[0126] According to an embodiment of the present disclosure, by processing the second intermediate pixels to generate a third intermediate image, and then combining the third intermediate image and the fourth intermediate image to generate a second target image, post-processing of the second intermediate image can be achieved, thereby achieving further processing and conversion of the image content, which helps to improve the effect of generating the second target image.

[0127] It can be understood that the above describes some specific implementations of the image generation method disclosed in the present invention, and how to obtain the second target image will be further described below in conjunction with FIG. 12 .

[0128] FIG12 schematically shows an example diagram of a process of obtaining a second target image according to an embodiment of the present disclosure.

[0129] As shown in FIG12 , third intermediate image 1202 and fourth intermediate image 1201 may be of the same size, and third intermediate image 1202 may include at least one fourth intermediate pixel. Fourth intermediate image 1201 may be an original image. An original image may be a good image, i.e., an image with few defective samples.

[0130] After obtaining third intermediate image 1202, it can be grayscaled and then binarized. At least one second target location can be determined based on at least one fourth intermediate pixel. The second target location can refer to a black portion in third intermediate image 1202. For each second target location, a second pixel value corresponding to each second target location can be determined in fourth intermediate image 1201. The second pixel value corresponding to the target location in third intermediate image 1202 is replaced with the fourth intermediate pixel. In other words, the black portion in third intermediate image 1202 is assigned to the corresponding location in fourth intermediate image 1201, resulting in second target image 1203. This second target image 1203 serves as the defect generation map for the specified location.

[0131] According to an embodiment of the present disclosure, by processing the third intermediate image and the fourth intermediate image, at least one second target position and a second pixel value corresponding to each second target position are determined, and each second pixel value is replaced according to the corresponding fourth intermediate pixel to generate a second target image. This can achieve conversion and reconstruction from the processed fourth intermediate pixel to the second target image, which helps to improve the efficiency and accuracy of generating the second target image.

[0132] The above are merely exemplary embodiments, but are not limited thereto. Other image generation methods known in the art may also be included, as long as they can improve the accuracy and pertinence of generating the first target image.

[0133] The image generation method of the present disclosure has been described above. After generating at least one of a first target image and a second target image using the image generation method, at least one of the first target image and the second target image can be used to train a second deep learning model to obtain an image processing model. The training method for the image processing model will be described below with reference to FIG13 .

[0134] FIG13 schematically shows a flowchart of a method for training an image processing model according to an embodiment of the present disclosure.

[0135] As shown in FIG13 , the training method of the image processing model includes operations S1310 to S1330 .

[0136] In operation S1310, at least one of the first target image and the second target image is input to a second deep learning model to obtain output information.

[0137] In operation S1320, a loss function value is determined according to an image category of at least one of the output information, the first target image, and the second target image.

[0138] In operation S1330, model parameters of the second deep learning model are adjusted according to the loss function value to obtain an image processing model.

[0139] At least one of the first target image and the second target image may be generated using an image generation method. The second deep learning model may include one of the following: a supervised model, a semi-supervised model, and an unsupervised model. At least one of the first target image and the second target image may be input into the second deep learning model to obtain output information for characterizing the prediction. The output information and the image category of at least one of the first target image and the second target image are input into a loss function to obtain a loss function value. The model parameters of the second deep learning model are adjusted according to the loss function value until a predetermined end condition is met. The second deep learning model obtained when the predetermined end condition is met is determined as the image processing model. The predetermined end condition may include the model iteration meeting a predetermined number of times or the loss function convergence.

[0140] The second deep learning model may refer to a model used to perform a specific image processing task. The second deep learning model can be configured according to actual business needs and is not limited here. The second deep learning model may include at least one of the following: a filter model, an edge detection model, a transformation model, a segmentation model, a feature extraction model, and an image reconstruction model. For example, the filter model can be used to perform denoising and smoothing tasks on the image to be processed, the edge detection model can be used to perform edge information detection tasks on the image to be processed, the transformation model can be used to analyze and process the image to be processed in the frequency domain, the segmentation model can be used to segment the image to be processed, the feature extraction model can be used to extract representative features from the image to be processed, and the image reconstruction model can be used to restore a high-quality image from the sampled data of the image to be processed.

[0141] According to an embodiment of the present disclosure, an image processing model is obtained by training by using at least one of the first target image and the second target image as a training sample. Therefore, the second deep learning model is trained using at least one of the first target image and the second target image, which reduces the number of model iterations, improves the training speed of the model, and improves the prediction accuracy of the model. As a result, the cost of training the second deep learning model is reduced, and the image processing model can be applied to different image processing tasks, which helps to improve the accuracy and effect of subsequent image processing.

[0142] The above are merely exemplary embodiments, but are not limited thereto. Other image processing model training methods known in the art may also be included, as long as they can improve the training speed and prediction accuracy of the model.

[0143] The above describes the training method for the image processing model of the present disclosure. After using at least one of the first target image and the second target image to train the second deep learning model to obtain the image processing model, the image processing model can also be used to process the image to be processed. The image processing method will be described below with reference to Figure 14.

[0144] FIG14 schematically shows a flow chart of an image processing method according to an embodiment of the present disclosure.

[0145] As shown in FIG. 14 , the image processing method includes operations S1410 to S1420 .

[0146] In operation S1410 , an image to be processed is acquired.

[0147] In operation S1420 , the image to be processed is input into an image processing model to obtain an image processing result.

[0148] After obtaining the image to be processed, a preprocessing operation can be performed on the image to obtain an intermediate processed image. After obtaining the intermediate processed image, the intermediate processed image can be input into an image processing model to obtain an image processing result. The image processing model can be trained using an image processing model training method.

[0149] Preprocessing operations can be configured according to actual business needs and are not limited here. Preprocessing operations may include at least one of the following: denoising, grayscale, edge detection, image enhancement, resizing, smoothing, binarization, and geometric correction. For example, denoising may refer to removing noise from an image using filters or other methods, grayscale may refer to converting a color image into a grayscale image, edge detection may refer to finding boundary information in an image through an edge detection algorithm, image enhancement may refer to enhancing the visual effect of an image by adjusting parameters such as contrast and brightness of the image, resizing may refer to scaling or cropping an image, smoothing may refer to using a smoothing filter to reduce detail information in an image, binarization may refer to converting a grayscale image into a black and white image, and geometric correction may refer to rotating, flipping, or performing perspective transformation on an image.

[0150] According to the embodiments of the present disclosure, when an image to be processed is processed using an image processing model, it is possible to more accurately determine the image processing to be implemented, thereby improving the accuracy of the image processing result and the processing efficiency of the image to be processed.

[0151] The above are merely exemplary embodiments, but are not limited thereto. Other image processing methods known in the art may also be included, as long as they can improve the accuracy of image processing results and the processing efficiency of the image to be processed.

[0152] FIG15 schematically shows a block diagram of an image generating apparatus according to an embodiment of the present disclosure.

[0153] As shown in FIG. 15 , the image generating apparatus 1500 may include a first determining module 1510 , a second determining module 1520 , and a generating module 1530 .

[0154] The first determination module 1510 is configured to determine a first noise image according to the initial noise image and the image category.

[0155] The second determining module 1520 is configured to determine a first intermediate image according to the initial noise image and the first noise image.

[0156] The generating module 1530 is configured to generate a first target image belonging to an image category based on the first intermediate image.

[0157] FIG16 schematically shows a block diagram of a training device for an image processing model according to an embodiment of the present disclosure.

[0158] As shown in FIG. 16 , the image processing model training apparatus 1600 may include a first input module 1610 , a third determination module 1620 , and an adjustment module 1630 .

[0159] The first input module 1610 is used to input at least one of the first target image and the second target image into the deep learning model to obtain output information.

[0160] The third determination module 1620 is configured to determine a loss function value according to the output information and an image category of at least one of the first target image and the second target image.

[0161] The adjustment module 1630 is configured to adjust the model parameters of the deep learning model according to the loss function value to obtain an image processing model. At least one of the first target image and the second target image is generated by an image generating device.

[0162] FIG17 schematically shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0163] As shown in FIG. 17 , the image processing apparatus 1700 may include an acquisition module 1710 and a second input module 1720 .

[0164] The acquisition module 1710 is used to acquire the image to be processed.

[0165] The second input module 1720 is used to input the image to be processed into the image processing model to obtain the image processing result. The image processing model is trained using the image processing model training device.

[0166] According to the modules of the embodiments of the present invention, any multiple or at least part of the functions of any multiple thereof can be implemented in one module. According to the modules of the embodiments of the present invention, any one or more can be split into multiple modules for implementation. According to the modules of the embodiments of the present invention, any one or more can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware of any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware and firmware or in a suitable combination of any several thereof. Alternatively, according to the modules of the embodiments of the present invention, one or more can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is run.

[0167] For example, any number of the first determination module 1510, the second determination module 1520, and the generation module 1530, or the first input module 1610, the third determination module 1620, and the adjustment module 1630, or the acquisition module 1710 and the second input module 1720 can be combined into a single module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in a single module. According to an embodiment of the present disclosure, the first determination module 1510, the second determination module 1520 and the generation module 1530, or the first input module 1610, the third determination module 1620 and the adjustment module 1630, or at least one of the acquisition module 1710 and the second input module 1720 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware and firmware, or in an appropriate combination of any of them. Alternatively, the first determination module 1510, the second determination module 1520 and the generation module 1530, or the first input module 1610, the third determination module 1620 and the adjustment module 1630, or at least one of the acquisition module 1710 and the second input module 1720 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0168] It should be noted that the image generating device part in the embodiment of the present disclosure corresponds to the image generating method part in the embodiment of the present disclosure, and the description of the image generating device part specifically refers to the image generating method part, which will not be repeated here. The training device part of the image processing model in the embodiment of the present disclosure corresponds to the training method part of the image processing model in the embodiment of the present disclosure, and the description of the training device part of the image processing model specifically refers to the training method part of the image processing model, which will not be repeated here. The image processing device part in the embodiment of the present disclosure corresponds to the image processing method part in the embodiment of the present disclosure, and the description of the image processing device part specifically refers to the image processing method part, which will not be repeated here.

[0169] Figure 18 schematically illustrates a block diagram of an electronic device suitable for implementing the image generation method, the image processing model training method, and the image processing method according to an embodiment of the present disclosure. The electronic device shown in Figure 18 is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

[0170] As shown in Figure 18, the computer electronic device 1800 according to an embodiment of the present disclosure includes a processor 1801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1802 or a program loaded from a storage portion 1809 into a random access memory (RAM) 1803. The processor 1801 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1801 may also include onboard memory for caching purposes. The processor 1801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0171] Various programs and data required for the operation of the electronic device 1800 are stored in the RAM 1803. The processor 1801, the ROM 1802, and the RAM 1803 are connected to each other via a bus 1804. The processor 1801 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 1802 and / or the RAM 1803. It should be noted that the programs may also be stored in one or more memories other than the ROM 1802 and the RAM 1803. The processor 1801 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0172] According to an embodiment of the present disclosure, electronic device 1800 may further include an input / output (I / O) interface 1805, which is also connected to bus 1804. Electronic device 1800 may further include one or more of the following components connected to I / O interface 1805: an input section 1806 including a keyboard, mouse, etc.; an output section 1807 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 1808 including a hard disk; and a communication section 1809 including a network interface card such as a LAN card or modem. Communication section 1809 performs communication processing via a network such as the Internet. A drive 1810 is also connected to I / O interface 1805 as needed. Removable media 1811, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 1810 as needed, so that computer programs read from the removable media can be installed into storage section 1808 as needed.

[0173] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1809, and / or installed from the removable medium 1811. When the computer program is executed by the processor 1801, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0174] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0175] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0176] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 1802 and / or the RAM 1803 described above and / or one or more memories other than the ROM 1802 and the RAM 1803 .

[0177] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the image generation method, image processing model training method and image processing method provided by the embodiment of the present disclosure.

[0178] When the computer program is executed by the processor 1801, the above functions defined in the system / device of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0179] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal over a network medium, downloaded and installed via the communication portion 1809, and / or installed from removable media 1811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0180] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0181] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, as well as the combination of boxes in the block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or may be implemented using a combination of dedicated hardware and computer instructions. It will be understood by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or coupled in various ways, and all such combinations and / or couplings fall within the scope of the present disclosure.

[0182] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for generating an image, comprising: Determining a first noise image according to the initial noise image and the image category; determining a first intermediate image according to the initial noise image and the first noise image; as well as A first target image belonging to the image category is generated based on the first intermediate image.

2. The method according to claim 1, wherein The initial noise image is determined based on the original image and random noise; The determining of the first noise image according to the initial noise image and the image category includes: Inputting the initial noise image and the image category into a trained deep learning model to obtain the first noise image; The trained deep learning model is obtained by training in the following way: Training a first deep learning model using a first training sample set to obtain an intermediate deep learning model, wherein the first training sample set includes a plurality of first training samples each having a sample image category, and the image category is the same as at least one of the plurality of sample image categories; and The intermediate deep learning model is fine-tuned using a second training sample set to obtain the trained deep learning model, wherein the second training sample set includes a plurality of second training samples that have the same image category as the first target image.

3. The method according to claim 2, wherein: The first training sample set includes a plurality of first training samples, each of the first training samples has a sample image category; The training of the first deep learning model using the first training sample set to obtain an intermediate deep learning model includes: Performing feature extraction processing on the multiple sample image categories respectively to obtain multiple sample image category vectors; Performing feature extraction processing on the iteration steps of the deep learning model to obtain an iteration step vector; Determining a plurality of sample fusion vectors according to the plurality of sample image category vectors and the iteration step number vector; and The first deep learning model is trained using the multiple first training samples and the multiple sample fusion vectors to obtain the intermediate deep learning model.

4. The method according to claim 3, wherein: The determining of a plurality of sample fusion vectors according to the plurality of sample image category vectors and the iteration step vector comprises: For each vector dimension of each sample image category vector, The elements are accumulated with the elements corresponding to the same vector dimension of the iteration step vector to obtain the multiple sample fusion vectors.

5. The method according to claim 3, wherein: The step of training the first deep learning model using the plurality of first training samples and the plurality of sample fusion vectors to obtain the intermediate deep learning model includes: For each of the first training samples, Performing a plurality of first cascade level processing on the first training sample and the sample fusion vector corresponding to the first training sample to obtain a first intermediate feature map corresponding to each of the plurality of first cascade levels; Performing a plurality of second cascade level processes on the first intermediate feature maps corresponding to the plurality of first cascade levels, to obtain a sample noise image; and The first deep learning model is trained using the plurality of sample noise images to obtain the intermediate deep learning model.

6. The method according to any one of claims 1 to 5, wherein The first intermediate image includes at least one first intermediate pixel; Generating a first target image belonging to the image category according to the first intermediate image includes: For each of the first intermediate pixels, process the first intermediate pixel to obtain at least one second intermediate pixel; The first target image is generated according to the at least one second intermediate pixel.

7. The method according to claim 1, further comprising: determining a masked noise image according to the initial noise image and the region mask image, wherein the region mask image includes the target region; determining a second noise image according to the masked noise image and the image category; determining a second intermediate image based on the masked noise image and the second noise image; and Based on the second intermediate image, a second target image belonging to the image category is generated.

8. The method according to claim 7, wherein: The initial noise image and the region mask image have the same size, the target region includes at least one first target position, and each pixel of the first target position has a preset pixel value; Determining the mask noise image according to the initial noise image and the region mask image includes: Determining a first pixel value corresponding to each of the first target positions in the initial noise image; as well as For each first target position, a preset pixel value corresponding to the first target position in the regional mask image is replaced with the first pixel value to obtain the mask noise image.

9. The method according to claim 7 or 8, wherein The determining, according to the masked noise image and the image category, a second noise image comprises: Inputting the region mask image, the mask noise image, and the image category into the trained deep learning model to obtain the second noise image; The masked noise image and the second noise image have the same dimension, and the dimensions of the masked noise image and the second noise image are preset numerical multiples of the dimension of the first noise image.

10. The method according to claim 7 or 8, wherein The second intermediate image comprises at least one third intermediate pixel; Generating a second target image belonging to the image category according to the second intermediate image includes: For each of the third intermediate pixels, process the third intermediate pixel to obtain a third intermediate image; and The second target image is generated based on the third intermediate image and the fourth intermediate image.

11. The method according to claim 10, wherein: The third intermediate image and the fourth intermediate image have the same size, and the third intermediate image includes at least one fourth intermediate pixel; Generating the second target image according to the third intermediate image and the fourth intermediate image includes: determining at least one second target position based on the at least one fourth intermediate pixel; For each of the second target positions, determining a second pixel value corresponding to each of the second target positions in the fourth intermediate image; and The second pixel value corresponding to the target position in the third intermediate image is replaced by the fourth intermediate pixel to obtain the second target image.

12. The method according to claim 1 or 7, wherein: The initial noise image includes an image of at least one of the following application scenarios: industrial image, medical image, cultural relic image, film and television image, traffic image and astronomical image.

13. A method for training an image processing model, comprising: Inputting at least one of the first target image and the second target image into a second deep learning model to obtain output information; determining a loss function value according to the output information and an image category of at least one of the first target image and the second target image; as well as Adjusting model parameters of the second deep learning model according to the loss function value to obtain an image processing model; Wherein, at least one of the first target image and the second target image is generated by using the method according to any one of claims 1 to 12.

14. An image processing method, comprising: Get the image to be processed; as well as Inputting the image to be processed into an image processing model to obtain an image processing result; The image processing model is trained using the method according to claim 13.

15. An image generating device, comprising: A first determining module, configured to determine a first noise image according to an initial noise image and an image category; a second determining module, configured to determine a first intermediate image based on the initial noise image and the first noise image; as well as A generating module is configured to generate a first target image belonging to the image category based on the first intermediate image.

16. A training device for an image processing model, comprising: A first input module is configured to input at least one of the first target image and the second target image into the deep learning model to obtain output information; a third determining module, configured to determine a loss function value according to the output information and an image category of at least one of the first target image and the second target image; as well as An adjustment module, configured to adjust model parameters of the deep learning model according to the loss function value to obtain an image processing model; At least one of the first target image and the second target image is generated using the apparatus according to claim 15 .

17. An image processing apparatus, comprising: An acquisition module, used for acquiring an image to be processed; as well as A second input module is used to input the image to be processed into the image processing model to obtain an image processing result; Wherein, the image processing model is trained using the device according to claim 16.

18. An electronic device comprising: one or more processors; a memory for storing one or more instructions, Wherein, when the one or more instructions are executed by the one or more processors, the one or more processors are enabled to implement the method of any one of claims 1 to 12 or claim 13 or claim 14.

19. A computer-readable storage medium having executable instructions stored thereon, wherein when the executable instructions are executed by a processor, the processor is enabled to implement any one of claims 1 to 13 or claim 14 or the method of claim 14.

20. A computer program product comprising computer executable instructions for implementing the method of any one of claims 1 to 12 or claim 13 or claim 14 when executed.

Images