Image conversion model training method, image conversion processing method and related device

Abstract

The invention provides an image conversion model training method, an image conversion processing method and a related device, and relates to the technical field of image conversion. In the process of performing image conversion on a plurality of training image samples by using a to-be-trained image conversion model, a self-supervised vision model is called to extract respective feature thermodynamic diagrams of a related source domain sample image, a target domain sample image, a target domain generated image and a source domain generated image; a thermodynamic diagram consistency loss function of the to-be-trained image conversion model is constructed, and then model parameter optimization is carried out for minimizing a target loss function (which relates to the thermodynamic diagram consistency loss function) of the to-be-trained image conversion model, so that when the corresponding target loss function is in a minimization state, the to-be-trained image conversion model is subjected to model parameter optimization; and the expected image conversion model is cut from the to-be-trained image conversion model, so that the image generation stability, the image edge contour definition and the image structure distribution controllability of the expected image conversion model are improved.

Description

Technical Field

[0001] This application relates to the field of image conversion technology, and more specifically, to an image conversion model training method, an image conversion processing method, and related apparatus. Background Technology

[0002] With the continuous development of science and technology, image conversion technology, as a core computer vision technology for performing tasks such as image generation, image translation, and image style transfer, is being used more and more widely in operational scenarios such as medical image enhancement, satellite remote sensing image synthesis, artistic style conversion, and industrial visual inspection. This has placed higher demands on the performance of image conversion models.

[0003] With the diversification of image transformation model architectures (e.g., Generative Adversarial Networks (GAN), Deep Convolution Generative Adversarial Networks (DCGAN), Cycle-Consistent Generative Adversarial Networks (CycleGAN), etc.), the performance of image transformation models should theoretically become more reliable. However, contrary to expectations, various image transformation model architectures still face a large number of key problems in practical applications. For example, most existing image conversion model architectures employ game-theoretic optimization mechanisms, which can lead to problems such as mode collapse, gradient oscillation, and large fluctuations in the quality of generated results during training, making it difficult to guarantee the stability of the generated images. Furthermore, current image conversion model architectures primarily focus on image domain conversion effects during training, neglecting the clarity of image edges and the details of image structure distribution. This can easily result in blurring, artifacts, and distortion at image edges, and the actual image structure distribution exhibits unpredictable randomness, failing to meet the requirements for image structure controllability and image generation quality. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an image conversion model training method, an image conversion processing method, a computer device, and a readable storage medium, which can introduce the thermal relationship between the real image and the generated image during the model training process of any image conversion model architecture. Figure 1Consistency constraints (which are used to align features of real and generated images at high-level semantic structures) are used to optimize model parameters so that the final trained desired image transformation model can ensure that the corresponding image generation results achieve global image domain transformation effects while possessing good image generation stability, image edge contour clarity, and controllable image structure distribution.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, this application provides an image conversion model training method, the method comprising: Multiple training image samples are acquired and sequentially input into the image conversion model to be trained for image conversion; wherein, each training image sample includes one source domain sample image and one target domain sample image; During the image transformation process of the image transformation model to be trained for each training image sample, a self-supervised visual model is invoked to extract feature heatmaps of the source domain sample image, target domain sample image, target domain generated image, and source domain generated image associated with the training image sample; wherein the target domain generated image is obtained by the image transformation model to be trained transforming the source domain sample image into the target domain, and the source domain generated image is obtained by the image transformation model to be trained transforming the target domain sample image into the source domain; Based on the feature heatmaps of the source domain sample image, the target domain sample image, the target domain generated image, and the source domain generated image, a heatmap is constructed for the image conversion model to be trained. Figure 1 Consistency loss function; The model parameters of the image conversion model to be trained are optimized with the aim of minimizing the objective loss function of the model, wherein the objective loss function relates to the thermal... Figure 1 Consistency loss function; When the target loss function is minimized, the image conversion model to be trained is pruned to obtain the corresponding desired image conversion model.

[0006] In an optional implementation, the step of constructing a heatmap for the image conversion model to be trained based on the feature heatmaps of the source domain sample image, the target domain sample image, the target domain generated image, and the source domain generated image is performed. Figure 1 The steps of the consistency loss function include: Based on the feature heatmaps of the source domain sample image and the target domain generated image, a first Dice loss function, a first crossover ratio (CRO) loss function, and a first edge preservation loss function are constructed for these two feature heatmaps. Based on the feature heatmaps of the target domain sample image and the source domain generated image, a second Dice loss function, a second crossover ratio (CRO) loss function, and a second edge preservation loss function are constructed for these two feature heatmaps. The first Dice loss function, the first cross-union ratio (CUNR) loss function, the first edge-preserving loss function, the second Dice loss function, the second CUNR loss function, and the second edge-preserving loss function are weighted and summed to obtain the thermodynamic loss function. Figure 1 Consistency loss function.

[0007] In an optional implementation, for any real sample image among the source domain sample images and the target domain sample images, the Dice loss function between the real sample image and the corresponding model-generated image is expressed by the following formula: ; The cross-union loss function between the real sample image and the corresponding model-generated image is expressed by the following formula: ; The edge-preserving loss function between the real sample image and the corresponding model-generated image is expressed by the following formula: ; in, The Dice loss function is used to represent the difference between the real sample image and the corresponding model-generated image. The cross-union loss function is used to represent the intersection-over-union ratio between the real sample image and the corresponding model-generated image. This is used to represent the edge-preserving loss function between the real sample image and the corresponding model-generated image. A feature heatmap used to represent the real sample image. A feature heatmap used to represent the model-generated image corresponding to the real sample image. and These are used to represent the horizontal and vertical directions of the image, respectively. For characteristic heatmap Image gradient in the horizontal axis direction, For characteristic heatmap Image gradient in the horizontal axis direction, For characteristic heatmap Image gradient along the vertical axis, For characteristic heatmap Image gradient along the vertical axis, Used to represent the L1 norm; the model-generated image corresponding to the source domain sample image is the target domain generated image; the model-generated image corresponding to the target domain sample image is the source domain generated image.

[0008] In an optional implementation, the step of cropping the image conversion model to obtain the corresponding desired image conversion model includes: Extract the image generator from the image conversion model to be trained, and use the extracted image generator as the desired image conversion model.

[0009] In an optional implementation, the target loss function of the image conversion model to be trained is composed of the original model architecture loss function of the image conversion model to be trained and the thermal loss function. Figure 1 The original model architecture loss function is obtained by weighted summation of consistency loss functions, and the model architecture type of the image conversion model to be trained is matched.

[0010] In an optional implementation, when the model architecture type of the image transformation model to be trained is a generative adversarial network or a deep convolutional generative adversarial network, the original model architecture loss function of the image transformation model to be trained is an adversarial loss function; wherein the adversarial loss function It can be expressed as follows: ; in, This represents the probability of the image domain authenticity judgment of the image transformation model to be trained for the input source domain sample image or target domain sample image. This represents the probability of image domain authenticity discrimination for the image generated by the image conversion model to be trained, based on the converted target domain or source domain. for The logarithm of for The logarithm of Used to represent mathematical expectation solution functions.

[0011] In an optional implementation, when the model architecture type of the image conversion model to be trained is a recurrent consistency generative adversarial network, the original model architecture loss function of the image conversion model to be trained is obtained by weighted summation of the adversarial loss function and the recurrent consistency loss function; Wherein, the adversarial loss function It can be expressed as follows: ; in, This represents the probability of the image domain authenticity judgment of the image transformation model to be trained for the input source domain sample image or target domain sample image. This represents the probability of image domain authenticity discrimination for the image generated by the image conversion model to be trained, based on the converted target domain or source domain. for The logarithm of for The logarithm of Used to represent mathematical expectation solution functions; The cycle consistency loss function It can be expressed as follows: ; in, Used to represent the source domain sample image input to the image transformation model to be trained; Used to represent the target domain sample image input to the image transformation model to be trained; The source domain reconstructed image is used to represent the image that matches the source domain sample image. It is obtained by reconstructing the source domain from the target domain generated image of the source domain sample image. The target domain reconstructed image, which represents the image matching the target domain sample image, is obtained by reconstructing the target domain from the source domain generated image of the target domain sample image. Used to represent the L1 norm.

[0012] Secondly, this application provides an image conversion processing method, the method comprising: Obtain the image to be converted and the reference image for conversion; The image to be converted and the reference image to be converted are input into the desired image conversion model. The desired image conversion model converts the actual image domain of the image to be converted to the target image domain corresponding to the reference image to obtain the corresponding desired image. The desired image conversion model is trained using any of the image conversion model training methods described in the foregoing embodiments.

[0013] Thirdly, this application provides a computer device, including a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the image conversion model training method described in any of the foregoing embodiments, or to implement the image conversion processing method described in the foregoing embodiments.

[0014] Fourthly, this application provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a computer device, it implements the image conversion model training method described in any of the foregoing embodiments, or the image conversion processing method described in the foregoing embodiments.

[0015] In this case, the beneficial effects of the embodiments of this application may include the following: In the process of performing image conversion on multiple training image samples using the image conversion model to be trained, this application extracts feature heatmaps of relevant source domain sample images, target domain sample images, target domain generated images (obtained by the image conversion model to convert the source domain sample images to the target domain), and source domain generated images (obtained by the image conversion model to convert the target domain sample images to the source domain) by calling a self-supervised visual model. These feature heatmaps reflect the actual distribution of high-level semantic regions and the details of region edge contours within their respective images. This process constructs the heatmap of the image conversion model to be trained. Figure 1 The consistency loss function (which describes the magnitude of the difference in heatmap distribution between the real image (e.g., a source domain sample image or a target domain sample image) and the generated image (e.g., a target domain generated image or a source domain generated image) is then used to minimize the objective loss function of the image transformation model to be trained (which involves the aforementioned heatmap). Figure 1 The model parameters are optimized with the aim of minimizing the corresponding target loss function, so that the desired image transformation model (i.e., the model component that truly realizes the image domain transformation function) is pruned from the image transformation model to be trained. This allows the introduction of heat between the real image and the generated image during the model training process of any image transformation model architecture. Figure 1 Consistency constraints (which employ thermal force) Figure 1 The model parameters are optimized by implementing a consistency loss function and effectively aligning the features of the real image and the generated image in the high-level semantic structure. This enhances the image transformation model's ability to retain and transfer image semantic structure information during the image transformation process (i.e., to retain as much effective semantic structure information of the real image as possible (including edge contour details and regional distribution of high-level semantic regions) and assign it to the generated image). This ensures that the corresponding image generation result achieves the global image domain transformation effect while possessing good image generation stability, image edge contour clarity, and controllable image structure distribution.

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic diagram of the composition of a computer device provided in the embodiments of this application; Figure 2 A schematic flowchart illustrating the image conversion model training method provided in this application embodiment; Figure 3 for Figure 2 A flowchart illustrating the sub-steps included in step S230; Figure 4 This is a schematic flowchart of the image conversion processing method provided in the embodiments of this application.

[0019] Icons: 10-Computer equipment; 11-Memory; 12-Processor; 13-Communication unit. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0023] In the description of this application, it should be understood that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are used only for the convenience of describing this application and simplifying the description, and are not intended to indicate or imply that the equipment or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0024] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0025] Furthermore, it is understood in the description of this application that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art will understand the specific meaning of the above terms in this application based on the specific circumstances.

[0026] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0027] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the computer device 10 provided in an embodiment of this application. In this embodiment, the computer device 10 can be used to train an image conversion model of any image conversion model architecture, and to use the trained image conversion model to perform image domain conversion processing on any image to be converted, so as to obtain a desired image that meets the expected image domain effect. The computer device 10 can be, but is not limited to, a smartphone, laptop computer, personal computer, server, etc.

[0028] In this embodiment, the computer device 10 may include a memory 11, a processor 12, and a communication unit 13. The memory 11, the processor 12, and the communication unit 13 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines.

[0029] In this embodiment, the memory 11 may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 11 is used to store computer programs, and the processor 12 can execute the computer programs accordingly after receiving execution instructions.

[0030] In this embodiment, the processor 12 can be an integrated circuit chip with signal processing capabilities. The processor 12 can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0031] In this embodiment, the communication unit 13 is used to establish a communication connection between the computer device 10 and other electronic devices through a network, and to send and receive data through the network, wherein the network includes wired communication networks and wireless communication networks. For example, the computer device 10 can obtain a model training sample set uploaded by other electronic devices for the image conversion model to be trained through the communication unit 13. The model training sample set may include multiple training image samples, each training image sample may include a source domain sample image and a target domain sample image, wherein the source domain sample images involved in each of the multiple training image samples belong to valid real images in the same image domain, and the target domain sample images involved in each of the multiple training image samples belong to valid real images in the same image domain, and the image domains corresponding to the source domain sample image and the target domain sample image under the same training image sample are different from each other.

[0032] Optionally, in this embodiment, the computer device 10 may pre-store a specific computer program related to the image conversion model training function in the memory 11, and by driving the processor 12 to execute the specific computer program, a thermal relationship between the real image and the generated image is introduced during the model training process of any image conversion model architecture. Figure 1 Consistency constraints are used to optimize model parameters, enhance the ability of the desired image transformation model to retain and transfer image semantic structure information during the image transformation process, and ensure that the image generation results based on the desired image transformation model achieve global image domain transformation effects while possessing good image generation stability, image edge contour clarity, and controllable image structure distribution.

[0033] Optionally, in this embodiment, the computer device 10 may pre-store a specific computer program related to the image conversion processing function in the memory 11, and by driving the processor 12 to execute the specific computer program, the trained desired image conversion model is used to perform stable image domain conversion processing on any image to be converted according to the reference conversion image, so that the final desired image can achieve the expected image domain effect while having good image edge contour clarity, and can maintain an image structure as similar as possible to the reference conversion image and the image to be converted.

[0034] Understandable, Figure 1 The block diagram shown is only a schematic diagram of one configuration of the computer device 10. The computer device 10 may also include components such as... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0035] In this application, to ensure that the computer device 10 can introduce thermal differences between the real image and the generated image during the image conversion model training process, Figure 1 By optimizing model parameters based on consistency constraints, the final desired image conversion model can enhance its ability to preserve and transfer semantic structural information during image conversion. This application provides an image conversion model training method to achieve the aforementioned objective. The image conversion model training method provided in this application is described in detail below.

[0036] Please refer to Figure 2 , Figure 2 This is a schematic flowchart of the image conversion model training method provided in an embodiment of this application. In this embodiment, Figure 2 The image conversion model training method shown may include steps S210 to S250.

[0037] Step S210: Obtain multiple training image samples and input them sequentially into the image conversion model to be trained for image conversion.

[0038] In this embodiment, each training image sample includes a source domain sample image and a target domain sample image; the model architecture type of the image conversion model to be trained may be, but is not limited to: generative adversarial network, deep convolutional generative adversarial network, recurrent consistent generative adversarial network, etc.; for the image conversion model to be trained, each training image sample (including a source domain sample image and a target domain sample image) input to the image conversion model to be trained belongs to a real image of a different image domain.

[0039] Step S220: During the image conversion process of the image conversion model to be trained for each training image sample, the self-supervised visual model is called to extract the feature heatmaps of the source domain sample image, target domain sample image, target domain generated image and source domain generated image associated with the training image sample.

[0040] In this embodiment, the image transformation model to be trained can call its own image generator to transform any received source domain sample image from the source domain to the target domain, obtaining a target domain generated image corresponding to the source domain sample image. Simultaneously, it can transform any received target domain sample image from the target domain to the source domain, obtaining a source domain generated image corresponding to the target domain sample image. Then, a pre-trained self-supervised vision model (i.e., a computer vision model implemented using a self-supervised learning mechanism, such as a feature extraction model based on Vision Transformer) can be called to extract feature heatmaps from the two real images (i.e., the source domain sample image and the target domain sample image) and the two generated images (i.e., the target domain generated image and the source domain generated image). This allows the extracted feature heatmaps to effectively reflect the actual distribution of high-level semantic regions and the details of region edge contours in the extracted images.

[0041] Step S230: Based on the feature heatmaps of the source domain sample image, target domain sample image, target domain generated image, and source domain generated image, construct a heatmap for the image conversion model to be trained. Figure 1 Consistency loss function.

[0042] In this embodiment, the heat Figure 1 The consistency loss function describes the difference in heatmap distribution between the real image (i.e., the input image of the image transformation model to be trained, such as source domain sample images and target domain sample images) and the generated image (e.g., target domain generated image and source domain generated image) obtained by the image transformation model to be trained. Specifically, the Dice loss function, the cross-union ratio loss function, and the edge-preserving loss function can be used to collaboratively construct a heatmap that effectively considers the differences in image structural information between the real and generated images in the source and target domains. Figure 1 Consistency loss function.

[0043] Alternatively, please refer to Figure 3 , Figure 3 yes Figure 2 A flowchart illustrating the sub-steps included in step S230 is provided. In this embodiment, step S220 may include sub-steps S231 to S233 to ensure the final thermal structure is constructed. Figure 1 Consistency loss function can comprehensively describe the differences in image structural information between real and generated images in the source and target domains from multiple dimensions.

[0044] Sub-step S231: Based on the feature heatmaps of the source domain sample image and the target domain generated image, construct the first Dice loss function, the first crossover ratio loss function, and the first edge preservation loss function for these two feature heatmaps.

[0045] Sub-step S232: Based on the feature heatmaps of the target domain sample image and the source domain generated image, construct a second Dice loss function, a second crossover ratio loss function, and a second edge preservation loss function for these two feature heatmaps.

[0046] In this embodiment, for any real sample image among the source domain sample images and the target domain sample images, the Dice loss function between the real sample image and the corresponding model-generated image is expressed by the following formula: ; The cross-union loss function between the real sample image and the corresponding model-generated image is expressed by the following formula: ; The edge-preserving loss function between the real sample image and the corresponding model-generated image is expressed by the following formula: ; in, The Dice loss function is used to represent the difference between the real sample image and the corresponding model-generated image. The cross-union loss function is used to represent the intersection-over-union ratio between the real sample image and the corresponding model-generated image. This is used to represent the edge-preserving loss function between the real sample image and the corresponding model-generated image. A feature heatmap used to represent the real sample image. A feature heatmap used to represent the model-generated image corresponding to the real sample image. and These are used to represent the horizontal and vertical directions of the image, respectively. For characteristic heatmap Image gradient in the horizontal axis direction, For characteristic heatmap Image gradient in the horizontal axis direction, For characteristic heatmap Image gradient along the vertical axis, For characteristic heatmap Image gradient along the vertical axis, Used to represent the L1 norm; the model-generated image corresponding to the source domain sample image is the target domain generated image; the model-generated image corresponding to the target domain sample image is the source domain generated image; Used to represent feature heatmaps With characteristic heatmap The number of pixels in the intersection region between them; Used to represent feature heatmaps The total number of pixels; Used to represent feature heatmaps The total number of pixels; Used to represent feature heatmaps With characteristic heatmap The number of pixels in the union region between them.

[0047] Sub-step S233 involves performing a weighted summation operation on the first Dice loss function, the first cross-union ratio loss function, the first edge-preserving loss function, the second Dice loss function, the second cross-union ratio loss function, and the second edge-preserving loss function to obtain the thermodynamic loss function. Figure 1 Consistency loss function.

[0048] In this embodiment, the computer device 10 can pre-configure a comprehensive representation weight for each of the first Dice loss function, the first intersection-over-union (IoU) loss function, the first edge-preserving loss function, the second Dice loss function, the second IoU loss function, and the second edge-preserving loss function. Then, by weighted summing these six loss functions, the final determined thermodynamic value is obtained. Figure 1 The consistency loss function can comprehensively describe the differences in image structural information between the real and generated images in the source and target domains from multiple dimensions. The comprehensive representation weights of the aforementioned six loss functions can be completely identical (e.g., each of the aforementioned six loss functions has a comprehensive representation weight of 1 or 1 / 6), partially identical, or completely different.

[0049] Therefore, by executing the above sub-steps S231 to S233, this application can ensure the final thermal structure is constructed. Figure 1 Consistency loss function can comprehensively describe the differences in image structural information between real and generated images in the source and target domains from multiple dimensions.

[0050] Step S240: To minimize the target loss function of the image conversion model to be trained, the model parameters of the image conversion model to be trained are optimized, wherein the target loss function involves thermal... Figure 1 Consistency loss function.

[0051] In this embodiment, the target loss function of the image conversion model to be trained is composed of the original model architecture loss function and the thermal loss function of the image conversion model to be trained. Figure 1 The loss function is obtained by weighted summation of consistency loss functions, and the original model architecture loss function matches the model architecture type of the image conversion model to be trained. The computer device 10 can pre-configure the original model architecture loss function and the thermal loss function. Figure 1 Each consistency loss function is configured with a loss impact weight, so as to affect the loss function of the original model architecture and the thermal model. Figure 1The target loss function is obtained by weighted summation of the consistency loss functions; the weights of the two loss functions can be the same (e.g., 0.5 or 1) or completely different.

[0052] Optionally, in one embodiment of this example, when the model architecture type of the image transformation model to be trained is a generative adversarial network or a deep convolutional generative adversarial network, the original model architecture loss function of the image transformation model to be trained is an adversarial loss function. Wherein, the adversarial loss function... It can be expressed as follows: ; in, This is used to represent the image domain authenticity discrimination probability of the image conversion model under training for input source domain sample images or target domain sample images (e.g., the discrimination probability that the image domain of the input source domain sample image belongs to the source domain, and the discrimination probability that the image domain of the input target domain sample image belongs to the target domain). This is used to represent the probability of the image domain authenticity of the image generated by the image conversion model to be trained for the converted target domain or the image generated by the source domain (e.g., the probability of determining that the image domain of the converted target domain generated image belongs to the target domain, and the probability of determining that the image domain of the converted source domain generated image belongs to the source domain). for The logarithm of for The logarithm of Used to represent mathematical expectation solution functions.

[0053] Optionally, in another embodiment of this invention, when the model architecture type of the image conversion model to be trained is a recurrent consistency generative adversarial network, the original model architecture loss function of the image conversion model to be trained is obtained by weighted summation of the adversarial loss function and the recurrent consistency loss function. The computer device 10 may pre-assign a comprehensive evaluation weight to each of the adversarial loss function and the recurrent consistency loss function, so that the corresponding original model architecture loss function can be obtained by weighted summation of these two loss functions; the comprehensive evaluation weights of these two loss functions can be the same (e.g., 0.5 or 1) or completely different.

[0054] In this process, the adversarial loss function It can be expressed using the following formula: ; in, This represents the probability of the image domain authenticity judgment of the image transformation model to be trained for the input source domain sample image or target domain sample image. This represents the probability of image domain authenticity discrimination for the image generated by the image conversion model to be trained, based on the converted target domain or source domain. for The logarithm of for The logarithm of Used to represent mathematical expectation solution functions.

[0055] The cycle consistency loss function It can be expressed using the following formula: ; in, Used to represent the source domain sample image input to the image transformation model to be trained; Used to represent the target domain sample image input to the image transformation model to be trained; The source domain reconstructed image, which is used to represent the source domain sample image, is obtained by reconstructing the target domain from the source domain to the source domain from the target domain generated image of the source domain sample image through an image generator (which belongs to the image transformation model to be trained); The target domain reconstructed image, which is used to represent the target domain sample image, is obtained by reconstructing the target domain from the source domain to the target domain using the aforementioned image generator; Used to represent the L1 norm.

[0056] Step S250: When the target loss function is minimized, the image conversion model to be trained is cropped to obtain the corresponding desired image conversion model.

[0057] In this embodiment, when the image conversion model to be trained is trained using multiple training image samples, and the target loss function of the image conversion model to be trained is minimized, it indicates that the image domain conversion capability and the ability to preserve and transfer image semantic structure information of the image generator of the image conversion model to be trained have reached the qualified standard. At this time, the image generator in the image conversion model to be trained can be extracted by model pruning, so that the extracted image generator can be directly used as the desired image conversion model. This ensures that the desired image conversion model can have good image semantic structure information preservation and transfer capability during the image conversion process, and ensures that the image generation result based on the desired image conversion model can achieve the global image domain transformation effect while having good image generation stability, image edge contour clarity and controllable image structure distribution.

[0058] Therefore, by executing the above steps S210 to S250, this application can introduce a thermal relationship between the real image and the generated image during the model training process of any image conversion model architecture. Figure 1 Consistency constraints are used to optimize model parameters to enhance the image semantic structure information retention and transfer capability of the desired image transformation model during the image transformation process. This ensures that the image generation results based on the desired image transformation model achieve global image domain transformation effects while possessing good image generation stability, image edge contour clarity, and controllable image structure distribution.

[0059] In this application, to ensure that the computer device 10 can use a trained desired image conversion model to perform stable image domain conversion processing on any image to be converted according to a reference conversion image, so that the final desired image can achieve the expected image domain effect while having good image edge contour clarity and maintaining an image structure as similar as possible to the reference conversion image and the image to be converted, this application provides an image conversion processing method to achieve the aforementioned objective. The image conversion processing method provided in this application will be described in detail below.

[0060] Please refer to Figure 4 , Figure 4 This is a schematic flowchart of the image conversion processing method provided in an embodiment of this application. In this embodiment, Figure 4 The image conversion processing method shown may include steps S310 and S320.

[0061] Step S310: Obtain the image to be converted and the reference image for conversion.

[0062] In this embodiment, the image domains of the image to be converted and the conversion reference image are different from each other; the image to be converted is the image that needs to be converted to the target image domain corresponding to the conversion reference image.

[0063] Step S320: Input the image to be converted and the conversion reference image into the desired image conversion model. The desired image conversion model converts the actual image domain of the image to be converted to the target image domain corresponding to the conversion reference image to obtain the corresponding desired image.

[0064] In this embodiment, the desired image conversion model is trained using the image conversion model training method described above to ensure that the desired image conversion model has image domain conversion capability and the ability to preserve and transfer image semantic structure information.

[0065] Therefore, by executing the above steps S310 to S320, this application can use the trained desired image conversion model to perform stable image domain conversion processing on any image to be converted according to the reference conversion image, so that the final desired image can achieve the expected image domain effect while having good image edge contour clarity, and can maintain an image structure as similar as possible to the reference conversion image and the image to be converted.

[0066] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0067] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. If the various functions provided in this application are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, including several instructions to cause the computer device 10 (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned readable storage medium includes: USB flash drives, mobile hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0068] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for training an image conversion model, characterized in that, The method includes: Multiple training image samples are acquired and sequentially input into the image conversion model to be trained for image conversion; wherein, each training image sample includes one source domain sample image and one target domain sample image; During the image transformation process of the image transformation model to be trained for each training image sample, a self-supervised visual model is invoked to extract feature heatmaps of the source domain sample image, target domain sample image, target domain generated image, and source domain generated image associated with the training image sample; wherein the target domain generated image is obtained by the image transformation model to be trained transforming the source domain sample image into the target domain, and the source domain generated image is obtained by the image transformation model to be trained transforming the target domain sample image into the source domain; Based on the feature heatmaps of the source domain sample image, the target domain sample image, the target domain generated image, and the source domain generated image, a heatmap consistency loss function is constructed for the image conversion model to be trained. To minimize the target loss function of the image conversion model to be trained, the model parameters of the image conversion model to be trained are optimized, wherein the target loss function is related to the heatmap consistency loss function; When the target loss function is minimized, the image conversion model to be trained is pruned to obtain the corresponding desired image conversion model.

2. The method according to claim 1, characterized in that, The step of constructing a heatmap consistency loss function for the image transformation model to be trained based on the feature heatmaps of the source domain sample image, the target domain sample image, the target domain generated image, and the source domain generated image includes: Based on the feature heatmaps of the source domain sample image and the target domain generated image, a first Dice loss function, a first crossover ratio (CRO) loss function, and a first edge preservation loss function are constructed for these two feature heatmaps. Based on the feature heatmaps of the target domain sample image and the source domain generated image, a second Dice loss function, a second crossover ratio (CRO) loss function, and a second edge preservation loss function are constructed for these two feature heatmaps. The heatmap consistency loss function is obtained by performing a weighted summation operation on the first Dice loss function, the first intersection-over-union (IoU) loss function, the first edge-preserving loss function, the second Dice loss function, the second IoU loss function, and the second edge-preserving loss function.

3. The method according to claim 2, characterized in that, For any real sample image in the source domain sample image and the target domain sample image, the Dice loss function between the real sample image and the corresponding model-generated image is expressed by the following formula: ； The cross-union loss function between the real sample image and the corresponding model-generated image is expressed by the following formula: ； The edge-preserving loss function between the real sample image and the corresponding model-generated image is expressed by the following formula: ； in, The Dice loss function is used to represent the difference between the real sample image and the corresponding model-generated image. The cross-union loss function is used to represent the intersection-over-union ratio between the real sample image and the corresponding model-generated image. This is used to represent the edge-preserving loss function between the real sample image and the corresponding model-generated image. A feature heatmap used to represent the real sample image. A feature heatmap used to represent the model-generated image corresponding to the real sample image. and These are used to represent the horizontal and vertical directions of the image, respectively. For characteristic heatmap Image gradient along the horizontal axis, For characteristic heatmap Image gradient along the horizontal axis, For characteristic heatmap Image gradient along the vertical axis, For characteristic heatmap Image gradient along the vertical axis, Used to represent the L1 norm; the model-generated image corresponding to the source domain sample image is the target domain generated image; the model-generated image corresponding to the target domain sample image is the source domain generated image.

4. The method according to claim 1, characterized in that, The step of cropping the image conversion model to obtain the corresponding desired image conversion model includes: Extract the image generator from the image conversion model to be trained, and use the extracted image generator as the desired image conversion model.

5. The method according to any one of claims 1-4, characterized in that, The target loss function of the image conversion model to be trained is obtained by weighted summation of the original model architecture loss function and the heatmap consistency loss function of the image conversion model to be trained, wherein the original model architecture loss function matches the model architecture type of the image conversion model to be trained.

6. The method according to claim 5, characterized in that, When the model architecture type of the image conversion model to be trained is a generative adversarial network or a deep convolutional generative adversarial network, the original model architecture loss function of the image conversion model to be trained is an adversarial loss function; wherein the adversarial loss function It can be expressed as follows: ； in, This represents the probability of the image domain authenticity judgment of the image transformation model to be trained for the input source domain sample image or target domain sample image. This represents the probability of image domain authenticity discrimination for the image generated by the image conversion model to be trained, based on the converted target domain or source domain. for The logarithm of for The logarithm of Used to represent mathematical expectation solution functions.

7. The method according to claim 6, characterized in that, When the model architecture type of the image conversion model to be trained is a recurrent consistency generative adversarial network, the original model architecture loss function of the image conversion model to be trained is obtained by weighted summation of the adversarial loss function and the recurrent consistency loss function; Wherein, the adversarial loss function It can be expressed as follows: ； in, This represents the probability of the image domain authenticity judgment of the image transformation model to be trained for the input source domain sample image or target domain sample image. This represents the probability of image domain authenticity discrimination for the image generated by the image conversion model to be trained, based on the converted target domain or source domain. for The logarithm of for The logarithm of Used to represent mathematical expectation solution functions; The cycle consistency loss function It can be expressed as follows: ； in, Used to represent the source domain sample image input to the image transformation model to be trained; Used to represent the target domain sample image input to the image transformation model to be trained; The source domain reconstructed image is used to represent the image that matches the source domain sample image. It is obtained by reconstructing the source domain from the target domain generated image of the source domain sample image. The target domain reconstructed image, which is used to represent the target domain sample image, is obtained by reconstructing the target domain from the source domain generated image of the target domain sample image. Used to represent the L1 norm.

8. An image conversion processing method, characterized in that, The method includes: Obtain the image to be converted and the reference image for conversion; The image to be converted and the conversion reference image are input into the desired image conversion model, and the desired image conversion model converts the actual image domain of the image to be converted to the target image domain corresponding to the conversion reference image to obtain the corresponding desired image; wherein, the desired image conversion model is trained using the image conversion model training method described in any one of claims 1-7.

9. A computer device, characterized in that, It includes a processor and a memory, the memory storing a computer program that can be executed by the processor, the processor executing the computer program to implement the image conversion model training method according to any one of claims 1-7, or to implement the image conversion processing method according to claim 8.

10. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a computer device, it implements the image conversion model training method according to any one of claims 1-7, or the image conversion processing method according to claim 8.

Images