Image processing model training method and apparatus
By acquiring the first, second, and third sample images from the training data, using feature map extraction to determine similarity, and constructing a block response loss, the problem of lack of paired sample images in the image style transfer model is solved, achieving a more accurate style transfer effect and generating more realistic and clearer images.
Patent Information
- Application Number
- CN202211164717.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-23
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-09-23
AI Technical Summary
Existing image style transfer models suffer from a lack of paired sample images, resulting in a completely unsupervised learning and training process. This leads to inaccurate style transfer results, making the resulting fused images less realistic, clear, and natural.
By acquiring the first, second, and third sample images from the training data, similarity is determined using feature map extraction, and a block response loss is constructed to guide model training, ensuring accurate transfer of the content features of the target image to the style features of the reference image.
It improves the accuracy of image style transfer, making the fused images more realistic, clear, and natural, overcoming the problem of lack of paired sample images, and avoiding completely unsupervised learning.
Smart Images

Figure CN117274035B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of image processing, and more specifically to an image processing model training method and apparatus, an image processing method, a computing device, a computer-readable storage medium, and a computer program product. Background Technology
[0002] With the development of deep learning and the emergence of large-scale image datasets, many automated image processing methods have appeared and received widespread attention, including image style transfer. However, related image style transfer models and methods have some drawbacks, such as: the lack of paired sample images, which leads to a completely unsupervised learning and training process; the style transfer effect is not accurate enough, making the resulting fused image less realistic, clear, and natural; and so on. These problems and drawbacks limit the popularization and development of image style transfer. Summary of the Invention
[0003] In view of the above, this disclosure provides an image processing model training method and apparatus, an image processing method, a computing device, a computer-readable storage medium, and a computer program product, thereby alleviating, reducing, or even eliminating some or all of the above-mentioned problems, as well as other possible problems.
[0004] According to one aspect of this disclosure, an image processing model training method is proposed, comprising: acquiring training data, the training data including a first sample image, a second sample image, and a third sample image, wherein the first sample image and the third sample image have the same content features; inputting the first sample image and the second sample image into a preset image processing model to obtain a first target image that integrates the content features of the first sample image and the style features of the second sample image; determining a first similarity between the second sample image and the third sample image through feature map extraction; determining a first loss of the image processing model based on the first similarity, the second sample image, and the first target image; determining a target loss of the image processing model based at least on the first loss; and iteratively updating the parameters of the image processing model based on the target loss until the target loss satisfies a preset condition.
[0005] According to some embodiments of this disclosure, determining the target loss of the image processing model based at least on the first loss includes: dividing the first target image into a plurality of first image blocks and dividing the second sample image into a plurality of second image blocks; for each first image block in the first target image, calculating a second similarity between each second image block in the second sample image and the first image block; for each first image block in the first target image, selecting a similar image block from the plurality of second image blocks based on the second similarity between each second image block in the second sample image and the first image block; determining a second loss of the image processing model based on each first image block in the first target image and its similar image blocks; and determining the target loss of the image processing model based at least on the first loss and the second loss.
[0006] According to some embodiments of this disclosure, the first image block and the second image block have the same area. Calculating the second similarity between each second image block in the second sample image and the first image block includes: for each second image block in the second sample image, calculating the Euclidean distance between the second vector corresponding to the second image block and the first vector corresponding to the first image block, wherein the second vector is composed of the pixel values of each pixel in the second image block, and the first vector is composed of the pixel values of each pixel in the first image block; and determining the second similarity based on the Euclidean distance.
[0007] According to some embodiments of this disclosure, determining the second loss of the image processing model based on each first image patch and its similar image patches in the first target image includes: for each first image patch and its similar image patches in the first target image, calculating the Manhattan distance between a first vector corresponding to each first image patch and a third vector corresponding to its similar image patches, wherein the third vector is composed of the pixel values of each pixel point of the similar image patch; and determining the second loss of the image processing model based on the sum of the Manhattan distances corresponding to all first image patches and their similar image patches.
[0008] According to some embodiments of this disclosure, determining the target loss of the image processing model based at least on the first loss includes: inputting the first sample image and the third sample image into a preset image processing model to obtain a second target image that integrates the content features of the first sample image and the style features of the third sample image; determining a third loss of the image processing model based on the third sample image and the second target image; and determining the target loss of the image processing model based at least on the first loss and the third loss.
[0009] According to some embodiments of this disclosure, determining the target loss of the image processing model based at least on the first loss and the second loss includes: inputting the first sample image and the third sample image into a preset image processing model to obtain a second target image that integrates the content features of the first sample image and the style features of the third sample image; determining a third loss of the image processing model based on the third sample image and the second target image; and determining the target loss of the image processing model based at least on the first loss, the second loss, and the third loss.
[0010] According to some embodiments of this disclosure, determining the target loss of the image processing model based at least on the first loss, the second loss, and the third loss includes: determining a first adversarial loss of the image processing model based on the second target image; determining a second adversarial loss of the image processing model based on the first target image; determining a joint block adversarial loss of the image processing model based on the first target image and the second sample image; determining a content loss of the image processing model based on the first sample image and the first target image; and determining the target loss of the image processing model based on at least one of the first adversarial loss, the second adversarial loss, the joint block adversarial loss, and the content loss, as well as the first loss, the second loss, and the third loss.
[0011] According to some embodiments of this disclosure, determining the target loss of the image processing model based on at least one of the first adversarial loss, the second adversarial loss, the joint block adversarial loss, and the content loss, as well as the first loss, the second loss, and the third loss, includes: determining the target loss as a weighted sum of the first loss, the second loss, the third loss, the first adversarial loss, the second adversarial loss, the joint block adversarial loss, and the content loss.
[0012] According to some embodiments of this disclosure, determining the first similarity between the second sample image and the third sample image through feature map extraction includes: extracting a first feature map and a second feature map from the second sample image and the third sample image, respectively; calculating a relationship matrix between the first feature map and the second feature map; and determining the first similarity between the second sample image and the third sample image based on the relationship matrix.
[0013] According to some embodiments of this disclosure, determining the first loss of the image processing model based on the first similarity, the second sample image, and the first target image includes: determining a pseudo-label image corresponding to the first target image based on the second sample image and the first similarity; and determining the first loss of the image processing model based on the first target image and the pseudo-label image.
[0014] According to some embodiments of this disclosure, the acquisition of training data includes: acquiring the second sample image and the third sample image; extracting content features of the third sample image from the third sample image; and generating the first sample image based on the content features of the third sample image.
[0015] According to some embodiments of this disclosure, the second sample image includes a first cartoon image, the third sample image includes a second cartoon image, and the first sample image includes a line drawing corresponding to the second cartoon image.
[0016] According to some embodiments of this disclosure, the image processing model includes a content feature extractor, a style feature extractor, and a content-style fusion unit. The step of inputting the first sample image and the second sample image into a preset image processing model to obtain a first target image that fuses the content features of the first sample image and the style features of the second sample image includes: inputting the first sample image into the content feature extractor to obtain the content features of the first sample image; inputting the second sample image into the style feature extractor to obtain the style features of the second sample image; and inputting the content features of the first sample image and the style features of the second sample image into the content-style fusion unit to obtain a first target image that fuses the content features of the first sample image and the style features of the second sample image.
[0017] According to another aspect of this disclosure, an image processing method is proposed, comprising: acquiring a first image and a second image to be processed; inputting the first image and the second image into an image processing model pre-trained using an image processing model training method according to some embodiments of this disclosure, so as to obtain a first fused image that integrates the content features of the first image and the style features of the second image.
[0018] According to some embodiments of this disclosure, the image processing method further includes: acquiring a third image to be processed; inputting the first image and the third image into the image processing model to obtain a second fused image that integrates the content features of the first image and the style features of the third image; and performing interpolation processing on the first fused image and the second fused image to obtain a third fused image that integrates the content features of the first image, at least a portion of the style features of the second image, and at least a portion of the style features of the third image.
[0019] According to another aspect of this disclosure, an image processing method is proposed, comprising: acquiring an original image to be processed and a first reference image and a second reference image; inputting the original image into a content feature extractor of an image processing model pre-trained using an image processing model training method according to some embodiments of this disclosure to obtain content features of the original image; inputting the first reference image and the second reference image into a style feature extractor of the pre-trained image processing model to obtain a first style feature of the first reference image and a second style feature of the second reference image; synthesizing the first style feature and the second style feature into a third style feature that integrates at least a portion of the first style feature and at least a portion of the second style feature by interpolation; and inputting the content features of the original image and the third style feature into a content style fusion unit of the pre-trained image processing model to obtain a target fused image that integrates the content features of the original image and the third style feature.
[0020] According to another aspect of this disclosure, an image processing model training apparatus is proposed, comprising: a training data acquisition module configured to acquire training data, the training data including a first sample image, a second sample image, and a third sample image, wherein the first sample image and the third sample image have the same content features; an input module configured to input the first sample image and the second sample image into a preset image processing model to obtain a first target image that integrates the content features of the first sample image and the style features of the second sample image; a first determination module configured to determine a first similarity between the second sample image and the third sample image through feature map extraction; a second determination module configured to determine a first loss of the image processing model based on the first similarity, the second sample image, and the first target image; a third determination module configured to determine a target loss of the image processing model based at least on the first loss; and an iteration module configured to iteratively update the parameters of the image processing model based on the target loss until the target loss satisfies a preset condition.
[0021] According to another aspect of this disclosure, a computing device is proposed, comprising: a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform an image processing model training method according to some embodiments of this disclosure.
[0022] According to another aspect of this disclosure, a computer-readable storage medium is proposed that stores computer-readable instructions thereon, which, when executed, implement an image processing model training method according to some embodiments of this disclosure.
[0023] According to another aspect of this disclosure, a computer program product is proposed, comprising a computer program that, when executed by a processor, implements an image processing model training method according to some embodiments of this disclosure.
[0024] The goal of the image processing model training method and apparatus according to some embodiments of this disclosure is to transfer the style features of a reference image to a target image while preserving the content features of the target image. During model training, when collecting training samples, in addition to the sample image pairs consisting of the target image sample (i.e., the first sample image) and the reference image sample (i.e., the second sample image), a third sample image with the same content features as the target image sample is introduced. This third sample image, together with the reference image sample, is used to obtain pseudo-sample labels through image feature extraction and similarity calculation, thereby constructing a block response loss (i.e., the first loss) to guide model training. On the one hand, obtaining such pseudo-sample labels can overcome the problem of lacking paired sample images (i.e., as input samples: the target image and the reference image, and as the fused image as the label) in related technologies, thus avoiding the completely unsupervised learning and training process caused by this. On the other hand, the block response loss can more accurately guide the position matching of the style features of the reference image in the target image, thereby ensuring the accuracy of the style transfer effect of the trained model, and making the fused image obtained by the model more realistic, clear, and natural.
[0025] These and other aspects of this application will be clearly understood from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description
[0026] The various aspects, features, and advantages of this disclosure will be readily understood from the following detailed description and accompanying drawings, in which: Figure 1 An example implementation environment of an image processing model training method according to some embodiments of the present disclosure is illustrated schematically; Figure 2 A flowchart illustrating an image processing model training method according to some embodiments of the present disclosure is shown schematically. Figure 3 A structural block diagram of an image processing model according to some embodiments of the present disclosure is shown schematically; Figure 4A An example structural block diagram of an image processing model according to some embodiments of the present disclosure is shown schematically; Figure 4B A schematic diagram illustrating the principle of determining a first loss for an image processing model according to some embodiments of the present disclosure is shown. Figure 5 A flowchart illustrating the determination of the target loss of an image processing model according to some embodiments of the present disclosure is shown schematically. Figure 6 A comparative diagram illustrating the use of an image processing model according to some embodiments of the present disclosure and the use of related techniques to determine a second loss is shown. Figure 7A A flowchart illustrating an image processing method according to some embodiments of the present disclosure is shown schematically; Figure 7B A flowchart of another image processing method according to some embodiments of the present disclosure is illustrated schematically; Figures 8A-8F The diagram illustrates the effect of using an image processing method according to some embodiments of the present disclosure. Figure 9 An example block diagram of an image processing model training apparatus according to some embodiments of the present disclosure is shown schematically; Figure 10 An example block diagram of a computing device according to some embodiments of the present disclosure is shown schematically.
[0027] It should be noted that the above-mentioned figures are merely schematic and illustrative, and are not necessarily drawn to scale. Detailed Implementation
[0028] Several embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings to enable those skilled in the art to implement this disclosure. This disclosure may be embodied in many different forms and for various purposes and should not be limited to the embodiments set forth herein. These embodiments are provided to make this disclosure comprehensive and complete, and to fully convey the scope of this disclosure to those skilled in the art. The embodiments described do not limit this disclosure.
[0029] It will be understood that although the terms first, second, third, etc., may be used herein to describe various elements, components, and / or parts, these elements, components, and / or parts should not be limited by these terms. These terms are used only to distinguish one element, component, or part from another. Therefore, the first element, component, or part discussed below may be referred to as the second element, component, or part without departing from the teachings of this disclosure.
[0030] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprising” and / or “including” as used in this specification designate the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.
[0031] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.
[0032] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is to say, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.
[0033] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all promotional information and operations / steps, nor do they necessarily need to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.
[0034] Those skilled in the art will understand that the accompanying drawings are merely example interfaces of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing this application, and therefore cannot be used to limit the scope of protection of this application.
[0035] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, intelligent transportation, and automatic control.
[0036] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, instruction-based learning, and active learning.
[0037] Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.
[0038] Before detailing the embodiments of this disclosure, some related concepts will be explained for clarity.
[0039] 1. Content Features: The content features of an image can include its semantic information, outline, shape, etc. Taking comics as an example, the content features of a comic can be its line drawing (or a simplified version of the comic), which only contains black lines.
[0040] 2. Stylistic Features: The stylistic features of an image can include its color, texture, and style. Taking comics as an example, a stylistic feature of a comic can be its screentone, which is a series of gray-toned dots used to represent the comic's style. A single comic may have multiple screentone styles.
[0041] 3. Image style transfer: This refers to transferring style features, texture features, etc., from one image to another target image. The challenge lies in distinguishing between the texture features and content features of the original image. During style transfer, the synthesized image needs to retain both the texture of the style image and the content of the target image. In other words, image style transfer is the process of fusing a content image and a style image to generate a new image; the generated new image simultaneously possesses the content features and style features of the specified image.
[0042] Figure 1 An example implementation environment 100 of an image processing model training method according to some embodiments of the present disclosure is illustrated schematically. Figure 1 As shown, the implementation environment 100 may include a terminal device 110 and a server 120, as well as a network 130 for connecting the terminal device 110 and the server 120. In some embodiments, the terminal device 110 may be used to implement an image processing model training method according to the present disclosure. For example, the terminal device 110 may be deployed with corresponding programs or instructions for performing various methods provided in the present disclosure. Optionally, the server 120 may also be used to implement various methods according to the present disclosure.
[0043] Terminal device 110 and third-party terminal device 140 can be any type of mobile computing device, including mobile computers (e.g., Microsoft® Surface® devices, personal digital assistants (PDAs), laptops, notebook computers, and devices such as Apple iPads). TM Tablet computers, netbooks, etc. Figure 1 The mobile phones shown (e.g., cellular phones, smartphones such as Microsoft Windows® phones, Apple iPhones, and devices implementing Google® Android) TMOperating systems such as phones, Palm® devices, Blackberry® devices, etc.), and wearable computing devices (such as smartwatches, head-mounted devices, including smart glasses such as Google® Glass). TM Terminal device 110 may be a fixed computing device, such as a desktop computer, game console, smart TV, etc., or other types of mobile devices. In some embodiments, terminal device 110 may also be a fixed computing device, such as a desktop computer, game console, smart TV, etc.
[0044] Server 120 can be a single server or a server cluster, or it can be a cloud server or cloud server cluster capable of providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. It should be understood that the server mentioned herein is typically a server computer with ample storage and processor resources, but other embodiments are also possible. Optionally, server 120 can also be a regular desktop computer, including a host, monitor, etc.
[0045] Examples of network 130 include combinations of local area networks (LANs), wide area networks (WANs), personal area networks (PANs), and / or communication networks such as the Internet. Server 120 and terminal device 110 may include at least one communication interface (not shown) capable of communicating via network 130. Such a communication interface may be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), wired or wireless (such as an IEEE 802.11 wireless LAN (WLAN)) wireless interface, Wi-MAX interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth. TM Interfaces, near field communication (NFC) interfaces, etc.
[0046] like Figure 1As shown, terminal device 110 may include a display screen 111, and terminal users can interact with terminal application 112 via the display screen 111. Terminal device 110 may interact with server 120, for example, via network 130, such as sending data to or receiving data from it. Terminal application 112 may be a local application, a web application, or a lightweight application (LiteApp, such as a mobile app or WeChat app). If the terminal application is a local application that requires installation, terminal application 112 can be installed on terminal device 110. If terminal application 112 is a web application, it can be accessed through a browser. If terminal application 112 is an app, it can be opened directly on terminal device 110 by searching for relevant information about the terminal application (such as the name of the terminal application) or scanning the graphic code of the terminal application (such as a barcode or QR code), without the need to install terminal application 112.
[0047] Figure 1 The example implementation environment is merely illustrative, and the image processing model training method according to this disclosure is not limited to the example implementation environment and server shown. It should be understood that although server 120 and terminal device 110 are shown and described as separate structures herein, they can be different components of the same computing device. Optionally, all steps of the image processing model training method according to some embodiments of this disclosure can also be implemented on the server 120 side, or can be implemented jointly on the terminal device 110 side and the server 120 side.
[0048] Figure 2 A flowchart illustrating an image processing model training method according to some embodiments of the present disclosure is shown schematically. In some embodiments, such as Figure 1 As shown, the image processing model training method according to this disclosure can be executed on the terminal device 110 side. In other embodiments, the image processing model training method according to this disclosure can also be executed by a combination of server 120 and terminal device 110.
[0049] like Figure 2 As shown, the image processing model training method according to some embodiments of this disclosure may include: S210, acquire training data, the training data including a first sample image, a second sample image and a third sample image, wherein the first sample image and the third sample image have the same content features; S220, the first sample image and the second sample image are input into a preset image processing model to obtain a first target image that integrates the content features of the first sample image and the style features of the second sample image; S230, determine the first similarity between the second sample image and the third sample image by feature map extraction; S240, based on the first similarity, the second sample image, and the first target image, determine the first loss of the image processing model; S250, at least based on the first loss, determine the target loss of the image processing model; S260, Based on the target loss, the parameters of the image processing model are iteratively updated until the target loss meets the preset conditions.
[0050] The following is combined Figure 3 Detailed description of steps S210-S260, wherein Figure 3 A structural block diagram of an image processing model according to some embodiments of the present disclosure is shown schematically.
[0051] In step S210, training data is acquired, which includes a first sample image 310, a second sample image 320, and a third sample image 330, wherein the first sample image 310 and the third sample image 330 have the same content features.
[0052] In some embodiments, acquiring training data may include: acquiring the second sample image 320 and the third sample image 330; extracting content features from the third sample image 330; and generating the first sample image 310 based on the content features of the third sample image. This ensures that the acquired first sample image 310 and the third sample image 330 have the same content features.
[0053] In some embodiments, the second sample image 320 may include a first cartoon image, the third sample image 330 may include a second cartoon image, and the first sample image 310 may include a line drawing corresponding to the second cartoon image. In some embodiments, the first sample image 310, the second sample image 320, and the third sample image 330 may also include other types of images, such as (but not limited to) pictures, photographs, drawings, color illustrations, etc. That is, the images described in this disclosure can be images understood by those skilled in the art in a general sense, and this disclosure does not limit the type of image.
[0054] Therefore, we will continue to combine the following. Figure 4A and Figure 4B Detailed description of steps S210-S260, wherein Figure 4A An example structural block diagram of an image processing model according to some embodiments of the present disclosure is schematically shown, and Figure 4BA schematic diagram illustrating the principle of determining a first loss for an image processing model according to some embodiments of the present disclosure is provided. For purposes of ease of description and clarity, cartoon illustrations and line drawings may also be used hereinafter to represent the corresponding sample images, and images with the same meaning are referred to by the same reference numerals, but this in no way implies that the images described in this disclosure are limited to the field of cartoons.
[0055] In step S220, the first sample image 310 and the second sample image 320 are input into a preset image processing model 400 to obtain a first target image 312 that integrates the content features of the first sample image 310 and the style features of the second sample image 320.
[0056] In some embodiments, the image processing model 400 may include a content feature extractor 410, a style feature extractor 420, and a content-style fusion unit 430. The content feature extractor 410 and style feature extractor 420 can be considered as feature extraction networks (e.g., linear networks, CNNs, RNNs, or other more complex network structures), typically using pooling layers to gradually reduce the size of the input data. The content-style fusion unit 430 can also be considered as a neural network, using upsampling / deconvolutional layers to gradually recover the details and corresponding spatial dimensions of the target image. The content feature extractor 410, style feature extractor 420, and content-style fusion unit 430 can be the same network or different networks. In some embodiments, the content feature extractor 410 may include, for example, a content encoder (CEnc) for encoding the content features of the first sample image 310; the style feature extractor 420 may include, for example, a style encoder (SEnc) for encoding the style features of the second sample image 320; and the content-style fusion unit 430 may include, for example, a stylegram integration decoder (SIDec) for decoding the combined feature encodings to obtain the target image. Optionally, in some embodiments, the image processing model 400 may also include, for example, other types of encoders / decoders, and the number of encoders / decoders is not limited.
[0057] In some embodiments, step S220 may include: inputting a first sample image 310 into a content feature extractor 410 to obtain content features of the first sample image 310; inputting a second sample image 320 into a style feature extractor 420 to obtain style features of the second sample image 320; and inputting the content features of the first sample image 310 and the style features of the second sample image 320 into a content-style fusion processor 430 to obtain a first target image 312 that fuses the content features of the first sample image 310 and the style features of the second sample image.
[0058] In some embodiments, the style feature encoding of the sample image extracted by the style feature extractor 420 may not include spatial location information; that is, the extracted style feature encoding of the sample image may be one-dimensional and location-independent, which makes the sampling area unaffected by location information. Optionally, in some embodiments, the style feature encoding of the sample image extracted by the style feature extractor 420 may include spatial location information, depending on the training objective and the specific application scenario.
[0059] For example, for the second sample image 320, a style encoder can be used to obtain a one-dimensional style feature encoding through feature mapping, global average pooling, and 1x1 convolution. This one-dimensional style feature encoding is then modulated and convolved according to multiple weights and normalized. Then, the content style fusion unit 430 fuses these normalized feature encodings with the two-dimensional content feature encoding of the first sample image 310 extracted by the content feature extractor 410. Using multiple weights for modulation can convey more information, allowing these feature encodings to be better fused in the content style fusion unit 430. Specifically, given a second sample image 320... (H and W represent the height and width of the image, respectively), the style encoder can output a style feature code. (That is, here it is a one-dimensional vector, and d represents the length of the vector), then we can get the following formula: Where y2 represents the second sample image 320, and SEnc indicates processing using a style encoder. Since style feature encoding needs to be position-independent, it can be obtained through convolutional layers without padding, global pooling layers, and fully connected layers. After extracting the content feature encoding from the content encoder, the style hybrid decoder can receive both the style feature encoding and the content feature encoding as input, and then decode them to obtain the first target image 312.
[0060] In step S230, the first similarity 470 between the second sample image 320 and the third sample image 330 is determined by feature map extraction.
[0061] In some embodiments, step S230 may include: extracting a first feature map 451 and a second feature map 452 from the second sample image 320 and the third sample image 330, respectively; calculating a relationship matrix 460 between the first feature map 451 and the second feature map 452; and determining a first similarity 470 between the second sample image 320 and the third sample image 330 based on the relationship matrix 460.
[0062] Calculating the first similarity 470 (also known as the mapping relationship) between the second sample image 320 and the third sample image 330 directly based on the first sample image 310 is very difficult. Moreover, an image can typically contain multiple style features (e.g., style patterns applied to hair and face are usually different). Therefore, in some embodiments, a block response method can be used to calculate the first similarity 470 between the second sample image 320 and the third sample image 330, while ensuring the consistency of style features between the sample images and the target image.
[0063] like Figure 4B As shown, the second sample image 320 and the third sample image 330 can be input into the neural network 440, respectively. The neural network 440 can include a recurrent neural network (e.g., a long short-term memory recurrent neural network), a feedforward neural network, a convolutional neural network, or other forms of neural networks, as long as it can extract feature maps. Here, the neural network 440 can be, for example, VGG-16 (Visual Geometry Group-16), because feature maps extracted by pre-trained VGG-16 generally model the response well. The feature map output from the ReLU4-1 layer in VGG-16 is taken. As a feature map for computation, the receptive field of each unit in this feature map is 15×15. The receptive field is defined as the size of the region mapped on the input image by a pixel on the feature map output by each layer of the convolutional neural network; in other words, it is the region on the input image corresponding to a point on the feature map. A first feature vector 461 is extracted from the first feature map 451, and a second feature vector 462 is extracted from the second feature map 452. Then, based on the first feature vector 461 and the second feature vector 462, a relational matrix can be calculated between the first feature map 451 of the second sample image 320 and the second feature map 452 of the third sample image 330 using the following formula. : in and Let represent the u-th position of the first feature map 451 and the v-th position of the second feature map 452, respectively, and T represent the transpose of the matrix. Then, the softmax function can be used to normalize the resulting relation matrix 460. Afterwards, the first similarity 470 between the second sample image 320 and the third sample image 330 can be determined based on the normalized relation matrix 460, where the degree of similarity is represented by the intensity of color; darker colors indicate greater similarity.
[0064] At step S240, a first loss of the image processing model 400 is determined based on the first similarity 470, the second sample image 320, and the first target image 312.
[0065] To ensure the accuracy and controllability of the image processing model 400, some loss functions are needed to guarantee the training of the image processing model 400 to minimize the loss. Examples of these loss functions are illustrated below with reference to the accompanying figures.
[0066] In some embodiments, step S240 may include: determining a pseudo-label image 321 corresponding to the first target image 312 based on the second sample image 320 and the first similarity 470; and determining a first loss of the image processing model 400 based on the first target image 312 and the pseudo-label image 321.
[0067] like Figure 4B As shown, image scaling can use downsampling to generate a thumbnail 320' of the second sample image 320. For example, the second sample image 320 can be scaled to 64×64 pixels (thumbnail 320') and a third feature vector 463 can be extracted from it, such that the third feature vector 463 has the same size as the feature vector of the feature map extracted above. Then, the normalized relation matrix 460 and the downsampled... (The second sample image 320) is multiplied to obtain (Pseudo-label image 321): In other words, in Each pixel in is Based on relation matrix arrive A linear combination. In this way, the aligned reference diagram is... Maintain Content features of (third sample image 330) and The stylistic characteristics. Next, Treating it as a pseudo-real image to guide the prediction of image processing model 400, a first loss (or block response loss) is constructed: Where G represents the mapping function, corresponding to image processing model 400; x1 represents the first sample image 310, and G(x1,y2) represents the first target image 312; it should be noted that the corresponding downsampling processing of G(x1,y2) is to match the mapping function. Maintain consistency.
[0068] At step S250, the target loss of the image processing model 400 is determined based at least on the first loss.
[0069] The target loss refers to an overall loss during the training process using the image processing model 400. This target loss may include only the first loss or other types of losses (which will be discussed below). Therefore, the target loss can be determined by assigning a corresponding weight to each loss and then using the weighted sum of all losses as the target loss, where the weights represent the importance of the respective losses.
[0070] In some embodiments, step S250 may include: inputting the first sample image 310 and the third sample image 330 into a preset image processing model 400 to obtain a second target image 313 (e.g., fused with the content features of the first sample image 310 and the style features of the third sample image 330). Figure 3 and Figure 4A (as shown in the figure); determine the third loss (not shown) of the image processing model 400 based on the third sample image 330 and the second target image 313; determine the target loss of the image processing model 400 based at least on the first loss and the third loss.
[0071] The third loss can be understood as follows: if the image processing model 400 receives paired training data... and As input, the output should be and While the processes are the same, a loss inevitably occurs during reconstruction. Here, Euclidean distance can be used to calculate the third loss (also known as pixel-level loss, meaning the third loss is determined based on the pixel value loss of each corresponding pixel in the second target image 313 relative to the third sample image 330), as shown in the following formula: The function G has the meaning as described above, where y1 represents the third sample image 330 and G(x1, y1) represents the second target image 313.
[0072] In some embodiments, step S250 may also be combined with Figure 4A The second loss shown is relevant. This will be discussed in conjunction with the following text. Figure 5 and Figure 6 To explain in more detail.
[0073] In step S260, based on the target loss, the parameters of the image processing model 400 are iteratively updated until the target loss meets the preset conditions.
[0074] In some embodiments, in the initial stage of optimization, a larger weight and iteration step size can be used to allow the parameters to converge quickly to the vicinity of the optimal solution. Subsequently, the weights are gradually reduced while the iteration step size is decreased, allowing the parameters to converge to the optimal solution slowly and accurately. The process is considered complete when the maximum number of iterations is reached or the target loss no longer decreases after several optimizations, thus completing the training of the image processing model 400. It should be understood that common functions (such as the sigmoid function) can be used for model iteration, or any other method can be used, as long as the loss value of the loss function gradually decreases. This disclosure does not exhaustively list all such methods. This process can be viewed as a supervised (semi-supervised) training process.
[0075] Thus, by utilizing the image processing model training method and image processing method according to some embodiments of this disclosure, when collecting training samples during model training, in addition to the sample image pair consisting of the target image sample (i.e., the first sample image) and the reference image sample (i.e., the second sample image), a third sample image with the same content features as the target image sample is introduced. This third sample image is used together with the reference image sample to obtain pseudo-sample labels through image feature extraction and similarity calculation, thereby constructing a block response loss (i.e., the first loss) to guide the training of the model. On the one hand, the acquisition of such pseudo-sample labels can overcome the problem of lack of paired sample images (i.e., as input samples: the target image and the reference image, and as the fused image as the label) in related technologies, thereby avoiding the completely unsupervised learning and training process caused by this. On the other hand, the block response loss can more accurately guide the position matching of the style features of the reference image in the target image, thereby ensuring the accuracy of the style transfer effect of the trained model, and making the fused image obtained by the model more realistic, clear, and natural.
[0076] Figure 5 A flowchart illustrating the determination of the target loss of an image processing model according to some embodiments of the present disclosure is shown schematically.
[0077] like Figure 5 As shown, step S250 described above may further include: S510, the first target image is divided into multiple first image blocks and the second sample image is divided into multiple second image blocks; S520, for each first image block in the first target image, calculate the second similarity between each second image block in the second sample image and the first image block; S530, for each first image block in the first target image, based on the second similarity between each second image block in the second sample image and the first image block, select a similar image block from the plurality of second image blocks; S540, determine the second loss of the image processing model based on each first image patch and its similar image patches in the first target image; S550, at least based on the first loss and the second loss, determine the target loss of the image processing model.
[0078] The following is combined Figure 4A and Figure 6 Detailed description of steps S510-S550, wherein Figure 6 A comparative diagram illustrating the use of an image processing model according to some embodiments of the present disclosure and the use of related techniques to determine a second loss is shown.
[0079] In step S510, the first target image 312 is divided into multiple first image blocks and the second sample image 320 is divided into multiple second image blocks.
[0080] like Figure 4A As shown in the upper right corner, the multiple first image blocks and multiple second image blocks are represented by small squares. In some embodiments, the areas of the first image blocks and the second image blocks may be the same (as illustrated). Alternatively, in some embodiments, the areas of the first image blocks and the second image blocks may not be the same.
[0081] In some embodiments, to make the image processing model 400 more refined, it may also be divided into smaller image blocks (but not single pixels, since single pixels have no content features and / or style features) to transfer more local style features. This disclosure does not limit the size of the first and second image blocks.
[0082] In step S520, for each first image block in the first target image 312, a second similarity between each second image block in the second sample image 320 and the first image block is calculated.
[0083] In some embodiments, calculating the second similarity between each second image block in the second sample image 320 and the first image block may include: for each second image block in the second sample image 320, calculating the Euclidean distance between the second vector corresponding to the second image block and the first vector corresponding to the first image block, wherein the second vector is composed of the pixel values of each pixel point of the second image block, and the first vector is composed of the pixel values of each pixel point of the first image block; and determining the second similarity based on the Euclidean distance.
[0084] In some embodiments, Euclidean distance can be viewed as the distance between n-dimensional vectors. Therefore, the method for calculating the Euclidean distance may include: calculating the one-dimensional distance between the pixel value of a pixel at a position in the second vector and the pixel value of the corresponding pixel at a position in the first vector, and then combining all the one-dimensional distances as the Euclidean distance between the second vector and the first vector.
[0085] In step S530, for each first image block in the first target image 312, a similar image block to the first image block is selected from the plurality of second image blocks based on the second similarity between each second image block in the second sample image 320 and the first image block.
[0086] To illustrate the effectiveness of the method for finding similar image patches, the following will use... Figure 6 Let's take an example to illustrate. For each first image block in the first target image 312... In the second sample image 320, find the image patch that is most similar to them. Figure 6 As shown, the problem to be solved is to find the image patch most similar to image patch (a), where image patch (a) is a first image patch selected from the first target image 312, and image patches (b), (c), and (d) are second image patches selected from different second sample images 320. Figure 6 The textures and styles of the selected image patches are only illustrative; the focus is on the Euclidean distances calculated using different methods. Related techniques, such as the Gram matrix method, are used, but the Gram matrix focuses more on measuring the correspondence between pixels. For the lattice features of an image, since the intensity is the same in a local image patch, the Gram matrix tends to overlook this. Using the Gram matrix method, the distances to the most similar image patches to image patch (a) are all relatively large (e.g., the distance between (a) and (b) is 2.31, the distance between (a) and (d) is 3.13, and the distance between (a) and (c) is 4.24). This disclosure, however, uses... Distance is a more suitable method because it is density-sensitive, so utilizing... The distances to the image patches most similar to image patch (a) are very small (e.g., the distance between (a) and (c) is 0.002, the distance between (a) and (d) is 0.004, and the distance between (a) and (b) is 0.11), which demonstrates the effectiveness and efficiency of the method disclosed herein.
[0087] In some embodiments, finding similar image patches The method can be explained by the following formula: in It is a similar image patch from the second sample image 320, p j It is each of the second image blocks in the second sample image 320. For example, in Figure 4A In the upper right corner, a pair of similar image patches have been found and connected by a thin line.
[0088] At step S540, a second loss of the image processing model 400 is determined based on each first image block and its similar image blocks in the first target image 312.
[0089] In some embodiments, step S540 may include: for each first image block and its similar image blocks in the first target image 312, calculating the Manhattan distance between the first vector corresponding to each first image block and the third vector corresponding to its similar image blocks, wherein the third vector is composed of the pixel values of each pixel point of the similar image block; and determining the second loss of the image processing model 400 based on the sum of the Manhattan distances corresponding to all first image blocks and their similar image blocks.
[0090] In some embodiments, the second loss (or pattern regularization loss) can be described by the following formula: in This represents similar image blocks to the first image block. Using this second loss, each first image block in the first target image 312 and its similar image blocks can be aligned.
[0091] At step S550, the target loss of the image processing model 400 is determined based at least on the first loss and the second loss.
[0092] In some embodiments, step S550 may include: inputting the first sample image 310 and the third sample image 330 into a preset image processing model 400 to obtain a second target image 313 (e.g., fused with the content features of the first sample image 310 and the style features of the third sample image 330). Figure 3 and Figure 4A (as shown in the figure); determine the third loss (not shown) of the image processing model 400 based on the third sample image 330 and the second target image 313; determine the target loss of the image processing model 400 based at least on the first loss, the second loss and the third loss.
[0093] The third loss here has the same meaning as above, so it will not be repeated. Based on the second loss, severe artifacts during image processing model training can be mitigated or eliminated—artifacts here can be understood as various forms of images that do not exist in the sample images but appear on the target image—thereby optimizing the image processing model training method.
[0094] In some embodiments, determining the target loss of the image processing model 400 based at least on the first loss, the second loss, and the third loss may include: determining a first adversarial loss (not shown) of the image processing model 400 based on the second target image 313; determining a second adversarial loss (not shown) of the image processing model 400 based on the first target image 312; determining a joint block adversarial loss (not shown) of the image processing model 400 based on the first target image 312 and the second sample image 320; determining a content loss (not shown) of the image processing model 400 based on the first sample image 310 and the first target image 312; and determining the target loss of the image processing model 400 based on at least one of the first adversarial loss, the second adversarial loss, the joint block adversarial loss, and the content loss, as well as the first loss, the second loss, and the third loss.
[0095] In some embodiments, the first adversarial loss can be described by the following formula: Where G(x1, y1) represents the second target image 313, D represents the discriminator of the generative adversarial network (GAN), and E represents the mathematical expectation. In some embodiments, the second adversarial loss can be described by the following formula: Where G(x1, y2) represents the first target image 312, D represents the discriminator of the generative adversarial network, and E represents the mathematical expectation. By using the first adversarial loss and the second adversarial loss, the reconstructed cartoon image and the transferred style features can be made more realistic, so that they can maintain a distribution consistent with the real cartoon image.
[0096] In some embodiments, the joint block adversarial loss can be described by the following formula: Where G(x1, y2) represents the first target image 312, y2 represents the second sample image 320, crop represents a randomly selected block, and crops represents multiple randomly selected blocks, D p Let E represent the block discriminator of the generative adversarial network, and E represent the mathematical expectation. By utilizing the joint block adversarial loss, the transfer of local style features can take into account more details, thereby making the image processing model 400 more accurate.
[0097] In some embodiments, content loss can be described by the following formula: Where G(x1, y2) represents the first target image 312, x1 represents the first sample image 310; B represents The binary mask diagram, that is, if in If a pixel is black, it is set to 1; otherwise (i.e., white), it is set to 0. By utilizing content loss, the content features in the first target image 312 and the first sample image 310 can be kept consistent. Optionally, in some embodiments, more complex constraints—not just a binary mask (i.e., not entirely black and white)—can be used to ensure the consistency of content features between the first sample image 310 and the first target image 312, since the color depth of the content features in the line drawing obtained under different conditions is not uniform.
[0098] In some embodiments, determining the target loss of the image processing model 400 based on at least one of the first adversarial loss, the second adversarial loss, the joint block adversarial loss, and the content loss, as well as the first loss, the second loss, and the third loss, may include: determining the target loss as a weighted sum of the first loss, the second loss, the third loss, the first adversarial loss, the second adversarial loss, the joint block adversarial loss, and the content loss.
[0099] In some embodiments, the weighted sum of all the above losses can be used to determine the target loss as follows: Where λ1, λ2, λ3, λ4, λ5, λ6, and λ7 represent the weights of the first loss, second loss, third loss, first adversarial loss, second adversarial loss, joint block adversarial loss, and content loss, respectively. In actual image processing model training, the weights can be balanced according to different objectives and application scenarios.
[0100] Therefore, iteratively updating the parameters of the image processing model based on the target loss during training can make the trained image processing model more accurate and efficient. At the same time, because various possible losses are taken into account, serious artifacts will not be generated during style transfer, and the resulting fused image can be realistic, clear and natural enough.
[0101] Utilize combination Figures 2-6 The image processing model training method described above can obtain, for example: Figure 7A and Figure 7B The image processing method shown is as follows. Among them, Figure 7A A flowchart illustrating an image processing method according to some embodiments of the present disclosure is shown schematically, and Figure 7B A flowchart of another image processing method according to some embodiments of the present disclosure is shown schematically.
[0102] like Figure 7A As shown, an image processing method according to some embodiments of this disclosure may include: S710a, acquire the first and second images to be processed; S720a, the first image and the second image are input into an image processing model pre-trained using an image processing model training method according to some embodiments of the present disclosure, so as to obtain a first fused image that integrates the content features of the first image and the style features of the second image.
[0103] In some embodiments, an image processing method according to some embodiments of this disclosure may further include: acquiring a third image to be processed; inputting the first image and the third image into the image processing model to obtain a second fused image that integrates the content features of the first image and the style features of the third image; and performing interpolation processing on the first fused image and the second fused image to obtain a third fused image that integrates the content features of the first image, at least a portion of the style features of the second image, and at least a portion of the style features of the third image. Image interpolation processing methods may include, for example, nearest neighbor interpolation, bilinear interpolation, bisquare interpolation, bicubic interpolation, and other higher-order interpolation methods.
[0104] In some embodiments, the number of images to be processed can be more than three. Taking four reference images as an example: after obtaining the first and second fused images, a fourth image to be processed is also obtained; the first and fourth images are input into the image processing model to obtain a third fused image that fuses the content features of the first image and the style features of the fourth image; interpolation processing is performed on the first, second, and third fused images to obtain a fourth fused image that fuses the content features of the first image, at least a portion of the style features of the second image, at least a portion of the style features of the third image, and at least a portion of the style features of the fourth image. Similarly, the number of images to be processed can be, for example, five, six, etc., and the final fused image is a final fused image that fuses the content features of the first image and a portion of the style features of the other images; the specific method is similar to that described above and will not be repeated here. Based on this, it is possible to transfer the style features of multiple different images to a single image. Therefore, there are no restrictions on the type and number of images when acquiring images to be processed and training models, which avoids the problem of relatively small scope of use and lack of flexibility, thereby meeting the needs of large-scale image processing and effectively expanding the scope of use and flexibility of the image processing method.
[0105] like Figure 7B As shown, another image processing method according to some embodiments of this disclosure may include: S710b, acquire the original image to be processed, as well as the first reference image and the second reference image; S720b, the original image is input into a content feature extractor of an image processing model pre-trained using an image processing model training method according to some embodiments of the present disclosure, so as to obtain the content features of the original image; S730b, the first reference image and the second reference image are respectively input into the style feature extractor of the pre-trained image processing model to obtain the first style feature of the first reference image and the second style feature of the second reference image; S740b, by interpolation, synthesizes the first style feature and the second style feature into a third style feature that integrates at least a portion of the first style feature and at least a portion of the second style feature; S750b inputs the content features and third style features of the original image into the content style fusion unit of the pre-trained image processing model to obtain a target fused image that fuses the content features and third style features of the original image.
[0106] This is Figure 7A The difference between the image processing methods and their embodiments lies in that... Figure 7BThe image processing method first interpolates the first and second style features to obtain the third style feature, and then fuses the third style feature with the content features of the original image to obtain the target fused image. This is similar to... Figure 7A The image interpolation process is different in the embodiments.
[0107] In some embodiments, the number of reference images acquired can be more than two. Taking three reference images as an example: the original image to be processed, as well as a first reference image, a second reference image, and a third reference image, can be acquired. After obtaining the style features of these reference images respectively, the first style features, the second style features, and the third style features are synthesized by interpolation to form a fourth style feature that integrates at least a portion of the first style features, at least a portion of the second style features, and at least a portion of the third style features. This fourth style feature and the content features of the original image are then input into the content style fusion unit of the pre-trained image processing model to obtain the target fused image. Similarly, the number of reference images acquired can be, for example, four, five, six, etc., and the final style feature obtained is a fused style feature that integrates a portion of the style features of each reference image, where the weights of each style feature can be pre-set and pre-adjusted; the specific method is similar to that described above and will not be repeated here. Therefore, using... Figure 7B The image processing methods can further enrich the diversity of image processing methods and increase their flexibility.
[0108] use Figure 7A and Figure 7B The image processing effects obtained by the image processing methods are combined in the following text. Figures 8A-8F Further explanation. Figures 8A-8F The illustration schematically shows the effect of using an image processing method according to some embodiments of the present disclosure. It should be noted that... Figures 8A-8F The illustrated diagrams are merely illustrative and exemplary and do not imply any limitation on the methods disclosed herein.
[0109] Figure 8A A comparison diagram showing the effects of using an image processing method according to some embodiments of the present disclosure and a method using related techniques is presented. The image processing method according to some embodiments of the present disclosure uses... Figure 8A The left three columns show how a fused image is obtained by using the first image (which has content features) and the second image (which has style features to be transferred). Figure 8AThe six columns on the right represent fused images obtained using image processing methods based on related technologies. These methods may include, for example, traditional style transfer methods such as NST, unsupervised image-to-image methods such as CycleGAN and DRIT++, instance-based image-to-image methods such as Adain, MUNIT, and RSIC, and so on. It can be seen that, compared to image processing methods based on related technologies, the image processing method according to some embodiments of this disclosure achieves the best results. The resulting fused image possesses the content features of the first image and the style features of the second image, and the fused image is clear, natural, and unobtrusive.
[0110] Figure 8B This demonstrates the effect of generating different fused images by combining different first images and different second images. For example... Figure 8B As shown, the first column on the left is the first image, the first row at the top is the second image, and the resulting fused image is shown in the middle of the graph. The results demonstrate that the image processing method according to some embodiments of this disclosure can extract different style features (i.e., halftone style) from different second images (i.e., reference images) and transfer them well to the first image. This proves the superiority of the method disclosed herein.
[0111] Figure 8C This demonstrates the effect of combining style features from different reference images and applying them to an input image. Because the content feature encoding and style feature encoding of this method are separate, the style feature encoding can be flexibly manipulated, and the style features of different cartoon images can be combined using linear interpolation and then applied to a line drawing. Figure 8C As shown, the first column from the left is the original image, the second column from the left is the first reference image, and the first column from the right is the second reference image. The resulting fused target image is shown in the middle of the figure. It can be seen that, from left to right, the style features of the fused target image closer to the first reference image are more similar to the style features of the first reference image, while the style features of the fused target image closer to the second reference image are more similar to the style features of the second reference image. This demonstrates the great flexibility of the method disclosed herein.
[0112] Figure 8D , Figure 8E and Figure 8F The results of image processing on other types of line art and comic illustrations are shown. For example... Figure 8D , Figure 8E and Figure 8F As shown, the first row consists of the first image, the third row consists of the second image, the second row consists of the fused image, and the labels (a)-(f) in each column represent examples under different conditions. From Figure 8DAs can be seen, the image processing model 400 disclosed herein can use not only images containing only human heads, but also images including other parts of the human body and more background information. From Figure 8E It can be seen that the image processing model 400 of this disclosure can not only use images that include only humans, but also images that do not include humans (e.g., the reference image can be an image that only includes background information). From Figure 8F It can be seen that the image processing model 400 of this disclosure can also use non-human images (including only animals, flowers, etc.). Therefore, regardless of the type of image to be processed, and regardless of the content and style characteristics of the image to be processed, good results can be obtained by using the image processing method according to some embodiments of this disclosure, which proves the versatility of the method of this disclosure.
[0113] according to Figure 7A and Figure 7B Image processing methods and combined Figures 8A-8F The image processing effects shown demonstrate that the image processing method according to some embodiments of this disclosure can quickly match suitable style features (dot style) to obtain the final fused image, thereby greatly accelerating the image processing process and improving image processing efficiency. Furthermore, during image processing, users can obtain the image processing effects in real time by inputting different types of images and / or multiple images, thereby improving human-computer interaction efficiency, meeting the real-time image processing needs in more application scenarios, and realizing a convenient and efficient image processing process.
[0114] Figure 9 An example block diagram of an image processing model training apparatus 900 according to some embodiments of the present disclosure is shown schematically. Figure 9 The image processing model training device 900 shown can correspond to Figure 1 The terminal device 110 shown is shown.
[0115] like Figure 9As shown, the image processing model training device 900 may include a training data acquisition module 910, an input module 920, a first determination module 930, a second determination module 940, a third determination module 950, and an iteration module 960. The training data acquisition module 910 can be configured to acquire training data, which includes a first sample image, a second sample image, and a third sample image, wherein the first sample image and the third sample image have the same content features. The input module 920 can be configured to input the first sample image and the second sample image into a preset image processing model to obtain a first target image that integrates the content features of the first sample image and the style features of the second sample image. The first determination module 930 can be configured to determine a first similarity between the second sample image and the third sample image through feature map extraction. The second determination module 940 can be configured to determine a first loss of the image processing model based on the first similarity, the second sample image, and the first target image. The third determination module 950 can be configured to determine a target loss of the image processing model based at least on the first loss. The iteration module 960 can be configured to iteratively update the parameters of the image processing model based on the target loss until the target loss meets a preset condition.
[0116] It should be noted that the various modules described above can be implemented in software, hardware, or a combination of both. Multiple different modules can be implemented within the same software or hardware architecture, or a single module can be implemented by multiple different software or hardware architectures.
[0117] Using an image processing model training apparatus according to some embodiments of the present disclosure, when collecting training samples during model training, in addition to the sample image pairs consisting of the target image sample (i.e., the first sample image) and the reference image sample (i.e., the second sample image), a third sample image with the same content features as the target image sample is introduced. This third sample image is used together with the reference image sample to obtain pseudo-sample labels through image feature extraction and similarity calculation, thereby constructing a block response loss (i.e., the first loss) to guide the training of the model. On the one hand, the acquisition of such pseudo-sample labels can overcome the problem of lack of paired sample images (i.e., as input samples: the target image and the reference image, and as the fused image as the label) in related technologies, thereby avoiding the completely unsupervised learning and training process caused by this. On the other hand, the block response loss can more accurately guide the position matching of the style features of the reference image in the target image, thereby ensuring the accuracy of the style transfer effect of the trained model, and making the fused image obtained by the model more realistic, clear, and natural.
[0118] Figure 10An example block diagram of a computing device 1000 according to some embodiments of the present disclosure is schematically shown. The computing device 1000 may represent a device for implementing the various apparatuses or modules described herein and / or performing the various methods described herein. The computing device 1000 may be, for example, a server, desktop computer, laptop computer, tablet, smartphone, smartwatch, wearable device, or any other suitable computing device or computing system, which may include various levels of devices ranging from full-resource devices with large storage and processing resources to low-resource devices with limited storage and / or processing resources. In some embodiments, the above regarding... Figure 9 The image processing model training device 900 described can be implemented in one or more computing devices 1000.
[0119] like Figure 10 As shown, the example computing device 1000 includes a processing system 1001 communicatively coupled to each other, one or more computer-readable media 1002, and one or more I / O interfaces 1003. Although not shown, the computing device 1000 may also include a system bus or other data and command transmission system that couples the various components to each other. The system bus may include any or a combination of different bus architectures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any of the various bus architectures. Alternatively, it may also include control and data lines.
[0120] Processing system 1001 represents the functionality of performing one or more operations using hardware. Therefore, processing system 1001 is illustrated as including hardware elements 1004 that can be configured as processors, function blocks, etc. This may include other logic devices implemented in hardware as application-specific integrated circuits (ASICs) or formed using one or more semiconductors. Hardware element 1004 is not limited by the materials in which it is formed or the processing mechanism employed therein. For example, a processor may consist of semiconductors and / or transistors (e.g., integrated circuits (ICs)). In such a context, processor-executable instructions may be electronically executable instructions.
[0121] Computer-readable medium 1002 is illustrated as including memory / storage device 1005. Memory / storage device 1005 represents a memory / storage device associated with one or more computer-readable media. Memory / storage device 1005 may include volatile media (such as random access memory (RAM)) and / or non-volatile media (such as read-only memory (ROM), flash memory, optical disk, magnetic disk, etc.). Memory / storage device 1005 may include fixed media (e.g., RAM, ROM, fixed hard disk drive, etc.) and removable media (e.g., flash memory, removable hard disk drive, optical disk, etc.). Exemplarily, memory / storage device 1005 may be used to store first audio of a first category of users mentioned in the above embodiments, a queue list of requests, etc. Computer-readable medium 1002 may be configured in various other ways as further described below.
[0122] One or more I / O (input / output) interfaces 1003 represent functions that allow a user to type commands and information into a computing device 1000 and also allow information to be displayed to the user and / or sent to other components or devices using various input / output devices. Examples of input devices include keyboards, cursor control devices (e.g., mice), microphones (e.g., for voice input), scanners, touch functionality (e.g., capacitive or other sensors configured to detect physical touch), cameras (e.g., capable of detecting non-touch-related movements as gestures using visible or invisible wavelengths (such as infrared frequencies), network interface cards (NICs), receivers, and so on). Examples of output devices include display devices (e.g., monitors or projectors), speakers, printers, haptic-responsive devices, network interface cards (NICs), transmitters, and so on. Exemplarily, in the embodiments described above, both a first-category user and a second-category user can input through the input interfaces on their respective terminal devices to initiate requests and record audio and / or video, and can view various notifications and watch videos or listen to audio, etc., through the output interfaces.
[0123] The computing device 1000 also includes an image processing model training strategy 1006. The image processing model training strategy 1006 can be stored as computing program instructions in a memory / storage device 1005, or it can be hardware or firmware. The image processing model training strategy 1006 can be implemented together with the processing system 1001, etc., regarding... Figure 9 The image processing model training device 900 is described, outlining the full functionality of each module.
[0124] This document describes various technologies in the general context of software, hardware, components, or program modules. Generally, these modules include routines, programs, objects, elements, components, data structures, etc., that perform specific tasks or implement specific abstract data types. The terms "module," "function," etc., as used herein generally refer to software, firmware, hardware, or a combination thereof. The technologies described herein are platform-independent, meaning that these technologies can be implemented on a variety of computing platforms with various processors.
[0125] Implementations of the described modules and technologies may be stored on or transmitted across some form of computer-readable medium. The computer-readable medium may include a variety of media accessible by the computing device 1000. By way of example and not limitation, the computer-readable medium may include "computer-readable storage media" and "computer-readable signal media".
[0126] In contrast to simple signal transmission, carrier waves, or signals themselves, a "computer-readable storage medium" refers to a medium and / or device capable of persistently storing information, and / or a tangible storage device. Therefore, a computer-readable storage medium refers to a non-signal-bearing medium. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented using methods or techniques suitable for storing information (such as computer-readable instructions, data structures, program modules, logic elements / circuits, or other data). Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage devices, hard disks, cassette tapes, magnetic tapes, disk storage devices or other magnetic storage devices, or other storage devices, tangible media, or articles of art suitable for storing desired information and accessible by a computer.
[0127] "Computer-readable signal medium" refers to a signal-bearing medium configured to transmit instructions, such as via a network, to computing device 1000. A signal medium typically embodies computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves, data signals, or other transmission mechanisms. Signal media also include any information transmission medium. By way of example and not limitation, signal media includes wired media such as wired networks or direct connections, and wireless media such as acoustic, RF, infrared, and other wireless media.
[0128] As previously described, hardware element 1004 and computer-readable medium 1002 represent instructions, modules, programmable device logic, and / or fixed device logic implemented in hardware, which in some embodiments can be used to implement at least some aspects of the techniques described herein. Hardware elements may include components of integrated circuits or systems-on-a-chip, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations or other hardware devices in silicon. In this context, hardware elements can serve as processing devices for executing program tasks defined by instructions, modules, and / or logic embodied by the hardware element, and as hardware devices for storing instructions for execution, such as the previously described computer-readable storage medium.
[0129] The foregoing combinations can also be used to implement the various techniques and modules described herein. Therefore, software, hardware, or program modules and other program modules can be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage medium and / or by one or more hardware elements 1004. The computing device 1000 can be configured to implement specific instructions and / or functions corresponding to the software and / or hardware modules. Thus, modules can be implemented at least partially in hardware as modules executable as software by the computing device 1000, for example, by using the computer-readable storage medium and / or hardware elements 1004 of the processing system. Instructions and / or functions can be executed / operated by, for example, one or more computing devices 1000 and / or processing system 1001 to implement the techniques, modules, and examples described herein.
[0130] The techniques described herein can be supported by these various configurations of the computing device 1000, and are not limited to specific examples of the techniques described herein.
[0131] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as a computer program. For example, embodiments of this disclosure provide a computer program product including a computer program carried on a computer-readable medium, the computer program containing program code for performing at least one step in the method embodiments of this disclosure.
[0132] In some embodiments of this disclosure, one or more computer-readable storage media are provided, having stored computer-readable instructions thereon that, when executed, implement an image processing model training method according to some embodiments of this disclosure. The various steps of the image processing model training method according to some embodiments of this disclosure can be programmed into computer-readable instructions and stored in the computer-readable storage medium. When such a computer-readable storage medium is read or accessed by a computing device or computer, the computer-readable instructions therein are executed by a processor on the computing device or computer to implement the method described in some embodiments of this disclosure.
[0133] In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example that is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0134] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed (including substantially simultaneously or in reverse order depending on the functions involved), as will be understood by those skilled in the art to which embodiments of this disclosure pertain.
[0135] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.
[0136] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.
[0137] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by hardware associated with program instructions. The program can be stored in a computer-readable storage medium, and when executed, the program includes performing one or a combination of the steps of the method embodiments.
[0138] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0139] Although this disclosure has been described in conjunction with some embodiments, it is not intended to be limited to the specific forms set forth herein. Rather, the scope of this disclosure is limited only by the appended claims. Additionally, although individual features may be included in different claims, these may be advantageously combined, and their inclusion in different claims does not imply that such a combination of features is not feasible and / or advantageous. The order of features in the claims does not imply that the features must be in any particular order of their operation. Furthermore, in the claims, the word "comprising" does not exclude other elements, and the terms "a" or "an" do not exclude a plurality. Reference numerals in the claims are provided only by way of explicit example and should not be construed as limiting the scope of the claims in any way.
[0140] It is understood that the specific embodiments of this disclosure involve image data such as various sample images used for model training, as well as original images and reference images to be processed. When the embodiments involving such data described in this disclosure are applied to specific products or technologies, user permission or consent is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.
Claims
1. A method for training an image processing model, comprising: Acquire training data, which includes a first sample image, a second sample image, and a third sample image, wherein the first sample image and the third sample image have the same content features; The first sample image and the second sample image are input into a preset image processing model to obtain a first target image that integrates the content features of the first sample image and the style features of the second sample image. The first similarity between the second sample image and the third sample image is determined by feature map extraction; Based on the second sample image and the first similarity, a pseudo-label image corresponding to the first target image is determined; Based on the first target image and the pseudo-label image, determine the first loss of the image processing model; The target loss of the image processing model is determined based at least on the first loss; Based on the target loss, the parameters of the image processing model are iteratively updated until the target loss meets the preset conditions.
2. The method according to claim 1, wherein, Determining the target loss of the image processing model based at least on the first loss includes: The first target image is divided into multiple first image blocks, and the second sample image is divided into multiple second image blocks; For each first image patch in the first target image, calculate the second similarity between each second image patch in the second sample image and the first image patch; For each first image block in the first target image, based on the second similarity between each second image block in the second sample image and the first image block, a similar image block to the first image block is selected from the plurality of second image blocks; The second loss of the image processing model is determined based on each first image patch and its similar image patches in the first target image; The target loss of the image processing model is determined based at least on the first loss and the second loss.
3. The method according to claim 2, wherein the first image block and the second image block have the same area. The calculation of the second similarity between each second image patch in the second sample image and the first image patch includes: For each second image block in the second sample image, calculate the Euclidean distance between the second vector corresponding to the second image block and the first vector corresponding to the first image block, wherein the second vector is composed of the pixel values of each pixel in the second image block, and the first vector is composed of the pixel values of each pixel in the first image block; The second similarity is determined based on the Euclidean distance.
4. The method according to claim 2, wherein determining the second loss of the image processing model based on each first image patch in the first target image and its similar image patches comprises: For each first image block and its similar image blocks in the first target image, calculate the Manhattan distance between the first vector corresponding to each first image block and the third vector corresponding to its similar image block, wherein the third vector is composed of the pixel values of each pixel point of the similar image block; The second loss of the image processing model is determined based on the sum of the Manhattan distances corresponding to all first image patches and their similar image patches.
5. The method according to claim 1, wherein, Determining the target loss of the image processing model based at least on the first loss includes: The first sample image and the third sample image are input into a preset image processing model to obtain a second target image that integrates the content features of the first sample image and the style features of the third sample image; The third loss of the image processing model is determined based on the third sample image and the second target image; The target loss of the image processing model is determined based at least on the first loss and the third loss.
6. The method according to claim 2, wherein, Determining the target loss of the image processing model based at least on the first loss and the second loss includes: The first sample image and the third sample image are input into a preset image processing model to obtain a second target image that integrates the content features of the first sample image and the style features of the third sample image; The third loss of the image processing model is determined based on the third sample image and the second target image; The target loss of the image processing model is determined based at least on the first loss, the second loss, and the third loss.
7. The method according to claim 6, wherein, Determining the target loss of the image processing model based at least on the first loss, the second loss, and the third loss includes: Based on the second target image, determine the first adversarial loss of the image processing model; Based on the first target image, determine the second adversarial loss of the image processing model; Based on the first target image and the second sample image, the joint block adversarial loss of the image processing model is determined; Based on the first sample image and the first target image, determine the content loss of the image processing model; The target loss of the image processing model is determined based on at least one of the first adversarial loss, the second adversarial loss, the joint block adversarial loss, and the content loss, as well as the first loss, the second loss, and the third loss.
8. The method according to claim 7, wherein, The step of determining the target loss of the image processing model based on at least one of the first adversarial loss, the second adversarial loss, the joint block adversarial loss, and the content loss, as well as the first loss, the second loss, and the third loss, includes: The target loss is determined by the weighted sum of the first loss, the second loss, the third loss, the first adversarial loss, the second adversarial loss, the joint block adversarial loss, and the content loss.
9. The method according to claim 1, wherein, The step of determining the first similarity between the second sample image and the third sample image through feature map extraction includes: Extract a first feature map and a second feature map from the second sample image and the third sample image, respectively; Calculate the relationship matrix between the first feature map and the second feature map; Based on the relationship matrix, the first similarity between the second sample image and the third sample image is determined.
10. The method according to claim 1, wherein, The acquisition of training data includes: Obtain the second sample image and the third sample image; Extract the content features of the third sample image from the third sample image; The first sample image is generated based on the content features of the third sample image.
11. The method of claim 10, wherein the second sample image includes a first cartoon image, the third sample image includes a second cartoon image, and the first sample image includes a line drawing corresponding to the second cartoon image.
12. The method according to claim 1, wherein the image processing model includes a content feature extractor, a style feature extractor, and a content-style fusion unit. The step of inputting the first sample image and the second sample image into a preset image processing model to obtain a first target image that integrates the content features of the first sample image and the style features of the second sample image includes: The first sample image is input into the content feature extractor to obtain the content features of the first sample image; The second sample image is input into the style feature extractor to obtain the style features of the second sample image; The content features of the first sample image and the style features of the second sample image are input into the content-style fusion processor to obtain a first target image that combines the content features of the first sample image and the style features of the second sample image.
13. An image processing method, comprising: Obtain the first and second images to be processed; The first image and the second image are input into an image processing model pre-trained using the method described in any one of claims 1-12 to obtain a first fused image that integrates the content features of the first image and the style features of the second image.
14. The image processing method according to claim 13, further comprising: Obtain the third image to be processed; The first image and the third image are input into the image processing model to obtain a second fused image that combines the content features of the first image and the style features of the third image; Interpolation processing is performed on the first fused image and the second fused image to obtain a third fused image that incorporates the content features of the first image, at least a portion of the style features of the second image, and at least a portion of the style features of the third image.
15. An image processing method, comprising: Acquire the original image to be processed, as well as the first reference image and the second reference image; The original image is input into the content feature extractor of the image processing model pre-trained using the method of claim 12 to obtain the content features of the original image; The first reference image and the second reference image are respectively input into the style feature extractor of the pre-trained image processing model to obtain the first style feature of the first reference image and the second style feature of the second reference image; The first style feature and the second style feature are combined into a third style feature by interpolation, which integrates at least a portion of the first style feature and at least a portion of the second style feature; The content features and third-style features of the original image are input into the content-style fusion unit of the pre-trained image processing model to obtain a target fused image that combines the content features and third-style features of the original image.
16. An image processing model training device, comprising: The training data acquisition module is configured to acquire training data, which includes a first sample image, a second sample image, and a third sample image, wherein the first sample image and the third sample image have the same content features. The input module is configured to input the first sample image and the second sample image into a preset image processing model to obtain a first target image that integrates the content features of the first sample image and the style features of the second sample image. The first determining module is configured to determine a first similarity between the second sample image and the third sample image by feature map extraction; The second determining module is configured to determine a pseudo-label image corresponding to the first target image based on the second sample image and the first similarity, and to determine a first loss of the image processing model based on the first target image and the pseudo-label image. The third determining module is configured to determine the target loss of the image processing model based at least on the first loss. The iteration module is configured to iteratively update the parameters of the image processing model based on the target loss until the target loss meets a preset condition.
17. A computing device, comprising: Memory and processor The memory contains a computer program that, when executed by the processor, causes the processor to perform the method of any one of claims 1-15.
18. A computer-readable storage medium storing computer-readable instructions thereon, which, when executed, implement the method of any one of claims 1-15.
19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-15.
Citation Information
Patent Citations
Model training method and device, style migration method and device, equipment and storage medium
CN112785493A
