Image style conversion method and related equipment
By determining the weight parameters of the image and fusing them with the basic color lookup table, an adaptive target color lookup table is generated, which solves the problems of high resource consumption and monotonous image style in the display system, realizes personalized image style conversion, and improves the user experience.
Patent Information
- Application Number
- CN202411025760.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-15
AI Technical Summary
The existing technology displays images with a single style, which cannot meet the personalized needs of users and leads to a poor user experience. In addition, the existing image style transformation methods have problems such as high resource consumption and slow display system operation.
By obtaining the target transformation style of the currently displayed image, determining the weight parameters of the basic color lookup table, and fusing the weight parameters and the basic color lookup table, an adaptive target color lookup table is generated, thereby achieving personalized transformation of image style.
It enables the provision of diverse image style options without increasing resource consumption, thereby enhancing the user's personalized experience and the operating efficiency of the display system.
Smart Images

Figure CN118967856A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image style conversion method and related equipment. Background Art
[0002] With the development of display technology, the intelligence of display devices has been continuously improved, and images can be presented to users.
[0003] However, the image style displayed in the prior art is relatively single and cannot meet the personalized needs of users, thus bringing a bad usage experience to users. Summary of the invention
[0004] In view of this, the purpose of this application is to propose an image style transfer method and related equipment to solve the above technical problems.
[0005] Based on the above purpose, the first aspect of the present application provides an image style transfer method, comprising:
[0006] Acquire a target conversion style of a currently displayed image, and determine a basic color lookup table corresponding to the currently displayed image;
[0007] Performing prediction processing using the target conversion style and the current display image to determine a weight parameter of the current display image relative to the basic color lookup table;
[0008] The weight parameter and the basic color lookup table are fused to obtain a target color lookup table, and a target display image corresponding to the target conversion style is obtained through the target color lookup table based on the current display image.
[0009] Optionally, determining a basic color lookup table corresponding to the currently displayed image includes:
[0010] Acquire a prediction data set, wherein the prediction data set includes a plurality of style images, each style image corresponding to a plurality of pixel combinations;
[0011] Selecting a target pixel combination from the plurality of pixel combinations according to a preset value interval;
[0012] The target pixel combination is input into a trained convolutional neural network model, an output result is obtained through the convolutional neural network model, and the basic color lookup table is obtained based on the target pixel combination and the output result.
[0013] Optionally, obtaining the basic color lookup table based on the target pixel combination and the output result includes:
[0014] The target pixel combination and the output result are stored correspondingly to obtain an initial color lookup table;
[0015] Obtaining a true output label corresponding to the target pixel combination, and constructing a loss function using the true output label and the output result;
[0016] The initial color lookup table is adjusted according to the loss function to obtain the basic color lookup table.
[0017] Optionally, obtaining a prediction data set includes:
[0018] Get sample images of multiple styles and perform the following operations on each style:
[0019] Performing instance segmentation on the sample image to obtain a foreground image and a background image;
[0020] Performing incremental processing on the background image to obtain an incrementally processed background image, and performing incremental processing on the foreground image to obtain an incrementally processed foreground image;
[0021] The incrementally processed background image and the incrementally processed foreground image are combined to obtain the prediction data set.
[0022] Optionally, the prediction data set is obtained using a trained diffusion model, the diffusion model includes a latent diffusion model and a low-rank adaptation model, and the diffusion model is obtained by training the low-rank adaptation model using sample images of the multiple styles after freezing the latent diffusion model.
[0023] Optionally, the using the target conversion style and the current display image to perform prediction processing to determine a weight parameter of the current display image relative to the basic color lookup table includes:
[0024] Performing embedded coding on the target conversion style to obtain an embedded coding result;
[0025] Performing feature extraction on the currently displayed image to obtain a feature-extracted image, and performing position encoding on the feature-extracted image to obtain a position encoding result;
[0026] Natural language processing is performed on the embedded coding result and the position coding result to obtain a weight parameter of the current display image relative to the basic color lookup table.
[0027] Optionally, the weight parameter is obtained using a pre-trained large language model, where the large language model includes a normalized exponential function layer for classification prediction.
[0028] Optionally, the fusing the weight parameter and the basic color lookup table to obtain a target color lookup table includes:
[0029] The target color lookup table is obtained by performing weighted summation processing using the weight parameter and the basic color lookup table.
[0030] Optionally, obtaining a target display image corresponding to the target conversion style through the target color lookup table based on the current display image includes:
[0031] Acquire a feature extraction image corresponding to the currently displayed image, and perform splicing processing on the feature extraction image and the currently displayed image to obtain a spliced image;
[0032] Acquire a pixel combination of the stitched image;
[0033] Searching the target pixel combination corresponding to the pixel combination of the spliced image from the target color lookup table;
[0034] A target display image corresponding to a target conversion style is obtained based on the target pixel combination.
[0035] Optionally, the input dimensions corresponding to the target color lookup table include: a first red sub-pixel, a first green sub-pixel, a first blue sub-pixel and a feature extraction image corresponding to any target pixel among the plurality of pixels of the stitched image;
[0036] The output dimensions corresponding to the target color lookup table include: a second red sub-pixel, a second green sub-pixel, and a second blue sub-pixel corresponding to any one of the target pixels in the target display image.
[0037] Optionally, the target display image is obtained using a trained pipeline model, which is obtained by back-propagation training based on a loss function constructed using a pre-acquired training current display image and a training output target display image obtained by converting the style of the training target, and a pre-acquired actual target display image.
[0038] Based on the same inventive concept, the second aspect of the present application provides an image style conversion device, comprising:
[0039] an acquisition module, configured to acquire a target conversion style of a currently displayed image, and determine a basic color lookup table corresponding to the currently displayed image;
[0040] A prediction module, configured to perform prediction processing using the target conversion style and the current display image to determine a weight parameter of the current display image relative to the basic color lookup table;
[0041] The fusion module is configured to fuse the weight parameter and the basic color lookup table to obtain a target color lookup table, and obtain a target display image corresponding to the target conversion style through the target color lookup table based on the current display image.
[0042] Based on the same inventive concept, the third aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable by the processor, wherein the processor implements the method described in the first aspect above when executing the computer program.
[0043] Based on the same inventive concept, the fourth aspect of the present application provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the method described in the first aspect above.
[0044] Based on the same inventive concept, the fifth aspect of the present application provides a computer program product, including computer program instructions, which, when executed on a computer, enable the computer to execute the method described in the first aspect.
[0045] From the above, it can be seen that the image style conversion method and related equipment provided by the present application determine the image style specified by the user by obtaining the target conversion style of the currently displayed image, and then use the target conversion style and the current displayed image for prediction processing to determine the weight parameters of the current displayed image relative to the basic color lookup table, and fuse the weight parameters and the basic color lookup table to obtain an adaptive target color lookup table, and then obtain the target display image of the target conversion style that the user wants to view based on the current displayed image through the target color lookup table, thereby meeting the user's personalized needs and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0047] Figure 1A This is a first schematic diagram of an original image before style transformation according to an embodiment of the present application;
[0048] Figure 1B A first schematic diagram of an image after style transformation according to an embodiment of the present application;
[0049] Figure 1CA second schematic diagram of an original image before style transformation according to an embodiment of the present application;
[0050] Figure 1D A second schematic diagram of an image after style transformation according to an embodiment of the present application;
[0051] Figure 1E A flowchart of an image style conversion method according to an embodiment of the present application;
[0052] Figure 1F A schematic diagram of a process of obtaining a basic color lookup table according to an embodiment of the present application;
[0053] Figure 1G A schematic diagram of a process of obtaining a prediction data set according to an embodiment of the present application;
[0054] Figure 1H A schematic diagram of the training process of the diffusion model of an embodiment of the present application;
[0055] Fig. 1I A schematic diagram of a potential diffusion model of an embodiment of the present application;
[0056] Figure 1J A schematic diagram of a low-rank adaptive model according to an embodiment of the present application;
[0057] Figure 1K A schematic diagram of a process for obtaining a target display image according to an embodiment of the present application;
[0058] Figure 1L A schematic diagram of an image style conversion process according to an embodiment of the present application;
[0059] Figure 2 This is a structural block diagram of an image style conversion device according to an embodiment of the present application;
[0060] Figure 3 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0061] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.
[0062] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0063] It is understandable that before using the technical solutions of each embodiment of the present application, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.
[0064] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to the electronic device, application, server, storage medium or other software or hardware that performs the operation of the technical solution of the present application according to the prompt message.
[0065] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0066] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation method of this application. Other methods that meet relevant laws and regulations may also be applied to the implementation method of this application.
[0067] With the development of display technology, the intelligence of display devices is constantly improving, and images can be presented to users. However, the image style displayed in the prior art is relatively single, which cannot meet the personalized needs of users, thus bringing a bad user experience. Some works will adjust the global brightness and contrast of the image to make the displayed image more visually effective. A few works mention multi-style image display, but its idea is the geometric deformation of the image (such as mirroring) rather than the image content. There is no real change in image style, and the display process is not convenient and universal enough. Image style transformation, for example: the original image before style transformation is like Figure 1A As shown, the image after style transformation is Figure 1B As shown, or the original image before style transformation is as follows Figure 1C As shown, the image after style transformation is Figure 1D shown.
[0068] However, in order to display multiple image styles, images of different styles are often generated through a color lookup table (LUT). The inventor of the present application found that in this way, if one image style is represented by several (n) LUTs, then m image styles need to store m times (i.e., m*n) LUT files. From the perspective of user personalization, accessing more and more LUT tables will make the display system run slower and slower, seriously affecting the interactive experience. However, if one style corresponds to several groups of LUTs, then in order to ensure the running speed of the display system (less resource occupation), the types of image styles that can be available will be relatively limited, which is not conducive to the popularization of different image styles liked by different users.
[0069] An embodiment of the present application provides an image style conversion method, which determines the weight parameters of the current display image relative to the basic color lookup table, and then fuses the weight parameters and the basic color lookup table to obtain an adaptive target color lookup table. Then, based on the current display image, the target color lookup table is used to obtain a target display image of the target conversion style that the user wants to view, thereby meeting the user's personalized needs and improving the user experience.
[0070] like Figure 1E As shown, the method of this embodiment includes:
[0071] Step 101: Acquire a target conversion style of a currently displayed image, and determine a basic color lookup table corresponding to the currently displayed image.
[0072] In this step, style conversion buttons corresponding to different types of image styles are set on the control device (e.g., a remote controller), for example, a button corresponding to style 1, a button corresponding to style 2, ..., a button corresponding to style n. The user triggers the button corresponding to the style he wants to see on the control device, so that the display terminal obtains the target conversion style of the currently displayed image.
[0073] Alternatively, only one conversion style button is set on the control device. When the user triggers the conversion style button on the control device, a conversion style page is displayed on the display terminal. The conversion style page displays conversion styles corresponding to different types of image styles. The user can interact with the display terminal through the control device to select the target conversion style in the conversion style page. In addition, the user can also select the target conversion style in the conversion style page through voice commands or touch commands.
[0074] In order to reduce the resource usage of the display system and obtain more images of different styles, a limited LUT table can be used as the basic color lookup table (basic LUTs) for all these style images. This will not only not affect the stylization quality of the image, but also greatly reduce the memory usage of the display system.
[0075] Step 102: Perform prediction processing using the target conversion style and the current display image to determine a weight parameter of the current display image relative to the basic color lookup table.
[0076] In this step, in order to achieve flexible image enhancement, a weight predictor can be used to understand the global context of the image, such as brightness, color, and hue, so as to perform prediction processing based on the target conversion style and the current display image, thereby predicting the weight parameters of the current display image relative to the basic color lookup table, so as to fuse multiple basic color lookup tables into an adaptive target color lookup table, which is used to effectively convert the image style of the current display image, wherein the weight predictor can be a convolutional neural network (CNN) model.
[0077] Step 103: fusing the weight parameter and the basic color lookup table to obtain a target color lookup table, and obtaining a target display image corresponding to the target conversion style through the target color lookup table based on the current display image.
[0078] In this step, in order to generate a display image with a specified image style, the weight parameters and the basic color lookup table are fused, and the basic color lookup table is fused into an image-adaptive color lookup table (i.e., a target color lookup table). The weight parameters of each pixel can be used to represent the complex pixel mapping relationship more comprehensively and accurately. Then, based on the current display image, the target display image with the target conversion style that the user wants to view can be obtained through the target color lookup table, thereby meeting the user's personalized needs and improving the user experience.
[0079] Through the above scheme, the target conversion style of the current display image is obtained to determine the image style specified by the user, and then the target conversion style and the current display image are used for prediction processing to determine the weight parameters of the current display image relative to the basic color lookup table, and the weight parameters and the basic color lookup table are fused to obtain an adaptive target color lookup table, and then the target display image of the target conversion style that the user wants to view is obtained through the target color lookup table based on the current display image, thereby meeting the user's personalized needs and improving the user experience.
[0080] In some embodiments, in step 101, determining a basic color lookup table corresponding to the currently displayed image includes:
[0081] Step A1, obtaining a prediction data set, wherein the prediction data set includes a plurality of style images, and each style image corresponds to a plurality of pixel combinations.
[0082] Step A2: selecting a target pixel combination from the plurality of pixel combinations according to a preset value interval.
[0083] Step A3: input the target pixel combination into a trained convolutional neural network model, obtain an output result through the convolutional neural network model, and obtain the basic color lookup table based on the target pixel combination and the output result.
[0084] In the above scheme, in order to solve the resource occupation problem of the display system, maintain the operating speed of the display system, and solve the problem that the types of image styles are limited and it is not conducive to popularizing different image styles liked by different users, the present application uses a limited LUT table that can become the basic color lookup table (basic LUTs) for all these style images. It will not only not affect the stylized quality of the image, but also greatly reduce the memory occupation of the display system. The previous color lookup table is m*n LUTs, and the basic color lookup table of this application is p, m*n>p.
[0085] In actual operation, the input is to traverse all possible pixel combinations. For example, the input of n (for example, 3) dimensions is 256 power 3 inputs (i.e., multiple pixel combinations). In order to reduce the color lookup table, the 256 intervals can be 2 4 (i.e. the preset value interval) value, then the input length is 2 8-4 +1, that is, the input value interval is (0, 16, 32...), and by selecting the target pixel combination from 256 power-of-3 inputs in this way, the input dimension will become 17 power-of-n.
[0086] In addition, the training process of the Convolutional Neural Networks (CNN) model is as follows Figure 1F As shown in ①, m types of training data (style 1, style 2, ..., style m) are obtained, the image order of the training data is shuffled, and sent to the initial convolutional neural network model (such as CNN model training in SRLUT, MuLUT, and RCLUT algorithms) for initial convolutional neural network model weight training (i.e., CNN weight training) to obtain the trained CNN weights. At this time, the convolutional neural network training is completed.
[0087] In some embodiments, in step A3, obtaining the basic color lookup table based on the target pixel combination and the output result includes:
[0088] Step B1, storing the target pixel combination and the output result correspondingly to obtain an initial color lookup table.
[0089] Step B2, obtaining a true output label corresponding to the target pixel combination, and constructing a loss function using the true output label and the output result.
[0090] Step B3: adjusting the initial color lookup table according to the loss function to obtain the basic color lookup table.
[0091] In the above scheme, if Figure 1F As shown in ②, the target pixel combination (i.e., LUT input) is input into the trained convolutional neural network model (i.e., CNN weight) to obtain the output result (i.e., LUT output), and the target pixel combination and the output result are stored correspondingly in the LUT table to form the required initial color lookup table (i.e., LUT file).
[0092] Then as Figure 1F As shown in ③, the real output label corresponding to the target pixel combination and the output result are used to construct a loss function. The initial color lookup table is fine-tuned through the loss function to obtain a basic color lookup table, so that the basic color lookup table with precision loss after conversion is more accurate.
[0093] In some embodiments, in step A1, obtaining a prediction dataset includes:
[0094] Step C1, obtaining sample images of multiple styles, and performing the following operations on the sample images of each style:
[0095] Step C11, performing instance segmentation on the sample image to obtain a foreground image and a background image.
[0096] Step C12, performing incremental processing on the background image to obtain an incrementally processed background image, and performing incremental processing on the foreground image to obtain an incrementally processed foreground image.
[0097] Step C13, combining the incrementally processed background image and the incrementally processed foreground image to obtain the prediction data set.
[0098] In the above scheme, if Figure 1G As shown, an instance segmentation algorithm is first used to determine whether a target object exists in a sample image. If so, in order to improve the amplification effect of the sample image with a target object, the present application first performs instance segmentation on the sample image (i.e., the image source) with a target object by using an instance segmentation algorithm to distinguish between a foreground image with a target object and a background image without a target object, wherein the target object can be a person or an object other than a person, and the target object here is preferably a person.
[0099] Then, based on the background image without the target object (i.e., the background image), the specified style image is generated through the trained diffusion model (i.e., the fine-tuned stylized Diffusion model) to obtain the stylized background image. For the foreground image with the target object (i.e., the person image), the person image is stylized and generated through the image stylization algorithm, and then the stylized person is pasted on the generated background image to obtain the final required stylized sample (i.e., the prediction dataset).
[0100] In addition, if Figure 1F As shown, m types of training data (style 1, style 2, ..., style m) used to train a convolutional neural network for generating a basic color lookup table can also be obtained through the above-mentioned trained diffusion model.
[0101] In some embodiments, the prediction data set is obtained using a trained diffusion model, which includes a latent diffusion model and a low-rank adaptation model. The diffusion model is obtained by training the low-rank adaptation model using sample images of the multiple styles after freezing the latent diffusion model.
[0102] In the above scheme, the prediction data set and Figure 1F The generation of m types of training data (style 1, style 2, ..., style m) for training the convolutional neural network for generating the basic color lookup table shown in the figure will use a variety of algorithms, including large image models (such as stable diffusion model, Stable Diffusion), image stylization algorithms (such as style transfer, style-swap: "Fast Patch-based Style Transfer of Arbitrary Style" and "Neural Algorithm of Artistic Style", etc.), among which, the prediction data set or training data is obtained by using the trained diffusion model, and the training process mainly includes two steps, fine-tuning the generation model, and generating the prediction data set or training data.
[0103] like Figure 1H As shown in the figure, first draw a small number (such as about 10) sample images (i.e., manually fine-tuned small sample style change images) for each required style image, and send these sample images to a stable diffusion model (i.e., the Duffusion basic model of natural images) for fine-tuning, and then you can get a diffusion model that can generate images of different styles (i.e., a fine-tuned stylized diffusion model).
[0104] The fine-tuning method uses the Low-Rank Adaptation (Lora) fine-tuning method. Each style of sample image is named a name, which is used as a prompt word for the stable diffusion model fine-tuning input and model inference. Then, the sample images of different styles after manual debugging and their corresponding names are input into the stable diffusion model, and the model performs low-rank adaptation fine-tuning through the denoising-denoising steps.
[0105] Diffusion models include latent diffusion model and low-rank adaptation (Lora) model. The structure of latent diffusion model is as follows: Fig. 1I As shown, it includes a perception compression module, an implicit space generation module and a conditional input mechanism module.
[0106] like Figure 1J As shown in the figure, while keeping the original network layer weights frozen, the low-rank adaptation model is injected and combined with the original potential diffusion model output, and only some parameters of the low-rank adaptation model are fine-tuned. It mainly includes two parts:
[0107] The weights of the original potential diffusion model (PretrainedWeights): The network weights of this part belong to the original network, and the weights of this part will be frozen and will not participate in further parameter changes.
[0108] Low-rank adaptive model: The diffusion model is composed of the product of two low-rank matrices (i.e., A and B) through low-rank decomposition. During the training process, the diffusion model trains and fine-tunes the low-rank adaptive model. Therefore, the low-rank adaptive model is like a patch package.
[0109] In some embodiments, step 102 includes:
[0110] Step D1, performing embedded coding on the target conversion style to obtain an embedded coding result.
[0111] Step D2, extracting features from the currently displayed image to obtain a feature-extracted image, and performing position encoding on the feature-extracted image to obtain a position encoding result.
[0112] Step D3, performing natural language processing on the embedded coding result and the position coding result to obtain a weight parameter of the current display image relative to the basic color lookup table.
[0113] In the above scheme, if Figure 1K As shown in the figure, the target conversion style (i.e., the corresponding image style label) is embedded (embedding) encoded to convert it from characters to digital expressions to obtain embedded encoding results. At the same time, the current display image (i.e., the original image) is subjected to CNN feature extraction to obtain features (i.e., feature extraction image), and then positionally encoded (i.e., block encoded) and further feature fused and extracted together with the embedded encoding results converted to digital expressions to obtain the weight distribution (i.e., weight parameters) of the current display image in basic LUTs (i.e., basic color lookup tables).
[0114] In some embodiments, the weight parameters are obtained using a pre-trained large language model, where the large language model includes a normalized exponential function layer for classification prediction.
[0115] In the above scheme, if Figure 1K As shown, after the features (i.e., feature extraction image) of the current display image (i.e., the original image) are extracted by CNN features, they are positionally encoded (i.e., block encoded) and, together with the embedded encoding results converted into digital expressions, are further fused and extracted by a large language model (i.e., Transformer). Classification prediction is performed through a normalized exponential function (Softmax) layer to obtain the weight distribution (i.e., weight parameter) of the current display image in the basic LUTs (i.e., basic color lookup table).
[0116] In some embodiments, in step 103, the fusion processing of the weight parameter and the basic color lookup table to obtain a target color lookup table includes:
[0117] The target color lookup table is obtained by performing weighted summation processing using the weight parameter and the basic color lookup table.
[0118] In the above scheme, if Figure 1K As shown, the weights (i.e., weight parameters) corresponding to the basic LUTs (i.e., basic color lookup table) are obtained by classification prediction through the normalized exponential function (Softmax) layer to be fused to obtain a content- and label-adapted 4D-LUT (i.e., target color lookup table). The fusion method is to multiply the weight parameters (p outputs) by the output values in the basic LUTs table (p), and then add and sum them to obtain the final Fusion 4D LUT (i.e., target color lookup table). The complex pixel mapping relationship can be relatively more comprehensively and accurately represented by the weight parameters of each pixel, and then the target display image with the target conversion style that the user wants to view can be obtained based on the current display image through the target color lookup table, thereby meeting the user's personalized needs and improving the user experience.
[0119] In some embodiments, in step 103, obtaining a target display image corresponding to the target conversion style through the target color lookup table based on the current display image includes:
[0120] Step E1, obtaining a feature extraction image corresponding to the current display image, and splicing the feature extraction image and the current display image to obtain a spliced image.
[0121] Step E2, obtaining pixel combinations of the stitched image.
[0122] Step E3: searching the target pixel combination corresponding to the pixel combination of the spliced image from the target color lookup table.
[0123] Step E4, obtaining a target display image corresponding to the target conversion style based on the target pixel combination.
[0124] In the above scheme, if Figure 1KAs shown, a 1-channel feature extraction image (i.e., context map) corresponding to the current display image is obtained, and the context map is spliced with the current display image (i.e., the original image) to obtain a spliced image, and then a target pixel combination corresponding to the pixel combination of the spliced image is searched from a target color lookup table (i.e., Fusion 4D LUTs) according to the pixel combination of the spliced image, and a target display image (i.e., style image output) corresponding to the target conversion style is obtained based on the target pixel combination.
[0125] In some embodiments, the input dimensions corresponding to the target color lookup table include: a first red sub-pixel, a first green sub-pixel, a first blue sub-pixel, and a feature extraction image corresponding to any target pixel among a plurality of pixels of the stitched image;
[0126] The output dimensions corresponding to the target color lookup table include: a second red sub-pixel, a second green sub-pixel, and a second blue sub-pixel corresponding to any one of the target pixels in the target display image.
[0127] In the above scheme, each pixel is composed of red sub-pixels, green sub-pixels and blue sub-pixels. Target color lookup table (4D LUT): When image enhancement display is combined with LUT, it is usually 3DLUT, and each dimension (1D) represents the value of an image channel. At this time, 3D LUT is a commonly used LUT method in the industry. The dimension of the target color lookup table (4D LUT) refers to the input dimension (R, G, B, C), that is, (first red sub-pixel, first green sub-pixel, first blue sub-pixel, feature extraction image), and the output dimension corresponding to the target color lookup table (4D LUT) is (R, G, B), that is, (second red sub-pixel, second green sub-pixel, second blue sub-pixel).
[0128] In some embodiments, the target display image is obtained using a trained pipeline model, which is obtained by back-propagation training based on a loss function constructed using a pre-acquired training current display image and a training output target display image obtained by a training target conversion style, and a pre-acquired actual target display image.
[0129] In the above scheme, if Figure 1K As shown in the figure, during the training process, the obtained training output target display image (i.e., the training style image output) is compared with the pre-acquired actual target display image (i.e., the original image Ground Truth) to obtain the loss, and the loss function (i.e., MSE loss) is obtained. Then, back-propagation training is performed to obtain the trained pipeline model (i.e., pipeline).
[0130] In some embodiments, Figure 1LAs shown, this application specifically describes the image style transfer process using an embodiment:
[0131] The user releases a style parameter change signal (i.e., target conversion style) through the button of the remote control. After receiving the signal, the transmission parameters will be changed and passed to the image style generation algorithm together with the image source to be displayed (i.e., the current display image) to obtain the style-converted image (i.e., the target display image). Finally, the image is displayed on the screen of the display terminal. For example, the style of the displayed image is style 1, style 2, ..., or style n.
[0132] It allows users to interact with the terminal and see the picture style they want according to their preferences, thereby improving the user experience and increasing the cost-effectiveness of the display terminal and screen.
[0133] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.
[0134] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0135] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides an image style conversion device.
[0136] refer to Figure 2 , the image style conversion device comprises:
[0137] The acquisition module 201 is configured to acquire a target conversion style of a currently displayed image and determine a basic color lookup table corresponding to the currently displayed image;
[0138] A prediction module 202 is configured to perform prediction processing using the target conversion style and the current display image to determine a weight parameter of the current display image relative to the basic color lookup table;
[0139] The fusion module 203 is configured to fuse the weight parameter and the basic color lookup table to obtain a target color lookup table, and obtain a target display image corresponding to the target conversion style through the target color lookup table based on the current display image.
[0140] In some embodiments, the acquisition module 201 includes:
[0141] An acquisition unit is configured to acquire a prediction data set, wherein the prediction data set includes a plurality of style images, each style image corresponding to a plurality of pixel combinations;
[0142] A selection unit, configured to select a target pixel combination from the plurality of pixel combinations according to a preset value interval;
[0143] An input unit is configured to input the target pixel combination into a trained convolutional neural network model, obtain an output result through the convolutional neural network model, and obtain the basic color lookup table based on the target pixel combination and the output result.
[0144] In some embodiments, the input unit is specifically configured as:
[0145] The target pixel combination and the output result are stored correspondingly to obtain an initial color lookup table;
[0146] Obtaining a true output label corresponding to the target pixel combination, and constructing a loss function using the true output label and the output result;
[0147] The initial color lookup table is adjusted according to the loss function to obtain the basic color lookup table.
[0148] In some embodiments, the acquisition unit is specifically configured to:
[0149] Get sample images of multiple styles and perform the following operations on each style:
[0150] Performing instance segmentation on the sample image to obtain a foreground image and a background image;
[0151] Performing incremental processing on the background image to obtain an incrementally processed background image, and performing incremental processing on the foreground image to obtain an incrementally processed foreground image;
[0152] The incrementally processed background image and the incrementally processed foreground image are combined to obtain the prediction data set.
[0153] In some embodiments, the prediction data set is obtained using a trained diffusion model, which includes a latent diffusion model and a low-rank adaptation model. The diffusion model is obtained by training the low-rank adaptation model using sample images of the multiple styles after freezing the latent diffusion model.
[0154] In some embodiments, the prediction module 202 is specifically configured to:
[0155] Performing embedded coding on the target conversion style to obtain an embedded coding result;
[0156] Performing feature extraction on the currently displayed image to obtain a feature-extracted image, and performing position encoding on the feature-extracted image to obtain a position encoding result;
[0157] Natural language processing is performed on the embedded coding result and the position coding result to obtain a weight parameter of the current display image relative to the basic color lookup table.
[0158] In some embodiments, the weight parameters are obtained using a pre-trained large language model, where the large language model includes a normalized exponential function layer for classification prediction.
[0159] In some embodiments, the fusion module 203 is specifically configured to:
[0160] The target color lookup table is obtained by performing weighted summation processing using the weight parameter and the basic color lookup table.
[0161] In some embodiments, the fusion module 203 is specifically configured to:
[0162] Acquire a pixel combination of the stitched image;
[0163] Searching the target pixel combination corresponding to the pixel combination of the spliced image from the target color lookup table;
[0164] A target display image corresponding to a target conversion style is obtained based on the target pixel combination.
[0165] In some embodiments, the input dimensions corresponding to the target color lookup table include: a first red sub-pixel, a first green sub-pixel, a first blue sub-pixel, and a feature extraction image corresponding to any target pixel among a plurality of pixels of the stitched image;
[0166] The output dimensions corresponding to the target color lookup table include: a second red sub-pixel, a second green sub-pixel, and a second blue sub-pixel corresponding to any one of the target pixels in the target display image.
[0167] In some embodiments, the target display image is obtained using a trained pipeline model, which is obtained by back-propagation training based on a loss function constructed using a pre-acquired training current display image and a training output target display image obtained by a training target conversion style, and a pre-acquired actual target display image.
[0168] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.
[0169] The device of the above embodiment is used to implement the corresponding image style conversion method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.
[0170] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the image style conversion method described in any of the above embodiments is implemented.
[0171] Figure 3 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 301, a memory 302, an input / output interface 303, a communication interface 304, and a bus 305. The processor 301, the memory 302, the input / output interface 303, and the communication interface 304 are connected to each other in communication within the device through the bus 305.
[0172] The processor 301 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.
[0173] The memory 302 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 302 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 302 and called and executed by the processor 301.
[0174] The input / output interface 303 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.
[0175] The communication interface 304 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0176] The bus 305 comprises a pathway for transmitting information between the various components of the device (eg, the processor 301 , the memory 302 , the input / output interface 303 , and the communication interface 304 ).
[0177] It should be noted that, although the above device only shows the processor 301, the memory 302, the input / output interface 303, the communication interface 304 and the bus 305, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.
[0178] The electronic device of the above embodiment is used to implement the corresponding image style conversion method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.
[0179] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the image style conversion method described in any of the above embodiments.
[0180] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.
[0181] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the image style conversion method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0182] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a computer program product, including computer program instructions. When the computer program instructions are run on a computer, the computer executes the method described in any of the above embodiments, which has the beneficial effects of the corresponding method embodiments and will not be repeated here.
[0183] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.
[0184] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.
[0185] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.
[0186] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.
Claims
1. A method for image style transfer, characterized in that: include: Acquire a target conversion style of a currently displayed image, and determine a basic color lookup table corresponding to the currently displayed image; Performing prediction processing using the target conversion style and the current display image to determine a weight parameter of the current display image relative to the basic color lookup table; The weight parameter and the basic color lookup table are fused to obtain a target color lookup table, and a target display image corresponding to the target conversion style is obtained through the target color lookup table based on the current display image.
2. The method according to claim 1, characterized in that The determining of a basic color lookup table corresponding to the currently displayed image comprises: Acquire a prediction data set, wherein the prediction data set includes a plurality of style images, each style image corresponding to a plurality of pixel combinations; Selecting a target pixel combination from the plurality of pixel combinations according to a preset value interval; The target pixel combination is input into a trained convolutional neural network model, an output result is obtained through the convolutional neural network model, and the basic color lookup table is obtained based on the target pixel combination and the output result.
3. The method according to claim 2, characterized in that The obtaining the basic color lookup table based on the target pixel combination and the output result comprises: The target pixel combination and the output result are stored correspondingly to obtain an initial color lookup table; Obtaining a true output label corresponding to the target pixel combination, and constructing a loss function using the true output label and the output result; The initial color lookup table is adjusted according to the loss function to obtain the basic color lookup table.
4. The method according to claim 2, characterized in that: The step of obtaining a prediction data set includes: Get sample images of multiple styles and perform the following operations on each style: Performing instance segmentation on the sample image to obtain a foreground image and a background image; Performing incremental processing on the background image to obtain an incrementally processed background image, and performing incremental processing on the foreground image to obtain an incrementally processed foreground image; The incrementally processed background image and the incrementally processed foreground image are combined to obtain the prediction data set.
5. The method according to claim 4, characterized in that The prediction data set is obtained by using a trained diffusion model, wherein the diffusion model includes a latent diffusion model and a low-rank adaptation model, and the diffusion model is obtained by training the low-rank adaptation model using sample images of the multiple styles after freezing the latent diffusion model.
6. The method according to claim 1, characterized in that The step of performing prediction processing by using the target conversion style and the current display image to determine a weight parameter of the current display image relative to the basic color lookup table includes: Performing embedded coding on the target conversion style to obtain an embedded coding result; Performing feature extraction on the currently displayed image to obtain a feature-extracted image, and performing position encoding on the feature-extracted image to obtain a position encoding result; Natural language processing is performed on the embedded coding result and the position coding result to obtain a weight parameter of the current display image relative to the basic color lookup table.
7. The method according to claim 6, characterized in that The weight parameters are obtained using a pre-trained large language model, where the large language model includes a normalized exponential function layer for classification prediction.
8. The method according to claim 1, characterized in that The step of fusing the weight parameter with the basic color lookup table to obtain a target color lookup table includes: The target color lookup table is obtained by performing weighted summation processing using the weight parameter and the basic color lookup table.
9. The method according to claim 1, characterized in that: The step of obtaining a target display image corresponding to the target conversion style through the target color lookup table based on the current display image includes: Acquire a feature extraction image corresponding to the currently displayed image, and perform splicing processing on the feature extraction image and the currently displayed image to obtain a spliced image; Acquire a pixel combination of the stitched image; Searching the target pixel combination corresponding to the pixel combination of the spliced image from the target color lookup table; A target display image corresponding to a target conversion style is obtained based on the target pixel combination.
10. The method according to claim 9, characterized in that The input dimensions corresponding to the target color lookup table include: a first red sub-pixel, a first green sub-pixel, a first blue sub-pixel and a feature extraction image corresponding to any target pixel among the plurality of pixels of the stitched image; The output dimensions corresponding to the target color lookup table include: a second red sub-pixel, a second green sub-pixel, and a second blue sub-pixel corresponding to any one of the target pixels in the target display image.
11. The method according to claim 1, characterized in that: The target display image is obtained using a trained pipeline model, wherein the pipeline model is obtained by back-propagation training based on a loss function constructed using a pre-acquired training current display image and a training output target display image obtained by converting the style of a training target, and a pre-acquired actual target display image.
12. An image style conversion device, characterized in that: include: an acquisition module, configured to acquire a target conversion style of a currently displayed image, and determine a basic color lookup table corresponding to the currently displayed image; A prediction module, configured to perform prediction processing using the target conversion style and the current display image to determine a weight parameter of the current display image relative to the basic color lookup table; The fusion module is configured to fuse the weight parameter and the basic color lookup table to obtain a target color lookup table, and obtain a target display image corresponding to the target conversion style through the target color lookup table based on the current display image.
13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 11 is implemented.
14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 11.
15. A computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 11.
Citation Information
Cited By
Pathological image self-supervision dyeing normalization method and device
CN119399208A
Virtual staining method and system for pathological section image
CN120259484A
Adaptive color management method and device based on scene recognition
CN120411548A
Electronic device and method for generating color lookup table
CN121883617A
