Image processing method and apparatus, and computer device and storage medium

Through model fusion technology, multiple image style models are fused to the stylized processing information of the images to be processed, and a variety of image style models are solved, which solves the problem of inefficiency in the processing of various models in the prior art, and realizes efficient multi-style image processing.

WO2024245229A1PCT designated stage expired Publication Date: 2024-12-05BEIJING ZITIAO NETWORK TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/095724
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-31
Filing Date
2024-05-28
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

When performing multiple stylized processing of images in the prior art, multiple network models are required, resulting in inefficiency and excessive data volume, making it difficult to efficiently process.

Method used

By receiving the stylized processing information of the image to be processed, the model fusion processing of each image style is performed based on the stylization degree information, and the target image processing model is obtained, and multiple style images can be generated in just one processing.

Benefits of technology

Improve image processing efficiency, reduce model deployment and storage requirements, and achieve more efficient multi-style processing without reducing image processing quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024095724_05122024_PF_FP_ABST
    Figure CN2024095724_05122024_PF_FP_ABST
Patent Text Reader

Abstract

An image processing method and apparatus, and a computer device and a storage medium. The method comprises: acquiring an image to be processed (S101); receiving stylization processing information which is set for said image, wherein the stylization processing information comprises stylization degree information respectively corresponding to a plurality of image styles in preset image styles (S102); on the basis of the stylization processing information, performing model fusion processing on image processing models respectively corresponding to the image styles, so as to obtain a target image processing model (S103); and processing said image on the basis of the target image processing model, so as to obtain a target style image matching the stylization degree information (S104).
Need to check novelty before this filing date? Find Prior Art

Description

Image processing method, device, computer equipment and storage medium

[0001] This application claims priority to Chinese patent application No. 202310633052.1 filed on May 31, 2023, the entire text of which is incorporated herein by reference as a part of this application. Technical Field

[0002] The present disclosure relates to an image processing method, apparatus, computer equipment, and storage medium. Background Art

[0003] With the rapid development of artificial intelligence technology, more and more people are beginning to use network models to stylize images to improve image processing efficiency.

[0004] In related technologies, when stylizing an image to be processed, if one wants to make the image to be processed have a certain image style, one often needs to use a network model corresponding to the image style to process the image to be processed. However, when one needs to make the image to be processed have multiple image styles, one needs to use multiple network models accordingly, which is inefficient, and deploying multiple models at the same time results in too much data.

[0005] Summary of the Invention

[0006] The embodiments of the present disclosure at least provide an image processing method, apparatus, computer equipment, and storage medium.

[0007] In a first aspect, an embodiment of the present disclosure provides an image processing method, comprising:

[0008] Get the image to be processed;

[0009] receiving stylization processing information set for the image to be processed; wherein the stylization processing information includes stylization degree information corresponding to a plurality of image styles in a preset image style;

[0010] Based on the stylized processing information, the image processing models corresponding to the respective image styles are subjected to model fusion processing to obtain a target image processing model;

[0011] The image to be processed is processed based on the target image processing model to obtain a target style image that matches the stylization degree information.

[0012] In one possible implementation, for any image style, the method further includes adjusting the image processing model corresponding to the image style according to the following steps:

[0013] For any image style, obtain a pre-trained image processing model corresponding to the image style;

[0014] Fine-tune the target parameter module among the multiple parameter modules of the pre-trained image processing model.

[0015] In one possible implementation, the fine-tuning of a target parameter module among the multiple parameter modules of the pre-trained image processing model includes:

[0016] For any of the target parameter modules, based on a parameter matrix decomposition size matching the target parameter module, the parameter matrix of the target parameter module is decomposed to obtain a plurality of parameter matrices after decomposition;

[0017] Parameter adjustment is performed on the multiple parameter matrices after the decomposition process.

[0018] In one possible implementation, the image processing models corresponding to the respective image styles have the same parameter modules except for a target parameter module, where the target parameter module is a module adjusted during fine-tuning of the image processing model.

[0019] The image processing models corresponding to the respective image styles are subjected to model fusion processing based on the stylized processing information to obtain a target image processing model, including:

[0020] Based on the stylization degree information corresponding to each image style, the target parameter modules in the image processing models corresponding to each image style are fused to obtain a fused parameter module;

[0021] The target image processing model is determined based on other parameter modules except the target parameter module and the fusion parameter module in any image processing model.

[0022] In a possible implementation, the stylization degree information is represented by a stylization degree parameter;

[0023] The image processing models corresponding to the respective image styles are subjected to model fusion processing based on the stylized processing information to obtain a target image processing model, including:

[0024] Normalizing the stylization degree parameters corresponding to the multiple image styles to obtain normalized degree parameters corresponding to the multiple image styles;

[0025] Based on the normalization degree parameters corresponding to the multiple image styles, the image processing models corresponding to the respective image styles are subjected to model fusion processing to obtain a target image processing model.

[0026] In a possible implementation, the receiving stylized processing information set for the image to be processed includes:

[0027] Performing image feature extraction on the image to be processed to obtain image features corresponding to the image to be processed;

[0028] Determining stylized recommendation information corresponding to the image to be processed based on image features corresponding to the image to be processed, and displaying the stylized recommendation information;

[0029] In response to a selection operation on the stylized recommended information, the stylized recommended information is determined as stylized processing information set for the image to be processed.

[0030] In a possible implementation, determining stylized recommendation information corresponding to the image to be processed based on image features corresponding to the image to be processed includes:

[0031] Determining feature similarity between image features corresponding to the image to be processed and standard image features corresponding to a preset standard image;

[0032] Based on the feature similarity and the standard stylization processing information corresponding to the preset standard image, stylization recommendation information corresponding to the image to be processed is determined.

[0033] In a second aspect, an embodiment of the present disclosure further provides an image processing device, including:

[0034] An acquisition module, used for acquiring an image to be processed;

[0035] A receiving module, configured to receive stylized processing information set for the image to be processed; wherein the stylized processing information includes stylized degree information corresponding to a plurality of image styles in a preset image style;

[0036] a fusion module configured to perform model fusion processing on the image processing models corresponding to the respective image styles based on the stylized processing information to obtain a target image processing model;

[0037] A processing module is used to process the image to be processed based on the target image processing model to obtain a target style image that matches the stylization degree information.

[0038] In one possible implementation, for any image style, the fusion module is further configured to adjust the image processing model corresponding to the image style according to the following steps:

[0039] For any image style, obtain a pre-trained image processing model corresponding to the image style;

[0040] Fine-tune the target parameter module among the multiple parameter modules of the pre-trained image processing model.

[0041] In one possible implementation, the fusion module, when fine-tuning the target parameter module among the multiple parameter modules of the pre-trained image processing model, is configured to:

[0042] For any of the target parameter modules, based on a parameter matrix decomposition size matching the target parameter module, the parameter matrix of the target parameter module is decomposed to obtain a plurality of parameter matrices after decomposition;

[0043] Parameter adjustment is performed on the multiple parameter matrices after the decomposition process.

[0044] In one possible implementation, the image processing models corresponding to the respective image styles have the same parameter modules except for a target parameter module, where the target parameter module is a module adjusted during fine-tuning of the image processing model.

[0045] The fusion module, when performing model fusion processing on the image processing models corresponding to the respective image styles based on the stylized processing information to obtain a target image processing model, is used to:

[0046] Based on the stylization degree information corresponding to each image style, the target parameter modules in the image processing models corresponding to each image style are fused to obtain a fused parameter module;

[0047] The target image processing model is determined based on other parameter modules except the target parameter module and the fusion parameter module in any image processing model.

[0048] In a possible implementation, the stylization degree information is represented by a stylization degree parameter;

[0049] The fusion module, when performing model fusion processing on the image processing models corresponding to the respective image styles based on the stylized processing information to obtain a target image processing model, is used to:

[0050] Normalizing the stylization degree parameters corresponding to the multiple image styles to obtain normalized degree parameters corresponding to the multiple image styles;

[0051] Based on the normalization degree parameters corresponding to the multiple image styles, the image processing models corresponding to the respective image styles are subjected to model fusion processing to obtain a target image processing model.

[0052] In a possible implementation, the receiving module, when receiving the stylized processing information set for the image to be processed, is configured to:

[0053] Performing image feature extraction on the image to be processed to obtain image features corresponding to the image to be processed;

[0054] Determining stylized recommendation information corresponding to the image to be processed based on image features corresponding to the image to be processed, and displaying the stylized recommendation information;

[0055] In response to a selection operation on the stylized recommended information, the stylized recommended information is determined as stylized processing information set for the image to be processed.

[0056] In one possible implementation, the receiving module, when determining the stylized recommendation information corresponding to the image to be processed based on the image features corresponding to the image to be processed, is configured to:

[0057] Determining feature similarity between image features corresponding to the image to be processed and standard image features corresponding to a preset standard image;

[0058] Based on the feature similarity and the standard stylization processing information corresponding to the preset standard image, stylization recommendation information corresponding to the image to be processed is determined.

[0059] In a third aspect, an embodiment of the present disclosure further provides a computer device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are performed.

[0060] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are executed.

[0061] In order to make the above-mentioned objects, features and advantages of the present disclosure more obvious and easy to understand, optional embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0063] FIG1 shows a flow chart of an image processing method provided by an embodiment of the present disclosure;

[0064] FIG2 shows a schematic diagram of the architecture of an image processing device provided by an embodiment of the present disclosure;

[0065] FIG3 shows a schematic structural diagram of a computer device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0067] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0068] The term "and / or" herein simply describes an association relationship, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, and the existence of B alone. In addition, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0069] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0070] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0071] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0072] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0073] Research has found that when stylizing an image to be processed, if you want the image to have a certain image style, you often need to use the network model corresponding to the image style to process the image. However, when you need to make the image to have multiple image styles, you need to use multiple network models accordingly, which is inefficient.

[0074] Based on the above research, the present disclosure provides an image processing method, apparatus, computer equipment and storage medium, which receives stylized processing information set for the image to be processed, and based on the stylized processing information, performs model fusion processing on the image processing models corresponding to each image style to obtain a target image processing model. Therefore, when performing image processing on the image to be processed, only one processing using the target image processing model is required to complete the image processing process of the image to be processed, and a target style image with multiple image styles is obtained. Compared with the related technology of using multiple network models for processing in sequence, the image processing efficiency is higher.

[0075] To facilitate understanding of this embodiment, an image processing method disclosed in an embodiment of the present disclosure is first introduced in detail. The execution subject of the image processing method provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities. The computer device includes, for example, a terminal device or a server or other processing device. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc. In some possible implementations, the image processing method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0076] 1 , which is a flow chart of an image processing method provided by an embodiment of the present disclosure, includes S101 to S104, wherein:

[0077] S101: Acquire an image to be processed.

[0078] S102: Receive stylization processing information set for the image to be processed; wherein the stylization processing information includes stylization degree information corresponding to a plurality of image styles in a preset image style.

[0079] S103: Based on the stylized processing information, the image processing models corresponding to the respective image styles are subjected to model fusion processing to obtain a target image processing model.

[0080] S104: Processing the image to be processed based on the target image processing model to obtain a target style image that matches the stylization degree information.

[0081] The following is a detailed description of the above steps.

[0082] For S101 and S102,

[0083] The image to be processed may be an image uploaded after user authorization or an image obtained after user photography; the preset image style may include anime style, realistic style, etc.; the stylization degree information may be represented by a stylization degree parameter, and the parameter range of the stylization degree parameter may be a 1-point system (the sum of the stylization degree parameters corresponding to each image style is less than or equal to 1), a 100-point system (the sum of the stylization degree parameters corresponding to each image style is less than or equal to 100), etc. The specific parameter range can be set according to user needs in actual applications, and the embodiments of the present disclosure do not limit this.

[0084] For example, when uploading an image to be processed, the user can trigger the upload button in the upload page corresponding to the image to be processed, and select the image to be processed in the picture selection page displayed after triggering the upload button and upload it; in addition, the user can also trigger the capture button in the upload page corresponding to the image to be processed, and perform image capture in the capture page after triggering the capture button, and after the user initiates the upload operation for the captured image, the captured image is uploaded as the image to be processed.

[0085] Specifically, the received stylized processing information set for the image to be processed may be set and uploaded by the user according to needs.

[0086] For example, the user may set the stylization degree parameters corresponding to each preset image style in the processing page corresponding to the image to be processed, and upload the stylization degree parameters corresponding to each preset image style after the setting is completed.

[0087] In actual applications, since there are many preset image styles, it may be difficult for users to obtain satisfactory image processing effects through a single image processing operation, so multiple image processing operations are often required. Therefore, in order to improve image processing efficiency, a reference is provided for users to set stylized processing information, and stylized recommendation information can be automatically generated so that users can set stylized information under the guidance of the stylized recommendation information.

[0088] In a possible implementation, when receiving the stylized processing information set for the image to be processed, the following steps A1 to A3 may be further performed:

[0089] A1: performing image feature extraction on the image to be processed to obtain image features corresponding to the image to be processed.

[0090] Here, when performing image feature extraction on the image to be processed, the image to be processed can be input into a trained image feature extraction network, so that the image features output by the image feature extraction network in at least one preset feature dimension can be obtained. The network type of the image feature extraction network can be a neural network model capable of deep learning, such as Convolutional Neural Networks (CNN). When training the image feature extraction network, a supervised training method can be adopted.

[0091] A2: Based on the image features corresponding to the image to be processed, determine stylized recommendation information corresponding to the image to be processed, and display the stylized recommendation information.

[0092] Here, after determining the stylized recommendation information corresponding to the image to be processed, the determined stylized recommendation information may be displayed on a processing page corresponding to the image to be processed, so as to provide a reference for the user to set stylized information.

[0093] In a possible implementation, when determining stylized recommendation information corresponding to the image to be processed based on the image features corresponding to the image to be processed, the following steps A21 to A22 may be performed:

[0094] A21: Determine feature similarity between image features corresponding to the image to be processed and standard image features corresponding to a preset standard image.

[0095] Here, the preset standard image can be a pre-set image corresponding to standardized style processing information; the standard image features corresponding to the preset standard image can be obtained after feature extraction through the image feature extraction network, and the image features corresponding to the image to be processed have the same feature dimensions as the standard image features corresponding to the preset standard image.

[0096] Specifically, when determining the feature similarity between the image features corresponding to the image to be processed and the standard image features corresponding to the preset standard images, the feature similarity between the image features corresponding to the image to be processed and the standard image features corresponding to each preset standard image can be determined respectively.

[0097] Exemplarily, taking the preset standard images as standard images 1 to 4 as an example, when determining the feature similarity between the image features corresponding to the image to be processed and the standard image features corresponding to the preset standard images, the feature similarity between the image to be processed and the standard image features corresponding to standard images 1 to 4 can be determined respectively.

[0098] A22: Determine stylized recommendation information corresponding to the image to be processed based on the feature similarity and the standard stylized processing information corresponding to the preset standard image.

[0099] Specifically, when determining the stylized recommendation information corresponding to the image to be processed based on the feature similarity and the standard stylized processing information corresponding to the preset standard image, the target standard image among the multiple preset standard images can be first determined based on the feature similarity, and the standard stylized processing information corresponding to the target standard image can be used as the stylized recommendation information corresponding to the image to be processed.

[0100] Continuing with the above example, taking the example where the feature similarities between the image to be processed and the standard image features corresponding to standard images 1 to 4 are 0.6, 0.7, 0.5, and 0.9, respectively, standard image 4 can be used as the target standard image, and the standard stylized processing information corresponding to standard image 4 can be used as the stylized recommendation information corresponding to the image to be processed.

[0101] A3: In response to a selection operation on the stylized recommendation information, determine the stylized recommendation information as stylized processing information set for the image to be processed.

[0102] In this way, by determining the stylized recommendation information corresponding to the image to be processed, the user can quickly set the stylized recommendation information that matches the image to be processed by performing a selection operation on the stylized recommendation information, thereby improving image processing efficiency.

[0103] For S103,

[0104] The image processing model corresponding to the image style is a model used to generate images of this image style. The parameter modules other than the target parameter module in the image processing models corresponding to each image style can be the same. The target parameter module is the module adjusted by the image processing model during fine-tuning.

[0105] Among them, the target parameter module can be a different module for different image processing models when performing stylization processing. For example, each image processing model may include an image feature extraction module, a pooling module, etc., and the image feature extraction module performs image feature extraction, and the pooling module performs pooling processing. The module structure and parameters may be the same. The real reason why different image processing models can generate images of different styles lies in the stylization processing operation performed by the stylization processing module. Then the stylization processing module can be the target parameter module.

[0106] The target parameter module can be set by the developer according to the experimental results. When conducting the experiment, the parameter modules can be selected by permutation and combination to generate multiple parameter module experimental combinations, and each parameter module experimental combination can be used for training separately, so that the target experimental combination can be selected from the multiple parameter module experimental combinations according to the training results, and the parameter modules contained in the target experimental combination can constitute the target parameter module.

[0107] In one possible implementation, for any image style, when adjusting the image processing model corresponding to the image style, the following steps B1 to B2 may be performed:

[0108] B1: For any image style, obtain a pre-trained image processing model corresponding to the image style.

[0109] Here, in order to improve the training speed of the image processing model and be able to complete the network parameter adjustment of the image processing model in a small sample scenario with less sample data, a model parameter adjustment paradigm of a pre-trained model plus fine-tuning processing can be used.

[0110] B2: Fine-tune the target parameter module among the multiple parameter modules of the pre-trained image processing model.

[0111] In this way, by fine-tuning only the target parameter module among the multiple parameter modules of the pre-trained image processing model during fine-tuning, the training speed can be further improved and the training scenario with small samples can be better adapted.

[0112] In one possible implementation, when fine-tuning the target parameter module among the multiple parameter modules of the pre-trained image processing model, the following steps B21 to B22 may be performed:

[0113] B21: For any of the target parameter modules, based on a parameter matrix decomposition size matching the target parameter module, decompose the parameter matrix of the target parameter module to obtain a plurality of parameter matrices after decomposition.

[0114] Specifically, after decomposing the parameter matrix of the target parameter module to obtain multiple parameter matrices after decomposition, a new image processing model can be obtained; wherein, the new image processing model can be expressed by the following formula: W2=W1+alpha*A*B

[0115] Among them, W2 represents a new image processing model obtained by decomposing the parameter matrix of the target parameter module to obtain multiple parameter matrices after decomposition; W1 represents other parameter modules except the target parameter module; A and B represent parameter matrices obtained by decomposing the parameter matrix of the target parameter module; alpha represents a preset stylization degree parameter, and the preset stylization degree parameters in the new image processing models corresponding to each image style can be the same or set separately for different image styles.

[0116] B22: Perform parameter adjustment on the multiple parameter matrices after the decomposition process.

[0117] The parameter matrix decomposition size may be pre-set for the target parameter module, for example, 4, 8, etc.

[0118] For example, taking the parameter matrix size before decomposition as 128*128 and the parameter matrix decomposition size as 4 as an example, after the parameter matrix of the target parameter module is decomposed, two parameter matrices of 128*4 and 4*128 can be obtained, so that the number of parameters in the parameter matrix obtained after decomposition is only 1 / 16 of the number of parameters in the parameter matrix before decomposition, which greatly reduces the number of parameters in the parameter matrix and facilitates rapid adjustment of network parameters.

[0119] Specifically, when performing parameter adjustment, the parameters of the multiple parameter matrices after the decomposition processing can be adjusted according to the preset parameter adjustment rounds, and the parameter adjustment can be ended after the preset parameter adjustment rounds are completed; or, other parameter adjustment end conditions can be set to end the parameter adjustment of the parameter matrix after detecting that the parameter adjustment conditions are met. The other parameter adjustment end conditions can be detecting that the network accuracy after the pre-trained image processing model parameter adjustment reaches the preset accuracy, the distance to the start of parameter adjustment reaches a preset time, etc. The embodiment of the present disclosure does not limit how to adjust the network parameters, and is based on whether the model accuracy of the network model can be improved in actual applications.

[0120] In this way, by adjusting the parameters of the multiple parameter matrices after the decomposition processing, the speed of the pre-trained image processing model during fine-tuning processing can be improved, and the multiple parameter matrices after the decomposition processing occupy less space when stored and are easier to deploy.

[0121] In addition, when adjusting parameters, in addition to adjusting the parameters of the multiple parameter matrices after decomposition processing, other network parameters can also be adjusted, for example, other parameter modules other than the target parameter module can also be adjusted to improve model accuracy.

[0122] In one possible implementation, when the image processing models corresponding to the respective image styles are subjected to model fusion processing based on the stylized processing information to obtain a target image processing model, the following steps C1 to C2 may be performed:

[0123] C1: Based on the stylization degree information corresponding to each image style, the target parameter modules in the image processing models corresponding to each image style are fused to obtain a fused parameter module.

[0124] For example, taking the number of image styles as 3, the formula for obtaining the fusion parameter module can be: W α =alpha0*A0*B0+alpha1*A1*B1+alpha2*A2*B2

[0125] Among them, W αRepresents the fused parameter module after fusion, alpha0 represents the stylization degree information corresponding to the first image style, A0 represents the first parameter matrix obtained by decomposing the parameter matrix of the target parameter module corresponding to the first image style, and B0 represents the second parameter matrix obtained by decomposing the parameter matrix of the target parameter module corresponding to the first image style; alpha1 represents the stylization degree information corresponding to the second image style, A1 represents the first parameter matrix obtained by decomposing the parameter matrix of the target parameter module corresponding to the second image style, and B1 represents the second parameter matrix obtained by decomposing the parameter matrix of the target parameter module corresponding to the second image style; alpha2 represents the stylization degree information corresponding to the third image style, A2 represents the first parameter matrix obtained by decomposing the parameter matrix of the target parameter module corresponding to the third image style, and B2 represents the second parameter matrix obtained by decomposing the parameter matrix of the target parameter module corresponding to the third image style; Among them, A can be initialized by Gaussian noise distribution before parameter adjustment, and B can be initialized with 0 as the initial value before parameter adjustment.

[0126] C2: Determine the target image processing model based on other parameter modules except the target parameter module and the fusion parameter module in any image processing model.

[0127] Continuing from the above example, when determining the target image processing model based on other parameter modules except the target parameter module and the fusion parameter module, the following formula can be used: new =W0+W α

[0128] Among them, W new represents the target image processing model; W0 represents other parameter modules except the target parameter module, W α Represents the fusion parameter module.

[0129] In this way, since other parameter modules except the target parameter module can be the same, when storing the image processing models corresponding to each image style, there is no need to store the complete model parameters corresponding to each image style. It is only necessary to store multiple parameter matrices after decomposition processing, which occupies less storage space.

[0130] In actual applications, the stylization degree parameter set by the user may not match the coefficient (i.e., alpha in the above formula) used when the image processing models corresponding to each image style are fused together. For example, the stylization degree parameter set by the user is 100 points, while the coefficient actually used when performing model fusion processing is 1 point (i.e., the sum of the coefficients is 1). Therefore, in order to perform model fusion processing on the image processing models corresponding to each image style without affecting the user's interactive experience, thereby obtaining the target image processing model, the stylization degree parameter can be processed accordingly before model fusion processing.

[0131] In one possible implementation, when the image processing models corresponding to the respective image styles are subjected to model fusion processing based on the stylized processing information to obtain a target image processing model, the following steps D1 to D2 may be performed:

[0132] D1: Normalizing the stylization degree parameters corresponding to the multiple image styles respectively to obtain normalized degree parameters corresponding to the multiple image styles respectively.

[0133] D2: Based on the normalization degree parameters corresponding to the multiple image styles, the image processing models corresponding to the respective image styles are subjected to model fusion processing to obtain a target image processing model.

[0134] Here, since the coefficient actually used in the model fusion processing is a 1-point system, the stylization degree parameters corresponding to the various image styles set by the user can be normalized respectively, so as to obtain normalization degree parameters that meet the 1-point system. Based on the normalization degree parameters corresponding to the various image styles, the image processing models corresponding to each image style are subjected to model fusion processing to obtain the target image processing model.

[0135] Specifically, when the image processing models corresponding to each image style are fused based on the normalization degree parameters corresponding to the multiple image styles to obtain the target image processing model, the normalization degree parameters corresponding to each image style can be used as alpha in the above formula, so that the model fusion processing can be performed through the formula to obtain the target image processing model.

[0136] In this way, by setting a normalization processing mechanism for the stylization degree parameter, the interactive page can be set more flexibly at the user interaction level, and the set interactive page can be made more in line with the user's actual habits.

[0137] For S104,

[0138] Specifically, when the image to be processed is processed based on the target image processing model to obtain a target style image that matches the stylization degree information, the image to be processed can be input into the target image processing model to obtain a target style image corresponding to the image to be processed output by the target image processing model.

[0139] The image processing method provided by the embodiment of the present disclosure receives stylized processing information set for the image to be processed, and based on the stylized processing information, performs model fusion processing on the image processing models corresponding to each image style to obtain a target image processing model. Therefore, when performing image processing on the image to be processed, only one processing using the target image processing model is required to complete the image processing process of the image to be processed, and a target style image with multiple image styles is obtained. Compared with the related art of using multiple network models for processing in sequence, the image processing efficiency is higher.

[0140] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0141] Based on the same inventive concept, an image processing device corresponding to the image processing method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned image processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0142] 2 is a schematic diagram of the architecture of an image processing device provided by an embodiment of the present disclosure, wherein the device includes: an acquisition module 201, a receiving module 202, a fusion module 203, and a processing module 204; wherein,

[0143] An acquisition module 201 is used to acquire an image to be processed;

[0144] The receiving module 202 is configured to receive stylization processing information set for the image to be processed; wherein the stylization processing information includes stylization degree information corresponding to a plurality of image styles in the preset image style;

[0145] A fusion module 203 is configured to perform model fusion processing on the image processing models corresponding to the respective image styles based on the stylized processing information to obtain a target image processing model;

[0146] The processing module 204 is configured to process the image to be processed based on the target image processing model to obtain a target style image that matches the stylization degree information.

[0147] In a possible implementation, for any image style, the fusion module 203 is further configured to adjust the image processing model corresponding to the image style according to the following steps:

[0148] For any image style, obtain a pre-trained image processing model corresponding to the image style;

[0149] Fine-tune the target parameter module among the multiple parameter modules of the pre-trained image processing model.

[0150] In one possible implementation, the fusion module 203, when fine-tuning the target parameter module among the multiple parameter modules of the pre-trained image processing model, is configured to:

[0151] For any of the target parameter modules, based on a parameter matrix decomposition size matching the target parameter module, the parameter matrix of the target parameter module is decomposed to obtain a plurality of parameter matrices after decomposition;

[0152] Parameter adjustment is performed on the multiple parameter matrices after the decomposition process.

[0153] In one possible implementation, the image processing models corresponding to the respective image styles have the same parameter modules except for a target parameter module, where the target parameter module is a module adjusted during fine-tuning of the image processing model.

[0154] The fusion module 203 performs model fusion processing on the image processing models corresponding to the respective image styles based on the stylized processing information to obtain a target image processing model, and is used to:

[0155] Based on the stylization degree information corresponding to each image style, the target parameter modules in the image processing models corresponding to each image style are fused to obtain a fused parameter module;

[0156] The target image processing model is determined based on other parameter modules except the target parameter module and the fusion parameter module in any image processing model.

[0157] In a possible implementation, the stylization degree information is represented by a stylization degree parameter;

[0158] The fusion module 203 performs model fusion processing on the image processing models corresponding to the respective image styles based on the stylized processing information to obtain a target image processing model, and is used to:

[0159] Normalizing the stylization degree parameters corresponding to the multiple image styles to obtain normalized degree parameters corresponding to the multiple image styles;

[0160] Based on the normalization degree parameters corresponding to the multiple image styles, the image processing models corresponding to the respective image styles are subjected to model fusion processing to obtain a target image processing model.

[0161] In a possible implementation, the receiving module 202, when receiving the stylized processing information set for the image to be processed, is configured to:

[0162] Performing image feature extraction on the image to be processed to obtain image features corresponding to the image to be processed;

[0163] Determining stylized recommendation information corresponding to the image to be processed based on image features corresponding to the image to be processed, and displaying the stylized recommendation information;

[0164] In response to a selection operation on the stylized recommended information, the stylized recommended information is determined as stylized processing information set for the image to be processed.

[0165] In one possible implementation, when determining stylized recommendation information corresponding to the image to be processed based on image features corresponding to the image to be processed, the receiving module 202 is configured to:

[0166] Determining feature similarity between image features corresponding to the image to be processed and standard image features corresponding to a preset standard image;

[0167] Based on the feature similarity and the standard stylization processing information corresponding to the preset standard image, stylization recommendation information corresponding to the image to be processed is determined.

[0168] The image processing device provided by the embodiment of the present disclosure receives stylized processing information set for the image to be processed, and based on the stylized processing information, performs model fusion processing on the image processing models corresponding to each image style to obtain a target image processing model. Therefore, when performing image processing on the image to be processed, only one processing using the target image processing model is required to complete the image processing process of the image to be processed, and a target style image with multiple image styles is obtained. Compared with the related art of using multiple network models for processing in sequence, the image processing efficiency is higher.

[0169] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0170] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Referring to Figure 3, there is a schematic diagram of the structure of the computer device 300 provided in the embodiment of the present disclosure, including a processor 301, a memory 302, and a bus 303. Among them, the memory 302 is used to store execution instructions, including a memory 3021 and an external memory 3022; the memory 3021 here is also called an internal memory, which is used to temporarily store the calculation data in the processor 301, as well as the data exchanged with the external memory 3022 such as a hard disk. The processor 301 exchanges data with the external memory 3022 through the memory 3021. When the computer device 300 is running, the processor 301 communicates with the memory 302 through the bus 303, so that the processor 301 executes the following instructions:

[0171] Get the image to be processed;

[0172] receiving stylization processing information set for the image to be processed; wherein the stylization processing information includes stylization degree information corresponding to a plurality of image styles in a preset image style;

[0173] Based on the stylized processing information, the image processing models corresponding to the respective image styles are subjected to model fusion processing to obtain a target image processing model;

[0174] The image to be processed is processed based on the target image processing model to obtain a target style image that matches the stylization degree information.

[0175] In one possible implementation, in the instructions of the processor 301, for any image style, the method further includes adjusting the image processing model corresponding to the image style according to the following steps:

[0176] For any image style, obtain a pre-trained image processing model corresponding to the image style;

[0177] Fine-tune the target parameter module among the multiple parameter modules of the pre-trained image processing model.

[0178] In one possible implementation, in the instructions of the processor 301, the fine-tuning of the target parameter module among the multiple parameter modules of the pre-trained image processing model includes:

[0179] For any of the target parameter modules, based on a parameter matrix decomposition size matching the target parameter module, the parameter matrix of the target parameter module is decomposed to obtain a plurality of parameter matrices after decomposition;

[0180] Parameter adjustment is performed on the multiple parameter matrices after the decomposition process.

[0181] In one possible implementation, in the instructions of the processor 301, the image processing models corresponding to the respective image styles have the same parameter modules except for a target parameter module, where the target parameter module is a module adjusted during fine-tuning of the image processing model.

[0182] The image processing models corresponding to the respective image styles are subjected to model fusion processing based on the stylized processing information to obtain a target image processing model, including:

[0183] Based on the stylization degree information corresponding to each image style, the target parameter modules in the image processing models corresponding to each image style are fused to obtain a fused parameter module;

[0184] The target image processing model is determined based on other parameter modules except the target parameter module and the fusion parameter module in any image processing model.

[0185] In a possible implementation, in the instruction of the processor 301, the stylization degree information is represented by a stylization degree parameter;

[0186] The image processing models corresponding to the respective image styles are subjected to model fusion processing based on the stylized processing information to obtain a target image processing model, including:

[0187] Normalizing the stylization degree parameters corresponding to the multiple image styles to obtain normalized degree parameters corresponding to the multiple image styles;

[0188] Based on the normalization degree parameters corresponding to the multiple image styles, the image processing models corresponding to the respective image styles are subjected to model fusion processing to obtain a target image processing model.

[0189] In a possible implementation, in the instructions of the processor 301, the receiving of stylized processing information set for the image to be processed includes:

[0190] Performing image feature extraction on the image to be processed to obtain image features corresponding to the image to be processed;

[0191] Determining stylized recommendation information corresponding to the image to be processed based on image features corresponding to the image to be processed, and displaying the stylized recommendation information;

[0192] In response to a selection operation on the stylized recommended information, the stylized recommended information is determined as stylized processing information set for the image to be processed.

[0193] In a possible implementation, in the instructions of the processor 301, determining stylized recommendation information corresponding to the image to be processed based on image features corresponding to the image to be processed includes:

[0194] Determining feature similarity between image features corresponding to the image to be processed and standard image features corresponding to a preset standard image;

[0195] Based on the feature similarity and the standard stylization processing information corresponding to the preset standard image, stylization recommendation information corresponding to the image to be processed is determined.

[0196] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program executes the steps of the image processing method described in the above method embodiment. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0197] The embodiments of the present disclosure also provide a computer program product, which carries program code. The instructions included in the program code can be used to execute the steps of the image processing method described in the above method embodiment. For details, please refer to the above method embodiment and will not be repeated here.

[0198] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0199] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0200] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0201] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0202] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0203] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present disclosure, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims.

Claims

1. An image processing method, comprising: Get the image to be processed; Receiving stylized processing information set for the image to be processed; wherein the stylized processing information includes stylized degree information corresponding to a plurality of image styles in the preset image style; Based on the stylized processing information, the image processing models corresponding to the image styles are subjected to model fusion processing to obtain a target image processing model; The image to be processed is processed based on the target image processing model to obtain a target style image that matches the stylization degree information.

2. The method according to claim 1, wherein: For any image style, the method further includes adjusting the image processing model corresponding to the image style according to the following steps: For any image style, obtain a pre-trained image processing model corresponding to the image style; Fine-tune a target parameter module among the multiple parameter modules of the pre-trained image processing model.

3. The method according to claim 2, wherein: The fine-tuning of a target parameter module among the plurality of parameter modules of the pre-trained image processing model comprises: For any of the target parameter modules, based on the parameter matrix decomposition size matching the target parameter module, the parameter matrix of the target parameter module is decomposed to obtain a plurality of parameter matrices after decomposition; Parameter adjustment is performed on the multiple parameter matrices after the decomposition process.

4. The method according to any one of claims 1 to 3, wherein: The image processing models corresponding to the respective image styles have the same parameter modules except for the target parameter module, where the target parameter module is a module adjusted during the fine-tuning process of the image processing model; Based on the stylized processing information, the image processing models corresponding to the respective image styles are subjected to model fusion processing to obtain a target image processing model, including: Based on the stylization degree information corresponding to each image style, target parameter modules in the image processing models corresponding to each image style are fused to obtain a fused parameter module; The target image processing model is determined based on other parameter modules except the target parameter module in any image processing model and the fusion parameter module.

5. The method according to any one of claims 1 to 3, wherein: The stylization degree information is represented by a stylization degree parameter; Based on the stylized processing information, the image processing models corresponding to the respective image styles are subjected to model fusion processing to obtain a target image processing model, including: Normalizing the stylization degree parameters corresponding to the multiple image styles respectively to obtain the normalized degree parameters corresponding to the multiple image styles respectively; Based on the normalization degree parameters corresponding to the multiple image styles, the image processing models corresponding to the image styles are subjected to model fusion processing to obtain a target image processing model.

6. The method according to any one of claims 1 to 5, wherein: The receiving stylized processing information set for the image to be processed includes: Extracting image features from the image to be processed to obtain image features corresponding to the image to be processed; Determining stylized recommendation information corresponding to the image to be processed based on image features corresponding to the image to be processed, and displaying the stylized recommendation information; In response to a selection operation on the stylized recommended information, the stylized recommended information is determined as stylized processing information set for the image to be processed.

7. The method according to claim 6, wherein: The determining, based on the image features corresponding to the image to be processed, stylized recommendation information corresponding to the image to be processed includes: Determine the feature similarity between the image features corresponding to the image to be processed and the standard image features corresponding to the preset standard image; Based on the feature similarity and the standard stylization processing information corresponding to the preset standard image, stylization recommendation information corresponding to the image to be processed is determined.

8. An image processing device, comprising: An acquisition module, used for acquiring an image to be processed; A receiving module, configured to receive stylized processing information set for the image to be processed; wherein the stylized processing information includes stylized degree information corresponding to a plurality of image styles in the preset image style; A fusion module, configured to perform model fusion processing on the image processing models corresponding to the respective image styles based on the stylized processing information to obtain a target image processing model; a processing module, used for processing the image to be processed based on the target image processing model, A target style image matching the stylization degree information is obtained.

9. A computer device comprising: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the image processing method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are executed.

Citation Information

Patent Citations

  • Image processing method and device, interactive display device and electronic equipment

    CN111986076A

  • Method and device for generating stylized image, electronic equipment and storage medium

    CN114429418A

  • Certificate identification model configuration method and device, electronic equipment and storage medium

    CN114581706A

  • Image processing method and electronic equipment

    CN114615421A

  • Generation model for generating personalized commodity description based on lightweight fine tuning

    CN114862493A