Image processing method and device, computer equipment and storage medium
Through model fusion technology, multiple image style processing models are fused into one target model, solving the inefficiency problem caused by the sequential processing of multiple models in the prior art, and achieving efficient image processing of multiple image styles.
Patent Information
- Application Number
- CN202310633052.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-13
AI Technical Summary
In the prior art, when processing images, multiple network models are required to process sequentially, resulting in inefficiency, especially when images need to have multiple image styles at the same time, the data volume is too large and the deployment is complex.
By receiving the stylized processing information, the image processing models corresponding to multiple image styles are fused to obtain the target image processing model, and the target image with multiple image styles can be generated in just one processing.
Improve image processing efficiency, reduce the number of models and data volume, and simplify the deployment process.
Smart Images
Figure CN119130778A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method, apparatus, computer equipment and storage medium. Background Art
[0002] With the rapid development of artificial intelligence technology, more and more people are beginning to use network models to stylize images to improve image processing efficiency.
[0003] In the related art, when stylizing an image to be processed, if one wants to make the image to be processed have a certain image style, one often needs to use a network model corresponding to the image style to process the image to be processed. However, if one needs to make the image to be processed have multiple image styles, one needs to use multiple network models accordingly, which is inefficient, and deploying multiple models at the same time results in too much data. Summary of the invention
[0004] The embodiments of the present disclosure at least provide an image processing method, apparatus, computer equipment, and storage medium.
[0005] In a first aspect, an embodiment of the present disclosure provides an image processing method, including:
[0006] Get the image to be processed;
[0007] Receiving stylized processing information set for the image to be processed; wherein the stylized processing information includes stylized degree information corresponding to a plurality of image styles in the preset image style;
[0008] Based on the stylized processing information, the image processing models corresponding to the image styles are subjected to model fusion processing to obtain a target image processing model;
[0009] The image to be processed is processed based on the target image processing model to obtain a target style image that matches the stylization degree information.
[0010] In a possible implementation manner, for any image style, the method further includes adjusting the image processing model corresponding to the image style according to the following steps:
[0011] For any image style, obtain a pre-trained image processing model corresponding to the image style;
[0012] Fine-tune a target parameter module among the multiple parameter modules of the pre-trained image processing model.
[0013] In a possible implementation manner, the fine-tuning of a target parameter module among a plurality of parameter modules of the pre-trained image processing model includes:
[0014] For any of the target parameter modules, based on the parameter matrix decomposition size matching the target parameter module, the parameter matrix of the target parameter module is decomposed to obtain a plurality of parameter matrices after decomposition;
[0015] Parameter adjustment is performed on the multiple parameter matrices after the decomposition process.
[0016] In a possible implementation manner, the image processing models corresponding to the respective image styles have the same parameter modules except for the target parameter module, and the target parameter module is a module adjusted during the fine-tuning process of the image processing model;
[0017] Based on the stylized processing information, the image processing models corresponding to the respective image styles are subjected to model fusion processing to obtain a target image processing model, including:
[0018] Based on the stylization degree information corresponding to each image style, target parameter modules in the image processing models corresponding to each image style are fused to obtain a fused parameter module;
[0019] The target image processing model is determined based on other parameter modules except the target parameter module in any image processing model and the fusion parameter module.
[0020] In a possible implementation manner, the stylization degree information is represented by a stylization degree parameter;
[0021] Based on the stylized processing information, the image processing models corresponding to the respective image styles are subjected to model fusion processing to obtain a target image processing model, including:
[0022] Normalizing the stylization degree parameters corresponding to the multiple image styles respectively to obtain the normalized degree parameters corresponding to the multiple image styles respectively;
[0023] Based on the normalization degree parameters corresponding to the multiple image styles, the image processing models corresponding to the image styles are subjected to model fusion processing to obtain a target image processing model.
[0024] In a possible implementation manner, the receiving stylized processing information set for the image to be processed includes:
[0025] Extracting image features from the image to be processed to obtain image features corresponding to the image to be processed;
[0026] Determining stylized recommendation information corresponding to the image to be processed based on image features corresponding to the image to be processed, and displaying the stylized recommendation information;
[0027] In response to a selection operation on the stylized recommended information, the stylized recommended information is determined as stylized processing information set for the image to be processed.
[0028] In a possible implementation manner, determining stylized recommendation information corresponding to the image to be processed based on image features corresponding to the image to be processed includes:
[0029] Determine the feature similarity between the image features corresponding to the image to be processed and the standard image features corresponding to the preset standard image;
[0030] Based on the feature similarity and the standard stylization processing information corresponding to the preset standard image, stylization recommendation information corresponding to the image to be processed is determined.
[0031] In a second aspect, the present disclosure also provides an image processing device, including:
[0032] An acquisition module, used for acquiring an image to be processed;
[0033] A receiving module, configured to receive stylized processing information set for the image to be processed; wherein the stylized processing information includes stylized degree information corresponding to a plurality of image styles in the preset image style;
[0034] A fusion module, configured to perform model fusion processing on the image processing models corresponding to the respective image styles based on the stylized processing information to obtain a target image processing model;
[0035] A processing module is used to process the image to be processed based on the target image processing model to obtain a target style image that matches the stylization degree information.
[0036] In a possible implementation manner, for any image style, the fusion module is further configured to adjust the image processing model corresponding to the image style according to the following steps:
[0037] For any image style, obtain a pre-trained image processing model corresponding to the image style;
[0038] Fine-tune a target parameter module among the multiple parameter modules of the pre-trained image processing model.
[0039] In a possible implementation, the fusion module, when fine-tuning the target parameter module among the multiple parameter modules of the pre-trained image processing model, is used to:
[0040] For any of the target parameter modules, based on the parameter matrix decomposition size matching the target parameter module, the parameter matrix of the target parameter module is decomposed to obtain a plurality of parameter matrices after decomposition;
[0041] Parameter adjustment is performed on the multiple parameter matrices after the decomposition process.
[0042] In a possible implementation manner, the image processing models corresponding to the respective image styles have the same parameter modules except for the target parameter module, and the target parameter module is a module adjusted during the fine-tuning process of the image processing model;
[0043] The fusion module, when performing model fusion processing on the image processing models corresponding to the respective image styles based on the stylized processing information to obtain a target image processing model, is used to:
[0044] Based on the stylization degree information corresponding to each image style, target parameter modules in the image processing models corresponding to each image style are fused to obtain a fused parameter module;
[0045] The target image processing model is determined based on other parameter modules except the target parameter module in any image processing model and the fusion parameter module.
[0046] In a possible implementation manner, the stylization degree information is represented by a stylization degree parameter;
[0047] The fusion module, when performing model fusion processing on the image processing models corresponding to the respective image styles based on the stylized processing information to obtain a target image processing model, is used to:
[0048] Normalizing the stylization degree parameters corresponding to the multiple image styles respectively to obtain the normalized degree parameters corresponding to the multiple image styles respectively;
[0049] Based on the normalization degree parameters corresponding to the multiple image styles, the image processing models corresponding to the image styles are subjected to model fusion processing to obtain a target image processing model.
[0050] In a possible implementation manner, the receiving module, when receiving the stylized processing information set for the image to be processed, is configured to:
[0051] Extracting image features from the image to be processed to obtain image features corresponding to the image to be processed;
[0052] Determining stylized recommendation information corresponding to the image to be processed based on image features corresponding to the image to be processed, and displaying the stylized recommendation information;
[0053] In response to a selection operation on the stylized recommended information, the stylized recommended information is determined as stylized processing information set for the image to be processed.
[0054] In a possible implementation manner, when determining the stylized recommendation information corresponding to the image to be processed based on the image features corresponding to the image to be processed, the receiving module is configured to:
[0055] Determine the feature similarity between the image features corresponding to the image to be processed and the standard image features corresponding to the preset standard image;
[0056] Based on the feature similarity and the standard stylization processing information corresponding to the preset standard image, stylization recommendation information corresponding to the image to be processed is determined.
[0057] In a third aspect, an embodiment of the present disclosure further provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are performed.
[0058] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are executed.
[0059] The image processing method, apparatus, computer equipment and storage medium provided by the embodiments of the present disclosure receive stylized processing information set for the image to be processed, and based on the stylized processing information, perform model fusion processing on the image processing models corresponding to each image style to obtain a target image processing model. Therefore, when performing image processing on the image to be processed, only one processing using the target image processing model is required to complete the image processing process of the image to be processed, and a target style image with multiple image styles is obtained. Compared with the related art of using multiple network models for processing in sequence, the image processing efficiency is higher.
[0060] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.
[0062] Figure 1 A flowchart of an image processing method provided by an embodiment of the present disclosure is shown;
[0063] Figure 2 A schematic diagram of the architecture of an image processing device provided by an embodiment of the present disclosure is shown;
[0064] Figure 3 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0065] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.
[0066] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0067] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.
[0068] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0069] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.
[0070] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0071] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.
[0072] Research has found that when performing stylized processing on an image to be processed, if you want to make the image to be processed have a certain image style, you often need to use the network model corresponding to the image style to process the image to be processed. However, when you need to make the image to be processed have multiple image styles, you need to use multiple network models accordingly, which is inefficient.
[0073] Based on the above research, the present disclosure provides an image processing method, apparatus, computer equipment and storage medium, which receives stylized processing information set for the image to be processed, and based on the stylized processing information, performs model fusion processing on the image processing models corresponding to each image style to obtain a target image processing model. Therefore, when performing image processing on the image to be processed, only one processing using the target image processing model is required to complete the image processing process of the image to be processed, and a target style image with multiple image styles is obtained. Compared with the related technology of using multiple network models for processing in sequence, the image processing efficiency is higher.
[0074] To facilitate understanding of the present embodiment, an image processing method disclosed in the present embodiment is first introduced in detail. The execution subject of the image processing method provided in the present embodiment is generally a computer device with certain computing capabilities, and the computer device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the image processing method can be implemented by a processor calling a computer-readable instruction stored in a memory.
[0075] See also Figure 1 FIG. 1 is a flowchart of an image processing method provided by an embodiment of the present disclosure, wherein the method includes S101 to S104, wherein:
[0076] S101: Acquire an image to be processed.
[0077] S102: Receive stylized processing information set for the image to be processed; wherein the stylized processing information includes stylized degree information corresponding to a plurality of image styles in a preset image style.
[0078] S103: Based on the stylized processing information, image processing models corresponding to the respective image styles are subjected to model fusion processing to obtain a target image processing model.
[0079] S104: Processing the image to be processed based on the target image processing model to obtain a target style image that matches the stylization degree information.
[0080] The following is a detailed description of the above steps.
[0081] For S101 and S102,
[0082] The image to be processed may be an image uploaded after user authorization or an image obtained after user photography; the preset image style may include anime style, realistic style, etc.; the stylization degree information may be represented by a stylization degree parameter, and the parameter range of the stylization degree parameter may be 1 point (the sum of the stylization degree parameters corresponding to each image style is less than or equal to 1), 100 points (the sum of the stylization degree parameters corresponding to each image style is less than or equal to 100), etc. The specific parameter range may be set according to user needs in actual applications, and the embodiments of the present disclosure are not limited to this.
[0083] Exemplarily, when uploading an image to be processed, the user can trigger the upload button in the upload page corresponding to the image to be processed, and select the image to be processed in the picture selection page displayed after the upload button is triggered and upload it; in addition, the capture button can be triggered in the upload page corresponding to the image to be processed, and the image can be captured in the capture page after the capture button is triggered, and after the user initiates the upload operation for the captured image, the captured image is uploaded as the image to be processed.
[0084] Specifically, the received stylized processing information set for the image to be processed may be set and uploaded by the user according to needs.
[0085] Exemplarily, the user may set the stylization degree parameters corresponding to each preset image style in the processing page corresponding to the image to be processed, and upload the stylization degree parameters corresponding to each preset image style after the setting is completed.
[0086] In actual applications, since there are multiple preset image styles, it may be difficult for users to obtain satisfactory image processing effects through a single image processing operation, so multiple image processing operations are often required. Therefore, in order to improve image processing efficiency and provide users with a reference for setting stylized processing information, stylized recommendation information can also be automatically generated so that users can set stylized information under the guidance of the stylized recommendation information.
[0087] In a possible implementation manner, when receiving the stylized processing information set for the image to be processed, the following steps A1 to A3 may also be performed:
[0088] A1: extracting image features from the image to be processed to obtain image features corresponding to the image to be processed.
[0089] Here, when performing image feature extraction on the image to be processed, the image to be processed can be input into a trained image feature extraction network, so that the image features output by the image feature extraction network in at least one preset feature dimension can be obtained. The network type of the image feature extraction network can be a neural network model capable of deep learning, such as Convolutional Neural Networks (CNN). When training the image feature extraction network, a supervised training method can be adopted.
[0090] A2: Based on the image features corresponding to the image to be processed, determine the stylized recommendation information corresponding to the image to be processed, and display the stylized recommendation information.
[0091] Here, after the stylized recommendation information corresponding to the image to be processed is determined, the determined stylized recommendation information can be displayed in a processing page corresponding to the image to be processed, so as to provide a reference for the user to set the stylized information.
[0092] In a possible implementation manner, when determining the stylized recommendation information corresponding to the image to be processed based on the image features corresponding to the image to be processed, the following steps A21 to A22 may be performed:
[0093] A21: Determine the feature similarity between the image features corresponding to the image to be processed and the standard image features corresponding to the preset standard image.
[0094] Here, the preset standard image may be a pre-set image corresponding to standardized style processing information; the standard image features corresponding to the preset standard image may be obtained after feature extraction through the image feature extraction network, and the image features corresponding to the image to be processed have the same feature dimensions as the standard image features corresponding to the preset standard image.
[0095] Specifically, when determining the feature similarity between the image features corresponding to the image to be processed and the standard image features corresponding to the preset standard images, the feature similarities between the image features corresponding to the image to be processed and the standard image features corresponding to each preset standard image can be determined respectively.
[0096] Exemplarily, taking the preset standard images as standard images 1 to 4 as an example, when determining the feature similarity between the image features corresponding to the image to be processed and the standard image features corresponding to the preset standard images, the feature similarities between the image to be processed and the standard image features corresponding to the standard images 1 to 4 can be determined respectively.
[0097] A22: Determine stylized recommendation information corresponding to the image to be processed based on the feature similarity and the standard stylized processing information corresponding to the preset standard image.
[0098] Specifically, when determining the stylized recommendation information corresponding to the image to be processed based on the feature similarity and the standard stylized processing information corresponding to the preset standard image, the target standard image among the multiple preset standard images can be first determined based on the feature similarity, and the standard stylized processing information corresponding to the target standard image can be used as the stylized recommendation information corresponding to the image to be processed.
[0099] Continuing with the above example, taking the example that the feature similarities between the image to be processed and the standard image features corresponding to standard images 1 to 4 are 0.6, 0.7, 0.5, and 0.9 respectively, standard image 4 can be used as the target standard image, and the standard stylized processing information corresponding to standard image 4 can be used as the stylized recommendation information corresponding to the image to be processed.
[0100] A3: In response to a selection operation on the stylized recommended information, determining the stylized recommended information as stylized processing information set for the image to be processed.
[0101] In this way, by determining the stylized recommendation information corresponding to the image to be processed, the user can quickly set the stylized recommendation information matching the image to be processed through a selection operation on the stylized recommendation information, thereby improving the image processing efficiency.
[0102] For S103,
[0103] The image processing model corresponding to the image style is a model used to generate an image of the image style. The other parameter modules except the target parameter module in the image processing models corresponding to each image style can be the same. The target parameter module is the module adjusted by the image processing model during fine-tuning.
[0104] Among them, the target parameter module can be a different module when different image processing models perform stylized processing. For example, each image processing model may include an image feature extraction module, a pooling module, etc., and the image feature extraction module is performing image feature extraction, and the pooling module is performing pooling processing. The module structure and parameters may be the same. The real reason why different image processing models can generate images of different styles lies in the stylized processing operation performed by the stylized processing module. Then the stylized processing module can be the target parameter module.
[0105] The target parameter module can be set by the developer according to the experimental results. When conducting the experiment, the parameter modules can be selected by permutation and combination to generate multiple parameter module experimental combinations, and each parameter module experimental combination can be used for training respectively, so that the target experimental combination can be selected from the multiple parameter module experimental combinations according to the training results, and the parameter modules included in the target experimental combination can constitute the target parameter module.
[0106] In a possible implementation manner, for any image style, when adjusting the image processing model corresponding to the image style, the following steps B1 to B2 may be performed:
[0107] B1: For any image style, obtain a pre-trained image processing model corresponding to the image style.
[0108] Here, in order to improve the training speed of the image processing model and to be able to complete the network parameter adjustment of the image processing model in a small sample scenario with less sample data, a model parameter adjustment paradigm of a pre-trained model plus fine-tuning processing can be used.
[0109] B2: Fine-tune the target parameter module among the multiple parameter modules of the pre-trained image processing model.
[0110] In this way, by fine-tuning only the target parameter module among the multiple parameter modules of the pre-trained image processing model during fine-tuning, the training speed can be further improved to better adapt to the training scenario of small samples.
[0111] In a possible implementation manner, when fine-tuning the target parameter module among the multiple parameter modules of the pre-trained image processing model, the following steps B21 to B22 may be performed:
[0112] B21: For any of the target parameter modules, based on the parameter matrix decomposition size matching the target parameter module, decompose the parameter matrix of the target parameter module to obtain a plurality of parameter matrices after decomposition.
[0113] Specifically, after decomposing the parameter matrix of the target parameter module to obtain multiple parameter matrices after decomposition, a new image processing model can be obtained; wherein the new image processing model can be expressed by the following formula:
[0114] Among them, represents a new image processing model obtained by decomposing the parameter matrix of the target parameter module to obtain multiple parameter matrices after decomposition; represents other parameter modules except the target parameter module; A and B represent parameter matrices obtained by decomposing the parameter matrix of the target parameter module; represents preset stylization degree parameters, and the preset stylization degree parameters in the new image processing models corresponding to each image style can be the same or can be set separately for different image styles.
[0115] B22: Perform parameter adjustment on the multiple parameter matrices after the decomposition process.
[0116] The parameter matrix decomposition size may be pre-set for the target parameter module, for example, 4, 8, etc.
[0117] Exemplarily, taking the size of the parameter matrix before decomposition as 128*128 and the parameter matrix decomposition size as 4 as an example, after decomposing the parameter matrix of the target parameter module, two parameter matrices of 128*4 and 4*128 can be obtained, so that the number of parameters in the parameter matrix obtained after the decomposition is only 1 / 16 of the number of parameters in the parameter matrix before the decomposition, which greatly reduces the number of parameters in the parameter matrix and facilitates rapid adjustment of network parameters.
[0118] Specifically, when performing parameter adjustment, the parameters of the multiple parameter matrices after the decomposition processing can be adjusted according to preset parameter adjustment rounds, and the parameter adjustment can be ended after the preset parameter adjustment rounds are completed; alternatively, other parameter adjustment end conditions can be set to end the parameter adjustment of the parameter matrix after detecting that the parameter adjustment conditions are met. The other parameter adjustment end conditions can be detecting that the network accuracy after the pre-trained image processing model parameter adjustment reaches the preset accuracy, the distance to the start of parameter adjustment reaches a preset time, etc. The embodiments of the present disclosure do not limit how to adjust the network parameters, and are based on how to improve the model accuracy of the network model in actual applications.
[0119] In this way, by adjusting the parameters of the multiple parameter matrices after the decomposition process, the speed of the pre-trained image processing model during fine-tuning can be improved, and the multiple parameter matrices after the decomposition process occupy less space when stored and are easier to deploy.
[0120] In addition, when adjusting parameters, in addition to adjusting the parameters of the multiple parameter matrices after decomposition processing, other network parameters can also be adjusted, for example, other parameter modules other than the target parameter module can also be adjusted to improve model accuracy.
[0121] In a possible implementation, when the image processing models corresponding to the respective image styles are subjected to model fusion processing based on the stylized processing information to obtain a target image processing model, the following steps C1 to C2 may be performed:
[0122] C1: Based on the stylization degree information corresponding to each image style, the target parameter modules in the image processing models corresponding to each image style are fused to obtain a fused parameter module.
[0123] For example, taking the number of image styles as 3, the formula for obtaining the fusion parameter module can be:
[0124] Among them, represents the fused parameter module after fusion, represents the stylization degree information corresponding to the first image style, represents the first parameter matrix obtained after decomposing the parameter matrix of the target parameter module corresponding to the first image style, represents the second parameter matrix obtained after decomposing the parameter matrix of the target parameter module corresponding to the first image style; represents the stylization degree information corresponding to the second image style, represents the first parameter matrix obtained after decomposing the parameter matrix of the target parameter module corresponding to the second image style, represents the second parameter matrix obtained after decomposing the parameter matrix of the target parameter module corresponding to the second image style; represents the stylization degree information corresponding to the third image style, represents the first parameter matrix obtained after decomposing the parameter matrix of the target parameter module corresponding to the third image style, represents the second parameter matrix obtained after decomposing the parameter matrix of the target parameter module corresponding to the third image style; Among them, A can be initialized by Gaussian noise distribution before parameter adjustment, and B can be initialized with 0 as the initial value before parameter adjustment.
[0125] C2: Determine the target image processing model based on other parameter modules except the target parameter module in any image processing model and the fusion parameter module.
[0126] Following the above example, when determining the target image processing model based on other parameter modules except the target parameter module and the fusion parameter module, the following formula can be used:
[0127] Among them, represents the target image processing model; represents other parameter modules except the target parameter module, and represents the fusion parameter module.
[0128] In this way, since other parameter modules except the target parameter module can be the same, when storing the image processing models corresponding to each image style, there is no need to store the complete model parameters corresponding to each image style. It is only necessary to store multiple parameter matrices after decomposition processing, which occupies less storage space.
[0129] In actual applications, the stylization degree parameter set by the user may not match the coefficient (i.e., in the above formula) used in the model fusion processing of the image processing models corresponding to each image style. For example, the stylization degree parameter set by the user is 100 points, while the coefficient actually used in the model fusion processing is 1 point (i.e., the sum of the coefficients is 1). Therefore, in order to be able to perform model fusion processing on the image processing models corresponding to each image style without affecting the user's interactive experience, thereby obtaining the target image processing model, the stylization degree parameter may be processed accordingly before the model fusion processing.
[0130] In a possible implementation, when the image processing models corresponding to the respective image styles are subjected to model fusion processing based on the stylized processing information to obtain a target image processing model, the following steps D1 to D2 may be performed:
[0131] D1: Normalizing the stylization degree parameters corresponding to the multiple image styles respectively to obtain normalized degree parameters corresponding to the multiple image styles respectively.
[0132] D2: Based on the normalization degree parameters corresponding to the multiple image styles, the image processing models corresponding to the image styles are subjected to model fusion processing to obtain a target image processing model.
[0133] Here, since the coefficient actually used in the model fusion processing is a 1-point system, the stylization degree parameters corresponding to the various image styles set by the user can be normalized respectively, so as to obtain the normalization degree parameters that meet the 1-point system, and then based on the normalization degree parameters corresponding to the various image styles, the image processing models corresponding to each image style are subjected to model fusion processing to obtain the target image processing model.
[0134] Specifically, based on the normalization degree parameters corresponding to the multiple image styles, the image processing models corresponding to each image style are subjected to model fusion processing to obtain the target image processing model. The normalization degree parameters corresponding to each image style can be used as the above formula, so that the model fusion processing can be performed through the formula to obtain the target image processing model.
[0135] In this way, by setting a normalization processing mechanism for the stylization degree parameter, the interactive page can be set more flexibly at the user interaction level, and the set interactive page can be made more in line with the actual habits of the user.
[0136] For S104,
[0137] Specifically, when the image to be processed is processed based on the target image processing model to obtain a target style image matching the stylization degree information, the image to be processed can be input into the target image processing model to obtain a target style image corresponding to the image to be processed output by the target image processing model.
[0138] The image processing method provided by the embodiment of the present disclosure receives stylized processing information set for the image to be processed, and based on the stylized processing information, performs model fusion processing on the image processing models corresponding to each image style to obtain a target image processing model. Therefore, when performing image processing on the image to be processed, only one processing using the target image processing model is required to complete the image processing process of the image to be processed, and a target style image with a variety of image styles is obtained. Compared with the related art of using a variety of network models for processing in sequence, the image processing efficiency is higher.
[0139] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0140] Based on the same inventive concept, an image processing device corresponding to the image processing method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned image processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0141] Reference Figure 2 , which is a schematic diagram of the architecture of an image processing device provided by an embodiment of the present disclosure, the device includes: an acquisition module 201, a receiving module 202, a fusion module 203, and a processing module 204; wherein,
[0142] An acquisition module 201 is used to acquire an image to be processed;
[0143] The receiving module 202 is used to receive stylized processing information set for the image to be processed; wherein the stylized processing information includes stylized degree information corresponding to a plurality of image styles in the preset image style;
[0144] A fusion module 203 is used to perform model fusion processing on the image processing models corresponding to the image styles based on the stylized processing information to obtain a target image processing model;
[0145] The processing module 204 is used to process the image to be processed based on the target image processing model to obtain a target style image that matches the stylization degree information.
[0146] In a possible implementation manner, for any image style, the fusion module 203 is further configured to adjust the image processing model corresponding to the image style according to the following steps:
[0147] For any image style, obtain a pre-trained image processing model corresponding to the image style;
[0148] Fine-tune a target parameter module among the multiple parameter modules of the pre-trained image processing model.
[0149] In a possible implementation, the fusion module 203, when fine-tuning the target parameter module among the multiple parameter modules of the pre-trained image processing model, is used to:
[0150] For any of the target parameter modules, based on the parameter matrix decomposition size matching the target parameter module, the parameter matrix of the target parameter module is decomposed to obtain a plurality of parameter matrices after decomposition;
[0151] Parameter adjustment is performed on the multiple parameter matrices after the decomposition process.
[0152] In a possible implementation manner, the image processing models corresponding to the respective image styles have the same parameter modules except for the target parameter module, and the target parameter module is a module adjusted during the fine-tuning process of the image processing model;
[0153] The fusion module 203, when performing model fusion processing on the image processing models corresponding to the image styles based on the stylized processing information to obtain a target image processing model, is used to:
[0154] Based on the stylization degree information corresponding to each image style, target parameter modules in the image processing models corresponding to each image style are fused to obtain a fused parameter module;
[0155] The target image processing model is determined based on other parameter modules except the target parameter module in any image processing model and the fusion parameter module.
[0156] In a possible implementation manner, the stylization degree information is represented by a stylization degree parameter;
[0157] The fusion module 203, when performing model fusion processing on the image processing models corresponding to the image styles based on the stylized processing information to obtain a target image processing model, is used to:
[0158] Normalizing the stylization degree parameters corresponding to the multiple image styles respectively to obtain the normalized degree parameters corresponding to the multiple image styles respectively;
[0159] Based on the normalization degree parameters corresponding to the multiple image styles, the image processing models corresponding to the image styles are subjected to model fusion processing to obtain a target image processing model.
[0160] In a possible implementation manner, the receiving module 202, when receiving the stylized processing information set for the image to be processed, is configured to:
[0161] Extracting image features from the image to be processed to obtain image features corresponding to the image to be processed;
[0162] Determining stylized recommendation information corresponding to the image to be processed based on image features corresponding to the image to be processed, and displaying the stylized recommendation information;
[0163] In response to a selection operation on the stylized recommended information, the stylized recommended information is determined as stylized processing information set for the image to be processed.
[0164] In a possible implementation manner, when determining the stylized recommendation information corresponding to the image to be processed based on the image features corresponding to the image to be processed, the receiving module 202 is configured to:
[0165] Determine the feature similarity between the image features corresponding to the image to be processed and the standard image features corresponding to the preset standard image;
[0166] Based on the feature similarity and the standard stylization processing information corresponding to the preset standard image, stylization recommendation information corresponding to the image to be processed is determined.
[0167] The image processing device provided by the embodiment of the present disclosure receives stylized processing information set for the image to be processed, and based on the stylized processing information, performs model fusion processing on the image processing models corresponding to each image style to obtain a target image processing model. Therefore, when performing image processing on the image to be processed, only one processing using the target image processing model is required to complete the image processing process of the image to be processed, and a target style image with multiple image styles is obtained. Compared with the related art of using multiple network models for processing in sequence, the image processing efficiency is higher.
[0168] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.
[0169] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Figure 3As shown, it is a schematic diagram of the structure of a computer device 300 provided in an embodiment of the present disclosure, including a processor 301, a memory 302, and a bus 303. Among them, the memory 302 is used to store execution instructions, including a memory 3021 and an external memory 3022; the memory 3021 here is also called an internal memory, which is used to temporarily store the operation data in the processor 301, and the data exchanged with the external memory 3022 such as a hard disk. The processor 301 exchanges data with the external memory 3022 through the memory 3021. When the computer device 300 is running, the processor 301 communicates with the memory 302 through the bus 303, so that the processor 301 executes the following instructions:
[0170] Get the image to be processed;
[0171] Receiving stylized processing information set for the image to be processed; wherein the stylized processing information includes stylized degree information corresponding to a plurality of image styles in the preset image style;
[0172] Based on the stylized processing information, the image processing models corresponding to the image styles are subjected to model fusion processing to obtain a target image processing model;
[0173] The image to be processed is processed based on the target image processing model to obtain a target style image that matches the stylization degree information.
[0174] In a possible implementation manner, in the instructions of the processor 301, for any image style, the method further includes adjusting the image processing model corresponding to the image style according to the following steps:
[0175] For any image style, obtain a pre-trained image processing model corresponding to the image style;
[0176] Fine-tune a target parameter module among the multiple parameter modules of the pre-trained image processing model.
[0177] In a possible implementation manner, in the instruction of the processor 301, the fine-tuning of the target parameter module among the multiple parameter modules of the pre-trained image processing model includes:
[0178] For any of the target parameter modules, based on the parameter matrix decomposition size matching the target parameter module, the parameter matrix of the target parameter module is decomposed to obtain a plurality of parameter matrices after decomposition;
[0179] Parameter adjustment is performed on the multiple parameter matrices after the decomposition process.
[0180] In a possible implementation, in the instructions of the processor 301, other parameter modules except a target parameter module in the image processing models corresponding to the respective image styles are the same, and the target parameter module is a module adjusted during the fine-tuning process of the image processing model;
[0181] Based on the stylized processing information, the image processing models corresponding to the respective image styles are subjected to model fusion processing to obtain a target image processing model, including:
[0182] Based on the stylization degree information corresponding to each image style, target parameter modules in the image processing models corresponding to each image style are fused to obtain a fused parameter module;
[0183] The target image processing model is determined based on other parameter modules except the target parameter module in any image processing model and the fusion parameter module.
[0184] In a possible implementation manner, in the instruction of the processor 301, the stylization degree information is represented by a stylization degree parameter;
[0185] Based on the stylized processing information, the image processing models corresponding to the respective image styles are subjected to model fusion processing to obtain a target image processing model, including:
[0186] Normalizing the stylization degree parameters corresponding to the multiple image styles respectively to obtain the normalized degree parameters corresponding to the multiple image styles respectively;
[0187] Based on the normalization degree parameters corresponding to the multiple image styles, the image processing models corresponding to the image styles are subjected to model fusion processing to obtain a target image processing model.
[0188] In a possible implementation manner, in the instruction of the processor 301, the receiving stylized processing information set for the image to be processed includes:
[0189] Extracting image features from the image to be processed to obtain image features corresponding to the image to be processed;
[0190] Determining stylized recommendation information corresponding to the image to be processed based on image features corresponding to the image to be processed, and displaying the stylized recommendation information;
[0191] In response to a selection operation on the stylized recommended information, the stylized recommended information is determined as stylized processing information set for the image to be processed.
[0192] In a possible implementation manner, in the instruction of the processor 301, determining the stylized recommendation information corresponding to the image to be processed based on the image features corresponding to the image to be processed includes:
[0193] Determine the feature similarity between the image features corresponding to the image to be processed and the standard image features corresponding to the preset standard image;
[0194] Based on the feature similarity and the standard stylization processing information corresponding to the preset standard image, stylization recommendation information corresponding to the image to be processed is determined.
[0195] The present disclosure also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps of the image processing method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.
[0196] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the image processing method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.
[0197] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).
[0198] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.
[0199] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0200] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0201] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0202] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. An image processing method, characterized in that: include: Get the image to be processed; Receiving stylized processing information set for the image to be processed; wherein the stylized processing information includes stylized degree information corresponding to a plurality of image styles in the preset image style; Based on the stylized processing information, the image processing models corresponding to the image styles are subjected to model fusion processing to obtain a target image processing model; The image to be processed is processed based on the target image processing model to obtain a target style image that matches the stylization degree information.
2. The method according to claim 1, characterized in that For any image style, the method further includes adjusting the image processing model corresponding to the image style according to the following steps: For any image style, obtain a pre-trained image processing model corresponding to the image style; Fine-tune a target parameter module among the multiple parameter modules of the pre-trained image processing model.
3. The method according to claim 2, characterized in that The fine-tuning of a target parameter module among the plurality of parameter modules of the pre-trained image processing model comprises: For any of the target parameter modules, based on the parameter matrix decomposition size matching the target parameter module, the parameter matrix of the target parameter module is decomposed to obtain a plurality of parameter matrices after decomposition; Parameter adjustment is performed on the multiple parameter matrices after the decomposition process.
4. The method according to claim 1, characterized in that: The image processing models corresponding to the respective image styles have the same parameter modules except for the target parameter module, where the target parameter module is a module adjusted during the fine-tuning process of the image processing model; Based on the stylized processing information, the image processing models corresponding to the respective image styles are subjected to model fusion processing to obtain a target image processing model, including: Based on the stylization degree information corresponding to each image style, target parameter modules in the image processing models corresponding to each image style are fused to obtain a fused parameter module; The target image processing model is determined based on other parameter modules except the target parameter module in any image processing model and the fusion parameter module.
5. The method according to claim 1, characterized in that The stylization degree information is represented by a stylization degree parameter; Based on the stylized processing information, the image processing models corresponding to the respective image styles are subjected to model fusion processing to obtain a target image processing model, including: Normalizing the stylization degree parameters corresponding to the multiple image styles respectively to obtain the normalized degree parameters corresponding to the multiple image styles respectively; Based on the normalization degree parameters corresponding to the multiple image styles, the image processing models corresponding to the image styles are subjected to model fusion processing to obtain a target image processing model.
6. The method according to claim 1, characterized in that The receiving stylized processing information set for the image to be processed includes: Extracting image features from the image to be processed to obtain image features corresponding to the image to be processed; Determining stylized recommendation information corresponding to the image to be processed based on image features corresponding to the image to be processed, and displaying the stylized recommendation information; In response to a selection operation on the stylized recommended information, the stylized recommended information is determined as stylized processing information set for the image to be processed.
7. The method according to claim 6, characterized in that The determining, based on the image features corresponding to the image to be processed, stylized recommendation information corresponding to the image to be processed includes: Determine the feature similarity between the image features corresponding to the image to be processed and the standard image features corresponding to the preset standard image; Based on the feature similarity and the standard stylization processing information corresponding to the preset standard image, stylization recommendation information corresponding to the image to be processed is determined.
8. An image processing device, characterized in that: include: An acquisition module, used for acquiring an image to be processed; A receiving module, configured to receive stylized processing information set for the image to be processed; wherein the stylized processing information includes stylized degree information corresponding to a plurality of image styles in the preset image style; A fusion module, configured to perform model fusion processing on the image processing models corresponding to the respective image styles based on the stylized processing information to obtain a target image processing model; A processing module is used to process the image to be processed based on the target image processing model to obtain a target style image that matches the stylization degree information.
9. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the image processing method as described in any one of claims 1 to 7 are performed.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are executed.