Image processing method and device, storage medium and electronic equipment

Abstract

The invention discloses an image processing method and device, a storage medium and electronic equipment. The method comprises the steps that an auxiliary processing area and a to-be-processed first image are displayed, the auxiliary processing area is used for obtaining at least one piece of auxiliary information referenced when the first image is processed, input of user-defined auxiliary information is allowed in the auxiliary processing area, and the at least one piece of auxiliary information comprises the user-defined auxiliary information; in response to a style processing request triggered for the first image, using the auxiliary processing area to obtain style auxiliary information, at least one piece of auxiliary information including the style auxiliary information, and the style auxiliary information being used for representing a target style to which the first image is expected to be processed; the first image is processed into a second image of the target style, and the image content similarity between the first image and the second image is larger than or equal to a preset threshold value. The technical problem of low image processing efficiency is solved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to an image processing method, device, storage medium and electronic device. Background Art

[0002] In image creation scenarios, there is often a need to transform the image style of an input image. However, the existing image style transformation methods can only provide a limited number of fixed style options, which results in the inability to complete the style transformation of the corresponding image when the target style required by the user is not included in the optional styles provided, resulting in a low efficiency of image style transformation.

[0003] Therefore, there is a technical problem in the related art that the efficiency of image style transfer is low. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, device, storage medium and electronic device to at least solve the technical problem of low image processing efficiency in the related art.

[0005] According to one aspect of an embodiment of the present application, an image processing method is provided, comprising: displaying an auxiliary processing area, and a first image to be processed, wherein the auxiliary processing area is used to obtain at least one auxiliary information for reference when processing the first image, and the auxiliary processing area is configured to allow input of custom auxiliary information, and the at least one auxiliary information includes the custom auxiliary information; in response to a style processing request triggered on the first image, obtaining style auxiliary information using the auxiliary processing area, wherein the at least one auxiliary information includes the style auxiliary information, and the style auxiliary information is used to indicate a target style into which the first image is expected to be processed; processing the first image into a second image of the target style, wherein the image content similarity between the first image and the second image is greater than or equal to a preset threshold.

[0006] According to another aspect of an embodiment of the present application, an image processing device is also provided, including: a display unit, used to display an auxiliary processing area, and a first image to be processed, wherein the auxiliary processing area is used to obtain at least one auxiliary information for reference when processing the first image, and the auxiliary processing area is configured to allow input of custom auxiliary information, and the at least one auxiliary information includes the custom auxiliary information; an acquisition unit, used to obtain style auxiliary information using the auxiliary processing area in response to a style processing request triggered on the first image, wherein the at least one auxiliary information includes the style auxiliary information, and the style auxiliary information is used to indicate a target style into which the first image is expected to be processed; a processing unit, used to process the first image into a second image of the target style, wherein the image content similarity between the first image and the second image is greater than or equal to a preset threshold.

[0007] As an optional scheme, the above-mentioned acquisition unit includes: a first acquisition module, which is used to acquire the above-mentioned style auxiliary information based on the at least one valid information when the above-mentioned custom auxiliary information in the above-mentioned auxiliary processing area is at least one valid information input; or, a second acquisition module, which is used to acquire the default auxiliary information corresponding to the above-mentioned first image when the above-mentioned custom auxiliary information in the above-mentioned auxiliary processing area is blank information, and determine the above-mentioned default auxiliary information as the above-mentioned style auxiliary information, wherein the style similarity between the style represented by the above-mentioned default auxiliary information and the original style of the above-mentioned first image is greater than or equal to a first threshold.

[0008] As an optional scheme, the above-mentioned first acquisition module includes: a first determination submodule, which is used to determine the above-mentioned one custom auxiliary information as the above-mentioned style auxiliary information when the above-mentioned at least one valid information includes one custom auxiliary information; or, a second determination submodule, which is used to determine each of the above-mentioned N custom auxiliary information as the above-mentioned style auxiliary information when the above-mentioned at least one valid information includes N custom auxiliary information, so as to obtain N above-mentioned style auxiliary information, wherein N is an integer greater than 1; or, a third determination submodule, which is used to perform similarity measurement on the styles represented by the above-mentioned M custom auxiliary information when the above-mentioned at least one valid information includes M custom auxiliary information, and classify the above-mentioned M custom auxiliary information as target auxiliary information using the measurement result of the above-mentioned similarity measurement, and determine the above-mentioned target auxiliary information as the above-mentioned style auxiliary information, wherein M is an integer greater than 1.

[0009] As an optional solution, the device further includes: a third acquisition module, which is used to acquire the first video to be processed before the default auxiliary information corresponding to the first image is acquired and the default auxiliary information is determined as the style auxiliary information; an extraction module, which is used to extract video key frames from the first video in response to a style processing request triggered by the first video before the default auxiliary information corresponding to the first image is acquired and the default auxiliary information is determined as the style auxiliary information, wherein the video key frames are video frames with key content among multiple first video frames contained in the first video, and the key content is used to represent the video content of the first video; a first determination module, which is used to determine the image corresponding to the video key frame as the first image before the default auxiliary information corresponding to the first image is acquired and the default auxiliary information is determined as the style auxiliary information; the device further includes: a first processing module, which is used to process all the multiple first video frames into second video frames of the target style after the default auxiliary information corresponding to the first image is acquired and the default auxiliary information is determined as the style auxiliary information, so as to obtain a second video of the target video, wherein the second video contains multiple second video frames.

[0010] As an optional solution, the above-mentioned device also includes: an execution module, which is used to perform the following steps before the above-mentioned first image is input as the input image to the style transfer model to obtain the above-mentioned second image output by the above-mentioned style transfer model based on the above-mentioned style auxiliary information, until a trained style transfer model is obtained: a fourth acquisition module, which is used to obtain a current image sample from the above-mentioned multiple image samples, and a current standard image corresponding to the current image sample when the current image sample belongs to the sample style, before the above-mentioned first image is input as the input image to the style transfer model to obtain the above-mentioned second image output by the above-mentioned style transfer model based on the above-mentioned style auxiliary information, wherein the above-mentioned sample style is different from the original style of the above-mentioned current image sample; a noise adding module, which is used to perform noise processing on the above-mentioned current image sample before the above-mentioned first image is input as the input image to the style transfer model to obtain the above-mentioned second image output by the above-mentioned style transfer model based on the above-mentioned style auxiliary information, and input the above-mentioned current image sample after noise adding into the current a style transfer model to obtain a sample image of the sample style output by the current style transfer model; a fifth acquisition module, used to obtain new image samples from the multiple image samples when the image content similarity between the sample image of the sample style and the current standard image is less than a training threshold before the first image is input as an input image into the style transfer model to obtain the second image output by the style transfer model based on the style auxiliary information, and use the new image samples as the current image samples to perform the next training on the current style transfer model; a second determination module, used to determine the current style transfer model as the trained style transfer model when the image content similarity between the sample image of the sample style and the current standard image is greater than or equal to the training threshold before the first image is input as an input image into the style transfer model to obtain the second image output by the style transfer model based on the style auxiliary information.

[0011] As an optional solution, the above-mentioned device also includes: a sixth acquisition module, which is used to obtain the above-mentioned multiple image samples before the above-mentioned first image is input as the input image to the style transfer model to obtain the above-mentioned second image output by the above-mentioned style transfer model based on the above-mentioned style auxiliary information; a clustering module, which is used to cluster the above-mentioned multiple image samples according to image style to obtain a P-type image sample set, wherein P is an integer greater than 1, the above-mentioned image sample set contains at least one of the above-mentioned image samples, and the image style similarity between image samples in the same above-mentioned image sample set is greater than or equal to a second threshold; a training module, which is used to select different types of image sample sets from the above-mentioned P-type image sample set in turn before the above-mentioned first image is input as the input image to the style transfer model to obtain the above-mentioned second image output by the above-mentioned style transfer model based on the above-mentioned style auxiliary information, and iteratively train the initial style transfer model until a trained style transfer model is obtained.

[0012] As an optional solution, the acquisition unit includes: a third determination module, which is used to input the above-mentioned valid information as input information into the style recognition model when the above-mentioned custom auxiliary information in the above-mentioned auxiliary processing area is valid input information, so as to obtain the target style information output by the above-mentioned style recognition model, wherein the above-mentioned style recognition model is a neural network model trained using multiple information samples and used to identify the style to which the above-mentioned input information belongs; a seventh acquisition module, which is used to obtain the above-mentioned style auxiliary information based on the above-mentioned target style information; or, a fourth determination module, which is used to input the above-mentioned first image as input information into the style recognition model when the above-mentioned custom auxiliary information in the above-mentioned auxiliary processing area is blank information, so as to obtain the first style information output by the above-mentioned style recognition model, wherein the above-mentioned first style information is used to represent the style to which the above-mentioned first image belongs; an eighth acquisition module, which is used to obtain the second style information associated with the above-mentioned first style information, wherein the style similarity between the style represented by the above-mentioned second style information and the style to which the above-mentioned first image belongs is greater than or equal to a third threshold; and a ninth acquisition module, which is used to obtain the above-mentioned style auxiliary information based on the above-mentioned second style information.

[0013] As an optional scheme, the above-mentioned seventh acquisition module includes: a first optimization sub-module, which is used to optimize the above-mentioned target style information in combination with the media comment information associated with the above-mentioned valid information when the media comment information associated with the above-mentioned valid information is acquired, so as to obtain the above-mentioned style auxiliary information; the above-mentioned ninth acquisition module includes: a second optimization sub-module, which is used to optimize the above-mentioned target style information in combination with the media comment information associated with the above-mentioned second style information when the media comment information associated with the above-mentioned second style information is acquired, so as to obtain the above-mentioned style auxiliary information.

[0014] As an optional solution, the above-mentioned device also includes: a display module, which is used to determine the preset number of second images ranked before the above-mentioned style matching degree from the second images whose number is greater than the above-mentioned preset display number according to the style matching degree between the above-mentioned second images and the above-mentioned target style after the above-mentioned first image is processed into the above-mentioned second image of the target style, when the number of images of the above-mentioned second images is greater than the above-mentioned preset display number, and display them, wherein the above-mentioned preset display number is a pre-set display number corresponding to the image after the above-mentioned first image is expected to be processed into the above-mentioned target style.

[0015] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above image processing method.

[0016] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the image processing method through the computer program.

[0017] In an embodiment of the present application, an auxiliary processing area and a first image to be processed are displayed, wherein the auxiliary processing area is used to obtain at least one auxiliary information for reference when processing the first image, and the auxiliary processing area is set to allow input of custom auxiliary information, and the at least one auxiliary information includes the custom auxiliary information; in response to a style processing request triggered on the first image, the style auxiliary information is obtained using the auxiliary processing area, wherein the at least one auxiliary information includes the style auxiliary information, and the style auxiliary information is used to indicate the target style into which the first image is expected to be processed; the first image is processed into a second image of the target style, wherein the image content similarity between the first image and the second image is greater than or equal to a preset threshold. By setting an auxiliary processing area where custom auxiliary information is allowed to be input, the style of the first image to be processed can be transformed according to the target style information input by the user, and the second image of the target style can be obtained, and the image content remains consistent during the transformation process. In this way, the user can input corresponding, custom auxiliary information in the auxiliary processing area according to the image creation needs, thereby achieving the purpose of quickly and efficiently realizing the style transformation of the original image, thereby achieving the technical effect of improving the efficiency of image style transformation, and solving the technical problem of low efficiency of image style transformation in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a schematic diagram of an application environment of an optional image processing method according to an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of a process of an optional image processing method according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of another optional image processing method according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of another optional image processing method according to an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of another optional image processing method according to an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of another optional image processing method according to an embodiment of the present application;

[0025] Figure 7 is a schematic diagram of another optional image processing method according to an embodiment of the present application;

[0026] Figure 8 is a schematic diagram of another optional image processing method according to an embodiment of the present application;

[0027] Fig. 9 is a schematic diagram of another optional image processing method according to an embodiment of the present application;

[0028] Fig.10 is a schematic diagram of another optional image processing method according to an embodiment of the present application;

[0029] Fig.11 is a schematic diagram of another optional image processing method according to an embodiment of the present application;

[0030] Fig.12 is a schematic diagram of another optional image processing method according to an embodiment of the present application;

[0031] Fig.13 is a schematic diagram of another optional image processing method according to an embodiment of the present application;

[0032] Fig.14 is a schematic diagram of another optional image processing method according to an embodiment of the present application;

[0033] Fig.15 is a schematic diagram of another optional image processing method according to an embodiment of the present application;

[0034] Fig.16 is a schematic diagram of another optional image processing method according to an embodiment of the present application;

[0035] Fig.17 is a schematic diagram of another optional image processing method according to an embodiment of the present application;

[0036] Fig.18 is a schematic diagram of another optional image processing device according to an embodiment of the present application;

[0037] Fig.19 A schematic structural diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0039] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0040] According to one aspect of the embodiments of the present application, an image processing method is provided. Optionally, as an optional implementation, the image processing method can be but is not limited to being applied to Figure 1 In the environment shown, the environment may include, but is not limited to, a client 102 and a server 112 . The client 102 may include, but is not limited to, a display 104 , a processor 106 , and a memory 108 . The server 112 includes a database 114 and a processing engine 116 .

[0041] The specific process can be as follows:

[0042] Step S102, after displaying the first image 1001 to be processed and the auxiliary processing area 1002, the client 102 obtains a style processing request triggered on the first image 1001;

[0043] Steps S104-S106, the client 102 sends a style processing request to the server 112;

[0044] Step S108, the server 112 responds to the style processing request and obtains style auxiliary information through the processing engine 116 to determine the target style to be processed;

[0045] Step S110, processing the first image into a second image of the target style according to the target style;

[0046] Steps S112-S114, sending the second image to the client 102 via the network 110;

[0047] In step S114, the client 102 performs processing operations such as data reception through the processor 106, and displays the second image 1003 (as well as the first image 1001 and the auxiliary processing area 1002) through the display 104, and stores the processing and related processing data generated during the actual process in the memory 108.

[0048] remove Figure 1 In addition to the examples shown, the above steps can be completed independently by the client or the server, or by the client and the server working together, such as the client 102 executing the above steps S108, thereby reducing the processing pressure of the server 112. The client 102 includes but is not limited to smart phones, laptops, tablet computers, desktop computers, smart TVs, car-mounted devices, etc., and the present application does not limit the specific implementation of the client 102. The server 112 can be a single server or a server cluster consisting of multiple servers, or a cloud server.

[0049] Optionally, as an optional implementation, as Figure 2 As shown, the image processing method can be executed by an electronic device, such as Figure 1 The client or server shown in the figure includes:

[0050] S202, displaying an auxiliary processing area and a first image to be processed, wherein the auxiliary processing area is used to obtain at least one auxiliary information for reference when processing the first image, and the auxiliary processing area is configured to allow input of custom auxiliary information, and the at least one auxiliary information includes custom auxiliary information;

[0051] S204, in response to a style processing request triggered on the first image, obtaining style auxiliary information using the auxiliary processing area, wherein the at least one auxiliary information includes style auxiliary information, and the style auxiliary information is used to indicate a target style into which the first image is expected to be processed;

[0052] S206: Process the first image into a second image of a target style, wherein the image content similarity between the first image and the second image is greater than or equal to a preset threshold.

[0053] Optionally, in this embodiment, the above-mentioned image processing method can be applied to, but is not limited to, image generation and creation scenarios. In this scenario, a widespread demand is to perform image style transformation on an input image. For example, a user wants to transform a current image into another image of a desired style, wherein the content of the image before and after the transformation does not substantially change.

[0054] However, the image style transfer method in the prior art can often only provide a limited number of fixed style options, which results in the inability to complete the style transfer of the corresponding image when the target style required by the user does not exist in the provided optional styles, thereby resulting in the problem of low efficiency of image style transfer.

[0055] For the above problem, the image processing method provided by this embodiment is used to set an auxiliary processing area that allows the input of custom auxiliary information, so that the style of the first image to be processed can be transformed according to the target style information input by the user, and the second image of the target style can be obtained, and the image content remains consistent during the transformation process. In this way, the user can input corresponding, custom auxiliary information in the auxiliary processing area according to the image creation needs, thereby achieving the purpose of quickly and efficiently realizing the style transformation of the original image, thereby achieving the technical effect of improving the efficiency of image style transformation, so as to solve the problem of low efficiency of the above-mentioned image style transformation.

[0056] Optionally, in this embodiment, the first image to be processed may be, but is not limited to, an original image input by a user for style transformation, may be, but is not limited to, an image captured by a capturing device, and may also be, but is not limited to, an image corresponding to a video frame extracted from a video by an extraction device.

[0057] Optionally, in this embodiment, the auxiliary processing area may be, but is not limited to, at least one auxiliary information that is pre-set for obtaining reference when processing the first image, wherein the at least one auxiliary information may be, but is not limited to, including custom auxiliary information input by a user, and may also be, but is not limited to, including blank information.

[0058] It can be understood that, when the user does not input any information in the auxiliary processing area, it is determined that at least one auxiliary information includes blank information, that is, the blank information is used to indicate that the user has not input any custom auxiliary information.

[0059] Optionally, in this embodiment, the above-mentioned customized auxiliary information may include, but is not limited to, text information, image information, and other types of information that can indicate the target style.

[0060] Optionally, in this embodiment, after determining the original first image of the first style to be processed, in response to the style processing request triggered on the first image, style auxiliary information is obtained using the auxiliary processing area, and based on the style auxiliary information, a target style representing the desired processing style of the first image is determined, wherein the target style can be but is not limited to a second style that is different from the first style.

[0061] For example, when at least one auxiliary information includes custom input text information, the above text information is used for retrieval and matching to obtain the target style corresponding to the text information, and the first image is processed into a second image of the target style; and, when at least one auxiliary information includes custom input image information, the above image information is used for recognition and matching to obtain the target style corresponding to the image information, and the first image is processed into a second image of the target style.

[0062] As a further example, when at least one auxiliary information does not include the input custom auxiliary information, style matching is performed on the first style of the first image to obtain a matched target style, and the first image is processed into a second image of the target style.

[0063] It should be noted that the above style matching may include, but is not limited to, determining a style that is relatively similar to the first style, and may also include, but is not limited to, determining a style that is completely dissimilar to the first style.

[0064] To further illustrate, a schematic diagram of an optional image processing method is shown as follows: Figure 3 As shown, the (client) display interface 302 includes a first display area 302 and a second display area 304, wherein the first display area 302 is used to display the first image to be processed, and the second display area 304 is an auxiliary processing area, which is used to obtain input custom auxiliary information or determine blank information when there is no input, and then used to determine style auxiliary information and target style.

[0065] It should be noted that if Figure 3 As shown, the user can, but is not limited to, upload and display the first image to be processed in the first display area 304 in the display interface 302, and input customized auxiliary information in the second display area 306, and click the OK button in the display interface 304 to trigger the style processing request.

[0066] To further illustrate, a schematic diagram of an optional image processing method is shown as follows: Figure 4 As shown in (a) of FIG. 4 , the user inputs a first image 406 to be processed in a first display area 404 in a display interface 402, and inputs a custom auxiliary information reference image 410 in a second display area 408 in the display interface 402, and clicks the OK button to trigger a style processing request for the first image 406, wherein the first display area 404 is used to display the image to be processed, the second display area 408 is an auxiliary processing area, the first image 406 is a first style (indicated by diagonal stripes filling), and the reference image 410 is a second style (indicated by straight stripes filling). After the above style processing request is responded to and processed, as shown in FIG. Figure 4As shown in (b) of FIG. 4 , a second image 416 is displayed on a third display area 414 on the display interface 412, wherein the third display area 414 is used to display the image after the style transformation, the second image 416 is in the second style (illustrated by straight line stripe filling), and the substantial content is unchanged compared with the first image 406. The user can, but is not limited to, click the return button on the display interface 412 to jump to the display interface 402 to adjust the relevant information of the image style transformation.

[0067] Through the embodiment provided by the present application, an auxiliary processing area and a first image to be processed are displayed, wherein the auxiliary processing area is used to obtain at least one auxiliary information for reference when processing the first image, and the auxiliary processing area is set to allow input of custom auxiliary information, and the at least one auxiliary information includes the custom auxiliary information; in response to a style processing request triggered on the first image, the style auxiliary information is obtained using the auxiliary processing area, wherein the at least one auxiliary information includes the style auxiliary information, and the style auxiliary information is used to indicate the target style into which the first image is expected to be processed; the first image is processed into a second image of the target style, wherein the image content similarity between the first image and the second image is greater than or equal to a preset threshold. By setting an auxiliary processing area where custom auxiliary information is allowed to be input, the style of the first image to be processed can be transformed according to the target style information input by the user, and the second image of the target style can be obtained, and the image content remains consistent during the transformation process. In this way, the user can input corresponding, custom auxiliary information in the auxiliary processing area according to the image creation needs, thereby achieving the purpose of quickly and efficiently realizing the style transformation of the original image, thereby achieving the technical effect of improving the efficiency of image style transformation.

[0068] As an optional solution, the auxiliary processing area is used to obtain style auxiliary information, including:

[0069] S1, when the customized auxiliary information in the auxiliary processing area is at least one valid information input, acquiring style auxiliary information based on the at least one valid information; or,

[0070] S2, when the custom auxiliary information in the auxiliary processing area is blank information, obtaining default auxiliary information corresponding to the first image, and determining the default auxiliary information as style auxiliary information, wherein the style similarity between the style represented by the default auxiliary information and the original style of the first image is greater than or equal to a first threshold.

[0071] Optionally, in this embodiment, the at least one valid information may be, but is not limited to, at least one of the following: at least one valid text information, at least one valid image information.

[0072] It should be noted that, when the customized auxiliary information in the auxiliary processing area is at least one valid information input, the style auxiliary information is acquired based on the at least one valid information.

[0073] Optionally, in this embodiment, when no information is input in the auxiliary processing area or the input information is invalid information, it is determined that the customized auxiliary information in the auxiliary processing area is blank information.

[0074] It should be noted that, when the custom auxiliary information in the auxiliary processing area is blank information, the default auxiliary information corresponding to the first image is obtained, and the default auxiliary information is determined as style auxiliary information, wherein the style similarity between the style represented by the default auxiliary information and the original style of the first image is greater than or equal to a first threshold.

[0075] To further illustrate, when the first input image is a photo of a certain clothing and the custom copy information is blank information, the system searches and matches according to the default original style of the first image, and obtains other styles that are different from the original style and have a similarity greater than or equal to a first threshold, and then determines the default auxiliary information and style auxiliary information based on the other styles, and generates images of other styles suitable for the clothing according to the style auxiliary information as a reference effect display diagram, so as to find a more suitable style visual effect for the clothing.

[0076] It should be noted that the first threshold can be set according to actual needs but is not limited to it. This embodiment does not impose any additional restrictions on this.

[0077] Through the embodiments provided by the present application, when the customized auxiliary information in the auxiliary processing area is at least one valid information input, style auxiliary information is obtained based on the at least one valid information; or, when the customized auxiliary information in the auxiliary processing area is blank information, default auxiliary information corresponding to the first image is obtained, and the default auxiliary information is determined as the style auxiliary information, wherein the style similarity between the style represented by the default auxiliary information and the original style of the first image is greater than or equal to a first threshold.

[0078] As an optional solution, obtaining style auxiliary information based on at least one valid information includes:

[0079] S1, when at least one valid information includes a piece of custom auxiliary information, determining the piece of custom auxiliary information as style auxiliary information; or,

[0080] S2, when at least one valid information includes N pieces of custom auxiliary information, determining each piece of the N pieces of custom auxiliary information as style auxiliary information to obtain N pieces of style auxiliary information, wherein N is an integer greater than 1; or

[0081] S3, when at least one valid information includes M pieces of custom auxiliary information, perform similarity measurement on the styles represented by the M pieces of custom auxiliary information, and classify the M pieces of custom auxiliary information as target auxiliary information using the measurement result of the similarity measurement, and determine the target auxiliary information as style auxiliary information, where M is an integer greater than 1.

[0082] Optionally, in this embodiment, when the at least one valid information includes only one piece of customized auxiliary information, the customized auxiliary information is determined as style auxiliary information.

[0083] To further illustrate, a schematic diagram of an optional image processing method is shown as follows: Figure 5 As shown, in the input page 502, the user inputs the image 506 to be processed, and inputs the image 508 as a reference image (the expected style information is not input). After the input is confirmed and the system's generation model determines a custom auxiliary information (as well as style auxiliary information, and target style) based on the image 508, the image 510 (processed based on the target style) that is consistent with the style of the image 508 is displayed on the image display page 504.

[0084] It should be noted that in Figure 5 In the input page 502 shown, if the user inputs the image 506 to be processed and inputs valid text information of the expected style but does not input a reference image, the valid text information is determined as the above-mentioned custom auxiliary information and the above-mentioned style auxiliary information.

[0085] Optionally, in this embodiment, when at least one valid information includes N custom auxiliary information, each of the N custom auxiliary information is determined as style auxiliary information to obtain N style auxiliary information, and corresponding N target styles are determined based on the N auxiliary information, respectively corresponding to the N second images after the style transformation.

[0086] To further illustrate, a schematic diagram of an optional image processing method is shown as follows: Figure 6As shown, in the input page 602, the user inputs a to-be-processed image 606, as well as two valid text information "cartoon style" and "urban style", and an image 608 as a reference image. After the input is determined and the system's generation model determines a custom auxiliary information corresponding to the valid text information "cartoon style" and another custom auxiliary information corresponding to the valid text information "urban style" based on the two valid text information "cartoon style" and "urban style" and an image 608, and another custom auxiliary information corresponding to the valid information image 608 (as well as the respective corresponding style auxiliary information and target style), an image 610 having the same style as the first valid text information (i.e., "cartoon style"), an image 612 having the same style as the second valid text information (i.e., "urban style"), and an image 614 having the same style as the image 608 as a reference image are displayed on the image display page 604.

[0087] Optionally, in this embodiment, when at least one valid information includes M custom auxiliary information, similarity measurement is performed based on the M custom auxiliary information to obtain a classified target auxiliary information, and the classified target auxiliary information is determined as style auxiliary information.

[0088] To further illustrate, a schematic diagram of an optional image processing method is shown as follows: Figure 7 As shown, in the input page 702, the user inputs the image 706 to be processed, the valid text information "cartoon style", and the image 708 as the reference image. After the input is determined and the system's generation model processes the valid text information "cartoon style" and the image 708 to determine a target custom auxiliary information (and the corresponding style auxiliary information and target style), the processed target style image 710 is displayed on the image display page 704. Among them, the above target custom auxiliary information is not classified and determined based on the valid text information "cartoon style" and the image 708 after similarity measurement.

[0089] It should be noted that the result of the above-mentioned similarity measurement classification can be but is not limited to being a certain customized auxiliary information among M customized auxiliary information (randomly or according to the preset highest weight), and can also be but is not limited to being a customized auxiliary information determined after the fusion of M customized auxiliary information. The above-mentioned fusion can be but is not limited to intersection processing.

[0090] Through the embodiments provided by the present application, when at least one valid information includes one custom auxiliary information, the one custom auxiliary information is determined as style auxiliary information; or, when at least one valid information includes N custom auxiliary information, each of the N custom auxiliary information is determined as style auxiliary information to obtain N style auxiliary information, where N is an integer greater than 1; when at least one valid information includes M custom auxiliary information, a similarity measurement is performed on the styles represented by the M custom auxiliary information, and the M custom auxiliary information are classified as target auxiliary information using the measurement result of the similarity measurement, and the target auxiliary information is determined as style auxiliary information, where M is an integer greater than 1.

[0091] As an optional solution, before obtaining the default auxiliary information corresponding to the first image and determining the default auxiliary information as the style auxiliary information, the method further includes:

[0092] S1, obtaining a first video to be processed;

[0093] S2, in response to a style processing request triggered on the first video, extracting a video key frame from the first video, wherein the video key frame is a video frame having key content among a plurality of first video frames contained in the first video, and the key content is used to represent the video content of the first video;

[0094] S3, determining the image corresponding to the video key frame as the first image;

[0095] After obtaining the default auxiliary information corresponding to the first image and determining the default auxiliary information as the style auxiliary information, the method further includes:

[0096] S4, processing the multiple first video frames into second video frames of a target style to obtain a second video of the target video, wherein the second video includes multiple second video frames.

[0097] Optionally, in this embodiment, the first image may be, but is not limited to, an image corresponding to a video key frame of the first video to be processed, wherein the video key frame is a video frame having key content among multiple first video frames contained in the first video, and the key content is used to represent the video content of the first video.

[0098] Optionally, in this embodiment, the above-mentioned image processing method is applied to the scenario of video style transition, a video key frame is extracted from the video, and the first image corresponding to the video key frame is obtained. The above-mentioned image processing method is used to obtain the corresponding default auxiliary information for the first image when the custom auxiliary information is blank information, thereby achieving the purpose of obtaining the most suitable style effect and its target style for the video key frame, and then using the style effect / target style word to process each video frame of the video where the video key frame is located into the style effect, thereby achieving the technical effect of video stylization.

[0099] Through the embodiments provided by the present application, before obtaining the default auxiliary information corresponding to the first image and determining the default auxiliary information as the style auxiliary information, the method also includes: obtaining a first video to be processed; in response to a style processing request triggered on the first video, extracting video key frames from the first video, wherein the video key frames are video frames with key content among multiple first video frames contained in the first video, and the key content is used to represent the video content of the first video; determining the image corresponding to the video key frames as the first image; after obtaining the default auxiliary information corresponding to the first image and determining the default auxiliary information as the style auxiliary information, the method also includes: processing multiple first video frames into second video frames of a target style to obtain a second video of the target video, wherein the second video contains multiple second video frames.

[0100] As an optional solution, processing the first image into a second image of a target style includes:

[0101] S1, inputting the first image as an input image into a style transfer model to obtain a second image output by the style transfer model based on style auxiliary information, wherein the style transfer model is a neural network model trained using multiple image samples and used to convert the original style of the input image into a desired style.

[0102] Optionally, in this embodiment, a pre-trained style transfer model is used to convert the input first image into a second image of a target style based on style auxiliary information, wherein the style transfer model may be, but is not limited to, a neural network model trained using multiple image samples and used to convert the original style of the input image into a desired style, and may be, but is not limited to, a model trained based on the instructpix2pix model.

[0103] As an optional solution, before inputting the first image as an input image into the style transfer model to obtain a second image output by the style transfer model based on the style auxiliary information, the method further includes:

[0104] S1, perform the following steps until a trained style transfer model is obtained:

[0105] S2, obtaining a current image sample from a plurality of image samples, and a current standard image corresponding to the current image sample when the current image sample belongs to a sample style, wherein the sample style is different from an original style of the current image sample;

[0106] S3, performing noise processing on the current image sample, and inputting the noised current image sample into the current style transfer model to obtain a sample image of the sample style output by the current style transfer model;

[0107] S4, when the image content similarity between the sample image of the sample style and the current standard image is less than the training threshold, a new image sample is obtained from the multiple image samples, and the new image sample is used as the current image sample to perform the next training on the current style transfer model;

[0108] S5, when the image content similarity between the sample image of the sample style and the current standard image is greater than or equal to the training threshold, determining the current style transfer model as the trained style transfer model.

[0109] Optionally, in this embodiment, the style transfer model may be, but is not limited to, a neural network model trained using multiple image samples and used to convert the original style of an input image into a desired style, and may be, but is not limited to, a model trained based on the instructpix2pix model.

[0110] Optionally, in this embodiment, in the process of using multiple image samples to train the style transfer model, the current image sample among the multiple image samples is used for noise processing to generate a noise map, the current image sample and the noise-added effect map are first concatenated and input into the neural network, and a first result is generated after latent space representation, and the first result and style representation information are further combined for calculation and processing to obtain a sample image of the style of the current style transfer model output sample.

[0111] Optionally, in this embodiment, a current standard image of the sample style corresponding to the current image sample is obtained. When the content similarity between the current standard image and the sample image output by the current style transfer model is low, it is determined that the training of the current image sample is unqualified, and a new image sample is further obtained from multiple image samples for the next training.

[0112] It should be noted that, when the image content similarity between the sample image of the sample style and the current standard image is greater than or equal to the training threshold, the current style transfer model is determined as the trained style transfer model.

[0113] As an optional solution, before inputting the first image as an input image into the style transfer model to obtain a second image output by the style transfer model based on the style auxiliary information, the method further includes:

[0114] S1, obtain multiple image samples;

[0115] S2, clustering the multiple image samples according to the image style to obtain a P-type image sample set, where P is an integer greater than 1, the image sample set contains at least one image sample, and the image style similarity between the image samples in the same image sample set is greater than or equal to a second threshold;

[0116] S3, selecting image sample sets of different categories from the P category image sample set in turn, and iteratively training the initial style transfer model until a trained style transfer model is obtained.

[0117] Optionally, in this embodiment, the above-mentioned clustering processing can be but is not limited to being used to indicate clustering of multiple image samples of initial various image styles, wherein, according to a preset number of cluster centers, a preset number of clusters and a preset number of cluster centers are obtained, and according to the image style of each image sample, the image samples are clustered to the cluster center closest to them, to obtain a preset number of image sample sets, each image sample set has a different image style, and the preset number can be but is not limited to P, where P is a positive integer.

[0118] It should be noted that, image sample sets of different categories are selected from the P category image sample set in turn, and the initial style transfer model is iteratively trained until a trained style transfer model is obtained.

[0119] As an optional solution, the auxiliary processing area is used to obtain style auxiliary information, including:

[0120] S1, when the customized auxiliary information in the auxiliary processing area is valid input information, input the valid information as input information into the style recognition model to obtain target style information output by the style recognition model, wherein the style recognition model is a neural network model trained using multiple information samples and used to identify the style of the input information;

[0121] S2, obtaining style auxiliary information based on the target style information; or,

[0122] S3, when the customized auxiliary information in the auxiliary processing area is blank information, inputting the first image as input information into the style recognition model to obtain first style information output by the style recognition model, wherein the first style information is used to indicate the style to which the first image belongs;

[0123] S4, obtaining second style information associated with the first style information, wherein a style similarity between the style represented by the second style information and the style to which the first image belongs is greater than or equal to a third threshold;

[0124] S5: Acquire style auxiliary information based on the second style information.

[0125] Optionally, in this embodiment, the style transfer model can be, but is not limited to, used to identify the custom auxiliary information of the auxiliary processing area to obtain the target style information. It can be, but is not limited to, directly identifying and processing the input valid information to obtain the target style information when the custom auxiliary information is valid information input, wherein the target style information is used to obtain the style auxiliary information, and the valid information can be, but is not limited to, valid text information and valid image information. It can also be, but is not limited to, first identifying the input original first image to obtain the first style information to which the first image belongs when the custom auxiliary information is blank information, and then obtaining the second style information that is different from the first style information but relatively similar (the style similarity is greater than or equal to the third threshold), wherein the second style information is used to obtain the style auxiliary information.

[0126] It can be understood that in this embodiment, when the customized auxiliary information in the auxiliary processing area is valid text information input, the above-mentioned style recognition model is used to determine the target style words that match the valid text information from a pre-constructed style word representation vocabulary, wherein the style word representation vocabulary stores a plurality of mapping relationships between text information and target style words, and the target style words are used to indicate the above-mentioned target style information.

[0127] Optionally, in this embodiment, when the custom auxiliary information in the auxiliary processing area is valid text information input, the above-mentioned style recognition model is used to determine target style words that match the recognized image semantic content of the valid image information from a pre-constructed image semantic representation vocabulary, wherein the image semantic representation vocabulary stores a plurality of mapping relationships between image semantic contents and target style words, and the target style words are used to indicate the above-mentioned target style information.

[0128] Optionally, in this embodiment, when the custom auxiliary information in the auxiliary processing area is blank information, the above-mentioned style recognition model is used to determine a second style word that matches the first style word corresponding to the first style information of the first image from a pre-constructed style image library, wherein the style image library stores a mapping relationship between multiple style information that are associated (style similarity is greater than or equal to a third threshold), and the second style word is used to indicate the above-mentioned second style information.

[0129] As an optional solution, style auxiliary information is obtained based on the target style information, including:

[0130] S1, when the media comment information associated with the effective information is obtained, the target style information is optimized in combination with the media comment information associated with the effective information to obtain style auxiliary information;

[0131] Acquiring style auxiliary information based on the second style information includes:

[0132] S2: When the media comment information associated with the second style information is obtained, the target style information is optimized in combination with the media comment information associated with the second style information to obtain style auxiliary information.

[0133] Optionally, in this embodiment, the media comment information can be, but is not limited to, manually supplemented text information. Since different users have different understandings of the concept of style, for example, when the image of a certain animated film or television work is used as effective image information, if it is simply identified as an animated style, the style of the transformed image will not be accurate enough, because the animated style also includes 2D style, 3D style and many subdivided styles, so it is necessary to combine the manually supplemented media comment information. To further illustrate, if the above-mentioned animated film or television work is pre-associated with the media comment information "2D combat style", the target style information is optimized based on the media comment information, and the optimized style auxiliary information is "2D combat animation style" instead of the original "animation style".

[0134] It should be noted that if the valid information obtained is the valid text information of the name of the above-mentioned animation film and television work, the same optimization method for the above-mentioned media review information is applicable and will not be repeated here.

[0135] As an optional solution, after processing the first image into a second image of a target style, the method further includes:

[0136] S1, when the number of second images is greater than the preset display number, according to the style matching degree between the second images and the target style, determine the second images with a preset display number ranked before the style matching degree from the second images whose number is greater than the preset display number, and display them, wherein the preset display number is the preset display number corresponding to the image after the first image is expected to be processed into the target style.

[0137] Optionally, in this embodiment, when the original first image and custom auxiliary information are obtained, style conversion processing is performed on the first image based on the target style indicated by the above custom auxiliary information to obtain multiple second images after style conversion, and the second images are sorted according to the style matching degree between the second images and the target style, and a preset number of second images are determined for display.

[0138] As an optional solution, the above image processing method is applied to a style transfer application scenario for user-defined input. In this scenario, image generation based on diffusion models has become the mainstream method of image creation, and a widespread demand in image creation is to perform image style transfer on the input image.

[0139] It should be noted that in this scenario, existing image style transfer methods can only provide limited style options, such as pre-arrangement through labels and other forms for selection, do not support custom image styles, and cannot provide the best style configuration, which makes the user experience very inconvenient. On the other hand, existing image style transfer models often use a small number of style images to fine-tune specific style words. This method is prone to overfitting the style of the head of the image and underfitting the style of the tail of the image, that is, the model loses its original ability to generate and create various objects, or the style generation effect is not stable enough, resulting in style overfitting problems.

[0140] In response to the above problems, this embodiment, based on the above image processing method, faces the diverse and open style conversion needs of users and provides a style conversion capability based on large-scale film and television drama (or other fields, such as clothing or cultural relics exhibitions, etc., which will be explained below using the film and television drama as an example) style characterization and generation model training to meet the image conversion needs of the above users with or without clear styles. This embodiment first constructs a style characterization model for film and television dramas and a large-scale available style characterization method; at the same time, it fine-tunes the text-to-image generation model for the image and text data set that is improved by enriching the content description for the missing style, and then generates the final model through fine-tuning the style conversion model; in the application, a method for optimal style generation based on homomodal or cross-modal retrieval is designed for indefinite styles. When training the style generation model, this embodiment redesigns the style data and model training method to reduce the problem that the head and tail style generation effects of the model are not stable enough under a large amount of style training.

[0141] Specifically, in this embodiment, a schematic diagram of an optional image processing method in the "guessing mode" is shown as follows: Figure 8 As shown, in the input page 802, the user inputs the image 806 to be processed, but does not enter the text information corresponding to the expected style column or the image information corresponding to the reference image column. Then, based on the original style of the input image 806, other optimal styles that best suit the image 806 are found and style conversion is performed, and the image 810 after style conversion is displayed on the image display page 804.

[0142] In this embodiment, another optional image processing method in the "text style constraint mode" is shown as follows: Fig. 9As shown, in the input page 902, the user inputs the image 906 to be processed, and inputs the text information "cartoon style" corresponding to the expected style column, and does not input the image information corresponding to the reference image column, then the target style is determined according to the input text information "cartoon style", and the image 906 is converted according to the target style, and the image 908 and the image 910 of the target style displayed on the image display page 904 are obtained. It should be noted that the number of output effect images displayed on the above-mentioned image display page 904 is not limited to 2, and can also be but not limited to other numbers that do not exceed the number limit.

[0143] In this embodiment, a schematic diagram of another optional image processing method in the "image reference style constraint mode" is shown as follows: Fig.10 As shown, in the input page 1002, the user inputs the image 1006 to be processed in the input image column, and inputs the image 1008 as the reference image in the reference image column, but does not input text information in the expected style column, then the target style (i.e., the style of the image 1008) is determined according to the input image 1008, and the style conversion is performed on the image 1006 to be processed according to the target style, and the target style image 1010 and the target style image 1012 displayed on the image display page 1004 are obtained. It should be noted that the number of output effect images displayed on the image display page 1004 is not limited to 2, and can also be but not limited to other numbers that do not exceed the number limit.

[0144] It is understandable that in the above-mentioned image processing methods in various modes, user-defined input is supported, so various possible input situations are processed and the complete image style conversion process is completed. For example, when the user simultaneously inputs the text information of the expected style column and the image information of the reference image column, it is possible but not limited to displaying half of the converted images generated by the style words based on the text information and half of the converted images generated by the style words based on the image information on the image display page; it is also possible but not limited to obtaining a target style word after processing based on the text information and the image information, and then displaying all the converted images generated based on the target style word.

[0145] To further illustrate, a schematic diagram of an optional image processing method covering the above three modes is shown as follows: Fig.11As shown, it includes three optional branches S1, S2 and S3. Specifically, when branch S1 is triggered, the user style word is obtained according to the text information input by the user, and the target style word matching the user style word is retrieved and determined from the word-style library, and the style transformation of the original image is performed according to the image style of the target style word; when branch S2 is triggered, the style graph is obtained according to the image information input by the user, and the style representation of the style graph is identified from the graph-style library, and then the target style word matching it is obtained based on the style representation, and the style transformation of the original image is performed according to the image style of the target style word; when branch S3 is triggered, the semantic representation of the original image is obtained, and the target style word matching the semantic representation is retrieved from the semantic-style library, and the style transformation of the original image is performed according to the image style of the target style word.

[0146] It should be noted that when there are style words, the input style word CLIP text representation emb_text is extracted, the style word closest to emb_x in the style vocabulary (or the closest top3 style words) is queried, and the matching target style word is output. When there is a style map, the style model representation emb_img of the input style map is extracted, emb_img is queried with the closest style map representation (top10 style map) in the style map library, and the style name corresponding to these style map representations with the most occurrences is taken as the target style word. When there are no style words and style images, the guessing mode is executed, that is, the CLIP semantic image representation emb_img2 is extracted from the input image, emb_img2 is queried with the closest image representation (top10 semantic map) in the semantic style vocabulary, and the style name corresponding to these semantic map representations with the most occurrences is taken as the target style word.

[0147] It should be noted that before processing the three different style input methods, a corresponding search database needs to be prepared, wherein the search database includes the above-mentioned word-style library, image-style library and semantic-style library.

[0148] Optionally, in this embodiment, a style word representation vocabulary (i.e., a word-style library) is constructed by using film and television drama style words and manual annotation expansion methods. Since the style words input by the user are an open set, that is, the system does not know in advance what the user has input, there are differences between the words input by the user and the words used to identify the style in training. For example, for a certain animation, the user may input the Chinese 3D animation style or the animation style of the animation, and the system needs to correctly map both inputs. This embodiment eliminates this difference through vocabulary expansion and similarity retrieval, so it is necessary to first construct style keywords that cover as many possible descriptions as possible.

[0149] Optionally, in this embodiment, a semantic representation vocabulary of style images (i.e., a semantic-style library) is constructed, which associates the semantic representation of the image with style words. When the user has no style input, only the image to be converted is input. In order to obtain the best style conversion effect, the system performs semantic analysis on the input image to find the style of the image with relevant semantic content, thereby obtaining a more harmonious generation effect.

[0150] It should be noted that for style word (expected style) input, the existing style words are retrieved through text embedding to determine the generated style words to which they belong; for reference style image input, the system obtains the reference image style representation embedding, determines the style category to which it belongs, and obtains the corresponding style word; for input style information is empty (at this time, the user hopes to obtain the most appropriate style conversion map for the input image), the most appropriate style word is obtained according to the input image content. After obtaining the style word, a new style image is generated based on the style transfer model.

[0151] To further illustrate, Fig.12 As shown, the method for associating style images with style words in this embodiment requires the preparation of an image style characterization model in advance, and finally obtains: an image semantics-style association inventory, an expanded text-style association inventory through the above-mentioned image style characterization model, manual style word expansion, etc.

[0152] Optionally, in this embodiment, the above recognition of text information, reference image information and original image information to obtain target style words can be achieved by, but not limited to, a pre-trained style recognition model.

[0153] Optionally, in this embodiment, the style recognition model can be trained in, but is not limited to, the following ways:

[0154] First, we collect multiple TV series and extract video frames based on the title of the series, remove duplicate images, and remove pictures without style information, such as black screen frames, pure text and logo frames (manual removal is used in this stage, or the existing black screen logo text classification model can identify such frames and delete them; removing duplicate images includes removing close-ups of the same characters, removing similar scenes in the same plot, etc., and about 20 to 30 valid images can be retained in an episode of a TV series), thereby forming M TV series image sets, each of which is the style image of the series.

[0155] Based on the above M film and television drama styles, the style recognition model is trained to support the input of a picture and output style categories (1 to M, indicating which film and television drama): Based on resnet18, the film and television drama classification model is trained, and the resnet18 pre-trained parameters trained based on the open source imagenet1000 classification data set are used as the model parameter initialization. During training, all model parameters need to be fine-tuned (note that when using this model as a style model, the fully connected layer outputs M, not 1000), and other layer parameters are not updated. SGD can be used for training according to the existing deep learning classification model method, which will not be described in detail here. After this style model is trained, the averagepool outputs a 1x512 vector that can be used as the image style embedding.

[0156] Then extract the style embedding for each frame, which is the input of the classification layer of the classification model (i.e. Fig.13 The output of the pool in (i.e., embedding). For example, Fig.13 The figure shows the model structure of Resnet18 based on the open source imagenet1000 classification. This embodiment can but is not limited to using this model for style classification, so the output of the fullyconnected module is changed to M (instead of 1000).

[0157] Calculate the mean of the style embedding for each drama image as the style representation of the drama

[0158] For each style record (style name: s0, style representation: emb0).

[0159] K-Means is used to cluster all style representations emb0, and the number of cluster centers is M / 10.M is the total number of film and television drama styles. M / 10 clusters (M / 10 cluster centers emb_center) are obtained, and then the Euclidean distance is calculated based on each style emb0 and all emb_centers, and the cluster center with the smallest distance is taken as the cluster to which the style belongs. Record data: (style name: s0, style representation: emb0, cluster to which it belongs: k).

[0160] Manually supplement style words. Since different people have different concepts of style, some people think that a certain anime is cartoon style, but cartoon style includes 2D, 3D and many other subdivided styles, so manual style nouns are needed. This step is used to improve the application to supplement manual style words for each film and television drama style (such as 3D Chinese comics battle style, 3D comics), thereby improving the recall rate when retrieving style words provided by users (for example, when a user enters "3D battle style", the corresponding target style can be correctly matched).

[0161] According to the above method, all style information is obtained and style representation record data is saved: (style name: s0, style representation: emb0, cluster: k, artificial style word: xxx). So far, style information and style-related film and television drama pictures have been collected.

[0162] In order to avoid binding style words with certain semantics, after collecting style images, it is necessary to describe the image content, rather than using the same style word to describe pictures with different semantic contents but the same style as the existing solution 3dreambooth. This step manually describes the image, such as style description: "A certain anime battle style", "3D Chinese comic style", semantic description: "Two people stand opposite each other on the battle stage, the person on the left is wearing blue clothes, the person on the right has red hair and wears black clothes, and there is a woman holding a scepter in the distance."

[0163] Optionally, in this embodiment, the style conversion of the original image according to the target style may be implemented, but is not limited to, by a pre-trained style conversion model, wherein the style conversion model may be, but is not limited to, an improved training model based on a style base model.

[0164] Optionally, in this embodiment, the style base model is trained and generated as follows:

[0165] Based on the above-mentioned film and television style data, the stable-diffusion model is fine-tuned to fine-tune the new style of the target into the style base model so that the subsequent style transfer model can support the generation of new styles. In this stage, a training set is formed from a large amount of film and television data, and the style of film and television dramas is trained into the basic stable-diffusion.

[0166] like Fig.14As shown in the figure, the training process of the stable-diffusion model is as follows: for the image after adding noise (embedding, VAE (Vector Adaptive Modulation, VAE for short) encoding is used to map to the latent feature space), the latent space representation at time T is obtained through the diffusion process, and the image features (i.e., the original image features without adding noise) are restored through T times of DenoisingU-Net (denoising process), and the restored encoding features are decoded by VAE to obtain the image. For text, the text embedding is obtained through the CLIP text branch and then input into U-Net as the input feature processing that needs to be denoised by U-Net. Diffusion sampling is used to map the VAE-encoded features of the noisy image to the latent space representation at time T, and the subsequent denoising process of the image learns to produce a fitting of the noise representation so that the original image minus the noise representation obtains the real image representation, and the real image is obtained through the decoder D. During the inference application process, the input is a pure noise image (compared to the image plus noise image during training).

[0167] This embodiment fine-tunes the style of the stable-diffusion model, and the training process is described as follows.

[0168] The fine-tuned image-text pair is (image, description text), and a total of E rounds (such as 100) iterations (epochs) are performed on the full data (film and TV drama image-text pairs). In each round of iteration: the full amount of image data is trained once in the model, which is called an iteration. Due to the limited memory resources, it is impossible to train all data at the same time, so it is necessary to train the full amount of data in batches, that is, update the model once every bs image-text pairs are trained until all image-text pairs have been trained and completed a round of iteration. Due to the imbalance of data between styles, the training is based on the M / 10 clusters obtained from the above clustering. When sampling each batch of samples: randomly select bs clusters, and randomly select 1 image-text pair from the data of the extracted cluster category to form a training sample.

[0169] For a certain image-text pair sample, the original image is used to add noise and then input into the VAE encoder as a noise map. The text is used to generate constraints. The loss is calculated by generating the output image and the original image. During training:

[0170] Initialize parameters before training the first batch of the first round: use the open source trained model (stable-diffusionv1-5) parameters for encoder, decoder, text_encoder, and U-Net, and only the U-Net parameters need to be updated in this training, and the others are not updated. Initialize with a learning rate of 0.0004, and after every 10 rounds of learning, the learning rate becomes 0.1 times the original, for a total of E rounds of training.

[0171] A random seed i is taken to generate a noise image, which is superimposed on the original image to generate image x, and image x is represented in the latent space to generate ZT.

[0172] The text information is clipped to obtain text representation, which is input into the generation model (the text representation is used as KV information), and T DenoisingU-Net forward calculations are performed on ZT under the KV constraint. After the first forward calculation, ZT-1 is obtained. Finally, after T times, the U-Net output is predicted Z0, and the predicted image is obtained through the decoder.

[0173] Calculate batch loss: Calculate the loss (this step is the MSE loss of the predicted image and the original image), and count the total loss of the batch of samples.

[0174] Using the SGD random gradient descent method, the loss is fed back to the model to obtain the gradient of the model parameters (U-Net) and update the parameters.

[0175] Complete all N / bs batches of training and end an iteration.

[0176] It should be noted that when calculating the prediction loss of an image, as shown in the following formula, the MSE loss is calculated by outputting the predicted image and the mean square error of the image of the image-text pair. The following y is the pixel value of each point in the image of the image-text pair. The p indicates the predicted pixel.

[0177]

[0178] Optionally, in this embodiment, after obtaining a basic model with the ability to generate a new style, in order to input an image and output a target style map, the style conversion model needs to be fine-tuned. This embodiment is based on the open source instructpix2pix model, and the model structure is as follows: Fig.15 As shown. This style transfer model has the same structure as the above-mentioned SD model, with the main difference being the input. During training, the SD input is the original image plus a noise image, while the instructpix2pix model input is the original image spliced ​​with the rendering result image plus noise; during application, the SD model input is a noise image, while the instructpix2pix model input is the original image spliced ​​with the noise image. The Instructpix2pix model ultimately predicts a rendering result image. When this embodiment is applied, it is hoped that the output rendering result image will be an image of the target style. After the above-mentioned basic model is fine-tuned, the new basic model carries the target new style information, and the rendering model needs to be updated on the new basic model. In this stage, the rendering model is fine-tuned and updated using the new basic model.

[0179] To further illustrate, Fig.16 As shown in (a) of FIG. 1 , it is the input image for SD model training, and the input is the original image plus the noise image; and as shown in FIG. Fig.16 As shown in (b), it is the input image for instructpix2pix model training (stacked up and down to represent splicing), and the input is the original image spliced ​​with the rendered result image plus noise.

[0180] This embodiment fine-tunes the style of the instructpix2pix model, and the training process is described as follows.

[0181] The instructpix2pix parameters pre-trained under open source data are used as initialization parameters. The fine-tuning process is the same as the above SD fine-tuning process, except that sample balance sampling is not required, and only the U-Net structure needs to be trained during training. The process is described as follows:

[0182] Initialize parameters before training the first batch of the first round: use the parameters of the open source trained model (instructpix2pix) for encoder, decoder, text_encoder, and U-Net, and only the U-Net parameters need to be updated in this training, and the others are not updated. Initialize with a learning rate of 0.0004, and then after every 10 rounds of learning, the learning rate becomes 0.1 times the original, and a total of E2 rounds of training are conducted.

[0183] Take a random seed i to generate a noise image, press Figure 6 In the process on the right, the original image and the noise-added effect image are concatenated end to end (to obtain image x) and input into the network, and image x is represented in the latent space to generate ZT.

[0184] The text information is clipped to obtain text representation, which is input into the generation model (the text representation is used as KV information), and T DenoisingU-Net forward calculations are performed on ZT under the KV constraint. After the first forward calculation, ZT-1 is obtained. Finally, after T times, the U-Net output is predicted Z0, and the predicted image is obtained through the decoder.

[0185] Calculate batch loss: Calculate the loss (this step is the MSE loss of the predicted image and the generated effect image), and count the total loss of the batch of samples.

[0186] Using the SGD random gradient descent method, the loss is fed back to the model to obtain the gradient of the model parameters (U-Net) and update the parameters.

[0187] Complete all N / bs batches of training and end an iteration.

[0188] It should be noted that when calculating the prediction loss of an image, as shown in the following formula, the model loss also samples the above-mentioned MSE loss. The difference is that y is the rendering effect image in the rendering training sample, and p is also the result image predicted by the model.

[0189]

[0190] It should be noted that if Fig.17 As shown, during the reasoning of the rendering model, the original image 1702 and the noise image 1704 are spliced ​​(different from the splicing of the original image and the rendering image with added noise), and input into the fine-tuned instructpix2pix model 1706, wherein the fine-tuned instructpix2pix model 1706 combines the input style word text 1708 to perform image style conversion to generate an image after image style conversion, wherein the style word text 1708 can be, but is not limited to, the edited text required for the image style conversion application to indicate the image style to be converted, such as a cartoon style.

[0191] It should be noted that in this embodiment, the hyperparameters of model training, such as learning rate, number of samples, cluster center data, etc., can be adjusted according to actual needs; the quantities in the application process, such as topK, etc., can be adjusted according to actual needs. If the style to be generated has been trained in the open source SD model, the rendering model does not need to be fine-tuned; the manual image annotation description stage can be replaced by automatic annotation using an open source model, such as using the BLIP-captioning method to annotate the image (but some incorrect annotation information will be introduced).

[0192] It should be noted that, in this embodiment, the above-mentioned image processing method is used to train the basic generation model through large-scale Chinese scene style data so that the generation model has the ability to recognize the Chinese style, thereby driving the subsequent style conversion model training that supports Chinese. By content-based annotation of training samples and a data processing and model training method that improves the effect of large-scale unbalanced style learning, the generation effect of head and tail styles is improved, and the problem of binding style with scene content is greatly reduced; the entire process of image style conversion model application under different input conditions is met: including the film and television style representation extraction and style description extraction methods used therein.

[0193] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0194] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0195] According to another aspect of the embodiments of the present application, an image processing device for implementing the above-mentioned image processing method is also provided. Fig.18 As shown, the device comprises:

[0196] A display unit 1802 is used to display an auxiliary processing area and a first image to be processed, wherein the auxiliary processing area is used to obtain at least one auxiliary information referenced when processing the first image, and the auxiliary processing area is configured to allow input of custom auxiliary information, and the at least one auxiliary information includes custom auxiliary information;

[0197] An acquiring unit 1804 is configured to acquire style auxiliary information using the auxiliary processing area in response to a style processing request triggered on the first image, wherein the at least one auxiliary information includes style auxiliary information, and the style auxiliary information is used to indicate a target style into which the first image is expected to be processed;

[0198] The processing unit 1806 is configured to process the first image into a second image of a target style, wherein the image content similarity between the first image and the second image is greater than or equal to a preset threshold.

[0199] As an optional solution, the acquisition unit 1804 includes:

[0200] A first acquisition module is used to acquire style auxiliary information based on at least one valid information when the customized auxiliary information in the auxiliary processing area is at least one valid information input; or

[0201] The second acquisition module is used to acquire default auxiliary information corresponding to the first image when the customized auxiliary information in the auxiliary processing area is blank information, and determine the default auxiliary information as style auxiliary information, wherein the style similarity between the style represented by the default auxiliary information and the original style of the first image is greater than or equal to a first threshold.

[0202] As an optional solution, the first acquisition module includes:

[0203] A first determination submodule is configured to determine a piece of custom auxiliary information as style auxiliary information when at least one valid information includes a piece of custom auxiliary information; or

[0204] A second determination submodule is used to determine each of the N pieces of custom auxiliary information as style auxiliary information when at least one valid information includes N pieces of custom auxiliary information, so as to obtain N pieces of style auxiliary information, wherein N is an integer greater than 1; or

[0205] The third determination submodule is used to perform similarity measurement on the styles represented by the M custom auxiliary information when at least one valid information includes M custom auxiliary information, and classify the M custom auxiliary information as target auxiliary information using the measurement result of the similarity measurement, and determine the target auxiliary information as style auxiliary information, wherein M is an integer greater than 1.

[0206] As an optional solution, the device also includes:

[0207] A third acquisition module, used for acquiring the first video to be processed before acquiring the default auxiliary information corresponding to the first image and determining the default auxiliary information as the style auxiliary information;

[0208] an extraction module, configured to extract a video key frame from the first video in response to a style processing request triggered on the first video before obtaining the default auxiliary information corresponding to the first image and determining the default auxiliary information as the style auxiliary information, wherein the video key frame is a video frame having key content among a plurality of first video frames contained in the first video, and the key content is used to represent the video content of the first video;

[0209] A first determination module is used to determine the image corresponding to the video key frame as the first image before obtaining the default auxiliary information corresponding to the first image and determining the default auxiliary information as the style auxiliary information;

[0210] The device also includes:

[0211] The first processing module is used to process multiple first video frames into second video frames of a target style after obtaining default auxiliary information corresponding to the first image and determining the default auxiliary information as style auxiliary information, so as to obtain a second video of the target video, wherein the second video includes multiple second video frames.

[0212] As an optional solution, the processing unit 1806 includes:

[0213] The second processing module is used to input the first image as an input image into the style transfer model to obtain a second image output by the style transfer model based on the style auxiliary information, wherein the style transfer model is a neural network model trained using multiple image samples and used to convert the original style of the input image into a desired style.

[0214] As an optional solution, the device also includes:

[0215] The execution module is used to input the first image as an input image into the style transfer model to obtain a second image output by the style transfer model based on the style auxiliary information, and then perform the following steps until a trained style transfer model is obtained:

[0216] a fourth acquisition module, configured to acquire a current image sample from a plurality of image samples and a current standard image corresponding to the current image sample when the current image sample belongs to the sample style, before inputting the first image as an input image into the style transfer model to obtain a second image output by the style transfer model based on the style auxiliary information, wherein the sample style is different from an original style of the current image sample;

[0217] A denoising module, used for, before inputting the first image as an input image into the style transfer model to obtain a second image output by the style transfer model based on the style auxiliary information, performing denoising processing on the current image sample, and inputting the denoised current image sample into the current style transfer model to obtain a sample image of the sample style output by the current style transfer model;

[0218] a fifth acquisition module, configured to, before inputting the first image as an input image into the style transfer model to obtain a second image output by the style transfer model based on the style auxiliary information, acquire a new image sample from the plurality of image samples when the image content similarity between the sample image of the sample style and the current standard image is less than a training threshold, and use the new image sample as the current image sample to perform next training on the current style transfer model;

[0219] The second determination module is used to determine the current style transfer model as a trained style transfer model when the image content similarity between the sample image of the sample style and the current standard image is greater than or equal to a training threshold before the first image is input as an input image to the style transfer model to obtain a second image output by the style transfer model based on the style auxiliary information.

[0220] As an optional solution, the device further includes:

[0221] A sixth acquisition module, configured to acquire a plurality of image samples before inputting the first image as an input image into the style transfer model to obtain a second image output by the style transfer model based on the style auxiliary information;

[0222] A clustering module, configured to perform clustering processing on a plurality of image samples according to image styles to obtain a P-type image sample set before inputting the first image as an input image into the style transfer model to obtain a second image output by the style transfer model based on the style auxiliary information, wherein P is an integer greater than 1, the image sample set includes at least one image sample, and the image style similarity between the image samples in the same image sample set is greater than or equal to a second threshold;

[0223] The training module is used to select image sample sets of different categories from the P category image sample set in turn before inputting the first image as an input image into the style transfer model to obtain the second image output by the style transfer model based on the style auxiliary information, and iteratively train the initial style transfer model until a trained style transfer model is obtained.

[0224] As an optional solution, the acquisition unit 1804 includes:

[0225] A third determination module is used to input the valid information as input information into a style recognition model when the customized auxiliary information in the auxiliary processing area is valid information, so as to obtain target style information output by the style recognition model, wherein the style recognition model is a neural network model trained by using a plurality of information samples and used to identify the style of the input information;

[0226] A seventh acquisition module, used to acquire style auxiliary information based on the target style information; or,

[0227] a fourth determination module, configured to input the first image as input information into the style recognition model when the customized auxiliary information in the auxiliary processing area is blank information, to obtain first style information output by the style recognition model, wherein the first style information is used to indicate the style to which the first image belongs;

[0228] an eighth acquisition module, configured to acquire second style information associated with the first style information, wherein a style similarity between the style represented by the second style information and the style to which the first image belongs is greater than or equal to a third threshold;

[0229] The ninth acquisition module is used to acquire style auxiliary information based on the second style information.

[0230] As an optional solution, the seventh acquisition module includes: a first optimization submodule, which is used to optimize the target style information in combination with the media comment information associated with the effective information when the media comment information associated with the effective information is acquired, so as to obtain style auxiliary information;

[0231] The ninth acquisition module includes: a second optimization submodule for optimizing the target style information in combination with the media comment information associated with the second style information when the media comment information associated with the second style information is acquired, so as to obtain style auxiliary information.

[0232] As an optional solution,

[0233] The device also includes:

[0234] A display module is used to determine, after processing a first image into a second image of a target style, a preset number of second images ranked before a preset number of display according to the style matching between the second images and the target style, from the second images whose number is greater than the preset number of display, and display them, wherein the preset number of display is a preset number of display corresponding to the image after the first image is expected to be processed into the target style.

[0235] For the specific implementation, reference may be made to the example shown in the above-mentioned image processing method, which will not be described in detail in this example.

[0236] According to another aspect of the embodiment of the present application, an electronic device for implementing the above-mentioned image processing method is also provided. The electronic device can be but is not limited to being Figure 1 The client 102 or server 112 shown in FIG. 1 is a client 102 or a server 112 shown in FIG. 1 . In this embodiment, the electronic device is used as the client 102 as an example. Fig.19 As shown, the electronic device includes a memory 1902 and a processor 1904. The memory 1902 stores a computer program, and the processor 1904 is configured to execute the steps in any of the above method embodiments through the computer program.

[0237] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0238] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0239] S1, displaying an auxiliary processing area and a first image to be processed, wherein the auxiliary processing area is used to obtain at least one auxiliary information for reference when processing the first image, and the auxiliary processing area is configured to allow input of custom auxiliary information, and the at least one auxiliary information includes custom auxiliary information;

[0240] S2, in response to a style processing request triggered on the first image, obtaining style auxiliary information using the auxiliary processing area, wherein the at least one auxiliary information includes style auxiliary information, and the style auxiliary information is used to indicate a target style into which the first image is expected to be processed;

[0241] S3, processing the first image into a second image of a target style, wherein the image content similarity between the first image and the second image is greater than or equal to a preset threshold.

[0242] Alternatively, a person skilled in the art may understand that: Fig.19 The structure shown is for illustration only. Fig.19 The structure of the electronic device is not limited. Fig.19 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig.19 Different configurations are shown.

[0243] Among them, the memory 1902 can be used to store software programs and modules, such as program instructions / modules corresponding to the image processing method and device in the embodiments of the present application. The processor 1904 executes various functional applications and data processing by running the software programs and modules stored in the memory 1902, that is, realizing the above-mentioned image processing method. The memory 1902 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1902 may further include a memory remotely located relative to the processor 1904, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 1902 can be specifically, but not limited to, used to store style auxiliary information and other information. As an example, such as Fig.19 As shown, the memory 1902 may include, but is not limited to, the display unit 1802, the acquisition unit 1804, and the processing unit 1806 in the image processing device. In addition, it may also include, but is not limited to, other module units in the image processing device, which will not be described in detail in this example.

[0244] Optionally, the transmission device 1906 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired network and a wireless network. In one example, the transmission device 1906 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 1906 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0245] In addition, the electronic device further includes: a display 1908 for displaying the style auxiliary information and other information; and a connection bus 1910 for connecting various module components in the electronic device.

[0246] In other embodiments, the client or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes in the form of network communication. A peer-to-peer network may be formed between the nodes, and any form of computing device, such as a server, a client, or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0247] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program, the computer program comprising a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit, various functions provided by the embodiments of the present application are executed.

[0248] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0249] It should be noted that the computer system of the electronic device is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0250] The computer system includes a central processing unit (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) or the program loaded from the storage part to the random access memory (RAM). In the random access memory, various programs and data required for system operation are also stored. The central processing unit, the read-only memory and the random access memory are connected to each other through a bus. The input / output interface (I / O interface) is also connected to the bus.

[0251] The following components are connected to the input / output interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a local area network card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the input / output interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that the computer program read therefrom is installed into the storage part as needed.

[0252] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit, various functions defined in the system of the present application are executed.

[0253] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of an electronic device reads a computer program from the computer-readable storage medium, and the processor executes the computer program, so that the electronic device executes the methods provided in the above-mentioned various optional implementations.

[0254] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0255] S1, displaying an auxiliary processing area and a first image to be processed, wherein the auxiliary processing area is used to obtain at least one auxiliary information for reference when processing the first image, and the auxiliary processing area is configured to allow input of custom auxiliary information, and the at least one auxiliary information includes custom auxiliary information;

[0256] S2, in response to a style processing request triggered on the first image, obtaining style auxiliary information using the auxiliary processing area, wherein the at least one auxiliary information includes style auxiliary information, and the style auxiliary information is used to indicate a target style into which the first image is expected to be processed;

[0257] S3, processing the first image into a second image of a target style, wherein the image content similarity between the first image and the second image is greater than or equal to a preset threshold.

[0258] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the electronic device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0259] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0260] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0261] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0262] In the several embodiments provided in the present application, it should be understood that the recorded client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0263] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0264] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0265] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: include: Displaying an auxiliary processing area and a first image to be processed, wherein the auxiliary processing area is used to obtain at least one auxiliary information for reference when processing the first image, the auxiliary processing area is configured to allow input of custom auxiliary information, and the at least one auxiliary information includes the custom auxiliary information; In response to a style processing request triggered on the first image, obtaining style auxiliary information using the auxiliary processing area, wherein the at least one auxiliary information includes the style auxiliary information, and the style auxiliary information is used to indicate a target style into which the first image is expected to be processed; The first image is processed into a second image of the target style, wherein the image content similarity between the first image and the second image is greater than or equal to a preset threshold.

2. The method according to claim 1, characterized in that The obtaining style auxiliary information by using the auxiliary processing area includes: In the case where the customized auxiliary information is at least one valid information input in the auxiliary processing area, acquiring the style auxiliary information based on the at least one valid information; or, When the custom auxiliary information in the auxiliary processing area is blank information, default auxiliary information corresponding to the first image is obtained, and the default auxiliary information is determined as the style auxiliary information, wherein the style similarity between the style represented by the default auxiliary information and the original style of the first image is greater than or equal to a first threshold.

3. The method according to claim 2, characterized in that The acquiring the style auxiliary information based on the at least one valid information includes: In a case where the at least one valid information includes one piece of user-defined auxiliary information, determining the one piece of user-defined auxiliary information as the style auxiliary information; or, In the case where the at least one valid information includes N pieces of custom auxiliary information, each piece of the N pieces of custom auxiliary information is determined as the style auxiliary information to obtain N pieces of the style auxiliary information, wherein N is an integer greater than 1; or In a case where the at least one valid information includes M pieces of custom auxiliary information, a similarity measurement is performed on the styles represented by the M pieces of custom auxiliary information, and the M pieces of custom auxiliary information are classified as target auxiliary information using a measurement result of the similarity measurement, and the target auxiliary information is determined as the style auxiliary information, wherein M is an integer greater than 1.

4. The method according to claim 2, characterized in that: Before acquiring the default auxiliary information corresponding to the first image and determining the default auxiliary information as the style auxiliary information, the method further includes: Get the first video to be processed; In response to a style processing request triggered on the first video, extracting a video key frame from the first video, wherein the video key frame is a video frame having key content among a plurality of first video frames included in the first video, and the key content is used to represent video content of the first video; Determine the image corresponding to the video key frame as the first image; After acquiring the default auxiliary information corresponding to the first image and determining the default auxiliary information as the style auxiliary information, the method further includes: The multiple first video frames are all processed into second video frames of the target style to obtain a second video of the target video, wherein the second video contains multiple second video frames.

5. The method according to claim 1, characterized in that The processing of the first image into a second image of the target style includes: The first image is input as an input image to a style transfer model to obtain the second image output by the style transfer model based on the style auxiliary information, wherein the style transfer model is a neural network model trained using multiple image samples and used to convert the original style of the input image into a desired style.

6. The method according to claim 5, characterized in that Before inputting the first image as an input image into a style transfer model to obtain the second image output by the style transfer model based on the style auxiliary information, the method further includes: Perform the following steps until you get a trained style transfer model: Acquire a current image sample from the multiple image samples, and a current standard image corresponding to the current image sample when the current image sample belongs to a sample style, wherein the sample style is different from an original style of the current image sample; Performing noise processing on the current image sample, and inputting the noised current image sample into the current style transfer model to obtain a sample image of the sample style output by the current style transfer model; When the image content similarity between the sample image of the sample style and the current standard image is less than a training threshold, obtaining a new image sample from the multiple image samples, and using the new image sample as the current image sample to perform next training on the current style transfer model; In a case where the image content similarity between the sample image of the sample style and the current standard image is greater than or equal to the training threshold, the current style transfer model is determined as the trained style transfer model.

7. The method according to claim 5, characterized in that Before inputting the first image as an input image into a style transfer model to obtain the second image output by the style transfer model based on the style auxiliary information, the method further includes: acquiring the plurality of image samples; According to the image style, clustering processing is performed on the multiple image samples to obtain a P-type image sample set, where P is an integer greater than 1, the image sample set contains at least one of the image samples, and the image style similarity between the image samples in the same image sample set is greater than or equal to a second threshold; Image sample sets of different categories are selected from the P category image sample sets in turn, and the initial style transfer model is iteratively trained until a trained style transfer model is obtained.

8. The method according to claim 1, characterized in that The obtaining style auxiliary information by using the auxiliary processing area includes: In the case where the custom auxiliary information in the auxiliary processing area is valid input information, the valid information is input as input information into a style recognition model to obtain target style information output by the style recognition model, wherein the style recognition model is a neural network model trained using a plurality of information samples and used to recognize the style to which the input information belongs; Acquire the style auxiliary information based on the target style information; or, When the customized auxiliary information in the auxiliary processing area is blank information, inputting the first image as input information into a style recognition model to obtain first style information output by the style recognition model, wherein the first style information is used to indicate the style to which the first image belongs; Acquire second style information associated with the first style information, wherein a style similarity between a style represented by the second style information and a style to which the first image belongs is greater than or equal to a third threshold; The style auxiliary information is acquired based on the second style information.

9. The method according to claim 8, characterized in that The acquiring of the style auxiliary information based on the target style information includes: when the media comment information associated with the valid information is acquired, optimizing the target style information in combination with the media comment information associated with the valid information to acquire the style auxiliary information; The acquiring of the style auxiliary information based on the second style information includes: when media comment information associated with the second style information is acquired, optimizing the target style information in combination with the media comment information associated with the second style information to acquire the style auxiliary information.

10. The method according to any one of claims 1 to 9, characterized in that After processing the first image into a second image of the target style, the method further includes: When the number of the second images is greater than the preset display number, according to the style matching degree between the second images and the target style, the second images whose style matching degree is ranked before the preset display number are determined from the second images whose number is greater than the preset display number, and are displayed, wherein the preset display number is a preset display number corresponding to the image after the first image is expected to be processed into the target style.

11. An image processing device, characterized in that: include: A display unit, configured to display an auxiliary processing area and a first image to be processed, wherein the auxiliary processing area is used to obtain at least one auxiliary information referenced when processing the first image, the auxiliary processing area is configured to allow input of custom auxiliary information, and the at least one auxiliary information includes the custom auxiliary information; an acquisition unit, configured to acquire style auxiliary information using the auxiliary processing area in response to a style processing request triggered on the first image, wherein the at least one auxiliary information includes the style auxiliary information, and the style auxiliary information is used to indicate a target style into which the first image is expected to be processed; A processing unit is used to process the first image into a second image of the target style, wherein the image content similarity between the first image and the second image is greater than or equal to a preset threshold.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein the computer program executes the method described in any one of claims 1 to 10 when executed by an electronic device.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 10 through the computer program.

Images