Method for training rotation point output model and processing cartoon image, related device and computer program product

Abstract

The invention provides a method for training a rotation point output model and processing a cartoon image, a related device and a computer program product. According to the application, a cartoon face image can be generated at least based on a real face image; the cartoon face image serves as input, a rotation point determined based on the real face image serves as output, the initial rotation point output model is trained, and the rotation point is used for rotating the presentation posture of the real face in the real face image to the target presentation posture; and obtaining a rotation point output model in response to determining that the training of the initial rotation point output model meets the predetermined training requirements. In the implementation mode provided by the invention, the rotation point determined based on the real face image can be used as the training sample to train the rotation point output model for processing the cartoon face image, so that the marking cost for the cartoon face image is reduced, and the processing capability of the rotation point output model is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device, computer-readable medium and computer program product for training a rotation point output model and processing cartoon images. Background Art

[0002] In order to meet users' more diversified and personalized image browsing and image interaction needs, image processing technology, as an important branch of computer technology, has also developed with the development of computer technology.

[0003] Based on image processing technology, users can obtain images generated by artificial intelligence models through "image generation" to enrich users' browsing experience and interactive fun. For example, users can use image processing technology to process real face images into "cartoon face images" of various styles.

[0004] Cartoon face images refer to face images that have been artistically processed, usually with simplified lines and colors, making them look more like hand-painted or cartoon-style artworks. Therefore, for cartoon face images, how to improve the user's reading experience and how to improve the quality of use of cartoon face images are worthy of attention and urgent needs. Summary of the invention

[0005] Multiple aspects of the present application provide a method, device, electronic device, computer-readable storage medium and computer program product for training a rotation point output model, which can use rotation points determined based on real face images as training samples to train a rotation point output model for processing cartoon face images, so as to reduce the labeling cost for cartoon face images while improving the processing capability of the rotation point output model.

[0006] Multiple aspects of the present application provide a method, device, electronic device, computer-readable storage medium and computer program product for processing cartoon images, which can use the rotation point model trained in the above process to process cartoon images. When the cartoon image includes a "cartoon face", high-quality "rotation" and "adjustment" can be achieved to facilitate more accurate identification of the "features" included therein.

[0007] In one aspect of the present application, a method for training a rotation point output model is provided, comprising: generating a cartoon face image based on a real face image; taking the cartoon face image as input and the rotation points determined based on the real face image as output, training an initial rotation point output model, wherein the rotation points are used to rotate the presentation posture of the real face in the real face image to a target presentation posture; in response to determining that the training of the initial rotation point output model meets predetermined training requirements, obtaining the rotation point output model.

[0008] In another aspect of the present application, there is provided an apparatus for training a rotation point output model, comprising: a cartoon image generation unit, configured to generate a cartoon face image based on a real face image; an output model training unit, configured to use the cartoon face image as input and the rotation point determined based on the real face image as output to train an initial rotation point output model; and an output model determination unit, configured to obtain a rotation point output model in response to determining that the training of the initial rotation point output model meets predetermined training requirements.

[0009] In another aspect of the present application, a method for processing a cartoon image is provided, comprising: using a rotation point output model to process a cartoon image to be processed to obtain a set of rotation points, wherein the rotation points are used to rotate the presentation posture of a cartoon face in the cartoon image to be processed to a target presentation posture, wherein the rotation point output model is trained based on the above-mentioned method for training a rotation point output model; in response to the number of rotation points included in the set of rotation points being greater than or equal to a quantity threshold, rotating the cartoon face using the set of rotation points to obtain a processed cartoon image.

[0010] In another aspect of the present application, a device for processing a cartoon image is provided, comprising: a rotation point acquisition unit, configured to process the cartoon image to be processed using a rotation point output model to obtain a set of rotation points, wherein the rotation points are used to rotate the presentation posture of a cartoon face in the cartoon image to be processed to a target presentation posture, wherein the rotation point output model is trained based on the above-mentioned device for training the rotation point output model; and a cartoon image processing unit, configured to rotate the cartoon face using a set of rotation points in response to the number of rotation points included in the set of rotation points being greater than or equal to a number threshold, to obtain a processed cartoon image.

[0011] Another aspect of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for training a rotating point output model and the method for processing a cartoon image as provided above.

[0012] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method for training a rotation point output model and the method for processing a cartoon image as provided above.

[0013] In another aspect of the present application, a computer program product includes a computer program having computer program instructions stored thereon, and when the computer program is executed by a processor, the computer program can implement the method for training a rotation point output model and the method for processing a cartoon image as provided above.

[0014] The solution provided by the embodiment of the present application can generate a cartoon face image based on a real face image during the training of the rotation point output model; the cartoon face image is used as input, and the rotation point determined based on the real face image is used as output to train the initial rotation point output model, wherein the rotation point is used to rotate the presentation posture of the real face in the real face image to the target presentation posture; in response to determining that the training of the initial rotation point output model meets the predetermined training requirements, the rotation point output model is obtained. Thus, the rotation point determined based on the real face image can be used as a training sample to train the rotation point output model for processing the cartoon face image, so as to reduce the annotation cost for the cartoon face image while improving the processing capability of the rotation point output model.

[0015] The solution provided by the embodiment of the present application can process the cartoon image to be processed using a rotation point output model to obtain a set of rotation points, wherein the rotation points are used to rotate the presentation posture of the cartoon face in the cartoon image to be processed to the target presentation posture, wherein the rotation point output model is trained based on the above-mentioned method for training the rotation point output model; in response to the number of rotation points included in the set of rotation points being greater than or equal to a quantity threshold, the cartoon face is rotated using a set of rotation points to obtain a processed cartoon image. Thus, the rotation point model trained in the above process can be used to process the cartoon image, and when the cartoon image includes a "cartoon face", high-quality "rotation" and "adjustment" can be achieved for it, so as to more accurately identify the "features" included therein. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0018] Figure 1 A flowchart of a process for training a rotation point output model provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of a process of processing a cartoon image provided by an embodiment of the present application;

[0020] Figure 3aA flowchart of a process of training a rotation point output model implemented in a specific application scenario provided by an embodiment of the present application;

[0021] Figure 3b A flowchart of a process for processing a cartoon image in a specific application scenario provided by an embodiment of the present application;

[0022] Figure 4 A schematic diagram of the structure of a device for training a rotation point output model provided in one embodiment of the present application;

[0023] Figure 5 A schematic diagram of the structure of a device for processing cartoon images provided in one embodiment of the present application;

[0024] Figure 6 The present invention is a schematic diagram of the structure of an electronic device suitable for implementing the solution in the embodiment of the present application.

[0025] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0028] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0029] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0030] As discussed above, for cartoon face images, how to improve the user's reading experience and how to improve the quality of use of cartoon face images are worthy of attention and urgent needs.

[0031] In this regard, in order to improve the quality and effect of image processing and analysis, you can choose to process the image to a certain extent so that the image can be better recognized by users, deep learning models, etc. For example, when processing an image including a face image, you can choose to rotate the "face" based on the rotation point included in the "face content" to adjust the "presentation posture" of the "face" so that the face can be viewed more clearly and comprehensively.

[0032] That is, after processing, the image can be expected to expose more "facial content" to users and deep learning models to facilitate users to obtain the content therein, or enable the deep learning model to obtain the (image) features included therein, such as facial features, with better quality.

[0033] In this context, for "cartoon face images", how to efficiently and efficiently determine the rotation points included therein (for example, the pixel positions of each image used to rotate the cartoon face) is worthy of attention and an urgent need.

[0034] In the present application, at least a training rotation point output model is selected, and then the trained rotation point output model is used to process a cartoon face image to output the ability of the rotation point that can be used to rotate the "face" in the face image. Thus, the rotation point in the cartoon face image can be determined efficiently and with quality using the rotation point output model.

[0035] And in this process, a cartoon face image can be generated based on a real face image; then, the cartoon face image is used as input, and the rotation point determined based on the real face image is used as output to train the initial rotation point output model, wherein the rotation point is used to rotate the presentation posture of the real face in the real face image to the target presentation posture; finally, in response to determining that the training of the initial rotation point output model meets the predetermined training requirements, a rotation point output model is obtained. Such a training method can use the rotation point determined based on the real face image as a training sample to train the rotation point output model for processing the cartoon face image, so as to reduce the annotation cost for the cartoon face image while improving the processing capability of the rotation point output model.

[0036] Subsequently, in the process of processing cartoon images, the rotation point output model can be used to process the cartoon image to be processed to obtain a set of rotation points, wherein the rotation points are used to rotate the presentation posture of the cartoon face in the cartoon image to be processed to the target presentation posture, wherein the rotation point output model is trained based on the above-mentioned method for training the rotation point output model; in response to the number of rotation points included in the set of rotation points being greater than or equal to a quantity threshold, the cartoon face is rotated using the set of rotation points to obtain a processed cartoon image. Thus, the rotation point model trained in the above process can be used to process cartoon images, and when a "cartoon face" is included in the cartoon image, high-quality "rotation" and "adjustment" can be achieved for it, so as to more accurately identify the "features" included therein.

[0037] In actual scenarios, the execution subject of the method can be a user device, or a device formed by integrating a user device and a network device through a network, or an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing (Cloud Computing), where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0038] When the execution subject is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.

[0039] It should be understood that in the technical solutions involved in this application, the acquisition, storage, use, processing, transportation, provision and disclosure of the personal information of the subjects involved (for example, real facial images involved in this application later, cartoon images that may include cartoon faces, etc.) are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0040] First, the process of training the rotation point output model is explained.

[0041] Figure 1 A process 100 for training a rotation point output model provided by an embodiment of the present application is shown. The process 100 includes at least the following processing steps:

[0042] (Step) S101, generating a cartoon face image based on a real face image.

[0043] In an embodiment of the present application, the execution entity can generate a cartoon face image with a real face image, that is, an image taken or acquired in a real scene, a real physical scene, and including a "real face (object)" (for example, a photo of a person with a real face, a scene photo, etc.).

[0044] For example, the execution subject can process real face images by calling Generative Adversarial Network (GAN), Variational Autoencoder (VAE), etc. to convert them into corresponding cartoon face images.

[0045] In some scenarios, the above process of processing real face images and generating cartoon face images can also be understood and described as "style conversion" of real face images. For example, cartoon face images can be considered as the result of converting real face images from "real style" to "cartoon style".

[0046] It should be understood that the obtained cartoon face image is "consistent" with the real face image in terms of image content, layout, arrangement, etc. That is, the real face image and the cartoon face image can actually be understood as two "face images" with the same content, layout, arrangement, etc., which only differ in the image "style".

[0047] In some embodiments, such a "cartoon face image" can also be understood as being generated based on a real face image, so that the face part is free from the image of the "real scene" face (for example, the "image" obtained by changing the face part into a different cartoon style).

[0048] In some embodiments, in order to improve the quality of the cartoon face image, a control module may be added to the diffusion model, and the diffusion model with the control module added may be used to generate the cartoon face image based on the real face image.

[0049] For example, the control module may be ControlNet.

[0050] ControlNet is a neural network structure for controlling diffusion models. ControlNet obtains additional input images and converts them into control charts through different preprocessors. These control charts are input into the Stable Diffusion diffusion model as additional conditions to guide the diffusion process and achieve more stable generation results.

[0051] Accordingly, by using the diffusion model with ControlNet attached, or in other words, by using the diffusion model controlled by ControlNet, it is possible to increase the control over the generated content without changing the core structure of the generation model. Thus, by using the diffusion model with ControlNet attached, or by using the diffusion model controlled by ControlNet, it is possible to more accurately and precisely generate cartoon face images that meet the requirements and are similar to real face images (for example, cartoon face images in different cartoon styles).

[0052] In some embodiments, in the process of selecting and determining the real face images as the "basis", samples with more characteristics or higher training difficulty can also be selected to enhance the training effect of the rotation point output model.

[0053] Specifically, after obtaining (a group of) candidate real face images that can be used as real face images, the execution subject can first determine the angular difference between the presentation posture of the real face in the candidate real face image and the target presentation posture.

[0054] Then, the execution subject selects the target candidate real face image whose angle difference is greater than or equal to the difference threshold as the real face image actually used.

[0055] That is, for a group of candidate real face images, if the angle difference between the presentation posture of the real face in one of the candidate real face images and the target presentation posture is greater than or equal to a difference threshold, the execution entity can respond to this and determine it as the target candidate real face image.

[0056] Subsequently, the execution subject may use these target candidate real face images as the "basis", ie, "real face images", to generate corresponding cartoon face images for the subsequent training process of the initial rotation point output model.

[0057] Accordingly, because the difference (angle difference) between the (current) presentation posture of the real face in the target candidate real face image and the target presentation posture is greater than or equal to the difference threshold, the reference value of the "rotation points" included therein may be more valuable than the posture being less than the difference threshold (for example, these "rotation points" can achieve a larger angle of rotation and a higher degree of difficulty of rotation).

[0058] Therefore, by selecting and determining the real face images as the "basis", it is also possible to select samples that are more characteristic or more difficult to train, so as to enhance the training effect of the rotation point output model.

[0059] S102, taking the cartoon face image as input and the rotation point determined based on the real face image as output, training an initial rotation point output model.

[0060] In an embodiment of the present application, after a cartoon face image is obtained based on the above S101, the initial rotation point output model can be trained using the cartoon face image and the rotation points determined based on the real face image as "samples".

[0061] As discussed above, the rotation point (determined based on the real face image) can be used to rotate the presentation posture of the real face in the real face image to a target presentation posture (for example, the target presentation posture may be, for example, "the face is facing the user", "the vertical score line of the face is perpendicular to the bottom edge of the image, and the face is facing the user").

[0062] In some scenarios, such “rotation points” may also be called “key points”, “rotation key points”, “face key points” or “adjustment key points”, etc.

[0063] For example, in some scenarios, the execution subject can adjust the presentation posture to the target presentation posture by adjusting the "face key points" to the corresponding angle.

[0064] In some embodiments, the target presentation posture may correspond to the subsequent purpose after the rotation. For example, in a scenario of extracting and recognizing features of a cartoon face, the target presentation posture may be "facing the user". Therefore, in these embodiments, such "rotation" and "adjustment" may also be referred to as normalization processing of the "cartoon face".

[0065] It should be understood that in the present application, because the cartoon face image is actually obtained through the "style conversion" of the real face image, the rotation point determined based on the real face image can also be used to adjust and rotate the cartoon face in the cartoon face image to "rotate" the presentation posture of the cartoon face to the target presentation posture similar to that of the real face.

[0066] Accordingly, the execution entity can choose to use the cartoon face image as input and the rotation point determined based on the real face image as output, and train the initial rotation point output model so that the rotation point output model has the corresponding "ability to extract rotation points from the cartoon face image".

[0067] That is, in this way, it is expected that by training the initial rotation point output model, it can learn the rotation points that should be extracted from the cartoon face image (or the logic of extracting these "rotation points").

[0068] Accordingly, when the training of the initial rotation point output model is completed and the rotation point output model is obtained, the rotation point output model can extract the corresponding rotation points from the input cartoon face image based on the "same logic", that is, it is used to rotate the cartoon face in the cartoon face image so that the cartoon face reaches those (pixel) points where the target presentation posture is achieved.

[0069] During the training process, the execution subject can complete the training of the initial rotation point output model by taking the cartoon face image as input, taking the rotation point determined based on the real face image as output, and using a predetermined loss function as supervision.

[0070] For example, during the training process, the execution entity can determine the training progress of the initial rotation point output model by detecting the change in the value of the loss function, and based on the change in the training progress, determine whether the training of the initial rotation point output model can be completed.

[0071] In some embodiments, in order to further improve the training effect of the initial rotation point output model, the execution entity can train the initial rotation point output model by using at least two, or multiple "inner" identical cartoon face images as training samples.

[0072] In some embodiments, in order to ensure the "inner consistency" of cartoon face images corresponding to the same "rotation point" at the same time, the execution entity may choose to process the cartoon face images by sample enhancement to obtain a set of enhanced cartoon face images.

[0073] Data augmentation can improve the generalization ability of the model by generating additional training samples. For example, data augmentation can generate a set of enhanced cartoon face images for cartoon face images by performing image augmentation such as rotation, translation, scaling, inversion, color conversion, brightness adjustment, contrast adjustment, noise addition, etc. on the cartoon face images.

[0074] Then, a set of enhanced cartoon face images is used as input and the rotation points determined based on real face images are used as output to train the initial rotation point output model.

[0075] For example, each enhanced cartoon face image in a set of enhanced cartoon face images may be obtained by replacing or splicing at least part of the cartoon face image, or by adjusting the presentation posture of the “cartoon face” included therein.

[0076] Thus, the enhanced cartoon face image can be used to change the presentation position and presentation posture of the actual content corresponding to the "rotation point". Thus, the initial rotation point output model can be more fully trained and better "learned" through a set of rotation points (i.e., the rotation points determined based on the real face image).

[0077] In some embodiments, in order to improve the training quality of the initial rotation point output model, the initial rotation model can be pre-trained, or "pre-trained", with a sample real face image as input and the sample rotation points corresponding to the sample real face image (i.e., the rotation points used to rotate the presentation posture of the real face in the sample real face image to the above-mentioned target presentation posture) as output.

[0078] By using such a "pre-trained" initial rotation point output model, since it already has the ability to determine the rotation point from the real face image to meet the requirements, the training task of processing the "cartoon face image" can be "converted" into enhanced training of the initial rotation point output model under a special type of "face image" (i.e., "cartoon face image"). Therefore, not only can the training workload of the rotation point output model be simplified through the strategies of "enhanced training" and "improved model", but the processing performance of the rotation point output model can also be enhanced through "enhanced training" by using the processing ability of the rotation point output model for "real face images" as a "base".

[0079] In other words, using the initial rotation point output model trained based on the "real face image" as a base, not only can the "rotation point output model" be trained faster, but also the trained rotation point output model can have better processing performance (for example, such a model may be able to refer to the "processing and recognition logic of real face images" to better identify and learn the "processing and recognition logic of real face images").

[0080] S103, in response to determining that the training of the initial rotation point output model meets the predetermined training requirement, obtaining the rotation point output model.

[0081] In an embodiment of the present application, if the execution entity determines that the training of the initial rotation point output model meets the predetermined training requirements, the execution entity can respond to this and complete the training of the initial rotation point output model to obtain the rotation point output model.

[0082] For example, in the case of supervision by a loss function, the execution subject may determine, for example, that the loss converges as the training of the initial rotation point output model meets the predetermined training requirements. For another example, in the scenario of "iterative training", the execution subject may also determine whether the training of the initial rotation point output model meets the predetermined training requirements by detecting whether the training rounds of the initial rotation point output model reach the target rounds (for example, when the target rounds are reached, it is determined that the training of the initial rotation point output model meets the predetermined training requirements).

[0083] Then, the method for training the rotation point output model provided by the present application can generate a cartoon face image based on a real face image; taking the cartoon face image as input and the rotation point determined based on the real face image as output, the initial rotation point output model is trained, wherein the rotation point is used to rotate the presentation posture of the real face in the real face image to the target presentation posture; in response to determining that the training of the initial rotation point output model meets the predetermined training requirements, the rotation point output model is obtained. In the implementation method provided by the present application, the rotation point determined based on the real face image can be used as a training sample to train the rotation point output model for processing the cartoon face image, so as to reduce the annotation cost for the cartoon face image while improving the processing capability of the rotation point output model.

[0084] In some embodiments, in order to improve the performance of the trained rotation point output model and enhance its scene applicability, in the process of training the initial rotation point output model, in addition to using the rotation points determined based on the real face image as training outputs, the initial rotation point output model can also be trained together with these rotation points and simultaneously using the face identification frame determined based on the real face image.

[0085] That is to say, the execution subject can also determine, based on the real face image, a face identification frame for identifying and indicating the real face in the real face image.

[0086] Then, in the process of executing the above S102, the execution subject, as an alternative or substitute, takes the cartoon face image as input and the mapping result of the rotation point and the face identification frame in the cartoon face image determined based on the real face image as output to train the initial rotation point output model.

[0087] Accordingly, after completing the training, the obtained rotation point output model can have a stronger recognition ability for the "position" and "area" of the cartoon face, and can output the ability to identify the recognized "position" and "area".

[0088] As a result, the initial rotation point can more accurately determine the position of the "face" and improve the training quality, while the trained rotation point output model can have the ability to output and indicate the position and area of ​​the cartoon face in the cartoon face image.

[0089] Next, the trained rotation point output model (for example, based on Figure 1 The use of the rotation point output model trained by the process 100 shown is explained.

[0090] For this, please refer to Figure 2 . Figure 2 A process 200 for processing a cartoon image provided by an embodiment of the present application is shown. The process 200 includes at least the following processing steps:

[0091] S201, using the rotation point output model to process the cartoon image to be processed to obtain a set of rotation points.

[0092] In an embodiment of the present application, the execution subject can determine whether the image is a cartoon image by analyzing the image after acquiring the image. If it is a cartoon image, the execution subject can use the rotation point output model to process the cartoon image to be processed to obtain a set of rotation points.

[0093] The rotation point output model can be based on the above example Figure 1 The embodiments discussed are trained and will not be repeated here.

[0094] Accordingly, after being processed by the rotation point output model, the obtained (a set of) rotation points can be used to rotate the presentation posture of the cartoon face in the cartoon image to be processed to the target presentation posture.

[0095] S202: In response to the number of rotation points included in a group of rotation points being greater than or equal to a number threshold, the cartoon face is rotated using the group of rotation points to obtain a processed cartoon image.

[0096] In an embodiment of the present application, if the number of rotation points in a set of rotation points determined based on the above S201 is greater than or equal to the quantity threshold (that is, the rotation points that meet the quantity requirement can be determined), the execution entity can use a set of rotation points to rotate the cartoon face (in the cartoon image) to obtain a processed cartoon image.

[0097] Generally, the quantity threshold may be predetermined based on the number of rotation points that can determine that the cartoon image is a cartoon face image and can satisfy the need to “rotate” the cartoon face in the cartoon face image.

[0098] It should be understood that because the function of the rotation point is to "rotate the cartoon face", accordingly, if a sufficient number of rotation points can be determined, then accordingly, the "cartoon image" is actually also highly likely and highly likely to be a "cartoon face image" that includes a cartoon face (because usually, if the cartoon face is not included, it may not be possible to determine a sufficient number of rotation points, because such rotation points are usually located above the "cartoon face").

[0099] Then, in the process of processing cartoon images, the rotation point output model can be used to process the cartoon image to be processed to obtain a set of rotation points, wherein the rotation points are used to rotate the presentation posture of the cartoon face in the cartoon image to be processed to the target presentation posture, wherein the rotation point output model is trained based on the above-mentioned method for training the rotation point output model; in response to the number of rotation points included in the set of rotation points being greater than or equal to the number threshold, the cartoon face is rotated using the set of rotation points to obtain a processed cartoon image. Thus, the rotation point model trained in the above process can be used to process cartoon images, and when a "cartoon face" is included in the cartoon image, high-quality "rotation" and "adjustment" can be achieved for it, so as to more accurately identify the "features" included therein.

[0100] As discussed above, the adjusted cartoon image can be used for various types of subsequent processing tasks.

[0101] For example, in an image classification task, the executor may choose to extract “features” based on the processed cartoon image, and then use the feature database to determine whether the “features” are of a certain type.

[0102] For example, in the task of further classifying a "cartoon (face) image", the execution subject may determine the number of cartoon face features hit by the processed cartoon image in the feature database. If the number is greater than or equal to the number threshold, the execution subject may respond to this and classify the processed cartoon image into the target classification.

[0103] For example, the target classification may be a classification corresponding to a feature database, that is, if the number of cartoon face features in the feature database included in the cartoon (face) image is greater than or equal to a quantity threshold, the cartoon (face) image may be classified into the target classification corresponding to the feature database.

[0104] Similarly, this processing method can also be applied to different tasks based on different requirements. For example, for the cartoon face extraction task, the execution subject can use the rotation point output model to output the rotation point and rotate the cartoon face in the cartoon image to achieve extraction.

[0105] Therefore, in some embodiments, based on the final task purpose, the rotation point output model may be used as a sub-model, a layer in the model, and a "complete model" for achieving the complete task.

[0106] For example, for classification tasks, the complete model can be described as a "classification model". Accordingly, the rotation point output model can be used as a sub-model or a layer of the "classification model". The "classification model" can also include a rotation model and a rotation layer located after or connected to the rotation point output model (for example, a model or layer for rotating cartoon faces based on rotation points), and a feature matching layer located after the rotation model and the rotation layer (for example, used to determine the number of cartoon face features in the feature database hit by the processed cartoon image), etc.

[0107] Therefore, by integrating the rotation point output model into the complete task model, it can be more widely applied to various tasks.

[0108] To facilitate understanding, we can use a specific scenario to fully illustrate the process of training the rotation point output model and processing cartoon images.

[0109] For this, please refer to Figure 3a and Figure 3b .

[0110] Figure 3a A flowchart of a process 300a for training a rotation point output model implemented in a specific application scenario provided in an embodiment of the present application.

[0111] Figure 3bA flowchart of a process 300b of processing a cartoon image implemented in a specific application scenario provided by an embodiment of the present application.

[0112] For example, in process 300a, a "server" (for example, a server used by a social platform that can provide image and picture sharing services) can be used as an execution entity (the "server" is not directly shown in the figure) to execute the process of training the rotating point output model (for example, training to obtain the rotating point output model 311').

[0113] Specifically, the execution subject may perform S301 based on the real face image 303 to determine the rotation point. For example, the rotation points 311-315 for the real face 302 may be determined.

[0114] Then, the execution subject may execute S302 to process the real face image 303 and generate a corresponding cartoon face image 305 .

[0115] It should be understood that the execution order of S301 and S302 is only for illustrative purposes and is not intended to limit the execution order of the two. For example, in other scenarios, the execution subject may first select to generate the cartoon face image 305 and then determine the rotation point.

[0116] Next, the execution subject may execute S303 to train the initial rotation point output model 311 using the cartoon face image 305 as input and the rotation points 311 - 315 as output.

[0117] Subsequently, when the training (of the initial rotation point output model 311) meets the training requirements, the execution subject may complete and stop the training to obtain the rotation point output model 311'.

[0118] Next, you can further refer to Figure 3b In process 300b, after receiving the cartoon image 307 (for example, it may be a cartoon image uploaded and submitted by a user to a server through a terminal device in the hope of being displayed and published on the platform), the execution subject may call the rotation point output model 311' trained based on the above process 300a to process the cartoon image 307 and output the rotation points included therein (for example, the rotation point output model 311' can determine the rotation points 321-325 included therein by processing the cartoon image 307).

[0119] Exemplarily, the number of rotation points (e.g., rotation points 321-325) determined by the rotation point output model 311' is greater than or equal to the number threshold. Accordingly, the execution subject can respond to this by executing S304 to rotate the cartoon face 308 in the cartoon image 307 using the rotation points 321-325 (i.e., adjust and rotate the cartoon face 308 from the presentation posture in the cartoon image 307 to the target presentation posture) to obtain a processed cartoon image 307'.

[0120] Subsequently, the execution subject may execute S305 to determine the number of cartoon face features hit by the cartoon image 307 ′ and located in the feature database 335 .

[0121] Exemplarily, the number of cartoon face features located in the feature database 335 hit by the cartoon image 307 ′ is greater than or equal to a number threshold.

[0122] Accordingly, the execution subject may execute S306 in response to this to classify the cartoon image 307 into the target classification (ie, the classification corresponding to the feature database 335 ).

[0123] The present application also provides a device for training a rotation point output model. The structure of the device is as follows: Figure 4 The device 400 shown. The device 400 includes: a cartoon image generating unit 410, configured to generate a cartoon face image based on a real face image; an output model training unit 420, configured to take the cartoon face image as input and the rotation point determined based on the real face image as output, and train the initial rotation point output model; an output model determining unit 430, configured to obtain the rotation point output model in response to determining that the training of the initial rotation point output model meets the predetermined training requirements.

[0124] In some embodiments, the device 400 also includes: an identification frame determination unit, configured to determine a face identification frame based on a real face image; and an output model training unit 420, further configured to train the initial rotation point output model with the cartoon face image as input and the mapping result of the rotation point and the face identification frame determined based on the real face image in the cartoon face image as output.

[0125] In some embodiments, the device 400 also includes: a candidate real face image acquisition unit, configured to acquire a group of candidate real face images; a real face image determination unit, configured to determine a target candidate real face image as a real face image in response to the fact that the angle difference between the presentation posture of the real face in the target candidate real face image and the target presentation posture in a group of candidate real face images is greater than or equal to a difference threshold.

[0126] In some embodiments, the device 400 also includes: an enhanced cartoon face image generation unit, configured to process the cartoon face image by sample enhancement to obtain a set of enhanced cartoon face images; and an output model training unit 420, further configured to train the initial rotation point output model with a set of enhanced cartoon face images as input and a rotation point determined based on a real face image as output.

[0127] In some embodiments, the initial rotation point output model is pre-trained with a sample real face image as input and a sample rotation point corresponding to the sample real face image as output.

[0128] In some embodiments, the cartoon image generation unit 410 is further configured to generate a cartoon face image based on a real face image using a diffusion model with an additional control module.

[0129] The present application also provides a device for processing cartoon images, the structure of which is as follows: Figure 5 The device 500 shown. The device 500 includes: a rotation point acquisition unit 510, configured to process the to-be-processed cartoon image using a rotation point output model to obtain a set of rotation points, wherein the rotation points are used to rotate the presentation posture of the cartoon face in the to-be-processed cartoon image to a target presentation posture, wherein the rotation point output model is trained by the device 400 described above; a cartoon image processing unit 520, configured to rotate the cartoon face using a set of rotation points in response to the number of rotation points included in the set of rotation points being greater than or equal to a quantity threshold, to obtain a processed cartoon image.

[0130] In some embodiments, the apparatus 500 further includes: a cartoon image classification unit configured to classify the processed cartoon image into a target classification in response to the number of cartoon face features located in a feature database hit by the processed cartoon image being greater than or equal to a quantity threshold.

[0131] Based on the same inventive concept, an electronic device, a readable storage medium and a computer program product are also provided in an embodiment of the present application. The method corresponding to the electronic device may be the method for training a rotation point output model and processing a cartoon image in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by at least one processor, and the instructions are executed by at least one processor so that at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0132] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0133] Figure 6 The structure of an electronic device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the electronic device 600 includes a central processing unit (CPU, Central Processing Unit) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 602 or the program loaded from the storage part 608 to the random access memory (RAM, Random Access Memory) 603. In RAM 603, various programs and data required for system operation are also stored. CPU 601, ROM 602 and RAM 603 are connected to each other through bus 604. Input / output (I / O, Input / Output) interface 605 is also connected to bus 604.

[0134] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 608 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet.

[0135] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 601, the above functions defined in the method of the present application are executed.

[0136] Another embodiment of the present application further provides a computer-readable storage medium and a computer program product, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application.

[0137] Specifically, the present embodiment can adopt any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, systems, devices or devices including but not limited to electricity, magnetism, light, electromagnetic, infrared, or semiconductors, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0138] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0139] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0140] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0143] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules and units is only a logical function division. There may be other division methods in actual implementation. For example, with units as an example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0144] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0145] In addition, each functional module or unit in each embodiment of the present application may be integrated into one processing module or unit, or each module or unit may exist physically separately, or two or more units may be integrated into one module or unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules or units.

[0146] The above-mentioned integrated modules and units implemented in the form of software function modules and units can be stored in a computer-readable storage medium. The above-mentioned software function modules and units are stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform some steps of the methods of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0148] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A method for training a rotation point output model, characterized in that: include: Generate cartoon face images based on real face images; Taking the cartoon face image as input and taking the rotation point determined based on the real face image as output, training an initial rotation point output model, wherein the rotation point is used to rotate the presentation posture of the real face in the real face image to a target presentation posture; In response to determining that the training of the initial rotation point output model meets a predetermined training requirement, a rotation point output model is obtained.

2. The method according to claim 1, characterized in that Also includes: Determine a face identification frame based on the real face image; and The method of taking the cartoon face image as input and the rotation point determined based on the real face image as output, training the initial rotation point output model, comprises: The cartoon face image is used as input, and the mapping result of the rotation point determined based on the real face image and the face identification frame in the cartoon face image is used as output to train the initial rotation point output model.

3. The method according to claim 1, characterized in that: Also includes: Obtain a set of candidate real face images; In response to the fact that, in the group of candidate real human face images, the angle difference between the presentation posture of the real human face in the target candidate real human face image and the target presentation posture is greater than or equal to a difference threshold, the target candidate real human face image is determined as the real human face image.

4. The method according to claim 1, characterized in that: Also includes: Processing the cartoon face images by sample enhancement to obtain a set of enhanced cartoon face images; as well as Taking the cartoon face image as input and the rotation point determined based on the real face image as output, training an initial rotation point output model includes: The initial rotation point output model is trained by taking the set of enhanced cartoon face images as input and the rotation points determined based on the real face images as output.

5. The method according to claim 4, characterized in that The initial rotation point output model is pre-trained with a sample real face image as input and a sample rotation point corresponding to the sample real face image as output.

6. The method according to any one of claims 1 to 5, characterized in that The method of generating a cartoon face image based on a real face image comprises: Cartoon face images are generated based on real face images using a diffusion model with an additional control module.

7. A method for processing cartoon images, characterized in that: include: Processing the cartoon image to be processed using a rotation point output model to obtain a set of rotation points, wherein the rotation points are used to rotate the presentation posture of the cartoon face in the cartoon image to be processed to a target presentation posture, wherein the rotation point output model is trained based on the method for training a rotation point output model according to any one of claims 1 to 6; In response to the number of rotation points included in the group of rotation points being greater than or equal to a number threshold, the cartoon face is rotated using the group of rotation points to obtain a processed cartoon image.

8. The method according to claim 7, characterized in that Also includes: In response to the number of cartoon face features located in a feature database hit by the processed cartoon image being greater than or equal to a quantity threshold, the processed cartoon image is classified as a target classification.

9. A device for training a rotation point output model, characterized in that: include: A cartoon image generating unit, configured to generate a cartoon face image based on a real face image; an output model training unit, configured to take the cartoon face image as input and the rotation point determined based on the real face image as output, and train an initial rotation point output model; The output model determination unit is configured to obtain a rotation point output model in response to determining that the training of the initial rotation point output model meets a predetermined training requirement.

10. A device for processing cartoon images, characterized in that: include: A rotation point acquisition unit, configured to process the cartoon image to be processed using a rotation point output model to obtain a set of rotation points, wherein the rotation points are used to rotate the presentation posture of the cartoon face in the cartoon image to be processed to a target presentation posture, wherein the rotation point output model is trained based on the device for training the rotation point output model according to claim 9; The cartoon image processing unit is configured to rotate the cartoon face using the set of rotation points in response to the number of rotation points included in the set of rotation points being greater than or equal to a number threshold, so as to obtain a processed cartoon image.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

12. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 8.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images