Online live streaming methods, devices, and electronic equipment based on image cartoonization
Through the method based on the generative adversarial network, the real-time and consistency problems of cartoon image generation in live broadcast scenes are solved, high-quality cartoon image generation is achieved, ensuring the consistency of the image in multiple dimensions, and solving the privacy protection problem.
Patent Information
- Application Number
- CN202011523785.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-22
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-12-22
AI Technical Summary
The existing cartoon image generation method cannot meet the problems of real-time generation and poor image color consistency in live broadcast scenes.
Using a method based on the generative adversarial network, an initial character cartoon avatar generation model is established, and the generator and discriminator are alternately trained, and the loss function is determined in combination with constraint rules to ensure the consistency of the generated cartoon images in dimensions such as color, position, texture, and edges. .
It achieves consistency in multiple dimensions, improves the quality of generated images, and can convert real face images into natural and similar cartoon avatars of characters in real time, solving the privacy protection problem.
Smart Images

Figure CN112561786B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of education, and is particularly applicable to the field of online education. More specifically, it relates to an online live broadcast method, device and electronic equipment based on image cartoonization. Background Art
[0002] With the development of the Internet, more and more online courses have emerged, and teachers imparting knowledge through online lectures or online classes has become an important way of learning.
[0003] However, in some existing online education systems, there is usually an interactive process between teachers and students during the specific course learning process. However, in related technologies, the face and each facial feature are located, and then the various organs of the face are transformed separately, and then spliced together. The above method has very high requirements on the stability of the detection model. For interactive scenes, the detection of the position also needs to be combined with manual annotation. Obviously, the above method cannot meet the live broadcast scene, and the generation process of generating and splicing separately is relatively costly. In addition, since there may be problems of incoordination between the separately generated facial features, there are also other problems such as how to splice the generated facial features together, the processing of the joints between the facial features and the face, and poor color consistency. Therefore, there is still a lot of room for improvement in how to generate more natural and realistic cartoon avatars.
[0004] Therefore, it is necessary to provide a more effective online live broadcast method that can perform image cartoon conversion more naturally and effectively. Summary of the invention
[0005] 1. Technical issues to be resolved
[0006] The present invention aims to solve the problems that the existing cartoon image generation method cannot meet the requirements of real-time generation in live broadcast scenes and the image color consistency is poor.
[0007] (II) Technical solution
[0008] To solve the above technical problems, one aspect of the present invention proposes an online live broadcast method based on image cartoonization, which is used to identify the original image in the live video stream in real time and cartoonize the face image in the original image. The method also includes: collecting a real face data set and a cartoon face data set to establish a training data set; based on a generative adversarial network, establishing an initial character cartoon avatar generation model, the initial character cartoon avatar generation model includes two groups of generators and discriminators, and the two groups of generators and discriminators are alternately trained using the training data set; according to constraint rules, a loss function is determined, and the alternating training is controlled using the loss function to obtain a final character cartoon avatar generation model; the original image to be converted in the current live video stream is obtained, and the face image in the original image is converted into a cartoon face image using the final character cartoon avatar generation model.
[0009] According to a preferred embodiment of the present invention, it also includes: pre-setting constraint rules, the constraint rules include setting consistency parameters and minimizing the loss function, the consistency parameters include at least two of color consistency, position consistency, texture consistency, and edge information consistency; the constraint rules also include judging the consistency of the facial skin color, hair color, and background color of the real face image within a specific scaling size range and the generated character cartoon avatar.
[0010] According to a preferred embodiment of the present invention, determining the loss function includes: selecting a consistency parameter to determine the loss function, and the determined loss function is:
[0011]
[0012] Among them, G is the first generator; F is the second generator; X is the real face image in the training sample, and Y is the cartoon face avatar in the training sample; is the first discriminator; is the second discriminator; , , , , , It is a configurable parameter; is the loss value of the discriminator; is the circulation loss value; It is the color loss value between the user's real face image and the facial skin color, hair color, and background color in the generated character cartoon image; It is the loss value between the user's real face image and the generated character cartoon image in terms of hair, facial contour and / or facial features; It represents the loss value of the avatar attributes between the user's real face image and the generated character cartoon image.
[0013] According to a preferred embodiment of the present invention, it also includes calculating , The calculation expression is as follows:
[0014]
[0015] in, It is the loss value between the user's real face image and the generated character cartoon image in terms of avatar attributes; It is used to calculate the avatar attribute value of the user's real face image; It is used to calculate the avatar attribute value of the generated user's character cartoon avatar; Is used to indicate taking and The mean of the Euclidean distances; It represents calculating and generating the Euclidean distance of the avatar attribute feature vectors before and after; constructing a first calculation model and a second calculation model, wherein the first calculation model is used to calculate the avatar attribute value of a real face image, and the second calculation model is used to calculate the avatar attribute value of a character cartoon avatar.
[0016] According to a preferred embodiment of the present invention, it also includes calculating , The calculation expression is as follows:
[0017]
[0018] in, It is the loss value between the user's real face image and the generated character cartoon image in terms of hair, facial contour and facial features. It is the feature value representing the edge extraction using the Candy operator on the user's real face image; is a feature value representing edge extraction using the Candy operator for the generated user's character cartoon avatar; Yes means The mean of Yes means and The Euclidean distance between .
[0019] According to a preferred embodiment of the present invention, it also includes calculating , The calculation expression is as follows:
[0020]
[0021] in, It is the color loss value between the user's real face image and the facial skin color, hair color, and background color in the generated character cartoon image; It means scaling the user's real face image to a color value within a specific range; It is a color value representing that the generated user's character cartoon avatar is within the specific range; Yes means The mean of Yes means and The Euclidean distance between .
[0022] According to a preferred embodiment of the present invention, it also includes: using the MobileNetV2 method to determine the position of the real face image contained in the original image in the historical video stream, and intercepting the real face image to establish a real face data set.
[0023] According to a preferred embodiment of the present invention, it also includes: using the MobileNetV2 method to determine the position of the real face image contained in the original image in the current live video stream, and intercepting a specific area corresponding to the real face image for replacement with a cartoon avatar.
[0024] According to a preferred embodiment of the present invention, it further includes: detecting the real face images and character cartoon avatars in the real face data set and the cartoon face data set, and extracting avatar attributes from the real face images and the character cartoon avatars respectively to determine the avatar attribute feature vectors before and after generating the cartoon avatar, and establishing training data sets for the first model and the second model; the avatar attributes include hair length, hair color, face shape, eye size and position, nose size and position, and whether there are facial expressions.
[0025] The second aspect of the present invention proposes an online live broadcast device based on image cartoonization, the device comprising: a collection module, used to collect a real face data set and a cartoon face data set to establish a training data set; an establishment module, based on a generative adversarial network, to establish an initial character cartoon avatar generation model, the initial character cartoon avatar generation model comprises two groups of generators and discriminators, and the two groups of generators and discriminators are alternately trained using the training data set; a control module, according to constraint rules, determines a loss function, and uses the loss function to control the alternating training to obtain a final character cartoon avatar generation model; a conversion module, used to obtain an original image to be converted in a current live video stream, and use the final character cartoon avatar generation model to convert the face image in the original image into a cartoon face image.
[0026] According to a preferred embodiment of the present invention, it also includes: a setting module for presetting constraint rules, the constraint rules include setting consistency parameters and minimizing the loss function, the consistency parameters include at least two of color consistency, position consistency, texture consistency, and edge information consistency; the constraint rules also include judging the consistency between the real human face image within a specific scaling size range and the facial skin color, hair color, and background color in the generated character cartoon avatar.
[0027] According to a preferred embodiment of the present invention, determining the loss function includes: selecting a consistency parameter to determine the loss function, and the determined loss function is:
[0028]
[0029] Among them, G is the first generator; F is the second generator; X is the real face image in the training sample, and Y is the cartoon face avatar in the training sample; is the first discriminator; is the second discriminator; , , , , , It is a configurable parameter; is the loss value of the discriminator; is the circulation loss value; It is the color loss value between the user's real face image and the facial skin color, hair color, and background color in the generated character cartoon image; It is the loss value between the user's real face image and the generated character cartoon image in terms of hair, facial contour and / or facial features; It represents the loss value of the avatar attributes between the user's real face image and the generated character cartoon image.
[0030] According to a preferred embodiment of the present invention, a calculation module is further included, wherein the calculation module is used to calculate , The calculation expression is as follows:
[0031]
[0032] in, It is the loss value between the user's real face image and the generated character cartoon image in terms of avatar attributes; It is used to calculate the avatar attribute value of the user's real face image; It is used to calculate the avatar attribute value of the generated user's character cartoon avatar; Is used to indicate taking and The mean of the Euclidean distances; It represents calculating and generating the Euclidean distance of the avatar attribute feature vectors before and after; constructing a first calculation model and a second calculation model, wherein the first calculation model is used to calculate the avatar attribute value of a real face image, and the second calculation model is used to calculate the avatar attribute value of a character cartoon avatar.
[0033] According to a preferred embodiment of the present invention, the calculation module is also used to calculate , The calculation expression is as follows:
[0034]
[0035] in, It is the loss value between the user's real face image and the generated character cartoon image in terms of hair, facial contour and facial features. It is the feature value representing the edge extraction using the Candy operator on the user's real face image; is a feature value representing edge extraction using the Candy operator for the generated user's character cartoon avatar; Yes means The mean of Yes means and The Euclidean distance between .
[0036] According to a preferred embodiment of the present invention, the calculation module is also used to calculate , The calculation expression is as follows:
[0037]
[0038] in, It is the color loss value between the user's real face image and the facial skin color, hair color, and background color in the generated character cartoon image; It means scaling the user's real face image to a color value within a specific range; It is a color value representing that the generated user's character cartoon avatar is within the specific range; Yes means The mean of Yes means and The Euclidean distance between .
[0039] According to a preferred embodiment of the present invention, it also includes: using the MobileNetV2 method to determine the position of the real face image contained in the original image in the historical video stream, and intercepting the real face image to establish a real face data set.
[0040] According to a preferred embodiment of the present invention, it also includes: using the MobileNetV2 method to determine the position of the real face image contained in the original image in the current live video stream, and intercepting a specific area corresponding to the real face image for replacement with a cartoon avatar.
[0041] According to a preferred embodiment of the present invention, it further includes: detecting the real face images and character cartoon avatars in the real face data set and the cartoon face data set, and extracting avatar attributes from the real face images and the character cartoon avatars respectively to determine the avatar attribute feature vectors before and after generating the cartoon avatar, and establishing training data sets for the first model and the second model; the avatar attributes include hair length, hair color, face shape, eye size and position, nose size and position, and whether there are facial expressions.
[0042] A third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory is used to store a computer executable program, and when the computer program is executed by the processor, the processor executes the online live broadcast method based on image cartoonization.
[0043] A fourth aspect of the present invention provides a computer-readable medium storing a computer-executable program, which, when executed, implements the online live broadcast method based on image cartoonization.
[0044] (III) Beneficial effects
[0045] Compared with the prior art, the present invention selects consistency parameters and redefines the loss function to control alternating training, thereby ensuring the consistency of images before and after the generation of character cartoon images in multiple dimensions such as color, position, edge, texture, and avatar attributes, and can further improve the quality of the generated image; it can convert real face images into similar character cartoon avatars in real time, preserve facial features to a certain extent, generate more natural character cartoon images, and solve the privacy issues of teachers or students. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 is a flowchart of an example of an online live broadcast method based on image cartoonization according to embodiment 1 of the present invention;
[0047] Figure 2 is a flowchart of another example of the online live broadcast method based on image cartoonization according to Embodiment 1 of the present invention;
[0048] Figure 3 is a flowchart of another example of the online live broadcast method based on image cartoonization according to Embodiment 1 of the present invention;
[0049] Figure 4 is a schematic diagram of an example of an online live broadcast device based on image cartoonization according to Embodiment 2 of the present invention;
[0050] Figure 5 is a schematic diagram of another example of an online live broadcast device based on image cartoonization according to Embodiment 2 of the present invention;
[0051] Figure 6 is a schematic diagram of another example of an online live broadcast device based on image cartoonization according to Embodiment 2 of the present invention;
[0052] Figure 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;
[0053] Figure 8 is a schematic diagram of a computer-readable recording medium according to an embodiment of the present invention. DETAILED DESCRIPTION
[0054] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.
[0055] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to the actual situation without departing from the main purpose of the present invention.
[0056] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processing unit devices and / or microcontroller devices.
[0057] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.
[0058] In order to solve the problems that the existing cartoon image generation methods cannot meet the real-time generation requirements in live broadcast scenes and the image color consistency is poor, the present invention provides an online live broadcast method based on image cartoonization, which can convert the real face image of a teacher or student into a cartoon avatar in real time during the live broadcast process and can effectively protect the privacy information of the teacher or student.
[0059] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.
[0060] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art may implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.
[0061] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to the actual situation without departing from the main purpose of the present invention.
[0062] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processing unit devices and / or microcontroller devices.
[0063] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.
[0064] Figure 1 It is a flowchart of an example of an online live broadcast method based on image cartoonization according to embodiment 1 of the present invention.
[0065] like Figure 1 As shown, the online live broadcast method includes the following steps:
[0066] Step S101, collect a real face data set and a cartoon face data set to establish a training data set.
[0067] Step S102, based on the generative adversarial network, establish an initial character cartoon avatar generation model, the initial character cartoon avatar generation model includes two groups of generators and discriminators, and use the training data set to alternately train the two groups of generators and discriminators.
[0068] Step S103, determining a loss function according to the constraint rules, and using the loss function to control the alternating training to obtain a final character cartoon avatar generation model.
[0069] Step S104, obtaining the original image to be converted in the current live video stream, and using the final character cartoon avatar generation model to convert the face image in the original image into a cartoon face image.
[0070] In this example, the online live broadcast method of the present invention is applied to the online live broadcast scenario of an online education APP. The local video stream is obtained from the mobile terminal and the real face is detected in real time as the original image for cartoon drawing. In this way, the real face image of the teacher or student during the live broadcast can be converted into a cartoon avatar in real time, and the privacy information of the teacher or student can be effectively protected.
[0071] It should be noted that the above is only described as a preferred example and cannot be understood as a limitation of the present invention. In other examples, a video stream can be obtained from a remote end and the locally obtained video stream can be sent to the remote end for cartoon conversion, wherein the remote end can be a server or another client.
[0072] First, in step S101, a real face dataset and a cartoon face dataset are collected to establish a training dataset.
[0073] In this example, real face images are obtained from historical video streams of online education to build a real face dataset.
[0074] Preferably, the MobileNetV2 method is used to determine the position of the real face image contained in the original image in the historical video stream, and the real face image is captured to establish a real face dataset.
[0075] For example, the original image in the historical video stream is input into the MobileNetV2 model, and an N*4 matrix is output, where N represents the number of detected faces, and a position vector is output for each face, for example, [x1, y1, x2, y2], where x1, y1, x2, and y2 respectively represent the position coordinates of the rectangular capture box that captures the real face image in the original image.
[0076] It should be noted that the above position vector may vary according to the capture frame and the capture method. The above is only described as an example and should not be construed as a limitation to the present invention.
[0077] Preferably, the real face images and character cartoon avatars in the real face data set and the cartoon face data set are detected, and avatar attributes are extracted from the real face images and the character cartoon avatars respectively to determine the avatar attribute feature vectors before and after generating the cartoon avatars.
[0078] Specifically, the avatar attributes include hair length, hair color, face shape, eye size and position, nose size and position, and whether there are facial expressions.
[0079] In addition, we crawled and collected cartoon portraits of people from the Internet to build a cartoon face dataset.
[0080] Furthermore, a training data set is established based on the real face data set and the cartoon face data set for training the model.
[0081] It should be noted that the above description is only given as an example and should not be construed as a limitation to the present invention.
[0082] Next, in step S102, based on the generative adversarial network, an initial character cartoon avatar generation model is established, and the initial character cartoon avatar generation model includes two groups of generators and discriminators, and the two groups of generators and discriminators are alternately trained using the training data set.
[0083] In this example, based on the CycleGAN method, a generative adversarial network is constructed to establish an initial character cartoon avatar generation model. The generative adversarial network includes two generators and two discriminators, specifically including a first generator, a first discriminator, a second generator and a second discriminator, which respectively realize the conversion of real face images to character cartoon avatars and the conversion of character cartoon avatars to real face images.
[0084] Specifically, a generator G (first generator) is trained using the corresponding data set in the training data set (including the real face data set) to generate a real face image x into a character cartoon avatar y. In addition, a discriminator DY is trained using the corresponding data set in the training data set (including the cartoon face data set) to discriminate whether a character cartoon avatar is a real picture or generated by the first generator G.
[0085] Furthermore, a generator F (second generator) is trained to generate a cartoon avatar y into a real face image x' (i.e., F(G(x))). In addition, a discriminator DX (second discriminator) is trained to discriminate whether a "real face image" is a real picture or generated by the second generator F. Thus, the character cartoon avatar and the real face image (i.e., the original image) are closer.
[0086] It should be noted that the loss functions of the generator and discriminator are the same as those of GAN. The discriminator D tries its best to detect the fake pictures (or fake images) generated by the generator G, and the generator tries its best to generate pictures to deceive the discriminator. In addition, using the CycleGAN method, there is no need to establish a one-to-one mapping between the source domain and the target domain.
[0087] Next, in step S103, a loss function is determined according to the constraint rules, and the alternating training is controlled using the loss function to obtain a final character cartoon avatar generation model.
[0088] like Figure 2 As shown, the step S201 of presetting constraint rules is also included.
[0089] In step S201, constraint rules are preset to control the alternating training, thereby ensuring that a more realistic and accurate character cartoon avatar is generated.
[0090] Specifically, the constraint rules include setting consistency parameters and minimizing the loss function, and the consistency parameters include at least two of color consistency, position consistency, texture consistency, edge information consistency, and avatar attribute consistency.
[0091] Furthermore, the constraint rules also include judging the consistency of the facial skin color, hair color, and background color between the real human face image within a specific zoom size range and the generated character cartoon avatar.
[0092] Preferably, a consistency parameter is selected to determine the loss function, thereby being able to control the alternating training and further ensuring the consistency of the images before and after the character cartoon avatar is generated in multiple dimensions such as color, position, texture, and edge to ensure the quality of image generation.
[0093] In one example, color consistency, position consistency, and avatar attribute consistency are selected to determine the loss function, and the loss function is:
[0094]
[0095] Among them, G is the first generator; F is the second generator; X is the real face image in the training sample, and Y is the cartoon face avatar in the training sample; is the first discriminator; is the second discriminator; , , , , , It is a configurable parameter; is the loss value of the discriminator; is the circulation loss value; It is the color loss value between the user's real face image and the facial skin color, hair color, and background color in the generated character cartoon image; It is the loss value between the user's real face image and the generated character cartoon image in terms of hair, facial contour and / or facial features; It represents the loss value of the avatar attributes between the user's real face image and the generated character cartoon image.
[0096] Specifically, it also includes calculation , The calculation expression is as follows:
[0097]
[0098] in, It is the loss value between the user's real face image and the generated character cartoon image in terms of avatar attributes; It is used to calculate the avatar attribute value of the user's real face image; It is used to calculate the avatar attribute value of the generated user's character cartoon avatar; Is used to indicate taking and The mean of the Euclidean distances; Indicates the Euclidean distance between the attribute feature vectors of the avatars before and after calculation.
[0099] In this example, the first calculation model is constructed And the second calculation model , the first calculation model For calculating the avatar attribute value of a real face image, the second calculation model Used to calculate the avatar attribute value of the character cartoon avatar, so that it can be calculated more accurately .
[0100] Preferably, the avatar attributes include hair length, hair color, face shape, eye size and position, nose size and position, and whether there are facial expressions.
[0101] Specifically, the real face images and character cartoon avatars in the real face data set and the cartoon face data set are detected, and avatar attributes are extracted from the real face images and the character cartoon avatars respectively to determine the avatar attribute feature vectors before and after generating the cartoon avatars.
[0102] More specifically, based on the above avatar attributes, the avatars used for training the model are selected from the training data. and The sub-training dataset is The sub-training dataset includes real face images (i.e., input features) and vectors of specific dimensions; the training model The sub-training dataset includes cartoon avatars of characters (i.e., input features) and vectors of specific dimensions.
[0103] For example, a real face image is input and a feature vector of 20 to 100 dimensions is output, preferably a 36-dimensional feature vector.
[0104] In this way, the avatar attributes can be supervised and the consistency of the generated character cartoon avatar and the original image in avatar attributes can be ensured.
[0105] Furthermore, it also includes calculation , The calculation expression is as follows:
[0106]
[0107] in, It is the loss value between the user's real face image and the generated character cartoon image in terms of hair, facial contour and facial features. It is the feature value representing the edge extraction using the Candy operator on the user's real face image; is a feature value representing edge extraction using the Candy operator for the generated user's character cartoon avatar; Yes means The mean of Yes means and The Euclidean distance between .
[0108] In this way, supervision of hair, facial contour, position of facial features, etc. can be achieved, and the consistency of the generated character cartoon avatar and the original image in terms of hair, facial contour, position of facial features, etc. can be guaranteed.
[0109] Preferably, it also includes calculating , The calculation expression is as follows:
[0110]
[0111] in, It is the color loss value between the user's real face image and the facial skin color, hair color, and background color in the generated character cartoon image; It means scaling the user's real face image to a color value within a specific range; It is a color value representing that the generated user's character cartoon avatar is within the specific range; Yes means The mean of Yes means and The Euclidean distance between .
[0112] In this example, Represents scaling image x to an image within a specific size range (such as a small 16*16 image). Therefore, use It can measure the consistency between the pixel value color of the generated cartoon avatar and the original image.
[0113] In this way, color supervision can be achieved, and the consistency of the generated character cartoon avatar and the original image in terms of facial skin color, hair color, background color, etc. can be guaranteed.
[0114] In another example, color consistency, position consistency, avatar attribute consistency, and edge information consistency are selected to determine the loss function, and the loss function is:
[0115]
[0116] Among them, G is the first generator; F is the second generator; X is the real face image in the training sample, and Y is the cartoon face avatar in the training sample; is the first discriminator; is the second discriminator; , , , , , , It is a configurable parameter; is the loss value of the discriminator; is the circulation loss value; It is the color loss value between the user's real face image and the facial skin color, hair color, and background color in the generated character cartoon image; It is the loss value between the user's real face image and the generated character cartoon image in terms of hair, facial contour and facial features. It is the loss value between the user's real face image and the generated character cartoon image in terms of avatar attributes; It is the loss value in color and shape between the user's real face image and the generated character cartoon image.
[0117] It should be noted that is a loss parameter that measures the consistency of attributes such as color and shape. Add , which can make the cartoon avatar generation result more realistic. For example, by adding an additional classification module inside the network, the consistency of the feature map before the classification layer can be used to measure the consistency of the overall style between the real face image (or original image) and the generated cartoon avatar, which is specifically reflected in different aspects such as color and shape. Therefore, it is possible to achieve supervision of shape and color, and further ensure the consistency of the shape and color between the generated character cartoon avatar and the original image.
[0118] Furthermore, the alternating training is controlled using the loss function determined above to obtain a final character cartoon head portrait generation model. Thus, by selecting a consistency parameter and re-determining the loss function to control the alternating training, the consistency of the image before and after the character cartoon image is generated in multiple dimensions such as color, position, edge, texture, and head portrait attributes can be ensured, and the quality of the generated image can be further improved.
[0119] It should be noted that the above description is only given as a preferred example and should not be construed as a limitation to the present invention.
[0120] Next, in step S104, the original image to be converted in the current live video stream is obtained, and the face image in the original image is converted into a cartoon face image using the final character cartoon avatar generation model.
[0121] like Figure 3As shown, the method further includes step S301 of detecting the position of the real face image contained in the original image in the current live video stream.
[0122] In step S301, the position of the real face image contained in the original image in the current live video stream is detected to determine the area of the real image to be captured.
[0123] Specifically, the MobileNetV2 method is used to determine the position of the real face image contained in the original image in the current live video stream, and to capture a specific area corresponding to the real face image for replacement with a cartoon avatar.
[0124] Furthermore, the final character cartoon avatar generation model in step S103 is used to input the original image to be converted of the current user, and the character cartoon avatar of the current user is output.
[0125] It should be noted that the above is only described as a preferred example and should not be understood as a limitation to the present invention.
[0126] Compared with the prior art, the present invention selects consistency parameters and redefines the loss function to control alternating training, thereby ensuring the consistency of images before and after the generation of character cartoon images in multiple dimensions such as color, position, edge, texture, and avatar attributes, and can further improve the quality of the generated image; it can convert real face images into similar character cartoon avatars in real time, preserve facial features to a certain extent, generate more natural character cartoon images, and solve the privacy issues of teachers or students.
[0127] Example 2
[0128] The following describes an apparatus embodiment of the present invention, which can be used to execute a method embodiment of the present invention. The details described in the apparatus embodiment of the present invention should be regarded as supplementary to the above method embodiment; details not disclosed in the apparatus embodiment of the present invention can be implemented with reference to the above method embodiment.
[0129] Reference Figures 4 to 6 , an online live broadcast device 400 based on image cartoonization according to embodiment 2 of the present invention will be described.
[0130] According to the second aspect of the present invention, the present invention also provides an online live broadcast device 400 based on image cartoonization, and the device 400 includes: a collection module 401, which is used to collect a real face data set and a cartoon face data set to establish a training data set; an establishment module 402, which is based on a generative adversarial network to establish an initial character cartoon avatar generation model, and the initial character cartoon avatar generation model includes two groups of generators and discriminators, and the two groups of generators and discriminators are alternately trained using the training data set; a control module 403, which determines a loss function according to constraint rules, and uses the loss function to control the alternating training to obtain a final character cartoon avatar generation model; a conversion module 404, which is used to obtain an original image to be converted in the current live video stream, and use the final character cartoon avatar generation model to convert the face image in the original image into a cartoon face image.
[0131] like Figure 5 As shown, it also includes: a setting module 501, which is used to preset constraint rules, and the constraint rules include setting consistency parameters and minimizing the loss function, and the consistency parameters include at least two of color consistency, position consistency, texture consistency, and edge information consistency; the constraint rules also include judging the consistency between the real face image within a specific zoom size range and the facial skin color, hair color, and background color in the generated character cartoon avatar.
[0132] Preferably, determining the loss function comprises: selecting a consistency parameter to determine the loss function, and the determined loss function is:
[0133]
[0134] Among them, G is the first generator; F is the second generator; X is the real face image in the training sample, and Y is the cartoon face avatar in the training sample; is the first discriminator; is the second discriminator; , , , , , It is a configurable parameter; is the loss value of the discriminator; is the circulation loss value; It is the color loss value between the user's real face image and the facial skin color, hair color, and background color in the generated character cartoon image; It is the loss value between the user's real face image and the generated character cartoon image in terms of hair, facial contour and / or facial features; It represents the loss value of the avatar attributes between the user's real face image and the generated character cartoon image.
[0135] like Figure 6 As shown, it also includes a calculation module 601, the calculation module 601 is used to calculate , The calculation expression is as follows:
[0136]
[0137] in, It is the loss value between the user's real face image and the generated character cartoon image in terms of avatar attributes; It is used to calculate the avatar attribute value of the user's real face image; It is used to calculate the avatar attribute value of the generated user's character cartoon avatar; Is used to indicate taking and The mean of the Euclidean distances; It represents calculating and generating the Euclidean distance of the avatar attribute feature vectors before and after; constructing a first calculation model and a second calculation model, wherein the first calculation model is used to calculate the avatar attribute value of a real face image, and the second calculation model is used to calculate the avatar attribute value of a character cartoon avatar.
[0138] Preferably, the calculation module 601 is also used to calculate , The calculation expression is as follows:
[0139]
[0140] in, It is the loss value between the user's real face image and the generated character cartoon image in terms of hair, facial contour and facial features. It is the feature value representing the edge extraction using the Candy operator on the user's real face image; is a feature value representing edge extraction using the Candy operator for the generated user's character cartoon avatar; Yes means The mean of Yes means and The Euclidean distance between .
[0141] Preferably, the calculation module 601 is also used to calculate , The calculation expression is as follows:
[0142]
[0143] in, It is the color loss value between the user's real face image and the facial skin color, hair color, and background color in the generated character cartoon image; It means scaling the user's real face image to a color value within a specific range; It is a color value representing that the generated user's character cartoon avatar is within the specific range; Yes means The mean of Yes means and The Euclidean distance between .
[0144] Preferably, it also includes: using the MobileNetV2 method to determine the position of the real face image contained in the original image in the historical video stream, and intercepting the real face image to establish a real face data set.
[0145] Preferably, it also includes: using the MobileNetV2 method to determine the position of the real face image contained in the original image in the current live video stream, and intercepting a specific area corresponding to the real face image for replacement with a cartoon avatar.
[0146] Preferably, it further includes: detecting the real face images and character cartoon avatars in the real face data set and the cartoon face data set, and extracting avatar attributes from the real face images and the character cartoon avatars respectively to determine the avatar attribute feature vectors before and after generating the cartoon avatars, and establishing training data sets for the first model and the second model; the avatar attributes include hair length, hair color, face shape, eye size and position, nose size and position, and whether there are facial expressions.
[0147] Compared with the prior art, the present invention selects consistency parameters and redefines the loss function to control alternating training, thereby ensuring the consistency of images before and after the generation of character cartoon images in multiple dimensions such as color, position, edge, texture, and avatar attributes, and can further improve the quality of the generated image; it can convert real face images into similar character cartoon avatars in real time, preserve facial features to a certain extent, generate more natural character cartoon images, and solve the privacy issues of teachers or students.
[0148] Example 3
[0149] The following describes an electronic device embodiment of the present invention, which can be regarded as a specific physical implementation of the method and device embodiments of the present invention. The details described in the electronic device embodiment of the present invention should be regarded as a supplement to the above method or device embodiments; details not disclosed in the electronic device embodiment of the present invention can be implemented with reference to the above method or device embodiments.
[0150] Figure 7 is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention, the electronic device includes a processor and a memory, the memory is used to store a computer executable program, when the computer program is executed by the processor, the processor executes Figure 1 method.
[0151] like Figure 7 As shown, the electronic device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.
[0152] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.
[0153] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).
[0154] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with an external device. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.
[0155] It should be understood that Figure 7 The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.
[0156] Figure 8 Schematic diagram of a computer readable recording medium according to an embodiment of the present invention. Figure 8As shown, a computer executable program is stored in a computer-readable recording medium, and when the computer executable program is executed, the above method of the present invention is implemented. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.
[0157] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0158] Through the above description of the implementation mode, it is easy for a person skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware. For computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), or it can be distributed and stored on the network, as long as it enables the electronic device to execute the method according to the present invention.
[0159] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or electronic device, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. An online live broadcast method based on image cartoonization, which is used to recognize the original image in the live video stream in real time and cartoonize the face image in the original image, characterized in that: The method further comprises: Collect real face datasets and cartoon face datasets to build training datasets; Based on a generative adversarial network, an initial character cartoon avatar generation model is established, wherein the initial character cartoon avatar generation model includes two groups of generators and discriminators, and the two groups of generators and discriminators are alternately trained using the training data set; Preset constraint rules, the constraint rules including: setting consistency parameters, minimizing the loss function, and judging the consistency of the facial skin color, hair color, and background color of the real face image within a specific zoom size range and the generated character cartoon avatar; According to the constraint rules, a consistency parameter is selected to determine a loss function, and the loss function is used to control the alternating training to obtain a final character cartoon avatar generation model; The loss function determined is: L total =λ1L GAN (G,D Y ,X,Y)+λ2L GAN (F,D X ,Y,X)+λ3L cycle +λ4L color +λ5L texture (G,X)+λ6L attribute (G,X) Among them, G is the first generator; F is the second generator; X is the real face image in the training sample, Y is the cartoon face avatar in the training sample; D X is the first discriminator; D Y is the second discriminator; λ1, λ2, λ3, λ4, λ5, λ6 are configurable parameters; L GAN is the loss value of the discriminator; L cycle is the circulation loss value; L color It is the color loss value between the user's real face image and the facial skin color, hair color, and background color in the generated character cartoon image; L texture L represents the loss value of the user's real face image and the generated character cartoon image in terms of hair, facial contour and / or facial features; attribute represents the loss value of the avatar attributes between the user's real face image and the generated character cartoon image; Calculate L attribute The calculation expression is as follows: Among them, L attribute A represents the loss value of the avatar attributes between the user's real face image and the generated character cartoon image; X It is used to calculate the avatar attribute value of the user's real face image; A Y It is used to calculate the avatar attribute value of the generated user's character cartoon avatar; It is used to indicate taking A X With A Y The mean of the Euclidean distances; ‖A X (x)-A Y (G(x))‖2 represents the Euclidean distance between the attribute feature vectors of the avatars before and after calculation; Calculate L texture The calculation expression is as follows: Among them, L texture It is the loss value of the hair, facial contour and facial features between the real face image of the user and the generated character cartoon image; Candy(x) is the feature value of edge extraction using the Candy operator on the real face image of the user; Candy(G(x)) is the feature value of edge extraction using the Candy operator on the generated character cartoon avatar of the user; is the mean of ‖Candy(x)-Candy(G(x))‖1; ‖Candy(x)-Candy(G(x))‖1 is the Euclidean distance between Candy(x) and Candy(G(x)); Calculate L color The calculation expression of (G,X) is as follows: Among them, L color is the color loss value between the user's real face image and the facial skin color, hair color, and background color in the generated character cartoon image; Re(x) is the color value representing scaling the user's real face image to a specific range; Re(G(x)) is the color value representing scaling the generated user's character cartoon avatar to within the specific range; is the mean of ‖Re(x)-Re(G(x))‖1; ‖Re(x)-Re(G(x))‖1 is the Euclidean distance between Re(x) and Re(G(x)); An original image to be converted in the current live video stream is obtained, and a face image in the original image is converted into a cartoon face image using the final character cartoon avatar generation model.
2. The method according to claim 1, characterized in that The consistency parameters include at least two of color consistency, position consistency, texture consistency, and edge information consistency.
3. The method according to claim 1, characterized in that Also includes: A first calculation model and a second calculation model are constructed, wherein the first calculation model is used to calculate the avatar attribute value of a real face image, and the second calculation model is used to calculate the avatar attribute value of a character cartoon avatar.
4. The method according to any one of claims 1 to 3, characterized in that: Also includes: Using the MobileNetV2 method, we determine the locations of real face images contained in the original images in the historical video streams, and capture the real face images to establish a real face dataset.
5. The method according to any one of claims 1 to 3, characterized in that: Also includes: Use the MobileNetV2 method to determine the position of the real face image contained in the original image in the current live video stream, and capture the specific area corresponding to the real face image to replace it with a cartoon avatar.
6. The method according to claim 4, characterized in that Also includes: Use the MobileNetV2 method to determine the position of the real face image contained in the original image in the current live video stream, and capture the specific area corresponding to the real face image to replace it with a cartoon avatar.
7. The method according to any one of claims 1 to 3, characterized in that: Further including: Detecting the real face images and the character cartoon avatars in the real face data set and the cartoon face data set, and extracting avatar attributes from the real face images and the character cartoon avatars respectively to determine the avatar attribute feature vectors before and after the cartoon avatars are generated, and establishing training data sets for the first model and the second model; The avatar attributes include hair length, hair color, face shape, eye size and position, nose size and position, and whether there are facial expressions or movements.
8. The method according to claim 4, characterized in that Further including: Detecting the real face images and the character cartoon avatars in the real face data set and the cartoon face data set, and extracting avatar attributes from the real face images and the character cartoon avatars respectively to determine the avatar attribute feature vectors before and after the cartoon avatars are generated, and establishing training data sets for the first model and the second model; The avatar attributes include hair length, hair color, face shape, eye size and position, nose size and position, and whether there are facial expressions or movements.
9. The method according to claim 5, characterized in that Further including: Detecting the real face images and the character cartoon avatars in the real face data set and the cartoon face data set, and extracting avatar attributes from the real face images and the character cartoon avatars respectively to determine the avatar attribute feature vectors before and after the cartoon avatars are generated, and establishing training data sets for the first model and the second model; The avatar attributes include hair length, hair color, face shape, eye size and position, nose size and position, and whether there are facial expressions or movements.
10. The method according to claim 6, characterized in that Further including: Detecting the real face images and the character cartoon avatars in the real face data set and the cartoon face data set, and extracting avatar attributes from the real face images and the character cartoon avatars respectively to determine the avatar attribute feature vectors before and after the cartoon avatars are generated, and establishing training data sets for the first model and the second model; The avatar attributes include hair length, hair color, face shape, eye size and position, nose size and position, and whether there are facial expressions or movements.
11. An online live broadcast device based on image cartoonization, characterized in that: The device comprises: The collection module is used to collect real face data sets and cartoon face data sets to establish training data sets; Establishing a module, based on a generative adversarial network, establishing an initial character cartoon avatar generation model, the initial character cartoon avatar generation model including two groups of generators and discriminators, and using the training data set to alternately train the two groups of generators and discriminators; A setting module is used to preset constraint rules, which include setting consistency parameters, minimizing the loss function, and judging the consistency of the facial skin color, hair color, and background color of the real face image within a specific zoom size range and the generated character cartoon avatar. A control module selects a consistency parameter to determine a loss function according to a constraint rule, and uses the loss function to control the alternating training to obtain a final character cartoon avatar generation model; The loss function determined is: L total =λ1l GAN (G,D Y ,X,Y)+λ2L GAN (F,D X ,Y,X)+λ3L cycle +λ4L color +λ5L texture (G,X)+λ6L attribute (G,X) Among them, G is the first generator; F is the second generator; X is the real face image in the training sample, Y is the cartoon face avatar in the training sample; D X is the first discriminator; D Y is the second discriminator; λ1, λ2, λ3, λ4, λ5, λ6 are configurable parameters; L GAN is the loss value of the discriminator; L cycle is the circulation loss value; L color It is the color loss value between the user's real face image and the facial skin color, hair color, and background color in the generated character cartoon image; L texture L represents the loss value of the user's real face image and the generated character cartoon image in terms of hair, facial contour and / or facial features; attribute represents the loss value of the avatar attributes between the user's real face image and the generated character cartoon image; Calculation module, used to calculate L attribute , L attribute The calculation expression is as follows: Among them, L attribute A represents the loss value of the avatar attributes between the user's real face image and the generated character cartoon image; X It is used to calculate the avatar attribute value of the user's real face image; A Y It is used to calculate the avatar attribute value of the generated user's character cartoon avatar; It is used to indicate taking A X With A Y The mean of the Euclidean distances; ‖A X (x)-A Y (G(x))‖2 represents the Euclidean distance between the attribute feature vectors of the avatars before and after calculation; Calculation module, used to calculate L texture , L texture The calculation expression is as follows: Among them, L texture It is the loss value of the hair, facial contour and facial features between the real face image of the user and the generated character cartoon image; Candy(x) is the feature value of edge extraction using the Candy operator on the real face image of the user; Candy(G(x)) is the feature value of edge extraction using the Candy operator on the generated character cartoon avatar of the user; is the mean of ‖Candy(x)-Candy(G(x))‖1; ‖Candy(x)-Candy(G(x))‖1 is the Euclidean distance between Candy(x) and Candy(G(x)); Calculation module, used to calculate L texture , L color The calculation expression of (G,X) is as follows: Among them, L color is the color loss value between the user's real face image and the facial skin color, hair color, and background color in the generated character cartoon image; Re(x) is the color value representing scaling the user's real face image to a specific range; Re(G(x)) is the color value representing scaling the generated user's character cartoon avatar to within the specific range; is the mean of ‖Re(x)-Re(G(x))‖1; ‖Re(x)-Re(G(x))‖1 is the Euclidean distance between Re(x) and Re(G(x)); The conversion module is used to obtain the original image to be converted in the current live video stream, and use the final character cartoon avatar generation model to convert the face image in the original image into a cartoon face image.
12. The device according to claim 11, characterized in that The consistency parameters include: at least two of: color consistency, position consistency, texture consistency, and edge information consistency.
13. The device according to claim 11, characterized in that Also includes A first calculation model and a second calculation model are constructed, wherein the first calculation model is used to calculate the avatar attribute value of a real face image, and the second calculation model is used to calculate the avatar attribute value of a character cartoon avatar.
14. The device according to any one of claims 11 to 13, characterized in that: Also includes: Using the MobileNetV2 method, we determine the locations of real face images contained in the original images in the historical video streams, and capture the real face images to establish a real face dataset.
15. The device according to any one of claims 11 to 13, characterized in that: Also includes: Use the MobileNetV2 method to determine the position of the real face image contained in the original image in the current live video stream, and capture the specific area corresponding to the real face image to replace it with a cartoon avatar.
16. The device according to claim 14, characterized in that Also includes: Use the MobileNetV2 method to determine the position of the real face image contained in the original image in the current live video stream, and capture the specific area corresponding to the real face image to replace it with a cartoon avatar.
17. The device according to any one of claims 11 to 13, characterized in that: Further including: Detecting the real face images and the character cartoon avatars in the real face data set and the cartoon face data set, and extracting avatar attributes from the real face images and the character cartoon avatars respectively to determine the avatar attribute feature vectors before and after the cartoon avatars are generated, and establishing training data sets for the first model and the second model; The avatar attributes include hair length, hair color, face shape, eye size and position, nose size and position, and whether there are facial expressions or movements.
18. The device according to claim 14, characterized in that Further including: Detecting the real face images and the character cartoon avatars in the real face data set and the cartoon face data set, and extracting avatar attributes from the real face images and the character cartoon avatars respectively to determine the avatar attribute feature vectors before and after the cartoon avatars are generated, and establishing training data sets for the first model and the second model; The avatar attributes include hair length, hair color, face shape, eye size and position, nose size and position, and whether there are facial expressions or movements.
19. The device according to claim 15, characterized in that Further including: Detecting the real face images and the character cartoon avatars in the real face data set and the cartoon face data set, and extracting avatar attributes from the real face images and the character cartoon avatars respectively to determine the avatar attribute feature vectors before and after the cartoon avatars are generated, and establishing training data sets for the first model and the second model; The avatar attributes include hair length, hair color, face shape, eye size and position, nose size and position, and whether there are facial expressions or movements.
20. The device according to claim 16, characterized in that Further including: Detecting the real face images and the character cartoon avatars in the real face data set and the cartoon face data set, and extracting avatar attributes from the real face images and the character cartoon avatars respectively to determine the avatar attribute feature vectors before and after the cartoon avatars are generated, and establishing training data sets for the first model and the second model; The avatar attributes include hair length, hair color, face shape, eye size and position, nose size and position, and whether there are facial expressions or movements.
21. An electronic device, comprising a processor and a memory, wherein the memory is used to store a computer executable program, wherein: When the computer executable program is executed by the processor, the processor performs the method according to any one of claims 1 to 10.
22. A computer-readable medium storing a computer-executable program, characterized in that: When the computer executable program is executed, the method according to any one of claims 1 to 10 is implemented.
