Method and device for automatically changing the eye features of people in a video
By applying transfer learning and deep learning algorithms to generate eye feature change models in videos, the high labor and time costs of existing technologies are solved, achieving efficient and natural automatic eye feature changes, thus improving the efficiency of film and television production and user experience.
Patent Information
- Application Number
- CN202310637176.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-05-31
AI Technical Summary
Existing methods for altering the eye features of people in videos require significant manpower and time, and can easily affect the viewing experience, making it difficult to achieve natural changes in eye features in film and television production.
By extracting target image data and change requirement types from video data, a target eye feature change model is generated using transfer learning. This model performs feature replacement and automatically changes eye features. By combining machine learning and deep learning algorithms such as the StarGANv2 model for training and transfer learning, automatic eye feature changes are achieved.
It improves the targeting and efficiency of eye feature alteration, reduces post-processing costs, enhances the user experience for video viewers, and achieves natural eye feature alteration effects.
Smart Images

Figure CN116824678B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of video data processing technology, and in particular to a method and apparatus for automatically changing the eye features of people in a video. Background Technology
[0002] In filmmaking and other video shooting scenarios, it is often necessary to film the same character at different stages of their life. When a plot spans a long period of time, realistically portraying the changes in an actor's expressions and demeanor at different ages on screen is a major challenge in film and television production. Before computer graphics and computer vision technologies were perfected and applied to the film and television industry, filmmakers typically used special shooting techniques and stunt doubles to achieve changes in a character's eye features.
[0003] However, existing methods for altering the eye features of individuals in videos have significant limitations. They require finding a stand-in actor with eye features extremely similar to the target actor during the filming stage, and then interspersing close-up shots of the stand-in actor's eyes into the target actor's video data in post-production. This method not only requires a large amount of manpower and time in post-production, but also makes it easy for viewers to only see close-up shots of the eyes and not the target actor's full face, which will affect the user experience of watching the video to some extent. Summary of the Invention
[0004] In view of this, embodiments of this application provide a method and apparatus for automatically changing the eye features of people in videos, so as to eliminate or improve one or more defects existing in the prior art.
[0005] One aspect of this application provides a method for automatically altering the eye features of a person in a video, including: Extract target image data containing the face of the target person and the corresponding target change requirement type from the video data, and obtain the target eye feature change model corresponding to the target person in advance based on the target change requirement type. The target image data is input into the target eye feature modification model, so that the target eye feature modification model performs feature replacement on the eye feature group in the target image data in accordance with the target modification requirement type, and outputs the eye feature modified image data corresponding to the target image data; The target image data is replaced with the eye feature change image data in the video data.
[0006] In some embodiments of this application, the target change request type is one of a plurality of change request types, wherein each of the change request types includes an emotion type and an age range, and any two different change request types may contain different emotion types and / or age ranges.
[0007] In some embodiments of this application, before extracting the target image data containing the target person's face from the video data, the method further includes: Using historical image data of other people's faces that do not include the target person, as well as label data corresponding to different types of change requirements, standard models for eye feature changes corresponding to each of the aforementioned change requirements are trained. Store the standard models of eye feature changes corresponding to each of the aforementioned change requirement types for transfer learning.
[0008] In some embodiments of this application, the step of using historical image data containing the faces of other persons not being the target person, and label data corresponding to different types of change requirements, to train a standard model for eye feature changes corresponding to each of the various change requirement types includes: Iterative training steps: Input each of the historical image data and the corresponding label data of the change requirement type into the preset machine learning model in sequence, so that the machine learning model first identifies the eye feature group in the historical image data, and then performs feature replacement on the eye feature group in the historical image data according to the corresponding change requirement type based on the label data, so that the machine learning model outputs the eye feature changed image data of each of the historical image data under each change requirement type in sequence. Based on the preset error threshold and the eye feature change image data output by the machine learning model, it is determined whether the current machine learning model meets the training end requirements. If not, the iterative training steps are returned until it is determined that the current machine learning model meets the training end requirements. Then the iteration stops, and the machine learning model is used as the standard model for eye feature change corresponding to its change requirement type.
[0009] In some embodiments of this application, the eye feature group includes at least one of the following: frontal eye features, side eye features, wrinkle features around the eyes, eye movement amplitude features, eye size features, iris color features, eye shadow size, and eye blood vessel distribution features.
[0010] In some embodiments of this application, before extracting the target image data containing the target person's face from the video data, the method further includes: Using historical image data containing the face of the target person and label data corresponding to different types of change requirements, transfer learning is performed on the standard model for eye feature change corresponding to each type of change requirement to train a special model for eye feature change of the target person under each type of change requirement. Store the specific eye feature change models corresponding to the target person under each of the various change requirement types.
[0011] In some embodiments of this application, the step of extracting target image data containing the face of a target person and the corresponding target change requirement type from video data, and obtaining a target eye feature change model corresponding to the target person pre-generated based on transfer learning, includes: Receive the target timestamp for the video data and the corresponding target change request type; Extract the target image data containing the face of the target person corresponding to the target timestamp from the video data; Among the various eye feature change-specific models, the eye feature change-specific model corresponding to the target change requirement type is selected as the current target eye feature change model for the target person.
[0012] Another aspect of this application provides an automatic alteration device for the eye features of a person in a video, comprising: The image and model acquisition module is used to extract target image data containing the face of the target person and the corresponding target change requirement type from the video data, and to obtain the target eye feature change model corresponding to the target person in advance based on the target change requirement type. An eye feature modification module is used to input the target image data into the target eye feature modification model, so that the target eye feature modification model performs feature replacement on the eye feature group in the target image data in accordance with the target modification requirement type, and outputs the eye feature modification image data corresponding to the target image data; An image data replacement module is used to replace the target image data with the eye feature change image data in the video data.
[0013] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for automatically changing the eye features of a person in a video.
[0014] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method for automatically changing the eye features of a person in a video.
[0015] The method for automatically changing the eye features of a person in a video provided in this application extracts target image data containing the face of the target person and the corresponding target change requirement type from the video data. Based on the target change requirement type, a target eye feature change model corresponding to the target person, generated in advance using transfer learning, is obtained. The target image data is input into the target eye feature change model, so that the target eye feature change model performs feature replacement on the eye feature group in the target image data in accordance with the target change requirement type, and outputs the eye feature change image data corresponding to the target image data. By replacing the target image data with the eye feature change image data in the video data, the eye features of the person in the video data can be automatically changed. By using a special target eye feature change model for the target person under the target change requirement type, the targeting and effect of the change can be effectively improved. On this basis, the use of transfer learning can also effectively improve the efficiency and automation of the change process, effectively reduce the manpower and time costs of post-processing, and effectively improve the user experience of video viewers, thereby meeting the needs of changing the eye movement features of the person applicable to movie settings and other scenarios.
[0016] Additional advantages, objectives, and features of this application will be set forth in part in the description which follows, and will in part become apparent to those skilled in the art upon review of the following description, or may be learned by practice of the application. The objectives and other advantages of this application can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.
[0017] Those skilled in the art will understand that the purposes and advantages that can be achieved with this application are not limited to those specifically described above, and that the above and other purposes that this application can achieve will be more clearly understood from the following detailed description. Attached Figure Description
[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. The components in the drawings are not drawn to scale but are merely for illustrating the principles of this application. For ease of illustration and description of certain parts of this application, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary device actually manufactured according to this application. In the drawings: Figure 1 This is a schematic diagram of the first step in an embodiment of the method for automatically changing the eye features of a person in a video according to this application.
[0019] Figure 2 This is a schematic diagram of the second process of an automatic alteration method for the eye features of a person in a video according to an embodiment of this application.
[0020] Figure 3 This is a flowchart illustrating the method of reversing aging in the eyes in a movie, as exemplified in one of the videos in this application.
[0021] Figure 4 This is a schematic diagram of the architecture of the StarGANv2 model used as an example in this application.
[0022] Figure 5 This is a schematic diagram illustrating the functional logic of a device that reverses the aging of the eyes in a movie, as illustrated in another example of this application.
[0023] Figure 6 This is a schematic diagram of the structure of an automatic alteration device for the eye features of a person in a video according to one embodiment of this application. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit it.
[0025] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the structures and / or processing steps closely related to the solution according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.
[0026] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.
[0027] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.
[0028] In the following description, embodiments of the present application will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.
[0029] To address the problems of existing methods for altering the eye features of people in videos, such as requiring advanced shooting skills, time-consuming post-processing, and potentially affecting the viewer experience, this application provides an automatic method for altering the eye features of people in videos, an automatic device for altering the eye features of people in videos to execute the method, a physical device, and a computer-readable storage medium. This significantly improves the efficiency of film and television shooting, results in more natural film and television presentation, and greatly reduces the cost of film and television production, while shortening the production cycle.
[0030] The following examples will provide a detailed description.
[0031] Based on this, embodiments of this application provide a method for automatically changing the eye features of a person in a video, which can be implemented by an automatic eye feature changing device in the video. See [link to relevant documentation]. Figure 1 The method for automatically changing the eye features of people in the video specifically includes the following: Step 100: Extract the target image data containing the target person's face and the corresponding target change requirement type from the video data, and obtain the target eye feature change model corresponding to the target person in advance based on the target change requirement type.
[0032] The target individuals are key figures whose facial features need to be presented in the final video data, such as main actors in a movie shooting scene.
[0033] In one or more embodiments of this application, the target image data containing the face of the target person obviously also includes the eyes of the target person, wherein the face can be the front or side view of the target person.
[0034] It is understood that the target change requirement type refers to the eye feature change requirement type data for the target image data. For example, the target change requirement type can specify to age or reverse the age of the eye features of the target person in the target image data. That is to say, in a video data, multiple target image data can be specified. The target change requirement types corresponding to these target image data can be the same or different, and can be set according to the actual application situation.
[0035] In one or more embodiments of this application, the target eye feature modification model, the standard eye feature modification model mentioned below, and the eye feature modification-specific model can all adopt models such as the StarGANv2 model, or models such as convolutional neural networks.
[0036] Step 200: Input the target image data into the target eye feature modification model, so that the target eye feature modification model performs feature replacement on the eye feature group in the target image data in accordance with the target modification requirement type, and outputs the eye feature modification image data corresponding to the target image data.
[0037] In step 200, inputting the target image data into the target eye feature modification model means that the target image data can be directly input into the target eye feature modification model, or the target image data can be preprocessed according to standards such as size and pixels, and then input into the target eye feature modification model in a standard format, so as to further improve the accuracy of feature modification by the target eye feature modification model.
[0038] Step 300: Replace the target image data with the eye feature change image data in the video data.
[0039] As can be seen from the above description, the automatic alteration method for human eye features in videos provided in this application embodiment can automatically alter the human eye features in video data. By adopting a specialized target eye feature alteration model for the target human under the target alteration requirement type, the method can effectively improve the targeting and effect of the alteration. On this basis, the use of transfer learning can also effectively improve the efficiency and automation of the alteration process, effectively reduce the manpower and time costs of post-processing, and effectively improve the user experience of video viewers, thereby meeting the alteration needs of human eye movement features applicable to scenarios such as movie settings.
[0040] To further improve the effectiveness and applicability of automatic alteration of eye features in videos, in an embodiment of this application, the target alteration requirement type is one of multiple alteration requirement types. Each alteration requirement type includes an emotion type and an age range, and any two different alteration requirement types may contain different emotion types and / or age ranges.
[0041] It is understood that the emotional types mentioned can at least include emotional expressions such as joy, anger, sorrow, and happiness, and can also be divided into other classification methods according to actual needs.
[0042] Furthermore, the age range is divided into intervals of at least one month to 30 years. For example, if the interval is 5 years, the age range could be: before age 5, between age 5 and 10, between age 10 and 15, etc.
[0043] To further improve the effectiveness and applicability of automatic alteration of eye features in videos, this application provides a method for automatic alteration of eye features in videos, see [link to relevant documentation]. Figure 2 The method for automatically changing the eye features of a person in the video also includes the following steps before step 100: Step 010: Using historical image data of other people's faces that do not contain the target person, and label data corresponding to different types of change requirements, train the standard model of eye feature change corresponding to each of the change requirements types.
[0044] In step 010, the historical image data of other people's faces that are not the target person can be obtained from existing historical datasets or historical databases to further reduce the difficulty and time required for data acquisition when training the standard model of eye feature change.
[0045] Meanwhile, the tag data corresponding to each historical image containing a person's face is the eye feature group of another historical image containing that person's face, and the change requirement type of the eye feature group of the previous historical image is different from the change requirement type of the eye feature group of the subsequent historical image. That is to say, the tag data corresponding to the change requirement type of the previous historical image specifically refers to the eye feature groups corresponding to different emotional types and age ranges of the same person in the previous historical image. For example: extract a historical image containing the face of Zhang San, who is not the target person, from the database, then extract Zhang San's eye feature group from this historical image and artificially determine that Zhang San's emotional type in the historical image is: joy (happiness), and determine that Zhang San's age in the historical image belongs to the age range of 45-50 years old. Then, another historical image containing Zhang San's face (not the target person) is extracted. The emotional type of Zhang San in this historical image is then manually determined to be "anger," and his age in the other historical image is determined to be between 15 and 20 years old. The eye feature set of Zhang San in this historical image is then extracted as label data. That is, the previous historical image with the emotional type "joy" (happiness) and an age range of 45-50 years old is used as training data, and the eye feature set of the second historical image with the emotional type "anger" and an age range of 15-20 years old is used as the label for this training data corresponding to the change requirement of "emotional type: anger and age range of 15-20 years old."
[0046] Step 020: Store the standard models of eye feature changes corresponding to each of the various change requirement types for transfer learning.
[0047] Step 010 can specifically consist of the following steps 011 and 012: Step 011 Iterative Training Step: Input each of the historical image data and the corresponding label data of the change requirement type into the preset machine learning model in sequence, so that the machine learning model first identifies the eye feature group in the historical image data, and then performs feature replacement on the eye feature group in the historical image data according to the corresponding change requirement type based on the label data, so that the machine learning model outputs the eye feature change image data of each of the historical image data under each change requirement type in sequence.
[0048] Step 012 determines whether the current machine learning model meets the training termination requirements based on the preset error threshold and the eye feature change image data output by the machine learning model. If not, it returns to the iterative training step until it is determined that the current machine learning model meets the training termination requirements. Then, it stops iterating and uses the machine learning model as the standard model for eye feature change corresponding to its corresponding change requirement type.
[0049] To further improve the effectiveness and reliability of automatic changes to the eye features of people in videos, the eye feature group includes at least one of the following: frontal eye features, side eye features, wrinkle features around the eyes, eye movement amplitude features, eye size features, iris color features, eye shadow size, and eye blood vessel distribution features.
[0050] In the method for automatically changing the eye features of a person in a video provided in this application embodiment, see [link to relevant documentation]. Figure 2 The method for automatically changing the eye features of a person in the video also includes the following content between steps 020 and 100: Step 030: Using historical image data containing the target person's face and label data corresponding to different types of change requirements, transfer learning is performed on the standard models for eye feature changes corresponding to each type of change requirement to train a dedicated model for eye feature changes for the target person under each type of change requirement.
[0051] In one example, we need to obtain the eye appearance of a person in their old age / youth / childhood from videos of them in middle age, but this data is lacking. Therefore, we can train eye models of multiple other people in their childhood, youth, middle age, and old age, and then input the middle-aged data of the original person to obtain the eye data of that person in their old age / youth / childhood. This belongs to the direct transfer learning method within transfer learning.
[0052] Step 040: Store the specific eye feature change models for the target person under each of the various change requirement types.
[0053] In the method for automatically changing the eye features of a person in a video provided in this application embodiment, see [link to relevant documentation]. Figure 2 Step 100 of the method for automatically changing the eye features of a person in the video specifically includes the following: Step 110: Receive the target timestamp for the video data and the corresponding target change request type.
[0054] Step 120: Extract the target image data containing the target person's face corresponding to the target timestamp from the video data.
[0055] Step 130: Among the various eye feature change models, select the eye feature change model corresponding to the target change requirement type as the current target eye feature change model for the target person.
[0056] In filmmaking, realistically portraying the changes in an actor's appearance, physique, and other physical attributes across a long timeframe has always been a major challenge, especially when the storyline spans a significant period. Special effects makeup, using stunt doubles, or casting younger actors have been common choices for filmmakers, but the results have often been unsatisfactory. With the continuous development of cinematic art, the need for films, including science fiction, to depict things that don't exist in the real world has become increasingly demanding. Before the advancement of video and image processing technologies, filmmakers typically used imaginative shooting techniques combined with post-production compositing to create these fictional elements.
[0057] However, this technique has limitations. With the increasing application of computer graphics and computer vision technologies in the film and television industry, more and more realistic and vivid aging and reverse-aging characters have appeared in the public eye in recent years. Digital facial aging and reverse-aging technology under CGI has begun to become a new option for film creation. Because it requires processing facial appearance frame by frame, the overall special effects production cycle of the film is relatively long. And from the final effect, the actors' faces, which appear to have returned to youth, are not realistic and believable enough, especially in terms of stiff and unnatural facial expressions and movements during dialogue. While moviegoers marvel at the technological advancements, they seem somewhat dissatisfied with this reverse-aging effect. When we need to achieve more complex video and image processing effects, such as reverse-aging eyes in movies, traditional methods are no longer sufficient. We need to research and implement more advanced methods and devices to meet new technical requirements.
[0058] Based on this, in one example of the method for automatically changing the eye features of a person in the video provided in this application, we will use the example of making the eyes of an actor in a movie look younger as an illustration: Based on artificial intelligence image recognition and model training technology, the desired anti-aging effect is imported into the target person's face and transferred to learn, ultimately resulting in a natural and distortion-free anti-aging eye model. This example can effectively achieve special effects that are impossible with traditional photography techniques, further enhancing the artistic expression of films. To achieve the aforementioned special effects, this example proposes a method for anti-aging eyes in film, including the following steps: Step 1. First, collect data on different emotions and angles of the person's face, including data from people other than the person and data from when the actor was young; the data on different emotions and angles of the person's face includes: the main features of the person's face and the area around the eyes.
[0059] Step 2. Extract specific data from the collected data of all types that match the plot at different time periods and mark them on the timeline.
[0060] Furthermore, in steps 1 and 2: data from different emotions and angles is collected: based on the expected angles and emotions presented in front of the camera according to the movie's special event point prediction model, all facial features and facial characteristics appearing in the shot are collected. Emphasis can be placed on strengthening the connection between the human gaze and the surrounding facial skin based on the algorithm in step 4, improving the realism of transfer learning. Data from different emotions and angles includes: frontal and side-view eye videos, wrinkles around the eyes, gaze movement amplitude, eyeball size, iris color and shadow size, and distribution of blood vessels in the eyes.
[0061] Step 3. Find data from the database at several key time points: In the first round, train the model with the same expression and angle; in the second round, use the actor's own data for transfer learning to obtain the model. The purpose of transfer learning is: 1. To reduce computing power; 2. To make fine adjustments to the original model to suit the current character.
[0062] Furthermore, in step 3: data is retrieved from the database at several key time points. The first round of model training involves obtaining the skeletal feature points of the target model using face recognition algorithms (such as MTCNN, PFLD, etc.). These skeletal feature points should be labeled before training, indicating the skeletal feature type and coordinates. The eyes and facial features of other people and actors in their younger years from the database are then substituted into the target model for training with the same expressions and angles.
[0063] In the first round, data from other individuals were used to train eye models for different genders and time periods.
[0064] For example, there could be a female-elderly-eye-tracking model, a male-youth-eyeball model, a female-child-detailed supplementary model, etc.
[0065] Specifically, by recording the following data from other time periods of a person's life: front and side view eye videos, eye movement amplitude, eyeball size, skin wrinkles around the eyes, iris color, and distribution of blood vessels in the eyeballs, the data is input into different pre-learned algorithm models to obtain eye data of the person for a specified time period and generate new eyeball image data.
[0066] Examples of algorithms for the female-elderly-eye-tracking model include: inputting the amplitude of eye movements and eye size, and using eye-tracking algorithms (Gaze-Tracking) to predict pupil position; examples of algorithms for the male-youth-eye-tracking model include: inputting frontal and side-view videos of the eyes, and using object detection algorithms (YOLO series) to detect open and closed eyes to obtain the blinking frequency and the degree of eyeball protrusion; examples of algorithms for the female-child-detail supplementation model include: inputting the wrinkles around the eyes, iris color, and blood vessel distribution of the eyes, and using segmentation models such as Deeplabv3 to re-predict data for different age groups.
[0067] The second round uses the actor's own data for transfer learning, including: using the actor's own eye expression data to replace the model data, optimizing it through deep learning algorithms, and finally obtaining the model. The reason for using transfer learning in this application is: We need to obtain the eye condition of a person in their old age / youth / childhood from videos of them in middle age, but this data is lacking. Therefore, we can train eye models of multiple other people in their childhood, youth, middle age, and old age, and then input the middle-aged data of the original person to obtain the eye data of that person in their old age / youth / childhood. This belongs to the direct transfer learning method within transfer learning.
[0068] Step 4. Using the facial expression and angle data at key time points trained in the previous step as input, perform interpolation on the images at intermediate time points.
[0069] In addition, to achieve the above method, this example also provides a device for achieving the effect of youthful eyes in movies, including the following: (1) Model training unit: Based on the facial expressions and emotions of the target person at key time points, data is searched from the database to train the model with the same facial expressions and angles; (2) Model learning unit: Combine the training unit with the actor's own data to perform transfer learning to obtain the model; (3) Calculation unit: Using the key time point data trained in the previous step as input, the difference is calculated for the intermediate time frame.
[0070] Furthermore, the model training unit is a combination of a face bone recognition device and a model training device: the face bone recognition device obtains the skeletal feature points of the target model through a face recognition algorithm; the model training device inputs the eyes and facial features of other people and actors in their youth from the database into the target model to train the model with the same expressions and angles.
[0071] Furthermore, the model learning unit is a combination of a model training unit and a transfer learning unit: the actor's own eye expression and facial features data are substituted into the training unit, and the model is finally obtained through transfer learning.
[0072] Furthermore, the computing unit is configured to calculate the difference between the trained data and the video signal at key time points in the movie and the footage at intermediate times.
[0073] Specifically, such as Figure 3 As shown, this example achieves the effect of reversing the aging of a movie character's eyes through a series of sequential, judgmental, and looping processing steps.
[0074] In step 101, time points are first marked on the film timeline. Based on the emotions that the film characters need to express at these time points, data from the same angle related to different emotions are collected to provide data support for subsequent model training. The collected data includes footage not of the actor themselves and footage from when the actor was a child.
[0075] In step 102, the data is adjusted until the angles are consistent before importing.
[0076] In step 103, the imported data is used for model training; here, the StarGANv2 model is used. For example... Figure 4 As shown, it consists of two parts: the first round is to train the model with the same expression and angle. The skeletal feature points of the target model are obtained through the face recognition algorithm. The eyes and facial features of other people and actors when they were young are substituted into the target model to train the model with the same expression and angle. The second round is to use the data of the actor when he was young for transfer learning. The eye data of the actor when he was young is used to replace the model data. The model is optimized through deep learning algorithm to finally obtain the corresponding model weights.
[0077] In step 104, the model obtained from transfer learning is substituted into the image captured by the camera to obtain the data result, and the difference is calculated with the data result of the intermediate image.
[0078] In step 105, if the error rate of the processed video signal in the focus area of the eyes and facial features is not less than the set value of 5%, it indicates that the model training is not perfect and needs to be imported back to the training unit to continue training with new data; if the error is less than the set value, the training can be considered complete, the processed video is output, and the process ends.
[0079] In the specific implementation of the above method, a device that supports the method is required. For example... Figure 5 As shown, this example employs an eye rejuvenation device comprising the following units: Module 201 collects and organizes data based on the emotional and perspective requirements at each time point, generating a video database related to the current time point as the data source for model learning; the data generated by 201 is then imported into Module 202. Module 202 first trains a model with the same expression and angle using the input data, then imports the actor's own data for transfer learning, and estimates the error of the final result. The unqualified model result is transmitted back to the previous module 201 to receive new data, retrain, and continuously improves the model in the loop until it reaches the allowable error range and is output.
[0080] The following example demonstrates the specific implementation of this embodiment by using the apparatus and method described herein to reverse the age of the eyes of an actor (A) playing a killer in a movie at different ages.
[0081] From a software perspective, this application also provides an apparatus for automatically changing the eye features of a person in a video, comprising all or part of the aforementioned method for automatically changing the eye features of a person in a video. See [link to relevant documentation]. Figure 6 The automatic alteration device for the eye features of the person in the video specifically includes the following components: The image and model acquisition module 10 is used to extract target image data containing the face of the target person and the corresponding target change requirement type from the video data, and to obtain the target eye feature change model corresponding to the target person in advance based on the target change requirement type.
[0082] The eye feature modification module 20 is used to input the target image data into the target eye feature modification model, so that the target eye feature modification model performs feature replacement on the eye feature group in the target image data in accordance with the target modification requirement type, and outputs the eye feature modification image data corresponding to the target image data.
[0083] The image data replacement module 30 is used to replace the target image data with the eye feature change image data in the video data.
[0084] The embodiments of the automatic alteration device for the eye features of people in videos provided in this application can be used to execute the processing flow of the embodiments of the automatic alteration method for the eye features of people in videos described above. Its functions will not be repeated here, but can be referred to the detailed description of the embodiments of the automatic alteration method for the eye features of people in videos described above.
[0085] The automatic alteration of eye features in a video can be performed on a client device. The specific implementation can be chosen based on the processing capabilities of the client device and limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are performed on the client device, the client device may further include a processor for the specific processing of the automatic alteration of eye features in the video.
[0086] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.
[0087] The server and the client device can communicate using any suitable network protocol, including those not yet developed as of the date of this application. Such network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Furthermore, such network protocols may also include RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer Protocol) protocols used on top of the aforementioned protocols.
[0088] As can be seen from the above description, the automatic eye feature changing device for people in videos provided in this application embodiment can automatically change the eye features of people in video data. By adopting a special target eye feature changing model for the target person under the target change requirement type, the device can effectively improve the targeting and effect of the change. On this basis, the use of transfer learning can also effectively improve the efficiency and automation of the change process, effectively reduce the manpower and time costs of post-processing, and effectively improve the user experience of video viewers, thereby meeting the needs for changing the eye movement features of people applicable to movie settings and other scenarios.
[0089] This application also provides an electronic device, which may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the automatic alteration method for the eye features of a person in a video mentioned in the above embodiments. The processor and the memory can be connected via a bus or other means, taking a bus connection as an example. The receiver can be connected to the processor and the memory via wired or wireless means.
[0090] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.
[0091] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the automatic alteration method for the eye features of a person in a video in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby realizing the automatic alteration method for the eye features of a person in a video in the above method embodiments.
[0092] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0093] The one or more modules are stored in the memory, and when executed by the processor, the method for automatically changing the eye features of people in the video in the embodiment is executed.
[0094] In some embodiments of this application, the user equipment may include a processor, a memory, and a transceiver unit. The transceiver unit may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected via a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.
[0095] As one implementation method, the functions of the receiver and transmitter in this application can be implemented by transceiver circuits or dedicated transceiver chips, and the processor can be implemented by dedicated processing chips, processing circuits or general-purpose chips.
[0096] As another implementation approach, the server provided in this application embodiment can be implemented using a general-purpose computer. That is, the program code implementing the processor, receiver, and transmitter functions is stored in memory, and the general-purpose processor implements the processor, receiver, and transmitter functions by executing the code in memory.
[0097] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned method for automatically changing the eye features of a person in a video. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.
[0098] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave.
[0099] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0100] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments. The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to the embodiments of this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A method for automatically changing the eye features of a person in a video, characterized in that, Applied to film shooting scenarios, the method includes: Using historical image data of other people's faces that do not include the target person, and label data corresponding to different types of change requirements, standard models for eye feature changes corresponding to each of the change requirements are trained; the target person includes actors in movie shooting scenes. Store the standard models of eye feature changes corresponding to each of the aforementioned change requirement types for transfer learning; Using historical image data containing the face of the target person and label data corresponding to different types of change requirements, transfer learning is performed on the standard model for eye feature change corresponding to each type of change requirement to train a special model for eye feature change of the target person under each type of change requirement. Store dedicated models for eye feature changes for the target person under each of the various change requirement types; Extract target image data containing the face of the target person and the corresponding target change requirement type from the video data, and obtain the target eye feature change model corresponding to the target person in advance based on the target change requirement type. The target image data is input into the target eye feature modification model, so that the target eye feature modification model performs feature replacement on the eye feature group in the target image data in accordance with the target modification requirement type, and outputs the eye feature modified image data corresponding to the target image data; The target image data is replaced with the eye feature change image data in the video data; The step of extracting target image data containing the target person's face from the video data and the corresponding target change requirement type, and obtaining the target eye feature change model corresponding to the target person pre-generated based on the target change requirement type, includes: Receive the target timestamp for the video data and the corresponding target change request type; Extract the target image data containing the face of the target person corresponding to the target timestamp from the video data; Among the various eye feature change-specific models, the eye feature change-specific model corresponding to the target change requirement type is selected as the current target eye feature change model for the target person.
2. The method for automatically changing the eye features of a person in a video according to claim 1, characterized in that, The target change request type is one of multiple change request types, wherein each change request type includes an emotion type and an age range, and any two different change request types may contain different emotion types and / or age ranges.
3. The method for automatically changing the eye features of a person in a video according to claim 1, characterized in that, The method employs historical image data containing faces of other individuals not related to the target person, along with label data corresponding to different types of alteration requirements, to train standard models for eye feature alteration for each of the alteration requirement types, including: Iterative training steps: Input each of the historical image data and the corresponding label data of the change requirement type into the preset machine learning model in sequence, so that the machine learning model first identifies the eye feature group in the historical image data, and then performs feature replacement on the eye feature group in the historical image data according to the corresponding change requirement type based on the label data, so that the machine learning model outputs the eye feature changed image data of each of the historical image data under each change requirement type in sequence. Based on the preset error threshold and the eye feature change image data output by the machine learning model, it is determined whether the current machine learning model meets the training end requirements. If not, the iterative training steps are returned until it is determined that the current machine learning model meets the training end requirements. Then the iteration stops, and the machine learning model is used as the standard model for eye feature change corresponding to its change requirement type.
4. The method for automatically changing the eye features of a person in a video according to claim 1 or 3, characterized in that, The eye feature group includes at least one of the following: frontal eye features, side eye features, wrinkle features around the eyes, eye movement amplitude, eye size, iris color, eye shadow size, and blood vessel distribution.
5. A device for automatically changing the eye features of a person in a video, characterized in that, Applied to film shooting scenarios, the device is used to perform: Using historical image data of other people's faces that do not include the target person, and label data corresponding to different types of change requirements, standard models for eye feature changes corresponding to each of the change requirements are trained; the target person includes actors in movie shooting scenes. Store the standard models of eye feature changes corresponding to each of the aforementioned change requirement types for transfer learning; Using historical image data containing the face of the target person and label data corresponding to different types of change requirements, transfer learning is performed on the standard model for eye feature change corresponding to each type of change requirement to train a special model for eye feature change of the target person under each type of change requirement. Store dedicated models for eye feature changes for the target person under each of the various change requirement types; The device includes: The image and model acquisition module is used to extract target image data containing the face of the target person and the corresponding target change requirement type from the video data, and to obtain the target eye feature change model corresponding to the target person in advance based on the target change requirement type. An eye feature modification module is used to input the target image data into the target eye feature modification model, so that the target eye feature modification model performs feature replacement on the eye feature group in the target image data in accordance with the target modification requirement type, and outputs the eye feature modification image data corresponding to the target image data; An image data replacement module is used to replace the target image data with the eye feature change image data in the video data; The step of extracting target image data containing the target person's face from the video data and the corresponding target change requirement type, and obtaining the target eye feature change model corresponding to the target person pre-generated based on the target change requirement type, includes: Receive the target timestamp for the video data and the corresponding target change request type; Extract the target image data containing the face of the target person corresponding to the target timestamp from the video data; Among the various eye feature change-specific models, the eye feature change-specific model corresponding to the target change requirement type is selected as the current target eye feature change model for the target person.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the automatic alteration method for the eye features of a person in a video as described in any one of claims 1 to 4.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method for automatically changing the eye features of a person in a video as described in any one of claims 1 to 4.
Citation Information
Patent Citations
A combined painting method and system based on facial expressions
CN109558851A
Image processing method, related equipment and computer readable storage medium
CN113822114A
Image generation method and system, electronic equipment and computer readable storage medium
CN115131857A
