A method and system for processing information of AIGC skit generation

By generating and adjusting the keyframe logo of the face action of the character in the AIGC storyboard, the problem of inability to express the expressions is solved, and the three-dimensional presentation and improvement of the fineness of emotional information is achieved, and the user's processing of expressions is met. need.

CN118632049BActive Publication Date: 2024-11-05GUANGZHOU HAND IN HAND INTERNET CO LTD
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411090096.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-05
Estimated Expiration
2044-08-09

AI Technical Summary

Technical Problem

The facial expressions of characters in the existing AIGC storyboard are not vivid and rich enough to effectively convey emotional information, resulting in insufficient processing precision and accuracy, which cannot meet user needs.

Method used

Through the detection of the target video storyboard, the character's speech text and plot expectations are obtained to convey emotional information, generate multiple facial action keyframe logos, and adjust the storyboard according to the user's choice to achieve the three-dimensional presentation of the character's facial expressions and Communication of emotional information.

Benefits of technology

The precision and accuracy of the equipment processing of facial images of characters in storyboards is improved, and the three-dimensional presentation of emotional information is realized, meeting users' needs for expression processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118632049B_ABST
    Figure CN118632049B_ABST
Patent Text Reader

Abstract

The present application provides an AIGC skit generation information processing method and system, which generates multiple facial action key frame identifiers through the expected emotional information conveyed by the character speech text of the target storyboard and the semantic emotional information of each line in the target storyboard; determines at least two target facial action key frame identifiers according to the user's second selection operation for the multiple facial action key frame identifiers, and the video time range corresponding to the at least two target facial action key frame identifiers includes at least the first video time range; adjusts the target storyboard according to the at least two target facial action key frame identifiers to obtain an updated target storyboard. Therefore, the present application realizes the function of presenting the expected emotional information conveyed by the character plot in three dimensions when the device processes the skit video data, which is conducive to improving the precision and accuracy of the facial images of the characters in the storyboards processed by the device.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of digital film and television development and production, and in particular to an AIGC short drama generation information processing method and system. Background Art

[0002] Currently, the facial expressions of the characters in the AIGC storyboards that contain images of the characters speaking are not vivid and rich enough, and cannot help enhance the three-dimensional presentation of the emotional information that the characters and the plot expect to convey. The fineness and accuracy of the processing of the facial images of the characters in the storyboards cannot meet user needs. Summary of the invention

[0003] The present application provides an AIGC short drama generation information processing method and system, which realizes the function of presenting the emotional information expected to be conveyed by the character plot in three dimensions when the device processes the short drama video data, which is conducive to improving the precision and accuracy of the device in processing the facial images of the characters in the storyboard.

[0004] In a first aspect, an embodiment of the present application provides an AIGC skit generation information processing method, the method comprising: detecting a first selection operation on an expression control for a target storyboard of a target video, obtaining a character speech text of the target storyboard, the character speech text including at least one line; and obtaining the expected emotional information of the character plot of the target storyboard; generating a plurality of facial action key frame identifiers according to the expected emotional information and the line semantic emotional information of the at least one line, the video time range corresponding to the plurality of facial action key frame identifiers including the first video time range of the at least one line, and including the second video time range and / or the third video time range, the second video time range being the time period before the first video time range, and the third video time range being the time period after the first video time range; determining at least two target facial action key frame identifiers according to a second selection operation of the user on the plurality of facial action key frame identifiers, the video time range corresponding to the at least two target facial action key frame identifiers including at least the first video time range; adjusting the target storyboard according to the at least two target facial action key frame identifiers to obtain an updated target storyboard.

[0005] In a second aspect, an embodiment of the present application provides an AIGC skit generation information processing system, comprising a server and a user device, wherein the server is used to execute the steps of the method as described in any one of the first aspects.

[0006] In a third aspect, an embodiment of the present application provides a server, comprising a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of the method as described in any one of the first aspects.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program or instructions, and when the computer program or instructions are executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0008] It can be seen that in the embodiment of the present application, a first selection operation on the expression control 321 for the target storyboard of the target video is detected, and the character speech text of the target storyboard is obtained, and the character speech text includes at least one line of dialogue; and the expected emotional information of the character plot of the target storyboard is obtained; a plurality of facial action key frame identifiers are generated according to the expected emotional information and the line semantic emotional information of at least one line of dialogue, and the video time range corresponding to the plurality of facial action key frame identifiers includes the first video time range of at least one line of dialogue, and includes the second video time range and / or the third video time range, the second video time range is the time period before the first video time range, and the third video time range is the time period after the first video time range; at least two target facial action key frame identifiers are determined according to the user's second selection operation on the plurality of facial action key frame identifiers, and the video time range corresponding to the at least two target facial action key frame identifiers includes at least the first video time range; the target storyboard is adjusted according to the at least two target facial action key frame identifiers to obtain an updated target storyboard. Therefore, the present application realizes the function of presenting the characters' plots in three dimensions and conveying emotional information when the device processes short drama video data, which is beneficial to improving the precision and accuracy of the device in processing the facial images of the characters in the storyboards. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0010] Figure 1 A schematic diagram of the structure of an AIGC skit generation information processing system provided in an embodiment of the present application;

[0011] Figure 2 A flowchart of an AIGC skit generation information processing method provided in an embodiment of the present application;

[0012] Figure 3 One of the schematic diagrams of the video generation page provided in the embodiment of the present application;

[0013] Figure 4 A schematic diagram of the distribution of each line in the video range of the target storyboard provided in an embodiment of the present application;

[0014] Figure 5 A schematic diagram of the video time range of the target storyboard and the voice audio corresponding to the target storyboard provided in an embodiment of the present application;

[0015] Figure 6 An example diagram of standard facial movements provided in an embodiment of the present application;

[0016] Figure 7 The second schematic diagram of the video generation page provided in the embodiment of the present application;

[0017] Figure 8 The third schematic diagram of the video generation page provided in the embodiment of the present application;

[0018] Fig. 9 Another schematic diagram of the distribution of the lines in the video range of the target storyboard provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0021] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0022] In the embodiments of the present application, "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent the following three situations: A exists alone; A and B exist at the same time; B exists alone. Among them, A and B can be singular or plural.

[0023] In the embodiment of the present application, the symbol " / " can indicate that the objects associated with each other are in an "or" relationship. In addition, the symbol " / " can also indicate a division sign, that is, performing a division operation. For example, A / B can indicate A divided by B.

[0024] In the embodiments of the present application, "at least one item" or similar expressions refer to any combination of these items, including any combination of single items or plural items, and refer to one or more, and multiple refers to two or more. For example, at least one item of a, b, or c can represent the following seven situations: a, b, c, a and b, a and c, b and c, a, b, and c. Among them, each of a, b, and c can be an element or a set containing one or more elements.

[0025] In the embodiments of the present application, "equal to" can be used in conjunction with greater than, and is applicable to the technical solution adopted when greater than, and can also be used in conjunction with less than, and is applicable to the technical solution adopted when less than. When equal to is used in conjunction with greater than, it is not used in conjunction with less than; when equal to is used in conjunction with less than, it is not used in conjunction with greater than.

[0026] AIGC (Artificial Intelligence Generated Content / AI-Generated Content) is translated into Chinese as artificial intelligence generated content. It is generally considered to be a concept proposed in contrast to PCG (professional generated content) and UCG (user generated content). The narrow concept of AIGC is a production method that uses AI to automatically generate content. In a broad sense, AIGC can be seen as an AI technology that has the ability to generate and create like humans, that is, generative AI, which can autonomously generate and create new texts, images, music, videos, 3D interactive content and other forms of content and data based on training data and generation algorithm models, as well as opening up new scientific discoveries and creating new value and meaning.

[0027] The present application provides an AIGC skit generation information processing method and system, which can adjust the facial expressions of characters in the AIGC storyboards that currently contain images of characters speaking through target facial action key frame markers, so that the emotional information that the characters' plots expect to convey is presented in three dimensions, meeting the user's requirements for the fineness and accuracy of the facial images of the characters in the storyboards.

[0028] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0029] See also Figure 1 , Figure 1 This is a schematic diagram of the structure of an AIGC skit generation information processing system provided in an embodiment of the present application. Figure 1 As shown, the AIGC skit generation information processing system 1 may include a server 10 and a user device 20. The server 10 may be a physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The user device 20 may be a PC, or a terminal device such as a smart phone, smart TV, tablet computer, or portable computer.

[0030] The server 10 stores the AIGC skit generation information processing program and performs the operations in the following AIGC skit generation information processing method. The method implemented when the AIGC skit generation information processing program running on the server 10 is executed can refer to the various embodiments of the AIGC skit generation information processing method of the present application, which will not be repeated here. The user device 20 is connected to the server 10 for communication. The user device 20 includes at least one display screen for displaying an interactive page and receiving a user's trigger operation on a touch control to implement functions such as parameter input and parameter selection.

[0031] Those skilled in the art will understand that Figure 1 The processing system structure shown in the figure does not constitute a limitation on the AIGC skit generation information processing system 1, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0032] Based on the above hardware structure, various embodiments of the AIGC skit generation information processing method of the present application are proposed.

[0033] See also Figure 2 , Figure 2 A flowchart of an AIGC skit generation information processing method provided in an embodiment of the present application is applied to Figure 1 The server 10 in the AIGC skit generation information processing system 1, the method comprising:

[0034] Step S201: The server detects a first selection operation on an expression control for a target storyboard of a target video.

[0035] In a specific implementation, after the server receives the first control selection message from the user device, the first control selection message carries the identification information of the expression control of the target storyboard. The server obtains the identification information of the target storyboard and executes the steps after step S201 in this embodiment for the target storyboard.

[0036] For example, see Figure 3 , Figure 3 One of the schematic diagrams of the video generation page provided in the embodiment of the present application. The video generation page 3 is displayed on the display screen of the user device, and the video generation page 3 includes a video setting display area 31, a video browsing display area 32, and a video storyboard display area 33. The video browsing display area 32 can display the target video, and the video browsing display area 32 also includes an expression control 321. When the user triggers the expression control 321, the server receives the expression instruction from the user terminal, and the server extracts multiple storyboards from the target video 322, and displays the thumbnails of the multiple storyboards in the video storyboard display area 33, and prompts the user to select a storyboard from the multiple storyboards as the current target storyboard to be processed. The video storyboard display area 33 also includes a confirmation control. After the server receives the first confirmation message from the user device, the first confirmation message carries the identification information of the storyboard, and the storyboard corresponding to the identification information is marked as the target storyboard, so as to perform subsequent processing steps on the target storyboard.

[0037] The video browsing display area 32 includes at least one video window 323, such as a target video window 3231 to be processed currently and a video window 3232 of a previously generated video. Each video window also includes a plurality of processing controls, such as a lip sync control 324, an expression control 321, a sharing control 325, an editing control 326, etc. Each video window also includes a user name, a generation time of the video, detailed information, etc.

[0038] Among them, the lip movements of the characters in the target video are synchronized with the audio, but the facial expressions of the characters are not vivid or even have no expressions at all. Exemplarily, the target video can be a video of a person speaking generated by AIGC. The target video includes at least one storyboard, and the target storyboard is one of the at least one storyboard. The target video can be a video of a person speaking generated by a server, or a video of a person speaking imported by a user.

[0039] Step S202: the server obtains the speech text of the character in the target storyboard.

[0040] The character speech text includes at least one line of dialogue.

[0041] The character's speech text can be identified based on the character's speech voice to obtain the speech text; when the character's speech text is pre-stored in the memory of the server, it can be directly obtained from the memory.

[0042] Step S203: the server obtains the desired emotional information of the character plot of the target storyboard.

[0043] The expected emotional information refers to the overall emotional information of the characters in the target storyboard. For example, common emotional information includes 7 types: neutral, happy, angry, sad, afraid, disgusted, and surprised. The expected emotional information in the target storyboard can be at least one of the 7 types of emotional information. The expected emotional information can be analyzed by machine learning, deep learning, knowledge graphs, etc. to extract the overall emotional information of the characters' speech texts in the target storyboard, and the overall emotional information is used as the expected emotional information.

[0044] Step S204: the server generates a plurality of facial action key frame identifiers according to the desired emotion information and the speech semantic emotion information of the at least one speech.

[0045] Among them, the video time range corresponding to the multiple facial action key frame identifiers includes the first video time range of the at least one line of dialogue, and includes the second video time range and / or the third video time range, the second video time range is the time period before the first video time range, and the third video time range is the time period after the first video time range. That is to say, in the storyboard video clip to which a line of dialogue belongs, the time of the character's facial action can be the speaking time corresponding to each line of dialogue, or it can be before or after speaking. The first video time range is when speaking, the second video time range is before speaking, and the third video time range is after speaking. Figure 4 , Figure 4 A schematic diagram of the distribution of each line in the video range of the target storyboard provided in the embodiment of the present application, such as Figure 4 As shown, the target storyboard includes the first line and the second line, the video time range corresponding to the first line and the second line is the first video time range, the video time range before the first video time range of the first line is the second video time range, and the video time range after the first video time range of the second line is the third video time range.

[0046] Step S205: The server determines at least two target facial motion key frame identifiers according to the second selection operation of the user on the multiple facial motion key frame identifiers.

[0047] In a specific implementation, the server receives a second control selection message from the user device, the second control selection message carries identification information of the target facial action key frame identifier, the server obtains the corresponding target facial action key frame identifier, and continues to execute the content of step S206.

[0048] The video time range corresponding to the at least two target facial action key frame identifiers at least includes the first video time range.

[0049] When the user selects the target storyboard, the server processes step S202 to step S204 and then displays the target storyboard. Figure 7 , Figure 7 The second schematic diagram of the video generation page provided in the embodiment of the present application is as follows: Figure 7 As shown, a plurality of facial motion key frame marks are displayed in the video setting display area 31. The facial motion key frame mark is a set of frame image sequences, which express the dynamic change process of the facial expression of the character between the key frames. The facial motion key frame mark can be played repeatedly. Preferably, the facial motion key frame marks of the same line are located in the same row.

[0050] It should be noted that for the same line, due to different selections of key frame images, the facial action change sequence in the generated facial action key frame identifier may also be different. Users can select the appropriate facial action key frame identifier from multiple facial action key frame identifiers for the same line. The same line may also involve multiple facial action key frame identifiers, see Fig. 9 The third key frame image and the fourth key frame image corresponding to the second line in the video are the key frame image group, and the frame images before the third frame image and after the fourth frame image in the second line are the remaining frame images. A facial motion key frame identifier is generated for each frame image group. Therefore, there are 3 facial motion key frame identifiers corresponding to the second line. The facial motion key frame identifier also includes the line identifier to which it belongs, the selection box, and the playback duration of the key frame ( Figure 7 not shown).

[0051] The video setting display area 31 also includes a confirmation control. When the server receives a second confirmation message from the user terminal, the second confirmation message carries identification information of the target facial motion key frame identifier. The server adjusts the target storyboard according to the target facial motion key frame identifier corresponding to the identification information to obtain an updated target storyboard.

[0052] Step S206: The server adjusts the target storyboard according to the at least two target facial action key frame identifiers to obtain an updated target storyboard.

[0053] Among them, see Figure 8 , Figure 8 The third schematic diagram of the video generation page provided in the embodiment of the present application divides the video storyboard display area 33 into two display windows, one is a window to be updated, and the other is an updated window. The window to be updated is used to display the video storyboard that has not been updated, and the updated window is used to display the updated video storyboard. After the target storyboard is updated, the updated target storyboard is displayed in the updated window, and the target storyboard is deleted from the window to be updated. Among them, for the storyboard that does not need to be adjusted, the third selection operation is performed on it to directly move it to the updated window.

[0054] Optionally, the third selection operation includes but is not limited to dragging, long pressing, etc.

[0055] It can be seen that in the embodiment of the present application, a first selection operation on the expression control 321 for the target storyboard of the target video is detected, and the character speech text of the target storyboard is obtained, and the character speech text includes at least one line of dialogue; and the expected emotional information of the character plot of the target storyboard is obtained; a plurality of facial action key frame identifiers are generated according to the expected emotional information and the line semantic emotional information of at least one line of dialogue, and the video time range corresponding to the plurality of facial action key frame identifiers includes the first video time range of at least one line of dialogue, and includes the second video time range and / or the third video time range, the second video time range is the time period before the first video time range, and the third video time range is the time period after the first video time range; at least two target facial action key frame identifiers are determined according to the user's second selection operation on the plurality of facial action key frame identifiers, and the video time range corresponding to the at least two target facial action key frame identifiers includes at least the first video time range; the target storyboard is adjusted according to the at least two target facial action key frame identifiers to obtain an updated target storyboard. Therefore, the present application realizes the function of presenting the characters' plots in three dimensions and conveying emotional information when the device processes short drama video data, which is beneficial to improving the precision and accuracy of the device in processing the facial images of the characters in the storyboards.

[0056] In some embodiments, generating multiple facial action key frame identifiers based on the desired emotional information and the semantic emotional information of the at least one line of dialogue includes: acquiring facial elements of the character in the target storyboard, the facial elements including at least one of human eyes, forehead, cheek muscle groups, lip shape, and facial angles; determining multiple key frame images of the target storyboard based on the distribution characteristics of the first video time range of the at least one line of dialogue in the video time range of the target storyboard; adjusting the facial elements in the multiple key frame images based on the desired emotional information and the semantic emotional information of the at least one line of dialogue to obtain adjusted key frame images; generating multiple facial motion key frame identifiers based on the adjusted multiple key frame images.

[0057] Wherein, determining the multiple key frame images of the target storyboard according to the distribution characteristics of the first video time range of the at least one line of dialogue in the video time range of the target storyboard means that the distribution of at least one first video time range is determined in the video time range of the target storyboard, such as Figure 4 The distribution of the first video time range of the first line and the second line in the video time range of the target storyboard. Each line in at least one line contains at least two key frame images. The key frame image in the frame image sequence corresponding to each line can be the first frame, the last frame, or any frame in the frame image sequence.

[0058] For example, the key frame determination method may be: since the target video is a video of a person speaking, the audio spectrum of the person speaking is obtained, and the key frame is determined according to the distribution characteristics of the audio spectrum in each line. Figure 5 , Figure 5 The video time range of the target split screen and the voice audio diagram corresponding to the target split screen provided in the embodiment of the present application are shown in FIG. Figure 5 As shown, in the storyboard video segment A to which the first line belongs, the amplitude of the voice audio change is relatively stable, so at least two frame images are arbitrarily selected as key frames, for example, the first frame image and the second frame image can be used as key frames, the first frame image is the starting frame image in the storyboard video segment A, and the second frame image is the ending frame image in the storyboard video segment A; in the storyboard video segment B to which the second line belongs, the amplitude of the voice audio increases significantly between the third frame image and the fourth frame image, so the third frame image and the fourth frame image can be used as key frames.

[0059] Among them, the semantic emotional information of the lines refers to the emotional information that each line directly points to. The semantic emotional information of the lines and the expected emotional information can be the same type of emotional information, for example, the expected emotional information and the semantic emotional information of the lines both express happy emotional information; they can also be completely opposite emotional information, for example, the expected emotional information expresses happy emotional information, while the semantic emotional information of the lines expresses sad emotional information.

[0060] It can be seen that in this embodiment, by determining the key frame image of each line of dialogue and adjusting the facial elements of the key frame according to the desired emotional information and the semantic emotional information of the lines, so as to form a plurality of facial motion key frame identifiers, the facial movements of the characters can be accurately adapted to the desired emotional information and the semantic emotional information of the lines, thereby improving the user's viewing experience.

[0061] In some embodiments, the facial elements in the multiple key frame images are adjusted according to the expected emotional information and the semantic emotional information of the at least one line, including: determining the semantic emotional information of each line in the at least one line, and determining the line emotional intensity feature vector according to the semantic emotional information; determining the expected emotional intensity feature vector according to the expected emotional information; performing the following operations ac for the key frame image group for each line, wherein the key frame image group includes at least 2 frame images in the storyboard video clip to which each line belongs among the multiple key frame images: a, determining the first character facial emotional intensity feature vector corresponding to the storyboard video clip according to the emotional intensity feature vector of the currently processed line and the expected emotional intensity feature vector; b, determining the first indicator parameter adjustment amount of each facial element in the facial element in the storyboard video clip according to the first character facial emotional intensity feature vector; c, performing the element adjustment operation according to the relative time position relationship of each key frame image in the storyboard video clip and the first indicator parameter adjustment amount to obtain the adjusted key frame image group.

[0062] Among them, the lines' semantic emotional information and expected emotional information both express a type of emotional information, such as neutral, happy, angry, sad, afraid, disgusted, and surprised, and the emotional intensity feature vector (specifically, it can include the lines' emotional intensity feature vector and the expected emotional intensity feature vector) indicates the emotional intensity and emotional direction for a specific emotion. The emotional intensity can be evaluated as the magnitude of the emotional expressiveness, which can be numerically expressed, such as 0-10. The larger the value, the greater the emotional intensity and the stronger the external expression. The emotional direction refers to the relationship between the expected emotion and the lines' semantic emotion. The emotional direction includes positive and negative. The positive direction refers to the expected emotion and the lines' semantic emotion being the same type of emotional information, for example, the expected emotional intensity feature vector is happy (+7), and the lines' emotional intensity feature vector is happy (+5). The negative direction refers to the expected emotion and the lines' semantic emotion being opposite emotional information, for example, the expected emotional intensity feature vector is happy (+7), and the lines' emotional intensity feature vector is sad (+2).

[0063] Among them, the character's facial emotion intensity feature vector is used to characterize the performance of the adjusted facial movements, which is jointly determined by the lines emotion intensity feature vector and the expected emotion intensity feature vector.

[0064] Specifically, the character's facial emotion intensity feature vector = β0 + β1 × the line emotion intensity feature vector + β2 × the expected emotion intensity feature vector + ε, formula (1), where β0, β1, β2 are regression coefficients, and ε is the deviation. The specific value is determined according to the actual training results. Therefore, according to formula (1), regardless of whether the line semantic emotion information is consistent with the expected emotional information, the line semantic emotion information can be corrected so that the final character's facial emotion intensity feature vector is consistent with the expected emotional information.

[0065] The adjustment amount of the index parameter of the facial element refers to the numerical expression of the facial element performance. For example, the adjustment amount of the index parameter can be the change range of the facial element performance of the current facial action compared with the standard facial action, that is, the adjustment amount corresponding to the standard facial action to the current facial action, wherein the standard facial action is shown in FIG. Figure 6 , the standard facial movement is when the distance between the upper and lower eyelids is the largest ( Figure 6 The maximum upper and lower eyelid distance is L1), the mouth is closed, the forehead is stretched, and the face angle is facing forward.

[0066] There is a mapping relationship between the character's facial emotion intensity feature vector and the indicator parameter adjustment amount. The specific indicator parameter adjustment amount can be obtained through the character's facial emotion intensity feature vector, and the adjustment range of each facial element is determined according to the size of the indicator parameter adjustment amount.

[0067] It can be seen that in this embodiment, by quantifying the expected emotional information and the semantic emotional information of the lines, the expected emotion intensity feature vector and the line emotion intensity feature vector are obtained, and then the character's facial emotion intensity feature vector is obtained based on the expected emotion intensity feature vector and the line emotion intensity feature vector. The character's facial emotion intensity feature vector and the reference index adjustment amount of the facial element have a mapping relationship. Therefore, the above steps can accurately quantify the index parameter adjustment amount of the facial element in the current storyboard video clip.

[0068] In some embodiments, the element adjustment operation includes the following steps: for each frame image within the time range of the first video, the human eye state, forehead state, cheek muscle group state, mouth shape state and the facial angle are adjusted according to the corresponding indicator parameter adjustment amount, the indicator parameter adjustment amount of the human eye state includes the distance between the upper and lower eyelids; the indicator parameter adjustment amount of the forehead state includes the frown amplitude value; the indicator parameter adjustment amount of the mouth shape state includes the position of the corner of the mouth; the indicator parameter adjustment amount of the facial angle includes forward, upward, downward, left and right.

[0069] Among them, in the process of generating animated expressions, the change mode of facial elements corresponding to each type of emotion can form a certain template or paradigm. For example, for the emotion of "happy", as the emotion increases, the distance between the upper and lower eyelids of the human eye may gradually increase from the normal eye opening size until it reaches the maximum value; the wrinkles between the eyebrows relax, and the forehead is in a relaxed state; the position of the corners of the mouth gradually rises in the mouth shape state; and the facial angle gradually rises. For the emotion of "sad", as the emotion increases, the distance between the upper and lower eyelids of the human eye may gradually decrease from the normal eye opening size to a specific value; the frown of the eyebrows deepens; the position of the corners of the mouth gradually goes down in the mouth shape state; and the facial angle gradually droops. Similarly, for the emotions of anger, sadness, fear, disgust, and surprise, different degrees of emotion correspond to different changes in facial elements. Therefore, the character facial emotion intensity feature vector can be quantified into ten levels of 1-10, and the adjustment amount of the index parameters corresponding to different facial elements can be quantified into percentages within 0-100%. For example, the specific facial element adjustment corresponding to the facial emotion intensity feature vectors of different characters under the "happy" emotion information is shown in Table 1 below.

[0070] Table 1 Details of the adjustment of facial elements corresponding to the facial emotion intensity feature vectors of different characters under the "happy" emotion information

[0071]

[0072] As shown in Table 1, when the desired emotional information is "happy" emotional information, the corresponding facial emotional intensity feature vector has a value range of [+1, +10]. The adjustment amount of the index parameters of the human eye state, forehead state, mouth shape state, and facial angle can be found in Table 1. Among them, the adjustment amount of the index parameters is compared with the standard facial action (see Figure 6 ), and adjust the facial elements of different characters according to the basic parameters of the facial elements of the characters and Table 1. Among them, for the upper and lower eyelid distance of the human eye, 0 represents the closed state, and the 1%~100% eye opening range is the difference between the upper and lower eyelid distance after adjustment and the upper and lower eyelid distance of the current character under the maximum eye opening condition (such as Figure 6 For the frown amplitude value of the forehead, 0 means the state of stretching, that is, no frowning, and the frown amplitude value of 1-100% is determined comprehensively according to the eyebrow angle and eyebrow distance; for the mouth corner position of the mouth shape, 0 means that the mouth corner is neither upturned nor drooped, and is in a straight state. There is an upper limit for the upturn or droop of the mouth corner position, and 1%-100% is the difference between the adjusted mouth corner position and the upturn upper limit (such as Figure 6 For the facial angle, 0 is horizontal and facing forward, and the angle can be adjusted directly in different directions.

[0073] For example, if the range of the size of the eyes of character A is [0-24] mm, the upper limit of the upward position of the corners of the mouth is the position of the nose, and under the emotion of "happy", if the character's facial emotion intensity feature vector is +4, then the corresponding index parameter adjustment of the upper and lower eyelids is 51%~60%, the corresponding upper and lower eyelid distance is 12.24-14.4mm, the frown amplitude value of the forehead is 0 (i.e., stretched state), the corners of the mouth are raised to 31%-40% of the distance between the mouth and the nose, and the facial angle is rotated upward by 16°-20°.

[0074] In addition, for other emotions such as neutral, angry, sad, afraid, disgusted, and surprised, the specific facial element adjustment corresponding to the facial emotion intensity feature vector of the character also has a template paradigm as shown in Table 1. The template paradigm of the facial element under each emotion can be generated according to actual needs, or the facial expression in the actual emotional expression process of humans can be analyzed by big data to determine the template paradigm of the facial element under each emotion.

[0075] It can be seen that in this embodiment, by template-ing the facial element adjustment conditions corresponding to the facial emotion intensity feature vectors of the characters under different emotions, when the facial emotion intensity feature vectors of the characters are known, the corresponding facial element templates can be quickly used as key frame images to improve video processing efficiency.

[0076] In some embodiments, after performing the element adjustment operation according to the relative time position relationship of each key frame image in the key frame image group in the storyboard video clip and the first indicator parameter adjustment amount to obtain the adjusted key frame image group, the method further includes: performing the following operations on the remaining frame image groups except the key frame image group in each line of dialogue: obtaining the first character facial action state in the first key frame image, and obtaining the second character facial action state in the second key frame image, the first key frame image is the first key frame image in the key frame image group, the second key frame image is the last key frame image in the key frame image group, and the character facial action state is used to characterize the indicator parameter adjustment amount of the facial element of the character in the current frame image; performing the element adjustment operation according to the expected emotion intensity feature vector and the first character facial action state to obtain the adjusted first remaining frame image group; performing the element adjustment operation according to the expected emotion intensity feature vector and the second character facial action state to obtain the adjusted second remaining frame image group.

[0077] Among them, see Fig. 9 , Fig. 9 Another schematic diagram of the distribution of the lines in the video range of the target storyboard provided in the embodiment of the present application is as follows: Fig. 9As shown, for the first line, the frame image before the first key frame image in the frame image sequence of the target storyboard is the first remaining frame image group, and for the frame images of the first remaining frame image group, the first character facial action state of the character in the first key frame image is obtained, and the first character facial action state is analyzed to obtain the index parameter adjustment amount of the facial element of the first key frame image; the facial element of the first key frame image is adjusted according to the index parameter adjustment amount of the first key frame image and the expected emotion intensity feature vector to obtain the adjusted first remaining frame image group.

[0078] For the second line, the frame images before the third key frame image and after the second key frame image in the frame image sequence of the target storyboard are the second remaining frame image group. For the frame images of the second remaining frame image group, the second character facial action state of the character in the third key frame image is obtained, and the second character facial action state is analyzed to obtain the index parameter adjustment amount of the facial elements of the third key frame image; the facial elements of the third key frame image are adjusted according to the index parameter adjustment amount of the third key frame image and the expected emotion intensity feature vector to obtain the adjusted second remaining frame image group; the frame images after the fourth key frame image in the frame image sequence of the target storyboard are the third remaining frame image group. For the frame images of the third remaining frame image group, the third character facial action state of the character in the fourth key frame image is obtained, and the third character facial action state is analyzed to obtain the index parameter adjustment amount of the facial elements of the fourth key frame image; the facial elements of the fourth key frame image are adjusted according to the index parameter adjustment amount of the fourth key frame image and the expected emotion intensity feature vector to obtain the adjusted third remaining frame image group.

[0079] It can be seen that in this embodiment, by adjusting the remaining frame image groups through the key frame images of each line in the target storyboard and the expected emotion intensity feature vector, it can be ensured that the remaining frame image groups other than the key frame image group in the target storyboard can also adjust the facial elements according to the expected emotion intensity feature vector and the line emotion intensity feature vector, thereby ensuring that the facial elements of each frame image in the target storyboard change smoothly.

[0080] In some embodiments, the element adjustment operation is performed according to the expected emotion intensity feature vector and the first character's facial action state to obtain an adjusted first remaining frame image group, including: determining the second character's facial emotion intensity feature vector in the storyboard video clip according to the indicator parameter adjustment amount corresponding to the first character's facial action state and the expected emotion intensity feature vector; determining the second indicator parameter adjustment amount of each facial element in the storyboard video clip according to the second character's facial emotion intensity feature vector; performing the element adjustment operation according to the relative time position relationship of each remaining frame image in the first remaining frame image group in the storyboard video clip and the two indicator parameter adjustment amounts to obtain the adjusted first remaining frame image group.

[0081] It can be seen that in this embodiment, by adjusting the remaining frame image groups through the first key frame image and the expected emotion intensity feature vector in the target storyboard, it can be ensured that the remaining frame image groups other than the key frame image group in the target storyboard can also adjust the facial elements according to the expected emotion intensity feature vector and the lines emotion intensity feature vector, thereby ensuring that the facial elements of each frame image in the target storyboard change smoothly.

[0082] In some embodiments, the element adjustment operation is performed according to the expected emotion intensity feature vector and the first character's facial action state to obtain an adjusted second remaining frame image group, including: determining the third character's facial emotion intensity feature vector in the storyboard video clip according to the indicator parameter adjustment amount corresponding to the second character's facial action state and the expected emotion intensity feature vector; determining the third indicator parameter adjustment amount of each facial element in the storyboard video clip according to the third character's facial emotion intensity feature vector; performing the element adjustment operation according to the relative time position relationship of each remaining frame image in the second remaining frame image group in the storyboard video clip and the three indicator parameter adjustment amounts to obtain the adjusted second remaining frame image group.

[0083] It can be seen that in this embodiment, by adjusting the remaining frame image groups through the second key frame image and the expected emotion intensity feature vector in the target storyboard, it can be ensured that the remaining frame image groups other than the key frame image group in the target storyboard can also adjust the facial elements according to the expected emotion intensity feature vector and the lines emotion intensity feature vector, thereby ensuring that the facial elements of each frame image in the target storyboard change smoothly.

[0084] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium may be a solid-state hard disk.

[0085] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps of any method described in the above method embodiments, and the above computer includes an electronic device.

[0086] An embodiment of the present application also provides an AIGC skit generation information processing system, which includes a computer storage medium that stores a computer program for electronic data exchange, and the computer program enables the computer to execute part or all of the steps of any method recorded in the above method embodiment, and the above computer includes an electronic device.

[0087] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0088] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0089] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0090] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0091] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0092] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the above-mentioned methods in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0093] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable memory, which may include a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0094] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An information processing method for generating AIGC skits, It is characterized in that The method comprises: Detecting a first selection operation on an expression control for a target storyboard of a target video, obtaining a character speech text of the target storyboard, wherein the character speech text includes at least one line; and obtaining expected emotional information of the character plot of the target storyboard, wherein the expected emotional information refers to the overall emotional information of the character speech text; Acquire facial elements of a character in the target storyboard, wherein the facial elements include at least one of human eyes, forehead, cheek muscle group, mouth shape, and facial angle; and determine a plurality of key frame images of the target storyboard according to distribution characteristics of the first video time range of the at least one line of dialogue in the video time range of the target storyboard; Determine the speech semantic emotion information of each speech in the at least one speech, determine the speech emotion intensity feature vector according to the speech semantic emotion information; and determine the expected emotion intensity feature vector according to the expected emotion information; wherein the emotion intensity feature vector is used to characterize the emotion intensity and emotion direction of the emotion, and the emotion direction includes positive and negative, the positive direction means that the expected emotion information and the speech semantic emotion information are the same type of emotion information, and the negative direction means that the expected emotion information and the speech semantic emotion information are opposite emotion information; For the key frame image group of each line of dialogue, the following operations ac are performed, wherein the key frame image group includes frame images between at least two key frame images in the storyboard video segment to which each line of dialogue belongs among the multiple key frame images: a. Determine the first character facial emotion intensity feature vector corresponding to the storyboard video clip according to the speech emotion intensity feature vector of the currently processed speech and the expected emotion intensity feature vector and formula (1); wherein formula (1) is character facial emotion intensity feature vector = β0 + β1 × speech emotion intensity feature vector + β2 × expected emotion intensity feature vector + ε, β0, β1, β2 are regression coefficients, and ε is the deviation; b. determining a first indicator parameter adjustment amount of each facial element in the storyboard video clip according to the first character's facial emotion intensity feature vector; c. performing an element adjustment operation according to the relative time position relationship of each key frame image in the key frame image group in the storyboard video segment and the first indicator parameter adjustment amount to obtain an adjusted key frame image group; For the remaining frame image groups except the key frame image group in each line, perform the following operations: Acquire a facial action state of a first person in a first key frame image, and acquire a facial action state of a second person in a second key frame image, wherein the first key frame image is the first key frame image in the key frame image group, and the second key frame image is the last key frame image in the key frame image group, and the facial action state of the person is used to characterize the index parameter adjustment amount of the facial element of the person in the current frame image; Performing the element adjustment operation according to the expected emotion intensity feature vector and the first character's facial action state to obtain an adjusted first residual frame image group; Performing the element adjustment operation according to the expected emotion intensity feature vector and the second character's facial action state to obtain an adjusted second residual frame image group; Generate multiple facial motion key frame identifiers according to the adjusted key frame image group, the adjusted first remaining frame image group, and the adjusted second remaining frame image group, wherein a single facial motion key frame identifier is a set of frame image sequences used to characterize the dynamic change process of the facial expression of the character between key frames, and the video time ranges corresponding to the multiple facial motion key frame identifiers include the first video time range of the at least one line of dialogue, and include the second video time range and / or the third video time range, the second video time range is the time period before the first video time range, and the third video time range is the time period after the first video time range; determining at least two target facial motion key frame identifiers according to a second selection operation of the user on the plurality of facial motion key frame identifiers, wherein the video time ranges corresponding to the at least two target facial motion key frame identifiers at least include the first video time range; The target storyboard is adjusted according to the at least two target facial motion key frame identifiers to obtain an updated target storyboard.

2. The method according to claim 1, It is characterized in that The step of performing the element adjustment operation according to the expected emotion intensity feature vector and the first character facial action state to obtain an adjusted first remaining frame image group includes: Determining a second character's facial emotion intensity feature vector in the storyboard video clip according to the index parameter adjustment amount corresponding to the first character's facial action state and the expected emotion intensity feature vector; Determining a second indicator parameter adjustment amount of each facial element in the facial elements in the storyboard video clip according to the second character's facial emotion intensity feature vector; The element adjustment operation is performed according to the relative time position relationship of each remaining frame image in the first remaining frame image group in the storyboard video segment and the two indicator parameter adjustment amounts to obtain the adjusted first remaining frame image group.

3. The method according to claim 1, It is characterized in that The step of performing the element adjustment operation according to the expected emotion intensity feature vector and the second character facial action state to obtain an adjusted second residual frame image group includes: Determining a facial emotion intensity feature vector of a third character in the storyboard video clip according to the index parameter adjustment amount corresponding to the facial action state of the second character and the expected emotion intensity feature vector; Determining a third indicator parameter adjustment amount of each facial element in the facial elements in the storyboard video clip according to the third character facial emotion intensity feature vector; The element adjustment operation is performed according to the relative time position relationship of each remaining frame image in the second remaining frame image group in the storyboard video segment and the three-index parameter adjustment amount to obtain the adjusted second remaining frame image group.

4. The method according to any one of claims 1 to 3, It is characterized in that The element adjustment operation includes the following steps: For each frame image within the time range of the first video, the human eye state, forehead state, cheek muscle group state, lip shape state and the facial angle are adjusted according to the corresponding index parameter adjustment amount. The adjustment amount of the index parameter of the human eye state includes the distance between the upper and lower eyelids; The adjustment amount of the index parameter of the forehead state includes a frown amplitude value; The adjustment amount of the index parameter of the mouth shape state includes the position of the mouth corners and the degree of opening and closing of the mouth shape; The index parameter adjustment amount of the facial angle includes forward, upward, downward, left, and right.

5. An AIGC skit generation information processing system, It is characterized in that The method comprises a server and a user device, wherein the server is used to execute the steps of the method according to any one of claims 1 to 4.

6. A server, It is characterized in that The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program or instructions, and when the computer program or instructions are executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Citation Information

Cited By

  • Video animation intelligent generation system based on AIGC

    CN119383425A

  • An intelligent video animation generation system based on AIGC

    CN119383425B

  • Suzhou evaluation long booklist generation method

    CN121389997A

  • A Method for Generating Long-Form Bibliographies of Suzhou Pingtan

    CN121389997B