Image processing method, device, equipment and storage medium

By analyzing the text content and generating target animations, the character's lips are dynamically changed, and the problem of monotonous video content in the prior art is solved, and the diversity and fun of broadcast information is improved.

CN114782597BActive Publication Date: 2024-11-22BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210356942.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-06
Publication Date
2024-11-22
Estimated Expiration
2042-04-06

AI Technical Summary

Technical Problem

The video content used in the prior art for broadcasting information is too monotonous and lacks diversity, making it difficult to effectively convey text information.

Method used

By obtaining text information and character images, analyzing text content to determine the lip animation sequence, and image processing of the character images to generate target animations, so that the character lips change dynamically based on the text content, improving the diversity of broadcast information.

Benefits of technology

The content of the broadcast text through target animation is realized, and the lips of the character image are dynamically changed, which significantly improves the diversity of the content of the broadcast text and enhances the fun and interactiveness of the video content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114782597B_ABST
    Figure CN114782597B_ABST
Patent Text Reader

Abstract

The present disclosure relates to an image processing method, device, equipment and storage medium, which relates to the field of network technology and can improve the diversity of broadcast information. The specific scheme includes: obtaining text information and character images, and the text information includes text content. Parsing the text content, determining a first lip animation sequence, the first lip animation sequence includes M first lip animations, and M is a positive integer. Performing image processing on the character image to generate a target animation, and the target animation includes the lips of the processed character image executing M first lip animations based on the text content.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of network technology, and in particular to an image processing method, device, equipment and storage medium. Background Art

[0002] With the development of network technology, video has gradually become an important carrier for disseminating information (such as news, mood status, introduction of items, etc.). More and more users choose to convert text information into videos and spread the content of text information through video broadcasts.

[0003] Currently, videos can be composed of text and pictures, and speech content can be generated using text to speech (TTS) technology, and the generated speech content can be used as audio in the video. However, the video content currently used to broadcast information is too monotonous. Therefore, how to improve the diversity of broadcast information has become an urgent problem to be solved. Summary of the invention

[0004] The present disclosure provides an image processing method, device, equipment and storage medium, which can improve the diversity of broadcast information. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a method for processing an image is provided, the method comprising:

[0006] Acquire text information and a character image, wherein the text information includes text content. Parse the text content to determine a first lip animation sequence, wherein the first lip animation sequence includes M first lip animations, where M is a positive integer. Perform image processing on the character image to generate a target animation, wherein the target animation includes the lips of the processed character image executing the M first lip animations based on the text content.

[0007] Optionally, the above-mentioned method of "parsing the text content to determine the first lip animation sequence" includes: parsing the text content to determine a phoneme sequence, the phoneme sequence includes P first-class phonemes and Q second-class phonemes, the lip animation corresponding to the first-class phonemes is an open-mouth type animation, and the lip animation corresponding to the second-class phonemes is a closed-mouth type animation, and P and Q are both positive integers. According to the P first-class phonemes and the Q second-class phonemes, the first lip animation sequence is determined, and the first lip animation sequence includes: P first open-mouth animations corresponding to the P first-class phonemes, Q first closed-mouth animations corresponding to the Q second-class phonemes, and P+Q=M.

[0008] Optionally, the above-mentioned method of "generating a first lip animation sequence based on P first-category phonemes and Q second-category phonemes" includes: determining a second lip animation sequence based on the P first-category phonemes and the Q second-category phonemes, the second lip animation sequence including P second opening animations and Q second closing animations, the P second opening animations corresponding to the P first opening animations, and the Q second closing animations corresponding to the Q first closing animations. A first lip animation sequence is generated based on every two adjacent lip animations in the second lip animation sequence and a preset strategy, the first lip animation sequence being the second lip animation sequence adjusted according to the preset strategy, and the preset strategy is used to adjust the duration of the lip animation.

[0009] Optionally, the second lip-sync animation sequence includes a first sub-lip-sync animation and a second sub-lip-sync animation, and the first sub-lip-sync animation and the second sub-lip-sync animation are any two adjacent lip-sync animations in the second lip-sync animation sequence; the preset strategy includes at least one of the following: if the first sub-lip-sync animation is an open-mouth type animation and the second sub-lip-sync animation is a closed-mouth type animation, then the duration of the second sub-lip-sync animation is reduced. If the first sub-lip-sync animation and the second sub-lip-sync animation are both open-mouth type animations, then the duration of the first sub-lip-sync animation and the duration of the second sub-lip-sync animation are reduced.

[0010] Optionally, the text information further includes N action identifiers, the action identifiers correspond to action animations, and N is a positive integer. The image processing method further includes: parsing the N action identifiers to determine a first action animation sequence, the first action animation sequence includes N first action animations. The target animation also includes the body parts of the processed character image executing the N first action animations.

[0011] Optionally, the N action identifiers include first-category identifiers and second-category identifiers, the action animation corresponding to the first-category identifier is a first-category action animation, the action animation corresponding to the second-category identifier is a second-category action animation, and the movement amplitude of the first-category action animation is greater than the movement amplitude of the second-category action animation.

[0012] Optionally, the above-mentioned method of "parsing N action identifiers to determine the first action animation sequence" includes: parsing N action identifiers to determine the second action animation sequence, the second action animation sequence includes N second action animations. Among them, there is at least one second-class identifier between every two first-class identifiers in the N action identifiers, and there is at least one second-class action animation between every two first-class action animations in the N second action animations; performing a first operation on each second action animation in the N second action animations to generate a first action animation sequence; the first operation includes: if the first sub-action animation is a first-class action animation, then adjusting the action animation of the first preset time period in the second sub-action animation to the second-class action animation to obtain the adjusted second sub-action animation, and the first sub-action animation and the adjusted second sub-action animation are animations in the first action animation sequence. Among them, the first sub-action animation and the second sub-action animation are any two adjacent animations in the N second action animations, and the first preset time period is the time period of executing the second sub-action animation that is immediately adjacent to the execution of the first sub-action animation.

[0013] Optionally, each of the M first lip animations includes multiple frames of lip action images. The above-mentioned method of "performing image processing on a character image to generate a target animation" includes: determining the lips of the character image according to a preset detection algorithm and the character image. According to each frame of the lip action image in each first lip animation, adjusting the lip action of the character image to generate the target animation.

[0014] Optionally, the image processing method further includes: obtaining a preset video, adding a target animation to the preset video, and generating a target video, wherein the target video includes the target animation.

[0015] According to a second aspect of the present disclosure, there is provided an image processing device, the image processing device comprising: an acquisition unit and a processing unit.

[0016] The acquisition unit is configured to acquire text information and a character image, wherein the text information includes text content. The processing unit is configured to parse the text content and determine a first lip animation sequence, wherein the first lip animation sequence includes M first lip animations, where M is a positive integer. The processing unit is further configured to perform image processing on the character image and generate a target animation, wherein the target animation includes the lips of the processed character image executing M first lip animations based on the text content.

[0017] Optionally, the processing unit is specifically configured to perform parsing of the text content to determine a phoneme sequence, the phoneme sequence includes P first-class phonemes and Q second-class phonemes, the lip animation corresponding to the first-class phonemes is an open-mouth type animation, and the lip animation corresponding to the second-class phonemes is a closed-mouth type animation, and P and Q are both positive integers. The processing unit is specifically configured to perform, according to the P first-class phonemes and the Q second-class phonemes, determining a first lip animation sequence, the first lip animation sequence includes: P first open-mouth animations corresponding to the P first-class phonemes, Q first closed-mouth animations corresponding to the Q second-class phonemes, and P+Q=M.

[0018] Optionally, the processing unit is specifically configured to determine a second lip animation sequence based on P first-category phonemes and Q second-category phonemes, the second lip animation sequence including P second opening animations and Q second closing animations; the P second opening animations correspond to the P first opening animations, and the Q second closing animations correspond to the Q first closing animations. The processing unit is specifically configured to generate a first lip animation sequence based on every two adjacent lip animations in the second lip animation sequence and a preset strategy, the first lip animation sequence being the second lip animation sequence adjusted according to the preset strategy, and the preset strategy is used to adjust the duration of the lip animation.

[0019] Optionally, the second lip-sync animation sequence includes a first sub-lip-sync animation and a second sub-lip-sync animation, and the first sub-lip-sync animation and the second sub-lip-sync animation are any two adjacent lip-sync animations in the second lip-sync animation sequence; the preset strategy includes at least one of the following: if the first sub-lip-sync animation is an open-mouth type animation and the second sub-lip-sync animation is a closed-mouth type animation, then the duration of the second sub-lip-sync animation is reduced. If the first sub-lip-sync animation and the second sub-lip-sync animation are both open-mouth type animations, then the duration of the first sub-lip-sync animation and the duration of the second sub-lip-sync animation are reduced.

[0020] Optionally, the text information further includes N action identifiers, the action identifiers correspond to action animations, and N is a positive integer. The processing unit is further configured to parse the N action identifiers to determine a first action animation sequence, the first action animation sequence includes N first action animations; wherein the target animation also includes the N first action animations executed by the limbs of the processed character image.

[0021] Optionally, the N action identifiers include first-category identifiers and second-category identifiers, the action animation corresponding to the first-category identifier is a first-category action animation, the action animation corresponding to the second-category identifier is a second-category action animation, and the movement amplitude of the first-category action animation is greater than the movement amplitude of the second-category action animation.

[0022] Optionally, the processing unit is specifically configured to perform parsing of N action identifiers to determine a second action animation sequence, wherein the second action animation sequence includes N second action animations. Among the N action identifiers, at least one second-class identifier is included between every two first-class identifiers, and among the N second action animations, at least one second-class action animation is included between every two first-class action animations. The processing unit is specifically configured to perform a first operation on each second action animation in the N second action animations to generate a first action animation sequence; the first operation includes: if the first sub-action animation is a first-class action animation, then the action animation of the first preset time period in the second sub-action animation is adjusted to the second-class action animation, and the adjusted second sub-action animation is obtained, and the first sub-action animation and the adjusted second sub-action animation are animations in the first action animation sequence; wherein the first sub-action animation and the second sub-action animation are any two adjacent animations in the N second action animations, and the first preset time period is the time period of executing the second sub-action animation that is adjacent to the execution of the first sub-action animation.

[0023] Optionally, each of the M first lip animations includes multiple frames of lip action images. The processing unit is further configured to determine the lips of the character image according to a preset detection algorithm and the character image. The processing unit is further configured to adjust the lip action of the character image according to each frame of the lip action image in each first lip animation to generate a target animation.

[0024] Optionally, the acquisition unit is further configured to acquire a preset video. The processing unit is further configured to add a target animation to the preset video to generate a target video, wherein the target video includes the target animation.

[0025] According to a third aspect of the present disclosure, a server is provided, the server comprising:

[0026] A processor. A memory for storing processor executable instructions. The processor is configured to execute the instructions to implement any one of the optional image processing methods in the first aspect above.

[0027] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which instructions are stored. When the instructions in the computer-readable storage medium are executed by a processor of a server, the server is enabled to execute any one of the optional image processing methods in the above-mentioned first aspect.

[0028] According to a fifth aspect of the present disclosure, there is provided a computer program product, comprising instructions, which, when executed on a computer, enable the computer to execute any one of the optional image processing methods in the first aspect.

[0029] According to a sixth aspect of the present disclosure, the present disclosure provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run a computer program or instructions to implement the image processing method described in the first aspect and any possible implementation method of the first aspect.

[0030] The technical solution provided by the present disclosure brings at least the following beneficial effects: obtaining text information and character images, the text information including text content. Afterwards, parsing the text content to determine a first lip animation sequence, the first lip animation sequence including M first lip animations. Then performing image processing on the character image to obtain a target animation, the target animation including the lips of the processed character image executing M first lip animations based on the text content. In this way, the text content can be broadcasted through the target animation, and the lips of the character image can change dynamically based on the text content, thereby improving the diversity of the broadcasted text content.

[0031] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0033] Figure 1 is a schematic diagram of a communication system according to an exemplary embodiment;

[0034] Figure 2 is a flowchart of a method for processing an image according to an exemplary embodiment;

[0035] Figure 3 is an example diagram of an image processing method according to an exemplary embodiment;

[0036] Figure 4 is a flowchart of another image processing method according to an exemplary embodiment;

[0037] Figure 5 is a flowchart of another image processing method according to an exemplary embodiment;

[0038] Figure 6 is a flowchart of another image processing method according to an exemplary embodiment;

[0039] Figure 7 is a schematic structural diagram of an image processing device according to an exemplary embodiment;

[0040] Figure 8 It is a schematic structural diagram of another image processing device according to an exemplary embodiment. DETAILED DESCRIPTION

[0041] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0042] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0043] It should be noted that the user information involved in this disclosure (including but not limited to user device information, user personal information, etc.) is all information authorized by the user or fully authorized by all parties.

[0044] First, the application scenarios of the embodiments of the present disclosure are introduced.

[0045] The image processing method of the disclosed embodiment is applied to the scene of voice playback. In the related art, a video can be composed of text and pictures, and voice content can be generated using speech synthesis (Text to Speech, TTS) technology, and the generated voice content can be used as audio in the video. However, the video content currently used for broadcasting information is too monotonous. Therefore, how to improve the diversity of videos for broadcasting information has become a problem to be solved urgently.

[0046] In order to solve the above problems, the embodiment of the present disclosure provides an image processing method to obtain text content and character images. The text content is parsed to determine multiple lip animations. Afterwards, the character image is processed to generate a target animation, which includes the lips of the processed character image performing multiple lip animations based on the text content. In this way, the lips of the character image can change dynamically based on the text content, thereby improving the diversity of the text content being reported.

[0047] Figure 1 A schematic diagram of a communication system provided in an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the communication system may include: a server 01 and a terminal 02. The server 01 may establish a connection with the terminal 02 via a wired network or a wireless network.

[0048] Among them, server 01 can be a data server of some multimedia resource service platforms, which can be used to store and process multimedia resources. For example, the multimedia resource service platform can be a short video application service platform, a news service platform, a live broadcast service platform, a shopping service platform, a takeaway service platform, a sharing service platform, a functional website, etc. Among them, the multimedia resources provided by the short video application service platform can be some short video works, the multimedia resources provided by the news service platform can be some news information, and the multimedia resources provided by the live broadcast service platform can be live broadcast works, etc., and the rest will not be repeated one by one. The present disclosure does not limit the specific type of the multimedia resource service platform.

[0049] As a possible implementation, the server 01 may include multiple multimedia resource service platforms, each of which uniquely corresponds to an application. The application is installed on the terminal and displays multimedia resources on the content display interface of the terminal 02. The server 01 is mainly used to store relevant data of the content community application installed on the terminal 02, and can send the corresponding data (which can be called the file to be transmitted) to the terminal when receiving a data acquisition request sent by the terminal 02.

[0050] In some embodiments, the server 01 may be a single server, or a server cluster composed of multiple servers. In some implementations, the server cluster may also be a distributed cluster. The present disclosure does not limit the specific implementation of the server 01.

[0051] In some other embodiments, the server 01 may also include a database or be connected to a database, and the multimedia resources of the multimedia resource service platform may be stored in the database. The terminal 02 may access the multimedia resources in the database through the server 01 .

[0052] Terminal 02 may be a mobile phone, tablet computer, desktop, laptop, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), augmented reality (AR)\virtual reality (VR) device, etc., which can install and use content community applications (such as Kuaishou), and the present disclosure does not impose any special restrictions on the specific form of the terminal. It can interact with the user through one or more methods such as keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device.

[0053] Optionally, the above Figure 1 In the communication system shown, the server 01 may be connected to at least one terminal 02. The present disclosure does not limit the number and type of the terminal 02.

[0054] It should be noted that Figure 1 The server 01 or the terminal 02 in the embodiment can be used as an execution subject to execute the method provided by the embodiment of the present disclosure. The following takes the server 01 as an execution subject as an example to illustrate the method provided by the embodiment of the present disclosure.

[0055] After introducing the application scenarios and implementation environments of the embodiments of the present disclosure, Figure 1 The implementation environment shown is used to introduce in detail the image processing method provided by the embodiment of the present disclosure.

[0056] Figure 2 FIG. 1 is a flowchart of a method for processing an image according to an exemplary embodiment. Figure 2 As shown, the method may include steps 201 to 203.

[0057] 201. The server obtains text information and character images.

[0058] The text information includes text content.

[0059] It should be noted that the embodiments of the present disclosure do not limit the text content. For example, the text content may be news content. For another example, the text content may be hot content. For another example, the text content may be story content. For another example, the text content may specifically be "Hello everyone, today I will tell you a story about cat and mouse. The following is the specific story content."

[0060] It should be noted that the embodiments of the present disclosure do not limit the character image. For example, the character image may be a cartoon character image (i.e., a virtual character image). For another example, the character image may be a character image of a real user. Optionally, the character image in the embodiments of the present disclosure may also be a cartoon animal image.

[0061] In a possible implementation, the terminal may receive text information and a character image input by a user. Afterwards, the terminal may send the text information and the character image to a server. The server may receive the text information and the character image from the terminal.

[0062] 202. The server parses the text content and determines a first lip animation sequence.

[0063] The first lip animation sequence includes M first lip animations. The text content includes M phonemes. The M phonemes correspond to the M first lip animations, and M is a positive integer.

[0064] In a possible design, all the preset mouth shape animations corresponding to each phoneme are stored in the server.

[0065] Exemplarily, if all the phonemes include: phoneme a, phoneme b, phoneme c, and phoneme d, the mouth shape animation corresponding to phoneme a is mouth shape animation a, the mouth shape animation corresponding to phoneme b is mouth shape animation b, the mouth shape animation corresponding to phoneme c is mouth shape animation c, and the mouth shape animation corresponding to phoneme d is mouth shape animation d.

[0066] It should be noted that in the embodiments of the present disclosure, each mouth shape animation is composed of multiple frames of mouth shape action images. The mouth shape animation can be pre-produced by developers (such as animators) and input into the server.

[0067] In a possible implementation manner, the server processes the text content through TTS technology to generate the voice content corresponding to the text content. Then, the server processes the voice content to determine M phonemes. The server determines the first mouth shape animation sequence from the preset mouth shape animations according to the M phonemes, and the first mouth shape animation sequence is composed of M mouth shape animations corresponding to the M phonemes.

[0068] Exemplarily, the text content is "Hello everyone". Among them, "da" is composed of phoneme a and phoneme b, "jia" is composed of phoneme c, phoneme d, and phoneme a, and "hao" is composed of phoneme e, phoneme a, and phoneme f. Among them, the mouth shape animation corresponding to phoneme a is mouth shape animation a, the mouth shape animation corresponding to phoneme b is mouth shape animation b, the mouth shape animation corresponding to phoneme c is mouth shape animation c, the mouth shape animation corresponding to phoneme d is mouth shape animation d, the mouth shape animation corresponding to phoneme e is mouth shape animation e, and the mouth shape animation corresponding to phoneme f is mouth shape animation f. Then the first mouth shape animation sequence is: mouth shape animation a - mouth shape animation b - mouth shape animation c - mouth shape animation d - mouth shape animation a - mouth shape animation e - mouth shape animation a - mouth shape animation f.

[0069] 203. The server performs image processing on the person image to generate a target animation.

[0070] Among them, the target animation includes the lips of the processed person image performing M first mouth shape animations based on the text content. That is to say, the target animation can broadcast the text content according to the first animation sequence.

[0071] In a possible implementation manner, the server determines the lips of the person image according to a preset detection algorithm and the person image. The preset detection algorithm is used to identify the lips in the person image.

[0072] It should be noted that the embodiments of the present disclosure do not limit the preset detection algorithm. For example, the preset detection algorithm can be a target detection algorithm. Another example is that the preset detection algorithm can be a clustering algorithm (such as the k-means clustering algorithm). Taking the following preset detection algorithm as the k-means clustering algorithm as an example, the process of determining the lips of the person image is introduced.

[0073] For example, the lips of the person image include multiple (for example, 3) anchor points with the target color (such as pure green). The server can obtain the color information of all the pixels in the person image, and then determine the positions of the 3 pure green anchor points by using the k-means clustering algorithm. Then, the server determines the lips of the person image based on the positions of the 3 anchor points. Figure 3 As shown, the character image 301 includes anchor point 302 , anchor point 303 , and anchor point 304 .

[0074] Afterwards, the server adjusts the lip movements of the character image according to each frame of the lip movement image in each first lip animation to obtain the target animation.

[0075] Exemplarily, the server adjusts the lip movements of the character image according to each frame of the lip movement image in each first lip animation through affine transformation to generate multiple frames of character animation images. The lip movements of each frame of the character animation image in the multiple frames of character animation images correspond to each frame of the lip movement image. Finally, the server combines the multiple frames of character animation images to generate the target animation.

[0076] The technical solution provided by the above embodiment brings at least the following beneficial effects: each of the M first lip animations includes multiple frames of lip action images. According to the preset detection algorithm and the character image, the lips of the character image are determined. Afterwards, according to each frame of the lip action image in each first lip animation, the lip action of the character image is adjusted to obtain the target animation. In this way, the target animation can execute the first lip animation, and the lips of the character image can change dynamically based on the text content, thereby improving the diversity of the broadcast text content.

[0077] In a possible design, the execution order and execution duration of the M first lip animations in the first animation sequence correspond to the M phonemes of the text content.

[0078] In a possible implementation, the server determines the broadcasting time of each of the M phonemes according to the phoneme duration detection algorithm and the text content, wherein the broadcasting time includes: the time when the broadcasting starts, the duration of the broadcasting, and the time when the broadcasting ends.

[0079] It should be noted that the embodiments of the present disclosure do not limit the phoneme duration detection algorithm. For example, the phoneme duration detection algorithm may be the Fastspeech algorithm. For another example, the phoneme duration detection algorithm may be the Tacotron algorithm.

[0080] For example, if the text content is composed of phoneme a-phoneme b-phoneme c. By parsing the text content with the Fastspeech algorithm, it can be determined that the broadcast time of phoneme a is time a-time b, the broadcast time of phoneme b is time c-time d, and the broadcast time of phoneme c is time e-time f. Then the playback time of the lip animation corresponding to phoneme a (such as animation a) is time a-time b, the playback time of the lip animation corresponding to phoneme b (such as animation b) is time c-time d, and the playback time of the lip animation corresponding to phoneme c (such as animation c) is time e-time f.

[0081] It should be noted that time b and time c can be the same or different, and similarly, time d and time e can be the same or different. That is, phoneme a, phoneme b and phoneme c can be broadcasted continuously, animation a, animation b and animation c can be played continuously, or phoneme a, phoneme b and phoneme c can be broadcasted at intervals, that is, animation a, animation b and animation c can be played at intervals.

[0082] The technical solution provided by the above embodiment brings at least the following beneficial effects: obtaining text information and character images, the text information including text content. Afterwards, parsing the text content to determine a first lip animation sequence, the first lip animation sequence including M first lip animations. Then, image processing is performed on the character image to obtain a target animation, the target animation including the lips of the processed character image executing M first lip animations based on the text content. In this way, the text content can be broadcasted through the target animation, and the lips of the character image can change dynamically based on the text content, thereby improving the diversity of the broadcasted text content.

[0083] In an practicable manner, the server may obtain a preset video, and then add the target animation to the preset video to generate the target video, wherein the target video includes the target animation.

[0084] In one practicable manner, the server pastes the target animation onto a preset video through a transparent channel to generate a target video. The server may also add the voice content corresponding to the text content to the preset video. In other words, the target video may also include the voice content corresponding to the text content.

[0085] The technical solution provided by the above embodiment brings at least the following beneficial effects: obtaining a preset video. Adding a target animation to the preset video to generate a target video, wherein the target video includes the target animation. In this way, the target animation can be added to the target video, and the target animation can dynamically broadcast text content, thereby improving the diversity of the broadcast text content.

[0086] In one practicable manner, if Figure 4 As shown, step 202 may include steps 401 and 402.

[0087] 401. The server parses the text content and determines the phoneme sequence.

[0088] The phoneme sequence includes P first-category phonemes and Q second-category phonemes. The lip animation corresponding to the first-category phonemes is an open-mouth type animation, and the lip animation corresponding to the second-category phonemes is a closed-mouth type animation. P and Q are both positive integers, and P+Q=M.

[0089] It should be noted that the first category of phonemes is not limited in the embodiments of the present disclosure. For example, the first category of phonemes may be vowel phonemes (such as a, e, i, o, u). For another example, the first category of phonemes may be a phoneme combination including vowel phonemes (such as uan, uang, ai, ei, etc.). The second category of phonemes is not limited in the embodiments of the present disclosure. For example, the second category of phonemes may be consonant phonemes (such as h, m, n, q, etc.). For another example, the second category of phonemes may be silent phonemes (i.e., there is no text content in this time period, such as commas, periods, exclamation marks, etc.).

[0090] For example, the text content is "hello everyone", and the factor sequence is: daj-ia-h-ao. The phoneme sequence includes three first-class phonemes, namely phoneme a, phoneme ia and phoneme ao. The phoneme sequence also includes three second-class phonemes, namely phoneme d, phoneme j and phoneme h.

[0091] In a possible implementation, the server processes the text content by using the TTS technology to generate voice content corresponding to the text content. Afterwards, the server processes the voice content to determine P first-category phonemes and Q second-category phonemes.

[0092] 402. The server determines a first lip animation sequence according to the P first-category phonemes and the Q second-category phonemes.

[0093] The first lip animation sequence includes: P first opening animations corresponding to P first-category phonemes, and Q first closing animations corresponding to Q second-category phonemes.

[0094] In a possible implementation, the server determines P first opening animations and Q first closing animations from the preset lip animations according to the correspondence between the P first-category phonemes and the Q second-category phonemes and the phonemes and the preset lip animations.

[0095] In another possible implementation, the server determines a second lip animation sequence according to P first-category phonemes and Q second-category phonemes.

[0096] The second lip animation sequence includes P second opening animations and Q second closing animations, wherein the P second opening animations correspond to the P first opening animations, and the Q second closing animations correspond to the Q first closing animations. Optionally, the duration of the second opening animation is different from the duration of the first opening animation, and the duration of the second closing animation is different from the duration of the first closing animation.

[0097] Afterwards, the server generates a first lip animation sequence according to every two adjacent lip animations in the second lip animation sequence and the preset strategy, wherein the first lip animation sequence is the second lip animation sequence adjusted according to the preset strategy, and the preset strategy is used to adjust the duration of the lip animation.

[0098] For example, if the second lip animation sequence includes: lip animation a-lip animation b-lip animation c, wherein lip animation a is adjacent to lip animation b, and lip animation b is adjacent to lip animation c. The playing time of lip animation a is 00:01.01-00:02.10, the playing time of lip animation b is 00:02.11-00:03.90, and the playing time of lip animation c is 00:03.91-00:04.90. The first lip animation sequence includes: lip animation a1-lip animation b1-lip animation c1, wherein the playing time of lip animation a1 is 00:01.01-00:01.50, the playing time of lip animation b1 is 00:02.11-00:03.90, and the playing time of lip animation c1 is 00:04.00-00:04.90. That is to say, compared with the second lip animation sequence, the playing time of the lip animation a1 and the lip animation c1 in the first lip animation sequence are both reduced.

[0099] The technical solution provided by the above embodiment brings at least the following beneficial effects: according to P first-category phonemes and Q second-category phonemes, a second lip animation sequence is determined, the second lip animation sequence includes P second opening animations and Q second closing animations, the duration of the second opening animation is different from the duration of the first opening animation, and the duration of the second closing animation is different from the duration of the first closing animation. Afterwards, according to every two adjacent lip animations in the second lip animation sequence and the preset strategy, the duration of the lip animation is adjusted to generate the first lip animation sequence. In this way, the duration of adjacent lip animations can be reasonably adjusted to ensure the continuity of two adjacent lip animations and reduce the abruptness between adjacent lip animations.

[0100] In a possible design, the second lip animation sequence includes a first sub-lip animation and a second sub-lip animation, and the first sub-lip animation and the second sub-lip animation are any two adjacent lip animations in the second lip animation sequence. In the embodiment of the present disclosure, two lip animations are adjacent, which means that the difference between the end time of the previous lip animation and the start time of the next lip animation is less than a preset time difference (for example, 1 second, 0.1 second, 0.01 second, etc.). The preset strategy includes at least one of the following: a first strategy, a second strategy.

[0101] The first strategy is: if the first sub-lip animation is an open mouth type animation and the second sub-lip animation is a closed mouth type animation, then the duration of the second sub-lip animation is reduced.

[0102] For example, if lip animation a is adjacent to lip animation b, the play time of lip animation a is 00:01.01-00:02.10, and the play time of lip animation b is 00:02.11-00:03.90. If lip animation a is an open mouth type animation, and lip animation b is a closed mouth type animation, the play time of lip animation b is adjusted to 00:02.80-00:03.90. Alternatively, if lip animation a is a closed mouth type animation, and lip animation b is an open mouth type animation, the play time of lip animation a is adjusted to 00:01.01-00:01.90.

[0103] Optionally, a transition animation is added to the duration of the reduced second sub-lip animation, and the transition animation is composed of a closed-mouth animation and an open-mouth animation.

[0104] The start time of the transition animation is the end time of the first sub-lip-sync animation, and the duration of the transition animation is the reduced duration of the second sub-lip-sync animation.

[0105] For example, if lip animation a is adjacent to lip animation b, the play time of lip animation a is 00:01.01-00:02.10, and the play time of lip animation b is 00:02.11-00:03.90. If lip animation a is an open mouth type animation, and lip animation b is a closed mouth type animation, the play time of lip animation b is adjusted to 00:02.80-00:03.90, and a transition animation is inserted between 00:02.11-00:02.19. 00:02.11 is the start time of the transition animation, and 00:02.11-00:02.19 is the duration of the transition animation.

[0106] Optionally, if the first sub-lip shape animation is an open mouth type animation and the second sub-lip shape animation is a closed mouth type animation, the duration of the second sub-lip shape animation is reduced and the duration of the first sub-lip shape animation is lengthened.

[0107] It is understandable that if the first sub-lip animation is an open mouth type animation and the second sub-lip animation is a closed mouth type animation, the duration of the second sub-lip animation is reduced. In this way, the coherence between the open mouth type animation and the closed mouth type animation can be guaranteed, and the abruptness between the lip animations can be reduced.

[0108] The second strategy is: if the first sub-lip-shaped animation and the second sub-lip-shaped animation are both opening type animations, then the duration of the first sub-lip-shaped animation and the duration of the second sub-lip-shaped animation are reduced.

[0109] For example, if lip animation a is adjacent to lip animation b, the playing time of lip animation a is 00:01.01-00:02.10, and the playing time of lip animation b is 00:02.11-00:03.90. If lip animation a and lip animation b are both open mouth type animations, the playing time of lip animation a is adjusted to 00:01.01-00:01.90, and the playing time of lip animation b is adjusted to 00:02.80-00:03.90.

[0110] Optionally, a transition animation is added to the duration of the first sub-lip animation and the duration of the second sub-lip animation, which are reduced. The transition animation is composed of a closed-mouth animation and an open-mouth animation.

[0111] It is understandable that if the first sub-lip animation and the second sub-lip animation are both opening type animations, the duration of the first sub-lip animation and the duration of the second sub-lip animation are reduced. In this way, by reducing the duration of the two opening type animations, the coherence between the two opening type animations can be guaranteed, and the abruptness between the lip animations can be reduced.

[0112] The technical solution provided by the above embodiment brings at least the following beneficial effects: the text content is parsed to determine a phoneme sequence, the phoneme sequence includes P first-class phonemes and Q second-class phonemes, the lip animation corresponding to the first-class phonemes is an open-mouth type animation, and the lip animation corresponding to the second-class phonemes is a closed-mouth type animation, and P and Q are both positive integers. Afterwards, based on the P first-class phonemes and the Q second-class phonemes, a first lip animation sequence is determined, and the first lip animation sequence includes: P first open-mouth animations corresponding to the P first-class phonemes, and Q first closed-mouth animations corresponding to the Q second-class phonemes. In this way, by classifying the phonemes, the classification of the lip animations can be completed. In this way, the first lip animation sequence can include multiple open-mouth animations and multiple closed-mouth animations, thereby ensuring the continuity of two adjacent lip animations and reducing the abruptness between adjacent lip animations.

[0113] In an practicable manner, in order to further improve the diversity of the broadcast text content, an action animation can be added to the character image in the target animation. The text information can also include N action identifiers, and the N action identifiers correspond to the N first action animations, where N is a positive integer. Each of the N action identifiers corresponds to a plurality of phonemes. In other words, the execution order and execution duration of the N first action animations correspond to the phonemes of the text content.

[0114] It should be noted that the action identifier can be marked by the developer in combination with the text content. Alternatively, the action identifier can be marked by the server according to the text content. This embodiment of the disclosure is not limited to this.

[0115] It should be noted that the present disclosure does not limit the process by which the server marks the action identifier for part of the text in the text content according to the text content. For example, the server can determine the action corresponding to the text content and the action identifier corresponding to the action according to natural language understanding or a preset recognition strategy, and the preset recognition strategy is used to identify the action corresponding to the text content.

[0116] Specifically, Figure 5 As shown, before step 203 , the image processing method may further include step 501 .

[0117] 501. The server parses N action identifiers to determine a first action animation sequence.

[0118] The first action animation sequence includes N first action animations.

[0119] In the disclosed embodiment, the server stores the correspondence between all action identifiers and preset action animations.

[0120] Exemplarily, if all action identifiers include: action identifier a, action identifier b, action identifier c and action identifier d, action identifier a corresponds to action animation a, action identifier b corresponds to action animation b, action identifier c corresponds to action animation c, and action identifier d corresponds to action animation d.

[0121] In a possible implementation manner, the server determines N first action animations from preset action animations according to N action identifiers.

[0122] In one possible design, the N action identifiers include first-category identifiers and second-category identifiers, the action animation corresponding to the first-category identifier is the first-category action animation, the action animation corresponding to the second-category identifier is the second-category action animation, and the movement amplitude of the first-category action animation is greater than the movement amplitude of the second-category action animation.

[0123] It should be noted that the amplitude of motion refers to the value of movement of the body or a part of the body (which can be expressed by measuring degrees, scales and line segments).

[0124] Optionally, the motion amplitude of the first type of motion animation is greater than a preset motion amplitude threshold, and the motion amplitude of the second type of motion animation is less than the preset motion amplitude threshold.

[0125] For example, the first type of action animation may include: waving animation, malfunction animation, running animation, jumping animation, rolling animation, etc. The second type of action animation may include: slight shaking animation, breathing animation, akimbo animation, etc.

[0126] The technical solution provided by the above embodiment brings at least the following beneficial effects: N action identifiers include first-class identifiers and second-class identifiers, the action animation corresponding to the first-class identifier is the first-class action animation, the action animation corresponding to the second-class identifier is the second-class action animation, and the movement amplitude of the first-class action animation is greater than the movement amplitude of the second-class action animation. In this way, it can be ensured that the action animation with a larger movement amplitude is combined with the action animation with a smaller movement amplitude, thereby reducing the abruptness between the action animations and improving the coherence between the action animations.

[0127] Optionally, the first type of action animation is an animation used to represent emotional information, and the second type of action animation is an animation used to represent emotional information.

[0128] In a possible design, the first type of identification includes multiple identifications, each of which corresponds to an action animation. The second type of identification includes at least one identification, each of which may correspond to multiple action animations.

[0129] Exemplarily, the first type of identification includes identification a, identification b and identification c, and the second type of identification includes identification d. Among them, identification a corresponds to action animation a, identification b corresponds to action animation b, identification c corresponds to action animation c. Identification d corresponds to any one of action animation d, action animation e, action animation f, and action animation g.

[0130] The server may determine the target action animation from the second category of action animations according to the action animations adjacent to the target action animation, and the target action animation is any animation in the second category of action animations.

[0131] Exemplarily, if identifier a is a second-category identifier, the action animations corresponding to identifier a include: action animation d, action animation e, and action animation f, and the target action animation can be any one of action animation d, action animation e, and action animation f.

[0132] It should be noted that the server can determine the target action animation according to the duration of the action animations adjacent to the target action animation. In this way, it can be avoided that the target action animation has a long duration and affects the display of other action animations. In this way, the coherence between the action animations can be improved.

[0133] In another possible implementation, the server parses the N action identifiers to determine a second action animation sequence, where the second action animation sequence includes N second action animations.

[0134] Exemplarily, the server determines N second action animations from preset action animations according to the N action identifiers.

[0135] Among them, at least one second type identifier is included between every two first type identifiers in the N action identifiers, and at least one second type action animation is included between every two first type action animations in the N second action animations.

[0136] Exemplarily, if N is 4, the 4 action identifiers include: second-category identifier-first-category identifier-second-category identifier-first-category identifier, and the 4 action animations include: second-category action animation-first-category action animation-second-category action animation-first-category action animation.

[0137] It is understandable that the second type of action animation exists between the first type of action animation, that is, the second type of action animation exists after the first type of action animation. In this way, the continuity between the multiple action animations can be guaranteed.

[0138] Afterwards, the server performs a first operation on each of the N second action animations to generate a first action animation sequence. The first operation includes: if the first sub-action animation is a first-category action animation, adjusting the action animation of the first preset period in the second sub-action animation to the second-category action animation, the first sub-action animation and the adjusted second sub-action animation, the adjusted second sub-action animation being the animation in the first action animation sequence.

[0139] The first sub-action animation and the second sub-action animation are any two adjacent animations among the N second action animations, and the first preset time period is a time period of executing the second sub-action animation that is adjacent to executing the first sub-action animation.

[0140] For example, if the second action animation sequence includes: action animation a-action animation b-action animation c, wherein action animation a is adjacent to action animation b, and action animation b is adjacent to action animation c. The play time of action animation a is 00:01.01-00:02.10, the play time of action animation b is 00:02.11-00:03.90, and the play time of action animation c is 00:03.91-00:04.90. If action animation a is a first type of action animation, 00:02.11-00:02.94 of the play time of action animation b is used as the first preset time period, and the second type of action animation (such as breathing animation) is added to 00:02.11-00:02.94. That is, the adjusted action animation b (i.e., action animation b1) includes breathing animation and action animation b, and the adjusted action animation b plays breathing animation at 00:02.11-00:02.94, and plays action animation b at 00:02.95-00:03.90. That is, the first action animation sequence generated according to the second action animation sequence includes action animation a-action animation b1 (breathing animation+action animation b)-action animation c.

[0141] In a possible design, if the second sub-motion animation is the first type of motion animation, the motion animation of the first preset period in the second sub-motion animation is adjusted to the second type of motion animation. If the second sub-motion animation is the second type of motion animation, the second sub-motion animation is not adjusted.

[0142] Optionally, the second sub-action animation can be the previous action of the first sub-action animation. Alternatively, the second sub-action animation can be the next action of the first sub-action animation.

[0143] The technical solution provided by the above embodiment brings at least the following beneficial effects: N action identifiers are parsed to determine the second action animation sequence, and the second action animation sequence includes N second action animations. A first operation is performed on each action animation in the N second action animations to generate a first action animation sequence; the first operation includes: if the first sub-action animation is a first-category action animation, the action animation of the first preset time period in the second sub-action animation is adjusted to the second-category action animation to obtain the adjusted second sub-action animation, the first sub-action animation and the adjusted second sub-action animation are animations in the first action animation sequence, and the first preset time period is the time period immediately adjacent to the execution of the first sub-action animation in the time period of executing the second sub-action animation. In this way, it can be ensured that there is an animation with a smaller movement amplitude between the execution of the first sub-action animation and the second sub-action animation, thereby ensuring that the action animation with a larger movement amplitude is combined with the action animation with a smaller movement amplitude, reducing the abruptness between the action animations and improving the coherence between the action animations.

[0144] In the disclosed embodiment, the server processes the character image to generate the target animation, which may also include: the server processes the character image according to the first action animation sequence and the first lip animation sequence to generate the target animation. The target animation also includes the body parts of the processed character image performing N first action animations.

[0145] It should be noted that the embodiment of the present disclosure does not limit the order in which the server executes step 501 and step 202. For example, the server may first execute step 202 and then execute step 501. For another example, the server may first execute step 501 and then execute step 202. For another example, the server may execute step 202 and step 501 at the same time.

[0146] The technical solution provided by the above embodiment brings at least the following beneficial effects: the text information also includes N action identifiers, the action identifiers correspond to the action animations, and N is a positive integer. The N action identifiers are parsed to determine the first action sequence, and the first action sequence includes N first action animations. Among them, the target animation also includes the body parts of the processed character image executing the N first action animations. In this way, in the target animation, not only the lips of the character image can change dynamically based on the text content, but also the body parts of the character image can execute the action animation, further improving the diversity of the broadcast text content.

[0147] The following describes the image processing method of the embodiment of the present disclosure in conjunction with specific examples. Figure 6 As shown, the server obtains text content with action identifiers. Afterwards, the server parses the text content with action identifiers to generate a phoneme sequence, a second action animation sequence, and speech corresponding to the text content. Afterwards, the server determines a first lip animation sequence based on the phoneme sequence. The server adjusts the second action animation sequence based on two adjacent action animations to generate a first action animation sequence. Then, the server processes the character image based on the first lip animation sequence and the first action animation sequence to generate a target animation. Then, the target animation can broadcast the speech corresponding to the text content.

[0148] It is understandable that the above method can be implemented by an image processing device. In order to implement the above functions, the image processing device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the modules and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present disclosure.

[0149] The embodiments of the present disclosure may divide the functional modules of the image processing device and the like according to the above method examples. For example, each functional module may be divided according to each function, or two or more functions may be integrated into one processing module. The above integrated modules may be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments of the present disclosure is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0150] Figure 7 is a structural block diagram of an image processing device according to an exemplary embodiment. Figure 7 The image processing device 70 includes an acquisition unit 71 and a processing unit 72.

[0151] The acquisition unit 71 is configured to acquire text information and character images, wherein the text information includes text content. The processing unit 72 is configured to parse the text content and determine a first lip animation sequence, wherein the first lip animation sequence includes M first lip animations, where M is a positive integer. The processing unit 72 is also configured to perform image processing on the character image and generate a target animation, wherein the target animation includes the lips of the processed character image executing M first lip animations based on the text content.

[0152] Optionally, the processing unit 72 is specifically configured to perform parsing of the text content to determine a phoneme sequence, the phoneme sequence includes P first-class phonemes and Q second-class phonemes, the lip animation corresponding to the first-class phonemes is an open-mouth type animation, and the lip animation corresponding to the second-class phonemes is a closed-mouth type animation, and P and Q are both positive integers. The processing unit 72 is specifically configured to perform, based on the P first-class phonemes and the Q second-class phonemes, determining a first lip animation sequence, the first lip animation sequence includes: P first open-mouth animations corresponding to the P first-class phonemes, Q first closed-mouth animations corresponding to the Q second-class phonemes, and P+Q=M.

[0153] Optionally, the processing unit 72 is specifically configured to determine a second lip animation sequence based on P first-category phonemes and Q second-category phonemes, the second lip animation sequence including P second opening animations and Q second closing animations; the P second opening animations correspond to the P first opening animations, and the Q second closing animations correspond to the Q first closing animations. The processing unit 72 is specifically configured to generate a first lip animation sequence based on every two adjacent lip animations in the second lip animation sequence and a preset strategy, the first lip animation sequence being the second lip animation sequence adjusted according to the preset strategy, and the preset strategy is used to adjust the duration of the lip animation.

[0154] Optionally, the second lip-sync animation sequence includes a first sub-lip-sync animation and a second sub-lip-sync animation, and the first sub-lip-sync animation and the second sub-lip-sync animation are any two adjacent lip-sync animations in the second lip-sync animation sequence; the preset strategy includes at least one of the following: if the first sub-lip-sync animation is an open-mouth type animation and the second sub-lip-sync animation is a closed-mouth type animation, then the duration of the second sub-lip-sync animation is reduced. If the first sub-lip-sync animation and the second sub-lip-sync animation are both open-mouth type animations, then the duration of the first sub-lip-sync animation and the duration of the second sub-lip-sync animation are reduced.

[0155] Optionally, the text information further includes N action identifiers, the action identifiers correspond to action animations, and N is a positive integer. The processing unit 72 is further configured to parse the N action identifiers to determine a first action animation sequence, the first action animation sequence includes N first action animations; wherein the target animation also includes the N first action animations executed by the limbs of the processed character image.

[0156] Optionally, the N action identifiers include first-category identifiers and second-category identifiers, the action animation corresponding to the first-category identifier is a first-category action animation, the action animation corresponding to the second-category identifier is a second-category action animation, and the movement amplitude of the first-category action animation is greater than the movement amplitude of the second-category action animation.

[0157] Optionally, the processing unit 72 is specifically configured to perform parsing of N action identifiers to determine a second action animation sequence, wherein the second action animation sequence includes N second action animations. Among them, at least one second-class identifier is included between every two first-class identifiers in the N action identifiers, and at least one second-class action animation is included between every two first-class action animations in the N second action animations. The processing unit 72 is specifically configured to perform a first operation on each second action animation in the N second action animations to generate a first action animation sequence; the first operation includes: if the first sub-action animation is a first-class action animation, then the action animation of the first preset time period in the second sub-action animation is adjusted to the second-class action animation, and the adjusted second sub-action animation is obtained, and the first sub-action animation and the adjusted second sub-action animation are animations in the first action animation sequence; wherein the first sub-action animation and the second sub-action animation are any two adjacent animations in the N second action animations, and the first preset time period is the time period of executing the second sub-action animation that is adjacent to the execution of the first sub-action animation.

[0158] Optionally, each of the M first lip animations includes multiple frames of lip action images. The processing unit 72 is further configured to determine the lips of the character image according to a preset detection algorithm and the character image. The processing unit 72 is further configured to adjust the lip action of the character image according to each frame of the lip action image in each first lip animation to generate a target animation.

[0159] Optionally, the acquisition unit 71 is further configured to acquire a preset video. The processing unit 72 is further configured to add a target animation to the preset video to generate a target video, wherein the target video includes the target animation.

[0160] Regarding the image processing device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0161] Figure 8 FIG. 8 is a schematic diagram of the structure of an image processing device 80 provided by the present disclosure. Figure 8 The image processing device 80 may include at least one processor 801 and a memory 803 for storing executable instructions of the processor 801. The processor 801 is configured to execute the instructions in the memory 803 to implement the image processing method in the above embodiment.

[0162] In addition, the image processing device 80 may further include a communication bus 802 and at least one communication interface 804 .

[0163] The processor 801 may be a GPU, a microprocessing unit, an ASIC, or one or more integrated circuits for controlling the execution of the program of the disclosed solution.

[0164] The communication bus 802 may include a pathway for transmitting information between the above-mentioned components.

[0165] The communication interface 804 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0166] The memory 803 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processing unit via a bus. The memory may also be integrated with the processing unit and be a volatile storage medium in the GPU.

[0167] The memory 803 is used to store instructions for executing the solution of the present disclosure, and the execution is controlled by the processor 801. The processor 801 is used to execute the instructions stored in the memory 803, so as to realize the functions in the method of the present disclosure.

[0168] In a specific implementation, as an embodiment, the processor 801 may include one or more GPUs, such as Figure 8 GPU0 and GPU1 in .

[0169] In a specific implementation, as an embodiment, the image processing device 80 may include multiple processors, such as Figure 8 801 and processor 807 in the embodiment of the present invention. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-GPU) processor. The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0170] In a specific implementation, as an embodiment, the image processing device 80 may further include an output device 805 and an input device 806. The output device 805 communicates with the processor 801 and may display information in a variety of ways. For example, the output device 805 may be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device 806 communicates with the processor 801 and may receive user input in a variety of ways. For example, the input device 806 may be a mouse, a keyboard, a touch screen device, or a sensor device.

[0171] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the image processing device 80, and may include more or less components than those shown in the figure, or combine certain components, or adopt a different component arrangement.

[0172] The present disclosure also provides a computer-readable storage medium having instructions stored thereon. When the instructions in the storage medium are executed by a processor of a server, the server is enabled to execute the group communication method provided by the above-mentioned embodiment of the present disclosure.

[0173] The embodiment of the present disclosure also provides a computer program product including instructions, which, when executed on a server, enables the server to execute the image processing method provided by the embodiment of the present disclosure.

[0174] The present disclosure also provides a communication system. Figure 1 As shown, the system includes a server 01 and a terminal 02. The server 01 and the terminal 02 are respectively used to execute the corresponding steps in the above-mentioned embodiments of the present disclosure, so that the communication system can solve the technical problems solved by the embodiments of the present disclosure and achieve the technical effects achieved by the embodiments of the present disclosure, which will not be repeated here.

[0175] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0176] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for processing an image, characterized in that: The method comprises: Acquire text information and character images, wherein the text information includes text content; Parsing the text content to determine a phoneme sequence, the phoneme sequence includes P first-category phonemes and Q second-category phonemes, the lip animation corresponding to the first-category phonemes is an open-mouth type animation, the lip animation corresponding to the second-category phonemes is a closed-mouth type animation, and P and Q are both positive integers; Determine a second lip animation sequence according to the P first-category phonemes and the Q second-category phonemes, wherein the second lip animation sequence includes P second opening animations and Q second closing animations, wherein the P second opening animations correspond to the P first opening animations, and the Q second closing animations correspond to the Q first closing animations; According to every two adjacent lip animations in the second lip animation sequence and the preset strategy, a first lip animation sequence is generated, wherein the first lip animation sequence is the second lip animation sequence adjusted according to the preset strategy, and the preset strategy is used to adjust the duration of the lip animation; the first lip animation sequence includes M first lip animations, and the M first lip animations include: P first opening animations corresponding to the P first-category phonemes, and Q first closing animations corresponding to the Q second-category phonemes, where M is a positive integer, and P+Q=M; The character image is subjected to image processing to generate a target animation, wherein the target animation includes the lips of the processed character image executing the M first lip animations based on the text content.

2. The method according to claim 1, characterized in that The second lip animation sequence includes a first sub-lip animation and a second sub-lip animation, and the first sub-lip animation and the second sub-lip animation are any two adjacent lip animations in the second lip animation sequence; the preset strategy includes at least one of the following: If the first sub-lip animation is the open mouth type animation and the second sub-lip animation is the closed mouth type animation, then reducing the duration of the second sub-lip animation; If the first sub-lip-sync animation and the second sub-lip-sync animation are both the opening type animations, then the duration of the first sub-lip-sync animation and the duration of the second sub-lip-sync animation are reduced.

3. The method according to claim 1 or 2, characterized in that: The text information also includes N action identifiers, the action identifiers correspond to the action animations, and N is a positive integer; Before performing image processing on the character image to generate the target animation, the method further includes: Parsing the N action identifiers to determine a first action animation sequence, where the first action animation sequence includes N first action animations; The target animation also includes the N first action animations executed by the limbs of the processed character image.

4. The method according to claim 3, characterized in that: The N action identifiers include first-category identifiers and second-category identifiers, the action animation corresponding to the first-category identifier is a first-category action animation, the action animation corresponding to the second-category identifier is a second-category action animation, and the movement amplitude of the first-category action animation is greater than the movement amplitude of the second-category action animation.

5. The method according to claim 4, characterized in that The step of parsing the N action identifiers to determine a first action animation sequence includes: Parsing the N action identifiers to determine a second action animation sequence, wherein the second action animation sequence includes N second action animations; wherein at least one second-type identifier is included between every two of the first-type identifiers in the N action identifiers, and at least one second-type action animation is included between every two of the first-type action animations in the N second action animations; Performing a first operation on each of the N second action animations to generate the first action animation sequence; the first operation includes: If the first sub-action animation is the first type of action animation, then the action animation of the first preset period in the second sub-action animation is adjusted to the second type of action animation to obtain an adjusted second sub-action animation, and the first sub-action animation and the adjusted second sub-action animation are animations in the first action animation sequence; The first sub-action animation and the second sub-action animation are any two adjacent animations among the N second action animations, and the first preset time period is a time period of executing the second sub-action animation that is adjacent to executing the first sub-action animation.

6. The method according to claim 5, characterized in that Each of the M first lip-sync animations includes a plurality of frames of lip-sync action images; and performing image processing on the character image to generate a target animation includes: Determine the lips of the person image according to a preset detection algorithm and the person image; According to each frame of lip action image in each first lip animation, the lip action of the character image is adjusted to generate the target animation.

7. The method according to claim 6, characterized in that The method further comprises: Get preset video; The target animation is added to the preset video to generate a target video, wherein the target video includes the target animation.

8. An image processing device, characterized in that: include: An acquisition unit, configured to acquire text information and a character image, wherein the text information includes text content; a processing unit configured to parse the text content to determine a phoneme sequence, wherein the phoneme sequence includes P first-category phonemes and Q second-category phonemes, the lip animations corresponding to the first-category phonemes are open-mouth type animations, and the lip animations corresponding to the second-category phonemes are closed-mouth type animations, and P and Q are both positive integers; The processing unit is further configured to determine a second lip animation sequence according to the P first-category phonemes and the Q second-category phonemes, wherein the second lip animation sequence includes P second opening animations and Q second closing animations, wherein the P second opening animations correspond to the P first opening animations, and the Q second closing animations correspond to the Q first closing animations; The processing unit is further configured to generate a first lip animation sequence according to every two adjacent lip animations in the second lip animation sequence and a preset strategy, wherein the first lip animation sequence is the second lip animation sequence adjusted according to the preset strategy, and the preset strategy is used to adjust the duration of the lip animation; The first lip animation sequence includes M first lip animations, and the M first lip animations include: P first opening animations corresponding to the P first-category phonemes, and Q first closing animations corresponding to the Q second-category phonemes, where M is a positive integer, and P+Q=M; The processing unit is further configured to perform image processing on the character image to generate a target animation, wherein the target animation includes the lips of the processed character image executing the M first lip animations based on the text content.

9. The image processing device according to claim 8, characterized in that: The second lip animation sequence includes a first sub-lip animation and a second sub-lip animation, and the first sub-lip animation and the second sub-lip animation are any two adjacent lip animations in the second lip animation sequence; the preset strategy includes at least one of the following: If the first sub-lip animation is the open mouth type animation and the second sub-lip animation is the closed mouth type animation, then reducing the duration of the second sub-lip animation; If the first sub-lip-sync animation and the second sub-lip-sync animation are both the opening type animations, then the duration of the first sub-lip-sync animation and the duration of the second sub-lip-sync animation are reduced.

10. The image processing device according to claim 8 or 9, characterized in that: The text information also includes N action identifiers, the action identifiers correspond to the action animations, and N is a positive integer; The processing unit is further configured to parse the N action identifiers to determine a first action animation sequence, wherein the first action animation sequence includes N first action animations; wherein the target animation also includes the limbs of the processed character image executing the N first action animations.

11. The image processing device according to claim 10, characterized in that: The N action identifiers include first-category identifiers and second-category identifiers, the action animation corresponding to the first-category identifier is a first-category action animation, the action animation corresponding to the second-category identifier is a second-category action animation, and the movement amplitude of the first-category action animation is greater than the movement amplitude of the second-category action animation.

12. The image processing device according to claim 11, characterized in that: The processing unit is specifically configured to parse the N action identifiers to determine a second action animation sequence, wherein the second action animation sequence includes N second action animations; wherein at least one second-type identifier is included between every two of the first-type identifiers in the N action identifiers, and at least one second-type action animation is included between every two of the first-type action animations in the N second action animations; The processing unit is specifically configured to perform a first operation on each of the N second action animations to generate the first action animation sequence; the first operation includes: if the first sub-action animation is the first type of action animation, then the action animation of the first preset time period in the second sub-action animation is adjusted to the second type of action animation to obtain the adjusted second sub-action animation, and the first sub-action animation and the adjusted second sub-action animation are animations in the first action animation sequence; wherein, the first sub-action animation and the second sub-action animation are any two adjacent animations among the N second action animations, and the first preset time period is a time period in the time period of executing the second sub-action animation that is immediately adjacent to the time period of executing the first sub-action animation.

13. The image processing device according to claim 12, characterized in that: Each of the M first lip-sync animations includes multiple frames of lip-sync action images; The processing unit is further configured to determine the lips of the person image according to a preset detection algorithm and the person image; The processing unit is further configured to adjust the lip movements of the character image according to each frame of the lip movement image in each first lip animation to generate the target animation.

14. The image processing device according to claim 13, characterized in that: The acquisition unit is further configured to execute acquisition of a preset video; The processing unit is further configured to add the target animation to the preset video to generate a target video, wherein the target video includes the target animation.

15. A server, characterized in that: The server comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image processing method as described in any one of claims 1-7.

16. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of a server, the server is enabled to execute the image processing method according to any one of claims 1 to 7.

17. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.