Digital human expression processing method and apparatus, electronic device, and storage medium

Abstract

The application provides an expression processing method and device of a digital person, electronic equipment and a storage medium. The method comprises the following steps: obtaining a broadcast text to be broadcast by the digital person; obtaining emotion distribution information corresponding to the broadcast text; determining emotion intensity information associated with emotion labels; determining a time overlap area of adjacent emotion labels based on the emotion distribution information; determining global expression driving data corresponding to the broadcast text based on emotion intensity information of a broadcast time in the time overlap area and emotion intensity information of a broadcast time outside the time overlap area; and synchronously driving the expression of the digital person based on the global expression driving data during the broadcasting of the broadcast text by the digital person. Thus, the present application realizes smooth transition of the expression of the digital person in the case of coexistence of multiple emotion labels, that is, smooth switching between multiple emotions of the digital person, so that the digital person can simulate natural expression transition of a person during emotional transition.

Description

Technical Field

[0001] This application relates to the technical fields of intelligent cockpit, in-vehicle intelligent agent, digital human interaction, scene arrangement, etc., and in particular to a digital human expression processing method, device, electronic device and storage medium. Background Technology

[0002] In real conversations, different responses often involve emotional shifts, meaning a single sentence can contain two or more different emotions. Currently, digital humans cannot handle these emotional shifts; they either only capture one overall emotion or abruptly switch between multiple emotions, resulting in noticeable and unnatural changes in the digital human's facial expressions. Summary of the Invention

[0003] The purpose of this application is to at least partially solve one of the technical problems in the related art.

[0004] Therefore, the first objective of this application is to propose a digital human expression processing method so that the digital human can simulate the natural transition of human expressions during emotional shifts.

[0005] The second objective of this application is to propose a digital human facial expression processing device.

[0006] The third objective of this application is to propose an electronic device.

[0007] The fourth objective of this application is to provide a computer-readable storage medium.

[0008] The fifth objective of this application is to provide a computer program product.

[0009] To achieve the above objectives, a first aspect of this application proposes a method for processing the facial expressions of a digital human, comprising: acquiring a text to be read by the digital human; acquiring emotion distribution information corresponding to the text, the emotion distribution information including each emotion tag and the reading time covered by each emotion tag; determining emotion intensity information associated with the emotion tags, the emotion intensity information including the emotion intensity of each reading time covered by the emotion tag; determining a time overlap region of adjacent emotion tags based on the emotion distribution information; determining global facial expression driving data corresponding to the text based on the emotion intensity information of the reading times within the time overlap region and the emotion intensity information of the reading times outside the time overlap region; and synchronously driving the facial expressions of the digital human based on the global facial expression driving data during the process of the digital human reading the text.

[0010] To achieve the above objectives, a second aspect of this application provides a digital human expression processing device, comprising: a first acquisition module for acquiring a text to be broadcast by the digital human; a second acquisition module for acquiring emotion distribution information corresponding to the broadcast text, the emotion distribution information including each emotion tag and the broadcast time covered by each emotion tag; a first determination module for determining emotion intensity information associated with the emotion tag, the emotion intensity information including the emotion intensity of each broadcast time covered by the emotion tag; a second determination module for determining a time overlap region of adjacent emotion tags based on the emotion distribution information; a third determination module for determining global expression driving data corresponding to the broadcast text based on the emotion intensity information of the broadcast times within the time overlap region and the emotion intensity information of the broadcast times outside the time overlap region; and an expression driving module for synchronously driving the expression of the digital human based on the global expression driving data during the process of the digital human broadcasting the broadcast text.

[0011] To achieve the above objectives, a third aspect of this application provides an electronic device, comprising: a processor; and a memory communicatively connected to the processor; the memory storing computer execution instructions; and the processor executing the computer execution instructions stored in the memory to enable the processor to perform the digital human expression processing method described in the first aspect of the application.

[0012] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, the computer instructions being used to cause the computer to execute the digital human expression processing method described in the above aspect of the embodiment.

[0013] To achieve the above objectives, a fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the digital human expression processing method described in the above aspect of the embodiment.

[0014] The digital human expression processing method, apparatus, electronic device, and storage medium provided in this application acquire the text to be broadcast by the digital human, and obtain the emotion distribution information and emotion intensity information associated with the emotion tags corresponding to the broadcast text. Based on the emotion distribution information, the time overlap area of ​​adjacent emotion tags is determined, and based on the emotion intensity information of the broadcast time within the time overlap area and the emotion intensity information of the broadcast time outside the time overlap area, global expression driving data corresponding to the broadcast text is determined. Thus, during the digital human's broadcast of the text, the digital human's expressions can be synchronously driven based on the global expression driving data. Therefore, this solution can achieve smooth transitions in the digital human's expressions when multiple emotion tags coexist, that is, it enables the digital human to smoothly switch between multiple emotions, allowing the digital human to simulate the natural transition of human expressions during emotional shifts, improving the fluency and realism of the digital human.

[0015] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0016] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0017] Figure 1 A flowchart illustrating a digital human facial expression processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the emotional intensity time curve provided in the embodiments of this application; Figure 3 A flowchart illustrating another digital human facial expression processing method provided in an embodiment of this application; Figure 4 A flowchart illustrating another digital human facial expression processing method provided in an embodiment of this application; Figure 5 This is a schematic diagram of global sentiment intensity information provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a digital human expression processing device provided in an embodiment of this application. Detailed Implementation

[0018] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0019] The following description, with reference to the accompanying drawings, describes a digital human facial expression processing method and apparatus according to embodiments of this application.

[0020] Figure 1 This is a flowchart illustrating a digital human expression processing method provided in an embodiment of this application, as shown below. Figure 1 As shown, the digital human expression processing method of this application embodiment includes, but is not limited to, the following steps: S101, Obtain the text to be read by the digital human.

[0021] It should be noted that the execution subject of the digital human expression processing method provided in this application embodiment is an electronic device, which can be a terminal device. Optionally, the terminal device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be personal computers (PCs), televisions, etc. This application embodiment does not impose specific limitations.

[0022] It should be noted that the digital human in this application embodiment can be applied to scenarios requiring user responses, such as intelligent cockpits in vehicles, customer service scenarios, and consultation scenarios. In other words, the digital human can respond to user questions or requests, and the text it broadcasts can be the response text corresponding to the question or request.

[0023] In some embodiments, by receiving user input information and combining it with a large language model to parse and respond to the input information, a response text corresponding to the input information is generated as the broadcast text that the digital human needs to broadcast.

[0024] It should be noted that the digital human in this embodiment can also be applied to other scenarios where user response is not required, such as information display and educational scenarios. In other words, if the digital human can broadcast received information, the text it broadcasts can be the user's input.

[0025] In some embodiments, user input is received and used as the text to be read by the digital human.

[0026] In this embodiment of the application, no specific limitations are made on the application scenarios of digital humans.

[0027] S102, obtain the sentiment distribution information corresponding to the broadcast text. The sentiment distribution information includes each sentiment tag and the broadcast time covered by each sentiment tag.

[0028] In some embodiments, Natural Language Processing (NLP) technology can be used to perform sentiment analysis on the broadcast text to obtain sentiment tags for the broadcast text. Based on the position of the sentiment tags in the broadcast text, the broadcast time covered by each sentiment tag can be determined, so that each sentiment tag and the broadcast time covered by each sentiment tag can be used as sentiment distribution information.

[0029] Emotional tags are used to describe, classify, or label human emotional states, such as happiness, amazement, sadness, etc.

[0030] In some embodiments, semantic understanding and feature extraction are performed on the broadcast text to obtain the emotional information carried in the broadcast text, and emotional tags are extracted from the broadcast text based on the emotional information to obtain the emotional tags of the broadcast text.

[0031] Optionally, sentiment classification can be performed based on sentiment information, and the classification results can be used as sentiment labels. For example, by extracting features from the broadcast text and using the extracted features as sentiment information, sentiment classification can be performed based on the sentiment information to output a sentiment label for each sentiment.

[0032] In some embodiments, text-to-speech (TTS) can be used to convert the broadcast text into the speech that the data person needs to broadcast. Then, a speech timeline can be generated based on the speech that the data person needs to broadcast, and the broadcast time covered by the emotion tag can be determined based on the speech timeline.

[0033] S103, determine the emotional intensity information associated with the emotional label, the emotional intensity information includes the emotional intensity of each broadcast time covered by the emotional label.

[0034] In some embodiments, the emotional intensity information includes the emotional intensity of each broadcast time covered by the emotional tag. The emotional intensity information can be obtained by determining the emotional tag of the broadcast text from the broadcast text and generating the emotional intensity and broadcast time corresponding to the emotional tag.

[0035] In some embodiments, the emotional intensity information can be an emotional intensity time curve related to the broadcast time and emotional intensity, which can intuitively display the emotional intensity corresponding to the emotional tag at different times based on the emotional intensity time curve.

[0036] In some embodiments, an emotional intensity time curve segment can be generated with the broadcast time as the horizontal axis and the emotional intensity as the vertical axis. Optionally, the emotional intensity time curve can be generated by determining the coordinate points corresponding to the emotional intensity at each broadcast time and connecting each coordinate point.

[0037] In some embodiments, to make the generated emotional intensity time curve more accurate, when generating the emotional intensity time curve, a first time interval of emotional intensity enhancement and a second time interval of emotional intensity decay can be divided, and emotional intensity time curve segments in the first time interval and the second time interval can be generated respectively, so as to determine the emotional intensity time curve based on the emotional intensity time curve segments.

[0038] In some embodiments, time intervals can be divided based on the broadcast times of different words. The broadcast time of the first word marks the beginning of the first time interval, and the broadcast time of the last word marks the end of the second time interval. Since the emotional peak word corresponds to the highest emotional intensity, the broadcast time of the emotional peak word can be used as the end of the first time interval and the beginning of the second time interval.

[0039] In other words, the first time interval is from the time the first word is broadcast to the time the sentiment peak word is broadcast; the second time interval is from the time the sentiment peak word is broadcast to the time the last word is broadcast.

[0040] Alternatively, a smoothing algorithm can be used to smooth the curve connecting the coordinate points to obtain the emotional intensity time curve.

[0041] In some embodiments, an emotional intensity time curve segment is generated within the first time interval between the broadcast time of the first word and the broadcast time of the emotional peak word, starting from the emotional intensity corresponding to the first word and increasing to the emotional intensity corresponding to the emotional peak word.

[0042] In some embodiments, an emotional intensity time curve segment is generated within a second time interval between the broadcast time of the emotional peak word and the broadcast time of the last word, starting from the emotional intensity corresponding to the emotional peak word and decreasing to the emotional intensity corresponding to the last word.

[0043] Furthermore, by combining the emotional intensity time curve segments within the first time interval and the emotional intensity time curve segments within the second time interval, an emotional intensity time curve can be obtained.

[0044] Figure 2 This is a schematic diagram of the emotional intensity time curve provided in an embodiment of this application. Figure 2 It can be seen that the emotional intensity gradually increases before the broadcast of the emotional peak word, and gradually decreases after the broadcast of the emotional peak word.

[0045] In this embodiment of the application, if multiple emotion tags are identified from the broadcast text, an emotion intensity time curve associated with each emotion tag can be generated. That is, each emotion tag has its corresponding emotion intensity time curve, i.e., emotion intensity information.

[0046] S104. Based on sentiment distribution information, determine the temporal overlap region of adjacent sentiment tags.

[0047] In some embodiments, the sentiment distribution information includes the broadcast times covered by each sentiment tag, so the time overlap area of ​​adjacent sentiment tags can be determined based on the broadcast times covered by each sentiment tag.

[0048] Optionally, adjacent sentiment tags can be determined from each sentiment tag based on their position in the broadcast text, and the time overlap area of ​​adjacent sentiment tags can be determined based on the broadcast time of the first sentiment tag among the adjacent sentiment tags and the broadcast time of the second sentiment tag among the adjacent sentiment tags.

[0049] In some embodiments, it can be determined whether there is a time overlap between the broadcast time of the first emotion tag and the broadcast time of the second emotion tag, and the area where there is a time overlap is taken as the time overlap area of ​​adjacent emotion tags.

[0050] For example, the broadcast time of the first sentiment label is from time 1 to time 2, and the broadcast time of the second sentiment label is from time a to time b. If time a is between time 1 and time 2, and time 2 is between time a and time b, then the time overlap area of ​​the adjacent sentiment labels is between time a and time 2.

[0051] S105, based on the emotional intensity information of the broadcast time within the time overlap area and the emotional intensity information of the broadcast time outside the time overlap area, determine the global expression-driven data corresponding to the broadcast text.

[0052] In some embodiments, the emotional intensity information of the broadcast time within the time overlap region can be fused, and the fused emotional intensity information can be fused with the emotional intensity information of the broadcast time outside the time overlap region to obtain the global emotional intensity information corresponding to the broadcast text. Then, based on the global emotional intensity information, the global expression-driven data corresponding to the broadcast text can be determined.

[0053] In some embodiments, the emotional intensity information can be an emotional intensity time curve, which determines the global emotional intensity information corresponding to the broadcast text, that is, determines the global emotional intensity time curve.

[0054] Within the time overlap region, by fusing the emotional intensity time curves of adjacent emotional tags, a fused emotional intensity time curve can be obtained. By connecting the emotional intensity time curves outside the time overlap region with the fused emotional intensity time curve, a global emotional intensity time curve can be obtained.

[0055] In some embodiments, during the fusion of emotional intensity time curves, different weight values ​​can be assigned to each emotional intensity time curve, wherein the sum of the weight values ​​is 1, so that the emotional intensity time curves are not simply superimposed, and the fusion is more natural.

[0056] In some embodiments, if there are decaying emotional intensity time curves and rising emotional intensity time curves within the time overlap region, then the weight value of the decaying emotional intensity time curve can be set to gradually change from high to low with the broadcast time, and the weight value of the rising emotional intensity time curve can be set to gradually change from low to high with the broadcast time.

[0057] In some embodiments, the broadcast text consists of different words, so the facial expression driving data corresponding to each word can be determined to determine the global facial expression driving data corresponding to the broadcast text.

[0058] In some embodiments, the emotional intensity of a word at each broadcast moment can be determined based on global emotional intensity information, thereby generating expression-driven data based on the emotional intensity of that word.

[0059] Since the global emotion intensity information contains different emotion labels, and different emotion labels correspond to different expressions, there are two expressions in the time overlap region. Different expression-driven data can be generated in the time overlap region, and the expression-driven data can be fused to obtain the final expression-driven data in the time overlap region.

[0060] Furthermore, facial expression-driven data outside the time overlap region can be fused with facial expression-driven data within the time overlap region to obtain global facial expression-driven data corresponding to the broadcast text.

[0061] S106, During the process of the digital human broadcasting the text, the digital human's facial expressions are synchronously driven based on global facial expression driving data.

[0062] In some embodiments, global facial expression driving data can be input into the digital human's driving engine to render the digital human's expressions based on the global facial expression driving data, thereby achieving synchronized driving of the digital human's expressions. During the driving process, facial expressions and spoken audio can be aligned based on the broadcast time to improve the accuracy and effect of the digital human's broadcast.

[0063] In some embodiments, to improve the driving effect of the digital human's facial expressions, the digital human's facial expressions can be synchronously driven word by word. That is, the facial expression driving data for each word in the broadcast text is determined from the global facial expression driving data, and synchronous driving is performed based on the facial expression driving data of each word according to the broadcast time of the word.

[0064] The digital human expression processing method provided in this application obtains the text to be broadcast by the digital human, and acquires the emotion distribution information and emotion intensity information associated with the emotion tags corresponding to the broadcast text. Based on the emotion distribution information, the time overlap area of ​​adjacent emotion tags is determined, and based on the emotion intensity information of the broadcast time within the time overlap area and the emotion intensity information of the broadcast time outside the time overlap area, global expression driving data corresponding to the broadcast text is determined. Thus, during the digital human's broadcast of the text, the digital human's expression can be synchronously driven based on the global expression driving data. Therefore, this solution can achieve a smooth transition of the digital human's expression when multiple emotion tags coexist, that is, it enables the digital human to smoothly switch between multiple emotions, allowing the digital human to simulate the natural transition of human expressions during emotional shifts, improving the fluency and realism of the digital human.

[0065] Figure 3 This is a flowchart illustrating another digital human expression processing method provided in an embodiment of this application, as shown below. Figure 3 As shown, the digital human expression processing method of this application embodiment includes, but is not limited to, the following steps: S301, Obtain the text to be read by the digital human.

[0066] S302, obtain the sentiment distribution information corresponding to the broadcast text. The sentiment distribution information includes each sentiment tag and the broadcast time covered by each sentiment tag.

[0067] S303, determine the emotional intensity information associated with the emotional label, the emotional intensity information including the emotional intensity of each broadcast time covered by the emotional label.

[0068] In the embodiments of this application, steps S301-S303 can be implemented in any of the embodiments of this application, and no limitation is made here, nor will it be described in detail.

[0069] S304, based on sentiment distribution information, determine the first broadcast end time of the first sentiment tag among adjacent sentiment tags, and the first broadcast start time of the second sentiment tag among adjacent sentiment tags.

[0070] S305, determine the time overlap area of ​​adjacent emotion tags based on the end time of the first broadcast of the first emotion tag and the start time of the first broadcast of the second emotion tag.

[0071] In some embodiments, sentiment distribution information can indicate the broadcast times covered by different sentiment tags. Based on the broadcast times covered by the sentiment tags, adjacent sentiment tags are determined from multiple sentiment tags. Sentiment tags with broadcast times adjacent to each other can be identified as adjacent sentiment tags.

[0072] Among them, the adjacent sentiment tags include two sentiment tags. Let the sentiment tag with the earlier broadcast time be the first sentiment tag, and let the sentiment tag with the later broadcast time be the second sentiment tag. Then the first broadcast end time of the first sentiment tag and the first broadcast start time of the second sentiment tag can be determined.

[0073] Furthermore, the time overlap region of adjacent emotion tags can be determined based on the end time of the first broadcast of the first emotion tag and the start time of the first broadcast of the second emotion tag.

[0074] In some embodiments, it can be determined whether the start time of the first broadcast is earlier than the end time of the first broadcast, thereby determining whether the end time of the first broadcast overlaps with the start time of the first broadcast. If it is earlier, it is determined that there is a time overlap between adjacent sentiment tags.

[0075] In response to the fact that the first broadcast start time of the second sentiment tag is earlier than the first broadcast end time of the first sentiment tag, the region between the first broadcast start time of the second sentiment tag and the first broadcast end time of the first sentiment tag is determined as the candidate time overlap region.

[0076] In other words, the candidate time overlap region is the area from the start time of the first broadcast to the end time of the first broadcast.

[0077] Furthermore, the temporal overlap region of adjacent sentiment tags can be determined based on the candidate temporal overlap region. Optionally, regions whose sentiment intensity meets set conditions can be determined from the candidate temporal overlap region as the temporal overlap region of adjacent sentiment tags.

[0078] In some embodiments, the emotional intensity change information of each adjacent emotional tag within the candidate time overlap region can be determined based on the emotional intensity information associated with each adjacent emotional tag, and the region whose emotional intensity change information meets the set conditions can be identified from the candidate time overlap region as the time overlap region of adjacent emotional tags.

[0079] Optionally, the set conditions can be that the first emotional intensity of the first emotional label decreases to a first preset intensity threshold, and the second emotional intensity of the second emotional label increases to a second preset intensity threshold, and the first emotional intensity of the first emotional label continuously decreases to a third preset intensity threshold, and the second emotional intensity of the second emotional label continuously increases to a fourth preset intensity threshold.

[0080] In some embodiments, the temporal overlap region of adjacent emotion tags can be determined based on the time corresponding to the satisfaction of set conditions. The first moment when the first emotion intensity of the first emotion tag decreases to a first preset intensity threshold and the second emotion intensity of the second emotion tag increases to a second preset intensity threshold is determined as the second broadcast start time of the temporal overlap region of adjacent emotion tags.

[0081] The second broadcast end time is the time when the first emotion intensity of the first emotion tag continuously decreases to the third preset intensity threshold and the second emotion intensity of the second emotion tag continuously increases to the fourth preset intensity threshold.

[0082] In other words, the temporal overlap region of adjacent sentiment tags is the area from the start time of the second broadcast to the end time of the second broadcast. For example... Figure 5 As shown, Figure 5 If time b is the end time of the first broadcast and time a is the start time of the first broadcast, then the area between time b and time a is the candidate time overlap region.

[0083] Within the candidate time overlap region, if the emotional intensity change information of the first emotional label and the emotional intensity change information of the second emotional label meet the set conditions, it can be determined that time d is the end time of the second broadcast and time c is the start time of the second broadcast. Therefore, the time overlap region is between time c and time d.

[0084] S306, Based on the emotional intensity information of the broadcast time within the time overlap region and the emotional intensity information of the broadcast time outside the time overlap region, determine the global expression-driven data corresponding to the broadcast text.

[0085] S307: During the process of the digital human broadcasting the text, the digital human's facial expressions are synchronously driven based on global facial expression driving data.

[0086] In the embodiments of this application, steps S306-S307 can be implemented in any of the embodiments of this application, and no limitation is made here, nor will it be described in detail.

[0087] The digital human expression processing method provided in this application embodiment can determine the first broadcast end time of the first emotion tag among the adjacent emotion tags and the first broadcast start time of the second emotion tag among the adjacent emotion tags based on the emotion distribution information. Then, based on the first broadcast end time of the first emotion tag and the first broadcast start time of the second emotion tag, the time overlap region of the adjacent emotion tags can be determined. By determining the time overlap region of adjacent emotion tags and smoothly transitioning the emotion tags within the time overlap region, the digital human can smoothly switch between multiple emotions.

[0088] Figure 4 This is a flowchart illustrating another digital human expression processing method provided in an embodiment of this application, as shown below. Figure 4 As shown, the digital human expression processing method of this application embodiment includes, but is not limited to, the following steps: S401, Obtain the text to be read by the digital human.

[0089] S402, obtain the sentiment distribution information corresponding to the broadcast text. The sentiment distribution information includes each sentiment tag and the broadcast time covered by each sentiment tag.

[0090] S403, determine the emotional intensity information associated with the emotional label, including the emotional intensity of each broadcast time covered by the emotional label.

[0091] S404, based on sentiment distribution information, determines the temporal overlap region of adjacent sentiment tags.

[0092] In the embodiments of this application, steps S401-S404 can be implemented in any of the ways described in the embodiments of this application. No limitation is made here, and no further details will be provided.

[0093] S405, within the time overlap region, the emotional intensity information of adjacent emotional tags is fused to obtain the target emotional intensity of each broadcast moment within the time overlap region, while maintaining the emotional intensity of each broadcast moment outside the time overlap region unchanged, thus obtaining the global emotional intensity information corresponding to the broadcast text.

[0094] In some embodiments, after determining the temporal overlap region of adjacent emotion tags, the emotion intensity information of adjacent emotion tags within the temporal overlap region can be fused so that the emotion intensity within the temporal overlap region can transition naturally without abrupt changes.

[0095] In some embodiments, since the sentiment labels outside the time overlap region are independent sentiment labels, the sentiment intensity of each broadcast moment outside the time overlap region can be kept unchanged, and only the sentiment intensity of each broadcast moment within the time overlap region is fused.

[0096] In some embodiments, the first emotional intensity under the first emotional tag and the second emotional intensity under the second emotional tag corresponding to each broadcast time in the time overlap region can be obtained, thereby fusing the first emotional intensity and the second emotional intensity corresponding to the same broadcast time in the time overlap region to obtain the target emotional intensity of the same broadcast time.

[0097] In some embodiments, during the process of fusing the first emotional intensity and the second emotional intensity, different fusion weights can be set for the first emotional intensity and the second emotional intensity so that the first emotional intensity and the second emotional intensity can be naturally fused.

[0098] It should be noted that the emotional intensity of the first emotional label in adjacent emotional labels shows a decreasing trend, while the emotional intensity of the second emotional label in adjacent emotional labels shows an increasing trend. Therefore, the fusion weight of the first emotional label in adjacent emotional labels decreases as the emotional intensity of the first emotional label decreases, and the fusion weight of the second emotional label in adjacent emotional labels increases as the emotional intensity of the second emotional label increases. Furthermore, the sum of the fusion weights of the first and second emotional labels is 1.

[0099] In some embodiments, during the fusion of emotional intensity information of adjacent emotional tags, Bezier curve interpolation can be used to ensure a smooth transition without abrupt changes, thereby improving the effect of curve fusion.

[0100] Figure 5 This is a schematic diagram of global sentiment intensity information provided in an embodiment of this application. Figure 5 The global sentiment intensity information is represented by the global sentiment intensity time curve. Figure 5 The adjacent sentiment labels in the data are label 1 and label 2, and their respective sentiment intensity time curves are curve 1 and curve 2, respectively.

[0101] By fusing the emotional intensity information of adjacent emotional tags within the time overlap region and keeping the emotional intensity of each broadcast moment outside the time overlap region unchanged, a global emotional intensity time curve can be obtained.

[0102] S406, Based on the global emotional intensity information, determine the facial expression-driven data corresponding to each broadcast moment covered by the broadcast text.

[0103] S407, Based on the facial expression driving data corresponding to each broadcast time, determine the global facial expression driving data corresponding to the broadcast text.

[0104] In some embodiments, after obtaining global emotional intensity information, the emotional intensity corresponding to each broadcast moment can be determined based on the global emotional intensity information, thereby determining the expression-driven data corresponding to each broadcast moment based on the emotional intensity of each broadcast moment.

[0105] Optionally, the facial driving unit of the digital human can be determined based on the emotional intensity at each broadcast moment, thereby determining the expression driving data corresponding to each broadcast moment based on the facial driving unit.

[0106] For example, each emotion intensity corresponds to a different facial driving unit. Therefore, after determining the emotion intensity at each broadcast moment covered by the broadcast text, the facial driving unit for each broadcast moment can be determined based on this correspondence. The facial driving unit includes driving units for different emotion labels; for example, if the emotion label is "happy," the facial driving units include upturned corners of the mouth and outward-facing corners of the mouth.

[0107] Furthermore, the facial expression-driven data corresponding to each broadcast moment are concatenated in the order of the broadcast moments to obtain the global facial expression-driven data corresponding to the broadcast text.

[0108] In some embodiments, after determining the time overlap region, global expression-driven data corresponding to the broadcast text can be determined based on the expression-driven data of each broadcast moment within the time overlap region and the expression-driven data of the broadcast moments outside the time overlap region.

[0109] In some embodiments, the time overlap region includes a first emotion label and a second emotion label. Then, based on the emotion intensity information of the broadcast time in the time overlap region, the emotion intensity of each broadcast time in the time overlap region under the first emotion label can be determined. Thus, based on the emotion intensity of each broadcast time in the time overlap region under the first emotion label, the first expression-driven data of each broadcast time under the first emotion label can be determined.

[0110] Based on the emotional intensity information of the broadcast time within the time overlap region, the emotional intensity of each broadcast time within the time overlap region under the second emotional label is determined. Thus, based on the emotional intensity of each broadcast time within the time overlap region under the second emotional label, the second expression-driven data of each broadcast time under the second emotional label is determined.

[0111] Furthermore, for the same broadcast moment within the time overlap region, the first expression-driven data and the second expression-driven data corresponding to the same broadcast moment are fused to obtain the target expression-driven data corresponding to the same broadcast moment. In other words, by fusing the first expression-driven data and the second expression-driven data for each broadcast moment within the time overlap region, the expression-driven data corresponding to the time overlap region can be obtained.

[0112] In some embodiments, based on the emotional intensity of each broadcast moment under the corresponding emotional label outside the time overlap region, the expression-driven data of each broadcast moment under the corresponding emotional label is determined, so that global expression-driven data corresponding to the broadcast text can be obtained according to the target expression-driven data of each broadcast moment within the time overlap region and the expression-driven data of each broadcast moment outside the time overlap region.

[0113] For example, by concatenating the target facial expression-driven data for each broadcast moment within the time overlap area with the facial expression-driven data for each broadcast moment outside the time overlap area according to the broadcast moment, we can obtain the global facial expression-driven data corresponding to the broadcast text.

[0114] S408, during the process of the digital human broadcasting the text, synchronizes the digital human's facial expressions based on global facial expression driving data.

[0115] In some embodiments, the words corresponding to the broadcast text can be determined, and the expression driving data corresponding to the words can be determined from the global expression driving data based on the broadcast time of each word in the broadcast text. Thus, during the process of the digital human broadcasting the text, the expression driving data corresponding to the broadcast words can be used synchronously to drive the digital human's expression.

[0116] In other words, the digital human's expressions can be driven step by step according to the order of words to achieve synchronous driving of the digital human's expressions.

[0117] In the digital human expression processing method provided in this application embodiment, the target emotional intensity of each broadcast moment within the time overlap region is obtained by fusing the emotional intensity information of adjacent emotional tags within the time overlap region, while maintaining the emotional intensity of each broadcast moment outside the time overlap region unchanged. This yields the global emotional intensity information corresponding to the broadcast text. Further, based on the global emotional intensity information, expression-driven data corresponding to each broadcast moment covered by the broadcast text is determined. Based on the expression-driven data corresponding to each broadcast moment, global expression-driven data corresponding to the broadcast text is determined, thereby synchronously driving the digital human's expressions based on the global expression-driven data. This enables smooth transitions in the digital human's expressions even when multiple emotional tags coexist, allowing the digital human to smoothly switch between multiple emotions. This allows the digital human to simulate natural transitions in human expressions during emotional shifts, improving the smoothness and realism of the digital human.

[0118] Corresponding to the digital human expression processing methods proposed in the above embodiments, an embodiment of this application also proposes a digital human expression processing device. Since the digital human expression processing device proposed in this application corresponds to the digital human expression processing methods proposed in the above embodiments, the implementation methods of the above digital human expression processing methods are also applicable to the digital human expression processing device proposed in this application, and will not be described in detail in the following embodiments.

[0119] Figure 6 This is a schematic diagram of the structure of a digital human expression processing device provided in an embodiment of this application.

[0120] like Figure 6 As shown, the digital human's expression processing device 600 includes: The first acquisition module 601 is used to acquire the text to be read by the digital human. The second acquisition module 602 is used to acquire the sentiment distribution information corresponding to the broadcast text, wherein the sentiment distribution information includes each sentiment tag and the broadcast time covered by each sentiment tag; The first determining module 603 is used to determine the emotional intensity information associated with the emotional tag, wherein the emotional intensity information includes the emotional intensity of each broadcast time covered by the emotional tag; The second determining module 604 is used to determine the time overlap region of adjacent emotion tags based on the emotion distribution information. The third determining module 605 is used to determine the global expression driving data corresponding to the broadcast text based on the emotional intensity information of the broadcast time within the time overlap region and the emotional intensity information of the broadcast time outside the time overlap region. The expression driving module 606 is used to synchronously drive the expressions of the digital human based on the global expression driving data during the process of the digital human broadcasting the broadcast text.

[0121] In one possible implementation of this application embodiment, the second determining module 604 is further configured to: determine the first broadcast end time of the first emotional tag among the adjacent emotional tags and the first broadcast start time of the second emotional tag among the adjacent emotional tags based on the emotional distribution information; The time overlap region of the adjacent emotion tags is determined based on the end time of the first broadcast of the first emotion tag and the start time of the first broadcast of the second emotion tag.

[0122] In one possible implementation of this application embodiment, the second determining module 604 is further configured to: in response to the first broadcast start time of the second emotion tag being earlier than the first broadcast end time of the first emotion tag, determine the region between the first broadcast start time of the second emotion tag and the first broadcast end time of the first emotion tag as a candidate time overlap region; and determine the time overlap region of the adjacent emotion tags based on the candidate time overlap region.

[0123] In one possible implementation of this application, the second determining module 604 is further configured to: determine the emotional intensity change information of each of the adjacent emotional tags within the candidate time overlap region based on the emotional intensity information associated with each of the adjacent emotional tags; and identify, based on the emotional intensity change information, a region in the candidate time overlap region where the emotional intensity change information meets a set condition, as the time overlap region of the adjacent emotional tags.

[0124] In one possible implementation of this application embodiment, the second determining module 604 is further configured to: determine a first moment when the first emotion intensity of the first emotion tag decreases to a first preset intensity threshold and the second emotion intensity of the second emotion tag increases to a second preset intensity threshold, as the second broadcast start moment of the time overlap region of the adjacent emotion tags; and determine a second moment when the first emotion intensity of the first emotion tag continues to decrease to a third preset intensity threshold and the second emotion intensity of the second emotion tag continues to increase to a fourth preset intensity threshold, as the second broadcast end moment of the time overlap region of the adjacent emotion tags.

[0125] In one possible implementation of this application embodiment, the third determining module 605 is further configured to: fuse the emotional intensity information of adjacent emotional tags within the time overlap region to obtain the target emotional intensity of each broadcast moment within the time overlap region, and maintain the emotional intensity of each broadcast moment outside the time overlap region unchanged to obtain global emotional intensity information corresponding to the broadcast text; determine the expression-driven data corresponding to each broadcast moment covered by the broadcast text based on the global emotional intensity information; and determine the global expression-driven data corresponding to the broadcast text based on the expression-driven data corresponding to each broadcast moment.

[0126] In one possible implementation of this application embodiment, the third determining module 605 is further configured to: obtain the first emotional intensity under the first emotional tag and the second emotional intensity under the second emotional tag corresponding to each broadcast time in the time overlap region; and fuse the first emotional intensity and the second emotional intensity corresponding to the same broadcast time in the time overlap region to obtain the target emotional intensity of the same broadcast time.

[0127] In one possible implementation of this application embodiment, the third determining module 605 is further configured to: determine first expression-driven data for each broadcast time under the first emotion tag based on the emotion intensity of each broadcast time under the first emotion tag within the time overlap region; determine second expression-driven data for each broadcast time under the second emotion tag based on the emotion intensity of each broadcast time under the second emotion tag within the time overlap region; fuse the first expression-driven data and the second expression-driven data corresponding to the same broadcast time within the time overlap region to obtain target expression-driven data corresponding to the same broadcast time; determine expression-driven data for each broadcast time under the corresponding emotion tag based on the emotion intensity of each broadcast time outside the time overlap region under the corresponding emotion tag; and obtain global expression-driven data corresponding to the broadcast text based on the target expression-driven data for each broadcast time within the time overlap region and the expression-driven data for each broadcast time outside the time overlap region.

[0128] In one possible implementation of this application, the fusion weight corresponding to the first emotion tag among the adjacent emotion tags decreases as the emotion intensity of the first emotion tag decreases, and the fusion weight corresponding to the second emotion tag among the adjacent emotion tags increases as the emotion intensity of the second emotion tag increases.

[0129] In one possible implementation of this application embodiment, the expression driving module 606 is further configured to: determine the expression driving data corresponding to each word from the global expression driving data according to the broadcast time of each word in the broadcast text; and synchronously drive the expression of the digital human using the expression driving data corresponding to the broadcast words during the process of the digital human broadcasting the broadcast text.

[0130] The digital human expression processing device provided in this application acquires the text to be broadcast by the digital human, and obtains the emotion distribution information and emotion intensity information associated with the emotion tags corresponding to the broadcast text. Based on the emotion distribution information, the time overlap area of ​​adjacent emotion tags is determined, and based on the emotion intensity information of the broadcast time within the time overlap area and the emotion intensity information of the broadcast time outside the time overlap area, global expression driving data corresponding to the broadcast text is determined. Thus, during the digital human's broadcast of the text, the digital human's expression can be synchronously driven based on the global expression driving data. Therefore, this solution can achieve a smooth transition of the digital human's expression when multiple emotion tags coexist, that is, it enables the digital human to smoothly switch between multiple emotions, allowing the digital human to simulate the natural transition of human expressions during emotional shifts, improving the fluency and realism of the digital human.

[0131] It should be noted that the foregoing explanation of the digital human expression processing method embodiment also applies to the digital human expression processing device of this embodiment, and will not be repeated here.

[0132] To implement the above embodiments, this application also proposes an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0133] To implement the above embodiments, this application also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.

[0134] To implement the above embodiments, this application also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.

[0135] The collection, storage, use, processing, transmission, provision, and application of user personal information involved in this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0136] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.

[0137] This application is intended to provide an implementation scheme for users to selectively prevent the use or access to their personal information data. Specifically, this application is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information is de-identified to protect user privacy.

[0138] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0139] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0140] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0141] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0142] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0143] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0144] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0145] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for processing facial expressions in digital humans, characterized in that, The method further includes: Obtain the text to be read by the digital human; Obtain the sentiment distribution information corresponding to the broadcast text, wherein the sentiment distribution information includes each sentiment tag and the broadcast time covered by each sentiment tag; Determine the emotional intensity information associated with the emotional tag, wherein the emotional intensity information includes the emotional intensity of each broadcast time covered by the emotional tag; Based on the aforementioned sentiment distribution information, the temporal overlap region of adjacent sentiment tags is determined; Based on the emotional intensity information of the broadcast time within the time overlap region and the emotional intensity information of the broadcast time outside the time overlap region, the global expression driving data corresponding to the broadcast text is determined. During the process of the digital human broadcasting the text, the digital human's facial expressions are synchronously driven based on the global facial expression driving data; The step of determining the global expression-driven data corresponding to the broadcast text based on the emotional intensity information of the broadcast time within the time overlap region and the emotional intensity information of the broadcast time outside the time overlap region includes: The emotional intensity information of adjacent emotional tags is fused within the time overlap region to obtain the target emotional intensity of each broadcast moment within the time overlap region, while maintaining the emotional intensity of each broadcast moment outside the time overlap region unchanged, to obtain the global emotional intensity information corresponding to the broadcast text. Based on the global emotional intensity information, determine the facial expression-driven data corresponding to each broadcast moment covered by the broadcast text; Based on the facial expression-driven data corresponding to each broadcast time, the global facial expression-driven data corresponding to the broadcast text is determined.

2. The method according to claim 1, characterized in that, The step of determining the temporal overlap region of adjacent sentiment tags based on the sentiment distribution information includes: Based on the emotion distribution information, the first broadcast end time of the first emotion tag among the adjacent emotion tags and the first broadcast start time of the second emotion tag among the adjacent emotion tags are determined. The time overlap region of the adjacent emotion tags is determined based on the end time of the first broadcast of the first emotion tag and the start time of the first broadcast of the second emotion tag.

3. The method according to claim 2, characterized in that, The step of determining the time overlap region of the adjacent emotion tags based on the first broadcast end time of the first emotion tag and the first broadcast start time of the second emotion tag includes: In response to the fact that the first broadcast start time of the second emotion tag is earlier than the first broadcast end time of the first emotion tag, the region between the first broadcast start time of the second emotion tag and the first broadcast end time of the first emotion tag is determined as a candidate time overlap region; Based on the candidate time overlap region, the time overlap region of the adjacent sentiment tags is determined.

4. The method according to claim 3, characterized in that, The step of determining the temporal overlap region of adjacent sentiment tags based on the candidate temporal overlap region includes: Based on the sentiment intensity information associated with each of the adjacent sentiment tags, the sentiment intensity change information of each of the adjacent sentiment tags within the candidate time overlap region is determined; Based on the emotion intensity change information, regions in the candidate time overlap regions that meet the set conditions are identified as the time overlap regions of the adjacent emotion tags.

5. The method according to claim 4, characterized in that, The step of identifying regions from the candidate time overlap regions that satisfy set conditions based on the emotion intensity change information includes: The first moment when the first emotion intensity of the first emotion tag decreases to a first preset intensity threshold and the second emotion intensity of the second emotion tag increases to a second preset intensity threshold is determined as the second broadcast start time of the time overlap area of ​​the adjacent emotion tags; The second broadcast end time is the second time when the first emotion intensity of the first emotion tag continuously decreases to a third preset intensity threshold and the second emotion intensity of the second emotion tag continuously increases to a fourth preset intensity threshold.

6. The method according to claim 1, characterized in that, The process of fusing the emotional intensity information of adjacent emotional tags within the time overlap region to obtain the target emotional intensity at each broadcast time within the time overlap region includes: Obtain the first emotional intensity under the first emotional tag and the second emotional intensity under the second emotional tag for each broadcast time within the time overlap region; The first emotional intensity and the second emotional intensity corresponding to the same broadcast time within the time overlap region are fused to obtain the target emotional intensity for the same broadcast time.

7. The method according to claim 1, characterized in that, The step of determining the global expression-driven data corresponding to the broadcast text based on the emotional intensity information of the broadcast time within the time overlap region and the emotional intensity information of the broadcast time outside the time overlap region includes: Based on the emotional intensity of each broadcast time under the first emotional label within the time overlap region, the first expression driving data under the first emotional label for each broadcast time is determined; Based on the emotional intensity of each broadcast time under the second emotional label within the time overlap region, the second expression driving data under the second emotional label for each broadcast time is determined; For the same broadcast time within the time overlap region, the first expression-driven data and the second expression-driven data corresponding to the same broadcast time are fused to obtain the target expression-driven data corresponding to the same broadcast time. Based on the emotional intensity of each broadcast moment outside the time overlap region under the corresponding emotional label, determine the expression-driven data of each broadcast moment under the corresponding emotional label; Based on the target facial expression driving data for each broadcast moment within the time overlap region, and the facial expression driving data for each broadcast moment outside the time overlap region, global facial expression driving data corresponding to the broadcast text is obtained.

8. The method according to claim 6 or 7, characterized in that, The fusion weight corresponding to the first emotion tag among the adjacent emotion tags decreases as the emotion intensity of the first emotion tag decreases, while the fusion weight corresponding to the second emotion tag among the adjacent emotion tags increases as the emotion intensity of the second emotion tag increases.

9. The method according to any one of claims 1-7, characterized in that, During the process of the digital human broadcasting the text, the digital human's facial expressions are synchronously driven based on the global facial expression driving data, including: Based on the broadcast time of each word in the broadcast text, determine the corresponding expression driving data from the global expression driving data; During the process of the digital human broadcasting the text, the facial expression driving data corresponding to the broadcast words is used simultaneously to drive the digital human's facial expressions.

10. A digital human facial expression processing device, characterized in that, The device further includes: The first acquisition module is used to acquire the text to be read by the digital human; The second acquisition module is used to acquire the sentiment distribution information corresponding to the broadcast text, wherein the sentiment distribution information includes each sentiment tag and the broadcast time covered by each sentiment tag; The first determining module is used to determine the emotional intensity information associated with the emotional tag, wherein the emotional intensity information includes the emotional intensity of each broadcast time covered by the emotional tag; The second determining module is used to determine the temporal overlap region of adjacent emotion tags based on the emotion distribution information; The third determining module is used to determine the global expression driving data corresponding to the broadcast text based on the emotional intensity information of the broadcast time within the time overlap region and the emotional intensity information of the broadcast time outside the time overlap region. An expression-driven module is used to synchronously drive the expressions of the digital human based on the global expression-driven data during the process of the digital human broadcasting the text. The third determining module is further configured to: The emotional intensity information of adjacent emotional tags is fused within the time overlap region to obtain the target emotional intensity of each broadcast moment within the time overlap region, while maintaining the emotional intensity of each broadcast moment outside the time overlap region unchanged, to obtain the global emotional intensity information corresponding to the broadcast text. Based on the global emotional intensity information, determine the facial expression-driven data corresponding to each broadcast moment covered by the broadcast text; Based on the facial expression-driven data corresponding to each broadcast time, the global facial expression-driven data corresponding to the broadcast text is determined.

11. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-9.

13. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-9.

Images