Digital people broadcasting method and device, electronic equipment and storage medium

Abstract

The invention provides a digital people broadcasting method and device, electronic equipment and a storage medium, and relates to the technical field of computer technologies. The method comprises the following steps: determining current to-be-broadcasted data in response to an interruption request received in a digital human broadcasting process, the to-be-broadcasted data comprising at least one frame of to-be-broadcasted audio and video stream starting from a current progress frame; traversing the to-be-broadcasted data according to a time sequence, and determining an end frame meeting a smooth end condition in the to-be-broadcasted data, the smooth end condition comprising that the digital person is in natural pause when broadcasting to the end frame; according to the to-be-broadcasted data from the current progress frame to the end frame, determining broadcast replacement data; and carrying out digital human broadcast on the broadcast replacement data. According to the embodiment of the invention, the effect of smooth interruption can be realized, and the problems of obvious sound breakpoints, human frame skipping and the like are avoided.

Description

Background Art

[0002] Digital human technology is a comprehensive technology that integrates machine learning, natural language processing, speech synthesis and other technologies. It aims to create immersive, highly realistic virtual characters with human characteristics and interactive capabilities. With the continuous development of computer technology, digital humans are gradually being applied to all aspects of daily life.

[0003] In the related art, digital humans can be used as virtual anchors or news anchors to broadcast voice information. However, when the user pauses or exits the broadcast, the digital human broadcasting method provided by the related art will directly interrupt the digital human broadcast and action, the interruption effect is abrupt, the sound has obvious breakpoints, and the digital human screen has obvious frame skipping.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides a digital human broadcasting method, device, electronic device and storage medium, which at least to a certain extent overcome the problems of related technologies directly interrupting the digital human broadcasting and actions, resulting in a harsh interruption effect, obvious breakpoints in the sound, and obvious frame skipping of the digital human image.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a digital human broadcast is provided, comprising: in response to receiving an interruption request during the digital human broadcast process, determining current data to be broadcast, the data to be broadcast including at least one frame of audio and video stream to be broadcast starting from the current progress frame; traversing the data to be broadcast in chronological order, determining an end frame in the data to be broadcast that meets a smooth end condition, the smooth end condition including a natural pause when the digital human broadcasts to the end frame; determining broadcast replacement data according to the data to be broadcast from the current progress frame to the end frame; and performing digital human broadcast on the broadcast replacement data.

[0008] In some exemplary embodiments, in response to receiving an interruption request during the digital human broadcasting process, before determining the current data to be broadcasted, it also includes: obtaining audio information and phoneme information; generating digital human lip shape parameters according to the phoneme information; rendering the digital human lip shape parameters to obtain multiple frames of digital human lip images; sending an audio and video stream to the broadcasting end for digital human broadcasting, the audio and video stream including multiple frames of digital human lip images and audio data corresponding to the multiple frames of digital human lip images.

[0009] In some exemplary embodiments, the data to be broadcast further includes action frame information corresponding to the at least one frame of the audio and video stream to be broadcast. After obtaining the audio information and the phoneme information, the method further includes: obtaining action information; arranging the broadcast action of the digital human according to the action information to obtain multiple frames of action frame information, wherein any arranged action is composed of one action frame information or multiple consecutive action frame information, and the action includes at least one of facial expression action and body action;

[0010] The sending of the audio and video stream to the broadcasting end for the digital human broadcasting includes: sending the audio and video stream and the action frame information to the broadcasting end for the digital human broadcasting.

[0011] In some exemplary embodiments, when the data to be broadcast also includes action frame information corresponding to the at least one frame of the audio and video stream to be broadcast, the smooth ending condition also includes the end of not performing an action or performing a target action when the digital human broadcasts to the end frame.

[0012] In some exemplary embodiments, the broadcast replacement data is determined based on the data to be broadcast from the current progress frame to the end frame, including: if no action is performed when the digital human broadcasts to the end frame, the data to be broadcast from the current progress frame to the end frame is used as the broadcast replacement data.

[0013] In some exemplary embodiments, the broadcast replacement data is determined based on the data to be broadcast from the current progress frame to the end frame, including: if the end of the target action is executed when the digital human broadcasts to the end frame, the data to be broadcast from the current progress frame to the end frame is obtained; at the end of the data to be broadcast from the current progress frame to the end frame, the action frame information of the end of the target action that has not been executed and the audio and video stream in a silent state are added to obtain the broadcast replacement data, wherein the time length of the audio and video stream in the silent state is the same as the time length of the action frame information that has not been executed.

[0014] In some exemplary embodiments, in response to receiving an interruption request during the digital human broadcasting process, determining the current data to be broadcasted includes: in response to receiving an interruption request during the digital human broadcasting process, determining whether the digital human broadcasting process supports interruption; if interruption is supported, determining the current data to be broadcasted.

[0015] According to another aspect of the present disclosure, a digital human broadcasting device is also provided, comprising:

[0016] The module for determining the data to be broadcast is used to determine the current data to be broadcast in response to receiving an interruption request during the broadcasting process of the digital human, and the data to be broadcast includes at least one frame of audio and video stream to be broadcast starting from the current progress frame;

[0017] An end frame determination module is used to traverse the data to be broadcast in chronological order and determine an end frame in the data to be broadcast that satisfies a smooth end condition, wherein the smooth end condition includes a natural pause when the digital human broadcasts to the end frame;

[0018] A broadcast replacement data determination module, used for determining the broadcast replacement data according to the data to be broadcasted from the current progress frame to the end frame;

[0019] The broadcast replacement data sending module is used to perform digital human broadcasting on the broadcast replacement data.

[0020] In some exemplary embodiments, the digital human broadcasting device provided by the embodiments of the present disclosure further includes: an information acquisition module, used to acquire audio information and phoneme information; a parameter generation module, used to generate digital human lip shape parameters according to the phoneme information; an image rendering module, used to render the digital human lip shape parameters to obtain multiple frames of digital human lip images; an audio and video stream sending module, used to send an audio and video stream to the broadcasting end for digital human broadcasting, the audio and video stream including multiple frames of digital human lip images and audio data corresponding to the multiple frames of digital human lip images.

[0021] In some exemplary embodiments, the information acquisition module is also used to obtain action information; the digital human broadcasting device provided by the embodiment of the present disclosure also includes: an action frame information determination module, which is used to arrange the digital human broadcasting action according to the action information to obtain multiple frames of action frame information, wherein any arranged action is composed of one action frame information or multiple consecutive action frame information, and the action includes at least one of facial expression action and body movement; an audio and video stream sending module, which is used to send the audio and video stream and the action frame information to the broadcasting end for digital human broadcasting.

[0022] In some exemplary embodiments, when the data to be broadcast also includes action frame information corresponding to the at least one frame of the audio and video stream to be broadcast, the smooth ending condition also includes the end of not performing an action or performing a target action when the digital human broadcasts to the end frame.

[0023] In some exemplary embodiments, the broadcast replacement data determination module is used to use the to-be-broadcast data from the current progress frame to the end frame as the broadcast replacement data if no action is performed when the digital human broadcasts to the end frame.

[0024] In some exemplary embodiments, the broadcast replacement data determination module is used to obtain the data to be broadcasted from the current progress frame to the end frame if the end of the target action is executed when the digital human broadcasts to the end frame; at the end of the data to be broadcasted from the current progress frame to the end frame, add the action frame information of the end of the target action that has not been executed and the audio and video stream in a silent state to obtain the broadcast replacement data, wherein the time length of the audio and video stream in a silent state is the same as the time length of the action frame information that has not been executed.

[0025] In some exemplary embodiments, the module for determining the data to be broadcast is used to determine whether the digital human broadcast process supports interruption in response to receiving an interruption request during the digital human broadcast process; if interruption is supported, determine the current data to be broadcast.

[0026] According to another aspect of the present disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned digital human broadcasting methods by executing the executable instructions.

[0027] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any one of the above-mentioned digital human broadcasting methods is implemented.

[0028] According to another aspect of the present disclosure, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the digital human broadcasting method provided in various optional ways in any embodiment of the present disclosure.

[0029] The technical solution provided in the embodiments of the present disclosure can, when it is necessary to interrupt the digital human broadcast, traverse the data to be broadcast to find the end frame that meets the smooth end condition, and determine the broadcast replacement data based on the end frame, thereby achieving smooth interruption and avoiding problems such as obvious breakpoints in the sound and frame skipping of the human screen.

[0030] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0032] Figure 1 A schematic diagram showing a system architecture in an embodiment of the present disclosure;

[0033] Figure 2 A flow chart of a digital human broadcasting method in an embodiment of the present disclosure is shown;

[0034] Figure 3 Another flowchart of a digital human broadcasting method in an embodiment of the present disclosure is shown;

[0035] Figure 4 Another flowchart of a digital human broadcasting method in an embodiment of the present disclosure is shown;

[0036] Figure 5 A schematic diagram of a digital human broadcasting device in an embodiment of the present disclosure is shown;

[0037] Figure 6 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0038] Figure 7 A schematic diagram of a computer-readable storage medium in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0039] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0040] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0041] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0042] Figure 1 FIG. 2 shows an exemplary application system architecture diagram to which the digital human broadcasting method in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a server 101 , a network 102 and a broadcast terminal 103 .

[0043] In response to receiving an interruption request during the broadcasting process of the digital human, the server 101 can determine the current data to be broadcasted, which includes at least one frame of audio and video stream to be broadcasted starting from the current progress frame. Then, the server 101 can traverse the data to be broadcasted in chronological order to determine the end frame in the data to be broadcasted that meets the smooth end condition, and the smooth end condition includes that the digital human is in a natural pause when broadcasting to the end frame.

[0044] Then, the server 101 can determine the broadcast replacement data according to the data to be broadcasted from the current progress frame to the end frame. Finally, the server 101 can send the broadcast replacement data to the broadcast terminal 103, so that the broadcast terminal 103 replaces the data to be broadcasted with the broadcast replacement data to perform digital human broadcasting.

[0045] The network 102 is a medium for providing a communication link between the server 101 and the broadcasting terminal 103, and can be a wired network or a wireless network.

[0046] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0047] The server 101 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 101. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0048] Optionally, server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0049] The broadcasting terminal 103 may be a terminal device, which may be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, wearable devices, augmented reality devices, virtual reality devices, etc.

[0050] Optionally, the client of the application installed in different broadcast terminals 103 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.

[0051] Those skilled in the art will know that Figure 1 The number of servers 101, networks 102 and broadcast terminals 103 is only illustrative, and any number of servers 101, networks 102 and broadcast terminals 103 may be provided according to actual needs. The embodiments of the present disclosure are not limited to this.

[0052] Under the above system architecture, a digital human broadcasting method is provided in an embodiment of the present disclosure, and the method can be executed by any electronic device with computing and processing capabilities.

[0053] In some embodiments, the digital human broadcasting method provided in the embodiments of the present disclosure can be executed by the server of the above-mentioned system architecture; in other embodiments, the digital human broadcasting method provided in the embodiments of the present disclosure can be implemented by the server and the broadcasting terminal in the above-mentioned system architecture through interaction.

[0054] Figure 2 A flowchart of a digital human broadcasting method in an embodiment of the present disclosure is shown. Figure 2 As shown, the digital human broadcasting method provided in the embodiment of the present disclosure includes the following steps S202 to S208.

[0055] S202, in response to receiving an interruption request during the digital human broadcasting process, determining the current data to be broadcasted, the data to be broadcasted comprising at least one frame of audio and video stream to be broadcasted starting from the current progress frame.

[0056] In an exemplary embodiment, the audio and video stream to be broadcast may include video information corresponding to the digital human broadcast screen and audio information and phoneme information corresponding to the broadcast content, etc. The content of the audio and video stream may be determined according to the application scenario. The embodiment of the present disclosure does not limit the application scenario. For example, the application scenario may be news broadcast, entertainment live broadcast, etc.

[0057] Taking the news broadcast as an example, when the user is listening to the news broadcast, he can pause or turn off the broadcast at any time. At this time, an interruption request can be generated to interrupt the digital human broadcast.

[0058] In some embodiments, in response to receiving an interruption request during the digital human broadcasting process, before determining the current data to be broadcasted, the digital human broadcasting method provided by the embodiment of the present disclosure may also include: obtaining audio information and phoneme information; generating digital human lip shape parameters according to the phoneme information; rendering the digital human lip shape parameters to obtain multiple frames of digital human lip images; sending an audio and video stream to the broadcasting end for digital human broadcasting, the audio and video stream including multiple frames of digital human lip images and audio data corresponding to the multiple frames of digital human lip images.

[0059] It should be noted that phoneme information can be distinguished based on the physical characteristics of speech, for example, phoneme information can be used to explain the pronunciation of words and the composition of syllables, etc. Audio information refers to sound signals that can be recorded, stored, processed and transmitted electronically, which can be represented by waveforms, including parameters such as amplitude, frequency, and duration.

[0060] In an exemplary embodiment, the best lip shape corresponding to each phoneme information can be determined by a predefined phoneme-lip shape mapping rule or model, thereby determining the corresponding digital human lip shape parameters based on the phoneme information. Exemplarily, the digital human lip shape parameters can include a parameter matrix required for a digital human lip rendering model.

[0061] Then, rendering can be performed based on the digital human lip shape parameters to obtain multiple frames of digital human lip images. For example, the number of frames of the digital human lip images can be consistent with and correspond to the audio data, so that the dynamics of the digital human's lips when broadcasting can be matched with the audio data played at that time.

[0062] In some embodiments, after obtaining an audio / video stream including multiple frames of digital human lip images and audio data corresponding to the multiple frames of digital human lip images, the audio / video stream can be sent to the broadcasting end for the digital human to broadcast. After receiving the interruption request, the current playback progress of the broadcasting end, i.e., the current progress frame, can be determined. Finally, at least one frame from the current progress frame to the end of the audio / video stream obtained above can be used as the data to be broadcasted.

[0063] In an exemplary embodiment, when the digital human is reporting, in addition to the lips needing to change according to the audio data being played at the time, there may also be corresponding facial expressions and body movements.

[0064] In some embodiments, the data to be broadcast further includes action frame information corresponding to the at least one frame of the audio or video stream to be broadcast.

[0065] In an exemplary embodiment, after obtaining audio information and phoneme information, the digital human broadcasting method provided by the embodiment of the present disclosure may also include: obtaining action information; arranging the digital human broadcasting action according to the action information to obtain multiple frames of action frame information, wherein any arranged action is composed of one action frame information or multiple consecutive action frame information, and the action includes at least one of facial expression action and body movement.

[0066] In this case, sending the audio and video stream to the broadcasting end for the digital human broadcasting includes: sending the audio and video stream and the action frame information to the broadcasting end for the digital human broadcasting.

[0067] For example, the action frame information can be used to identify the action of the audio and video stream, so that the digital human can make corresponding facial expressions, body movements, etc. according to the played audio data during the broadcast process. In some possible implementations, the action information can be preset, and the action arrangement can be performed based on the action information to obtain the action frame information. For example, the facial expression can include smiling, closing the mouth, etc. The body movement can include spreading hands, waving hands, etc.

[0068] Any action can be completed by at least one frame of digital human expression image or body image, so any action can be composed of one action frame information or a plurality of continuous action frame information. Any action frame information is used to describe the digital human expression image or body image corresponding to the frame. After the action frame information is generated, each action frame information can be inserted into the corresponding position in the audio and video stream. Exemplarily, the insertion of each action frame information into the corresponding position in the audio and video stream can be realized by action frame index.

[0069] In some embodiments, in response to receiving an interruption request during the digital human broadcasting process, determining the current data to be broadcasted includes: in response to receiving an interruption request during the digital human broadcasting process, determining whether the digital human broadcasting process supports interruption; if interruption is supported, determining the current data to be broadcasted.

[0070] In an exemplary embodiment, whether the broadcasting process of the digital human supports interruption can be set based on experience or application scenarios. If the broadcasting process of the digital human supports interruption, smooth interruption can be achieved according to the method provided in the embodiment of the present disclosure. If the broadcasting process of the digital human does not support interruption, the playback end can send an instruction message to continue playing. In addition, the user can be prompted that interruption is not currently supported by means of a pop-up window prompt or the like.

[0071] S204, traversing the data to be broadcast in chronological order to determine an end frame in the data to be broadcast that satisfies a smooth end condition, wherein the smooth end condition includes a natural pause when the digital human broadcasts to the end frame.

[0072] In an exemplary embodiment, audio information, phoneme information, action frame information, etc. may be cached. When an interruption request is obtained, the playback progress frame of the broadcaster may be determined and the information in the cache may be obtained, thereby obtaining the to-be-broadcasted data including at least one frame of the audio and video stream to be broadcast starting from the current progress frame.

[0073] For example, the current playback progress of the broadcaster, i.e., the playback progress frame, can be determined by the action frame index, and the information in the cache can be obtained by the action frame index. Then, the action frame index can be postponed according to the subsequent calculation time and data transmission delay. The subsequent calculation time can represent the calculation time required to determine the end frame and obtain the broadcast replacement data. The data transmission delay can represent the time for sending the broadcast replacement data to the broadcast end.

[0074] Exemplarily, it can be determined whether each frame is a silent frame. If a currently traversed frame is a silent frame, then when the digital human broadcasts to this frame, it is at a natural pause, and this frame can be used as the end frame that meets the smooth end condition.

[0075] In some embodiments, when the data to be broadcast also includes action frame information corresponding to the at least one frame of the audio and video stream to be broadcast, the smooth ending condition also includes the end of not performing an action or performing a target action when the digital human broadcasts to the end frame.

[0076] For example, it can be determined whether there is action frame information inserted in each frame. If the currently traversed frame does not have action frame information inserted, no action is performed when the digital human broadcasts to the frame. In addition, if the currently traversed frame has action frame information inserted with the target action, it can be determined whether the end of the target action is performed in the frame.

[0077] The disclosed embodiment does not limit the judgment method of whether the target action is executed at the end, and the judgment method can be set based on experience or application scenarios. For example, when the target action is currently executed by 90% or 95%, it is considered that the target action is currently executed. Alternatively, if the number of frames that the target action has not yet been executed is not greater than 12 frames or 24 frames, it can be considered that the target action is currently executed.

[0078] In a possible implementation, it can be determined whether a frame currently being traversed is a silent frame. If it is not a silent frame, the traversal continues. If it is a silent frame, it is determined whether no action is performed when the digital human broadcasts to the frame. If no action is performed, the frame is determined to be an end frame. If an action is performed, it is determined whether the frame is the end of the target action. If the end of the target action is performed, the frame is determined to be an end frame. If the end of the target action is not performed, the traversal continues.

[0079] It should be noted that the target action is not limited in the embodiments of the present disclosure. For example, the target action may be an expression of closing the mouth or a body movement of spreading the hands.

[0080] S206: Determine the broadcast replacement data according to the to-be-broadcast data from the current progress frame to the end frame.

[0081] In an exemplary embodiment, since the data to be broadcasted is directly sent to the broadcast end for broadcasting, the interruption may cause abrupt interruption effects, obvious breakpoints in the sound, obvious frame skipping of the digital human screen, etc. Therefore, the broadcast replacement data that can achieve a smooth interruption effect can be determined first, and the push end can be instructed to discard the data to be broadcasted and send the broadcast replacement data to the broadcast end.

[0082] In some embodiments, the broadcast replacement data is determined based on the data to be broadcast from the current progress frame to the end frame, including: if no action is performed when the digital human broadcasts to the end frame, the data to be broadcast from the current progress frame to the end frame is used as the broadcast replacement data.

[0083] In an exemplary embodiment, if a frame in which the broadcast is paused and the digital human does not perform any action can be found when traversing the data to be played, the smooth interruption effect can be achieved by playing normally to this frame. Therefore, this frame can be directly used as the end frame, and the data to be broadcast from the current progress frame to the end frame can be used as the broadcast replacement data.

[0084] In some embodiments, the broadcast replacement data is determined based on the data to be broadcast from the current progress frame to the end frame, including: if the end of the target action is executed when the digital human broadcasts to the end frame, the data to be broadcast from the current progress frame to the end frame is obtained; at the end of the data to be broadcast from the current progress frame to the end frame, the action frame information of the end of the target action that has not been executed and the audio and video stream in a silent state are added to obtain the broadcast replacement data, wherein the time length of the audio and video stream in the silent state is the same as the time length of the action frame information that has not been executed.

[0085] In a possible implementation, under the premise that the digital human is in a natural pause when broadcasting to the frame, if the frame is at the end of the target action, the frame can be used as the end frame, and the data to be broadcasted from the current progress frame to the end frame is obtained, and the audio and video stream in a silent state is added to the end of the data to be broadcasted in this part to obtain the broadcast replacement data. Exemplarily, the audio and video stream in a silent state can include action frame information corresponding to the closing mouth action and silent audio information.

[0086] The embodiment of the present disclosure does not limit the time length of the supplemented audio and video stream in the silent state. Exemplarily, the time length of the audio and video stream in the silent state may be consistent with the time length of the supplemented action frame information.

[0087] In one possible implementation, after receiving an interruption request and querying the current playback progress, the original audio and video streams to be sent and the queue to be rendered can be discarded, and the audio information, digital human lip shape parameters and action information that need to be supplemented can be re-received, so as to re-render and overwrite the new digital human lip image, and push new broadcast replacement data.

[0088] S208, performing digital human broadcasting on the broadcast replacement data.

[0089] In an exemplary embodiment, the architecture of the digital human broadcast may include a scheduling system, a lip and action arrangement service module, and a picture rendering and push service module. The lip and action arrangement service can be used to generate audio data to be broadcast according to text information and action information. And it can obtain phoneme information according to the text information, and generate the digital human lip shape parameters according to the phoneme information. In addition, the lip and action arrangement service module can also arrange the action of each frame of the digital human according to the action information to obtain action frame information.

[0090] The image rendering and push service module can be used to render the digital human lip shape parameters to obtain multiple frames of digital human lip images. The rendered digital human lip images can be merged with the complete digital human image. Then, the image rendering and push service module can also segment the audio data and push the audio and video streams to the playback end frame by frame.

[0091] After receiving the interruption request, the scheduling system can determine the current data to be broadcast and determine the end frame. Then, the scheduling system can determine the broadcast replacement data according to the end frame. The scheduling system can send the broadcast replacement data to the broadcast terminal for the digital human broadcast.

[0092] Exemplarily, performing digital human broadcasting on the broadcast replacement data may include: sending the broadcast replacement data to the broadcast terminal, so that the broadcast terminal replaces the data to be broadcasted with the broadcast replacement data to perform digital human broadcasting.

[0093] Among them, if the digital human performs the end of the target action when the digital human broadcasts to the end frame, the scheduling system can receive the audio information, digital human lip shape parameters and action information that need to be supplemented, and re-render and overwrite the new digital human lip image through the picture rendering and push service module, and push the new broadcast replacement data to the playback end.

[0094] The method provided by the embodiment of the present disclosure can traverse the data to be broadcast to find the end frame that meets the smooth end condition when it is necessary to interrupt the broadcast of the digital human, and determine the broadcast replacement data according to the end frame, thereby achieving smooth interruption and avoiding problems such as obvious breakpoints in the sound and frame skipping of the human screen.

[0095] For example, a flowchart of a digital human broadcasting method can be as follows: Figure 3 As shown. Figure 3 The process may include steps S302 to S316. Among them, steps S308 to S314 are interruption processes.

[0096] S302, receiving a broadcast message.

[0097] S304, obtaining TTS (Text-to-Speech) audio information.

[0098] S306, acquiring multiple frames of digital human lip images according to the phoneme information, and arranging the digital human action frame information according to the action information.

[0099] S308, determine whether the digital human broadcasting process supports interruption. If yes, execute step S310. If no, execute step S314.

[0100] S310, buffering audio information, phoneme information and action frame information.

[0101] S312, start the interruption information monitoring thread to obtain the broadcast replacement data.

[0102] For example, the life span of the interruption information monitoring thread can be set according to the message duration. The message duration can refer to the broadcast duration corresponding to the data to be broadcasted.

[0103] S314, push the broadcast message frame by frame.

[0104] Exemplarily, the broadcast message may be data to be broadcast or broadcast replacement data.

[0105] S316, end.

[0106] It should be noted that the implementation of steps S302 to S316 can refer to the corresponding description of S202 to S208 above, which will not be repeated here.

[0107] For example, a flowchart of a digital human broadcasting method can be as follows: Figure 4 As shown. Figure 4 The process may include steps S402 to S426.

[0108] S402: Receive an interruption request.

[0109] S404, obtaining the current streaming progress a.

[0110] It should be noted that the current streaming progress a is the above-mentioned current progress frame.

[0111] S406, postpone the action frame index according to the subsequent calculation time and information transmission delay.

[0112] S408: Obtain cache information according to the action frame index.

[0113] S410, traverse the cache frames.

[0114] S412, determining whether there is any frame information that has not been traversed.

[0115] S414, take out a frame.

[0116] S416, determine whether it is an action frame. If yes, execute S418. If no, execute S420.

[0117] S418, determine whether it is the end of the action frame. If yes, execute S420. If no, execute S412.

[0118] S420, determine whether it is a silent frame. If so, execute S422. If not, execute S412.

[0119] S422, using the current traversal point index to set the end frame to b.

[0120] S424, determining the broadcast replacement data according to a and b, and pushing the broadcast replacement data frame by frame to the broadcast end.

[0121] S426, end.

[0122] It should be noted that the implementation of steps S402 to S426 can refer to the corresponding description of S202 to S208 above, which will not be repeated here.

[0123] It should be noted that the acquisition, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations, and various types of data such as personal identity data, operation data, behavioral data, etc. related to individuals, customers, and groups obtained in the embodiments of this disclosure have all been authorized.

[0124] Based on the same inventive concept, the present disclosure also provides a digital human broadcasting device, as described in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0125] Figure 5A schematic diagram of a digital human broadcasting device in an embodiment of the present disclosure is shown. Figure 5 As shown, the device comprises:

[0126] The module 501 for determining the data to be broadcast is used to determine the current data to be broadcast in response to receiving an interruption request during the broadcasting process of the digital human, and the data to be broadcast includes at least one frame of audio and video stream to be broadcast starting from the current progress frame;

[0127] The end frame determination module 502 is used to traverse the data to be broadcast in time order and determine the end frame in the data to be broadcast that meets the smooth end condition. The smooth end condition includes that the digital human is in a natural pause when broadcasting to the end frame;

[0128] The broadcast replacement data determination module 503 is used to determine the broadcast replacement data according to the to-be-broadcast data from the current progress frame to the end frame;

[0129] The broadcast replacement data sending module 504 is used to perform digital human broadcasting on the broadcast replacement data.

[0130] In some exemplary embodiments, the digital human broadcasting device provided by the embodiments of the present disclosure further includes:

[0131] An information acquisition module, used to acquire audio information and phoneme information;

[0132] A parameter generation module, used for generating digital human lip shape parameters according to the phoneme information;

[0133] An image rendering module is used to render the digital human lip shape parameters to obtain multiple frames of digital human lip images;

[0134] The audio and video stream sending module is used to send the audio and video stream to the broadcasting end for digital human broadcasting, and the audio and video stream includes multiple frames of digital human lip images and audio data corresponding to the multiple frames of digital human lip images.

[0135] In some exemplary embodiments, the information acquisition module is also used to acquire action information; the digital human broadcasting device provided by the embodiment of the present disclosure also includes:

[0136] An action frame information determination module is used to arrange the digital human broadcast action according to the action information to obtain multiple frames of action frame information, wherein any arranged action is composed of one action frame information or multiple consecutive action frame information, and the action includes at least one of an expression action and a body action;

[0137] The audio and video stream sending module is used to send the audio and video stream and the action frame information to the broadcasting end for digital human broadcasting.

[0138] In some exemplary embodiments, when the data to be broadcast also includes action frame information corresponding to the at least one frame of the audio and video stream to be broadcast, the smooth ending condition also includes the end of not performing an action or performing a target action when the digital human broadcasts to the end frame.

[0139] In some exemplary embodiments, the broadcast replacement data determination module 503 is used to use the to-be-broadcasted data from the current progress frame to the end frame as the broadcast replacement data if no action is performed when the digital human broadcasts to the end frame.

[0140] In some exemplary embodiments, the broadcast replacement data determination module 503 is used to obtain the data to be broadcasted from the current progress frame to the end frame if the end of the target action is executed when the digital human broadcasts to the end frame; at the end of the data to be broadcasted from the current progress frame to the end frame, add the action frame information of the end of the target action that has not been executed and the audio and video stream in a silent state to obtain the broadcast replacement data, wherein the time length of the audio and video stream in a silent state is the same as the time length of the action frame information that has not been executed.

[0141] In some exemplary embodiments, the module for determining the data to be broadcast 501 is used to determine whether the digital human broadcast process supports interruption in response to receiving an interruption request during the digital human broadcast process; if interruption is supported, determine the current data to be broadcast.

[0142] The device provided by the embodiment of the present disclosure can traverse the data to be broadcast to find an end frame that meets the smooth end condition when it is necessary to interrupt the broadcast of the digital human, and determine the broadcast replacement data according to the end frame, thereby achieving smooth interruption and avoiding problems such as obvious breakpoints in the sound and frame skipping of the human screen.

[0143] It should be noted that the above-mentioned to-be-broadcasted data determination module 501, the end frame determination module 502, the broadcast replacement data determination module 503, and the broadcast replacement data sending module 504 correspond to S202 to S208 in the method embodiment, and the examples and application scenarios implemented by the above-mentioned modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0144] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0145] The embodiment of the present disclosure provides an electronic device, which exemplarily includes: a processor and a memory. The memory can be used to store executable instructions of the processor. The processor is configured to perform the digital human broadcasting method provided by the embodiment of the present disclosure by executing the above executable instructions.

[0146] Refer to the following Figure 6 The electronic device 600 according to this embodiment of the present disclosure is described. Figure 6 The electronic device 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0147] like Figure 6 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include but are not limited to: at least one processing unit 610, at least one storage unit 620, and a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610).

[0148] The storage unit stores a program code, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps of various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 610 can perform the following steps of the above method embodiment:

[0149] In response to receiving an interruption request during the digital human broadcasting process, the current data to be broadcasted is determined, and the data to be broadcasted includes at least one frame of audio and video stream to be broadcasted starting from the current progress frame; the data to be broadcasted is traversed in time sequence to determine the end frame in the data to be broadcasted that meets the smooth end condition, and the smooth end condition includes that the digital human is in a natural pause when broadcasting to the end frame; according to the data to be broadcasted from the current progress frame to the end frame, the broadcast replacement data is determined; and the digital human broadcasts the broadcast replacement data.

[0150] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0151] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0152] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0153] The electronic device 600 may also communicate with one or more external devices 640 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 660. As shown, the network adapter 660 communicates with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0154] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0155] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart may be implemented as a computer program product, which includes: a computer program, which implements the above-mentioned digital human broadcasting method when executed by a processor.

[0156] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, and when the computer program is executed by a processor, the digital human broadcasting method provided in the embodiment of the present disclosure can be implemented. The computer-readable storage medium can be a readable signal medium or a readable storage medium.

[0157] Figure 7 A schematic diagram of a computer-readable storage medium in an embodiment of the present disclosure is shown. Figure 7As shown, a program product capable of implementing the above method of the present disclosure is stored on the computer-readable storage medium 700. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0158] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0159] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0160] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0161] In a specific implementation, the program code for performing the operation of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0162] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0163] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0164] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0165] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope of the present disclosure is indicated by the appended claims.

Claims

1. A digital human broadcasting method, characterized in that: include: In response to receiving an interruption request during the digital human broadcasting process, determining current data to be broadcasted, wherein the data to be broadcasted includes at least one frame of audio and video stream to be broadcasted starting from the current progress frame; Traversing the data to be broadcast in chronological order, determining an end frame in the data to be broadcast that satisfies a smooth end condition, wherein the smooth end condition includes a natural pause when the digital human broadcasts to the end frame; Determine the broadcast replacement data according to the to-be-broadcast data from the current progress frame to the end frame; The broadcast replacement data is broadcasted by a digital human.

2. The digital human broadcasting method according to claim 1, characterized in that: In response to receiving an interruption request during the digital human broadcasting process, before determining the current data to be broadcasted, the method further includes: Get audio information and phoneme information; Generate digital human lip shape parameters according to the phoneme information; Rendering the digital human lip shape parameters to obtain multiple frames of digital human lip images; An audio and video stream is sent to the broadcasting end for digital human broadcasting, wherein the audio and video stream includes multiple frames of digital human lip images and audio data corresponding to the multiple frames of digital human lip images.

3. The digital human broadcasting method according to claim 2, characterized in that: The data to be broadcast further includes action frame information corresponding to the at least one frame of the audio and video stream to be broadcast, and after obtaining the audio information and the phoneme information, the method further includes: Get action information; Arranging the broadcasting action of the digital human according to the action information to obtain multiple frames of action frame information, wherein any arranged action is composed of one action frame information or multiple consecutive action frame information, and the action includes at least one of facial expression action and body movement; The sending of the audio and video streams to the broadcasting terminal for digital human broadcasting includes: The audio and video streams and the action frame information are sent to the broadcasting end to perform digital human broadcasting.

4. The digital human broadcasting method according to claim 1 or 3, characterized in that: When the data to be broadcast further includes action frame information corresponding to the at least one frame of audio and video stream to be broadcast, the smooth ending condition further includes the end of the digital human not performing an action or performing a target action when broadcasting to the end frame.

5. The digital human broadcasting method according to claim 4, characterized in that: The step of determining the broadcast replacement data according to the data to be broadcasted from the current progress frame to the end frame includes: If no action is performed when the digital human broadcasts to the end frame, the data to be broadcasted from the current progress frame to the end frame is used as the broadcast replacement data.

6. The digital human broadcasting method according to claim 4, characterized in that: The step of determining the broadcast replacement data according to the data to be broadcasted from the current progress frame to the end frame includes: If the target action ends when the digital human broadcasts to the end frame, the data to be broadcasted from the current progress frame to the end frame is obtained; At the end of the data to be broadcast from the current progress frame to the end frame, add the action frame information of the end of the target action that has not been executed and the audio and video stream in a silent state to obtain the broadcast replacement data, wherein the time length of the audio and video stream in a silent state is the same as the time length of the action frame information that has not been executed.

7. The digital human broadcasting method according to claim 1, characterized in that: The step of determining the current data to be broadcasted in response to receiving an interruption request during the broadcasting process of the digital human includes: In response to receiving an interruption request during the digital human broadcasting process, determining whether the digital human broadcasting process supports interruption; If interruption is supported, the current data to be broadcast is determined.

8. A digital human broadcasting device, characterized in that: include: The module for determining the data to be broadcast is used to determine the current data to be broadcast in response to receiving an interruption request during the broadcasting process of the digital human, wherein the data to be broadcast includes at least one frame of audio and video stream to be broadcast starting from the current progress frame; An end frame determination module is used to traverse the data to be broadcast in chronological order and determine an end frame in the data to be broadcast that satisfies a smooth end condition, wherein the smooth end condition includes a natural pause when the digital human broadcasts to the end frame; A broadcast replacement data determination module, used for determining the broadcast replacement data according to the to-be-broadcast data from the current progress frame to the end frame; The broadcast replacement data sending module is used to perform digital human broadcasting on the broadcast replacement data.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the digital human broadcasting method described in any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the digital human broadcasting method described in any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the digital human broadcasting method as described in any one of claims 1 to 7.

Images