Video playing method and related equipment

Abstract

The embodiment of the invention discloses a video playing method and related equipment, and the method comprises the steps: in a process of playing a first video, in response to the obtained input information, determining the prediction time of generating a second video corresponding to the input information, determining a switching video frame from a plurality of video frames included in the first video based on the playing condition of the first video and a reserved duration determined based on the predicted time, generating a second video based on the switching video frame, and determining the actual time for generating the second video, and playing the first video and the second video according to the actual time and the predicted time. According to the embodiment of the invention, the switching video frame can be reasonably determined, so that the video playing effect is ensured.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video playback method and related equipment. Background Art

[0002] Currently, when a video is playing, usually some information is input to change the video playing. Therefore, during the video playing process, how to play the video based on the user's input information has become a hot research issue. Summary of the invention

[0003] The embodiments of the present application provide a video playback method and related devices, which can reasonably determine the switching of video frames to ensure the video playback effect.

[0004] In a first aspect, an embodiment of the present application provides a video playback method, the method comprising:

[0005] In the process of playing the first video, in response to obtaining input information, determining a predicted time for generating a second video corresponding to the input information, and determining a switching video frame from a plurality of video frames included in the first video based on the playing status of the first video and a reserved duration; the reserved duration is determined based on the predicted time;

[0006] generating a second video based on the switching video frame, and determining an actual time of generating the second video;

[0007] The first video and the second video are played according to the actual time and the predicted time.

[0008] In a second aspect, an embodiment of the present application provides a video playback device, the device comprising: a determination unit, a generation unit and a playback control unit, wherein:

[0009] The determining unit is used to determine, in response to obtaining input information during the process of playing the first video, a predicted time for generating a second video corresponding to the input information, and determine a switching video frame from a plurality of video frames included in the first video based on the playing status of the first video and a reserved duration; the reserved duration is determined based on the predicted time;

[0010] The generating unit is configured to generate a second video based on the switching video frame, and determine an actual time for generating the second video;

[0011] The playback control unit is used to play the first video and the second video according to the actual time and the predicted time.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the program includes instructions for executing the steps in the first aspect of the embodiment of the present application.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps described in the first aspect of the embodiment of the present application.

[0014] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect of the embodiment of the present application. The computer program product may be a software installation package.

[0015] The implementation of the embodiments of the present application has the following beneficial effects:

[0016] The video playback method and related equipment described in the present application, in the process of playing the first video, in response to obtaining input information, determine the predicted time of generating the second video corresponding to the input information, and determine the switching video frame from the multiple video frames included in the first video based on the playback status and reserved time of the first video, generate the second video based on the switching video frame, and determine the actual time of generating the second video, play the first video and the second video according to the actual time and the predicted time, so that, in the case of changing the playback of the first video according to the input information, first predict the predicted time required to generate the second video corresponding to the input information, and then determine the reasonable reserved time corresponding to the predicted time, and combine the reserved time and the current playback status of the first video to accurately and reasonably determine the switching video frame in the first video, and then dynamically control the playback of the first video and the second video based on the difference between the actual time and the predicted time. Since the second video is generated based on the switching video frame, the video playback method of the present application can ensure that the first video is naturally transitioned to the second video by switching the video frame, that is, it can ensure that the transition between the first video and the second video is more natural during playback, and thus, the video playback effect can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of a video playback method provided in an embodiment of the present application;

[0019] Figure 2 This is a demonstration schematic diagram of an 18-key-point model of the human body provided in an embodiment of the present application;

[0020] Figure 3 This is a demonstration schematic diagram of another 18-key-point model of the human body provided in an embodiment of the present application;

[0021] Figure 4 It is a structural schematic diagram of a time prediction model provided in an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of the structure of another time prediction model provided in an embodiment of the present application;

[0023] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0024] Figure 7 It is a block diagram of the functional units of a video playback device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0027] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0028] The electronic devices described in the embodiments of the present application may include smart phones (such as Android phones, iOS phones, Windows Phone phones, etc.), tablet computers, PDAs, driving recorders, laptop computers, mobile Internet devices (mobile internet devices, MID) or wearable devices (such as smart watches, Bluetooth headsets), etc. The above are only examples and not exhaustive, including but not limited to the above electronic devices. The electronic device may also include a server, which may be an outsourced server, a cloud server, an edge server, a conversational robot, a server cluster, etc., which are not limited here.

[0029] The following is an introduction to the relevant terms involved in this application.

[0030] The first video may specifically refer to a group of frames that are not generated in real time, that is, originally shot video frames, which are used for loop playback.

[0031] The second video can be specifically understood as a video generated based on the input information in response to the input information during the playing of the first video. For example, the first video is played in a loop, and once the user asks a question, the second video will be newly generated according to the response text of the dialogue robot.

[0032] The reserved duration is the duration between the first moment when the first video frame in the first video is being played when the input information is obtained and the second moment when the first video is played to switch the video frame. That is, within the reserved duration, the first video frame in the first video is played until the video frame is switched.

[0033] The switching video frame can be specifically understood as a video frame of the first video. Specifically, when the input information is obtained, the first video frame in the first video is being played, and the video frame in the first video that is sorted after the first video frame is determined as the second video frame, and then the second video frame corresponding to the reserved duration is determined from the second video frame, and the determined second video frame is used as the switching video frame. After playing the switching video frame, you can switch to playing the second video.

[0034] The transition frame is a transition frame generated by an interpolation algorithm based on a switching video frame in the first video. It is the first frame of the second video and is used to connect the first video with the second video to achieve a smooth transition between the first video and the second video.

[0035] Full-link response time: The full-link response time is composed of the automatic speech recognition (ASR) duration, the dialogue robot duration, the text to speech (TTS) duration, the digital human driving and rendering algorithm duration, and the network transmission duration. Specifically, for example, the full-link response time = ASR duration + dialogue robot duration + TTS duration + digital human driving and rendering algorithm duration + network transmission duration.

[0036] Stuttering means that the playback stops at a certain frame and the picture is still.

[0037] Turn back, that is, play to a certain frame and start playing in reverse, that is, play in the opposite direction of the original playback direction.

[0038] Frame skipping can be understood as a video consisting of a group of continuous images. The difference between each two adjacent frames is not much. If a few frames are lost in the middle, the video will skip a few frames during playback, resulting in obvious picture discontinuity.

[0039] In the related technologies, 2D digital humans are taken as an example. As a kind of digital human with real-life effects, 2D digital humans have been widely used. For example, some digital humans have been put into service in the fields of enterprise digital employees, product explainers, cultural and tourism explainers, and public services. The essence of 2D digital humans is the re-editing of videos. For example, only half of the face area is generated by the algorithm, and the other non-generated areas are taken from the original video. Then the generated part is merged with the original video to get a 2D digital human.

[0040] In actual applications, the 2D real-time digital human system responds to user questions in the form of images and voice in a very short time. When there are no questions, the digital human is in standby mode, and the first video is played in a loop. Once the user asks a question, the second video will be generated according to the response text of the dialogue robot, and the second video will be played after the video frame is switched. Since the digital human driving and rendering algorithms need some time to generate the second video, the first video needs to be played to wait for the algorithm to generate the second video.

[0041] Specifically, the following two playback modes are mainly used:

[0042] The first method: use a short video (first video) to loop. When the second video has been generated, it is necessary to wait until the short video loop is finished before switching to play the second video. That is, the first frame of the second video is determined based on a fixed frame in the first video. Although the short video loop switches quickly, the loop feeling is very serious and looks unnatural.

[0043] The second method: use a long video (first video) to loop. In this case, the first frame of the second video is determined based on a frame in the first video, but the frame is not fixed. Since the reserved time is manually specified, it is easy to have a video rewind phenomenon, which is unnatural and has a poor user experience.

[0044] In view of the defects of the related art, the current goals are: 1) shorten the full-link response time; 2) the second video is continuous and natural, without freezing, frame skipping, and frequent looking back. Since the digital human driving and rendering algorithm requires a period of time to generate the second video, it is necessary to continuously play the first video within the reserved duration to wait for the second video to be generated. If the reserved duration is too long, the response time will be increased, and if the reserved duration is too short, the playback will freeze and skip frames. Therefore, an embodiment of the present application provides a video playback method, the method comprising: in the process of playing a first video, in response to obtaining input information, determining a predicted time for generating a second video corresponding to the input information, and determining a switching video frame from a plurality of video frames included in the first video based on the playback status of the first video and the reserved duration; the reserved duration is determined based on the predicted time; generating a second video based on the switching video frame, and determining the actual time for generating the second video; playing the first video and the second video according to the actual time and the predicted time.

[0045] In the embodiment of the present application, when it is necessary to change the playback of the first video according to the input information, the predicted time required to generate the second video corresponding to the input information is first predicted, and then a reasonable reserved duration corresponding to the predicted time is determined. The switching video frame in the first video can be accurately and reasonably determined in combination with the reserved duration and the current playback status of the first video, and then the playback of the first video and the second video can be dynamically controlled based on the difference between the actual time and the predicted time. Since the second video is generated based on the switching video frame, the video playback method of the present application can ensure that the first video is naturally transitioned to the second video by switching the video frame, that is, it can ensure that the transition between the first video and the second video during playback is relatively natural, and thus, the video playback effect can be guaranteed.

[0046] The following is a detailed description of the embodiments of the present application.

[0047] See also Figure 1 , Figure 1: is a flow chart of a video playback method provided by an embodiment of the present application. As shown in the figure, the video playback method includes:

[0048] 101. During the playback of a first video, in response to obtaining input information, determine a predicted time for generating a second video corresponding to the input information, and determine a switching video frame from multiple video frames included in the first video based on the playback status and reserved duration of the first video.

[0049] The reserved duration is determined based on the predicted time. Input information can be understood as input information acquired by an electronic device, and the input information may include at least one of the following: text content, image content, video content, audio content, sensor acquisition, etc., which are not limited here. For example, the input information may include: voice, text, gesture, lip reading, etc.

[0050] The playback status of the first video may include at least one of the following: the playback order of the first video, the playback position of the first video, the playback rate of the first video, the playback resolution of the first video, etc., which are not limited here. The playback order can be understood as the video playback order, which can include a forward playback order or a reverse playback order. The forward playback order is to play in the normal playback order of the video, and the reverse playback order is to play in the opposite playback order of the normal playback order of the video. The playback position of the first video can be understood as which frame is currently being played.

[0051] In a specific implementation, during the playback of the first video, the user input information can be waited for. After the input information is detected, the input information can be obtained in response, and the switching video frame can be determined from the multiple video frames included in the first video based on the playback status of the first video and the reserved time. That is, the reserved time corresponds to the switching video frame. Since the reserved time corresponds to the input information, the precise prediction time is determined based on the input information, and the reasonable reserved time corresponding to the prediction time is determined. Therefore, a reasonable switching video frame is determined based on the reserved time to avoid freezes and frame skipping, so as to ensure the playback effect of the digital human.

[0052] In an embodiment of the present application, the original video can be converted into a looping video for the digital human. Specifically, an interpolation algorithm can be used to convert the original video into a looping video, and the looping video is used as the first video.

[0053] The original video may be an original video of a digital human, and the digital human may include a 2D digital human or a 3D digital human. A digital human not only refers to a virtual human image, but may also refer to a cartoon character, an animal-shaped digital human, an object-shaped digital human, etc., which are not limited here. Of course, the original video may not include a digital human, but may be processed into a looping video including a digital human.

[0054] In the specific implementation, since it is necessary to generate digital human videos of different lengths, a sufficiently long loop video is required. In addition, the shot video is ultimately of limited length, so a loop video that can be played in a loop is required.

[0055] In the specific implementation, for the original video of the digital human, since it is ensured that the character will return to the initial action within a period of time when the original video is shot, the conditions for forming a loop video are met. Therefore, it is only necessary to filter out the frames in a video that return to the initial action and select a certain duration to obtain an original video to be converted into a loop video. The following steps A1 to A3 can be referred to:

[0056] A1, we can use the pose key point model to annotate all frames with pose key points and obtain the key point coordinates (x i ,y i )∈P f , where P f Represents the key point set of each frame of a video clip f, i=1,2,...,n is the i-th key point of each frame, and n is the number of key points in each frame.

[0057] In the specific implementation, since there is usually only one person in the shot, the person's movements are not large, and the whole person keeps facing the camera, a 2D posture key point algorithm can be used. The 2D posture key point detection algorithm is a common technology in the field. For example, Figure 2 As shown, it provides an 18-key point model of the human body.

[0058] Among them, the human key point model is a subtask of human posture estimation. Take the COCO key point model as an example, the 18-point model trained in the COCO dataset. The key points and serial numbers defined in the COCO dataset are as follows: nose-0, neck-1, right shoulder-2, right elbow-3, right wrist-4, left shoulder-5, left elbow-6, left wrist-7, right hip-8, right knee-9, right ankle-10, left hip-11, left knee-12, left ankle-13, right eye-14, left eye-15, ear-16, left ear-17, background-18 (not shown in the figure), please refer to the details. Figure 3For example, the key point detection model can be applied to the VGGNet model. The first 10 layers of the VGGNet model are used to create feature maps for the input image and obtain multiple features. Then, these features can be input into two parallel branches of the convolution layer. The first branch predicts a set of confidence maps (18), each of which represents a specific part of the human body posture skeleton graph. The key point detection task corresponds to the first branch. The second branch predicts a set of Part Affinity Fields (PAF, 38), and PAF represents the degree of association between parts.

[0059] Specifically, taking the opencv function as an example, the input of the key point detection model can be an original image, without any processing. The image can be pre-processed in advance through the functions provided by opencv, including subtracting the mean, scaling, cropping, swapping channels, etc., and returning a 4-channel blob (blob can be simply understood as an N-dimensional array for the input of the neural network). This N-dimensional array is input into the VGGNet network to obtain a 2D confidence map, that is, the estimated location of the body part (such as the elbow joint, knee, etc.). Among them, the role of the confidence map is to show the possibility of the appearance of human body parts in grayscale.

[0060] A2, find the frame in the video that returns to the initial action. A certain frame can be used as a reference initial action frame, and all frames in the video that return to the initial action can be found based on this.

[0061] Specifically, step A2 may include the following steps B1-B3:

[0062] B1. Select a frame f0 as the reference initial action frame.

[0063] B2. Calculate the distance between each frame and f0, that is, the sum of the key point distances, according to the following formula:

[0064]

[0065] Where i represents the i-th frame in the video, j=1,2,...,n is the j-th key point in each frame, and n is the number of key points in each frame. is the key point set of frame f0, P i is the key point set of the i-th frame.

[0066] B3. Given a small positive number ε. If D i <ε, the i-th frame is marked as the frame returning to the initial action. If it is not found in one traversal, ε is increased to 1.05ε as the new ε, and the traversal is repeated again, and so on, until ≥1 frames returning to the initial action are found.

[0067] A3, select a video of a certain length and generate transition frames.

[0068] In a specific implementation, based on experience, a video of more than 1 minute can be selected as the loop period, so that it is difficult to perceive the sense of looping of the video. A video of more than 1 minute is selected from the first frame of the video to a frame that returns to the initial action found in step A2) as the last frame f e . Need to be f e Frame -> Create a transition frame at frame 1. By calling the interpolation algorithm, you can specify to insert k transition frames, where k is an odd number. The value of k depends on the transition duration. According to the frames per second (FPS) of the current video, the playback duration of each frame can be determined, and the k value can be calculated according to the required transition duration. The length of the transition duration can be determined subjectively. Generally, k=3 is taken.

[0069] Among them, video interpolation aims to improve the frame rate and smoothness of the video, making the video look more "smooth". The general process of transition frame generation (interpolation algorithm) is as follows: Based on deep learning solutions, two adjacent frames of the original video are usually used as the input of the neural network, combined with optical flow neural network, occlusion estimation and other technologies to predict the intermediate frame between the two frames. Deep learning methods can extract image semantic information, and often perform better in occlusion estimation and other aspects. The dense optical flow between the two frames can be calculated. The optical flow information can be used to warp the two frames of the image to the middle moment, thereby synthesizing the intermediate frame. Specifically, for example: based on the optical flow interpolation algorithm (TOFlow), in the video interpolation task, the number of input frames is 3 frames, thus, a loop video is obtained.

[0070] In some possible examples, the playing condition includes that a first video frame in the first video is being played when the input information is acquired; and a plurality of video frames included in the first video are arranged and played in a first order;

[0071] The above step 101, determining the switching video frame from the multiple video frames included in the first video based on the playback status of the first video, may include the following steps:

[0072] A21. Determine a video frame in the first video that is sorted after the first video frame as a second video frame;

[0073] A22. Determine a second video frame corresponding to the reserved duration from the second video frames, and use the determined second video frame as a switching video frame.

[0074] Among them, the first order may include the order corresponding to the forward playback order.

[0075] In a specific implementation, the playback status may include the first video frame in the first video being played when the input information is obtained, thereby, the video frame in the first video that is sorted after the first video frame can be determined as the second video frame, and then the second video frame corresponding to the reserved duration can be determined from the second video frame, and the determined second video frame can be used as the switching video frame, thereby, based on the first playback time of the first video frame in the first video being played when the input information is obtained, the second playback time corresponding to the switching video frame can be determined based on the first playback time and the reserved duration, and the corresponding second video frame can be obtained as the switching video frame based on the second playback time, so that the corresponding switching video frame can be determined based on the playback status of the first video and the reserved duration.

[0076] The reserved duration may be related to the input information. Of course, the reserved duration may also be related to the predicted time, that is, the reserved duration is determined based on the predicted time. Since the corresponding second video is generated based on the input information, the corresponding reserved duration may be determined based on the generation time of the second video. The predicted time may be understood as the time when the second video is predicted to be generated.

[0077] In some possible embodiments, the above step 101, determining the predicted time for generating the second video corresponding to the input information, may include the following steps:

[0078] B11. Obtain environmental parameters;

[0079] B12, obtaining attribute information of the response text corresponding to the input information;

[0080] B13. Input the environmental parameters and the attribute information into a time prediction model to obtain the predicted time.

[0081] In the embodiment of the present application, the environmental parameters may include at least one of the following: service concurrency, digital human driving and rendering algorithm generation efficiency, hardware-related parameters during the testing of digital human driving and rendering algorithms, network parameters, etc., which are not limited here. The hardware-related parameters during the testing of digital human driving and rendering algorithms may include at least one of the following: central processing unit (CPU) related parameters, graphics processing unit (GPU) related parameters, neural network processing unit (NPU) related parameters, memory related parameters, computing power, etc., which are not limited here. Network parameters may include at least one of the following: network bandwidth, network transmission rate, etc., which are not limited here.

[0082] Among them, CPU related parameters may include at least one of the following: main frequency, number of cores, number of threads, floating point operations per second (FLOPS), etc., which are not limited here. GPU related parameters may include at least one of the following: main frequency, number of cores, number of threads, FLOPS, etc., which are not limited here. NPU related parameters may include at least one of the following: main frequency, number of cores, number of threads, FLOPS, etc., which are not limited here. Memory related parameters may include at least one of the following: memory size, memory processing rate, memory latency, etc., which are not limited here.

[0083] In an embodiment of the present application, the attribute information of the response text corresponding to the input information may include at least one of the following: the audio length corresponding to the response text, the language type of the response text, the expression of the response text, etc., which are not limited here. The language type of the response text may include a local or national language or a special code (Morse code), and the language type of the response text may include at least one of the following: Chinese, English, Japanese, local dialects, etc., which are not limited here. The expression of the response text may include at least one of the following: gestures, dance, lip reading, audio, etc., which are not limited here.

[0084] In a specific implementation, the environment parameters can be obtained, and then the attribute information of the response text corresponding to the input information can be obtained. Then, the environment parameters and the attribute information are input into the time prediction model to obtain the predicted time. Since the time prediction is performed on the environment parameters and the attribute information of the response text corresponding to the input information, on the one hand, the depth of the prediction result is consistent with the actual network environment and device environment (software environment, hardware environment, etc.). On the other hand, the depth of the prediction result is also consistent with the characteristics of the response text corresponding to the input information. Therefore, the predicted time is more consistent with the actual situation, and then, the reserved time is guaranteed to be more consistent with the actual situation, so as to reasonably determine the switching video frame to ensure the digital human playback effect.

[0085] In some possible embodiments, the number of the predicted time is multiple; and the following steps may also be included:

[0086] C1. Determine the mean and standard deviation of multiple prediction times;

[0087] C2. Determine a reserved duration according to the average value and the standard deviation.

[0088] In the embodiment of the present application, the number of predicted times can be multiple. Specifically, multiple predictions can be made through the time prediction model to obtain multiple predicted times. The multiple predicted times are used to perform mean calculations and standard deviation calculations to obtain the average value and standard deviation respectively. The reserved time can be obtained based on the average value and standard deviation. Since the model is constantly learning and the environment is constantly changing, the prediction error can be reduced by calculating the average value of multiple predicted times. The standard deviation reflects the prediction fluctuation to a certain extent. The standard deviation is used to adjust the reserved time to a certain extent, so that the reserved time is more in line with the actual situation to ensure the playback effect of the digital human.

[0089] In the specific implementation, Where σ is t gen The standard deviation of is the average value, that is, the average prediction time, and s is the adjustment factor, which is a positive number. For example, the adjustment factor is 3.

[0090] For example, for a new computing environment, the environment parameters can be processed and input into the time prediction model to predict the required prediction time t gen The final reservation duration is Where σ is t gen The standard deviation of cache Determine the switching video frame. Since the input environment parameters come from the current computing environment and are relatively fixed, the prediction can be performed offline and will not take up the time for generating the second video in real time.

[0091] In the specific implementation, the reserved duration is determined based on the predicted time. Since the predicted time depth is deeply related to the attributes of the environment and the response text itself, the reserved duration also deeply conforms to the attributes of the response text itself corresponding to the environment and the input information. Based on this, the switching video frame corresponding to the reserved duration can be obtained from the first video. In this way, the switching video frame can be reasonably determined to ensure the digital human playback effect.

[0092] In some possible embodiments, the following steps may also be included:

[0093] D1. Obtaining sample data, wherein the sample data includes sample environment parameters and / or sample attribute information;

[0094] D2. preprocessing the sample data to obtain target sample data;

[0095] D3. Training a preset time prediction model with the target sample data to meet preset conditions, thereby obtaining the time prediction model.

[0096] In the embodiment of the present application, the sample environment parameters may include at least one of the following: service concurrency, digital human driving and rendering algorithm generation efficiency, hardware-related parameters during the testing of digital human driving and rendering algorithms, network parameters, etc., which are not limited here. The hardware-related parameters during the testing of digital human driving and rendering algorithms may include at least one of the following: CPU-related parameters, GPU-related parameters, NPU-related parameters, memory-related parameters, computing power, etc., which are not limited here. Network parameters may include at least one of the following: network bandwidth, network transmission rate, network delay, etc., which are not limited here.

[0097] It can be understood that the hardware-related parameters include not only front-end hardware-related parameters, but also back-end hardware-related parameters. The front-end may include a terminal, and the back-end may include various servers.

[0098] Among them, CPU-related parameters may include at least one of the following: main frequency, number of cores, number of threads, FLOPS, etc., which are not limited here; GPU-related parameters may include at least one of the following: main frequency, number of cores, number of threads, FLOPS, etc., which are not limited here; NPU-related parameters may include at least one of the following: main frequency, number of cores, number of threads, FLOPS, etc., which are not limited here; memory-related parameters may include at least one of the following: memory size, memory processing rate, memory latency, etc., which are not limited here.

[0099] The sample attribute information may include at least one of the following: the length of the audio corresponding to the response text, the language type of the response text, the expression of the response text, etc., which are not limited here. The language type of the response text may include a local or national language or a special code (Morse code), and the language type of the response text may include at least one of the following: Chinese, English, Japanese, local dialects, etc., which are not limited here. The expression of the response text may include at least one of the following: gestures, dance, lip reading, audio, etc., which are not limited here.

[0100] The sample data may include sample environment parameters, or the sample data may include sample attribute information, or the sample data may include sample environment parameters and sample attribute information.

[0101] The preprocessing may include at least one of the following: normalization, one-hot vectorization, value filling, etc., which are not limited here. The purpose of the preprocessing is to constrain each input parameter within a certain range, for example, between 0 and 1.

[0102] Among them, the preset conditions can be pre-set or system default. For example, the preset condition can be reaching a set number of training times. For another example, the preset condition can be model convergence. For another example, the preset condition can be that the accuracy of the model reaches a set accuracy. Among them, the set number of training times and the set accuracy can both be pre-set or system default.

[0103] The preset time prediction model can be preset or set by system default. The input (target sample data) of the time prediction model is denoted as x, and the output (prediction time) is denoted as y. By setting different input x, different training data can be obtained, such as: setting different concurrent request amounts, setting different lengths of TTS audio, setting different numbers of CPU cores, numbers of threads, setting different memory limits, using different GPU cards, and setting different speed limits on the network, different prediction times t can be obtained. gen The prediction time t gen That is y.

[0104] Among them, for model training, the sample data is preprocessed, such as normalization, one-hot vectorization, value filling, etc., to process the input x and output y so that they are numerical, on the same scale and have no null values. Use machine learning models or deep learning models for training (fitting a function of x->y) to obtain a time prediction model (such as a regression model), such as Figure 4 As shown, the time prediction model can be regarded as a function of x->y. The time prediction model can include a machine learning model, which can use a machine learning algorithm such as a gradient boosting decision tree (GBDT) or a boosting tree with stronger learning ability, or a deep neural network (DNN). Further, as Figure 5 As shown, the sample data can be preprocessed to obtain x, and then x can be input into the time prediction model to obtain y.

[0105] In a specific implementation, sample data (sample environmental parameters and / or sample attribute information) can be obtained, and then the sample data can be preprocessed to obtain target sample data. Each data in the target sample data is constrained within a certain range. The target sample data is then trained on a preset time prediction model to meet preset conditions and obtain a time prediction model.

[0106] In a specific implementation, the full-link response time of the second video can be predicted by a time prediction model to obtain a predicted time.

[0107] In an embodiment of the present application, the time prediction model can realize function adaptation, and then, the full-link response time of the second video corresponding to the response text is generated through function adaptive prediction.

[0108] Among them, the input x and output y of the function can be defined, and the input of the time prediction model is x, and the output is y. Use machine learning or deep learning methods to establish a prediction model. First, it is necessary to find the factors that affect the generation time as input x. The main factors affecting the time are: audio length, service concurrency, digital human driver and rendering algorithm generation efficiency, CPU / GPU / memory related parameters during the test of digital human driver and rendering algorithm (such as main frequency, number of cores, number of threads, FLOPS, memory size, etc.), the current server's CPU / GPU / memory related parameters (such as main frequency, number of cores, number of threads, FLOPS, memory size, etc.), current network parameters (such as: bandwidth, whether it is a backbone network, etc.). y can be understood as how long it takes to start playing before the second video can be stably and continuously obtained without pauses.

[0109] 102. Generate the second video based on the switching video frame, and determine the actual time of generating the second video.

[0110] In the embodiment of the present application, the second video may include a group of video frames or two groups of video frames. The actual time of the second video may be understood as the time when the second video is actually generated.

[0111] In a specific implementation, it is possible to detect in real time whether the second video is generated, and then, based on this method, the actual time when the second video is generated can be obtained.

[0112] In a specific implementation, a time prediction model can be used to predict the full-link response time of the second video corresponding to the input information generated based on the switching video frame to obtain at least one predicted time.

[0113] In an embodiment of the present application, a second video corresponding to the input information can be generated based on switching video frames, and the actual time of generating the second video can be determined. Since the second video is generated based on switching video frames, an association between the first video and the second video is established through switching video frames, which helps to complete a natural transition between the first video and the second video based on switching video frames based on actual time and predicted time.

[0114] 103. Play the first video and the second video according to the actual time and the predicted time.

[0115] In the embodiment of the present application, the first video and the second video can be dynamically played according to the sequence between the actual time and the predicted time. Since the association between the first video and the second video is established by switching video frames, a natural transition between the first video and the second video can be achieved based on the actual time and the predicted time by switching video frames.

[0116] In a specific implementation, one of the at least one predicted time may be selected to be compared with the actual time, or an average value of the at least one predicted time may be determined and the average value may be compared with the actual time.

[0117] In some possible embodiments, the above step 103, playing the first video and the second video according to the actual time and the predicted time, can be implemented as follows:

[0118] A31. When the actual time is earlier than or equal to the predicted time, continue playing the first video;

[0119] A32. When the switching video frame in the first video is played, the second video is played.

[0120] In the specific implementation, since the actual time refers to the actual time of generating the second video, and the predicted time is the predicted time of generating the second video, when the actual time is earlier than the predicted time, it means that before the predicted time arrives, the generation of the second video has been completed, but the switching video frame has not been played. In addition, when the actual time is earlier than the predicted time, it means that before the predicted time arrives, the second video has been generated. In order to ensure a natural transition between the first video and the second video based on the switching video frame, it is necessary to continue playing the first video until the switching video frame, that is, after playing the switching video frame, play the second video. Since the second video is generated based on the switching video frame, a natural transition between the first video and the second video is achieved, thereby ensuring the playback effect of the digital human.

[0121] Of course, when the actual time is equal to the predicted time, it means that the switching video frame has just been played and the second video has just been generated. In this case, the second video can be played directly after the switching video frame is played.

[0122] In some possible embodiments, the plurality of video frames included in the first video are arranged in a first order; the above step 103, playing the first video and the second video according to the actual time and the predicted time, can be implemented as follows:

[0123] B31. When the actual time is later than the predicted time, continue playing the first video;

[0124] B32. After the switching video frame in the first video is played, if the second video has not been generated, continue to play the video frames in the first video that are located after the switching video frame in the first video in the first order until the second video is generated;

[0125] B33, playing from the third video frame to the switching video frame in the second order; the third video frame refers to the video frame being played in the first video when the second video is generated;

[0126] B34. After the switching video frame is played, the second video is played in the first order.

[0127] The first order and the second order are both the order of frames in the video. The first order and the second order can be preset or set by the system by default. The first order and the second order are opposite to each other.

[0128] The third video frame may be understood as the video frame being played when the second video is generated.

[0129] In the specific implementation, since the actual time refers to the actual time of generating the second video, and the predicted time is the predicted time of generating the second video, when the actual time is later than the predicted time, the first video will continue to be played first. After the switching video frame in the first video is played, it will appear that the predicted time has arrived, but the second video has not been generated. At this time, in order to ensure the digital human playback effect, the video frames after the switching video frames in the first video continue to be played in the first order until the second video has been generated. Since the second video is generated based on the switching video frames, in order to achieve a natural transition from the first video to the second video by switching the video frames, it is necessary to return to the switching video frames, and then play the second video after the switching video frames are played. Then, when the second video is generated, it will be played from the third video frame to the switching video frame in the second order; the third video frame refers to the video frame being played in the first video when the second video is generated. Then, the first video can be naturally transitioned to the second video by switching the video frames to avoid frame skipping and stuttering, so as to ensure the digital human playback effect.

[0130] To illustrate, taking the first video being played in forward playback order as an example, when the actual time is later than the predicted time, the predicted time will arrive, but the second video has not yet been generated, then you can continue to play the frame after the switching video frame, that is, play the video frame after the switching video frame in the first video until the second video is generated. When the second video is generated, start from the current playback frame when the second video is generated and play it back in reverse playback order until you return to the switching video frame, that is, after playing the switching video frame, play the second video back in forward playback order again. By adjusting the reserved time, you can reduce the phenomenon of generating the second video later than the predicted time, while trying to ensure that the time consumption of the entire link is not significantly increased.

[0131] Another example is given below. For the processing strategy when the actual time is earlier or later than the predicted time, when playing, first play the first frame of the first video in frame order, and then play the second frame according to the reserved time t after the user asks a question. cache Determine the switching video frame f t , that is, switch the video frame f after playing t After that, the second video starts playing.

[0132] Further, taking another example, if the second video is generated earlier than the predicted time, it means that the switching video frame f has not yet been played. t , but the second video has been generated, because the second video is based on the switching video frame f t If generated, it will continue to play to f t Then, play the second video F gen If the second video is generated later than the prediction time, it will appear that the switching video frame f has passed. t , but the second video has not been generated, you need to wait for the second video to be generated, then you can continue to play f t The following video frame {f t+1 ,f t+2 ,...,f t+k After the second video is generated, the video frame being played at the moment when the second video is generated is played back in reverse order, that is, {f t+k ,f t+k-1 ,...,f t}, where k is greater than f t The number of frames to play after. Return to switch video frame f t Then go back again and play the second video F in the forward playback order gen By adjusting the reserved time, the phenomenon of generating the second video later than the prediction can be reduced, while ensuring that the full-link response time does not increase significantly as much as possible.

[0133] In some possible embodiments, the second video includes a first group of video frames and a second group of video frames; the first group of video frames includes m video frames generated based on the switching video frames and corresponding to the first sequence and the input information; the second group of video frames includes n video frames generated based on the switching video frames and corresponding to the second sequence and the input information; m and n are both positive integers;

[0134] Then the above step 103, playing the first video and the second video according to the actual time and the predicted time, can be implemented as follows:

[0135] C31. When the actual time is earlier than or equal to the predicted time, continue to play the first video in the first order until the switching video frame in the first video is played, and then play the first group of video frames in the first order;

[0136] C32: when the actual time is later than the predicted time, playing the first video in the first order;

[0137] C33. If the second group of video frames has not been generated after the switching video frame in the first video is played, continue to play the video frames after the switching video frame in the first video in the first order until the second group of video frames is generated;

[0138] C34. Play from the fourth video frame to the switching video frame in the second order, and after the switching video frame is played, play the second group of video frames in the second order; the fourth video frame refers to the video frame being played in the first video when the second group of video frames is generated.

[0139] The first sequence and the second sequence can be preset or set by system default, and the first sequence and the second sequence are opposite to each other. The fourth video frame can be understood as the video frame being played when the second set of video frames is generated.

[0140] Among them, the second video may include a first group of video frames and a second group of video frames, the first group of video frames may include m video frames corresponding to the first order and input information generated based on switching video frames, and the second group of video frames may include n video frames corresponding to the second order and input information generated based on switching video frames; m and n are both positive integers, m and n are equal or unequal, and since there is a difference in video generation directions between the first group of video frames and the second group of video frames, there may also be a difference in the number of video frames generated corresponding to the input information.

[0141] In an embodiment of the present application, the first group of video frames and the second group of video frames can be generated in parallel, that is, one process is used to generate the first group of video frames corresponding to the input information based on the switching video frames, and another process is used to generate the second group of video frames corresponding to the input information based on the switching video frames, or, that is, one thread is used to generate the first group of video frames corresponding to the input information based on the switching video frames, and another thread is used to generate the second group of video frames corresponding to the input information based on the switching video frames.

[0142] In a specific implementation, when the second video includes a first group of video frames and a second group of video frames, the actual time can be understood as the actual time when both the first group of video frames and the second group of video frames are generated, and the predicted time can be understood as the predicted time when both the first group of video frames and the second group of video frames are generated.

[0143] Among them, due to the difference in the generation method of the first group of video frames and the second group of video frames, the actual time of the first group of video frames and the second group of video frames may also be different. The first group of video frames can correspond to an actual time, and the second group of video frames can also correspond to an actual time. Although the difference between the two actual times is not large, there may be a certain difference. The actual time corresponding to the second video can be the latest time of the actual times corresponding to the two groups of video frames.

[0144] In the specific implementation, since the actual time refers to the actual time of generating the second video, and the predicted time is the predicted time of generating the second video, the first video is played in the first order. When the actual time is earlier than or equal to the predicted time, it means that the generation of the second video has been completed before the predicted time arrives, but the switching video frame has not been played. In order to ensure a natural transition between the first video and the second video based on the switching video frame, it is necessary to continue playing the first video until the switching video frame is played. The first video is played in the first order until the switching video frame in the first video is played. Then, the first group of video frames are played in the first order. Since the first group of video frames is generated based on the switching video frame, a natural transition between the first video and the first group of video frames is achieved, thereby ensuring the playback effect of the digital human.

[0145] In the specific implementation, since the actual time refers to the actual time of generating the second video, and the predicted time is the predicted time of generating the second video, the first video is played in the first order. When the actual time is later than the predicted time, it means that after the switching video frame in the first video is played, the predicted time has arrived, but the second group of video frames has not been generated. At this time, in order to ensure the digital human playback effect, after the switching video frame is played, the first video is played in the first order. If the second group of video frames has not been generated after the switching video frame in the first video is played, the video frames after the switching video frame in the first video continue to be played in the first order until the second group of video frames are played. Two groups of video frames are generated, and are played from the fourth video frame to the switching video frame in the second order. After the switching video frame is played, the second group of video frames are played in the second order. The fourth video frame refers to the video frame being played in the first video when the second group of video frames is generated. Since the second group of video frames is generated based on the switching video frames, and the second group of video frames corresponds to the second order opposite to the first order, the number of times of looking back can be reduced (one less time of looking back), making the overall playback effect more natural. In addition, the first video can be naturally transitioned to the second group of video frames by switching video frames to avoid frame skipping and stuttering, so as to ensure the playback effect of the digital human.

[0146] For example, in a specific implementation, when the second video includes two sets of video frames, the switching video frame f can be generated simultaneously. t The next frame f t+1 and the previous frame f t-1 Two sets of video frames are generated backward and forward respectively. At this time, if the second video returns earlier than the predicted time, it will be played to the switching video frame f t After that, the second video is played in the forward playback order. If the second video is generated later than the predicted time, it is played to the switching video frame f t Then, continue playing in reverse order, which will reduce the need to look back once and make the playback effect more natural overall.

[0147] In the specific implementation, once the actual time is later than the predicted time, there will be two turns back, which will cause the video to appear temporarily unnatural, and the user may notice that the forward playback order and the reverse playback order are played repeatedly. Therefore, if the computing resources allow, the switching video frame f can be generated at the same time. t The next frame f t+1 and the previous frame f t-1 Two groups of video frames are generated backward and forward respectively, namely, the first group of video frames and the second group of video frames, specifically, the first group of video frames {f t+1 ,f t+2 ,...,f t+l} and the first set of video frames {f t-1 ,f t-2 ,...,ft-l}, where l is the number of frames of the video to be generated. That is, when the actual time is earlier than or equal to the predicted time, the first video is played continuously until the switching video frame, and then the first group of video frames is played. Conversely, when the actual time is later than the predicted time, the first video is played continuously until the switching video frame, and then the second group of video frames is played in reverse order. Specifically, when the actual time of the second video is earlier than the predicted time, the switching video frame f is played. t Then, play the first set of video frames in forward order {f t+1 ,f t+2 ,...,f t+l}; If the actual time of the second video is later than the predicted time, play to the switching video frame f t After that, continue to play the second set of video frames in reverse order. t-1 ,f t-2 ,...,f t-l}, this way you can look back less often and the overall playback effect will be more natural.

[0148] Furthermore, in the embodiment of the present application, after playing the second video, the first video is returned to. If the playback order when returning to the first video is the forward playback order, the first video is continued to be played in the forward playback order. Conversely, if the playback order when returning to the first video is the reverse playback order, the first video is continued to be played in the reverse playback order. Similarly, the user is waiting for questions in the reversely played first video. In this way, a real-time 2D digital human playback system can be obtained for the real-time 2D digital human playback application scenario.

[0149] The video playback method described in the present application, in the process of playing the first video, in response to obtaining input information, determines the predicted time of generating the second video corresponding to the input information, and determines the switching video frame from the multiple video frames included in the first video based on the playback status of the first video and the reserved time, the reserved time is determined based on the predicted time, the second video is generated based on the switching video frame, and the actual time of generating the second video is determined, and the first video and the second video are played according to the actual time and the predicted time. In this way, in the case where it is necessary to change the playback of the first video according to the input information, the predicted time required to generate the second video corresponding to the input information is first predicted, and then a reasonable reserved time corresponding to the predicted time is determined. The switching video frame in the first video can be accurately and reasonably determined in combination with the reserved time and the current playback status of the first video, and then the playback of the first video and the second video is dynamically controlled based on the difference between the actual time and the predicted time. Since the second video is generated based on the switching video frame, the video playback method of the present application can ensure that the first video is naturally transitioned to the second video by switching the video frame, that is, it can ensure that the transition between the first video and the second video is more natural during playback, and thus, the video playback effect can be guaranteed.

[0150] In accordance with the above embodiment, please refer to Figure 6 , Figure 6 : is a structural diagram of an electronic device provided in an embodiment of the present application. As shown in the figure, the electronic device includes a processor, a memory, a communication interface, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the processor. In the embodiment of the present application, the program includes instructions for executing the following steps:

[0151] In the process of playing the first video, in response to obtaining input information, determining a predicted time for generating a second video corresponding to the input information, and determining a switching video frame from a plurality of video frames included in the first video based on the playing status of the first video and a reserved duration; the reserved duration is determined based on the predicted time;

[0152] generating a second video based on the switching video frame, and determining an actual time of generating the second video;

[0153] The first video and the second video are played according to the actual time and the predicted time.

[0154] In some possible examples, the playing condition includes that a first video frame in the first video is being played when the input information is acquired; and a plurality of video frames included in the first video are arranged and played in a first order;

[0155] In terms of determining the switching video frame from a plurality of video frames included in the first video based on the playback status of the first video, the above program includes instructions for performing the following steps:

[0156] Determine a video frame in the first video that is sorted after the first video frame as a second video frame;

[0157] A second video frame corresponding to the reserved duration is determined from the second video frames, and the determined second video frame is used as a switching video frame.

[0158] In some possible examples, in terms of playing the first video and the second video according to the actual time and the predicted time, the program includes instructions for performing the following steps:

[0159] When the actual time is earlier than or equal to the predicted time, continue playing the first video;

[0160] When the switching video frame in the first video is played, the second video is played.

[0161] In some possible examples, the plurality of video frames included in the first video are arranged in a first order; and in terms of playing the first video and the second video according to the actual time and the predicted time, the program includes instructions for performing the following steps:

[0162] When the actual time is later than the predicted time, continue playing the first video;

[0163] After the switching video frame in the first video is played, if the second video has not been generated, continue to play the video frames in the first video that are located after the switching video frame in the first video according to the first sequence until the second video is generated;

[0164] Playing from a third video frame to the switching video frame in a second order; the third video frame refers to the video frame being played in the first video when the second video is generated;

[0165] After the switching video frame is played, the second video is played according to the first sequence.

[0166] In some possible examples, the second video includes a first group of video frames and a second group of video frames; the first group of video frames includes m video frames generated based on the switching video frames and corresponding to the first sequence and the input information; the second group of video frames includes n video frames generated based on the switching video frames and corresponding to the second sequence and the input information; m and n are both positive integers;

[0167] In terms of playing the first video and the second video according to the actual time and the predicted time, the program includes instructions for performing the following steps:

[0168] When the actual time is earlier than or equal to the predicted time, the first video is continuously played in the first order until the switching video frame in the first video is played completely, and then the first group of video frames is played in the first order;

[0169] When the actual time is later than the predicted time, playing the first video in the first order;

[0170] If the second group of video frames has not been generated after the switching video frame in the first video is played, continue to play the video frames after the switching video frame in the first video in the first order until the second group of video frames is generated;

[0171] Play from the fourth video frame to the switching video frame in a second order, and after the switching video frame is played, play the second group of video frames in the second order; the fourth video frame refers to the video frame being played in the first video when the second group of video frames is generated.

[0172] In some possible examples, the number of the predicted time is multiple; the above program also includes instructions for executing the following steps:

[0173] Determine the mean and standard deviation for multiple forecast times;

[0174] The reserved duration is determined according to the average value and the standard deviation.

[0175] In some possible examples, in terms of determining the predicted time for generating the second video corresponding to the input information, the program includes instructions for performing the following steps:

[0176] Get environmental parameters;

[0177] Obtaining attribute information of the response text corresponding to the input information;

[0178] The environmental parameters and the attribute information are input into a time prediction model to obtain the predicted time.

[0179] The electronic device described in the present application, in the process of playing the first video, in response to obtaining input information, determines the predicted time of generating the second video corresponding to the input information, and determines the switching video frame from the multiple video frames included in the first video based on the playback status of the first video and the reserved time, the reserved time is determined based on the predicted time, the second video is generated based on the switching video frame, and the actual time of generating the second video is determined, and the first video and the second video are played according to the actual time and the predicted time. In this way, in the case where it is necessary to change the playback of the first video according to the input information, the predicted time required to generate the second video corresponding to the input information is first predicted, and then a reasonable reserved time corresponding to the predicted time is determined. In combination with the reserved time and the current playback status of the first video, the switching video frame in the first video can be accurately and reasonably determined, and then based on the difference between the actual time and the predicted time, the playback of the first video and the second video is dynamically controlled. Since the second video is generated based on the switching video frame, the video playback method of the present application can ensure that the first video is naturally transitioned to the second video by switching the video frame, that is, it can ensure that the transition between the first video and the second video is more natural during playback, and thus, the video playback effect can be guaranteed.

[0180] Figure 7700 is a functional unit block diagram of a video playback device 700 involved in an embodiment of the present application. The video playback device 700 includes: a determination unit 701, a generation unit 702 and a playback control unit 703, wherein:

[0181] The determining unit 701 is used to determine, in response to obtaining input information during the process of playing the first video, a predicted time for generating a second video corresponding to the input information, and determine a switching video frame from a plurality of video frames included in the first video based on the playing status of the first video and a reserved duration; the reserved duration is determined based on the predicted time;

[0182] The generating unit 702 is configured to generate a second video based on the switching video frame, and determine an actual time for generating the second video;

[0183] The playback control unit 703 is configured to play the first video and the second video according to the actual time and the predicted time.

[0184] In some possible examples, the playing condition includes that a first video frame in the first video is being played when the input information is acquired; and a plurality of video frames included in the first video are arranged and played in a first order;

[0185] In terms of determining the switching video frame from the multiple video frames included in the first video based on the playback status of the first video, the determining unit 701 is specifically used to:

[0186] Determine a video frame in the first video that is sorted after the first video frame as a second video frame;

[0187] A second video frame corresponding to the reserved duration is determined from the second video frames, and the determined second video frame is used as a switching video frame.

[0188] In some possible examples, in the aspect of playing the first video and the second video according to the actual time and the predicted time, the playing control unit 703 is specifically used to:

[0189] When the actual time is earlier than or equal to the predicted time, continue playing the first video;

[0190] When the switching video frame in the first video is played, the second video is played.

[0191] In some possible examples, the plurality of video frames included in the first video are arranged in a first order; in terms of playing the first video and the second video according to the actual time and the predicted time, the playing control unit 703 is specifically configured to:

[0192] When the actual time is later than the predicted time, continue playing the first video;

[0193] After the switching video frame in the first video is played, if the second video has not been generated, continue to play the video frames in the first video that are located after the switching video frame in the first video according to the first sequence until the second video is generated;

[0194] Playing from a third video frame to the switching video frame in a second order; the third video frame refers to the video frame being played in the first video when the second video is generated;

[0195] After the switching video frame is played, the second video is played according to the first sequence.

[0196] In some possible examples, the second video includes a first group of video frames and a second group of video frames; the first group of video frames includes m video frames generated based on the switching video frames and corresponding to the first sequence and the input information; the second group of video frames includes n video frames generated based on the switching video frames and corresponding to the second sequence and the input information; m and n are both positive integers;

[0197] In the aspect of playing the first video and the second video according to the actual time and the predicted time, the playing control unit 703 is specifically used for:

[0198] When the actual time is earlier than or equal to the predicted time, the first video is continuously played in the first order until the switching video frame in the first video is played completely, and then the first group of video frames is played in the first order;

[0199] When the actual time is later than the predicted time, playing the first video in the first order;

[0200] If the second group of video frames has not been generated after the switching video frame in the first video is played, continue to play the video frames after the switching video frame in the first video in the first order until the second group of video frames is generated;

[0201] Play from the fourth video frame to the switching video frame in a second order, and after the switching video frame is played, play the second group of video frames in the second order; the fourth video frame refers to the video frame being played in the first video when the second group of video frames is generated.

[0202] In some possible examples, the number of the predicted time is multiple; and the video playback device is further specifically used for:

[0203] Determine the mean and standard deviation for multiple forecast times;

[0204] The reserved duration is determined according to the average value and the standard deviation.

[0205] In some possible examples, in determining the predicted time for generating the second video corresponding to the input information, the determining unit 701 is specifically configured to:

[0206] Get environmental parameters;

[0207] Obtaining attribute information of the response text corresponding to the input information;

[0208] The environmental parameters and the attribute information are input into a time prediction model to obtain the predicted time.

[0209] The video playback device described in the present application, in the process of playing the first video, in response to obtaining input information, determines the predicted time of generating the second video corresponding to the input information, and determines the switching video frame from the multiple video frames included in the first video based on the playback status of the first video and the reserved time, the reserved time is determined based on the predicted time, the second video is generated based on the switching video frame, and the actual time of generating the second video is determined, and the first video and the second video are played according to the actual time and the predicted time. In this way, when it is necessary to change the playback of the first video according to the input information, the predicted time required to generate the second video corresponding to the input information is first predicted, and then a reasonable reserved time corresponding to the predicted time is determined. The switching video frame in the first video can be accurately and reasonably determined in combination with the reserved time and the current playback status of the first video, and then the playback of the first video and the second video can be dynamically controlled based on the difference between the actual time and the predicted time. Since the second video is generated based on the switching video frame, the video playback method of the present application can ensure that the first video is naturally transitioned to the second video by switching the video frame, that is, it can ensure that the transition between the first video and the second video is more natural during playback, and thus, the video playback effect can be guaranteed.

[0210] It can be understood that the functions of each program module of the video playback device of this embodiment can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here.

[0211] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps of any method described in the above method embodiments, and the above computer includes an electronic device.

[0212] The embodiment of the present application also provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, the computer program is operable to cause a computer to execute some or all of the steps of any method described in the method embodiment. The computer program product may be a software installation package, and the computer includes an electronic device.

[0213] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0214] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0215] In the several embodiments provided in the present application, it should be understood that the disclosed system can be implemented in other ways. For example, the system embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the system or unit can be electrical or other forms.

[0216] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0217] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0218] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0219] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, which can include: a flash drive, a read-only memory, a random access memory, a magnetic disk or an optical disk, etc.

[0220] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video playback method, characterized in that: include: In the process of playing the first video, in response to obtaining the input information, determining the predicted time of generating the second video corresponding to the input information, and determining the switching video frame from the multiple video frames included in the first video based on the playing status and the reserved duration of the first video; generating a second video based on the switching video frame, and determining an actual time of generating the second video; The first video and the second video are played according to the actual time and the predicted time.

2. The method according to claim 1, characterized in that: The playing condition includes that the first video frame in the first video is being played when the input information is acquired; the multiple video frames included in the first video are arranged and played in a first order; The step of determining a switching video frame from a plurality of video frames included in the first video based on the playback status of the first video includes: Determine a video frame in the first video that is sorted after the first video frame as a second video frame; A second video frame corresponding to the reserved duration is determined from the second video frames, and the determined second video frame is used as a switching video frame.

3. The method according to claim 1 or 2, characterized in that: Playing the first video and the second video according to the actual time and the predicted time includes: When the actual time is earlier than or equal to the predicted time, continue playing the first video; When the switching video frame in the first video is played, the second video is played.

4. The method according to claim 1 or 2, characterized in that: The plurality of video frames included in the first video are arranged in a first order; and playing the first video and the second video according to the actual time and the predicted time comprises: When the actual time is later than the predicted time, continue playing the first video; After the switching video frame in the first video is played, if the second video has not been generated, continue to play the video frames in the first video that are located after the switching video frame in the first video according to the first sequence until the second video is generated; Playing from a third video frame to the switching video frame in a second order; the third video frame refers to the video frame being played in the first video when the second video is generated; After the switching video frame is played, the second video is played according to the first sequence.

5. The method according to claim 1 or 2, characterized in that: The second video includes a first group of video frames and a second group of video frames; the first group of video frames includes m video frames generated based on the switching video frames and corresponding to the first sequence and the input information; the second group of video frames includes n video frames generated based on the switching video frames and corresponding to the second sequence and the input information; m and n are both positive integers; Playing the first video and the second video according to the actual time and the predicted time includes: When the actual time is earlier than or equal to the predicted time, the first video is continuously played in the first order until the switching video frame in the first video is played completely, and then the first group of video frames is played in the first order; When the actual time is later than the predicted time, playing the first video in the first order; If the second group of video frames has not been generated after the switching video frame in the first video is played, continue to play the video frames after the switching video frame in the first video in the first order until the second group of video frames is generated; Play from the fourth video frame to the switching video frame in a second order, and after the switching video frame is played, play the second group of video frames in the second order; the fourth video frame refers to the video frame being played in the first video when the second group of video frames is generated.

6. The method according to any one of claims 1 to 5, characterized in that: The determining and generating a predicted time of the second video corresponding to the input information includes: Get environmental parameters; Obtaining attribute information of the response text corresponding to the input information; The environmental parameters and the attribute information are input into a time prediction model to obtain the predicted time.

7. A video playback device, characterized in that: The device comprises: a determination unit, a generation unit and a playback control unit, wherein: The determining unit is used to determine, in response to obtaining input information during the process of playing the first video, a predicted time for generating a second video corresponding to the input information, and determine a switching video frame from a plurality of video frames included in the first video based on the playing status of the first video and a reserved duration; the reserved duration is determined based on the predicted time; The generating unit is configured to generate a second video based on the switching video frame, and determine an actual time for generating the second video; The playback control unit is used to play the first video and the second video according to the actual time and the predicted time.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store one or more programs and is configured to be executed by the processor, wherein the program comprises instructions for executing the steps in the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: A computer program for electronic data exchange is stored, wherein the computer program enables a computer to execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the method as claimed in any one of claims 1 to 6.

Images