Front-end processing method, device and medium for digital human video stream

Abstract

The present application discloses a front-end processing method, device, and medium for digital human video streams, relating to the field of video stream processing technology. The method comprises: based on a pre-built long link channel, receiving segmented digital human video stream data generated by server segmentation, and caching the segmented digital human video stream data into a front-end queue in the order of transmission; monitoring the playback progress of the current video container, and when the playback progress reaches a preset playback progress threshold, extracting the next segmented digital human video stream data from the front-end queue and loading it into another video container; when monitoring the end of playback of the current video container, switching to another video container and triggering the other video container to play the next pre-loaded segmented digital human video stream data, so as to achieve seamless connection of the segmented digital human video stream data.

Description

Technical Field

[0001] The present application relates to the technical field of video stream processing, and in particular to a front-end processing method, device and medium for digital human video stream. Background Art

[0002] With the widespread application of digital human technology in live broadcasting, intelligent customer service, virtual conferencing and other fields, the real-time generation and efficient playback of digital human video streams have become a core link in improving user experience. In existing technologies, digital human video streams usually rely on back-end servers for centralized processing and transmission, resulting in significant performance bottlenecks in the reception, assembly and playback processes of front-end devices. The back-end centralized processing mode requires the complete video stream data to be transmitted to the front-end via the network. The high latency problem is particularly prominent in real-time interactive scenarios, making it difficult to meet users' needs for instant feedback. At the same time, the transmission of complex video stream data requires extremely high network bandwidth stability. When the network fluctuates or the device performance is insufficient, it is easy to cause problems such as freezes and data loss, which seriously affects visual coherence.

[0003] Current front-end processing solutions mostly use a single video container sequential loading and playback mechanism, lacking support for preloading and dynamic switching of segmented data. During playback, the front-end must wait for the current video segment to be fully loaded and played before loading the next segment, resulting in noticeable playback gaps and abrupt transitions. Furthermore, existing technologies fail to fully consider the impact of device performance differences on rendering efficiency. Low-end devices are prone to resource overload when parsing highly complex video streams, while high-end devices are unable to fully utilize their hardware advantages to achieve better performance. This processing approach is difficult to adapt to diverse terminal environments, limiting the large-scale application of digital human technology.

[0004] Furthermore, existing methods have significant deficiencies in fault tolerance and dynamic adaptation. When a video stream fails to load or playback is interrupted, traditional solutions require reloading the entire data queue, resulting in a broken user experience. At the same time, fixed-layout video containers cannot adapt to diverse display scenarios such as vertical and split screens on mobile devices, resulting in wasted display area or screen obstruction. Although edge computing technology has been partially introduced to alleviate back-end pressure, its coordination mechanism with the front-end is not yet mature, and the caching strategy and dynamic resource allocation lack intelligent design, making it difficult to effectively reduce transmission latency and improve system robustness. Summary of the Invention

[0005] The embodiments of the present application provide a front-end processing method, device and medium for a digital human video stream to solve the above-mentioned technical problems.

[0006] On the one hand, an embodiment of the present application provides a front-end processing method for a digital human video stream, comprising:

[0007] Based on the pre-built long link channel, it receives the segmented digital human video stream data generated by the server, and caches the segmented digital human video stream data into the front-end queue in the transmission order;

[0008] By means of two stacked video containers, the segmented digital human video stream data in the front-end queue is alternately loaded and played;

[0009] Monitor the playback progress of the currently playing video container, and when the playback progress reaches a preset playback progress threshold, extract the next segmented digital human video stream data from the front-end queue and load it into another video container;

[0010] When the end of the playback of the currently playing video container is detected, switch to the other video container and trigger the other video container to play the preloaded next segmented digital human video stream data to achieve seamless connection of the segmented digital human video stream data.

[0011] In one implementation of the present application, before receiving the segmented digital human video stream data generated by segmentation on the server side based on the pre-built long link channel, the method further includes:

[0012] The server receives the intelligent question-answering request initiated by the user based on the front-end device, generates the corresponding answer text, and performs semantic analysis on the answer text to identify the key semantic nodes associated with the context;

[0013] Monitor the network bandwidth of the front-end device in real time, and dynamically adjust the maximum duration threshold of the segmented digital human video stream data based on the monitoring results. If the network bandwidth is lower than the preset threshold, shorten the length of the segmented digital human video stream data according to a preset ratio to reduce the amount of data transmitted in a single transmission;

[0014] According to the key semantic nodes and in combination with the adjusted maximum duration threshold, the segmentation points of the digital human video stream are dynamically adjusted so that each segmented digital human video stream data contains a complete semantic unit;

[0015] The answer text is divided into multiple sentences according to the segmentation points, a video stream segment corresponding to each sentence is generated, and the action parameters and expression parameters of the digital human are associated with each video stream segment to obtain segmented digital human video stream data.

[0016] In one implementation of the present application, sentiment analysis is performed on each sentence at the server end to identify the emotion type corresponding to the sentence, and an associated emotion feature tag is generated for the segmented digital human video stream data corresponding to the sentence;

[0017] Parsing the emotional feature tags of the segmented digital human video stream data on the front-end device, and adjusting the video playback speed and video playback volume corresponding to the segmented digital human video stream data according to the tag type in the parsing result and based on predefined playback parameter mapping rules;

[0018] The adjusted video playback speed and volume are applied in real time during video playback to ensure that the emotional expression of the playing video is synchronized with the video content.

[0019] In one implementation of the present application, two stacked video containers are used to alternately load and play the segmented digital human video stream data in the front-end queue, specifically including:

[0020] The display levels of the two video playback containers are controlled by the z-index attribute, so that when the currently playing video container loads and plays the current segmented digital human video stream data in the front-end queue, the display level of the currently playing video container is set to a visible state, and the display level of the other video container is set to a hidden state;

[0021] After the other video container completes preloading of the next segmented digital human video stream data in the front-end queue, setting the display level of the currently playing video container to a hidden state;

[0022] When the other video container is activated, the display level of the other video container is synchronously updated to a visible state, and a play instruction of the other video container is triggered.

[0023] In one implementation of the present application, after caching the segmented digital human video stream data into the front-end queue in the transmission order, the method further includes:

[0024] On the edge computing node, the action parameters and expression parameters of the digital human in the segmented digital human video stream data are dynamically compressed according to the hardware performance parameters of the front-end device;

[0025] Binding the optimized action parameters and expression parameters to the segmented digital human video stream data to generate optimized video stream segments adapted to the hardware performance parameters of the front-end device;

[0026] Through the edge computing node, the local cache stores the preset frequently used video stream segments and generates a temporary access link so that the front-end device preferentially pulls the video stream segments from the local cache of the edge computing node. If a hit is not found, a data acquisition request is initiated to the server.

[0027] Through the collaborative caching mechanism between the edge computing node and the front-end device, and according to the fluctuation of network bandwidth, the distribution ratio of the segmented digital human video stream data between the edge computing node and the front-end device is dynamically adjusted to reduce network transmission delay.

[0028] In one implementation of the present application, the present invention further includes:

[0029] When detecting that the loading of the currently playing video container fails or the playback is interrupted, switch to another video container and try to reload the same segmented digital human video stream data;

[0030] If the other video container also fails to load, a regeneration request is sent to the server to obtain an alternative version of the segmented digital human video stream data;

[0031] After the alternative version is loaded successfully, the alternative version is inserted into the original position of the faulty segmented digital human video stream data in the front-end queue and continues to be played, and the faulty segmented digital human video stream data is marked as low priority to delay the retry loading of the faulty segmented digital human video stream data.

[0032] In one implementation of the present application, the present invention further includes:

[0033] When the front-end device plays the segmented digital human video stream data, real-time information about the user's gaze area on the digital human image in the currently playing video container is obtained; the gaze area information is collected by the eye tracking device pre-deployed in the front-end device;

[0034] Based on the gaze area information, determine whether the user continues to gaze at a specific part of the digital human image for more than a preset gaze duration threshold; if so, pause the video playback of the currently playing video container, and send the user gaze behavior information and the playback progress information of the currently playing segmented digital human video stream data to the server;

[0035] The server analyzes the currently played segmented digital human video stream data, extracts detailed information related to the specific part, and generates a corresponding extended interpretation video stream segment based on the detailed information, so as to send the extended interpretation video stream segment and corresponding playback insertion point information to the front-end device;

[0036] Through the front-end device, in the currently playing video container, the extended interpretation video stream segment is inserted and played starting from the playback progress position corresponding to the playback insertion point information, and after the extended interpretation video stream segment is played, the remaining unplayed part of the segmented digital human video stream data continues to be played.

[0037] In one implementation of the present application, the present invention further includes:

[0038] Monitor the front-end device's screen orientation changes or user-triggered split-screen operations. If the front-end device switches to portrait mode, adjust the two video containers in the stacked layout to a parallel layout and dynamically adjust the size ratio of the two video containers.

[0039] In the split-screen mode, the other spare video container of the two video containers is hidden, the preloading operation of the other video container is paused, and the currently playing video container is kept for playing to adapt to the display area.

[0040] On the other hand, an embodiment of the present application further provides a front-end processing device for a digital human video stream, the device comprising:

[0041] at least one processor;

[0042] and, a memory communicatively coupled to the at least one processor;

[0043] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the front-end processing method of a digital human video stream as described above.

[0044] On the other hand, an embodiment of the present application further provides a non-volatile computer storage medium storing computer executable instructions. When the computer executable instructions are executed, the front-end processing method of a digital human video stream as described above is implemented.

[0045] The present invention provides a front-end processing method, device, and medium for digital human video streams, which have at least the following beneficial effects:

[0046] The server-generated segmented video stream data is received over a pre-established long link, avoiding the frequent handshake overhead associated with traditional polling or short connections and significantly reducing data transmission time from the server to the frontend. Segmented data is sequentially cached in a frontend queue. Combined with dynamic progress monitoring and preloading, this allows the video stream to reach the frontend and complete rendering quickly in smaller data units, effectively reducing user-perceived playback latency and making it particularly suitable for real-time interactive scenarios. Two video containers in a stacked layout alternately load and play segmented data. By preloading the next video segment and instantly switching the container level upon completion, this completely eliminates the loading gaps and screen freezes associated with traditional single-container playback. This dual-container collaborative mechanism not only conserves frontend cache resources but also optimizes hardware resource utilization by parallelizing loading and playback, adapting to the smooth operation requirements of terminals with varying performance. By monitoring playback completion events and automatically triggering container switching, multiple video streams appear seamlessly to the user, enhancing interactive immersion. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0048] Figure 1 A schematic diagram of an application scenario of a front-end processing method for a digital human video stream provided in an embodiment of the present application;

[0049] Figure 2 A schematic flow chart of a front-end processing method for a digital human video stream provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of a process for collaborative caching between edge computing nodes and front-end devices provided in an embodiment of the present application;

[0051] Figure 4 A schematic structural diagram of a front-end processing device for a digital human video stream provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of the internal structure of a front-end processing device for a digital human video stream provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0054] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0055] The front-end processing method of a digital human video stream provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Figure 1 As shown, the application environment may include: a front-end device 101 carried by the user, a communication network 102, a server 103, and a database 104. The communication network 102 can serve as a data transmission channel, providing a communication link for communication between the front-end device 101 and the server 103. After the server 103 segments the digital human video stream and generates segmented digital human video stream data, it transmits the segmented digital human video stream data to the front-end device 101 carried by the user based on the communication network 102 and stores it in the front-end queue of the front-end device 101.

[0056] The database 104 is connected to the server 103 and can be used to store the original digital human video stream and the segmented digital human video stream data generated after segmentation. The database 106 can be integrated on the server, or placed on the cloud or other network servers.

[0057] Figure 2 A schematic flow chart of a front-end processing method for a digital human video stream provided in an embodiment of the present application.

[0058] The analysis method involved in the embodiments of the present application can be implemented by a terminal device or a server, and the present application does not impose any special restrictions on this. For ease of understanding and description, the following embodiments are described in detail using a server as an example.

[0059] It should be noted that the server can be a single device or a system composed of multiple devices, that is, a distributed server, and this application does not make any specific restrictions on this.

[0060] like Figure 2 As shown, the embodiment of the present application provides a front-end processing method for a digital human video stream, comprising:

[0061] Step 201: Based on the pre-built long link channel, receive the segmented digital human video stream data generated by the server, and cache the segmented digital human video stream data into the front-end queue according to the transmission order.

[0062] It should be noted that the digital human video stream (Digital Human Video Stream) in the embodiment of the present application refers to the video data stream generated by a virtual character (digital human) generated by a computer in real time or offline state.

[0063] In this embodiment, a long link channel is implemented via the WebSocket protocol, establishing a persistent bidirectional communication connection with the server, avoiding the overhead associated with frequent handshakes associated with traditional short HTTP connections. As can be appreciated, the persistence of the long link channel enables the server to push segmented digital human video stream data to the front-end device in real time, significantly reducing transmission latency. It should be noted that when generating the segmented digital human video stream data, the server performs semantic analysis on the user's input answer text, identifying key contextual semantic nodes, such as the end of a question sentence and emotional turning points, and segments the video stream based on a dynamically adjusted maximum duration threshold. This maximum duration threshold is determined based on the real-time monitored front-end network bandwidth. For example, if the network bandwidth is low, the server will shorten the duration of a single video stream to ensure that the data volume of each segment is suitable for the current transmission conditions. If the network bandwidth is good, the server will appropriately increase the duration of each video stream, thereby increasing the content transmitted by the single-end video stream.

[0064] Specifically, the segmented digital human video stream data includes video frames, audio streams, and digital human motion and expression parameters, such as gesture trajectories and smile amplitude. After receiving these segmented digital human video streams via a long link, the front-end device uses a first-in, first-out (FIFO) strategy to store the data in the front-end queue in the order of transmission. For example, the front-end queue is managed using a ring buffer structure. When the queue capacity reaches a preset upper limit, the oldest received segment is automatically removed to free up memory resources.

[0065] In this embodiment, when the server receives a request initiated by the user triggering the front end, such as intelligent chat Q&A, a question is asked, and the answer is returned based on the question. The answer is divided into multiple sentences according to the sentence segmentation logic. Then, the algorithm generates specific movements and facial shapes of the digital human based on each sentence and pays attention to the connection and continuation of the movement of the previous sentence. It is pushed to the front-end client through the video stream of Life Good in the form of a long link (WebSocket).

[0066] The front-end stores the segmented digital human video stream data received from the server through the long link channel in the form of a queue. When it detects that there is only one data piece in the front-end queue, it can fill the data into the encapsulated video container component and execute the loading and playback.

[0067] Figure 3 Schematic diagram of the process of the collaborative caching method between the edge computing node and the front-end device provided in the embodiment of the present application. Figure 3 ,301, on the edge computing node, dynamically compress the action parameters and expression parameters of the digital human in the segmented digital human video stream data according to the hardware performance parameters of the front-end device.

[0068] In this embodiment, a collaborative optimization mechanism between edge computing nodes and front-end devices achieves a deep integration of data compression, cache management, and dynamic distribution. It should be understood that an edge computing node refers to a computing unit deployed at the edge of the network, such as a CDN node or local server close to the front-end device, which possesses real-time processing and caching capabilities. After the segmented digital human video stream data is cached in the front-end queue in the order of transmission, the edge computing node dynamically adjusts the data compression strategy based on the hardware performance parameters of the front-end device, such as CPU computing power, GPU rendering capabilities, and memory capacity. For example, if the front-end device is a low-performance mobile terminal, the edge computing node will downsample the digital human's motion parameters and expression parameters to reduce the data volume to adapt to the device's processing power. Motion parameters include limb movement trajectory and gesture frequency, and expression parameters include facial muscle movement amplitude. If it is a high-performance PC, the original parameters are retained to ensure rendering accuracy. It should be noted that hardware performance parameters are actively reported by the front-end device during initialization, or dynamically obtained by the edge computing node through probe requests, such as WebRTC performance testing.

[0069] 302. Bind the compressed action parameters and expression parameters with the segmented digital human video stream data to generate optimized video stream segments adapted to the hardware performance parameters of the front-end device.

[0070] In this embodiment, the edge computing node rebinds the compressed action parameters and expression parameters to the original video stream segment to generate an optimized video stream segment. For example, for a video stream containing a digital human waving his hands, the edge computing node can only retain the hand trajectory data of the key frames and remove redundant intermediate frames, thereby reducing the computing resources required for rendering. It should be noted that the optimized video stream segmentation format needs to be compatible with the decoder of the front-end device, such as H.264 encoding, to avoid additional conversion overhead. This process is coordinated with the dynamic segmentation logic of the server. The server divides the video stream according to semantic units, and the edge computing node is further optimized according to device performance to form a two-level adaptation mechanism.

[0071] 303. Through the edge computing node, the local cache stores the preset frequently used video stream segments and generates a temporary access link so that the front-end device preferentially pulls the video stream segments from the local cache of the edge computing node. If there is no hit, a data acquisition request is initiated to the server.

[0072] In this embodiment, the edge computing node identifies frequently used video stream segments, such as digital human action clips corresponding to frequently asked questions, by analyzing historical access data, and stores them in a local cache. For example, for answers to product function introductions that frequently appear in intelligent customer service scenarios, the edge computing node pre-stores the corresponding video stream segments. It should be noted that the cache strategy is managed using the Least Recently Used (LRU) algorithm, and low-frequency data is automatically eliminated when cache space is insufficient. For segments that hit the cache, the edge computing node generates a temporary access link, such as a CDN URL with a time-limited signature, and the front-end device directly pulls the data through the link to avoid repeated requests to the server. If there is no hit, the front-end device falls back to the server to obtain the original data, and the edge computing node records the access frequency of the segment to update the cache strategy.

[0073] 304. Through the collaborative caching mechanism of edge computing nodes and front-end devices, and according to the fluctuation of network bandwidth, the distribution ratio of segmented digital human video stream data between edge computing nodes and front-end devices is dynamically adjusted to reduce network transmission delay.

[0074] In this embodiment, the collaborative caching mechanism refers to the edge computing node and the front-end device dynamically adjusting the data source ratio according to the real-time network conditions. For example, when the network bandwidth is low, the edge computing node increases the push ratio of the local cache segment and reduces the amount of data pulled from the server; when the network bandwidth is sufficient, the front-end device is allowed to pull a mixture of edge cache and server data to ensure the freshness of the content. Specifically, the edge computing node dynamically calculates the distribution ratio by monitoring network bandwidth fluctuations, such as based on the TCP congestion window size or the throughput reported by the front-end. For example, if it is detected that the network delay exceeds the threshold, the edge computing node will give priority to distributing the cached lightweight segments and suspend the transmission of non-critical data, such as high-precision expression parameters.

[0075] Step 202: monitor the playback progress of the currently playing video container. When the playback progress reaches a preset playback progress threshold, extract the next segmented digital human video stream data from the front-end queue and load it into another video container.

[0076] The front-end device uses two video containers to create a stacked layout, using the data in the front-end queue as the data source and assigning values ​​to the two video containers separately. For example, the upper layer in the layout is the first segmented digital human video stream data in the queue, and the lower layer in the layout is the second segmented digital human video stream data.

[0077] In this embodiment, after monitoring the playback progress of the currently playing video container, the display level of the two video playback containers is controlled via the z-index property. For example, in the initial state, container A has a z-index of 2, indicating a visible state, while container B has a z-index of 1, indicating a hidden state. It is understandable that the design of the stacked layout allows the two containers to visually overlap in the same display area, with the user only perceiving the playback screen of the top-level video container. It should be noted that while container A is playing the current segmented digital human video stream data, container B preloads the next segmented digital human video stream data. After container A finishes playing, the display level of container A is set to hidden, and the display level of container B is raised to visible, thereby triggering container B to play the preloaded next segmented digital human video stream data.

[0078] In this embodiment, using emotional characteristics as one of the indicators for generating the digital human video stream will affect the speed of the digital human video stream generation. However, a completely lack of emotional sound will reduce the user experience. To reduce this impact, after the digital human video stream is generated, it is segmented into multiple sentences based on the answer to the question using a segmentation logic. Each sentence corresponds to a video stream. At the same time, the emotional characteristics of each sentence are determined and marked on the corresponding video stream.

[0079] After receiving the video stream, the front-end device identifies the emotion feature tag corresponding to the video stream, and each emotion feature tag corresponds to the video playback speed and video playback volume.

[0080] For example, when you are happy, the video playback speed and volume of the video stream will be increased appropriately; when you are sad, the video playback speed and volume of the video stream will be decreased appropriately; when there is no emotional change, the video playback speed and volume of the video stream will be played normally.

[0081] In this embodiment, when container A loads and plays the first segmented digital human video stream data in the front-end queue, container B is in a standby state. The resource loading status is monitored through the video monitoring event loadedmetadata, and the video playback progress is monitored through timeupdate, and the upper and lower video containers are dynamically switched alternately. When the playback progress of container A triggers the preloading condition, container B starts loading the next segmented digital human video stream data. For example, in the digital human live broadcast scene, when container A plays the digital human explanation video corresponding to the current sentence, container B has preloaded the action data of the next sentence, such as gesture changes. It should be noted that the alternating loading mechanism of the dual containers cooperates with the fault-tolerant strategy. If a segment fails to load, the system can immediately switch to the backup container and try to reload it to avoid playback interruption.

[0082] In this embodiment, when a video stream is loaded into the video container, it is mounted in the topmost z-index video container. When the loaded callback is triggered, the currently playing video container is played, and the playback progress is monitored via onTimeupdate. When the playback progress reaches 90%, the data queue is checked for subsequent data. If not, playback ends or waits for a response. If subsequent data is available, the data is sequentially retrieved and loaded into the second video player, which then loads the data. It should be noted that if the resources of a single video stream are large, this threshold can be appropriately reduced.

[0083] In this embodiment, the detection of loading failure or playback interruption is achieved through the error event and stalled event of the front-end video container. For example, when the video container cannot load segmented data due to network packet loss or data corruption, the error event will be triggered; if the playback freeze exceeds the preset time due to insufficient buffering, it will be marked as an interrupted state through a custom timer. It should be noted that once a failure of the current playback container (such as container A) is detected, the system immediately raises the z-index level of another video container (such as container B) to a visible state and attempts to reload the same segmented data. Specifically, when reloading, the optimized lightweight version is pulled from the edge computing node first. If the edge cache does not hit, it falls back to the server to request the original data.

[0084] If container B still fails to load the same segmented data, the front-end device sends a regeneration request to the server. The request carries the unique identifier of the faulty segment and the cause of the failure. The unique identifier may be a segment ID and a timestamp, and the cause of the failure may be a network timeout or data verification failure. After receiving the request, the server generates an alternative version based on the type of failure. For example, for loading failures caused by insufficient bandwidth, the server can generate a low-resolution version or a simplified video stream that only retains key action frames; if the data is damaged, the segment is re-rendered and a redundant check code is attached. It should be noted that the server quickly reconstructs the content based on the original semantic nodes to ensure that the alternative version is semantically consistent with the context.

[0085] Once the alternate version is successfully loaded, the front-end device inserts it into the queue at the original location of the faulty segment and immediately triggers playback. For example, in a virtual meeting, if a digital human's response fails to load due to network jitter, the alternate version plays seamlessly, avoiding interruptions in the conversation. For example, an alternate version might include a simplified video with only basic lip syncing. The system also deletes the original faulty segment.

[0086] Step 203: When the playback of the currently played video container is detected to be finished, switch to another video container and trigger the other video container to play the preloaded next segmented digital human video stream data to achieve seamless connection of the segmented digital human video stream data.

[0087] This application uses the ended event to monitor the video playback completion node, and uses the play event and stop event to control playback and stop respectively.

[0088] In this embodiment, after the currently playing video container completes playback and triggers the ended event, the play event of another video container is immediately triggered, and the z-index values ​​of the two video containers are swapped to switch the display levels of the two video containers. When the other video container reaches the 90% threshold, the data in the front-end queue is checked to fill the first video container, and the corresponding two levels are switched. This is repeated in this way, achieving orderly playback of multiple video streams in a queued manner. Ultimately, the playback of multiple video streams is visually seamless, just like playing a single video, and offers improved performance.

[0089] In this embodiment, the eye tracking device refers to an infrared camera or dedicated sensor module integrated into a front-end device, such as AR glasses or high-end mobile terminals. It captures the user's eye movement trajectory and pupil focus position to calculate the projection area of ​​the gaze on the screen in real time. For example, in a digital human teaching scenario, when a user gazes at the gesture area of ​​the digital human in the video container, the eye tracking device samples the gaze coordinates at a millisecond frequency and converts them into gaze area information relative to the digital human image through a coordinate mapping algorithm, such as the left hand holding area or facial expression area. It should be noted that this information is encapsulated in JSON format and contains fields such as the region boundary coordinates, gaze start timestamp, and duration.

[0090] The front-end device determines whether the user continues to gaze at a specific part for more than a preset time threshold based on the gaze area information. Specifically, the system maintains a sliding time window, such as the line of sight data in the last few seconds. If the cumulative gaze time in the same area exceeds the threshold, the interaction logic is triggered. For example, in a virtual shopping guide scenario, if the user continues to gaze at the product held by the digital person for more than the threshold, the system determines that the user needs to obtain detailed information about the product. At this time, the front-end device immediately pauses the video stream of the currently playing container, and sends the user's gaze behavior information and the progress information of the currently playing segment, such as timestamp and segment ID, to the server based on the long link channel. It should be noted that when playback is paused, the container level remains visible, but the video frame is frozen to maintain a static display of the picture.

[0091] It is understandable that after receiving the gaze behavior information, the server performs a multimodal analysis on the currently playing segmented digital human video stream data. For example, if the user's gaze area is the drug packaging held by the digital human, the server extracts the text information on the packaging through image recognition, such as ingredients and usage, and combines voice recognition technology to parse the drug-related explanation content in the current segment. Subsequently, the server generates an extended explanation video stream segment, which may include a rotating display of the drug's 3D model, a detailed instruction pop-up window, or supplementary explanation audio. It should be noted that the server ensures that the extended segment is semantically coherent with the original content based on the context of the original answer text, such as inserting a side effect description after the drug introduction.

[0092] Specifically, the server returns the generated extended interpretation video stream segment and playback insertion point information to the front-end device, such as the starting timestamp of the "drug display" node in the original segment. The front-end device inserts the extended segment at the corresponding position of the current playback container to form a mixed playback stream. For example, in an online education scenario, when the user looks at the formula derivation area explained by the digital human, the extended segment displays a step-by-step calculation animation of the formula in picture-in-picture form. After the extended segment is played, the system automatically continues to play the remaining content from the insertion point of the original segment. It should be noted that the extended segment can be directly superimposed and played in the current container without enabling a backup container to avoid layout conflicts.

[0093] In this embodiment, the screen orientation change is detected by the gyroscope or orientation sensor of the front-end device, and the event generated by it (such as orientationchange) triggers the layout adjustment logic. For example, when the mobile device switches from landscape to portrait mode, the system obtains the screen aspect ratio change information in real time. Split-screen operation refers to a multi-window display mode actively triggered by the user (such as the Android split-screen function), and the front-end device monitors the split-screen status through the operating system API (such as WindowManager). It should be noted that this monitoring mechanism runs independently from the eye tracking logic, but shares the same event processing thread to avoid resource conflicts.

[0094] Specifically, when it detects that the device has switched to portrait mode, the system adjusts the two stacked video containers to a side-by-side layout. For example, the originally stacked containers A (z-index: 2) and B (z-index: 1) are repositioned vertically: container A occupies the upper part of the screen (e.g., 60% of the height), and container B occupies the lower part (e.g., 40% of the height), with both widths adapted to the screen width. It is understandable that when dynamically adjusting the size ratio, the system calculates the container size based on the available screen area to ensure that the display ratio of the digital human image is not distorted. It should be noted that the hierarchical control logic temporarily fails in this scenario because both containers are visible, but the preloading mechanism still operates according to the original rules, that is, container B continues to load subsequent segmented data to prepare for possible landscape mode recovery.

[0095] In this embodiment, if the user triggers a split-screen operation, such as shrinking the application window to half the screen, the system immediately hides the backup video container (such as container B) and suspends its preloading task to release computing resources. For example, in split-screen mode, the size of the current playback container (container A) is dynamically compressed to the split-screen area, such as 50% of the screen width, and redundant UI elements (such as control buttons) are removed to maximize the display area. It should be noted that after hiding container B, the unloaded segmented data in the front-end queue is still cached in order, but preloading is resumed only when the split screen is exited. In addition, the edge computing node gives priority to pushing lightweight segments in this mode to ensure low resource consumption in split-screen scenarios.

[0096] Figure 4 This is a schematic diagram of the structure of a front-end processing device for a digital human video stream provided in an embodiment of the present application. Figure 4As shown, the front-end processing device for a digital human video stream disclosed in the present application includes a server module, a WebSocket module, and a front-end module. The server module includes a video stream segmentation generation mechanism, so that the digital human video stream is segmented on the WebSocket server through the video stream segmentation generation mechanism. The WebSocket module includes a long link channel, specifically, the long link channel is implemented through the WebSocket protocol, and then the segmented digital human video stream data is pushed to the front-end device in real time through the long link channel. The front-end module includes a front-end queue for caching the received segmented digital human video stream data and pushing it to the WebSocket client.

[0097] It should be noted that the front-end device in this application uses two video containers to form a stacked layout, and uses a time event listener to monitor the playback progress of the currently playing video container. Then, according to the playback progress, it alternately plays the segmented digital human video stream data in the front-end queue and renders a natural video stream in real time to enhance the user's viewing experience.

[0098] In some embodiments, based on the pre-built long link channel, before receiving the segmented digital human video stream data generated by the server, the method further includes:

[0099] The server receives the intelligent question-answering request initiated by the user based on the front-end device, generates the corresponding answer text, and performs semantic analysis on the answer text to identify the key semantic nodes associated with the context;

[0100] Monitor the network bandwidth of front-end devices in real time and dynamically adjust the maximum duration threshold of segmented digital human video stream data based on the monitoring results;

[0101] According to the key semantic nodes and the adjusted maximum duration threshold, the segmentation points of the digital human video stream are dynamically adjusted so that each segmented digital human video stream data contains a complete semantic unit;

[0102] The answer text is divided into multiple sentences according to the adjusted segmentation points, and a video stream segment corresponding to each sentence is generated. The action parameters and expression parameters of the digital human are associated with each video stream segment to obtain segmented digital human video stream data.

[0103] In some embodiments, further comprising:

[0104] On the server side, sentiment analysis is performed on each sentence to identify the emotion type corresponding to the sentence and generate associated emotion feature tags for the segmented digital human video stream data corresponding to the sentence;

[0105] Parsing the emotional feature tags of the segmented digital human video stream data on the front-end device, and adjusting the video playback speed and video playback volume corresponding to the segmented digital human video stream data according to the tag type in the parsing result and based on the predefined playback parameter mapping rules;

[0106] The adjusted video playback speed and volume are applied in real time during video playback to ensure that the emotional expression of the playing video is synchronized with the video content.

[0107] In some embodiments, after monitoring the playback progress of the currently playing video container, the method further includes:

[0108] The display level of the two video playback containers is controlled through the z-index property. When the currently playing video container loads and plays the current segmented digital human video stream data in the front-end queue, the display level of the currently playing video container is set to visible, and the display level of the other video container is set to hidden.

[0109] After the currently playing video container completes playing the currently segmented digital human video stream data, setting the display level of the currently playing video container to a hidden state;

[0110] When another video container is activated, the display level of the other video container is synchronously updated to a visible state, and a play instruction of the other video container is triggered.

[0111] In some embodiments, after caching the segmented digital human video stream data into the front-end queue in the transmission order, the method further includes:

[0112] On the edge computing node, the action parameters and expression parameters of the digital human in the segmented digital human video stream data are dynamically compressed according to the hardware performance parameters of the front-end device;

[0113] Binding the compressed action parameters and expression parameters to the segmented digital human video stream data to generate optimized video stream segments that adapt to the hardware performance parameters of the front-end device;

[0114] Through the edge computing node, the local cache stores the preset frequently used video stream segments and generates a temporary access link so that the front-end device can first pull the video stream segment from the local cache of the edge computing node. If it misses, it will initiate a data acquisition request to the server.

[0115] Through the collaborative caching mechanism between edge computing nodes and front-end devices, and according to network bandwidth fluctuations, the distribution ratio of segmented digital human video stream data between edge computing nodes and front-end devices is dynamically adjusted to reduce network transmission delay.

[0116] In some embodiments, further comprising:

[0117] When detecting that the loading of the currently playing video container fails or the playback is interrupted, switch to another video container and try to reload the same segmented digital human video stream data;

[0118] If another video container also fails to load, a regeneration request is sent to the server to obtain an alternative version of the segmented digital human video stream data;

[0119] After the alternative version is loaded successfully, the alternative version is inserted into the original position of the faulty segmented digital human video stream data in the front-end queue and continues to be played, and the faulty segmented digital human video stream data is deleted.

[0120] In some embodiments, further comprising:

[0121] When the front-end device plays the segmented digital human video stream data, it obtains the user's gaze area information of the digital human image in the currently playing video container in real time;

[0122] Based on the gaze area information, determine whether the user continues to gaze at a specific part of the digital human image for more than a preset gaze duration threshold. If so, pause the video playback of the current video container and send the user's gaze behavior information and the playback progress information of the currently playing segmented digital human video stream data to the server;

[0123] The server analyzes the currently played segmented digital human video stream data, extracts detailed information related to specific parts, and generates corresponding extended interpretation video stream segments based on the detailed information, so as to send the extended interpretation video stream segments and corresponding playback insertion point information to the front-end device;

[0124] Through the front-end device, in the currently playing video container, the extended interpretation video stream segment is inserted and played starting from the playback progress position corresponding to the playback insertion point information, and after the extended interpretation video stream segment is played, the remaining unplayed part of the segmented digital human video stream data continues to be played.

[0125] In some embodiments, further comprising:

[0126] Monitor the front-end device's screen orientation changes or user-triggered split-screen operations. If the front-end device switches to portrait mode, adjust the two video containers in the stacked layout to a parallel layout and dynamically adjust the size ratio of the two video containers.

[0127] In split-screen mode, the other spare video container of the two video containers is hidden, the preloading operation of the other video container is paused, and the currently playing video container is kept for playing to adapt to the display area.

[0128] The above is an embodiment of the method proposed in this application. Based on the same inventive concept, this application embodiment also provides a front-end processing device for digital human video stream, whose structure is as follows: Figure 5 shown.

[0129] Figure 5 This is a schematic diagram of the internal structure of a front-end processing device for a digital human video stream provided in an embodiment of the present application. Figure 5 As shown, the equipment includes:

[0130] at least one processor;

[0131] and, a memory communicatively coupled to the at least one processor;

[0132] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0133] Based on the pre-built long link channel, it receives the segmented digital human video stream data generated by the server, and caches the segmented digital human video stream data into the front-end queue in the transmission order;

[0134] Monitor the playback progress of the currently playing video container. When the playback progress reaches a preset playback progress threshold, extract the next segment of the digital human video stream data from the front-end queue and load it into another video container.

[0135] When the playback of the currently playing video container is detected to be finished, switch to another video container and trigger the other video container to play the preloaded next segmented digital human video stream data to achieve seamless connection of the segmented digital human video stream data.

[0136] The present application also provides a non-volatile computer storage medium storing computer-executable instructions. When the computer-executable instructions are executed, they can:

[0137] Based on the pre-built long link channel, it receives the segmented digital human video stream data generated by the server, and caches the segmented digital human video stream data into the front-end queue in the transmission order;

[0138] Monitor the playback progress of the currently playing video container. When the playback progress reaches a preset playback progress threshold, extract the next segment of the digital human video stream data from the front-end queue and load it into another video container.

[0139] When the playback of the currently playing video container is detected to be finished, switch to another video container and trigger the other video container to play the preloaded next segmented digital human video stream data to achieve seamless connection of the segmented digital human video stream data.

[0140] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device and medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0141] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects to their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0142] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0144] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0146] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0147] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0148] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0149] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

Claims

1. A front-end processing method for a digital human video stream, characterized in that: The method comprises: Based on the pre-built long link channel, it receives the segmented digital human video stream data generated by the server, and caches the segmented digital human video stream data into the front-end queue in the transmission order; Monitor the playback progress of the currently playing video container, and when the playback progress reaches a preset playback progress threshold, extract the next segmented digital human video stream data from the front-end queue and load it into another video container; When the playback of the currently playing video container is detected to have ended, switching to the other video container and triggering the other video container to play the preloaded next segmented digital human video stream data to achieve seamless connection of the segmented digital human video stream data; Before receiving the segmented digital human video stream data generated by segmentation at the server based on the pre-built long link channel, the method further includes: The server receives the intelligent question-answering request initiated by the user based on the front-end device, generates the corresponding answer text, and performs semantic analysis on the answer text to identify the key semantic nodes associated with the context; Real-time monitoring of the network bandwidth of the front-end device, and dynamic adjustment of the maximum duration threshold of the segmented digital human video stream data according to the monitoring results; Dynamically adjust the segmentation points of the digital human video stream according to the key semantic nodes and the adjusted maximum duration threshold, so that each segmented digital human video stream data contains a complete semantic unit; The answer text is divided into multiple sentences according to the adjusted segmentation points, a video stream segment corresponding to each sentence is generated, and the action parameters and expression parameters of the digital human are associated with each video stream segment to obtain segmented digital human video stream data; After caching the segmented digital human video stream data into the front-end queue in the transmission order, the method further includes: On the edge computing node, the action parameters and expression parameters of the digital human in the segmented digital human video stream data are dynamically compressed according to the hardware performance parameters of the front-end device; Binding the compressed action parameters and expression parameters to the segmented digital human video stream data to generate optimized video stream segments adapted to the hardware performance parameters of the front-end device; Through the edge computing node, the local cache stores the preset frequently used video stream segments and generates a temporary access link so that the front-end device preferentially pulls the video stream segments from the local cache of the edge computing node. If a hit is not found, a data acquisition request is initiated to the server. Through the collaborative caching mechanism between the edge computing node and the front-end device, and according to the fluctuation of network bandwidth, the distribution ratio of the segmented digital human video stream data between the edge computing node and the front-end device is dynamically adjusted to reduce network transmission delay.

2. The front-end processing method of a digital human video stream according to claim 1, characterized in that: The method further comprises: Perform sentiment analysis on each sentence on the server side to identify the emotion type corresponding to the sentence, and generate associated emotion feature tags for the segmented digital human video stream data corresponding to the sentence; Parsing the emotional feature tags of the segmented digital human video stream data on the front-end device, and adjusting the video playback speed and video playback volume corresponding to the segmented digital human video stream data according to the tag type in the parsing result and based on predefined playback parameter mapping rules; The adjusted video playback speed and volume are applied in real time during video playback to ensure that the emotional expression of the playing video is synchronized with the video content.

3. The front-end processing method of a digital human video stream according to claim 1, characterized in that: After monitoring the playback progress of the currently playing video container, the method further includes: The display levels of the two video playback containers are controlled by the z-index attribute, so that when the currently playing video container loads and plays the current segmented digital human video stream data in the front-end queue, the display level of the currently playing video container is set to a visible state, and the display level of the other video container is set to a hidden state; After the currently playing video container completes playing the currently segmented digital human video stream data, setting the display level of the currently playing video container to a hidden state; When the other video container is activated, the display level of the other video container is synchronously updated to a visible state, and a play instruction of the other video container is triggered.

4. The front-end processing method of a digital human video stream according to claim 1, characterized in that: The method further comprises: When detecting that the loading of the currently playing video container fails or the playback is interrupted, switch to another video container and try to reload the same segmented digital human video stream data; If the other video container also fails to load, a regeneration request is sent to the server to obtain an alternative version of the segmented digital human video stream data; After the replacement version is loaded successfully, the replacement version is inserted into the original position of the faulty segmented digital human video stream data in the front-end queue and continues to be played, and the faulty segmented digital human video stream data is deleted.

5. The front-end processing method of a digital human video stream according to claim 1, characterized in that: The method further comprises: When the front-end device plays the segmented digital human video stream data, it obtains the user's gaze area information of the digital human image in the currently playing video container in real time; Based on the gaze area information, determine whether the user continues to gaze at a specific part of the digital human image for more than a preset gaze duration threshold; if so, pause the video playback of the currently playing video container, and send the user gaze behavior information and the playback progress information of the currently playing segmented digital human video stream data to the server; The server analyzes the currently played segmented digital human video stream data, extracts detailed information related to the specific part, and generates a corresponding extended interpretation video stream segment based on the detailed information, so as to send the extended interpretation video stream segment and corresponding playback insertion point information to the front-end device; Through the front-end device, in the currently playing video container, the extended interpretation video stream segment is inserted and played starting from the playback progress position corresponding to the playback insertion point information, and after the extended interpretation video stream segment is played, the remaining unplayed part of the segmented digital human video stream data continues to be played.

6. The front-end processing method of a digital human video stream according to claim 1, characterized in that: The method further comprises: Monitor the front-end device's screen orientation changes or user-triggered split-screen operations. If the front-end device switches to portrait mode, adjust the two video containers in the stacked layout to a parallel layout and dynamically adjust the size ratio of the two video containers. In the split-screen mode, the other spare video container of the two video containers is hidden, the preloading operation of the other video container is paused, and the currently playing video container is kept for playing to adapt to the display area.

7. A front-end processing device for digital human video stream, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the front-end processing method of a digital human video stream as described in any one of claims 1-6.

8. A non-volatile computer storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed, a front-end processing method for a digital human video stream according to any one of claims 1 to 6 is implemented.

Images