Digital human broadcasting control method, system, medium and terminal

Abstract

The application provides a digital human broadcast control method, system, medium and terminal, wherein the digital human broadcast control method comprises: acquiring streaming data of a data source associated with a data large screen and determining a variation characteristic quantity thereof over time; performing abnormality detection according to a preset abnormality criterion and determining an association relationship between each abnormal event; performing coding on the characteristics and the association relationship of the abnormal event to obtain a target event representation, and generating a broadcast text with the target event representation as a generation condition; determining a broadcast order according to a preset priority rule; determining a prosody boundary of the broadcast text according to the variation characteristic quantity and synthesizing into an audio stream; converting the audio stream into a facial animation parameter sequence of a digital human, determining a time offset therebetween and correcting the same to drive the digital human to render and output. The application can generate sound and picture synchronous broadcast content in real time according to the dynamic change of industrial data, and match the broadcast content with a data area observed by a user.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and industrial data visualization technology, and in particular to digital human broadcasting control methods, systems, media and terminals. Background Technology

[0002] As industrial digitalization and intelligentization continue to improve, industrial command centers, project scheduling halls, and production monitoring centers are gradually becoming important platforms for equipment operation management, project coordination and scheduling, and regional business analysis. To intuitively present core indicators such as equipment operating status, project implementation progress, and regional business distribution, these locations typically feature large data screens, centrally displaying business data through visualizations such as charts, maps, and lists. Meanwhile, with the development of artificial intelligence, natural language processing, and digital human technology, utilizing digital humans to assist in data explanation, information retrieval, and fault reporting has become an important technological direction for improving data display efficiency and human-computer interaction experience.

[0003] Existing data dashboards primarily use static, visual charts to present core indicators such as equipment operation data, project progress, and regional business distribution. During use, presenters typically need to manually switch display views based on tour or scheduling needs, and verbally explain the meaning and changes of the data based on their personal experience. Some existing solutions also incorporate digital humans into the data dashboards, using TTS (Text-to-Speech) technology to convert preset or temporarily generated text into speech, and combining the generated speech with facial animation to achieve voice broadcasting and visual display. Furthermore, existing digital humans can filter information to be broadcast according to preset business rules, receive user questions through voice interaction, and broadcast corresponding fault phenomena based on alarm information when equipment malfunctions.

[0004] However, the aforementioned existing technologies still have the following shortcomings. First, the content of traditional data dashboard explanations highly relies on the personal experience of the presenters, resulting in low standardization. Different presenters may interpret the same data differently. When data is updated, manual identification of data changes and corresponding adjustments to the explanation content are required, making it difficult to guarantee real-time delivery. Second, existing digital humans typically simply splice text-to-speech results with facial animations. In industrial scenarios where equipment status changes dozens of times per second, leading to high-frequency data jumps, issues such as mismatch between lip movements and voice, and broadcast delays, can easily occur. Third, the number of sensors in industrial scenarios is enormous, potentially reaching thousands. Broadcasting all the data can easily cause information overload. Relying on preset business rules to filter information is inefficient and makes it difficult to generate effective narrative content in a timely manner based on real-time data changes. Furthermore, existing voice interaction typically lacks spatial registration capabilities, failing to accurately determine the user's location and gaze direction, resulting in a mismatch between the digital human's explanation content and the user's current gaze area. Finally, existing fault reports usually only present the alarm phenomenon itself, lacking analysis of the root cause of the fault and the fault propagation path. Operation and maintenance personnel still need to manually review the operation logs and check each item one by one, resulting in low fault location efficiency.

[0005] Therefore, existing data dashboards and digital human solutions still have shortcomings in terms of standardized explanation content, response to data changes, audio-visual synchronization, massive information filtering, user spatial status awareness, and root cause analysis of faults. These limitations make it difficult to meet the requirements of real-time performance, accuracy, interactivity, and intelligent analysis capabilities in industrial command, project scheduling, and equipment maintenance scenarios. Therefore, there is an urgent need to provide a technical solution that can adapt to the dynamic changes in industrial data and improve the efficiency of data explanation, interactive response, and fault analysis. Summary of the Invention

[0006] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a digital human broadcasting control method, system, medium and terminal to solve the technical problem that existing data screens and digital human solutions are unable to generate audio-visual synchronized explanation content in real time, accurately and in line with the user's gaze area according to the dynamic changes of industrial data, and to effectively analyze the root causes and propagation paths of equipment failures.

[0007] To achieve the above and other related objectives, a first aspect of this application provides a digital human broadcast control method, comprising: acquiring streaming data from a data source associated with a data dashboard, the streaming data including real-time data streams collected in real time, and determining a change characteristic quantity of the streaming data over time, the change characteristic quantity representing the data change state of the data source; performing anomaly detection on the streaming data according to a preset anomaly criterion to obtain an anomaly event set, and determining the correlation relationship between each anomaly event in the anomaly event set; encoding the characteristics of the anomaly events and the correlation relationship to obtain a target event representation of the anomaly events, and using the target event representation as a generation bar. The system generates a broadcast text corresponding to the abnormal event; determines the broadcast order of each event to be broadcast in the broadcast queue consisting of the events to be broadcast corresponding to the broadcast text according to a preset priority rule; determines the prosodic boundary of the broadcast text according to the change feature quantity, and synthesizes the broadcast text into an audio stream according to the prosodic boundary; converts the audio stream into a facial animation parameter sequence of the digital human, determines the time offset between the audio stream and the facial animation parameter sequence, performs correction on the facial animation parameter sequence according to the time offset, and drives the rendering output of the digital human according to the corrected facial animation parameter sequence and the audio stream.

[0008] The following describes several preferred embodiments of the first aspect described above: each embodiment corresponds to a step in the above method, namely, determining the variable feature quantity, determining the prosodic boundary, performing anomaly detection, determining the correlation, generating the target event representation and broadcast text, determining the broadcast order, and correcting the facial animation parameter sequence; based on this, this application also provides a spatial interactive positioning implementation method that matches the broadcast content with the user's gaze area, and a fault root cause broadcasting implementation method that extends the broadcast content to the fault root cause and propagation path.

[0009] In some embodiments of the first aspect of this application, the change feature includes the data change rate of each of the data sources and the mutation event in the real-time data stream; determining the change feature of the streaming data over time includes: dividing the historical data corresponding to the streaming data into several time-series segments according to a preset length, performing segment encoding and superimposing position encoding on each of the time-series segments, inputting the segment into an encoder based on a self-attention mechanism to obtain a time-series dependency representation characterizing the time-series dependency relationship between the time-series segments, and performing regression processing on the time-series dependency representation to obtain the data change rate of each of the data sources at a future time; comparing the data change rate of each of the data sources with an adaptive threshold obtained by statistical analysis of the historical data of the data source, and determining the moment when the data change rate exceeds the adaptive threshold as the mutation event.

[0010] In some embodiments of the first aspect of this application, the prosodic boundary includes a strong boundary, a medium boundary, and a weak boundary; determining the prosodic boundary of the broadcast text based on the change feature quantity includes: determining the position in the broadcast text corresponding to the abrupt event as the strong boundary, the strong boundary corresponding to a pause and intonation change; determining the position in the broadcast text where the trend of the change rate of the data changes is reversed as the medium boundary, the pause degree corresponding to the medium boundary being less than that of the strong boundary; determining the positions in the broadcast text other than the strong boundary and the medium boundary as the weak boundary, the weak boundary not corresponding to a pause.

[0011] In some embodiments of the first aspect of this application, the step of performing anomaly detection on the streaming data according to a preset anomaly criterion to obtain an anomaly event set includes: presetting a probability density model for each data source, the probability density model being composed of several cascaded invertible transformations for mapping the data distribution of the data source to a baseline distribution; inputting the streaming data at the current moment into the probability density model to obtain the log-likelihood of the streaming data under a conditional distribution determined by corresponding historical data; using the sum of the negative value of the log-likelihood and the baseline value of the data source as an anomaly score, and determining the streaming data whose anomaly score satisfies the anomaly criterion as the anomaly event, and the anomaly events constituting the anomaly event set.

[0012] In some embodiments of the first aspect of this application, the association relationship includes the propagation path between the abnormal events; determining the association relationship between each abnormal event in the abnormal event set includes: constructing a dynamic topology graph that updates over time, using the sensor nodes corresponding to each data source as nodes and the physical connection relationship and data correlation relationship between each sensor node as edges; maintaining a memory vector for each sensor node that is updated with the received messages, and determining the inter-node messages based on the memory vector, the memory vectors of adjacent sensor nodes, and the connection relationship between adjacent sensor nodes, and aggregating the inter-node messages to obtain the node embedding of each sensor node; and determining the propagation path from the sensor node where the abnormal event occurred to other sensor nodes based on the node embedding of each sensor node.

[0013] In some embodiments of the first aspect of this application, the features of the anomalous event include temporal features, spatial features, and topological features; the step of encoding the features of the anomalous event and the correlation to obtain a target event representation of the anomalous event includes: encoding the temporal features by a first encoder to obtain a temporal feature vector, and encoding the spatial features and the topological features by a second encoder to obtain a graph structure feature vector; performing cross-modal attention fusion with the temporal feature vector and the graph structure feature vector as input to obtain fused features; performing vector quantization on the fused features to discretize the fused features into a semantic tag sequence composed of several semantic tags arranged in sequence, and using the semantic tag sequence as the target event representation.

[0014] In some embodiments of the first aspect of this application, generating a broadcast text corresponding to the abnormal event using the target event representation as a generation condition includes: determining the causal chain corresponding to the abnormal event based on the correlation, and constructing a prompt message from the target event representation, the abnormal event, and the causal chain; inputting the prompt message into a pre-trained language model to obtain the broadcast text organized hierarchically according to the current fault phenomenon, fault propagation path, root cause, and handling suggestions.

[0015] In some embodiments of the first aspect of this application, the priority rule is a score output by a scoring model; determining the broadcast order of each of the events to be broadcast in the broadcast queue according to a preset priority rule includes: determining the importance based on the degree of abnormality of the abnormal event corresponding to each of the events to be broadcast, determining the urgency based on the severity of the abnormality corresponding to each of the events to be broadcast, and determining the score of each of the events to be broadcast according to the importance, the time interval between the occurrence time of the event to be broadcast and the current time, and the urgency; wherein the score increases with the increase of the importance, decreases with the increase of the time interval, and increases with the increase of the urgency; performing multi-head attention calculation on the score of each of the events to be broadcast using the current broadcast capability as the query, the features of each of the events to be broadcast as the key, and the content representation of each of the events to be broadcast as the value; obtaining the priority by fusing the outputs of each attention head, and determining the broadcast order of each of the events to be broadcast in the broadcast queue from high to low according to the priority.

[0016] In some embodiments of the first aspect of this application, determining the time offset between the audio stream and the facial animation parameter sequence, and correcting the facial animation parameter sequence according to the time offset, includes: performing convolutional coding on the audio stream and the facial animation parameter sequence respectively to obtain an audio embedding sequence and a visual embedding sequence; performing cross-correlation operation on the audio embedding sequence and the visual embedding sequence, and determining the lag that maximizes the cross-correlation value as the time offset; shifting the values ​​of corresponding frames in the facial animation parameter sequence forward or backward according to the time offset, and performing interpolation smoothing on several adjacent frames in the corrected facial animation parameter sequence; wherein, the time of each frame in the facial animation parameter sequence is the sum of the time of the corresponding phoneme in the audio stream and the time offset, and the time of each phoneme in the audio stream is the sum of the time of the mutation event and the advance corresponding to the timing prediction.

[0017] In addition to the basic link for broadcast control described above, the first aspect of this application also provides the following additional implementation method for making the broadcast content correspond to the data area currently being viewed by the user:

[0018] In some embodiments of the first aspect of this application, the method further includes: acquiring multi-channel audio and video streams within the physical space where the data screen is located; performing beamforming scanning on the multi-channel audio streams, determining the sound source direction based on the output power in each scanning direction, and determining the sound source distance based on the arrival time difference between each channel, thereby obtaining a first position estimate of the user; detecting a face region from the video stream, and estimating the user's gaze direction vector based on the face region; mapping the first position estimate, the user's head pose, and the gaze direction vector to the same world coordinate system, and performing filtering preprocessing on the mapping result. The system measures and obtains the position of the user's gaze point in the world coordinate system; maps the gaze point to a target data area on the data screen according to the screen layout of the data screen; determines the digital human's posture parameters and gestures based on the target data area and the user's position, and highlights the target data area; wherein, the target event representation is also generated based on the target data area, so that the generated broadcast text corresponds to the target data area; and the broadcast order of each event to be broadcast in the broadcast queue is determined according to the change of the target data area relative to the previous moment.

[0019] In some embodiments of the first aspect of this application, the step of performing beamforming scanning on the multi-channel audio stream, determining the sound source direction based on the output power in each scanning direction, and determining the sound source distance in combination with the arrival time difference between each channel to obtain a first position estimate of the user includes: determining beamforming weights in each scanning direction according to the minimum variance distortion-free response criterion based on the covariance matrix of the multi-channel audio stream, and determining the scanning direction with the highest output power as the sound source direction; inputting the spectral features of the multi-channel audio stream into a deep neural network to obtain the probability distribution of the user at each spatial location, and obtaining a second position estimate based on the probability distribution; performing weighted fusion of the first position estimate and the second position estimate with adaptive fusion weights, and using the fusion result as the first position estimate.

[0020] In some embodiments of the first aspect of this application, detecting a face region from the video stream and estimating the user's gaze direction vector based on the face region includes: detecting and cropping the face region from the current frame of the video stream; extracting image features of the face region and performing global pooling on the image features to obtain a face feature vector; performing regression processing on the face feature vector to obtain a unit vector representing the gaze direction, and using the angle between the unit vector and the ground truth of the gaze direction as the training loss.

[0021] In some embodiments of the first aspect of this application, the step of uniformly mapping the first position estimate, the user's head pose, and the gaze direction vector to the same world coordinate system, and performing filtering prediction on the mapping result, includes: transforming the world coordinate system, the screen coordinate system of the data screen, and the user's user coordinate system using a homogeneous transformation matrix, wherein the homogeneous transformation matrix is ​​determined by the installation posture of the data screen and the posture of the image acquisition device; using the user's position and velocity in the world coordinate system as a state vector, performing prediction and updating on the state vector to obtain an updated state vector; determining the position of the gaze point in the world coordinate system according to the position indicated by the updated state vector and the gaze direction vector; wherein, when the user's motion satisfies a nonlinear condition, particle filtering is used instead of the prediction and the update.

[0022] In some embodiments of the first aspect of this application, determining the digital human's posture parameters and gestures based on the target data region and the user's position includes: performing inverse kinematics calculations on the digital human's eye position and the target point, with the center of the target data region as the target point, to obtain the digital human's head orientation; determining the digital human's body orientation based on the user's position in the world coordinate system, and transitioning from the current body orientation to the body orientation according to a preset interpolation coefficient; when the target data region changes relative to the previous moment, triggering the digital human to perform a gesture pointing to the target data region, and highlighting the target data region.

[0023] In addition to the aforementioned spatial interactive positioning, the first aspect of this application also provides the following additional implementation method for extending the broadcast content from alarm phenomena to the root cause of the fault and the fault propagation path:

[0024] In some embodiments of the first aspect of this application, the method further includes: acquiring historical operational data of the monitored object; performing causal discovery on the historical operational data based on conditional independence tests; constructing a causal graph characterizing the causal relationships between the monitored variables; acquiring real-time operational data of the monitored object using a sliding window; performing time-series causal tests on each pair of the monitored variables within the sliding window; and updating the causal graph based on the test results; responding to an alarm event; determining the ancestor nodes of the nodes corresponding to the alarm event in the causal graph; and extracting the propagation paths from each ancestor node to the node corresponding to the alarm event to obtain a root cause candidate set; and processing the root cause... The graph is encoded to obtain a graph representation vector. Using the graph representation vector, the root cause candidate set, and the alarm event as generation conditions, a broadcast text containing the fault phenomenon, fault propagation path, and root cause is generated. Each candidate node in the root cause candidate set is sorted according to a preset root cause score. The root cause score increases as the probability of the candidate node being the root cause increases, increases as the severity of the abnormality of the candidate node increases, and decreases as the time interval between the current time and the occurrence time of the alarm event increases. The digital human is driven to perform root cause broadcasting according to the broadcast text, and the causal graph is updated incrementally according to the real-time running data updated by the sliding window.

[0025] In some embodiments of the first aspect of this application, the step of performing causal discovery on the historical operating data based on conditional independence testing and constructing a causal graph representing the causal relationship between the monitored variables includes: initializing a completely undirected graph with each monitored variable as a node; performing conditional independence testing on adjacent node pairs in ascending order of condition set; deleting edges that satisfy conditional independence and recording the corresponding separation sets; performing directional inference on triples that satisfy a preset structure based on each separation set to obtain directed edges; and completing the orientation of the remaining edges according to the transitive orientation rule to obtain the causal graph; when there are unobserved confounding variables, outputting an ancestor graph containing some directed edges and bidirectional edges as the causal graph; or performing a greedy search of forward edge addition and backward edge deletion on the causal graph based on a scoring function.

[0026] In some embodiments of the first aspect of this application, the step of performing a time-series causality test on each pair of the monitored variables within the sliding window and updating the causal graph based on the test results includes: performing a Granger causality test on the first and second monitored variables within the sliding window to obtain a test result on whether the first monitored variable is a Granger cause of the second monitored variable; calculating the transfer entropy from the first monitored variable to the second monitored variable, and determining the first monitored variable that simultaneously satisfies the criteria corresponding to the Granger causality test and the transfer entropy as the causal parent node of the second monitored variable; adding a directed edge from the causal parent node to the second monitored variable in the causal graph, and determining the confidence level of the directed edge using the statistic of the Granger causality test.

[0027] In some embodiments of the first aspect of this application, the step of performing graph encoding on the causal graph to obtain a graph representation vector, and generating a broadcast text containing fault phenomena, fault propagation paths, and root causes using the graph representation vector, the root cause candidate set, and the alarm event as generation conditions, includes: performing node embedding and superimposing position encoding on the node features of each of the monitored variables to obtain an initial representation of each node; performing graph attention calculation using the adjacency relationship of the causal graph as a mask, wherein the positions of node pairs corresponding to non-existent edges participate in the graph attention calculation with a negative infinite bias to obtain a target layer representation of each node, and performing graph pooling on each of the target layer representations to obtain the graph representation vector; constructing prompt information according to a hierarchical structure of fault phenomena, fault propagation paths, root causes and root cause confidence, and handling suggestions, and inputting the graph representation vector, the root cause candidate set, and the alarm event into the prompt information and then inputting it into a pre-trained language model to obtain the broadcast text.

[0028] In some embodiments of the first aspect of this application, the probability of a candidate node being a root cause in the root cause scoring is determined by backpropagation of the confidence of the directed edges pointing to the candidate node from each parent node of the candidate node; the step of updating the causal graph incrementally according to the real-time running data after the sliding window update includes: re-performing the conditional independence test on each existing edge in the causal graph, updating the edges whose test results have changed, and adding newly discovered causal relationships within the sliding window to the causal graph according to their confidence levels meeting preset conditions.

[0029] To achieve the above and other related objectives, a second aspect of this application provides a data dashboard digital human assistant system, including a data acquisition module, a change feature extraction module, an anomaly analysis module, an event compression module, a narrative generation module, a broadcast arbitration module, a frame-level synchronous broadcast module, a spatial interactive positioning module, a fault root cause broadcast module, and a scene scheduling module;

[0030] The data acquisition module is used to acquire streaming data from the data source associated with the data dashboard, including real-time data streams collected in real time. The change feature extraction module is used to determine the change features of the streaming data over time, whereby the change features characterize the data change state of the data source. The anomaly analysis module is used to perform anomaly detection on the streaming data according to preset anomaly criteria, obtain a set of anomaly events, and determine the correlation between each anomaly event in the set. The event compression module is used to encode the features of the anomaly events and the correlation to obtain a target event representation of the anomaly events. The narrative generation module is used to generate a narrative based on the target event representation. The system generates broadcast text corresponding to the abnormal event; the broadcast arbitration module is used to determine the broadcast order of each of the events to be broadcast in the broadcast queue consisting of the events to be broadcast corresponding to the broadcast text according to a preset priority rule; the frame-level synchronous broadcast module is used to determine the prosodic boundary of the broadcast text according to the change feature quantity, and synthesize the broadcast text into an audio stream according to the prosodic boundary; the audio stream is converted into a facial animation parameter sequence of the digital human, the time offset between the audio stream and the facial animation parameter sequence is determined, the facial animation parameter sequence is corrected according to the time offset, and the rendering output of the digital human is driven by the corrected facial animation parameter sequence and the audio stream.

[0031] The spatial interaction positioning module is used to collect multi-channel audio and video streams within the physical space where the data screen is located, thereby determining the user's first position estimate and the user's gaze direction vector, mapping the user's gaze point to a target data area on the data screen, and determining the digital human's posture parameters and gestures based on the target data area and the user's position. The fault root cause broadcasting module is used to construct and update a causal graph representing the causal relationship between various monitored variables online, determine a root cause candidate set in the causal graph in response to an alarm event, and generate broadcast text containing fault phenomena, fault propagation paths, and root causes, and sort each candidate node in the root cause candidate set according to a preset root cause score. The scene scheduling module is used to determine the activation status and priority of the change feature extraction module, the anomaly analysis module, the event compression module, the narrative generation module, the broadcast arbitration module, the frame-level synchronous broadcasting module, the spatial interaction positioning module, and the fault root cause broadcasting module according to the scene mode.

[0032] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described digital human broadcasting control method.

[0033] To achieve the above and other related objectives, a fourth aspect of this application provides a computer program product comprising computer program code, which, when executed on a computer, enables the computer to implement the aforementioned digital human broadcast control method.

[0034] To achieve the above and other related objectives, a fifth aspect of this application provides an electronic terminal, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the above-described digital human broadcast control method.

[0035] As described above, the digital human broadcasting control method, system, medium, and terminal of this application have the following beneficial effects:

[0036] Firstly, this application uses the characteristics of streaming data changing over time to drive the generation of anomaly detection and broadcast text, so that the content of the explanation is triggered by the changes in the data itself, rather than relying on the personal experience of the presenter and manual switching of views. This improves the standardization of the content of the explanation and keeps the content of the explanation synchronized with the data updates.

[0037] Secondly, this application uses the change characteristics of data to determine the prosodic boundaries of the broadcast text, so that the pauses and changes in intonation in speech correspond to the locations where data changes abruptly and trends reverse, and key data is emphasized in auditory sense.

[0038] Third, this application performs cross-correlation calculations on the audio stream and facial animation parameter sequence to calculate the time offset and correct it accordingly, so that the audio and facial animation are aligned with each other at the frame level, avoiding the problems of lip-sync and sound mismatch and broadcast delay in high-frequency jump scenarios of industrial data.

[0039] Fourth, this application encodes the features and relationships of abnormal events and vectorizes them into semantic tag sequences to generate broadcast text. While retaining necessary information, it compresses the input scale of text generation and determines the broadcast order according to importance, urgency and time decay, thus avoiding information overload caused by broadcasting the full amount of massive sensor data.

[0040] Fifth, this application determines the user's gaze point and its corresponding target data area on the data screen by beamforming scanning of multi-channel audio streams and face detection in video streams. Based on this, the broadcast content, broadcast order, and the posture and gestures of the digital human are adjusted to match the narration content with the data area currently being focused on by the user.

[0041] Sixth, this application constructs and continuously updates a causal graph through offline causal discovery and online time-series causal verification, thereby determining the root cause candidate set and generating a broadcast text containing fault phenomena, fault propagation paths and root causes, enabling operation and maintenance personnel to directly locate the source of the fault based on the broadcast content; at the same time, each processing link is connected in a streaming manner according to data segments, shortening the waiting time from data change to broadcast output. Attached Figure Description

[0042] Figure 1 The diagram shown is a flowchart of a digital human broadcasting control method in one embodiment of this application.

[0043] Figure 2 The diagram shows a layered architecture of a data dashboard digital human assistant system in one embodiment of this application.

[0044] Figure 3 The diagram shows the data flow architecture of a data dashboard digital human assistant system in one embodiment of this application.

[0045] Figure 4 This is a schematic diagram showing the interaction relationships between the core subsystems in one embodiment of this application;

[0046] Figure 5 The diagram shown is a schematic representation of a multimodal semantic tag compression process in one embodiment of this application;

[0047] Figure 6 The diagram shown is a schematic representation of a causal narrative generation process in one embodiment of this application.

[0048] Figure 7 The diagram shows the architecture of the semantic compression and narrative generation engine in one embodiment of this application.

[0049] Figure 8 The diagram shown is a schematic representation of the frame-level synchronization engine in one embodiment of this application.

[0050] Figure 9 The diagram shows a schematic representation of the alignment architecture between acoustic and visual features in one embodiment of this application.

[0051] Figure 10 The diagram shown is a schematic diagram of a flow inference pipeline in one embodiment of this application;

[0052] Figure 11 The diagram shown is an architectural schematic of a cross-modal spatial localization engine in one embodiment of this application;

[0053] Figure 12 The diagram shows the architecture of a causal reasoning failure root cause engine in one embodiment of this application.

[0054] Figure 13 The diagram shown is a schematic block diagram of a data dashboard digital human assistant system according to an embodiment of this application;

[0055] Figure 14 This is a schematic diagram of an end-to-end data flow according to an embodiment of this application;

[0056] Figure 15 The diagram shown is a schematic diagram of the collaborative logic of the scene scheduler in one embodiment of this application;

[0057] Figure 16 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of this application. Detailed Implementation

[0058] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0059] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0060] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0061] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0062] Before providing a further detailed description of this application, the nouns and terms used in the embodiments of this application are explained, and the above nouns and terms are subject to the following interpretations:

[0063] <1> A digital human is a virtual image with human appearance and behavioral characteristics generated by computer graphics and artificial intelligence technologies. In this embodiment, the digital human is used to explain and broadcast data changes, abnormal events and fault information on a large data screen.

[0064] <2> A semantic token is a discrete token obtained by mapping continuous features to a pre-established token set through vector quantization. In this embodiment, several semantic tokens are arranged in sequence to form a semantic token sequence, which is used to represent abnormal events.

[0065] <3> A data dashboard refers to a large-format display device or interface used to centrally present data such as equipment operating status, project progress, and regional business distribution; in this embodiment of the application, the data dashboard is the display carrier that the digital human broadcasts to.

[0066] <4> Streaming data refers to a collection of data continuously output from each channel of a data source at sampling times and aligned to a unified time base. In this embodiment, the streaming data includes time-series sensor data streams, spatial distribution data of sensor devices, and topological relationship data between sensor devices. The time-series sensor data stream is the real-time data stream, which includes the measured values ​​output in real-time from each data source at the sampling time.

[0067] <5> A change feature is a higher-level parameter that characterizes the data change state of a data source. In this embodiment, the change feature includes the data change rate of each data source and the mutation events in the real-time data stream.

[0068] <6> The rate of change of data refers to the ratio of the change in value between the current time and the previous time to the time interval of the same data source; a mutation event refers to the event corresponding to the moment when the rate of change of data exceeds the adaptive threshold obtained from the historical data statistics of the data source.

[0069] <7> A temporal segment refers to several data segments obtained by dividing historical data into preset lengths and arranging them in chronological order; a temporal dependency representation refers to a representation that characterizes the temporal dependency relationship between the temporal segments, obtained by performing segment encoding and superimposing position encoding on each temporal segment, and then processing it through an encoder based on a self-attention mechanism.

[0070] <8> Anomaly criteria refer to the criteria used to determine whether the data from a data source deviates from its normal state; anomaly events refer to the data sources that are judged to be abnormal and the events that occur at the time of their occurrence, and the set of anomaly events is composed of all the aforementioned anomaly events.

[0071] <9> A probability density model is a model used to characterize the data distribution of a data source. In this embodiment, the probability density model is composed of several cascaded invertible transformations, used to map the data distribution of the data source to a baseline distribution, and to characterize the degree of deviation of the streaming data from the normal state by the log-likelihood of the streaming data under the baseline distribution.

[0072] <10> A dynamic topology graph refers to a graph structure with the sensor nodes corresponding to each of the data sources as nodes, the physical connection relationships and data correlation relationships between the sensor nodes as edges, and which is updated over time; a propagation path refers to the association path from the sensor node that has experienced an abnormal event to other sensor nodes, determined by the node embedding of each of the sensor nodes.

[0073] <11> The target event representation refers to the representation obtained after encoding the features of the abnormal event and the association relationship, which is used as a condition for text generation; in the embodiments of this application, the target event representation can be a semantic tag sequence.

[0074] <12> A causal chain refers to a chain determined by the propagation path indicated by the aforementioned relationship, which sequentially points from the upstream events that caused the anomaly to the anomaly itself; a prompt message refers to the input information constructed from the target event representation, the anomaly event, and the causal chain, which is used by the pre-trained language model to generate broadcast text.

[0075] <13> A broadcast queue refers to a sequence of events waiting to be broadcast, consisting of the events to be broadcast corresponding to the broadcast text; the broadcast order refers to the sequential order of the events to be broadcast in the broadcast queue; and the events to be broadcast refer to the events corresponding to the broadcast text that are waiting to be broadcast.

[0076] <14> Prosodic boundaries refer to the positions in the broadcast text where pauses or intonation changes need to be set; in this embodiment, the prosodic boundaries include strong boundaries, medium boundaries, and weak boundaries; audio stream refers to audio data obtained by speech synthesis of the broadcast text and arranged sequentially over time.

[0077] <15> The facial animation parameter sequence refers to the sequence of control parameters used to drive the facial movements of the digital human arranged in chronological order; the time offset refers to the amount of time misalignment between the audio stream and the facial animation parameter sequence.

[0078] <16> Frame-level synchronization refers to the process of aligning the audio stream and the facial animation of the digital human on the time axis using frames as the smallest unit. In this embodiment, frame-level synchronization is achieved by correcting the facial animation parameter sequence through the time offset.

[0079] <17> Beamforming scanning refers to the process of setting beamforming weights in different directions and calculating the output power in that direction, and taking the direction with the maximum output power as the sound source direction; first position estimation refers to the user's spatial position determined by the sound source direction and the distance from the sound source.

[0080] <18> The gaze direction vector is a unit vector representing the direction of the user's gaze; the gaze point is the position where the user's gaze intersects the plane of the data screen, determined by the user's position and the gaze direction vector; the target data area is the data area obtained by mapping the gaze point to the data screen according to the screen layout of the data screen.

[0081] <19> The world coordinate system refers to the coordinate system established with reference to the physical space where the data screen is located; the homogeneous transformation matrix refers to the matrix used to transform between the world coordinate system, the screen coordinate system of the data screen, and the user's user coordinate system.

[0082] <20> A cause-effect graph is a graph structure that represents the causal relationships between various monitored variables. Its nodes are the monitored variables, and the directed edges represent the causal directions. The monitored variables are the variables obtained by collecting data on the operating status of the monitored objects.

[0083] <21> The conditional independence test is the process of testing whether two variables are independent of each other given one or more other variables; the separating set is the set of conditions that make two variables independent in the conditional independence test.

[0084] <22> Transitive orientation rules refer to rules that infer the directions of the remaining edges based on the determined directed edges after the direction of some edges has been inferred; ancestor graphs refer to graphs containing some directed edges and bidirectional edges, used to characterize the causal relationships between monitored variables in the presence of unobserved confounding variables.

[0085] <23> Granger causality test is a test to determine whether the historical values ​​of one monitoring variable can significantly reduce the prediction error of another monitoring variable; transfer entropy is a measure of the amount of information that one monitoring variable transmits to another.

[0086] <24> The graph representation vector refers to the vector representation obtained after performing graph encoding on the causal graph, which is used as a condition for text generation; the root cause candidate set refers to the set of candidate root causes obtained after extracting the propagation path from each ancestor node to the node corresponding to the alarm event in the causal graph; graph pooling refers to the process of aggregating the representations of each node into the graph representation vector.

[0087] <25> Root cause score refers to the score used to rank each candidate node in the root cause candidate set; in the embodiments of this application, the root cause score increases as the probability of the candidate node being the root cause increases, increases as the severity of the anomaly of the candidate node increases, and decreases as the time interval between the current time and the time of occurrence of the alarm event increases.

[0088] <26> Scene mode refers to the category used to characterize the current usage scenario of the data dashboard; in this embodiment of the application, the scene mode includes visit mode, fault mode and daily mode; scene scheduler refers to the scheduling module that determines the activation status and priority of each module according to the scene mode.

[0089] <27> The correlation refers to the relationship between the abnormal events in terms of data transmission, physical connection, or trend of change; in the embodiments of this application, the correlation includes the propagation path between the abnormal events.

[0090] <28> The broadcast text refers to the text generated based on the target event representation and used to drive the digital human to perform voice broadcasting; in this embodiment of the application, the broadcast text is organized hierarchically according to the current fault phenomenon, fault propagation path, root cause and handling suggestions.

[0091] <29> The baseline distribution refers to the distribution obtained by mapping the original data distribution of the data source to the probability density model, which is used to infer the density of the original data; the log-likelihood refers to the logarithmic value of the likelihood function of the streaming data under the conditional distribution determined by the corresponding historical data, which is used to characterize the degree to which the data deviates from its normal state; the anomaly score is the sum of the negative value of the log-likelihood and the baseline value of the data source; the baseline value is the value used to eliminate differences between different data sources on the normal fluctuation scale.

[0092] <30> A memory vector is a vector maintained for each of the sensor nodes and updated with the messages received by that node; an inter-node message is information used to transmit between adjacent sensor nodes, determined based on the memory vector of the current node, the memory vectors of adjacent sensor nodes, and the connection relationships between adjacent sensor nodes; a node embedding is a vector representation of the corresponding sensor node and its connection relationships obtained by aggregating the inter-node messages.

[0093] <31> Cross-modal attention fusion refers to the process of generating a query from features of one modality and generating keys and values ​​from features of another modality and performing attention calculations, so that the features of different modalities are aligned with each other at the representation level; vector quantization refers to the process of mapping continuous fused features to corresponding labels in a pre-established label set, so that the features are passed downstream in the form of discrete labels.

[0094] <32> A pre-trained language model refers to a language model that has been pre-trained and is used to generate text based on input information; in this embodiment of the application, the prompt information is input into the pre-trained language model and then the broadcast text is output.

[0095] <33> The scoring model refers to a model used to determine the score of each of the events to be broadcast. The score increases with the importance level, decreases with the time interval, and increases with the urgency level. The importance level is a quantity that characterizes the importance of the content of the event to be broadcast, determined based on the degree of abnormality of the abnormal event corresponding to the event to be broadcast. The urgency level is a quantity that characterizes the urgency of the event to be broadcast, determined based on the severity of the abnormality corresponding to the event to be broadcast. The content representation refers to the vector representation of the content of the event to be broadcast, used as the value for the multi-head attention calculation.

[0096] The technical solutions of this application can be applied to places with large data screens, such as industrial command centers, project scheduling halls, and production monitoring centers. In particular, they can be applied to scenarios in the above-mentioned places where digital humans provide real-time explanations and broadcasts of data changes, abnormal events, and root causes of failures.

[0097] To facilitate understanding of the embodiments of this application, firstly, in conjunction with Figure 1 Detailed explanation. Figure 1 A flowchart illustrating a digital human broadcast control method according to an embodiment of this application is shown. The main flow of this method will be elaborated in steps S1 to S6 below. Before describing the specific steps, let's first combine... Figure 2 The system layered architecture applicable to the embodiments of this application is described in order to clarify the layer in which each step is located and the supporting relationship between each layer.

[0098] Figure 2 This paper illustrates the layered architecture of a data dashboard digital human assistant system according to an embodiment of this application, comprising four layers from top to bottom. The presentation layer, located at the top, includes four parts: dashboard visualization, digital human rendering, voice interaction, and spatial awareness. These are used to present data views on the dashboard, drive the digital human's image and actions, output voice, and receive the user's voice and location information, respectively. The core engine layer, located below the presentation layer, includes four engines: a frame-level synchronization engine, semantic compression and narrative generation, and cross-modal spatial localization and causal reasoning for fault root causes. Each engine is connected to a scene scheduler, meaning all four engines are uniformly coordinated by the scene scheduler. The data access layer includes a time-series database, message queue, log collection, video stream, and sensor gateway, used to access different types of data sources and provide streaming data to the upper layers. The infrastructure layer, located at the bottom, includes computing clusters, edge nodes, object storage, communication networks, and container orchestration, providing computing, storage, and communication support for the above layers.

[0099] In conjunction with the embodiments of this application, the four engines of the core engine layer correspond to different method chains of this application: the frame-level synchronization engine is used to perform processing such as determining changing feature quantities, determining prosodic boundaries, synthesizing audio streams, and converting and correcting facial animation parameter sequences; the semantic compression and narrative generation engine is used to perform processing such as anomaly detection, correlation determination, target event representation, and broadcast text generation; the cross-modal spatial localization engine is used to realize gaze-driven interactive localization; and the causal inference fault root cause engine is used to realize the fault root cause broadcast chain. The scene scheduler is used to determine the activation status and priority of each of the above engines.

[0100] Under the above system architecture, specifically, the method in this embodiment mainly includes the following steps:

[0101] Step S1: Obtain streaming data from the data source associated with the data dashboard. The streaming data includes real-time data streams collected in real time. Determine the change characteristics of the streaming data over time. The change characteristics represent the data change status of the data source.

[0102] In one implementation, the streaming data includes a time-series sensor data stream, spatial distribution data of sensor devices, and topological relationship data between sensor devices. These three data streams are aligned according to a unified time base and then processed together. Specifically, the time-series sensor data stream refers to a multi-channel time series composed of measurement values ​​continuously output by various sensor devices in the industrial field at sampling times; the spatial distribution data characterizes the installation location of each sensor device in the industrial field and the monitoring area it covers; and the topological relationship data characterizes the physical connection relationships and data correlation relationships between sensor devices.

[0103] Specifically, a data acquisition gateway is deployed in the industrial field. A programmable logic controller (PLC), a data acquisition and monitoring control system, a sensor interface, and a business database output device operation data, sensor readings, business data, and system logs, respectively. These are then distributed to subsequent stages via a message queue according to their topics. The time-series sensor data stream is cached in a sliding window format. The spatially distributed data is maintained in the form of a correspondence table between sensor device identifiers and installation location coordinates. The topological relationship data is maintained in the form of an adjacency list or adjacency matrix, distinguishing between physical connection edges determined by pipes, cables, and transmission links, and data correlation edges determined by the correlation of historical readings. All three are aligned according to a unified time base before being output to subsequent steps. The data acquisition can be implemented using a periodic polling method or a change-triggered subscription push method. The message queue can be implemented using a distributed publish-subscribe system or a streaming log system with persistent capabilities; this embodiment does not limit the implementation in this application.

[0104] It is worth noting that if only a single sensor reading is used as input for subsequent processing, the spatial attribution and propagation relationship of the abnormal event cannot be determined, and the broadcast content can only remain at the level of a certain numerically elevated value. This application acquires temporal, spatial, and topological data simultaneously, enabling subsequent anomaly scoring, propagation inference, and semantic compression to be performed on the same structured description. This transforms the abnormal event from an isolated numerical deviation into a narrative object with both locational attributes and correlations, avoiding the shortcomings of single-modal input that results in the broadcast content lacking spatial and causal orientation.

[0105] Step S2: Perform anomaly detection on the streaming data according to the preset anomaly criteria to obtain an anomaly event set, and determine the correlation between each anomaly event in the anomaly event set.

[0106] The anomaly criterion refers to the criteria used to determine whether the data from the data source deviates from its normal state; the abnormal event refers to the data source that is determined to be abnormal and the event consisting of the time of its occurrence; the correlation refers to the correlation between the abnormal events in terms of data transmission relationship, physical connection relationship or change trend.

[0107] Specifically, this step can pre-establish a probability density model for the data distribution of each data source. This probability density model consists of several cascaded invertible transformations, used to map the data distribution of the data source to a baseline distribution. The current streaming data is input into the probability density model to obtain the log-likelihood of the streaming data under a conditional distribution determined by the corresponding historical data. The sum of the negative value of the log-likelihood and the baseline value of the corresponding data source is used as the anomaly score. When the anomaly score satisfies the anomaly criterion, the corresponding streaming data is identified as an anomalous event, and the anomalous events constitute the anomaly event set. The anomaly criterion can also be implemented using data reconstruction error, the statistical distance between the data and its historical distribution, or the confidence level output by a trained discriminant model. When determining the association, a dynamic topology graph that updates over time can be constructed using the nodes corresponding to each data source as nodes and the physical connections and data relationships between nodes as edges. A memory vector updated with received messages is maintained for each node. Messages between nodes are determined based on the memory vectors of nodes, the memory vectors of their neighboring nodes, and the connections between neighboring nodes. After aggregating the messages between nodes, the node embedding of each node is obtained. Then, based on the node embedding of each node, the propagation path from the node where the abnormal event occurred to other nodes is determined, and this propagation path is used as the association. The association can also be determined by the correlation coefficient between data sources or by a pre-configured list of upstream and downstream data sources.

[0108] Step S3: Encode the features of the abnormal event and the correlation to obtain the target event representation of the abnormal event, and generate the broadcast text corresponding to the abnormal event using the target event representation as the generation condition.

[0109] The features of the abnormal event may include its temporal features, spatial features, and topological features; the target event representation refers to the representation obtained after encoding the features of the abnormal event and the association relationship, which is used as a condition for text generation.

[0110] Specifically, the features can be encoded separately: the temporal features are encoded by a first encoder to obtain a temporal feature vector, and the spatial features and topological features are encoded by a second encoder to obtain a graph structure feature vector; cross-modal attention fusion is performed using the temporal feature vector and the graph structure feature vector as input to obtain fused features; then vector quantization is performed on the fused features to discretize them into a semantic tag sequence composed of several semantic tags arranged in sequence, and the semantic tag sequence is used as the target event representation. Vector quantization refers to mapping continuous fused features to corresponding tags in a pre-established tag set, so that the features are passed downstream in the form of discrete tags. When generating the broadcast text, the causal chain corresponding to the abnormal event can be determined according to the correlation, and a prompt message can be constructed from the target event representation, the abnormal event, and the causal chain. The prompt message is input into a pre-trained language model to obtain the broadcast text organized hierarchically according to the current fault phenomenon, fault propagation path, root cause, and handling suggestions. The broadcast text can also be obtained by filling a pre-configured text template with the target event representation.

[0111] It is worth noting here that the features of anomalous events come from different modalities, and their value scales and statistical characteristics are different from each other. If they are directly concatenated and fed into the text generation stage, the parts with more information in each modality will dominate the generated results, while the parts with less information but necessary for causal expression will be submerged. Encoding them separately and then performing cross-modal attention fusion allows each modality to participate in the fusion with the same representation. Vector quantization further compresses the fused features into a limited number of discrete labels, so that the downstream text generation is conditioned on a limited sequence of labels, thereby reducing the input size of text generation while retaining the necessary information.

[0112] Step S4: Determine the broadcast order of each of the events to be broadcast in the broadcast queue consisting of the events to be broadcast corresponding to the broadcast text, according to the preset priority rules.

[0113] The event to be broadcast refers to the event corresponding to the broadcast text that is waiting to be broadcast; the priority rule refers to the rule used to determine the order of each event to be broadcast.

[0114] Specifically, this step can determine the priority of the events to be broadcast based on their importance, urgency, and the time interval between the occurrence time of the event and the current time. In one implementation, the importance is determined by the degree of anomalousness of the abnormal event corresponding to each event to be broadcast, the urgency is determined by the severity of the anomalous event corresponding to each event to be broadcast, and a score is determined for each event to be broadcast based on the importance, the time interval, and the urgency, wherein the score increases with increasing importance, decreases with increasing time interval, and increases with increasing urgency; then, using the current broadcasting capability as the query, the characteristics of each event to be broadcast as the key, and the content representation of each event to be broadcast as the value, multi-head attention calculation is performed on the scores of each event to be broadcast, and the priority is obtained by fusing the outputs of each attention head, and the broadcasting order of each event to be broadcast in the broadcasting queue is determined according to the priority from high to low. In another implementation, the broadcasting order can be determined directly according to a preset priority sorting table or a first-in-first-out method.

[0115] Step S5: Determine the prosodic boundary of the broadcast text based on the change feature quantity, and synthesize the broadcast text into an audio stream according to the prosodic boundary.

[0116] The prosodic boundary refers to the position in the broadcast text where a pause or intonation change needs to be set; the audio stream refers to the audio data obtained by speech synthesis of the broadcast text and arranged in chronological order.

[0117] Specifically, this step can first convert the broadcast text into a phoneme sequence and obtain the fundamental frequency, energy, and duration parameters corresponding to the phoneme sequence; then, the prosodic boundaries are determined based on the change feature quantities: in one implementation, the position in the broadcast text corresponding to the abrupt change indicated by the data change feature quantity is determined as a strong boundary, which corresponds to pauses and intonation changes; the position where the change trend of the change feature quantity in the broadcast text reverses is determined as a middle boundary, which corresponds to a pause degree less than that of the strong boundary; the positions in the broadcast text other than the strong boundary and the middle boundary are determined as weak boundaries, which do not correspond to pauses. The prosodic boundaries can also be divided into two or more levels, with the pause duration and intonation change amplitude corresponding to each level decreasing sequentially. During synthesis, the broadcast text can be divided into several text blocks according to the prosodic boundaries, and streaming speech synthesis can be performed on each text block sequentially, outputting audio data in blocks, so that the audio data can be output after the first block is synthesized, without waiting for the entire text to be synthesized.

[0118] It is worth noting that existing speech synthesis technologies typically determine pauses based on punctuation marks within the text itself. However, industrial data broadcast text is generated data-driven, and while its content is complete, the pauses indicated by punctuation do not correspond to the moments of change in the data itself, resulting in key data not being emphasized in speech. This step determines prosodic boundaries based on the data's change features, essentially mapping the moments of data change to pauses and intonation changes in speech, ensuring that the auditory emphasis aligns with the locations of data changes. Furthermore, segmenting and streaming the speech according to prosodic boundaries eliminates the need for the entire text to be synthesized before audio output, shortening the waiting time from data change to speech output.

[0119] Step S6: Convert the audio stream into a facial animation parameter sequence of the digital human, determine the time offset between the audio stream and the facial animation parameter sequence, perform correction on the facial animation parameter sequence according to the time offset, and drive the rendering output of the digital human according to the corrected facial animation parameter sequence and the audio stream.

[0120] The facial animation parameter sequence refers to the sequence of control parameters used to drive the facial movements of the digital human arranged in chronological order; the time offset refers to the amount of time misalignment between the audio stream and the facial animation parameter sequence.

[0121] Specifically, the audio features of the audio stream can be used as input to obtain an audio embedding sequence through convolutional encoding, and the facial animation parameter sequence can be used as input to obtain a visual embedding sequence through convolutional encoding. A cross-correlation operation is performed on the audio embedding sequence and the visual embedding sequence, and the lag value that maximizes the cross-correlation is determined as the time offset. The values ​​of corresponding frames in the facial animation parameter sequence are shifted forward or backward according to the time offset, and interpolation smoothing is performed on several adjacent frames in the corrected facial animation parameter sequence. It should be understood that cross-correlation is a sequence matching algorithm used to measure the similarity of two sequences at different relative time shifts. It obtains the cross-correlation result by progressively shifting one sequence forward or backward relative to another and calculating the correlation value corresponding to each lag value. The lag value corresponding to the maximum cross-correlation value represents the time offset between the audio embedding sequence and the visual embedding sequence.

[0122] In one implementation, the time of each frame in the facial animation parameter sequence is the sum of the time of the corresponding phoneme in the audio stream and the time offset. The time of each phoneme in the audio stream is the sum of the change time indicated by the data change feature and the prediction advance, thus establishing a sequential temporal relationship between the data change time, the phoneme time, and the facial animation frame time. In another implementation, an initial facial animation parameter sequence can be obtained by mapping from the audio stream, and then the initial sequence can be corrected and smoothed according to the time offset.

[0123] It's worth noting that facial animation parameters are typically generated by audio, but audio synthesis and facial animation generation are often performed separately, resulting in a misalignment in their frame correspondence. If frame numbers are directly concatenated, a noticeable asynchrony will occur between the pronunciation in the audio and the facial movements, especially noticeable when data jumps frequently or the broadcast content switches frequently. This step calculates and corrects the misalignment by performing cross-correlation operations at the frame data level, ensuring that the corrected facial animation parameter sequence corresponds to the audio stream at the frame level. Simultaneously, it sequentially connects the facial animation frame moments, phoneme moments, and data change moments, thus maintaining temporal consistency among data changes, speech, and facial expressions.

[0124] Steps S1 to S6 above provide a complete flowchart of the method according to the embodiments of this application. To facilitate a better understanding by those skilled in the art of digital human broadcasting control method provided by the embodiments of this application, the following description is in conjunction with… Figure 3 The data flow architecture diagram shown will be further explained.

[0125] Figure 3 The top-middle row, from left to right, consists of industrial data sensors, a data access layer (time series, logs, video), a core engine layer (four subsystems working together), and a presentation layer (large screen, digital human). Data is passed layer by layer from left to right. The data access layer connects downwards to edge preprocessing (anomaly detection, semantic compression), the core engine layer connects downwards to the scene scheduler (mode switching, subsystem collaboration), and the presentation layer connects downwards to user interaction (voice, gaze). The processing of these three data links includes: real-time data stream: "sensor data → time series prediction → prosodic coding → speech synthesis → digital human facial animation"; interactive data stream: "user voice and gaze → spatial positioning → target area → posture and content matching"; and fault data stream: "alarm logs and monitoring data → causal reasoning → propagation path → root cause broadcasting".

[0126] In conjunction with the embodiments of this application, the processing corresponding to the real-time data stream is steps S1 to S6 of the above method embodiments: determining the changing feature quantity from sensor data and determining the prosodic boundary accordingly, ultimately converting it into a facial animation parameter sequence and driving the digital human output; the processing corresponding to the interactive data stream is the method described in the following gaze-driven interactive positioning implementation method, determining the target data area from the user's voice and gaze, and then matching the broadcast content with the digital human's posture; the processing corresponding to the fault data stream is the method described in the following fault root cause broadcasting link implementation method, obtaining the propagation path from alarm logs and monitoring data through causal reasoning and generating root cause broadcasting. Furthermore, Figure 3 The edge preprocessing shown in the figure indicates that anomaly detection and semantic compression can be completed at the edge node side, thus allowing filtering and compression to be completed without uploading all the original data.

[0127] Furthermore, under the aforementioned three data links, the core subsystems are not independent of each other, but rather share intermediate quantities and serve as inputs and outputs to each other, with their interaction relationships as follows: Figure 4 As shown.

[0128] Figure 4 This diagram illustrates the interaction relationships between the core subsystems in this embodiment. At the top of the diagram is the scene scheduler, which connects downwards to the frame-level synchronization engine, the semantic compression and narrative generation engine, and the cross-modal spatial positioning engine. The frame-level synchronization engine outputs "change information" to the semantic compression and narrative generation engine; the semantic compression and narrative generation engine outputs "abnormal events" to the causal inference engine; and the cross-modal spatial positioning engine also outputs "spatial location" to the causal inference engine. The processing of each engine is essentially as follows: the frame-level synchronization engine processes "data to animation," the semantic compression and narrative generation engine processes "data to narrative," the cross-modal spatial positioning engine processes "space to interaction," and the causal inference engine processes "fault to root cause."

[0129] In conjunction with the embodiments of this application: First, the frame-level synchronization engine transmits "change information" to the semantic compression and narrative generation engine, indicating that the change feature quantity is not only used to determine the prosodic boundary, but also transmitted downstream as the input feature for anomaly detection. That is, the change feature quantity determined in step S1 of the above method embodiment is reused in step S2. Second, the semantic compression and narrative generation engine transmits "abnormal events" to the causal inference engine, indicating that abnormal events are the conditions that trigger root cause inference. That is, the processing in the above method embodiment that determines the root cause candidate set in the causal graph in response to alarm events has a clear upstream source. In addition, the cross-modal spatial positioning engine transmits "spatial location" to the causal inference engine, indicating that the area viewed by the user will also constrain the selection range of the root cause broadcast content. The above interaction relationship shows that the subsystems are not independent of each other, but share intermediate quantities and are inputs and outputs to each other.

[0130] The above combination Figures 1 to 4The overall flow of the method in the embodiments of this application and the system architecture on which it is based have been described. Based on this, some preferred embodiments of each step in the above method are further described below.

[0131] Further, regarding the determination of the change features in step S1 above, the change features include the data change rate of each of the data sources and the mutation events in the real-time data stream; determining the change features of the streaming data over time includes: dividing the historical data corresponding to the streaming data into several time-series segments according to a preset length, performing segment encoding and superimposing position encoding on each of the time-series segments, inputting the data into an encoder based on a self-attention mechanism to obtain a time-series dependency representation characterizing the time-series dependency relationship between the time-series segments, and performing regression processing on the time-series dependency representation to obtain the data change rate of each of the data sources at future times; comparing the data change rate of each of the data sources with an adaptive threshold obtained from the historical data of that data source, and determining the moment when the data change rate exceeds the adaptive threshold as the mutation event.

[0132] Specifically, this implementation specifies the change characteristics as two items: data change rate and abrupt change events. The data change rate characterizes the speed of data change, while the abrupt change event characterizes the moment when significant data jumps. When determining the data change rate, historical data is divided into several time-series segments according to a preset length. Each time-series segment is encoded to obtain a segment representation, and a positional encoding is superimposed to preserve the temporal order of each segment. The segment representations with positional encodings are input into an encoder based on a self-attention mechanism. The encoder outputs a temporal dependency representation characterizing the temporal dependencies between the time-series segments. Regression processing is then performed on the temporal dependency representation to obtain the data change rate at future moments. The adaptive threshold is obtained statistically from the historical data of the data source. For example, it can be the sum of the mean and standard deviation of the historical data change rate of the data source, or it can be the value of the historical data change rate at a preset quantile, thus allowing data from different data sources and with different dimensions to determine abrupt changes at their respective scales.

[0133] Therefore, a combination of temporal segment encoding and self-attention encoding is used to predict the data change rate, enabling the determination of the change rate to utilize temporal dependencies over a longer time range; and an adaptive threshold obtained from historical data statistics is used to determine abrupt events, making the abrupt event determination adaptable to the fluctuation scale of the data source itself. The above-mentioned process of predicting the data change rate is referred to as temporal prediction in this embodiment, and the unit performing this process is called a temporal predictor. Its output data change rate and abrupt events are transmitted downstream as change feature quantities.

[0134] Furthermore, regarding the determination of prosodic boundaries in step S5 above, the prosodic boundaries include strong boundaries, medium boundaries, and weak boundaries; determining the prosodic boundaries of the broadcast text based on the change feature quantity includes: determining the position in the broadcast text corresponding to the abrupt event as the strong boundary, the strong boundary corresponding to pauses and intonation changes; determining the position in the broadcast text where the trend of the data change rate changes reverses as the medium boundary, the medium boundary corresponding to a pause less severe than the strong boundary; and determining the positions in the broadcast text other than the strong boundary and the medium boundary as the weak boundary, the weak boundary not corresponding to pauses.

[0135] Specifically, this implementation divides prosodic boundaries into three levels: strong boundaries, medium boundaries, and weak boundaries. The broadcast text is generated from the target event representation, and there is a correspondence between the statements, data sources, and data items. Therefore, it is possible to determine the position of each statement in the text and its corresponding data source. The positions of statements corresponding to abrupt events are determined as strong boundaries, where a pause is set and an intonation change is applied. The positions where the trend of data change rate changes is reversed are determined as medium boundaries, where a pause with a less pronounced degree than that of strong boundaries is set. The remaining positions are determined as weak boundaries, where no pause is set. The pause duration and intonation change amplitude corresponding to each level of boundary decrease sequentially, so that the fluctuation of speech and the significance of data change correspond hierarchically. Thus, by setting prosodic boundaries in a hierarchical manner, the degree of emphasis in speech corresponds hierarchically to the significance of data change, and listeners can identify three types of situations—data abrupt changes, trend reversals, and smooth transitions—from the changes in pauses and intonation.

[0136] Further, regarding the anomaly detection in step S2 above, anomaly detection is performed on the streaming data according to a preset anomaly criterion to obtain an anomaly event set, including: a preset probability density model for each data source, the probability density model being composed of several cascaded invertible transformations, used to map the data distribution of the data source to a baseline distribution; inputting the streaming data at the current moment into the probability density model to obtain the log-likelihood of the streaming data under a conditional distribution determined by the corresponding historical data; using the sum of the negative value of the log-likelihood and the baseline value of the data source as an anomaly score, and determining the streaming data whose anomaly score satisfies the anomaly criterion as the anomaly event, and the anomaly event set being composed of each of the anomaly events.

[0137] For example, the probability density model can be implemented using a normalizing flow model, such as Real-valued Non-Volume Preserving (RealNVP), Masked Autoregressive Flow (MAF), Inverse Autoregressive Flow (IAF), Non-linear Independent Components Estimation (NICE), Glow, Neural Spline Flow (NSF), Continuous Normalizing Flow (CNF), or Conditional Normalizing Flow. These models map the original data distribution of the data source to a preset baseline distribution through one or more reversible transformations, and determine the probability density or log-likelihood corresponding to the original data based on the variable transformation relationship. This embodiment does not specifically limit the specific network structure, reversible transformation form, or baseline distribution type of the normalizing flow model.

[0138] More specifically, this implementation uses a probability density model for anomaly detection. A probability density model is established for each data source, consisting of several cascaded invertible transformations. This cascade maps the original data distribution of the data source to a baseline distribution, allowing the density of the original data under historical distributions to be inferred from the density values ​​under the baseline distribution. During operation, the current streaming data is input into the model to obtain the log-likelihood of the data under a conditional distribution determined by the corresponding historical data. The sum of the negative value of the log-likelihood and the baseline value of the data source is used as the anomaly score; a larger anomaly score indicates that the data deviates more from its historical distribution. This anomaly score is compared with the anomaly criteria; data that meets the criteria is identified as an anomaly event. The baseline value is used to eliminate differences in normal fluctuation scales between different data sources, allowing anomaly scores from different data sources to be compared under the same criteria.

[0139] Therefore, the probability density model established by the reversible transformation cascade can complete the modeling without abnormal samples, and the log-likelihood provides a quantified degree of deviation, making the anomaly judgment have a comparable scale, which is suitable for industrial data scenarios that lack labeled abnormal samples.

[0140] Furthermore, regarding the determination of the association in step S2 above, the association includes the propagation path between the abnormal events; determining the association between each abnormal event in the abnormal event set includes: constructing a dynamic topology graph that updates over time, using the sensor nodes corresponding to each data source as nodes and the physical connection relationships and data correlation relationships between each sensor node as edges; maintaining a memory vector for each sensor node that is updated with the received messages, and determining the inter-node messages based on the memory vector, the memory vectors of adjacent sensor nodes, and the connection relationships between adjacent sensor nodes, and aggregating the inter-node messages to obtain the node embedding of each sensor node; and determining the propagation path from the sensor node where the abnormal event occurred to other sensor nodes based on the node embedding of each sensor node.

[0141] Specifically, this implementation uses a dynamic topology graph to determine the propagation path between abnormal events. The nodes of the dynamic topology graph are the sensor nodes corresponding to each data source, and the edges consist of two types of relationships: one is the objectively existing physical connection relationship between sensor nodes, such as the connection between upstream and downstream equipment in the same production line, or the connection between power supply or material supply; the other is the data correlation relationship between sensor nodes, such as the correlation between the readings of two sensor nodes in historical data. The dynamic topology graph is updated constantly, that is, its edges and their weights are updated as data is collected, to adapt to situations such as equipment maintenance and topology changes. A memory vector is maintained for each sensor node, which is updated with the messages received by the node. When determining messages between nodes, the memory vector of the current node, the memory vectors of adjacent nodes, and the connection relationship between them are comprehensively considered. After aggregating the messages between nodes, the node embedding of each sensor node is obtained, and then the propagation path from the sensor node where the abnormal event occurred to other sensor nodes is determined based on the node embedding. The resulting propagation path indicates the order of transmission of the abnormality between data sources and is used as the correlation relationship to propagate downstream.

[0142] Therefore, by using a dynamically updated topology map to carry the connections between data sources, the propagation path can reflect not only the objectively connected device relationships, but also the statistical relationships related to the data, and can be updated as the device topology changes, providing causal basis for the broadcast text generated thereafter.

[0143] Further, regarding the generation of the target event representation in step S3 above, the features of the anomalous event include temporal features, spatial features, and topological features; encoding the features of the anomalous event and the correlation to obtain the target event representation of the anomalous event includes: encoding the temporal features by a first encoder to obtain a temporal feature vector, and encoding the spatial features and the topological features by a second encoder to obtain a graph structure feature vector; performing cross-modal attention fusion with the temporal feature vector and the graph structure feature vector as input to obtain a fused feature; performing vector quantization on the fused feature to discretize the fused feature into a semantic tag sequence composed of several semantic tags arranged in sequence, and using the semantic tag sequence as the target event representation.

[0144] Specifically, in this embodiment, the features involved in encoding include three categories: temporal features, spatial features, and topological features. Temporal features represent the value changes of an abnormal event over time; spatial features represent the location of the data source where the abnormal event occurred; and topological features represent the location of the data source in the dynamic topology graph and its connectivity. The first encoder encodes the temporal features to obtain a temporal feature vector, and the second encoder encodes the spatial and topological features to obtain a graph structure feature vector. Cross-modal attention fusion is then performed using both as inputs. Specifically, a query is generated from features of one modality, and keys and values ​​are generated from features of the other modality, with attention calculations performed to align the different modalities at the representation level, resulting in fused features. Vector quantization is then performed on the fused features, discretizing them into a sequence of semantic tags arranged sequentially. This sequence of semantic tags serves as the representation of the target event. Vector quantization maps continuous fused features to a finite set of tags, thus constraining the length and content of the target event representation within a predictable range.

[0145] For example, the first encoder can be a Long Short-Term Memory (LSTM) network, a Gated Recurrent Unit (GRU), a Temporal Convolutional Network (TCN), or a Transformer encoder to extract temporal correlation features of anomalous events over time. The second encoder can be a Graph Convolutional Network (GCN), a Graph Attention Network (GAT), a GraphSAGE network, or a Message Passing Neural Network (MPNN) to encode spatial and topological features based on the relationships between industrial objects.

[0146] Thus, cross-modal attention fusion aligns temporal, spatial, and topological features at the representation level, and vector quantization compresses the fusion result into a discrete label sequence, reducing the input size of subsequent text generation while preserving necessary information about anomalous events.

[0147] The generation process of the above target event representation can be combined with Figure 5 To gain a further understanding. Figure 5 This paper illustrates the multimodal semantic tag compression process according to an embodiment of this application. Temporal data, spatial data, and topological data are each encoded with independent features, that is, each is encoded by its own encoder to obtain feature vectors, without directly concatenating the three types of data; then cross-modal fusion is performed to align the three types of features at the representation level; then vector quantization mapping is performed to map the fused continuous features to a finite set of tags to obtain a semantic tag sequence, and this semantic tag sequence is used as the representation of the target event.

[0148] Furthermore, regarding the generation of the broadcast text in step S3 above, the generation of the broadcast text corresponding to the abnormal event is based on the target event representation as the generation condition, including: determining the causal chain corresponding to the abnormal event according to the correlation, and constructing prompt information from the target event representation, the abnormal event and the causal chain; inputting the prompt information into a pre-trained language model to obtain the broadcast text organized hierarchically according to the current fault phenomenon, fault propagation path, root cause and handling suggestions.

[0149] Specifically, in this embodiment, the causal chain corresponding to the abnormal event is determined based on the propagation path indicated by the association relationship. The causal chain points sequentially from the upstream event that caused the abnormality to the abnormality itself. A prompt message is constructed using the target event representation, the abnormal event itself, and the causal chain. This prompt message sequentially includes the content of the abnormal event, the position of the abnormal event in the association structure, and the causal chain determined by the association relationship. The prompt message is input into a pre-trained language model, which outputs a broadcast text. This broadcast text is organized hierarchically according to the current fault phenomenon, the fault propagation path, the root cause, and handling suggestions. The broadcast text can also be obtained by filling in a pre-configured text template with the target event representation, i.e., filling the corresponding slots with semantic tags from the target event representation according to a fixed sentence structure. This is suitable for scenarios with high real-time requirements for text generation but low requirements for sentence diversity. Therefore, because the broadcast text is organized according to the causal chain, the generated text provides the propagation path and root cause while describing the phenomenon, giving the broadcast content causal information for further handling, rather than just a restatement of the phenomenon.

[0150] The process of generating the above broadcast text can be combined with Figure 6To further understand, the semantic tag sequence represents the target event, the abnormal event description is the abnormal event itself, the fault causal chain is obtained from the causal chain determined by the correlation, and the broadcast context represents the scene information in which this broadcast takes place; these four are combined and used as input to perform natural language decoding to obtain the narrative text; the output narrative text is organized hierarchically according to the fault phenomenon, propagation path, root cause, and handling information.

[0151] Further, regarding the determination of the broadcast order in step S4 above, the priority rule is the score output by the scoring model; determining the broadcast order of each of the events to be broadcast in the broadcast queue according to the preset priority rule includes: determining the importance based on the degree of abnormality of the abnormal event corresponding to each of the events to be broadcast, determining the urgency based on the severity of the abnormality corresponding to each of the events to be broadcast, and determining the score of each of the events to be broadcast according to the importance, the time interval between the occurrence time of the event to be broadcast and the current time, and the urgency; wherein, the score increases with the increase of the importance, decreases with the increase of the time interval, and increases with the increase of the urgency; performing multi-head attention calculation on the score of each event to be broadcast using the current broadcast capability as the query, the features of each event to be broadcast as the key, and the content representation of each event to be broadcast as the value; obtaining the priority by fusing the outputs of each attention head, and determining the broadcast order of each of the events to be broadcast in the broadcast queue from high to low according to the priority.

[0152] Specifically, in this embodiment, importance is determined by the degree of anomalousness of the corresponding abnormal event, and urgency is determined by the severity of the anomaly. A score is then calculated by combining the time interval between the occurrence of the event and the current time: the score increases with importance, decreases with the time interval, and increases with urgency. Further, multi-head attention calculation is performed: using the current broadcasting capabilities (e.g., currently available broadcasting duration and channels) as the query, the characteristics of each event as the key, and the content representation of each event as the value, multiple attention heads calculate the attention weight of each event. The outputs of each attention head are then merged to obtain a priority, and the broadcasting order is determined from highest to lowest priority. The advantage of using multi-head attention instead of a single score ranking is that different attention heads can focus on different aspects such as timeliness, importance, and the suitability of the broadcast content, ensuring that the final order takes multiple factors into account simultaneously. Therefore, the broadcasting order is determined by combining scoring with multi-head attention, prioritizing the broadcasting of important, urgent, and recent events, and matching the broadcasting order with the currently available broadcasting capabilities.

[0153] Thus, the above-mentioned anomaly detection, correlation determination, target event representation generation, broadcast text generation, and broadcast order determination constitute a complete processing chain. The engine architecture of this chain is as follows: Figure 7 As shown.

[0154] Figure 7 This diagram illustrates the architecture of the semantic compression and narrative generation engine in an embodiment of this application, comprising five processing layers from top to bottom. The first layer is information entropy anomaly detection, which, from left to right, includes time-series data streams, probability density modeling, and anomaly event sets. The second layer is dynamic topology modeling, which, from left to right, includes sensor topology maps, dynamic graph encoding, propagation paths, and impact ranges. The third layer is multimodal semantic compression, which, from left to right, includes time-series spatial topological features, cross-modal fusion, and semantic tag sequences. The fourth layer is causal narrative generation, which, from left to right, includes semantic tags and causal chains, natural language decoding, and candidate narrative content. The fifth layer is broadcast queue arbitration, which, from left to right, includes importance information, timeliness and urgency fusion, and target broadcast content. The layers are interconnected from top to bottom. Therefore, it can be seen that… Figure 7 It covers the complete processing chain from anomaly detection to broadcast order determination, that is, the unfolding process of steps S2 to S4 in the above method embodiment.

[0155] Further, regarding the correction of the facial animation parameter sequence in step S6 above, determining the time offset between the audio stream and the facial animation parameter sequence, and performing correction on the facial animation parameter sequence according to the time offset, includes: performing convolutional coding on the audio stream and the facial animation parameter sequence respectively to obtain an audio embedding sequence and a visual embedding sequence; performing cross-correlation operation on the audio embedding sequence and the visual embedding sequence, and determining the lag that maximizes the cross-correlation value as the time offset; shifting the values ​​of corresponding frames in the facial animation parameter sequence forward or backward according to the time offset, and performing interpolation smoothing on several adjacent frames in the corrected facial animation parameter sequence; wherein, the time of each frame in the facial animation parameter sequence is the sum of the time of the corresponding phoneme in the audio stream and the time offset, and the time of each phoneme in the audio stream is the sum of the time of the mutation event and the advance corresponding to the timing prediction.

[0156] Specifically, in this embodiment, the audio stream and the facial animation parameter sequence are encoded through concatenated one-dimensional convolutions to obtain an audio embedding sequence and a visual embedding sequence in the same embedding space. Cross-correlation is performed on the two embedding sequences, i.e., the correlation between them is calculated for different time lags, and the lag that maximizes the cross-correlation value is determined as the time offset. The values ​​of corresponding frames in the facial animation parameter sequence are shifted forward or backward according to this time offset to align the visual sequence with the audio sequence in time. Since forward and backward shifts cause value jumps between frames, interpolation smoothing is performed on several adjacent frames in the corrected sequence to avoid facial jitter. In this embodiment, the three moments form a sequential temporal relationship: the moment of the data mutation event plus the advance corresponding to the temporal prediction yields the moment of the corresponding phoneme in the audio stream; the moment of the phoneme plus the time offset yields the moment of the corresponding frame in the facial animation parameter sequence. Thus, the moment of data change, the moment of speech pronunciation, and the moment of facial movement are incorporated into the same time chain. Therefore, the misalignment between audio and facial animation is calculated using cross-correlation operations, so that the alignment of the two does not depend on manual calibration; and the jitter caused by correction is avoided by interpolation smoothing; the timing connection of data change time, phoneme time and facial animation frame time makes the expression switching and data change have a definite sequential relationship.

[0157] The determination of the aforementioned variable features, the determination of prosodic boundaries, and the conversion and correction of facial animation parameter sequences are all uniformly completed by a frame-level synchronization engine in this embodiment. The engine is composed as follows: Figure 8 As shown. Figure 8 This diagram illustrates the composition of the frame-level synchronization engine according to an embodiment of this application. From left to right, the upper part of the diagram shows a timing predictor, a prosodic encoder, and a cross-modal alignment module, all connected sequentially. The timing predictor outputs the rate of change and abrupt change points; the prosodic encoder outputs the fundamental frequency, energy, and persistence information; and the cross-modal alignment module outputs facial deformation weights. The middle part of the diagram shows the frame synchronization controller, which internally includes offset calculation, animation timeline fine-tuning, and inter-frame interpolation smoothing, with the output from the cross-modal alignment module fed into the frame synchronization controller. The lower part of the diagram shows the streaming inference pipeline, which internally includes text generation, streaming speech synthesis, facial animation generation, and large-screen rendering output.

[0158] It should be understood that the timing predictor corresponds to the processing of determining the changing feature quantity in step S1, and its output change rate and abrupt change point are the data change rate and abrupt change event; the prosody encoder corresponds to the processing of determining the prosodic boundary in step S5, that is, determining prosodic parameters such as fundamental frequency, energy and duration information from the changing feature quantity; the cross-modal alignment module corresponds to the processing of converting the audio stream into a facial animation parameter sequence in step S6; the offset calculation, animation time axis fine-tuning and inter-frame interpolation smoothing in the frame synchronization controller correspond to the cross-correlation operation to obtain the time offset, the value to be shifted forward or backward according to the time offset, and the interpolation smoothing as described above; the streaming inference pipeline means that text generation, speech synthesis, facial animation generation and rendering output can be connected sequentially in a streaming manner, without having to wait for all text processing to be completed.

[0159] Preferably, Figure 8 A specific implementation of the mid-to-span modal alignment module is as follows: Figure 9 As shown. Figure 9 This diagram illustrates the alignment architecture of acoustic and visual features in an embodiment of this application. The lower left and right sides of the diagram represent two inputs: audio prosodic features and facial deformation features. The audio prosodic features are fed into an audio prosodic encoder, which outputs fundamental frequency, energy, persistence information, and speech rate variations. The facial deformation features are fed into a visual parameter encoder, which outputs parameters related to the mouth, eyes, eyebrows, and others. The outputs of both encoders are fed upwards into a shared embedding space. The label "contrastive learning and mapping" between the two encoders indicates that they are mapped to the same space through contrastive learning during the training phase.

[0160] For example, the contrastive learning loss function is as follows:

[0161] ;

[0162] in, represents the contrastive learning loss function, used to measure the degree of matching between audio features and corresponding visual features; the smaller the value, the higher the similarity of the matching audio-visual feature pairs and the more obvious the distinction between them and the non-matching feature pairs; N represents the total number of audio-visual sample pairs participating in this loss calculation, which is usually the number of sample pairs in a training batch; i represents the sample index, used to indicate the i-th audio-visual sample pair; j represents the candidate visual sample index, used to traverse each visual feature compared with the i-th audio prosodic feature vector for similarity comparison; Indicates the first Each audio prosodic feature vector can be used to characterize prosodic information such as speech rate, pitch, energy, rhythm, or pauses in audio. Indicates the first individual vision Eigenvectors, i.e., with The visual feature vectors that constitute positive sample pairs are used to characterize facial morphology, mouth shape, or expression state at the corresponding time or in the corresponding segment. This represents the feature vector of the j-th candidate visual BlendShape, when j=i. Usually with Matching positive samples; when j ≠ i, Usually as with Mismatched negative samples; This represents the similarity between the i-th audio prosodic feature vector and its corresponding visual BlendShape feature vector; This represents the similarity between the i-th audio prosody feature vector and the j-th candidate visual BlendShape feature vector; This represents the temperature parameter.

[0163] Furthermore, after training, the mapping from audio prosody features to BlendShape weights is as follows.

[0164] ;

[0165] in, This represents the predicted BlendShape weights. BlendShape weights are used to characterize the contribution of different facial shape primitives to the target facial shape and can be used to drive the mouth shape, expression and other facial movements of digital humans. This refers to the audio features extracted from the input audio, which may include the audio's spectral features, prosodic features, phoneme features, or contextual semantic features, and are used as input to the audio encoder. This refers to an audio encoder, which encodes the input audio features to extract high-level audio representations related to facial movements. This indicates a nested execution process, where audio features are first encoded using an audio encoder, and then the encoded result is decoded using a visual decoder to obtain the corresponding BlendShape weights. It should be noted that BlendShape refers to a pre-made set of standardized facial deformation primitives. These primitives are linearly superimposed with different weights to synthesize the target facial shape, and their weights are often used as driving parameters for digital human facial movements.

[0166] In conjunction with the embodiments of this application, Figure 9To illustrate a specific implementation of step S6, "converting the audio stream into a facial animation parameter sequence for the digital human," in the above method embodiments: the features on the audio side and the visual side are not directly concatenated, but are encoded separately and mapped to a shared embedding space, so that the features on both sides correspond to each other in this space, thereby allowing the visual parameters to be obtained from the audio side features. The fundamental frequency, energy, duration information, and speech rate changes output by the audio prosody encoder are connected to the processing of synthesizing the audio stream according to prosodic boundaries in step S5 of the above method embodiments; the mouth, eyes, eyebrows, and other parameters output by the visual parameter encoder represent the specific parameter categories included in the facial animation parameter sequence. The advantage of using a shared embedding space instead of parameter-by-parameter mapping is that the same audio segment can correspond to multiple combinations of lip shapes and expressions, and mapping in the shared space can preserve this many-to-many correspondence.

[0167] Based on the above processing steps, the text generation, speech synthesis, facial animation generation, and rendering output can be sequentially connected in a streaming manner, with the inference pipeline as follows: Figure 10 As shown. Figure 10 The diagram illustrates the streaming inference pipeline of an embodiment of this application. From left to right, the diagram shows four stages: data arrival event triggering, narrative text generation, streaming speech synthesis, and facial animation generation and large-screen rendering; from "facial animation generation and large-screen rendering" it connects downwards to "continuously transmitting intermediate results by data fragment and outputting synchronized broadcast".

[0168] In conjunction with the embodiments of this application, Figure 10 To illustrate the connection between the various processing stages of this application: each stage does not wait for the upstream to complete all processing before starting, but rather continuously transmits intermediate results according to data segments. That is, the current data segment can be output after narrative text generation, speech synthesis, and facial animation generation, while subsequent data segments are still being processed. As a result, the waiting time from the arrival of data to the synchronous broadcast output of the digital human is shortened. Correspondingly, in the above method embodiment, step S5, which divides the broadcast text into several text blocks according to prosodic boundaries, performs streaming speech synthesis on each text block in sequence, and outputs audio data in blocks, is the specific manifestation of this pipeline in the speech synthesis stage.

[0169] The above describes the broadcast link and its frame-level synchronization processing according to the embodiments of this application. In addition, the embodiments of this application also provide an interactive positioning implementation method that matches the user's gaze area, so that the broadcast content corresponds to the data area currently being viewed by the user:

[0170] Furthermore, the method further includes: acquiring multi-channel audio and video streams within the physical space where the data screen is located; performing beamforming scanning on the multi-channel audio streams, determining the sound source direction based on the output power in each scanning direction, and determining the sound source distance by combining the arrival time difference between each channel, thereby obtaining a first position estimate of the user; detecting a face region from the video stream, and estimating the user's gaze direction vector based on the face region; mapping the first position estimate, the user's head pose determined by the face region, and the gaze direction vector to the same world coordinate system, and performing filtering prediction on the mapping result. The system obtains the position of the user's gaze point in the world coordinate system; maps the gaze point to a target data area on the data screen according to the screen layout of the data screen; determines the digital human's posture parameters and gestures based on the target data area and the user's position, and highlights the target data area; wherein, the target event representation is also generated based on the target data area, so that the generated broadcast text corresponds to the target data area; and the broadcast order of each event to be broadcast in the broadcast queue is also determined according to the change of the target data area relative to the previous moment.

[0171] Specifically, this embodiment deploys a microphone array and an image acquisition device within the physical space of the data screen to acquire multi-channel audio and video streams, respectively. Beamforming scanning is performed on the multi-channel audio streams, i.e., beamforming weights are set according to different directions and the output power in that direction is calculated. The direction with the highest output power is determined as the sound source direction. Simultaneously, the sound source distance is determined based on the time difference and speed of sound reaching each channel. The user's first position estimate is obtained from the sound source direction and distance. A face region is detected from the current frame of the video stream, and the user's gaze direction vector is estimated from this face region. The first position estimate, the user's head pose, and the gaze direction vector are uniformly mapped to the world coordinate system. Filtering prediction is performed using the user's position and velocity as state vectors to obtain the position of the gaze point in the world coordinate system. Then, the gaze point is mapped to the target data area on the data screen according to the screen layout. After obtaining the target data region, this implementation establishes two connections with the broadcasting link: First, the target event representation is also generated based on the target data region, that is, when generating the broadcast text, the data region currently being gazed at by the user is considered simultaneously, so that the narration content corresponds to that region; second, the broadcasting order is also determined based on the change of the target data region relative to the previous moment, that is, when the user's gaze region changes, the order of events corresponding to the new gaze region in the broadcasting queue is advanced. Furthermore, the digital human's posture parameters and gestures are determined based on the target data region and the user's position, and the target data region is highlighted to ensure that the digital human's pointing direction aligns with the user's gaze direction.

[0172] This enables the data dashboard to have spatial registration capabilities. The narration content of the digital human changes according to the user's gaze area, the broadcast order is adjusted according to the switch of the gaze area, and the digital human's posture and gestures point to the area the user is looking at, thereby solving the problem of mismatch between the narration content and the user's attention area.

[0173] The following provides further explanation of the specific processes in the above-described spatial interactive positioning implementation method.

[0174] Furthermore, regarding sound source localization, beamforming scanning is performed on the multi-channel audio stream. The sound source direction is determined based on the output power in each scanning direction, and the sound source distance is determined by combining the arrival time difference between each channel, thereby obtaining a first position estimate of the user. This includes: determining the beamforming weights in each scanning direction according to the minimum variance distortion-free response criterion based on the covariance matrix of the multi-channel audio stream, and determining the scanning direction with the highest output power as the sound source direction; inputting the spectral features of the multi-channel audio stream into a deep neural network to obtain the probability distribution of the user at each spatial location, and obtaining a second position estimate based on the probability distribution; performing weighted fusion of the first position estimate and the second position estimate with adaptive fusion weights, and using the fusion result as the first position estimate.

[0175] Specifically, this embodiment provides two mutually verified paths for sound source localization and fuses them. The first path determines the beamforming weights according to the minimum variance distortion-free response criterion. This criterion minimizes the output power while maintaining the gain in the target direction, thus highlighting the sound source in the desired direction even in the presence of ambient noise and reverberation. The output power is calculated separately for different scanning directions, and the direction with the highest output power is determined as the sound source direction. The sound source distance is determined by combining the arrival time difference and sound velocity between each channel. The second path inputs the spectral characteristics of the multi-channel audio stream into a deep neural network, which outputs the probability distribution of the user at each spatial location, thereby obtaining a second location estimate. Finally, the results of the two paths are weighted and fused using adaptive fusion weights. The fusion weights can be determined based on the confidence level or historical accuracy of each path, and the fusion result serves as the first location estimate.

[0176] Therefore, by cross-validating the two paths of classical beamforming and depth spatial coding, a stable position estimate can still be given in scenarios with strong reverberation or high noise, and the positioning output can be maintained based on the fusion weight when one path fails.

[0177] Furthermore, regarding the estimation of gaze direction, detecting a face region from the video stream and estimating the user's gaze direction vector based on the face region includes: detecting and cropping the face region from the current frame of the video stream, extracting image features of the face region and performing global pooling on the image features to obtain a face feature vector; performing regression processing on the face feature vector to obtain a unit vector representing the gaze direction, and using the angle between the unit vector and the ground truth of the gaze direction as the training loss.

[0178] Specifically, in this embodiment, a face region is detected and cropped from the current frame of the video stream. A convolutional backbone network extracts image features from this face region, and global pooling is performed on the image features to obtain a face feature vector. A regression head then performs regression processing on the face feature vector, outputting a unit vector representing the gaze direction. During training, the angle between the unit vector and the ground truth of the gaze direction is used as the training loss; that is, the angle error is directly used as the optimization objective, thus directly constraining the deviation of the estimated unit vector in direction. Alternatively, the face region can be segmented into several image blocks and input into an encoder based on a self-attention mechanism. After global attention processing to obtain the face feature vector, regression is then performed to maintain stable direction estimation even under conditions of significant facial pose variations.

[0179] Therefore, by using the angle error as the training loss, the error in the gaze direction estimation is directly reflected as the direction deviation. Furthermore, since the estimation can be completed using only the image of the face region, it is convenient to deploy a single-channel image acquisition device within the space where the data screen is located.

[0180] Furthermore, regarding the unification of various coordinate systems, the first position estimate, the user's head pose determined by the face region, and the gaze direction vector are uniformly mapped to the same world coordinate system, and the mapping result is filtered and predicted. This includes: transforming the world coordinate system, the screen coordinate system of the data screen, and the user's user coordinate system using a homogeneous transformation matrix, wherein the homogeneous transformation matrix is ​​determined by the installation posture of the data screen and the posture of the image acquisition device; using the user's position and velocity in the world coordinate system as a state vector, performing prediction and updating on the state vector to obtain an updated state vector; determining the position of the gaze point in the world coordinate system according to the position indicated by the updated state vector and the gaze direction vector; wherein, when the user's motion satisfies nonlinear conditions, particle filtering is used instead of the prediction and updating.

[0181] Specifically, in this embodiment, the world coordinate system, screen coordinate system, and user coordinate system are mutually transformed using a homogeneous transformation matrix. This homogeneous transformation matrix is ​​determined by the installation orientation of the data screen and the orientation of the image acquisition device, and is fixed after installation. Thus, the position and orientation in the three coordinate systems can be uniformly expressed within the same coordinate system. Based on this, the user's position and velocity in the world coordinate system are used as the state vector, and prediction and updating are performed: the prediction step calculates the current state based on the state estimate from the previous moment and the motion model; the update step corrects the prediction result based on the current observation value, obtaining an updated state vector, which also smooths observation noise. Then, the position of the gaze point in the world coordinate system is determined according to the position indicated by the updated state vector and the gaze direction vector. When the user's movement meets nonlinear conditions, such as rapid movement, turning, or occlusion, particle filtering replaces the above prediction and updating to adapt to nonlinear motion situations. The gaze direction vector estimated from the face region is located in the camera coordinate system of the image acquisition device. It needs to be transformed to the user coordinate system based on the user's head pose and then mapped to the world coordinate system. Therefore, the head pose participates in the coordinate mapping. The head pose can be determined by facial key points in the face region.

[0182] Therefore, by using a homogeneous transformation matrix, the sound source location, head posture, and gaze direction are unified into the same coordinate system, allowing localization results from different sources to be compared and fused. Furthermore, by using filtering prediction to suppress observation noise and maintain tracking continuity in nonlinear situations such as rapid movement, the system can achieve this.

[0183] Furthermore, regarding the control of the digital human's posture, the posture parameters and gestures of the digital human are determined based on the target data region and the user's position, including: using the center of the target data region as the target point, performing inverse kinematics calculation on the position of the digital human's eyes and the target point to obtain the head orientation of the digital human; determining the body orientation of the digital human based on the user's position in the world coordinate system, and transitioning from the current body orientation to the body orientation according to a preset interpolation coefficient; when the target data region changes relative to the previous moment, triggering the digital human to perform a gesture pointing to the target data region, and highlighting the target data region.

[0184] Specifically, in this embodiment, the digital human's posture consists of three parts: head orientation, body orientation, and hand gestures. Head orientation is determined through inverse kinematics: using the center of the target data region as the target point and the position of the digital human's eyes as the endpoint, the rotation required to make the eyes point towards the target point is calculated, yielding the head orientation. Body orientation is determined based on the user's position in the world coordinate system; that is, the desired body orientation is determined by the direction from the digital human's position to the user's position, and the current body orientation is gradually transitioned to this desired orientation according to preset interpolation coefficients to avoid abrupt posture changes. When the target data region changes relative to the previous moment, the digital human is triggered to perform a hand gesture pointing towards the target data region, and the target data region is highlighted, ensuring consistency between the digital human's pointing direction, the highlight on the screen, and the user's gaze area.

[0185] Therefore, the digital human's head and body orientation, as well as its gestures, are determined based on the user's gaze area and location, so that the digital human's instructions correspond to the data area that the user is focusing on; interpolation transitions avoid posture jumps, and gestures and highlights are triggered synchronously to enhance the indication effect.

[0186] The aforementioned sound source localization, gaze direction estimation, coordinate system 1, and attitude control constitute the cross-modal spatial localization link in this application embodiment. The engine architecture in which this link resides is as follows: Figure 11 As shown. Figure 11 This diagram illustrates the architecture of the cross-modal spatial localization engine according to an embodiment of this application, comprising three layers from top to bottom. The top layer has two processing branches side-by-side: the left branch is for sound source localization, which internally includes the microphone array and user spatial position from left to right; the right branch is for gaze tracking, which internally includes camera video and user gaze direction from left to right. These two branches connect downwards to the middle layer. The middle layer is for multi-sensor fusion and spatial alignment, which internally includes position and orientation fusion, unified coordinate mapping, and target display area from left to right. The bottom layer is for digital human posture control, which internally includes head orientation, body orientation, and gesture guidance.

[0187] In conjunction with the embodiments of this application, the sound source localization branch corresponds to the beamforming scanning and position estimation described above; the gaze tracking branch corresponds to the face region detection and gaze direction vector estimation described above; the unified coordinate mapping of the intermediate layer corresponds to the transformation of each coordinate system to another using a homogeneous transformation matrix and the filtering prediction of the position described above; the target display area is the target data area described above, which is output to the bottom layer along with the user's position; the head orientation, body orientation, and gesture guidance of the bottom layer correspond to the digital human posture parameters and gesture actions described above. Figure 11The sound source localization and gaze tracking are two parallel inputs. The results of the two inputs are fused in the intermediate layer to obtain the target display area, indicating that the determination of the target data area depends on both the user's location and the direction of their gaze.

[0188] In addition to the aforementioned spatial interactive positioning, this application embodiment also provides a fault root cause broadcasting implementation method, used to extend the broadcast content from fault phenomena to fault root causes and fault propagation paths:

[0189] Furthermore, the method further includes: acquiring historical operational data of the monitored object; performing causal discovery on the historical operational data based on conditional independence tests; constructing a causal graph representing the causal relationships between various monitored variables; acquiring real-time operational data of the monitored object using a sliding window; performing time-series causal tests on each pair of monitored variables within the sliding window; and updating the causal graph based on the test results; responding to an alarm event; determining the ancestor node of the node corresponding to the alarm event in the causal graph; extracting the propagation path from each ancestor node to the node corresponding to the alarm event; and obtaining a root cause candidate set; and performing graph encoding on the causal graph. The code obtains a graph representation vector, and uses the graph representation vector, the root cause candidate set, and the alarm event as generation conditions to generate a broadcast text containing the fault phenomenon, fault propagation path, and root cause. The candidate nodes in the root cause candidate set are sorted according to a preset root cause score. The root cause score increases as the probability of the candidate node being a root cause increases, increases as the severity of the candidate node's anomaly increases, and decreases as the time interval between the current moment and the occurrence time of the alarm event increases. The digital human is driven to perform root cause broadcasting based on the broadcast text, and the causal graph is updated incrementally according to the real-time running data updated by the sliding window.

[0190] Specifically, this implementation describes the ability to broadcast root cause reports of link access failures, and its processing is divided into two stages. In the initialization stage, historical operational data of the monitored objects is acquired. Causal discovery is performed on the historical operational data based on conditional independence tests. This involves determining the dependencies between variables by examining whether each monitored variable is independent given other variables, and constructing a causal graph representing the causal relationships between the monitored variables. In the operational stage, real-time operational data is acquired using a sliding window. Within the sliding window, time-series causal tests are performed on each pair of monitored variables. This involves examining whether the historical values ​​of one variable help predict another variable over time, and updating the causal graph based on the test results, enabling the causal graph to adapt to changes in equipment status and topology. When an alarm event occurs, the ancestor nodes of the node corresponding to the alarm event are determined in the cause-effect graph, and the propagation paths from each ancestor node to the current node are extracted. The ancestor nodes constitute a root cause candidate set. Graph encoding is performed on the cause-effect graph to obtain a graph representation vector. Using this graph representation vector, the root cause candidate set, and the alarm event as generation conditions, a broadcast text containing the fault phenomenon, fault propagation path, and root cause is generated. At the same time, each candidate node in the root cause candidate set is sorted according to the root cause score. This score increases with the probability of the candidate node being the root cause and its abnormality severity, and decreases with the increase of time interval. Finally, the digital human is driven to perform root cause broadcast based on the broadcast text, and the cause-effect graph is incrementally updated according to the real-time running data updated by the sliding window.

[0191] This allows the broadcast content to extend from alarm phenomena to the root cause and propagation path of the fault, enabling maintenance personnel to directly locate the source of the fault based on the broadcast content. Since the cause-effect graph is built offline and updated incrementally by sliding window, this implementation method can adapt to changes in device status and topology.

[0192] The following provides further explanation of the specific processing steps in the above-described root cause reporting implementation method.

[0193] Furthermore, regarding the offline construction of the causal graph, causal discovery is performed on the historical operational data based on conditional independence tests to construct a causal graph representing the causal relationships between the monitored variables. This includes: initializing a completely undirected graph with each monitored variable as a node; performing conditional independence tests on adjacent node pairs according to the order of the condition sets from low to high; deleting edges that satisfy conditional independence and recording the corresponding separation sets; performing directional inference on triples that satisfy a preset structure based on each separation set to obtain directed edges; and completing the orientation of the remaining edges according to the transitive orientation rule to obtain the causal graph; when there are unobserved confounding variables, outputting an ancestor graph containing some directed edges and bidirectional edges as the causal graph; or performing a greedy search of forward edge addition and backward edge deletion on the causal graph using a scoring function as a criterion.

[0194] Specifically, this implementation adopts a constraint-based causal discovery method: A completely undirected graph is initialized with each monitored variable as a node. Then, conditional independence is checked for each pair of adjacent nodes in ascending order of the condition set. If two nodes are independent given a certain condition set, the edge between them is deleted, and the separation set that makes them independent is recorded. After deleting the edge, direction inference is performed on triples that satisfy a preset structure: if both nodes are connected to a third node but not to each other, and the third node is not in the separation set of the two nodes, then the triple is oriented as a confluence structure pointing from both ends to the middle. The remaining edges are then oriented according to the transitive orientation rule to obtain the causal graph. When there are unobserved confounding variables, an ancestor graph containing partially oriented edges and bidirectional edges is output as the causal graph. In this case, the reliability of the conclusion is traded for the degree of edge orientation. Alternatively, a scoring-based method can be used: using a scoring function as the criterion, a forward edge-adding search is performed first, followed by a backward edge-deleting search, until the score no longer increases, thus obtaining the causal graph.

[0195] Therefore, causal discovery based on conditional independence tests does not require prior specification of the causal direction; the results are determined by the data itself. When dealing with cases where there are unobserved variables, the ancestor graph is output, avoiding the conclusion of a definite direction when conditions are insufficient.

[0196] Furthermore, regarding the online updating of the causal graph, performing a time-series causal test on each pair of the monitored variables within the sliding window and updating the causal graph based on the test results includes: performing a Granger causality test on the first and second monitored variables within the sliding window to obtain a test result on whether the first monitored variable is a Granger cause of the second monitored variable; calculating the transit entropy from the first monitored variable to the second monitored variable, and determining the first monitored variable that simultaneously satisfies the criteria corresponding to the Granger causality test and the transit entropy as the causal parent node of the second monitored variable; adding a directed edge from the causal parent node to the second monitored variable in the causal graph, and determining the confidence level of the directed edge using the statistic of the Granger causality test.

[0197] Specifically, in this embodiment, a Granger causality test is performed on the first and second monitored variables within the sliding window: if incorporating the historical values ​​of the first monitored variable when predicting the second monitored variable significantly reduces the prediction error, then the first monitored variable is a Granger cause of the second monitored variable. The test process obtains a statistic by comparing the sum of squared residuals of the restricted and unrestricted models, and uses this statistic to determine whether the test is valid. Based on this, the transfer entropy from the first to the second monitored variable is further calculated. The transfer entropy measures the additional information about the future values ​​of the second monitored variable derived from the historical values ​​of the first monitored variable, given the historical values ​​of the second monitored variable itself. It does not depend on the linear hypothesis and therefore can capture nonlinear causal relationships between variables. Only when both the Granger causality test criterion and the transfer entropy criterion are satisfied is the first monitored variable identified as the causal parent node of the second monitored variable, and a corresponding directed edge is added to the causal graph. The confidence level of the directed edge is determined by the Granger causality test statistic, providing a basis for subsequent root cause scoring.

[0198] Therefore, using both linearity tests and information content measures as dual criteria to determine causal relationships not only retains the convenience of Granger causality tests in implementation, but also supplements the ability to identify nonlinear causal relationships by using transfer entropy, thus reducing the risk of mistaking accidental correlations for causality.

[0199] Furthermore, regarding the generation of the root cause broadcast text, graph encoding is performed on the causal graph to obtain a graph representation vector. Using the graph representation vector, the root cause candidate set, and the alarm event as generation conditions, a broadcast text containing the fault phenomenon, fault propagation path, and root cause is generated. This includes: performing node embedding and superimposing position encoding on the node features of each monitored variable to obtain an initial representation of each node; performing graph attention calculation using the adjacency relationship of the causal graph as a mask, wherein the positions of node pairs without edges participate in the graph attention calculation with a negative infinite bias to obtain the target layer representation of each node; performing graph pooling on each target layer representation to obtain the graph representation vector; constructing prompt information according to a hierarchical structure of fault phenomenon, fault propagation path, root cause and root cause confidence, and handling suggestions; and inputting the graph representation vector, the root cause candidate set, and the alarm event into the prompt information and then into a pre-trained language model to obtain the broadcast text.

[0200] Specifically, in this embodiment, the causal graph is encoded to obtain a graph representation vector: First, node embedding and position encoding are performed on the node features of each monitored variable, so that each node retains its position information in the graph while obtaining its own feature representation; then, graph attention calculation is performed using the adjacency relationship of the causal graph as a mask. That is, for node pairs with edges in the graph, their attention weights are calculated in the conventional way, while for node pairs without edges, they participate in the attention calculation with a negative infinite bias, so that their attention weights approach zero after normalization, thereby ensuring that attention is only transmitted between nodes with causal relationships; after several layers of graph attention calculation, the target layer representation of each node is obtained, and then graph pooling is performed on each target layer representation to obtain a graph representation vector representing the entire causal graph. When generating the broadcast text, the prompt information is constructed according to the hierarchical structure of fault phenomenon, fault propagation path, root cause and root cause confidence, and handling suggestions. The graph representation vector, root cause candidate set and alarm event are filled into the prompt information and then input into the pre-trained language model to obtain the broadcast text organized according to the above hierarchy.

[0201] Therefore, graph attention computation using adjacency as a mask ensures that information is transmitted only along edges of causal relationships, avoiding mutual interference between irrelevant nodes; prompts are constructed according to a fixed hierarchy, so that the generated broadcast text is structurally consistent, making it easy for maintenance personnel to obtain phenomena, paths, root causes and handling suggestions in a fixed order.

[0202] Furthermore, regarding the incremental update of the root cause score and the causal graph, the probability of a candidate node being a root cause in the root cause score is determined by backpropagation of the confidence of the directed edges pointing to the candidate node from each parent node of the candidate node; the incremental update of the causal graph based on the real-time running data after the sliding window update includes: re-performing the conditional independence test on each existing edge in the causal graph, updating the edges whose test results have changed, and adding newly discovered causal relationships within the sliding window to the causal graph when their confidence meets preset conditions.

[0203] Specifically, in this embodiment, the probability of a candidate node being a root cause is determined by backpropagation of the confidence scores of all directed edges pointing to that node: the probability of the candidate node itself being a root cause is obtained by combining the confidence scores of the edges pointing to it from each of its parent nodes, and the probability of each parent node itself is determined by the confidence scores of the edges further upstream, thus converging step by step along the reverse direction of the causal graph to obtain the probability of the candidate node being a root cause. This probability is then multiplied by the anomaly severity of the candidate node and a decay factor of the time interval between the current moment and the time of the alarm event to obtain the root cause score of the candidate node, which is used to sort the root cause candidate set. When incrementally updating the causal graph, the conditional independence test is re-executed on each existing edge in the causal graph, and the corresponding edge is updated if the test result changes; for newly discovered causal relationships within the sliding window, they are added to the causal graph when their confidence scores meet the preset conditions, thereby keeping the causal graph consistent with the latest operating data without rebuilding the entire graph.

[0204] Thus, the root cause probability is determined by backpropagation along the causal structure, enabling the root cause ranking to reflect the cumulative confidence level of the fault throughout the entire propagation chain; the causal graph is updated incrementally, allowing it to adapt to changes in device state and topology, and avoiding the computational overhead of rebuilding the entire graph.

[0205] The aforementioned offline cause-effect graph construction, online temporal cause verification, cause-effect graph encoding, and root cause scoring and ranking constitute the fault root cause broadcasting link in this application embodiment. The engine architecture in which this link resides is as follows: Figure 12 As shown. Figure 12 This diagram illustrates the architecture of the causal inference root cause engine of this application, comprising four processing layers from top to bottom. The first layer is offline causal graph construction, which, from left to right, includes historical logs and monitoring data, data preprocessing, causal relationship discovery, and the initial causal graph. The second layer is online time-series causal inference, which, from left to right, includes a real-time data window, time-series causal analysis, fault propagation path, and causal graph updates. The third layer is causal graph to natural language conversion, which, from left to right, includes the target causal graph, graph feature encoding, natural language decoding, and root cause broadcast text. The fourth layer is root cause scoring and ranking, which, from left to right, includes root cause probability, anomaly degree, timeliness information, and root cause ranking results. The layers are interconnected from top to bottom.

[0206] In conjunction with the embodiments of this application, the first layer of causal relationship discovery corresponds to the causal discovery based on conditional independence test described above; the second layer of temporal causal analysis corresponds to Granger causality test and transit entropy described above; the second layer of causal graph update and the fourth layer of root cause scoring and ranking correspond to the root cause probability backpropagation determination method and incremental update of the causal graph using a sliding window described above; the third layer of graph feature encoding and natural language decoding corresponds to graph attention calculation using adjacency as a mask and generation of broadcast text by constructing prompt information according to a hierarchical structure described above. The first and second layers in the figure are interconnected, indicating that the construction of the causal graph is divided into two stages: offline and online. The offline stage establishes an initial causal graph based on historical data, and the online stage continuously corrects the causal graph based on real-time data windows, so that the causal graph has both an initial structural foundation and can adapt to changes in device status and topology.

[0207] The digital human broadcasting control method provided in the embodiments of this application has been described above. Accordingly, the embodiments of this application also provide a system for performing the above method:

[0208] Figure 13 This is a schematic block diagram of a data dashboard digital human assistant system provided in an embodiment of this application. It should be noted that... Figure 13 Dashed boxes are used to represent the modules connected to the scene scheduling module 1310, including all modules from the data acquisition module 1301 to the fault root cause broadcasting module 1309; solid boxes are used to represent the system function modules that implement the corresponding processing functions, including all modules from the data acquisition module 1301 to the scene scheduling module 1310. For example... Figure 13 As shown, the system includes:

[0209] Data acquisition module 1301 is used to acquire streaming data from the data source associated with the data dashboard, wherein the streaming data includes real-time data streams collected in real time; wherein, Figure 13 The industrial data source shown is used to provide industrial data to be processed to the data acquisition module 1301. The industrial data source may correspond to at least one of the sensor data, log data, and monitoring data shown in the figure. The data acquisition module 1301 acquires the corresponding streaming data from the industrial data source and provides the acquired streaming data to subsequent functional modules for processing.

[0210] The change feature extraction module 1302 is used to determine the change feature quantity of the streaming data over time, wherein the change feature quantity characterizes the data change state of the data source.

[0211] The anomaly analysis module 1303 is used to perform anomaly detection on the streaming data according to preset anomaly criteria, obtain an anomaly event set, and determine the correlation between each anomaly event in the anomaly event set.

[0212] The event compression module 1304 is used to encode the features of the abnormal event and the correlation relationship to obtain the target event representation of the abnormal event.

[0213] The narrative generation module 1305 is used to generate broadcast text corresponding to the abnormal event, using the target event representation as the generation condition.

[0214] The broadcast arbitration module 1306 is used to determine the broadcast order of each event in the broadcast queue, which consists of the events to be broadcast corresponding to the broadcast text, according to a preset priority rule. After determining the broadcast order of each event, the broadcast arbitration module 1306 outputs the corresponding broadcast text sequentially according to the determined broadcast order, so that the broadcast text is provided to the frame-level synchronization broadcast module 1307 for subsequent broadcast processing in the determined broadcast order. Figure 13 The “broadcast text output in broadcast order” shown refers to the broadcast text after the broadcast arbitration module 1306 determines the broadcast order.

[0215] A frame-level synchronized broadcast module 1307 is used to determine the prosodic boundary of the broadcast text based on the changing feature quantity, synthesize the broadcast text into an audio stream according to the prosodic boundary, convert the audio stream into a facial animation parameter sequence of the digital human, determine the time offset between the audio stream and the facial animation parameter sequence, perform correction on the facial animation parameter sequence according to the time offset, and drive the rendering output of the digital human according to the corrected facial animation parameter sequence and the audio stream. Figure 13 The data screen and digital human shown on the right are the display and interactive objects of the system. The digital human is used to broadcast and present animations according to the audio stream and the corrected facial animation parameter sequence.

[0216] The spatial interaction positioning module 1308 is used to collect multi-channel audio and video streams within the physical space where the data screen is located, thereby determining the user's first position estimate and the user's gaze direction vector, mapping the user's gaze point to a target data area on the data screen, and determining the digital human's posture parameters and gestures based on the target data area and the user's position. Figure 13 The “user voice and gaze” shown indicates the source of external interactive information for the spatial interaction positioning module 1308. The spatial interaction positioning module 1308 determines the user's position, gaze direction, and target data area based on the audio stream and video stream.

[0217] The fault root cause broadcasting module 1309 is used to construct and update a causal graph representing the causal relationship between various monitored variables online. In response to an alarm event, it determines a root cause candidate set in the causal graph and generates a broadcast text containing the fault phenomenon, fault propagation path and root cause. It also sorts each candidate node in the root cause candidate set according to a preset root cause score.

[0218] The scene scheduling module 1310 is used to determine the activation status and priority of the data acquisition module 1301, the change feature extraction module 1302, the anomaly analysis module 1303, the event compression module 1304, the narrative generation module 1305, the broadcast arbitration module 1306, the frame-level synchronous broadcast module 1307, the spatial interaction positioning module 1308, and the fault root cause broadcast module 1309 according to the scene mode. Figure 13 The scene mode shown serves as the scheduling input for the scene scheduling module 1310, representing the current working scene of the system. Based on this, the scene scheduling module 1310 determines the activation status and priority of each of the above functional modules.

[0219] It should be understood that the specific processes by which each module performs the corresponding steps described above have been detailed in the above method embodiments, and will not be repeated here for the sake of brevity. Furthermore, the module division in the embodiments of this application is illustrative and merely a logical functional division; in actual implementation, there may be other division methods. Additionally, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or have two or more modules integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0220] The data flow between the modules of the above system is as follows: Figure 14 As shown. Figure 14 This diagram illustrates the end-to-end data flow of an embodiment of this application. The top layer is the data acquisition layer, including six data sources: device data, sensor readings, business data, system logs, video streams, and audio streams. Below this is the message middleware, including four types of messages: sensor data, business data, alarm data, and multimedia data. Next is the core processing engine, which internally comprises a front-end parallel frame-level synchronization engine, a semantic compression engine, and a spatial positioning engine, all of which are connected to the subsequent causal inference engine. The bottom layer is the display and interaction layer, including large-screen rendering, digital human animation, voice broadcasting, and voice interaction. The layers are connected sequentially from top to bottom by arrows.

[0221] In conjunction with the embodiments of this application, Figure 14To illustrate the correspondence between modules and data flow in the above system embodiments: the various types of data collected by the data acquisition layer, namely the streaming data of the data source associated with the data dashboard, are acquired by the data acquisition module 1301; the message middleware is used to transmit data between the data source and the core processing engine, enabling different types of data to enter the processing stage through a unified interface; the frame-level synchronization engine in the core processing engine corresponds to the frame-level synchronization broadcast module 1307, the semantic compression engine corresponds to the event compression module 1304 and the narrative generation module 1305, the spatial positioning engine corresponds to the spatial interaction positioning module 1308, and the causal reasoning engine corresponds to the fault root cause broadcast module 1309; the display and interaction layer corresponds to the presentation of digital human rendering output and user interaction results. The arrangement order of each engine in the figure is consistent with the transmission direction of data from acquisition to presentation, indicating that the processing process of this application embodiment is a data chain with a defined direction, wherein the output of the upstream link constitutes the input of the downstream link.

[0222] The scene scheduling module in the above system determines the activation status and priority of each module according to the identified scene mode, and its coordination logic is as follows: Figure 15 As shown. Figure 15 The collaborative logic of the scene scheduler in this embodiment is shown, comprising three layers from top to bottom. The top layer is the scene scheduler; below it is the pattern recognition layer, which contains three modes: visitor mode, fault mode, and daily mode. The visitor mode is characterized by the detection of visitors and the absence of fault alarms, the fault mode by the detection of fault alarms, and the daily mode by the absence of visitors and the absence of fault alarms. The bottom layer is the subsystem collaboration layer, which contains four stages from left to right: frame-level synchronization, semantic compression, spatial positioning, and causal reasoning. Their functions are labeled as outputting change information, generating narrative content, constraining content and posture, and generating root cause information, respectively.

[0223] In conjunction with the embodiments of this application, Figure 15 To illustrate the operation of the scene scheduling module in the above system embodiment: The scene scheduling module 1310 determines the current scene mode based on the detected visitor situation and fault alarm situation, and determines the activation status and priority of each module according to the scene mode. In fault mode, the causal reasoning link is activated, and the root cause information generated by it is incorporated into the broadcast content; in visitor mode, the constraint effect of the spatial positioning link is strengthened, that is, the broadcast content and the digital human's posture change according to the visitor's gaze area; in daily mode, the broadcast mainly consists of a rotation and summary of core indicators. In the figure, the links in the subsystem collaboration layer are connected sequentially from left to right, indicating that the engines work collaboratively in a determined order under the same scene mode, where the output of the spatial positioning link is used to constrain the content and posture of the subsequent links.

[0224] Figure 16This is a schematic block diagram of the electronic terminal provided in an embodiment of this application. Figure 16 As shown, the electronic terminal 1600 includes at least one processor 1601, a memory 1602, at least one network interface 1603, and a user interface 1605. The various components in the electronic terminal 1600 are coupled together via a bus system 1604. It is understood that the bus system 1604 is used to implement communication between these components. In addition to a data bus, the bus system 1604 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 16 The general will label all buses as bus systems.

[0225] The user interface 1605 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0226] It is understood that memory 1602 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), etc. Volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable categories of memory.

[0227] In this embodiment, the memory 1602 is used to store various types of data to support the operation of the electronic terminal 1600. Examples of this data include: any executable program for operation on the electronic terminal 1600, such as the operating system 16021 and application program 16022; the operating system 16021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 16022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The methods provided in this embodiment can be included in the application program 16022.

[0228] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 1601. The processor 1601 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware in the processor 1601 or by instructions in software form. The processor 1601 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 1601 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor 1601 may be a microprocessor or any conventional processor, etc. The steps of the methods provided in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory. The processor reads information from the memory and, in conjunction with its hardware, completes the steps of the aforementioned method.

[0229] In an exemplary embodiment, the electronic terminal 1600 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), etc., to perform the aforementioned method.

[0230] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to perform the method of any of the embodiments described above.

[0231] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when run on a computer, causes the computer to perform the method of any of the embodiments described above.

[0232] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0233] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0234] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0235] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0236] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0237] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0238] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0239] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0240] In summary, this application provides a digital human broadcasting control method, system, medium, and terminal, which can generate synchronized audio-visual broadcasting content that matches the user's gaze area in real time based on dynamic changes in industrial data, and analyze the root causes and propagation paths of equipment failures. Therefore, this application effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0241] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A digital human broadcasting control method, characterized in that, Applicable to industrial data dashboards, the method includes: Acquire streaming data from the data source associated with the data dashboard. The streaming data includes real-time data streams collected in real time. Determine the change characteristics of the streaming data over time. The change characteristics characterize the data change state of the data source. Anomaly detection is performed on the streaming data according to preset anomaly criteria to obtain an anomaly event set, and the correlation between each anomaly event in the anomaly event set is determined. Encode the features of the abnormal event and the correlation to obtain the target event representation of the abnormal event, and use the target event representation as the generation condition to generate the broadcast text corresponding to the abnormal event; According to the preset priority rules, the broadcast order of each of the events to be broadcast in the broadcast queue, which is composed of the events to be broadcast corresponding to the broadcast text, is determined. The prosodic boundary of the broadcast text is determined based on the change feature quantity, and the broadcast text is synthesized into an audio stream according to the prosodic boundary; The audio stream is converted into a sequence of facial animation parameters for a digital human. The time offset between the audio stream and the facial animation parameter sequence is determined. The facial animation parameter sequence is corrected according to the time offset. The digital human is then rendered and output according to the corrected facial animation parameter sequence and the audio stream.

2. The digital human broadcasting control method according to claim 1, characterized in that, The change characteristics include the data change rate of each of the data sources and the mutation events in the real-time data stream; Determining the time-varying characteristics of the streaming data includes: The historical data corresponding to the streaming data is divided into several time-series segments according to a preset length. Segment encoding is performed on each time-series segment and position encoding is superimposed. The segments are input into an encoder based on a self-attention mechanism to obtain a time-series dependency representation that represents the time-series dependency relationship between the segments. Regression processing is performed on the time-series dependency representation to obtain the data change rate of each data source at future times. The rate of change of data from each of the data sources is compared with an adaptive threshold obtained from the historical data of that data source, and the moment when the rate of change of data exceeds the adaptive threshold is determined as the mutation event.

3. The digital human broadcasting control method according to claim 1, characterized in that, The process of performing anomaly detection on the streaming data according to preset anomaly criteria yields a set of anomaly events, including: For each of the aforementioned data sources, a pre-defined probability density model is provided. The probability density model is composed of several cascaded invertible transformations and is used to map the data distribution of the data source to a baseline distribution. The current streaming data is input into the probability density model to obtain the log-likelihood of the streaming data under the conditional distribution determined by the corresponding historical data. The sum of the negative value of the log-likelihood and the baseline value of the data source is used as the anomaly score, and the streaming data whose anomaly scores satisfy the anomaly criteria are identified as the anomaly events, and the anomaly events constitute the anomaly event set.

4. The digital human broadcasting control method according to claim 1, characterized in that, The correlation includes the propagation path between the abnormal events; Determining the correlation between the abnormal events in the abnormal event set includes: A dynamic topology graph that updates over time is constructed using the sensor nodes corresponding to each of the data sources as nodes and the physical connection relationships and data correlation relationships between the sensor nodes as edges. Each sensor node maintains a memory vector that is updated with the received messages, and determines the inter-node messages based on the memory vectors, the memory vectors of adjacent sensor nodes, and the connection relationships between adjacent sensor nodes. The node embedding of each sensor node is obtained by aggregating the inter-node messages. Based on the node embedding of each of the sensor nodes, the propagation path from the sensor node where the abnormal event occurred to other sensor nodes is determined.

5. The digital human broadcasting control method according to claim 1, characterized in that, The characteristics of the abnormal events include temporal characteristics, spatial characteristics, and topological characteristics; The process of encoding the features of the abnormal event and the correlation to obtain the target event representation of the abnormal event includes: The first encoder encodes the temporal features to obtain a temporal feature vector, and the second encoder encodes the spatial features and the topological features to obtain a graph structure feature vector; Cross-modal attention fusion is performed using the temporal feature vector and the graph structure feature vector as input to obtain fused features; Vector quantization is performed on the fused features, discretizing the fused features into a semantic tag sequence composed of several semantic tags arranged in sequence, and the semantic tag sequence is used as the representation of the target event.

6. The digital human broadcasting control method according to claim 1, characterized in that, The step of generating broadcast text corresponding to the abnormal event, using the target event representation as a generation condition, includes: The causal chain corresponding to the abnormal event is determined based on the correlation, and a prompt message is constructed from the target event representation, the abnormal event, and the causal chain; The prompt information is input into a pre-trained language model to obtain the broadcast text, which is organized hierarchically according to the current fault phenomenon, fault propagation path, root cause, and handling suggestions.

7. The digital human broadcasting control method according to claim 2, characterized in that, Determining the time offset between the audio stream and the facial animation parameter sequence, and performing correction on the facial animation parameter sequence according to the time offset, includes: Convolutional encoding is performed on the audio stream and the facial animation parameter sequence respectively to obtain an audio embedding sequence and a visual embedding sequence; Perform a cross-correlation operation on the audio embedding sequence and the visual embedding sequence, and determine the hysteresis that maximizes the cross-correlation value as the time offset; The values ​​of the corresponding frames in the facial animation parameter sequence are shifted forward or backward according to the time offset, and interpolation smoothing is performed on several adjacent frames in the corrected facial animation parameter sequence. Wherein, the time of each frame in the facial animation parameter sequence is the sum of the time of the corresponding phoneme in the audio stream and the time offset, and the time of each phoneme in the audio stream is the sum of the time of the mutation event and the advance amount corresponding to the timing prediction.

8. A large-screen digital human assistant system, characterized in that, include: The data acquisition module is used to acquire streaming data from the data source associated with the data dashboard, including real-time data streams collected in real time. A change feature extraction module is used to determine the change feature quantity of the streaming data over time, wherein the change feature quantity characterizes the data change state of the data source; An anomaly analysis module is used to perform anomaly detection on the streaming data according to preset anomaly criteria, obtain an anomaly event set, and determine the correlation between each anomaly event in the anomaly event set; The event compression module is used to encode the features and the correlation of the abnormal event to obtain the target event representation of the abnormal event; The narrative generation module is used to generate broadcast text corresponding to the abnormal event, using the target event representation as the generation condition. The broadcast arbitration module is used to determine the broadcast order of each of the events to be broadcast in the broadcast queue, which is composed of the events to be broadcast corresponding to the broadcast text, according to a preset priority rule. A frame-level synchronous broadcasting module is used to determine the prosodic boundary of the broadcast text based on the change feature quantity, and synthesize the broadcast text into an audio stream according to the prosodic boundary; The audio stream is converted into a sequence of facial animation parameters for a digital human. The time offset between the audio stream and the facial animation parameter sequence is determined. The facial animation parameter sequence is corrected according to the time offset. The digital human is then rendered and output according to the corrected facial animation parameter sequence and the audio stream.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the digital human broadcasting control method as described in any one of claims 1 to 7.

10. An electronic terminal, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the digital human broadcasting control method as described in any one of claims 1 to 7.

Images