A multimodal AGI cross-border Chinese oral assessment teaching system and method

Abstract

The application provides a multimodal AGI cross-border Chinese oral assessment teaching system and method, comprising: calculating a network jitter value of a current transmission link according to a time delay fluctuation component in the first feature vector, and judging whether the network jitter value is greater than a pre-established jitter threshold; using a classifier to perform classification processing on the first text sequence to obtain a first assessment result reflecting the oral teaching effect; obtaining a transmission time delay of the first assessment result between the target edge computing node and the center server, and judging whether the transmission time delay is greater than a preset time delay threshold; if the transmission time delay is greater than the preset time delay threshold, generating a computing power scheduling instruction according to the transmission time delay; and using the computing power scheduling instruction to redivide a global computing power pool to obtain an updated computing power distribution matrix representing the resource proportion of each edge computing node.

Description

Technical Field

[0001] This invention relates to the field of information technology, and in particular to a multimodal AGI cross-border Chinese oral assessment teaching system and method. Background Technology

[0002] With the acceleration of globalization, the demand for international education of Chinese, as one of the most widely spoken languages, has exploded. Cross-border Chinese oral teaching has become an important bridge connecting China with the world's cultural exchanges, but existing online teaching systems face many bottlenecks in dealing with global scenarios. Current mainstream solutions mostly adopt a centralized cloud architecture, requiring all learners' voice data and video feeds to be transmitted to a single data center for processing, resulting in significant differences in the learning experience for learners in different regions. This centralized processing approach exposes serious latency issues in cross-border scenarios. When learners in Asia and those in Europe and America access the same server, network latency can differ by hundreds of milliseconds or even seconds. Since oral language teaching demands extremely high real-time performance, any significant delay disrupts the natural rhythm of teacher-student interaction, rendering pronunciation correction and lip-reading guidance ineffective. Adding to the complexity, when the system needs to process multiple modalities of data simultaneously, such as speech recognition, lip-reading capture, and facial expression analysis, the data volume increases exponentially. Relying solely on central cloud computing power allocation leads to bandwidth congestion and slow response times, especially during peak user periods, where remote learners often experience buffering or even connection interruptions. This cumulative effect of latency further exacerbates the imbalance in the allocation of teaching resources. Because data from users in different regions converges at the central node, the system struggles to provide differentiated services based on network conditions and time zone characteristics, resulting in inconsistent teaching quality. For example, when European learners access the system during peak evening study hours, their voice data needs to be transmitted across the ocean to an Asian server for recognition and evaluation before the results are returned. Network jitter and packet loss during this round trip severely impact evaluation accuracy, while the system's core computing power may be being used by users in other time zones during off-peak hours, leading to resource misallocation. Therefore, how to eliminate the interference of cross-border network latency on teaching interaction while ensuring the real-time processing capability of multimodal data, and at the same time realize the intelligent scheduling of teaching resources and the efficient allocation of computing power globally, has become a key issue in building a high-quality cross-border Chinese oral assessment and teaching system. Summary of the Invention

[0003] This invention provides a multimodal AGI cross-border Chinese oral assessment teaching system and method, mainly including: The first multimodal data collected by the cross-border teaching terminal is acquired. The first multimodal data includes a voice audio stream and a facial image frame sequence. Temporal features are extracted from the first multimodal data through a neural network to obtain a first feature vector used to characterize the link transmission behavior. The network jitter value of the current transmission link is calculated based on the delay fluctuation component in the first feature vector, and it is determined whether the network jitter value is greater than a pre-established jitter threshold. If the network jitter value is greater than the pre-established jitter threshold, then an edge node allocation instruction is generated for the first multimodal data, and a target edge computing node is determined from the candidate edge computing node set according to the edge node allocation instruction; The target edge computing node performs joint recognition processing on the speech audio stream and facial image frame sequence in the first multimodal data to obtain a first text sequence containing the pronunciation unit and lip shape alignment results; The first text sequence is classified using a classifier to obtain the first evaluation result that reflects the effectiveness of oral language teaching. The transmission latency of the first evaluation result between the target edge computing node and the central server is obtained, and it is determined whether the transmission latency is greater than a preset latency threshold. If the transmission delay is greater than the preset delay threshold, a computing power scheduling instruction is generated based on the transmission delay. The global computing power pool is re-divided using the aforementioned computing power scheduling instructions to obtain an updated computing power distribution matrix that characterizes the resource proportion of each edge computing power node; Based on the updated computing power distribution matrix, teaching resources matching the cross-border teaching terminal are distributed to the cross-border teaching terminal to obtain a resource allocation sequence containing the correspondence between teaching resources and target terminals.

[0004] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: This invention discloses a multimodal AGI cross-border Chinese oral communication assessment and teaching system and method. Addressing problems in cross-border Chinese oral communication teaching caused by unstable link transmission, large network jitter, and uneven distribution of edge computing power, resulting in high latency in speech and lip-reading recognition, delayed feedback of assessment results, and mismatched distribution of teaching resources, this invention acquires multimodal data including speech audio streams and facial image frame sequences collected by cross-border teaching terminals. It extracts feature vectors representing link transmission behavior through a neural network, calculates network jitter values ​​based on the latency fluctuation components, and compares them with a jitter threshold. When the jitter exceeds the threshold, an edge node allocation instruction is generated. The target node is determined from the candidate edge computing node set. This node performs joint recognition of speech and facial images to obtain a text sequence that aligns the pronunciation units with the lip shape. The text sequence is then processed by a classifier to output the oral assessment results. The system further determines whether the transmission latency of the assessment results back to the central server exceeds the threshold. If it does, a computing power scheduling instruction is generated to re-divide the global computing power pool, resulting in an updated computing power distribution matrix. Based on this matrix, matching teaching resources are distributed to the terminal. This achieves an integrated closed loop of link awareness, adaptive computing power scheduling, and precise distribution of personalized resources, significantly improving the real-time performance, accuracy, and teaching adaptability of cross-border oral assessment. Attached Figure Description

[0005] Figure 1 This is a flowchart of a multimodal AGI cross-border Chinese oral assessment teaching system and method according to the present invention.

[0006] Figure 2 This is a schematic diagram of a multimodal AGI cross-border Chinese oral assessment teaching system and method according to the present invention.

[0007] Figure 3 This is another schematic diagram of a multimodal AGI cross-border Chinese oral assessment teaching system and method according to the present invention. Detailed Implementation

[0008] The technical solutions of the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention.

[0009] like Figures 1-3 This embodiment of a multimodal AGI cross-border Chinese oral assessment teaching system and method may specifically include: S101. Obtain the first multimodal data collected by the cross-border teaching terminal. The first multimodal data includes a voice audio stream and a facial image frame sequence. Extract the temporal features of the first multimodal data through a neural network to obtain a first feature vector used to characterize the link transmission behavior.

[0010] First multimodal data, including audio streams and image frames, is acquired from the teaching end. A neural network is used to extract temporal features from the first multimodal data to obtain a first feature vector. Based on the first feature vector, a first feature value is extracted on the time axis. It is then determined whether the first feature value is greater than a preset threshold. If the first feature value is greater than the preset threshold, the first feature vector is determined to be abnormal, resulting in an abnormal feature vector. For the abnormal feature vector, the corresponding transmitter and receiver identifiers are obtained. A pre-established link state table is queried using the transmitter and receiver identifiers to obtain first link state data. Based on the first link state data, the packet loss rate and latency contained in the first link state data are analyzed. A decision tree algorithm is used to classify the packet loss rate and latency to obtain a link congestion level. The link congestion level is obtained, and it is determined whether the link congestion level matches a preset severe congestion condition. If the link congestion level matches the severe congestion condition, downsampling processing is performed on the audio stream and the image frames to obtain second multimodal data. For the second multimodal data, extract the second feature value of the second multimodal data, construct a second vector set based on the second feature value, and output the second vector set to the receiving end.

[0011] In one possible implementation, for online teaching scenarios, the teaching terminal collects the audio stream of the teacher's lecture and the whiteboard image frames as the first multimodal data. A recurrent neural network is used to extract temporal features from this data, mapping five consecutive seconds of audio and video data into a high-dimensional first feature vector. Then, the first eigenvalue of this vector on the time axis is extracted, for example, by calculating the energy fluctuation value of the feature vector. If this eigenvalue is greater than a preset threshold of 0.85, it indicates that the current audio and video data has drastic temporal fluctuations, possibly caused by network jitter, thus determining that the first feature vector is abnormal, and obtaining an abnormal feature vector. This neural network-based temporal analysis can accurately capture sudden interruptions in the teaching process, effectively improving the accuracy of anomaly detection.

[0012] For example, for the aforementioned abnormal feature vectors, their corresponding transmission end identifiers, such as the teacher's network address and receiving end identifiers, such as the student's network address, are extracted. The first link state data of the current network is obtained by querying a pre-established distributed link state table. This data is parsed to obtain specific network metrics, such as a packet loss rate of 5% and a latency of 120 milliseconds. Then, the packet loss rate and latency are used as input features and fed into a pre-trained decision tree algorithm model. This decision tree model evaluates the input features using node splitting rules and outputs a link congestion level of three. By introducing the decision tree algorithm to comprehensively classify multi-dimensional network metrics, the congestion level of the current teaching link can be quickly and objectively quantified, avoiding misjudgment based on a single metric.

[0013] It should be noted that after obtaining the link congestion level, it is determined whether it matches a preset severe congestion condition, such as whether the congestion level reaches level three or above. If it matches, downsampling processing is performed on the original audio stream and image frames, for example, reducing the image frame rate from 30 frames per second to 15 frames per second and the audio sampling rate from 44100 Hz to 22050 Hz, thereby obtaining a second multimodal data with a smaller data volume. Subsequently, the second feature value of this second multimodal data is extracted, such as the keyframe features after downsampling, and a second vector set is constructed and output to the student receiving end. This approach, by actively reducing data dimensionality and transmission volume during severe network congestion, maximizes the continuity of teaching audio and video, significantly improving the teaching experience in weak network environments.

[0014] S102. Calculate the network jitter value of the current transmission link based on the delay fluctuation component in the first feature vector, and determine whether the network jitter value is greater than a pre-established jitter threshold.

[0015] The first feature vector is obtained from the target node, and the delay fluctuation component contained in the first feature vector is extracted to obtain a first fluctuation value. Based on the first fluctuation value, the network jitter value of the current transmission link is calculated using a time series analysis tool to obtain a first jitter value. A pre-established jitter threshold is obtained, and it is determined whether the first jitter value is greater than the jitter threshold. If the first jitter value is greater than the jitter threshold, a first abnormal state identifier is determined. For the first abnormal state identifier, anomaly detection is performed on the historical delay value of the current transmission link using the isolated forest algorithm to obtain a second abnormal state identifier. Based on the second abnormal state identifier, the link-end state quantity of the current transmission link is extracted to obtain a first state quantity. Based on the first state quantity, it is determined whether the first state quantity matches a preset fault judgment value. If the first state quantity matches the fault judgment value, a target blocking instruction is determined.

[0016] In one possible implementation, for multimodal data transmission in cross-border teaching scenarios, network jitter is a key indicator for measuring link stability. After obtaining the first feature vector from the target node, the delay fluctuation component reflecting the arrival time difference of data packets is extracted from this vector to obtain the first fluctuation value.

[0017] Specifically, the first fluctuation value is smoothed and trend-fitted using an autoregressive moving average model in time series analysis tools, and the network jitter value of the current transmission chain is calculated as the first jitter value.

[0018] For example, a pre-established jitter threshold is set to 60 milliseconds. When the first jitter value input reaches 85 milliseconds, it is determined that there is a severe fluctuation in the current cross-border link, and thus the first abnormal state identifier is determined.

[0019] In one possible implementation, to eliminate the interference of occasional network congestion, the isolated forest algorithm is used to perform deep anomaly detection on the historical delay values ​​of the current transmission link based on the first anomaly identifier.

[0020] Specifically, using the historical latency sequence within the past 5 minutes as input data, the Isolation Forest algorithm constructs multiple isolation trees by randomly selecting features and split points, spatially partitioning the latency data. Because extremely high latency data is sparse in the data space, it can be isolated with fewer partitions in the isolation trees, thus obtaining a higher anomaly score. When the output anomaly score exceeds a set threshold, a second anomaly flag is obtained, indicating that there is a non-incidental, persistent anomaly in this cross-border teaching link.

[0021] For example, based on the second abnormal state identifier, the link-end state variables of the current transmission chain are further extracted to obtain a first state variable. The first state variable includes the number of consecutive unresponsive times of the transmission port and the connectivity status of the underlying routing nodes. The first state variable is matched and analyzed with a preset fault judgment value. If the number of consecutive unresponsive times in the first state variable reaches the set fault judgment value of 150 times, it is determined that the current link has experienced physical-level or routing-level blocking. At this time, a target blocking instruction is determined. This instruction is used to directly terminate the current abnormal audio and video data stream transmission process and release the occupied transmission channel resources.

[0022] S103. If the network jitter value is greater than the pre-established jitter threshold, then generate an edge node allocation instruction for the first multimodal data, and determine the target edge computing node from the candidate edge computing node set according to the edge node allocation instruction.

[0023] The jitter value of the current network environment is obtained. If the jitter value is greater than a pre-established judgment value, a first allocation order is generated for the first data packet. A candidate set containing multiple edge nodes is obtained based on the first allocation order. The edge nodes in the candidate set are clustered using a K-means clustering algorithm to obtain a first set of computing power points. The load rate and latency of each edge node in the first set of computing power points are obtained. A node state matrix is ​​constructed using the load rate and latency to obtain a first state matrix. Features are extracted from the first state matrix using a random forest algorithm to determine the availability of each edge node in the first set of computing power points. A matching degree is calculated based on the availability and the transmission rate of the first data packet. If the matching degree is greater than a preset matching threshold, a second set of computing power points is determined. The edge nodes in the second set of computing power points are sorted to obtain the priority of each edge node. A target computing power point is determined in the second set of computing power points based on the priority.

[0024] For example, obtaining and determining the jitter value of the current network environment is a fundamental step in ensuring stable data packet transmission. When the jitter value of the current transmission link is detected to reach 50 milliseconds, while the pre-established determination value is set at 30 milliseconds, since the jitter value is greater than the determination value, it indicates that there is a risk of congestion on the current link, and a first allocation order is then generated for the first data packet.

[0025] Specifically, a candidate set of 100 edge nodes is obtained according to the first allocation order. To quickly filter out nodes that are physically close and have similar network states, the mean clustering algorithm is used to cluster the edge nodes in the candidate set. The input is the initial geographical coordinates and basic response time of these 100 nodes. The number of cluster centers is set to 5. After iterative calculation, 5 different node clusters are output. The cluster with the closest comprehensive distance is selected as the first set of computing power points, which contains 20 edge nodes.

[0026] Specifically, real-time operational data for each edge node is obtained from the first set of computing power points. For example, a node's load rate is 60% and its latency is 15 milliseconds. A node state matrix is ​​constructed by integrating the load rates and latency of these 20 nodes, resulting in a first state matrix. Subsequently, a random forest algorithm is used to extract features from the first state matrix. The input is the first state matrix containing load and latency features. Through ensemble learning using multiple decision trees, an availability score for each edge node in the first set of computing power points is output. For example, a node's availability might be assessed as 85 points. This process accurately quantifies the actual carrying capacity of a node under current network fluctuations.

[0027] It should be noted that after obtaining the availability score, the matching score needs to be calculated in conjunction with the transmission rate of the first data packet. If the required transmission rate of the first data packet is 100 megabits per second, combined with the aforementioned availability score of 85, a matching score of 92 is calculated through weighted summation. If the preset matching threshold is 80, since 92 is greater than 80, the node is included in the second set of computing power points. By sorting all edge nodes in the second set of computing power points in descending order of matching score, the priority of each edge node is obtained. Finally, based on the priority, the node ranked first in the second set of computing power points is determined as the target computing power point, thereby achieving precise scheduling of computing resources and reliable offloading of data packets when network jitter is abnormal.

[0028] S104. The speech audio stream and facial image frame sequence in the first multimodal data are jointly recognized by the target edge computing node to obtain a first text sequence containing the pronunciation unit and lip shape alignment results.

[0029] First multimodal data from the edge is acquired, and separated to obtain an initial audio stream and an initial image frame. Based on the initial audio stream and the initial image frame, a timestamp synchronization comparison rule is used to filter and obtain a synchronized audio stream and a synchronized image frame. Acoustic vectors are extracted from the synchronized audio stream using a Hidden Markov Model to determine an acoustic vector sequence containing multiple phonemes. The synchronized image frame is cropped based on the timestamps of the acoustic vector sequence to obtain a target image frame. A convolutional neural network is used to extract the mouth region pixels of the target image frame to obtain a lip shape matrix. The acoustic vector sequence and the lip shape matrix are acquired. If the time span of the acoustic vector sequence matches the time span of the lip shape matrix, the distance between the phoneme and the lip shape matrix is ​​calculated to determine an alignment value. A preset mapping table is queried based on the alignment value to obtain a combined lip shape stream. Character conversion is performed on the combined lip shape stream to obtain a first text string.

[0030] Specifically, acquiring and separating the first multimodal data at the edge is fundamental to achieving edge-side audio-video collaborative analysis. The input first multimodal data, containing a mixed audio-video stream, is separated into an initial audio stream and initial image frames using a demultiplexing algorithm. Subsequently, a timestamp synchronization comparison rule is employed for filtering. With a timestamp tolerance threshold of 20 milliseconds, if the difference between the timestamp of an audio frame and the timestamp of an initial image frame is less than 20 milliseconds, they are considered synchronized. This results in the output of a synchronized audio stream and a synchronized image frame, ensuring consistency in the benchmark for subsequent feature extraction.

[0031] An example is provided using a Hidden Markov Model (HMM) for acoustic vector extraction from a synchronized audio stream. The input synchronized audio stream is processed by extracting its Mel-frequency cepstral coefficients (MFCCs). This is then decoded using the state transition probability matrix of the HMM, outputting an acoustic vector sequence containing 64 phonemes. Simultaneously, based on the start and end timestamps of this acoustic vector sequence, target image frames corresponding to the specified time periods are extracted from the synchronized image frames. A convolutional neural network is then used to process these target image frames, locating and extracting 64x64 pixels from the mouth region. This is then transformed into a two-dimensional mouth shape matrix, thereby achieving a quantitative representation of the visual features.

[0032] In one possible implementation, after obtaining the aforementioned acoustic vector sequence and lip shape matrix, cross-modal alignment analysis is required. First, it is determined whether the difference between the time span of the acoustic vector sequence and the time span of the lip shape matrix is ​​less than 50 milliseconds. If they match, a dynamic time warping algorithm is used to calculate the Euclidean distance between the feature vectors of the articulators and the feature vectors of the lip shape matrix, and this distance value is used as the alignment value. When the alignment value is less than a preset distance threshold of 15, a pre-constructed audiovisual mapping table is consulted based on this alignment value to obtain the corresponding articulation and lip shape combination stream. Finally, a natural language processing model is used to decode and convert the articulation and lip shape combination stream, outputting the first text string, thus completing the text-based parsing of the edge-end multimodal data.

[0033] S105. The first text sequence is classified using a classifier to obtain the first evaluation result that reflects the effectiveness of oral teaching.

[0034] A first text sequence is obtained and segmented using a word segmentation tool to obtain a second text sequence. Based on the second text sequence, colloquial words are extracted using word frequency statistics to obtain their frequencies. If the frequency of colloquial words is greater than a preset frequency threshold, the second text sequence is vectorized to obtain a text vector set. The text vector set is then acquired and classified using a support vector machine classifier to obtain classification labels. Based on the classification labels, a preset teaching evaluation standard library is matched to determine the first evaluation result corresponding to the classification label. If the first evaluation result meets preset achievement conditions, core evaluation words are extracted from the first evaluation result to obtain the final evaluation conclusion.

[0035] In one possible implementation, the system receives a first text sequence output from the front-end speech and lip-sync recognition algorithm and segments it using a Hidden Markov Model (HMM) word segmentation algorithm. The input is a continuous text containing the student's reading, which is segmented to obtain an independent vocabulary set, i.e., a second text sequence. This sequence is then traversed, and a pre-defined spoken language feature dictionary is used for comparison, counting the frequency of spoken words such as "ah," "um," and "that." If the count finds that a spoken word in a 50-word sequence appears 6 times, exceeding a preset frequency threshold of 5 times, the subsequent vectorization process is triggered.

[0036] For example, for a second text sequence that meets the threshold condition, a continuous bag-of-words model is used for vector transformation. The input is segmented vocabulary, mapped through a pre-trained word embedding matrix, and outputs a 128-dimensional text vector set. This text vector set is then input into a pre-trained support vector machine classifier. This classifier uses a radial basis function kernel to partition the text vectors in the high-dimensional space into a hyperplane, outputting corresponding classification labels, such as poor fluency or hesitant expression.

[0037] It should be noted that after obtaining the classification labels, they are mapped and matched with a locally constructed teaching evaluation standard library. This standard library pre-establishes a correspondence between labels and assessment levels. When a label indicating poor fluency is matched, the first assessment result is determined to be Level 3 in oral expression coherence. If this Level 3 result meets the preset intervention criteria, then a conditional random field algorithm is used to extract coherence as the core evaluation term from the descriptive text of the first assessment result, thereby generating the final assessment conclusion and completing the entire quantitative assessment process for oral expression ability in teaching.

[0038] S106. Obtain the transmission delay of the first evaluation result between the target edge computing node and the central server, and determine whether the transmission delay is greater than a preset delay threshold.

[0039] The transmission latency of the first evaluation result from the target edge computing node to the central server is obtained. If the transmission latency is greater than a preset latency threshold, a first latency exceedance state is determined. Based on the first latency exceedance state, the network packet loss rate of the target edge computing node in the transmission segment is obtained, and the network packet loss rate is classified using a random forest algorithm to obtain a packet loss severity level. For the packet loss severity level, the backup routing table corresponding to the central server is obtained, and a first routing path matching the packet loss severity level is extracted from the backup routing table. The first evaluation result is retransmitted using the first routing path to obtain the transmission latency of the second evaluation result. It is determined whether the transmission latency of the second evaluation result is less than the preset latency threshold. If the transmission latency of the second evaluation result is less than the preset latency threshold, a link recovery state is determined. Based on the link recovery state, the node address corresponding to the first routing path is extracted, and an updated master routing configuration table is generated.

[0040] In one possible implementation, in oral language teaching and assessment scenarios, edge computing nodes are typically deployed near student terminals to perform preliminary processing of spoken pronunciation audio and generate initial assessment results. Obtaining the transmission latency from this node to the central server is crucial for ensuring real-time feedback in oral language teaching interactions. Specifically, the transmission latency is calculated by encapsulating a sending timestamp in the header of the assessment result data packet and comparing it with the local time on the central server. If the measured transmission latency is 150 milliseconds, while the preset latency threshold is 100 milliseconds, since 150 is greater than 100, the first latency exceedance state is triggered. This state indicates a risk of congestion in the upload channel for the oral language assessment data, which may cause stuttering when students receive pronunciation correction guidance.

[0041] For example, based on the determined first latency exceeding the standard, it is necessary to further obtain the network packet loss rate of the target edge computing node in the transmission segment. Assuming that the proportion of lost spoken assessment data packets in the past minute, collected by a network monitoring probe, is 5, this value is used as an input feature and fed into a pre-trained random forest algorithm model. This algorithm model internally constructs 50 decision trees, each evaluating the packet loss rate and the number of concurrent connections according to a set splitting rule. After a comprehensive vote by all decision trees, if more than half of the decision trees determine that the current network condition has deteriorated, the packet loss severity level is output as level three. This process transforms a single network indicator into a specific business intervention level.

[0042] It should be noted that for the aforementioned three levels of packet loss severity, a pre-configured backup routing table from the central server is required. A first routing path that strictly matches the three levels is extracted from this table; this path avoids the currently congested network nodes. The first routing path is used to retransmit the blocked first evaluation results, and the transmission latency of the second evaluation results is continuously monitored. If the measured latency after retransmission drops to 80 milliseconds, and is determined to be less than the preset latency threshold of 100 milliseconds, then the network is considered to have entered a link recovery state. Based on this link recovery state, the addresses of relay nodes at each level included in the first routing path are extracted, an updated master routing configuration table is generated, and distributed to maintain the stability of oral teaching evaluation data transmission.

[0043] S107. If the transmission delay is greater than the preset delay threshold, then a computing power scheduling instruction is generated based on the transmission delay.

[0044] The system obtains a first data packet from the transmitting end, extracts the sending timestamp and receiving timestamp of the first data packet, and calculates the difference between the receiving timestamp and the sending timestamp to obtain an initial latency value. It then obtains a target latency threshold from a preset threshold library. If the initial latency value is greater than the target latency threshold, it calculates the difference between the initial latency value and the target latency threshold to obtain a latency excess value. The system obtains the service identifier corresponding to the latency excess value, constructs a two-dimensional feature matrix based on the service identifier and the latency excess value, and performs classification processing on the two-dimensional feature matrix using a support vector machine algorithm to obtain a computing power level label. Based on the computing power level label, it matches corresponding candidate node groups from a preset computing power pool, obtains the current load rate of each node in the candidate node group, and sorts the candidate node group in ascending order of the current load rate to obtain a target node sequence. The system selects the first node in the target node sequence as the target computing power node, and generates a computing power scheduling instruction set based on the network address of the target computing power node and the latency excess value. Obtain the first scheduling instruction from the computing power scheduling instruction set, and trigger the target computing power node to perform business data processing through the first scheduling instruction to obtain the business processing result.

[0045] For example, in a data interaction scenario of an edge computing network, after obtaining the first data packet from the transmitting end, the sending timestamp and the receiving timestamp upon arrival at the receiving end are extracted from the header of the data packet. Assuming the sending timestamp is recorded as 1000 milliseconds and the receiving timestamp as 1050 milliseconds, the difference between the two yields an initial latency value of 50 milliseconds. The target latency threshold corresponding to the current video streaming service is retrieved from a preset threshold library, which is 30 milliseconds. Since 50 milliseconds is greater than 30 milliseconds, the latency excess value is further calculated to be 20 milliseconds.

[0046] In one possible implementation, the service identifier corresponding to the aforementioned latency excess value is obtained; for example, this service identifier is identifier code 101 representing high-definition video rendering. The service identifier code 101 and the latency excess value 20 are used together as feature dimensions to construct a two-dimensional feature matrix. This two-dimensional feature matrix is ​​input into a pre-trained support vector machine (SVM) algorithm model. The SVM algorithm maps different features to the corresponding classification space by finding the optimal hyperplane, thereby outputting a computing power level label. Assuming that after classification, the output computing power level label is a high computing power demand label.

[0047] Specifically, based on the obtained high computing power demand tags, a search is performed from a pre-defined edge computing power pool to match candidate node groups that meet the tag requirements. Assuming this candidate node group contains three edge computing power nodes, their current load rates are obtained, for example, 60%, 20%, and 40% respectively. These three nodes are then sorted in ascending order of load rate to obtain a target node sequence. The first node in this sequence, i.e., the node with a load rate of 20%, is extracted as the target computing power node. Subsequently, a computing power scheduling instruction set is generated by combining the network address of this target computing power node with the aforementioned latency excess value of 20. The first scheduling instruction in this instruction set is extracted, and the previously delayed video rendering data packets are redirected to the target computing power node, triggering it to execute specific video rendering data processing tasks, ultimately obtaining the rendered business processing result.

[0048] S108. The global computing power pool is re-divided using the computing power scheduling instruction to obtain an updated computing power distribution matrix that represents the resource proportion of each edge computing power node.

[0049] The system obtains scheduling orders and a computing power pool, extracts the resource quantity of the node group in the computing power pool, and obtains a first state table. Based on the first state table, it uses a K-means clustering algorithm to cluster the node group, obtaining a first node cluster. By extracting the load rate and remaining quantity of the first node cluster, it calculates the matching degree between the load rate and the remaining quantity, determining a first matching degree sequence. If the matching degree in the first matching degree sequence is greater than a preset threshold, it extracts the task column corresponding to the matching degree, obtaining a first task column. It uses a decision tree algorithm to classify the first task column, obtaining a first priority sequence. Based on the first priority sequence and the first node cluster, it generates a partitioning graph for the computing power pool, obtaining a first partitioning graph. It extracts the proportion value of the node group from the first partitioning graph, generates a distribution matrix, and obtains an updated computing power distribution matrix.

[0050] In one possible implementation, after obtaining the scheduling order and the computing power pool, the number of central processing unit cores and memory capacity of the node group in the computing power pool are extracted as resource quantities to construct the first state table.

[0051] For example, the computing power pool contains 100 nodes. The first node is identified as having a 64-core processor and 256 gigabytes of memory, and this is recorded in the first state table. Based on this state table, the 100 nodes are clustered using the mean clustering algorithm. The input is the resource feature vector of each node, and the number of cluster centers is set to 3. After iterative calculation, the first node cluster is output in three categories: high resources, medium resources, and low resources. This approach categorizes physically dispersed computing power nodes according to their resource endowments, providing fundamental data support for subsequent precise scheduling.

[0052] It should be noted that, for the obtained first node cluster, the current load rate and remaining memory of each node within it are further extracted.

[0053] For example, a target node in a high-resource node cluster is found to have a load rate of 70% and a remaining memory of 80 gigabytes. The load rate and remaining memory are weighted and summed, with the load rate weight set to a negative value and the remaining memory weight set to a positive value. This yields a matching score of 85 for the node, forming the first matching score sequence. If the matching score of the target node in this sequence (85) is greater than a preset threshold of 80, it indicates that the node is in good condition. The pending task queue assigned to this node is then extracted, resulting in the first task queue. This process achieves a quantitative assessment of node health and capacity, ensuring that subsequent tasks are assigned to nodes with sufficient available resources.

[0054] In one possible implementation, after obtaining the first task list, a decision tree algorithm is used to classify it. The input consists of the business type identifier and data packet size of each task in the first task list. The decision tree's node splitting rules are used to determine the priority, and the output is a first priority sequence containing urgent and ordinary tasks.

[0055] For example, the video stream rendering task is determined to be of high priority. Subsequently, based on the mapping relationship between the first priority sequence and the first node cluster, a first partitioning map for the computing power pool is generated. This partitioning map is used to extract the percentage (60%) of high-priority tasks occupying high-resource nodes. This percentage is then filled into a two-dimensional matrix to generate a distribution matrix, ultimately obtaining the updated computing power distribution matrix. This series of operations achieves dynamic adaptation of computing power resources to business priorities, optimizing the overall resource configuration structure of the computing power network.

[0056] S109. Based on the updated computing power distribution matrix, distribute teaching resources that match the cross-border teaching terminal to the cross-border teaching terminal to obtain a resource allocation sequence that includes the correspondence between teaching resources and target terminals.

[0057] The updated computing power distribution matrix is ​​obtained, and each terminal number and its corresponding computing power value are extracted from the matrix. Based on each terminal number and its corresponding computing power value, a hierarchical clustering algorithm is used for clustering to obtain a terminal cluster set at the first computing power level. For the terminal cluster set at the first computing power level, a pre-established set of teaching packages is obtained, and the matching degree between the terminal cluster set at the first computing power level and the teaching package set is calculated. If the matching degree is greater than a preset matching threshold, the target teaching package corresponding to the terminal cluster set at the first computing power level is determined. The target teaching package corresponding to the terminal cluster set at the first computing power level is obtained, and its resource quantity is extracted. An initial allocation table is generated based on the resource quantity and the target terminals in the terminal cluster set at the first computing power level. The target teaching package is distributed to the target terminals according to the initial allocation table, resulting in a final allocation table containing the correspondence between target teaching packages and target terminals.

[0058] In one possible implementation, after obtaining the updated computing power distribution matrix, the terminal numbers and corresponding computing power values ​​contained therein are extracted as input data. A cohesive hierarchical clustering algorithm is then used to process this data. Specifically, each terminal is initially treated as an independent cluster, the Euclidean distance between the computing power values ​​of different terminals is calculated, the terminals with the smallest distances are found and merged, and this process is repeated until the hierarchical partitioning conditions are met.

[0059] For example, if 100 terminal numbers are extracted, and their computing power values ​​are distributed between 500 and 2000, hierarchical clustering can be used to divide terminals with computing power values ​​between 1500 and 2000 into a first-level computing power cluster. This clustering method can group terminals with equivalent computing capabilities together.

[0060] In one possible implementation, the obtained terminal cluster set at the first computing power level needs to be matched with a pre-established set of teaching packages. The teaching package set contains data such as 3D virtual simulation teaching packages. The computing power requirement indicators of the teaching packages are extracted, and a cosine similarity calculation is performed between these indicators and the average computing power value of the terminal cluster set at the first computing power level to determine the matching degree.

[0061] For example, a matching threshold of 80 is set. When the calculated matching degree between a certain 3D teaching package and the terminal set of the first computing power level reaches 85, since 85 is greater than the threshold of 80, the teaching package can be identified as the target teaching package. This process ensures that tasks with high computing power requirements can be accurately assigned to terminals with the corresponding capabilities.

[0062] It should be noted that after determining the target teaching package, its resource quantity is extracted, for example, 500 megabytes of storage space. An initial allocation table is generated based on this resource quantity and the target terminals in the first computing power level terminal cluster. This table records the mapping relationship between terminal identifiers and resource quantities. Subsequently, the target teaching package is sent to the target terminal according to the initial allocation table. After transmission, the actual correspondence between the teaching package and the target terminal is recorded to obtain the final allocation table. The above process realizes the dynamic adaptation of teaching resources and edge computing terminals, improving the success rate of teaching package distribution and resource utilization.

[0063] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A multimodal AGI cross-border Chinese oral assessment teaching system and method, characterized in that, The method includes: The first multimodal data collected by the cross-border teaching terminal is acquired. The first multimodal data includes a voice audio stream and a facial image frame sequence. Temporal features are extracted from the first multimodal data through a neural network to obtain a first feature vector used to characterize the link transmission behavior. The network jitter value of the current transmission link is calculated based on the delay fluctuation component in the first feature vector, and it is determined whether the network jitter value is greater than a pre-established jitter threshold. If the network jitter value is greater than the pre-established jitter threshold, then an edge node allocation instruction is generated for the first multimodal data, and a target edge computing node is determined from the candidate edge computing node set according to the edge node allocation instruction; The target edge computing node performs joint recognition processing on the speech audio stream and facial image frame sequence in the first multimodal data to obtain a first text sequence containing the pronunciation unit and lip shape alignment results; The first text sequence is classified using a classifier to obtain the first evaluation result that reflects the effectiveness of oral language teaching. The transmission latency of the first evaluation result between the target edge computing node and the central server is obtained, and it is determined whether the transmission latency is greater than a preset latency threshold. If the transmission delay is greater than the preset delay threshold, a computing power scheduling instruction is generated based on the transmission delay. The global computing power pool is re-divided using the aforementioned computing power scheduling instructions to obtain an updated computing power distribution matrix that characterizes the resource proportion of each edge computing power node; Based on the updated computing power distribution matrix, teaching resources matching the cross-border teaching terminal are distributed to the cross-border teaching terminal to obtain a resource allocation sequence containing the correspondence between teaching resources and target terminals.

2. The multimodal AGI cross-border Chinese oral assessment and teaching system and method according to claim 1, characterized in that, The first multimodal data collected by the cross-border teaching terminal includes a speech audio stream and a facial image frame sequence. Temporal features are extracted from the first multimodal data using a neural network to obtain a first feature vector characterizing the link transmission behavior, including: First multimodal data is obtained from the teaching end. The first multimodal data includes audio streams and image frames. A neural network is used to perform temporal feature extraction on the first multimodal data to obtain a first feature vector. Based on the first feature vector, extract the first feature value of the first feature vector on the time axis, determine whether the first feature value is greater than a preset threshold, and if the first feature value is greater than the preset threshold, determine that the first feature vector is abnormal and obtain the abnormal feature vector. For the abnormal feature vector, obtain the transmitting end identifier and receiving end identifier corresponding to the abnormal feature vector, and query the pre-established link state table through the transmitting end identifier and the receiving end identifier to obtain the first link state data; Based on the first link status data, the packet loss rate and latency contained in the first link status data are parsed, and the packet loss rate and latency are classified using a decision tree algorithm to obtain the link congestion level. The link congestion level is obtained, and it is determined whether the link congestion level matches a preset severe congestion condition. If the link congestion level matches the severe congestion condition, downsampling processing is performed on the audio stream and the image frame to obtain second multimodal data. For the second multimodal data, extract the second feature value of the second multimodal data, construct a second vector set based on the second feature value, and output the second vector set to the receiving end.

3. The multimodal AGI cross-border Chinese oral assessment and teaching system and method according to claim 1, characterized in that, The step of calculating the network jitter value of the current transmission link based on the delay fluctuation component in the first feature vector, and determining whether the network jitter value is greater than a pre-established jitter threshold, includes: Obtain the first feature vector from the target node, extract the time delay fluctuation component contained in the first feature vector, and obtain the first fluctuation value; Based on the first fluctuation value, the network jitter value of the current transmission link is calculated using time series analysis tools to obtain the first jitter value; Obtain a pre-established jitter threshold value, determine whether the first jitter value is greater than the jitter threshold value, and if the first jitter value is greater than the jitter threshold value, then determine the first abnormal state identifier; Based on the first abnormal state identifier, the historical delay value of the current transmission link is anomaly detected using the isolated forest algorithm to obtain the second abnormal state identifier; Based on the second abnormal state identifier, the link-end state quantity of the current transmission link is extracted to obtain the first state quantity; Based on the first state quantity, determine whether the first state quantity matches a preset fault determination value. If the first state quantity matches the fault determination value, then determine the target blocking command.

4. The multimodal AGI cross-border Chinese oral assessment and teaching system and method according to claim 1, characterized in that, If the network jitter value is greater than the pre-established jitter threshold, then an edge node allocation instruction is generated for the first multimodal data, and a target edge computing node is determined from the candidate edge computing node set according to the edge node allocation instruction, including: Obtain the jitter value of the current network environment; if the jitter value is greater than a pre-established judgment value, generate a first allocation order for the first data packet. According to the first allocation order, a candidate set containing multiple edge nodes is obtained, and the edge nodes in the candidate set are clustered using the K-means clustering algorithm to obtain the first computing power point set. The load rate and latency of each edge node are obtained from the first set of computing power points. A node state matrix is ​​constructed using the load rate and latency to obtain the first state matrix. The random forest algorithm is used to extract features from the first state matrix to determine the availability of each edge node in the first set of computing power points; The matching degree is calculated based on the availability and the transmission rate of the first data packet. If the matching degree is greater than a preset matching threshold, a second set of computing power points is determined. By sorting the edge nodes in the second set of computing power points, the priority of each edge node is obtained, and the target computing power point is determined in the second set of computing power points according to the priority.

5. The multimodal AGI cross-border Chinese oral assessment and teaching system and method according to claim 1, characterized in that, The step of jointly recognizing the speech audio stream and facial image frame sequence in the first multimodal data through the target edge computing node to obtain a first text sequence containing the alignment results of articulation units and lip shapes includes: Acquire the first multimodal data at the edge, separate the first multimodal data, and obtain the initial audio stream and the initial image frame; Based on the initial audio stream and the initial image frame, a timestamp synchronization comparison rule is used to filter and obtain a synchronized audio stream and a synchronized image frame. Acoustic vectors are extracted from the synchronized audio stream using a hidden Markov model to determine an acoustic vector sequence containing multiple phonemes. The target image frame is obtained by extracting the synchronized image frame based on the timestamp of the acoustic vector sequence, and the mouth region pixels of the target image frame are extracted using a convolutional neural network to obtain the mouth shape matrix. Obtain the acoustic vector sequence and the mouth shape matrix. If the time span of the acoustic vector sequence matches the time span of the mouth shape matrix, calculate the distance between the phonetic element and the mouth shape matrix to determine the alignment value. The pronunciation mouth shape combination stream is obtained by querying a preset mapping table based on the alignment value; The first text string is obtained by converting characters through the combination of pronunciation and lip movements.

6. The multimodal AGI cross-border Chinese oral assessment and teaching system and method according to claim 1, characterized in that, The process of classifying the first text sequence using a classifier to obtain a first evaluation result reflecting the effectiveness of oral language teaching includes: Obtain the first text sequence, and then segment the first text sequence using a word segmentation tool to obtain the second text sequence; Based on the second text sequence, the colloquial words in the second text sequence are extracted using a word frequency statistics method to obtain the frequency of colloquial words; If the frequency of the spoken words is greater than a preset frequency threshold, then the second text sequence is subjected to vector transformation processing to obtain a text vector set; The text vector set is obtained, and a support vector machine classifier is used to classify the text vector set to obtain classification labels; Based on the classification labels, a preset teaching evaluation standard library is matched to determine the first evaluation result corresponding to the classification labels; If the first evaluation result meets the preset compliance conditions, then the core evaluation words in the first evaluation result are extracted to obtain the final evaluation conclusion.

7. The multimodal AGI cross-border Chinese oral assessment teaching system and method according to claim 1, characterized in that, The step of obtaining the transmission latency of the first evaluation result between the target edge computing node and the central server, and determining whether the transmission latency is greater than a preset latency threshold, includes: The transmission latency of the first evaluation result from the target edge computing node to the central server is obtained. If the transmission latency is greater than the preset latency threshold, the first latency exceedance status is determined. Based on the first latency exceeding the standard state, the network packet loss rate of the target edge computing node in the transmission segment is obtained, and the network packet loss rate is classified by the random forest algorithm to obtain the packet loss severity level; For the packet loss severity level, obtain the backup routing table corresponding to the central server, and extract the first routing path that matches the packet loss severity level from the backup routing table; The first evaluation result is retransmitted using the first routing path to obtain the transmission delay of the second evaluation result; Determine whether the transmission delay of the second evaluation result is less than the preset delay threshold. If the transmission delay of the second evaluation result is less than the preset delay threshold, then determine the link recovery status. Based on the link recovery status, extract the node address corresponding to the first routing path and generate an updated master routing configuration table.

8. The multimodal AGI cross-border Chinese oral assessment and teaching system and method according to claim 1, characterized in that, If the transmission delay is greater than the preset delay threshold, a computing power scheduling instruction is generated based on the transmission delay, including: The first data packet is obtained from the transmission end. The sending timestamp and receiving timestamp of the first data packet are extracted. The difference between the receiving timestamp and the sending timestamp is calculated to obtain the initial delay value. Obtain the target latency threshold from the preset threshold library. If the initial latency value is greater than the target latency threshold, calculate the difference between the initial latency value and the target latency threshold to obtain the latency excess value. Obtain the service identifier corresponding to the latency excess value, construct a two-dimensional feature matrix based on the service identifier and the latency excess value, and use the support vector machine algorithm to classify the two-dimensional feature matrix to obtain the computing power level label; Based on the computing power level label, the corresponding candidate node group is matched from the preset computing power pool, the current load rate of each node in the candidate node group is obtained, and the candidate node group is sorted in ascending order of the current load rate to obtain the target node sequence. The first node in the target node sequence is selected as the target computing power node, and a computing power scheduling instruction set is generated based on the network address of the target computing power node and the latency excess value. Obtain the first scheduling instruction from the computing power scheduling instruction set, and trigger the target computing power node to perform business data processing through the first scheduling instruction to obtain the business processing result.

9. The multimodal AGI cross-border Chinese oral assessment and teaching system and method according to claim 1, characterized in that, The process of re-dividing the global computing power pool using the computing power scheduling instructions to obtain an updated computing power distribution matrix representing the resource proportion of each edge computing power node includes: Obtain the scheduling order and computing power pool, extract the resource amount of the node group in the computing power pool, and obtain the first state table; Based on the first state table, the node group is clustered using the K-means clustering algorithm to obtain the first node cluster; By extracting the load rate and remaining amount of the first node cluster, the matching degree between the load rate and the remaining amount is calculated, and a first matching degree sequence is determined. If the matching degree in the first matching degree sequence is greater than a preset threshold, then the task column corresponding to the matching degree is extracted to obtain the first task column; The first task column is classified using a decision tree algorithm to obtain the first priority sequence; Based on the first priority sequence and the first node cluster, a partitioning graph for the computing power pool is generated to obtain the first partitioning graph; The proportion of the node group is extracted from the first partitioning map to generate a distribution matrix, and the updated computing power distribution matrix is ​​obtained.

10. The multimodal AGI cross-border Chinese oral assessment and teaching system and method according to claim 1, characterized in that, The step of distributing teaching resources matching the cross-border teaching terminal to the cross-border teaching terminal according to the updated computing power distribution matrix, and obtaining a resource allocation sequence containing the correspondence between teaching resources and target terminals, includes: Obtain the updated computing power distribution matrix and extract each terminal number and its corresponding computing power value from the computing power distribution matrix; Based on each terminal number and its corresponding computing power value, a hierarchical clustering algorithm is used to perform clustering processing to obtain the terminal cluster set of the first computing power level; For the terminal cluster set of the first computing power level, obtain the pre-established teaching package set, and calculate the matching degree between the terminal cluster set of the first computing power level and the teaching package set; If the matching degree is greater than the preset matching threshold, then the target teaching package corresponding to the terminal cluster set of the first computing power level is determined; Obtain the target teaching package corresponding to the terminal cluster set of the first computing power level, extract the resource amount of the target teaching package, and generate an initial allocation table based on the resource amount and the target terminals in the terminal cluster set of the first computing power level. Based on the initial allocation table, the target teaching package is sent to the target end, and the final allocation table containing the correspondence between the target teaching package and the target end is obtained.

Images