A foreign language classroom learning behavior analysis method and system based on multi-modal data

Abstract

The present application relates to the technical fields of computer data processing and digital signal processing, and discloses a foreign language classroom learning behavior analysis method and system based on multi-modal data. The present application acquires classroom multi-modal data through an audio acquisition device and a video acquisition device, identifies a classroom acoustic interaction state, and dynamically calls a corresponding voice behavior analysis parameter set to adaptively adjust a voice activity detection window length, a time domain framing window length, a silence duration threshold, and a voice response delay determination threshold. Voice activity timing feature parameters of a target spatial observation node are extracted, individual statistical baselines and group statistical baselines are constructed, deviation data is calculated using double standardization distances, and an abnormal state marker signal is generated. The present application realizes audio and video feature fusion, dynamic parameter switching, and double baseline collaborative anomaly detection, and can improve the accuracy of voice activity detection and the reliability of anomaly recognition in a complex acoustic environment of a foreign language classroom.

Description

Technical Field

[0001] This invention relates to the field of SS technology, and in particular to a method and system for analyzing foreign language classroom learning behavior based on multimodal data. Background Technology

[0002] In foreign language classroom teaching scenarios, there are two typical complex acoustic interaction states: teacher-dominated state (one-way teacher explanation and demonstration) and student-dominated state (students repeating from multiple sound sources and engaging in group interactive discussions). Under these two states, the temporal speech behavior sequence characteristics exhibited by learners show fundamental statistical differences.

[0003] Learners' speech activities in foreign language classrooms have unique temporal structure characteristics that distinguish them from ordinary conference speech and natural conversational speech. In scenarios such as foreign language shadowing, oral imitation, and immediate response, there are numerous non-syntactic boundary pauses caused by language organization, pronunciation correction, and semantic retrieval. These pauses manifest as short, random, sudden silences in the temporal domain, and their statistical distribution differs significantly from normal syntactic pauses in natural speech.

[0004] Currently, existing technologies in the field of classroom behavior analysis mainly include the following categories: Publication number CN115547304A discloses a classroom analysis method and device. This scheme segments the audio data of individual sound sources in the classroom using voiceprint recognition, calculates the number of turn-taking between teachers and students, and between students, and determines the interaction structure type of the classroom based on the number of turns. This scheme only outputs static interaction structure labels and does not adjust the underlying feature analysis and decision-making logic of the system. For differences in data distribution under different interaction structures, uniform analysis parameters are still used for processing.

[0005] Publication No. CN120850228B discloses a method and system for processing multimodal classroom data based on deep learning. This scheme proposes a multimodal data alignment method that conforms to the teaching rhythm, assigning higher alignment weights to semantically key segments such as teacher explanations and teacher-student interactions, achieving non-linear alignment that conforms to the teaching rhythm. The rhythm perception in this scheme is only used to optimize alignment accuracy on the time axis, representing an adaptive adjustment at the data preprocessing level, and does not involve dynamic switching of the analysis parameters themselves.

[0006] Publication number CN120541781A discloses a classroom behavior analysis system based on an adaptive algorithm. This scheme collects multimodal classroom behavior data, extracts real-time student posture vectors, speech spectrum features, interaction response latency, and semantic tags for teaching content, and performs hierarchical association modeling and dynamic priority ranking through a multi-head attention mechanism. While this scheme uses a multi-head attention mechanism for multimodal fusion, it does not address dynamically adjusting the underlying analysis parameters based on the switching of classroom acoustic interaction states.

[0007] Publication number CN117079667A discloses a scene classification method, apparatus, device, and readable storage medium. This scheme acquires student and teacher audio datasets, rearranges and merges audio atomic segments in chronological order, performs a two-level merging and partitioning, and combines this with a scene classification judgment formula to obtain the time period information corresponding to each scene. The scene classification results of this scheme are used to output time period labels and are not used to drive the dynamic adjustment of system analysis parameters.

[0008] In the fields of general speech signal processing and anomaly detection, various adaptive parameter adjustment and statistical baseline modeling schemes exist. For example, adaptive speech enhancement schemes determine the scene to which an audio frame belongs through a scene recognition model and select different methods to control the output ratio of the audio signal according to the scene to achieve adaptive speech enhancement. However, its adaptive adjustment occurs at the signal preprocessing layer to optimize audio quality and does not involve the adjustment of judgment criteria at the behavior analysis decision layer. In the field of network security, there are schemes that use individual baselines to detect user behavior anomalies, but these belong to different technical fields and do not involve classroom acoustic behavior analysis scenarios.

[0009] In summary, the aforementioned existing technologies share the following common problems: First, existing classroom analysis systems process data with fixed parameters under different acoustic conditions. In the teacher-dominated state, the frequency of turn-taking is low and students mainly listen; in the student-dominated state, the frequency of turn-taking is high and students mainly produce speech actively. There are fundamental differences in the statistical distribution of speech behavior characteristics between the two states.

[0010] Second, the set of analysis parameters preset by the fixed parameter system can only match a single acoustic state. When the acoustic interaction state changes, a systematic mismatch occurs between the fixed parameters and the current data distribution, leading to an increase in the false alarm rate and false negative rate of anomaly detection.

[0011] Third, existing technologies fail to recognize the unique characteristics of non-syntactic boundary pause durations in foreign language classrooms. Traditional fixed-parameter speech activity detection mechanisms often misidentify non-syntactic boundary pause durations as speech end boundaries, leading to incorrect segmentation of continuous speech streams. This further causes distortion in speech response delay calculations, shifts in turn-taking statistics, and an increased false alarm rate for anomaly detection. Existing technologies do not design dynamically adjusted detection window mechanisms specifically for the non-syntactic boundary pause durations unique to foreign language classrooms.

[0012] Fourth, the adaptive adjustments in existing technologies all occur at the signal preprocessing layer or the label output layer, without involving the dynamic reconstruction of parameters at the analysis and decision layer. The results of scene classification are only used to output time period labels, and rhythm perception is only used to optimize time alignment; neither uses the results of state perception to drive the switching of the analysis parameter set. Summary of the Invention

[0013] Given that existing classroom behavior analysis systems use fixed parameters to process data under different acoustic interaction states, failing to recognize that switching acoustic interaction states can cause non-stationary drift in the statistical distribution of speech behavior features, and that fixed analysis parameters cannot simultaneously accommodate low-interaction-density long silence scenarios and high-interaction-density short burst scenarios, leading to increased false alarm and false negative rates in anomaly detection, a foreign language classroom learning behavior analysis method and system based on multimodal data is proposed.

[0014] The purpose is to address the non-stationary temporal feature drift caused by different acoustic interaction states in foreign language classrooms by dynamically reconstructing the underlying speech behavior analysis parameters through acoustic state perception, so that the analysis standard can be adaptively aligned with the current data distribution, thereby reducing the false alarm rate and false negative rate of the system under complex acoustic scene switching conditions.

[0015] In this application, learning behavior specifically refers to multimodal physical behavior data acquired by audio and video acquisition devices that can be objectively represented through time-series data, including quantifiable time-series feature sequences such as speech activity timing, turn-taking characteristics, response delay feature sequences, and non-syntactic boundary pause duration.

[0016] Learning behavior analysis refers to the technical process of performing low-level digital signal processing, statistical distribution modeling, and distribution deviation detection on the aforementioned multimodal data. It does not include any inference or evaluation of the learner's internal subjective state, learning attitude, or ability. In this application, "learner" is used only to identify the data source object in the classroom acoustic scenario.

[0017] The technical solution of this invention is a method for analyzing foreign language classroom learning behavior based on multimodal data, comprising: S1. Acquire multimodal data collected by audio and video acquisition devices deployed in the classroom. The multimodal data corresponds to at least one predefined spatial observation node, and the spatial observation node corresponds to an audio and video data observation channel of a predefined spatial area in the classroom.

[0018] S2. Identify the current acoustic interaction state of the classroom based on multimodal data. The acoustic interaction state includes the teacher's voice as the main state and the student's voice as the main state.

[0019] S3. Based on the identified acoustic interaction state, call the speech behavior analysis parameter set corresponding to the current acoustic interaction state from the pre-constructed parameter configuration set. Use the called speech behavior analysis parameter set to dynamically adjust the speech activity detection window length, temporal framing window length, silence duration threshold, or speech response delay judgment threshold, so that the framing boundary, silence segment segmentation result, and speech activity triggering time of the subsequent audio stream change synchronously, so as to reconstruct the data input structure for subsequent feature extraction. The teacher's speech as the main state and the student's speech as the main state correspond to different analysis parameters.

[0020] S4. Extract the speech activity temporal feature parameters of the target spatial observation node from the multimodal data. The speech activity temporal feature parameters include the duration of non-syntactic boundary pauses, speech response delay time and turn-taking frequency. Construct an individual statistical baseline based on the parameter values ​​of the target spatial observation node in the same historical acoustic interaction state, and construct a group statistical baseline based on the real-time parameter values ​​of all spatial observation nodes in the same acoustic interaction state in the current classroom.

[0021] S5. Using the invoked speech behavior analysis parameter set, calculate the deviation data of the current speech activity temporal characteristic parameters relative to the individual statistical baseline and the group statistical baseline.

[0022] S6. When the deviation data exceeds the preset threshold range, an abnormal state marker signal is generated, and based on the abnormal state marker signal, an incremental retraining of the abnormal time sequence segment cache, metadata marker, or analysis parameter set is triggered.

[0023] Furthermore, S2 identifies the current acoustic interaction state of the classroom based on multimodal data, including: extracting the spatial distribution features of speech energy and the turn-taking frequency from the multimodal data; when the proportion of teacher speech energy in the spatial distribution features of speech energy is higher than a preset first threshold and the turn-taking frequency is lower than a preset second threshold, it is identified as a teacher-dominated state; when the proportion of student speech energy in the spatial distribution features of speech energy is higher than a preset third threshold and the turn-taking frequency is higher than a preset fourth threshold, it is identified as a student-dominated state.

[0024] Furthermore, the spatial distribution characteristics of speech energy are extracted, including: extracting the spatial geometric boundary vectors of the teacher's activity area and the student's seating area from the video stream acquired by the video acquisition device; constructing a spatial region weighting matrix using the spatial geometric boundary vectors; and performing element-wise spatial weighting calculations on the multi-channel acoustic energy matrix acquired by the audio acquisition device to determine the spatial distribution characteristics of speech energy.

[0025] Furthermore, the multi-channel acoustic energy matrix is ​​output after beamforming the raw audio stream acquired by the audio acquisition device; the element-wise spatial weighted calculation is achieved by the Hadamard product operation of the weighted matrix and the acoustic energy matrix.

[0026] Furthermore, S3 dynamically adjusts the length of the speech activity detection window, the length of the temporal frame window, the silence duration threshold, or the speech response delay judgment threshold using the called speech behavior analysis parameter set. This includes: increasing the length of the temporal frame window or lengthening the silence duration threshold when the current classroom is identified as being dominated by teacher speech; and decreasing the length of the temporal frame window or shortening the length of the speech activity detection judgment window when the current classroom is identified as being dominated by student speech.

[0027] Furthermore, in S4, an individual statistical baseline for the target space observation nodes is constructed, including: acquiring historical data of the temporal characteristic parameters of the speech activities of the target space observation nodes within a historical time period and under the current acoustic interaction state; updating the moving mean and moving variance using a sliding window and exponential moving average method, with the moving mean update formula being: The formula for updating the sliding variance is: Using the updated moving mean and moving standard deviation as the individual statistical baseline, the updated formula for the moving standard deviation is: in, This is the updated current moving average. This is the historical moving average before the update. The updated current sliding variance, The historical sliding variance before the update. For the current observation value, For the updated sliding standard deviation, The preset learning rate parameter has a range of values. .

[0028] Furthermore, in S4, a group statistical baseline is constructed, including: acquiring real-time data of the temporal characteristic parameters of speech activities of all spatial observation nodes in the current classroom and in the same acoustic interaction state; calculating the arithmetic mean and the overall standard deviation of the real-time data, and using the arithmetic mean and the overall standard deviation as the group statistical baseline.

[0029] Furthermore, the speech behavior analysis parameter set includes a non-syntactic boundary pause duration threshold, a follow-up reading delay time threshold, and a speech response delay time threshold. When the acoustic interaction state is dominated by the teacher's speech, a first non-syntactic boundary pause duration threshold is used, which is greater than a second non-syntactic boundary pause duration threshold. When the acoustic interaction state is dominated by the student's speech, a second non-syntactic boundary pause duration threshold is used, which is less than a first non-syntactic boundary pause duration threshold.

[0030] Furthermore, in S5, the deviation data of the current speech activity temporal feature parameters relative to the individual statistical baseline is calculated, including: calculating the first standardized distance of the current speech activity temporal feature parameter value of the target spatial observation node relative to the individual statistical baseline. The first standardized distance is calculated as follows: in, The first standardized distance, For the current observation value, This is the individual statistical mean. For individual statistical standard deviations; Calculate the deviation data of the current speech activity temporal feature parameters relative to the population statistical baseline, including: Calculate the second standardized distance between the current speech activity temporal feature parameter value of the target spatial observation node and the population statistical baseline. The second standardized distance is calculated as follows: in, The second standardized distance, This is the group statistical mean. The standard deviation of the group; Deviation data exceeding the preset threshold range includes: The absolute value of the first standardized distance is greater than the preset first sensitivity coefficient, expressed as: in, The first sensitivity coefficient, ; Meanwhile, the absolute value of the second standardized distance is greater than the preset second sensitivity coefficient, expressed as: in, This is the second sensitivity coefficient. .

[0031] Furthermore, the present invention also provides a foreign language classroom learning behavior analysis system based on multimodal data, including a processor and a memory, wherein when the processor executes the program stored in the memory, it implements the steps of any of the above methods.

[0032] The analysis of speech behavior sequence features in this application is limited to objective acoustic response features and spatial interaction features in multimodal time-series data. The analysis results are only used as statistical anomaly deviation detection results at the data level and are not used, nor can they be used, to generate learning ability scores, teaching evaluation results, or conclusions on the merits or demerits of behavior.

[0033] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention uses an acoustic interactive state perception mechanism to dynamically call different sets of speech behavior analysis parameters based on whether the teacher's speech is the dominant state or the student's speech is the dominant state. This enables the system's analysis parameters to adaptively match the statistical distribution of speech behavior features in the current classroom, thereby solving the problem of parameter and data distribution mismatch in fixed parameter systems under classroom acoustic state switching conditions and reducing the false alarm rate and false negative rate in anomaly detection.

[0034] 2. This invention dynamically adjusts the length of the speech activity detection window, the length of the time-domain framing window, the silence duration threshold, and the speech response delay determination threshold, so that parameter switching directly affects the underlying digital signal processing flow. This enables adaptive adjustment of the subsequent audio stream framing processing, silence segment recognition, and speech activity detection process, thereby improving the stability and accuracy of speech activity feature extraction under different classroom acoustic interaction states.

[0035] 3. This invention addresses the non-syntactic boundary pause phenomenon unique to foreign language classroom scenarios. Through a state-aware driven dynamic detection window mechanism, it avoids misidentifying normal listening silence as abnormal response delay in teacher-led explanation scenarios and enhances the ability to capture transient bursts of speech in student-led reading and group interaction scenarios, thereby improving the accuracy of speech activity detection in complex acoustic scenarios in foreign language classrooms.

[0036] 4. This invention constructs a spatial region weighted matrix by combining the spatial geometric boundary vector extracted from the video stream with the multi-channel acoustic energy matrix after audio beamforming, thereby realizing spatial fusion of audio and video data at the feature level. It can effectively suppress reverberation interference in non-target areas under conditions of multiple people speaking simultaneously and sound field aliasing, improve the accuracy of speech energy spatial distribution characteristics, and thus enhance the reliability of acoustic state recognition and anomaly detection.

[0037] 5. This invention simultaneously constructs individual statistical baselines and group statistical baselines, and uses dual standardized distances for collaborative deviation detection. This enables the system to not only reflect the long-term behavioral patterns of observation nodes in the target space, but also to make real-time comparisons with the current overall distribution status of the classroom. An abnormal state marker signal is generated only when the individual baseline and the group baseline deviate simultaneously, thereby effectively distinguishing between normal individual fluctuations, common group changes, and sudden abnormal behaviors, and reducing misjudgments in anomaly detection.

[0038] 6. This invention uses a local edge computing architecture to complete multimodal data processing. The original audio and video data is deleted immediately after feature extraction, and only the anonymized feature values ​​are retained for subsequent analysis. This reduces the bandwidth consumption and privacy leakage risks caused by the transmission and storage of the original data, and improves the system's real-time processing capability and deployment adaptability in classroom scenarios. Attached Figure Description

[0039] Figure 1 This invention presents a hardware deployment architecture and multimodal data flow topology diagram for a foreign language classroom learning behavior analysis system based on multimodal data. Figure 2 This is a flowchart of the overall adaptive digital signal processing of a foreign language classroom learning behavior analysis method based on multimodal data according to the present invention. Figure 3 This is the logic diagram for the spatial region energy weighting calculation of the video involved in this invention; Figure 4 This is the dual statistical baseline collaborative verification and closed-loop control diagram of the present invention; Figure 5 This is a closed-loop diagram for the incremental parameter update of the present invention. Detailed Implementation

[0040] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0041] In this application, multimodal data refers to multi-channel synchronous audio timing data collected by a microphone array that is permanently and routinely deployed in the classroom, and video stream data collected by a permanently deployed wide-angle camera. The aforementioned data acquisition devices operate in a routine, unobtrusive manner, and personnel in the classroom do not need to wear any additional active or passive sensors.

[0042] A spatial observation node is a spatial data acquisition unit formed by the mapping of the beam coverage area of ​​a microphone array and the spatial area determined by a video acquisition device. Each spatial observation node corresponds to an audio and video data observation channel for a predefined spatial area in the classroom.

[0043] This system adopts an edge computing architecture, and all multimodal data processing is completed on edge computing terminals deployed locally in the classroom.

[0044] The original audio and video data is deleted immediately after feature extraction is completed in the edge computing terminal. The system only retains the voice behavior sequence feature values ​​extracted from the original data, and does not store or transmit any data that can be restored to the original recording or video.

[0045] The pre-built parameter configuration set is generated in advance during the initial stage of system deployment through offline classroom sample data statistics or experience initialization.

[0046] For example, the window length, silence duration threshold, and voice response delay judgment threshold in the parameter configuration set can be generated offline based on the statistical distribution characteristics of historical classroom sample data through grid search, empirical regression, or threshold optimization, so that each set of analysis parameters matches the data distribution characteristics of the corresponding acoustic interaction state.

[0047] The optimization objective of grid search can be set as follows: Under the corresponding acoustic interaction state, the false alarm rate and false negative rate of anomaly detection based on this parameter set are weighted and minimized.

[0048] The range of each parameter in the parameter search space can be set based on the basic physical constraints of human speech activity. For example, the search range of the temporal framing window length can be set from 10ms to 1000ms, and the search range of the silence duration threshold can be set from 0.1s to 5s.

[0049] In subsequent system operation, incremental retraining of the online analysis parameter set is triggered by abnormal state marker signals to perform dynamic model calibration.

[0050] For example, the first threshold, the second threshold, the third threshold, and the fourth threshold can be determined offline based on historical classroom sample data labeled with acoustic state types from multiple sessions, by statistically analyzing the distribution characteristics of the proportion of teacher speech energy and the frequency of turn-taking under different states.

[0051] For example, the first threshold can be set to 65%, the second threshold to 4 times / minute, the third threshold to 55%, and the fourth threshold to 8 times / minute. These values ​​are exemplary and can be adjusted according to the specific classroom acoustics and teaching scenario during actual deployment.

[0052] When an anomaly marker is triggered, the system adds the feature vector corresponding to the current anomaly fragment to the retraining cache pool. The anomaly fragments in the retraining cache pool must undergo time continuity verification or multiple repeated trigger verification before they can participate in the incremental retraining of the analysis parameter set.

[0053] When the cache pool reaches the preset size, the corresponding statistics in the parameter configuration set are recalculated and merged with the original parameters in an exponentially weighted manner.

[0054] In this approach, the incremental retraining of the analysis parameter set does not rely on manual labeling, but rather employs a self-supervised mechanism.

[0055] The feature vectors fed into the retraining cache pool have their corresponding abnormal state marker signals as labels, indicating that under the current parameter set, this behavior pattern belongs to a statistically low-probability event.

[0056] The goal of retraining is to update the thresholds and baseline calculation parameters in the parameter set. For example, by using the feature vectors in the cache pool and the corresponding standardized distances, the sliding window length of the individual baseline, the reference range of the group baseline, or the values ​​of various thresholds can be refitted or updated.

[0057] Update strategies can include sliding window reestimation, Bayesian updates, or exponentially weighted moving averages, so that the updated parameter set can better adapt to the latest distribution of student behavioral characteristics in the current classroom environment. The algorithm modules involved in this solution can be implemented in the following exemplary ways: the target detection module uses the YOLO series detection algorithms to extract the bounding boxes of teachers and students in real time; the beamforming module uses the delay summation beamforming algorithm or the MVDR minimum variance distortionless response adaptive beamforming algorithm to perform beamforming processing of multi-channel audio signals; the speech activity detection module uses the WebRTC-VAD-based short-time energy detection method to perform speech activity detection and determination. The above algorithms are all exemplary implementations and do not constitute a limitation on the scope of protection of this solution.

[0058] Since both offline sample data and online incremental data are anonymized feature values ​​rather than the original audio and video, the entire parameter generation and update process is completed without the data leaving the local edge computing terminal.

[0059] The core mechanism of parameter switching in this scheme for controlling the behavior of underlying digital signal processing is as follows: When the current acoustic interaction state is identified as the teacher's voice as the dominant state, the first set of analysis parameters is called. The system smooths out stray background noise in the environment by extending the time domain analysis window and the silence judgment period, thereby effectively reducing the probability of data missegmentation and false alarm rate in low interaction density scenarios at the physical level.

[0060] When the current acoustic interaction state is identified as student voice as the dominant state, the second set of analysis parameters is invoked. The system significantly improves the sensitivity of capturing transient high-frequency burst acoustic signals by dynamically tightening the time domain analysis window and shortening the sampling sliding window period, effectively avoiding the missed detection of short-term burst speech fragments.

[0061] It is important to note that after the parameter switch, the frame boundaries of the subsequent audio stream, the results of silence segmentation, and the timing of voice activity triggering all change synchronously, thereby altering the data input structure for subsequent feature extraction. In other words, the parameter switch not only adjusts a single threshold but also fundamentally reconstructs the input structure of the subsequent data processing flow.

[0062] If a fixed-parameter system is used, and a low-latency response threshold suitable for student interaction is consistently applied during teacher-dominated speech mode, the system may misinterpret normal periods of silent listening as abnormal speech response delays. Conversely, if a high-latency response threshold is continued to be used during student-dominated speech mode, the excessively wide threshold range will prevent the effective capture of actual abnormal speech response behaviors.

[0063] The mismatch between parameters and data distribution caused by acoustic state switching is a systemic defect that inevitably occurs in fixed-parameter systems in non-stationary classroom environments.

[0064] Example 1, referring to Figure 1 - Figure 4 This is the first embodiment of the present invention. Taking the scenario of students reading aloud together as an example, this embodiment provides a method for analyzing foreign language classroom learning behavior based on multimodal data. In this scenario, after the teacher plays a demonstration recording or leads the reading, all students in the class read along at the same time, and the classroom is in a state of high frequency and multiple sound sources concurrently.

[0065] Specifically, including: S1. System Deployment and Initialization: A microphone array and a wide-angle camera are fixedly deployed in the classroom, and a parameter configuration set is pre-built in the edge computing terminal. The parameter configuration set is pre-generated in the early stage of system deployment through offline classroom sample data statistics or empirical initialization methods, including a first analysis parameter set and a second analysis parameter set.

[0066] The first set of analytical parameters corresponds to the teacher's voice as the main state. Its time-domain frame window length is relatively long, the silence duration threshold is relatively long, and the voice response delay judgment threshold is relatively wide.

[0067] The second set of analysis parameters corresponds to the student's voice as the main state. Its time-domain framing window length is relatively short, the voice activity detection and judgment window length is relatively short, and the voice response delay judgment threshold is relatively tight.

[0068] S2. Multimodal data acquisition: Multi-channel synchronous audio timing data is acquired through a microphone array on the ceiling, and video stream data is acquired through wide-angle cameras at the front and back of the classroom.

[0069] The system pre-divides multiple spatial regions in the video image, with each spatial region corresponding to a student's seat location.

[0070] The microphone array uses beamforming technology to form multiple directional pickup beams, each covering a spatial area.

[0071] The spatial regions in the video image are mapped to the beam coverage areas of the microphone array, forming multiple predefined spatial observation nodes. Each spatial observation node corresponds to an independent audio and video data observation channel.

[0072] S3. Extraction of spatial distribution features of speech energy: The system uses a target detection algorithm to extract the bounding boxes of the teacher's body and the students' body in real time from the video stream acquired by the wide-angle camera. The spatial geometric boundary vector of the teacher's activity area is determined based on the position of the teacher's bounding box, and the spatial geometric boundary vector of the student's seat area is determined based on the position of the student's bounding box.

[0073] The system performs beamforming processing on the multi-channel raw audio streams acquired by the microphone array to obtain acoustic energy values ​​corresponding to multiple directional beams. These acoustic energy values ​​are then arranged spatially according to the beams to form a multi-channel acoustic energy matrix.

[0074] The system uses the spatial geometric boundary vectors of the teacher's activity area and the student's seating area obtained from the video stream to construct a spatial region weighting matrix. In this weighting matrix, elements corresponding to the teacher's activity area are assigned a first weight value, elements corresponding to the student's seating area are assigned a second weight value, and elements corresponding to other areas are assigned zero weight.

[0075] The system performs element-wise multiplication of the spatial region weighted matrix with the multi-channel acoustic energy matrix, preferably using the Hadamard product operation.

[0076] The matrix obtained after multiplication is the final spatial distribution feature of speech energy. This feature reflects the acoustic energy magnitude of each spatial region and also spatially assigns the acoustic energy using video information.

[0077] During the feature extraction process described above, the original video stream is immediately deleted from the edge computing terminal after the spatial geometric boundary vector is extracted, retaining no data that can be reconstructed into the original image. Similarly, the original multi-channel audio stream is immediately deleted after the acoustic energy matrix is ​​calculated, retaining only the anonymized energy distribution values ​​and weighted calculation results.

[0078] All subsequent processing steps are based on these anonymized feature values.

[0079] In scenarios where multiple people speak simultaneously and the sound field is extremely aliased, the aforementioned spatial region weighting mechanism accurately isolates acoustic reverberation interference from non-target regions by utilizing visual boundary features, thus decoupling the separation problem of aliased sound sources from the bottom up.

[0080] If only the sound source localization function of the microphone array is used without the help of video spatial boundary information, in the case of multiple sound sources in concurrency, the acoustic signals from adjacent seat areas will be severely aliased, which will lead to the deviation of the calculation results of the spatial distribution characteristics of speech energy, and thus affect the accuracy of subsequent acoustic state recognition and anomaly detection.

[0081] This solution addresses this problem in principle by spatially fusing audio and video data at the feature level.

[0082] S4. Acoustic Interaction State Recognition: Based on the spatial distribution characteristics of speech energy obtained in step S3, the system calculates the sum of element values ​​corresponding to the teacher's activity area and the sum of element values ​​corresponding to the student's seat area, and calculates the proportion of teacher's speech energy and the proportion of student's speech energy.

[0083] The system detects turn-switching events between different spatial observation nodes based on the changes in energy of each spatial observation node over time in the spatial distribution characteristics of speech energy.

[0084] When the energy of one space observation node decreases and the energy of another space observation node increases, a turn-by-turn switch is recorded.

[0085] The system counts the number of turn-taking cycles per unit time to obtain the turn-taking frequency. It then compares the calculated percentage of teacher voice energy with a preset first threshold, the percentage of student voice energy with a preset third threshold, and the turn-taking frequency with preset second and fourth thresholds.

[0086] In this embodiment, the system detects that the proportion of voice energy corresponding to the student's seat area is higher than a preset third threshold, and the real-time turn-taking frequency is higher than a preset fourth threshold. Therefore, the system identifies the current acoustic interaction state of the classroom as the student voice-dominated state.

[0087] S5. Dynamic switching of analysis parameters. Based on the student's voice status identified in step S4, the system calls the second set of analysis parameters corresponding to that status from the pre-built parameter configuration set.

[0088] For example, the temporal framing window length in the second set of analysis parameters is shortened to 150 milliseconds, the voice activity detection and determination window length is shortened to 300 milliseconds, and the voice response delay determination threshold is tightened to 1.5 seconds.

[0089] Meanwhile, the system updates the temporal frame window length of the underlying audio processing module to 150 milliseconds, the judgment window length of the voice activity detection module to 300 milliseconds, and the statistical discrimination threshold for voice response delay to 1.5 seconds.

[0090] The reconstruction of these parameters directly affects the underlying digital signal processing module, changing the system's behavior in acquiring, segmenting, and detecting audio data.

[0091] S6. Individual Statistical Baseline Construction. For the target spatial observation node CH-001, the system retrieves historical response delay feature sequences from the historical database over the past 30 days, under conditions where student speech was the primary mode. This historical data consists of anonymized feature values ​​extracted and stored from previous classes, and does not contain any original audio or video content. The system constructs the individual statistical baseline using a sliding window and exponential moving average method.

[0092] Specifically, the system maintains a sliding window that stores the response delay observations for the most recent time under the same conditions. When a new observation enters the window, the system updates the baseline parameters according to the following steps.

[0093] The system obtains the current observation value. The moving average before the update and preset learning rate parameters (For example, take) Update the current moving average according to the following formula. : Subsequently, the system retrieves the historical sliding variance before the update. Update the current sliding variance according to the following formula. : Then update the current sliding variance Take the arithmetic square root to obtain the updated moving standard deviation. : After the recursion is completed, the individual moving average of that node is... The individual statistical standard deviation is 0.55 seconds. The duration is 0.28 seconds, which together constitute the individual statistical baseline of this node in the current acoustic interaction state.

[0094] S7. Construction of Group Statistical Baseline: The system acquires the real-time response delay data of all spatial observation nodes in the current classroom and in the same acoustic interaction state, under the same student voice-dominated state in this classroom, and calculates the arithmetic mean and overall standard deviation of these values. The group statistical baseline is preferably constructed based on the set of spatial observation nodes in the same classroom stage, the same interactive task type, and the same acoustic interaction state.

[0095] Obtain the real-time mean of the group The time interval is 0.50 seconds, and the group statistical standard deviation is... It takes 0.32 seconds.

[0096] S8. Deviation Calculation: At the current moment, the system detects that the target space observation node CH-001 has generated a transient response delay. The triggering condition for this response is that the student corresponding to the node starts to speak after the teacher issues the follow-up reading instruction.

[0097] The system records the time delay observation for this response. (For example, assume 1.55 seconds). The system compares this observation with the individual statistical baseline constructed in step S6 and calculates the first standardized distance. : Simultaneously, the system compares the observed value with the population statistical baseline constructed in step S7 and calculates the second standardized distance. : S9. Dual-condition judgment: The system's preset first sensitivity coefficient Second sensitivity coefficient For example, and (If both are set to 3.0), the system compares the absolute values ​​of the two standardized distances calculated in step S8 with the corresponding sensitivity coefficients.

[0098] in, , If both conditions are met simultaneously, the system determines that the current observation belongs to a statistically low-probability event under the dual distribution model.

[0099] It should be noted that the standardized distance used in this embodiment is not limited to strictly satisfying the standard normal distribution. It is only used to measure the degree of deviation of the current observation value from the historical statistical baseline.

[0100] S10. Anomaly marker signal generation and closed-loop control. Based on the determination result of step S9, the system generates an anomaly status marker signal.

[0101] The marker signal contains the following information: trigger timestamp, spatial observation node identifier CH-001, current acoustic interaction state type is student voice-dominated state, abnormal feature type is voice response delay, current observation value 1.55 seconds, first normalized distance 3.57, second normalized distance 3.28.

[0102] Based on this marker signal, the system automatically performs the following control operations: The current abnormal time series segment is automatically cached and stored in the abnormal data cache area of ​​the edge computing terminal; metadata tags are added to the abnormal data stream to record the abnormal type, triggering conditions and time information; the abnormal time series segment is imported into the retraining cache pool.

[0103] Abnormal segments in the retraining cache pool must undergo time continuity verification or multiple repeated trigger verifications before they can participate in the incremental retraining of the analysis parameter set.

[0104] When the number of abnormal segments accumulated in the retraining cache pool reaches a preset threshold, the system automatically triggers the incremental retraining process of the analysis parameter set, recalculates the statistics related to the analysis parameter set in the parameter configuration set, and merges the newly calculated statistics with the original parameters in an exponential weighted manner.

[0105] Throughout the closed-loop control process, the data cached, labeled, and imported into the retraining cache pool are all anonymized feature values. The system output consists only of deviation data and abnormal state label signals, without any labels or evaluations of the learner's personal abilities, attitudes, or psychological states.

[0106] In this embodiment, the technical effect of dual baseline collaborative verification is reflected in the following: when only individual baselines are used, the individual baselines of the target space observation nodes may have adapted to a certain inherent response mode due to long-term exposure to this mode, resulting in the failure to detect sudden deviations; when only group baselines are used, the normal difference between the inherent response habits of individuals and the group mean may be misjudged as abnormal.

[0107] The dual baselines are used simultaneously: the individual baseline provides a reference benchmark for the long-term behavior pattern of the node itself, while the group baseline provides a reference level for the real-time distribution of all nodes in the current state.

[0108] The system generates an anomalous state marker signal only when the current observation deviates significantly from both the individual baseline and the population baseline.

[0109] The dual constraint can, in principle, distinguish between normal fluctuations in an individual, gradual evolution of an individual, and sudden deviations that require attention, thereby reducing the false alarm rate.

[0110] Example 2, refer to Figure 1 - Figure 4 This is the second embodiment of the present invention, which differs from the first embodiment in that: the classroom acoustic environment is a multi-spatial aliasing sound source interaction scenario, and the system performs abnormal deviation detection on a specific node marked as spatial observation node CH-002. In this scenario, the teacher divides students into several groups for dialogue practice, and students within the same group take turns speaking at high frequency, resulting in spatial aliasing of speech signals between different groups.

[0111] Specifically, after the system identifies the current state as student voice as the dominant state, it calls the corresponding second set of analysis parameters.

[0112] For example, the speech response delay time threshold in the second set of analysis parameters is tightened to 1.5 seconds, and the temporal framing window length is shortened to 150 milliseconds.

[0113] The system updates the parameters of the underlying audio processing module to the above values, putting the system in a high-sensitivity acquisition mode to adapt to the characteristics of dense turn-taking in group dialogue scenarios.

[0114] Using the spatial observation node CH-002 as the observation object, the system extracts historical anonymized feature data under the same conditions, and constructs individual statistical baselines using the same sliding window and exponential moving average method as in Example 1.

[0115] After the recursion is completed, the individual statistical mean of that node is obtained. The duration is 1.10 seconds, and the individual statistical standard deviation is... The system acquires real-time response delay data of all spatial observation nodes in the same acoustic interaction state within the current classroom at 0.35 seconds, calculates the arithmetic mean and the population standard deviation, and obtains the group's real-time mean. The time is 0.90 seconds, and the overall standard deviation of the population is [missing information]. It takes 0.42 seconds.

[0116] Specifically, at the current moment, the system detects a response delay generated by space observation node CH-002. For example, assume that the observed delay value of this response is... It takes 1.85 seconds. Calculate the first standardized distance. : Calculate the second standardized distance: In this example, exemplarily, the preset sensitivity threshold coefficient and All are taken as 3.0, of which , If both conditions are not met simultaneously, the system will not generate an abnormal state marker signal.

[0117] To illustrate the detection scenario where both conditions are met simultaneously, for example, suppose that at another moment, the space observation node CH-002 experiences a longer response delay observation. It took 2.65 seconds. The system recalculated the first standardized distance. : Recalculate the second standardized distance at this time, , If both conditions are met simultaneously, the system determines that the temporal feature stream of the space observation node CH-002 has a distribution deviation, generates an abnormal state marker signal, and drives the back-end memory to slice and isolate the current abnormal audio temporal segment of the node. The cached data consists of anonymized feature values ​​and does not contain any original audio or video records. The remaining steps are the same as in Example 1.

[0118] Example 3, referring to Figure 1 - Figure 4 This is the third embodiment of the present invention. This embodiment differs from the second embodiment in that the classroom acoustic environment is a scenario with a high proportion of single, fixed sound sources, where the teacher is explaining grammatical rules unidirectionally from the podium, and the students are listening. This embodiment is used to verify the system's ability to avoid false alarms under normal low-voice activity conditions.

[0119] Specifically, based on the spatial distribution characteristics of speech energy, the system detects that the proportion of speech energy in the fixed podium area is higher than a preset first threshold and the frequency of turn-taking is lower than a preset second threshold, thus identifying the current acoustic interaction state of the classroom as a teacher-dominated state.

[0120] The system automatically calls the first set of analysis parameters. For example, the time-domain frame window length in the first set of analysis parameters is adjusted to 500 milliseconds, the silence duration threshold is adjusted to 1.5 seconds, and the statistical discrimination threshold for voice response delay is adjusted to 5.0 seconds.

[0121] The system will update the above parameters of the underlying audio processing module synchronously.

[0122] During the teacher's explanation, taking the spatial observation node CH-003 as an example, the system detected that the spatial area corresponding to the node continuously maintained the visual orientation characteristics facing the podium area, and did not generate speech energy output that triggered the adaptive sampling threshold.

[0123] Since the system has dynamically widened the time-domain segmentation window and discrimination criteria based on the teacher's dominant voice state, the system determines that this time segment belongs to a normal low-voice activity state and does not generate an abnormal state marker signal. The remaining steps are the same as in Example 2.

[0124] This embodiment demonstrates the technical necessity of dynamic parameter switching. If a fixed parameter system is used, and the fixed parameter is a low-latency response threshold configured based on the student's voice as the primary state, then in the teacher's explanation scenario of this embodiment, the student's normal listening silence will be misjudged as an abnormal voice response delay.

[0125] This invention identifies the current state as dominated by teacher speech through acoustic state perception and dynamically adjusts the speech response delay judgment threshold accordingly. This enables the system to correctly distinguish between normal listening silence and genuine abnormal response delay, fundamentally eliminating false alarms caused by parameter and distribution mismatch. The remaining structure is the same as in Embodiment 2.

[0126] The technical advantages of this scheme compared to fixed-parameter systems are a natural conclusion drawn from recognized principles of signal processing and statistics, and can be demonstrated without relying on specific experimental data.

[0127] When the acoustic state of the classroom shifts from teacher-dominated to student-dominated, the distribution of students' speech behavior characteristics undergoes a non-stationary drift.

[0128] For systems that use fixed analysis parameters, the preset judgment threshold can only match the data distribution under a single acoustic state.

[0129] After the state switch, a systematic mismatch occurs between the fixed parameters and the current data distribution: The response delay threshold set when the teacher's voice is dominant is usually high to accommodate the time students need to think; this threshold will be too lenient when the student's voice is dominant, causing actual response anomalies to go undetected and increasing the false negative rate.

[0130] Conversely, the threshold set when students are the primary voice users is usually low to accommodate the need for frequent turn-taking; this threshold would be too strict when teachers are the primary voice users, causing normal low-voice activity to be misjudged as abnormal, resulting in a higher false alarm rate.

[0131] This solution identifies the current acoustic interaction status of the classroom through acoustic state perception and dynamically calls up the set of analysis parameters that match the current status, so that the analysis standard is realigned with the statistical distribution of the current data, thus eliminating the above-mentioned parameter and distribution mismatch problem in principle.

[0132] A dual verification mechanism using individual statistical baselines and population statistical baselines is employed: Individual baselines are constructed based on historical behavioral data of target spatial observation nodes, reflecting their long-term behavioral patterns; group baselines are constructed based on real-time behavioral data of all spatial observation nodes within the current classroom, reflecting the group reference level under the current acoustic conditions. The system generates anomaly marker signals only when the current observation deviates significantly from both the individual and group baselines simultaneously.

[0133] The dual constraint mechanism can effectively distinguish between normal fluctuations, gradual evolution, and sudden deviations that require attention, thereby reducing the false alarm rate.

[0134] By using spatial region weighted calculation to achieve the fusion of audio and video data, in the case of aliased sound field where multiple people speak at the same time, the system accurately removes acoustic reverberation interference from non-target areas by utilizing visual boundary features, decoupling the separation problem of aliased sound sources from the bottom layer, and further ensuring the accuracy of feature extraction and the reliability of the overall system detection.

[0135] Parameter switching not only adjusts a single threshold, but also fundamentally reconstructs the input structure of subsequent data processing, which is fundamentally different from simple rule combinations or parameter optimization.

[0136] Based on embodiments 1-3, the working principle of the present invention is as follows: the system collects multimodal data through a microphone array and a wide-angle camera, constructs a spatial region weighting matrix using the spatial geometric boundary vector extracted from the video stream, and performs element-wise spatial weighting on the multi-channel acoustic energy matrix after the audio stream has undergone beamforming processing to obtain the spatial distribution characteristics of speech energy that incorporates spatial location information.

[0137] The system calculates the proportion of teacher's and student's speech energy, as well as the turn-taking frequency, based on these characteristics to identify whether the current acoustic interaction in the classroom is dominated by teacher or student speech. Based on the identification results, the system calls the corresponding speech behavior analysis parameter set from a pre-built parameter configuration set, dynamically adjusting the temporal framing window length, silence duration threshold, speech activity detection judgment window length, and speech response delay judgment threshold of the underlying digital signal processing module. Then, the system extracts the temporal feature parameters of speech activity from the target spatial observation nodes, constructs individual statistical baselines using a sliding window and exponential moving average method, and simultaneously calculates the real-time group statistical baseline for all spatial observation nodes in the current classroom. Using the called parameter set, the system calculates the standardized distance of the current observation value relative to the dual baselines.

[0138] When both standardized distances exceed the preset sensitivity threshold, the system generates an anomaly status marker signal. Based on this signal, it triggers the caching of abnormal time-series segments and metadata marking, and imports the abnormal segments into the retraining cache pool. When the cache pool reaches the preset size, it triggers incremental retraining of the analysis parameter set. The entire process is completed on the edge computing terminal. The original audio and video data is deleted immediately after feature extraction, and only anonymized feature values ​​are retained for statistical analysis and parameter updates.

[0139] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for analyzing foreign language classroom learning behavior based on multimodal data, characterized in that, include: Acquire multimodal data collected by audio and video acquisition devices deployed in the classroom. The multimodal data corresponds to at least one predefined spatial observation node, and the spatial observation node corresponds to an audio and video data observation channel in a predefined spatial area within the classroom. Based on multimodal data, the current acoustic interaction state of the classroom is identified, including the teacher-dominated state and the student-dominated state. Based on the identified acoustic interaction state, the speech behavior analysis parameter set corresponding to the current acoustic interaction state is called from the pre-built parameter configuration set. The speech activity detection window length, time domain frame window length, silence duration threshold or speech response delay judgment threshold are dynamically adjusted using the called speech behavior analysis parameter set. Different analysis parameters correspond to the teacher speech-dominant state and the student speech-dominant state. The speech activity temporal feature parameters of the target spatial observation nodes are extracted from multimodal data. The speech activity temporal feature parameters include the duration of non-syntactic boundary pauses, speech response delay time and turn-taking frequency. Individual statistical baselines are constructed based on the parameter values ​​of the target spatial observation nodes under the same historical acoustic interaction state, and group statistical baselines are constructed based on the real-time parameter values ​​of all spatial observation nodes in the current classroom under the same acoustic interaction state. Using the invoked speech behavior analysis parameter set, the deviation data of the current speech activity temporal feature parameters relative to the individual statistical baseline and the group statistical baseline are calculated; When the deviation data exceeds the preset threshold range, an abnormal state marker signal is generated, and based on the abnormal state marker signal, an abnormal time sequence segment cache, metadata marker, or incremental retraining of the analysis parameter set is triggered.

2. The method for analyzing foreign language classroom learning behavior based on multimodal data according to claim 1, characterized in that: Identify the current acoustic interaction status in the classroom based on multimodal data, including: Extracting spatial distribution features of speech energy and turn-taking frequency from multimodal data; When the proportion of teacher voice energy in the spatial distribution characteristics of voice energy is higher than the preset first threshold and the turn-alternating frequency is lower than the preset second threshold, it is identified as a state where teacher voice is dominant. When the proportion of student speech energy in the spatial distribution characteristics of speech energy is higher than the preset third threshold, and the frequency of turn-taking is higher than the preset fourth threshold, it is identified as a state where student speech is dominant.

3. The method for analyzing foreign language classroom learning behavior based on multimodal data according to claim 2, characterized in that: Extracting spatial distribution features of speech energy, including: The spatial geometric boundary vectors of the teacher's activity area and the student's seating area are extracted from the video stream acquired by the video acquisition device. The spatial geometric boundary vectors are used to construct a spatial region weighting matrix. The multi-channel acoustic energy matrix acquired by the audio acquisition device is then spatially weighted element by element to determine the spatial distribution characteristics of speech energy.

4. The method for analyzing foreign language classroom learning behavior based on multimodal data according to claim 3, characterized in that: The multi-channel acoustic energy matrix outputs the raw audio stream acquired by the audio acquisition device after beamforming processing. The element-wise spatial weighted calculation is achieved by the Hadamard product operation of the weighting matrix and the acoustic energy matrix.

5. The method for analyzing foreign language classroom learning behavior based on multimodal data according to claim 1, characterized in that: The speech activity detection window length, temporal framing window length, silence duration threshold, or speech response delay determination threshold are dynamically adjusted using the invoked speech behavior analysis parameter set, including: When the current classroom is identified as being dominated by the teacher's voice, increase the length of the time-domain frame window or increase the silence duration threshold. When the current classroom is identified as being dominated by student voices, adjust the small-scale frame window length or shorten the voice activity detection and judgment window length.

6. The method for analyzing foreign language classroom learning behavior based on multimodal data according to claim 1, characterized in that: Construct individual statistical baselines for the target spatial observation nodes, including: Acquire historical data of the temporal characteristic parameters of speech activities of the target spatial observation node in the current acoustic interaction state within a historical time period; The moving mean and moving variance are updated using a sliding window and exponential moving average method. The formula for updating the moving mean is: The formula for updating the sliding variance is: Using the updated moving mean and moving standard deviation as the individual statistical baseline, the updated formula for the moving standard deviation is: in, This is the updated current moving average. This is the historical moving average before the update. The updated current sliding variance, The historical sliding variance before the update. For the current observation value, For the updated sliding standard deviation, The preset learning rate parameter has a range of values. .

7. The method for analyzing foreign language classroom learning behavior based on multimodal data according to claim 1, characterized in that: Constructing a population statistical baseline includes: Acquire real-time data of the temporal characteristic parameters of speech activities of spatial observation nodes that are in the same acoustic interaction state within the current classroom; Calculate the arithmetic mean and population standard deviation of the real-time data, and use the arithmetic mean and population standard deviation as the statistical baseline of the population.

8. The method for analyzing foreign language classroom learning behavior based on multimodal data according to claim 1, characterized in that: The speech behavior analysis parameter set includes the non-syntactic boundary pause duration threshold, the follow-up delay time threshold, and the speech response delay time threshold; When the acoustic interaction state is dominated by the teacher's voice, the first non-syntactic boundary pause duration threshold is used, which is greater than the second non-syntactic boundary pause duration threshold. When the acoustic interaction state is dominated by student speech, a second non-syntactic boundary pause duration threshold is used, which is less than the first non-syntactic boundary pause duration threshold.

9. The method for analyzing foreign language classroom learning behavior based on multimodal data according to claim 1, characterized in that: Calculate the deviation data of the current speech activity temporal feature parameters relative to the individual statistical baseline, including: Calculate the first standardized distance between the current speech activity temporal feature parameter value of the target spatial observation node and the individual statistical baseline. The first standardized distance is calculated as follows: in, The first standardized distance, For the current observation value, This is the individual statistical mean. For individual statistical standard deviations; Calculate the deviation data of the current speech activity temporal feature parameters relative to the population statistical baseline, including: Calculate the second standardized distance between the current speech activity temporal feature parameter value of the target spatial observation node and the population statistical baseline. The second standardized distance is calculated as follows: in, The second standardized distance, This is the group statistical mean. The standard deviation of the group; Deviation data exceeding the preset threshold range includes: The absolute value of the first standardized distance is greater than the preset first sensitivity coefficient, expressed as: in, The first sensitivity coefficient, ; Meanwhile, the absolute value of the second standardized distance is greater than the preset second sensitivity coefficient, expressed as: in, This is the second sensitivity coefficient. .

10. A foreign language classroom learning behavior analysis system based on multimodal data, comprising a processor and a memory, wherein when the processor executes a program stored in the memory, it implements the steps of any one of claims 1 to 9.

Images