A sign language public opinion tendency feedback control method and system based on emotional color recognition

Abstract

The application discloses a sign language public opinion tendency feedback control method and system based on emotional color recognition, relates to the technical field of sign language recognition feedback, and comprises the following steps: aligning and structuring processing of autism sign language multi-modal time sequence data, extracting stable state components, constructing state continuity association structure, and generating standard state keeping tendency values; performing propagation reconstruction and resampling to generate second state keeping tendency values; determining stable intervals and trigger positions based on cosine similarity changes and time difference detection, and realizing equivalent control; using the same for emotional recognition and training feedback; generating reward values based on error feedback and random gradient update; matching and sorting the reward values and tendency values, and screening the optimal interval to output a control result; and solving the problems of large emotional recognition result fluctuation, unstable state switching and difficult stable capture of emotional changes in autism sign language expression in the prior art.

Description

Technical Field

[0001] This invention relates to the field of sign language recognition feedback technology, and more specifically, to a sign language public opinion tendency feedback control method and system based on emotional color recognition. Background Technology

[0002] Existing sign language public opinion feedback technology based on emotion color recognition mainly relies on a multimodal sign language recognition framework. It collects continuous sign language video streams through camera equipment, converts the video streams into frame sequence data, and then extracts hand trajectory features, skeletal posture features, and facial expression features by combining human posture estimation and hand key point detection methods. These features are then input into a temporal modeling network for dynamic expression modeling, and finally output emotion category labels and emotion intensity values, thereby realizing the recognition of emotional information in sign language expression and feedback of public opinion trends. This type of technology has formed a relatively mature application system in the fields of sign language translation and emotion computing.

[0003] Based on the aforementioned existing technologies, some improved methods introduce a "state retention tendency value" to characterize the ability to continuously determine the emotional or expressive state of sign language over time. This state retention tendency value is used to characterize whether the system tends to maintain the currently identified emotional or expressive state without switching in continuous time segments. Essentially, it is used to control the degree to which the emotion recognition model accepts state change signals in the time series, thereby affecting whether the emotional state is continuously identified and whether state updates are triggered. In the process of sign language public opinion tendency feedback, this value directly affects the stability of the system output and the sensitivity of state switching.

[0004] The state retention tendency value itself has two different value characteristics: high and low. Both high and low state retention tendency values ​​will have different effects on the sign language public opinion tendency feedback process based on emotion color recognition. The high state retention tendency value tends to maintain the continuity of the current state judgment result, while the low state retention tendency value tends to enhance the sensitivity of state switching. Therefore, the value distribution of this parameter will directly affect the system's response speed and stability to emotion changes, thus having a key impact on the continuity and consistency of public opinion tendency feedback results.

[0005] When the state retention tendency value is low, the system is more sensitive to subtle changes in sign language expression or short-term emotional fluctuations. It is easy to judge short-term local changes as state switching signals, which leads to frequent changes in emotion recognition results. This results in fragmented and discontinuous feedback of public opinion tendency. At the same time, in the autism sign language expression scenario, it is easy to misjudge unstable movements as emotional changes, thereby reducing the stability of recognition results and the consistency of training feedback.

[0006] When the state retention tendency value is high, the system tends to prolong the duration of the current state determination time, which has a strong inhibitory effect on emotional changes or action deviations that occur in a short period of time, thereby reducing the frequency of state switching. However, at the same time, it will cause real emotional mutations or expression changes to be delayed or even ignored, making it impossible for the system to capture rapid state changes such as "resistance to acceptance" or "calm to excitement" when autistic users express them in a timely manner, thereby reducing the sensitivity and timeliness of emotion recognition.

[0007] In practical applications, existing technologies often lack a unified modeling mechanism for controlling state retention tendency values. They cannot dynamically adjust the state retention strength according to changes in sign language expressions at different time scales. This makes it difficult to simultaneously consider state stability and sensitivity to change in the sign language public opinion tendency feedback process based on emotional color recognition. As a result, problems such as excessive state stability leading to information lag or excessive state sensitivity leading to feedback fluctuations are likely to occur. Consequently, the overall public opinion tendency feedback results are insufficient in terms of consistency, continuity, and robustness.

[0008] To address the above problems, this invention proposes a solution. Summary of the Invention

[0009] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a sign language opinion tendency feedback control method and system based on emotion color recognition. By constructing a state-maintaining tendency value adaptive control mechanism based on multi-scale frequency domain stable component extraction and propagation reconstruction, and combining stable interval detection and reward feedback optimization, the continuous consistency modeling and dynamic adjustment of sign language emotional states are realized, thereby solving the problems of large fluctuations in emotion recognition results, unstable state switching, and difficulty in stably capturing emotional changes in autistic sign language expressions in the prior art.

[0010] To achieve the above objectives, the present invention provides the following technical solution: A sign language public opinion tendency feedback control method based on emotion color recognition includes the following steps: Aligning and structuring multimodal temporal data of autism sign language, constructing a state continuity association structure, and extracting stable state components by combining similarity constraints and frequency domain decomposition; performing cross-scale aggregation and posterior correction based on the stable components to output standard state maintenance tendency values; propagating and reconstructing the standard state maintenance tendency values ​​to generate state maintenance extension sequences and multi-path propagation chains, and obtaining several second state maintenance tendency values ​​through resampling; determining the stable state maintenance interval and state change trigger position based on continuous frame cosine similarity changes and adjacent time segment differences, and maintaining the stable state... A fixed interval mapping is used as the acquisition time window, and the time interval corresponding to the state change trigger position is mapped as the state update trigger delay, realizing equivalent control of the state maintenance tendency value. The equivalent control of each second state maintenance tendency value is applied to the sign language emotion recognition and training feedback process, and the acquisition, recognition and feedback are performed and recorded. Feature extraction is performed on the application process of each second state maintenance tendency value, and iterative updates are performed based on stochastic gradient optimization. The corresponding state maintenance reward value is generated through error feedback. The state maintenance reward value and the second state maintenance tendency value are matched, sorted and filtered by interval, and the value corresponding to the optimal stable interval is extracted as the optimal state maintenance tendency value and used for sign language training feedback control.

[0011] A sign language public opinion tendency feedback control system based on emotion color recognition includes an alignment modeling module, a propagation reconstruction module, an interval control module, a feedback execution module, an error optimization module, and an interval filtering module. The alignment modeling module aligns and structures multimodal temporal data of autism sign language, constructs a state continuity association structure, extracts stable state components by combining similarity constraints and frequency domain decomposition, performs cross-scale aggregation and posterior correction based on the stable components, and outputs a standard state maintenance tendency value. The propagation reconstruction module reconstructs the standard state maintenance tendency value, generates a state maintenance extension sequence and a multi-path propagation chain, and obtains several second state maintenance tendency values ​​through resampling. The interval control module determines the stable state maintenance region based on the cosine similarity change of consecutive frames and the difference detection of adjacent time segments. The system employs a multi-stage approach, which maps the state-maintaining interval to the state change trigger position and the state-update trigger delay to achieve equivalent control of the state-maintaining tendency value. A feedback execution module applies the equivalent control of each second state-maintaining tendency value to the sign language emotion recognition and training feedback process, performing collection, recognition, and feedback, and recording the data. An error optimization module extracts features from the application process of each second state-maintaining tendency value, iteratively updates it based on stochastic gradient optimization, and generates the corresponding state-maintaining reward value through error feedback. An interval filtering module matches and sorts the state-maintaining reward value with the second state-maintaining tendency value, extracts the value corresponding to the optimal stable interval as the optimal state-maintaining tendency value, and uses it for sign language training feedback control.

[0012] An electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a sign language opinion tendency feedback control method based on emotion color recognition.

[0013] A computer-readable storage medium storing a computer program that, when executed by a processor, implements a sign language public opinion tendency feedback control method based on emotion color recognition.

[0014] The technical effects and advantages of the sign language public opinion tendency feedback control method and system based on emotion color recognition of the present invention are as follows: 1. This invention improves the sign language emotion recognition process from traditional single-temporal modeling to "multi-path state propagation modeling" by constructing a frequency domain stable component extraction mechanism for standard state-preserving tendency values ​​and a cross-scale propagation reconstruction mechanism. This enables the simultaneous characterization of stable expression parts and instantaneous change parts of sign language actions at different time scales, so that the emotion state no longer depends on a single frame or short segment for judgment, but is uniformly modeled based on the continuous evolution of stable state components. At the same time, the propagation extension sequence and resampling mechanism enhance the multi-path consistency of state expression, so that the system can still maintain the stable capture ability of core emotional states when faced with irregular fluctuations in the sign language expressions of autistic users, thereby improving the continuity and noise resistance of emotion recognition results.

[0015] 2. This invention introduces a stable state interval detection, state change trigger location mapping, and reward feedback-driven optimal interval selection mechanism, transforming the state maintenance tendency value from a static parameter into an adaptively optimizable control variable. This allows the system to dynamically balance the relationship between "state maintenance stability" and "state change sensitivity" during training feedback. Simultaneously, a closed-loop adjustment mechanism is formed through error feedback and stochastic gradient optimization, enabling the differences between individuals with autism in sign language expression to be incorporated into a unified optimization process. This achieves personalized adaptive adjustment of the state maintenance tendency value, thereby significantly improving the stability, accuracy, and generalization ability of sign language emotion recognition and public opinion tendency feedback control. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a sign language public opinion tendency feedback control method based on emotional color recognition according to the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1, Figure 1 This invention presents a sign language public opinion tendency feedback control method based on emotion color recognition, comprising the following steps: S1 aligns and structures the multimodal temporal data of autism sign language, constructs a state continuity association structure, and extracts stable state components by combining similarity constraints and frequency domain decomposition. Based on the stable components, it performs cross-scale aggregation and posterior correction, and outputs the standard state preservation tendency value.

[0019] In this embodiment, the alignment and structuring of multimodal temporal data of autism sign language are performed to construct a state continuity association structure. Stable state components are extracted by combining similarity constraints and frequency domain decomposition. Based on these stable components, cross-scale aggregation and posterior correction are performed to output a standard state preservation propensity value. Specifically: The hand trajectory, posture skeleton and facial expression multimodal data in the autism sign language training process are time-stamped and aligned, and uniformly mapped to the same time axis; Spatial normalization and joint topology reconstruction are performed on the aligned multimodal data to generate structured state sequence units; Cosine similarity analysis is performed on the structured state sequence units to construct a continuous correlation structure of states in adjacent time slices; The state continuity correlation structure is subjected to frequency domain decomposition to separate the low-frequency stable component and the high-frequency disturbance component, while retaining the low-frequency stable component. The low-frequency stable components are subjected to multi-timescale sliding aggregation and posterior correction fusion based on different autism sample distributions to obtain the stable state baseline sequence. The steady-state baseline sequence is subjected to time compression mapping to generate a continuous state preservation strength sequence, and the standard state preservation tendency value is output.

[0020] In this embodiment, timestamp synchronization alignment involves performing frame-level correspondence on the hand trajectory sequence, posture skeleton sequence, and facial expression sequence collected during autism sign language training, according to a unified time reference. This enables different modalities to form comparable state description sequences under the same time index, thereby eliminating the time misalignment problem caused by inconsistent sampling times of multi-source acquisition devices.

[0021] In this embodiment, spatial normalization is to uniformly transform the hand trajectory and key points of the posture skeleton into a relative coordinate system with the center of the human pelvis as the origin. This unifies the spatial scale differences between different individuals and at different shooting distances into the same geometric expression space, thereby ensuring that subsequent state comparisons are based on relative motion relationships rather than absolute spatial positions.

[0022] In this embodiment, joint topology reconstruction involves reconstructing key points of human posture into a structured graph connection form according to the connection relationship of human skeleton, and using the connection relationship between joints as a fixed topology, so that posture data is transformed from a point set into a state graph sequence with structural constraints, which is used to describe the overall motion structure changes of the action.

[0023] In this embodiment, cosine similarity analysis measures the degree of directional consistency between state vectors of adjacent time slices. By comparing the degree of consistency in the direction of state change, a basis for judging state continuity is formed, which is used to characterize whether sign language actions and emotional expressions maintain a consistent evolutionary trend between adjacent time slices.

[0024] In this embodiment, the state continuity association structure is a time connection structure built based on the similarity relationship between adjacent time slices, where each time slice is a node and the similarity relationship between adjacent time slices is a connecting edge, used to express the continuity relationship and change path of sign language state in the time dimension.

[0025] In this embodiment, the frequency domain decomposition process converts the time change sequence corresponding to the state continuity association structure into a frequency expression form, and separates the slowly changing part from the rapidly fluctuating part. The slowly changing part corresponds to long-term stable emotions and action trends, while the rapidly fluctuating part corresponds to instantaneous action interference and noise changes.

[0026] In this embodiment, the low-frequency stable component is a slowly changing component extracted from the frequency domain decomposition result. It represents the trend structure of the sign language emotional state remaining consistent over a long period of time and is used to reflect whether the state has a sustainable stability basis.

[0027] In this embodiment, the multi-timescale sliding aggregation process involves repeatedly pooling low-frequency stable components under sliding windows of different time lengths, so that short-time stable structures and long-time stable structures are preserved at the same time, thereby forming a consistent expression result across time scales.

[0028] In this embodiment, the posterior correction fusion is based on the differences in state distribution among different autism samples, and makes consistent adjustments to the multi-timescale aggregation results so that the stable state expression results under the differences in individual expression tend to a unified reference standard, thereby reducing the impact of individual differences on the state judgment results.

[0029] In this embodiment, the stable state reference sequence is a continuous time state sequence obtained after multi-timescale aggregation and posterior correction. It expresses the standardized stable state evolution trajectory under a unified correction rule and is used as the basic reference sequence for generating subsequent state maintenance strength.

[0030] In this embodiment, the time compression mapping process compresses the steady-state baseline sequence in the time dimension, integrates the steady-state changes within multiple time slices into a continuous intensity change expression, and transforms the original discrete-time state into a continuously changing state intensity sequence.

[0031] In this embodiment, the continuous state maintenance intensity sequence is a numerical expression of the time compression mapping result. It represents the system's tendency to determine the degree to which the current sign language emotion or action state persists in a continuous time process. This tendency is used to reflect whether the system continuously recognizes the time continuity of the current state without change.

[0032] In this embodiment, the standard state retention tendency value is a standardized expression extracted from the continuous state retention strength sequence. It is used to characterize the system's tendency to determine whether the current sign language state continues to hold in the time series. Essentially, it controls the system's sensitivity to the time delay of "whether the current state needs to be maintained or updated". A high value indicates that the system is more inclined to maintain the existing state and delay the state update, while a low value indicates that the system is more inclined to quickly accept the state change and update the current judgment.

[0033] It should be noted that in this step, the combination of time alignment and spatial normalization makes the multimodal sign language data comparable under a unified coordinate system and a unified time axis, thereby avoiding the impact of scale and time deviations between different sensor data on the results of state continuity analysis.

[0034] It should be noted that the combination of frequency domain decomposition and multi-timescale aggregation in this step enables the system to capture both short-term fluctuations and long-term stable trends, avoiding misjudgment or omission of subtle but continuous state changes in autism sign language expressions due to reliance on only a single timescale.

[0035] It should be noted that this step introduces a posterior correction fusion mechanism, which enables different autistic individuals to generate stable state reference sequences under a unified standard, even when there are significant differences in their expression rhythm, movement amplitude, and emotional expression. This improves the consistency and transferability of state retention tendency values ​​in cross-individual training scenarios.

[0036] S2, propagation reconstruction of the standard state-preserving tendency value, generating a state-preserving extended sequence and a multi-path propagation chain, and obtaining several second-state-preserving tendency values ​​through resampling.

[0037] In this embodiment, the propagation and reconstruction of the standard state-preservation tendency value to generate a state-preservation extension sequence and a multi-path propagation chain, followed by resampling to obtain several second state-preservation tendency values, specifically: The standard state hold-up tendency value sequence is expanded into an initial state hold-up curve in chronological order, and the curve is then segmented and smoothed to eliminate local abrupt change points. A sliding time window coverage sequence is constructed based on the smoothed state preservation curve. The state preservation strength is repeatedly sampled and expanded under different window spans to generate a multi-scale state preservation expansion sequence. The multi-scale state-preserving extended sequence is processed by path unfolding, and the state-preserving change paths within each time window are connected according to the time progression relationship to generate a set of multi-path state propagation chains. The set of multi-path state propagation chains is perturbed and reconstructed. Under the condition of maintaining the original state trend constraints, the time steps are rearranged and combined to generate a set of state-preserving extended sequences. The set of state-preserving extended sequences is resampled at equal intervals to map each extended sequence to a standard time resolution, resulting in several second state-preserving tendency value sequences. Each second-state hold-up tendency sequence is bound to the standard state hold-up tendency sequence to form a propagated and reconstructed state hold-up sequence group.

[0038] In this embodiment, the standard state maintenance tendency value sequence is a time-seriesd data set output by step S1. It consists of state maintenance strength values ​​corresponding to consecutive time slices arranged in chronological order, and is used to represent the trajectory of the system's tendency to determine the continued validity of the current sign language state in different time slices.

[0039] In this embodiment, the initial state hold curve is a continuous curve expression formed by expanding the standard state hold tendency value sequence in chronological order. Its essence is the continuous interpolation connection result of discrete state hold tendency values ​​on the time axis, which is used to describe the trend of state hold intensity changing over time.

[0040] In this embodiment, the segmented smoothing process involves dividing the initial state hold curve into local time intervals and weakening the abrupt differences between adjacent state hold tendency values ​​within each time interval. This transforms the curve from a high-frequency jump structure to a low-disturbance continuous structure, thereby eliminating unstable changes caused by sampling errors or short-term action fluctuations.

[0041] In this embodiment, the sliding time window coverage sequence is a set of multiple time coverage intervals formed by sliding and trunculating the initial state maintenance curve segment by segment with a fixed time step. Each time window corresponds to a continuous state maintenance intensity segment, which is used to observe the local stability of the state maintenance characteristics in different time ranges.

[0042] In this embodiment, the multi-scale state-preserving extended sequence is a multi-layer time structure sequence formed by repeatedly truncating and superimposing the sliding time window coverage sequence under different window span conditions. It expresses the multiple manifestations of the same state-preserving change under short and long time scales, and is used to enhance the ability to describe state changes with different duration periods.

[0043] In this embodiment, the path unfolding process connects the state change segments within each time window in the multi-scale state preservation extension sequence in chronological order, so that the state preservation change trajectory within each window forms a continuous path structure, which is used to describe the propagation direction and continuity of the state preservation strength during the time evolution process.

[0044] In this embodiment, the multi-path state propagation chain set is a set structure formed by combining multiple state-preserving change paths generated at different time window scales. Each propagation chain corresponds to a state-preserving evolution trajectory at a certain time scale, which is used to express the parallel propagation structure of state-preserving changes at multiple time scales.

[0045] In this embodiment, the perturbation reconstruction process involves locally rearranging and combining the temporal order in the multi-path state propagation chain while maintaining the original state change trend. This results in multiple possible arrangements of the connection relationships between adjacent time slices, which are used to generate extended structures with different time evolution paths but maintaining a consistent overall trend.

[0046] In this embodiment, the set of state-preserving extended sequences is a set of multiple time-rearranged paths after perturbation and reconstruction. It represents the state-preserving change extension trajectory obtained by different time arrangements under the constraint of maintaining the original state-preserving trend, and is used to enhance the diversity of state propagation paths.

[0047] In this embodiment, the equal-interval resampling process involves re-extracting time points from each sequence in the state-preserving extended sequence set at a uniform time interval and aligning and reconstructing them at a standard time resolution, so that sequences of different lengths and time steps are uniformly converted into comparable standard time series.

[0048] In this embodiment, the second state-preservation tendency value sequence is a standardized time series obtained after equal-interval resampling. It is formed by rearranging the state-preservation strength values ​​corresponding to each time slice and is used to represent the re-expression result of the state-preservation tendency after propagation reconstruction and time perturbation.

[0049] In this embodiment, the propagation-reconstructed state-preserving sequence group is a data set formed by binding the second state-preserving tendency value sequence and the original standard state-preserving tendency value sequence according to time index. It is used to compare and analyze the original state-preserving expression and the propagation-expanded state-preserving expression on the same time axis.

[0050] It should be noted that this step introduces a multi-scale window expansion and path unfolding mechanism, so that the original state retention tendency value is no longer limited to a single time trajectory expression, but is expanded into a multi-path time propagation structure, thereby being able to characterize the propagation stability and evolutionary differences of state retention at different time scales.

[0051] It should be noted that this step, through a combination of perturbation reconstruction and equal-interval resampling, enables the state-preserving sequence to generate multiple temporal arrangements while maintaining the overall trend, thereby providing a richer candidate expression space for the subsequent screening of the second state-preserving tendency value.

[0052] It should be noted that the beneficial effect of this step is that by propagating and reconstructing the standard state-maintaining tendency value and expanding it through multiple paths, the system can simultaneously observe state-maintaining behavior under different time structures in autism sign language training scenarios, thereby improving the comprehensive modeling ability of state persistence and change elasticity, and enhancing the stability and robustness of subsequent control parameter selection.

[0053] S3, based on the cosine similarity change of consecutive frames and the difference detection of adjacent time segments, determines the stable state interval and the state change trigger position, and maps the stable state interval to the acquisition time window, and maps the time interval corresponding to the state change trigger position to the state update trigger delay, thereby realizing equivalent control of the state maintenance tendency value.

[0054] In this embodiment, the method of determining the stable state interval and the state change trigger position based on the cosine similarity change of consecutive frames and the difference detection of adjacent time segments, and mapping the stable state interval to the acquisition time window, and mapping the time interval corresponding to the state change trigger position to the state update trigger delay, to achieve equivalent control of the state maintenance tendency value, specifically: Based on the adjacent difference sequences of the second state-preserving tendency value sequence group on the time axis, a continuous frame cosine similarity sequence is constructed, and the difference amplitude of the state vectors of adjacent time slices is calculated to generate a state change detection sequence. The continuous frame cosine similarity sequence is segmented by a sliding window. Within each window, the intervals of continuous increase in similarity and the intervals of stable fluctuation are extracted to form a set of intervals where the candidate state remains stable. Threshold segmentation is performed on the difference amplitude in the state change detection sequence to extract the continuous difference low fluctuation interval and the position of abrupt jump point, forming a set of candidate positions for state change triggering; The set of candidate stable intervals and the set of candidate positions that trigger state changes are time-aligned and matched to filter out effective stable intervals that satisfy the adjacency relationship between the end point of the stable interval and the mutation point. The effective stable intervals are merged into consecutive segments on the time axis to generate a final state stable interval sequence, and its time length is output as the acquisition time window. The time interval difference processing is performed on the set of candidate locations for triggering state changes, and the time interval sequence between adjacent mutation points is output as the state update trigger delay sequence. A state-maintaining tendency value control mapping relationship is constructed based on the acquisition time window sequence and the state update trigger delay sequence to achieve equivalent control output for the second state-maintaining tendency value.

[0055] In this embodiment, the second state preservation tendency value sequence group is a set of multiple time series output by step S2. It is composed of state preservation strength sequences obtained under different propagation reconstruction paths arranged by time index, and is used to express the evolution results of the same state preservation mechanism under different time extension paths.

[0056] In this embodiment, the continuous frame cosine similarity sequence is a time series formed by measuring the directional consistency of adjacent time slice state vectors in the second state tendency value sequence group. It is used to characterize the continuous consistency change trend of the state in time by comparing the directional offset of adjacent state vectors in the feature space.

[0057] In this embodiment, the adjacent time slice state vector is a vector expression formed by organizing the multi-dimensional state preservation feature values ​​corresponding to each time slice according to a unified feature structure. It includes the state preservation strength and its change features corresponding to the time slice, and is used as the basic object for similarity comparison.

[0058] In this embodiment, the state change detection sequence is a joint change description sequence formed by synchronizing the cosine similarity sequence of consecutive frames and the difference amplitude sequence of adjacent time slices. It is used to comprehensively reflect the consistency decrease and structural jump of the state in the continuous time process.

[0059] In this embodiment, the sliding window segmentation process is a process of segmenting the continuous frame cosine similarity sequence over a fixed time span, so that each time window corresponds to a local state change trajectory, which is used to analyze the stability of the state within the local time range.

[0060] In this embodiment, the continuously increasing similarity interval is the time period in which the cosine similarity value continuously increases or remains unchanged within the continuous time slices of the sliding time window. It is used to represent the process in which the state gradually tends to be consistent and stable within this time range.

[0061] In this embodiment, the stable fluctuation range is the time period in which the cosine similarity fluctuates within a small range and the overall change is limited within the sliding time window. It is used to represent a situation where the state remains relatively stable within this time range but there are slight disturbances.

[0062] In this embodiment, the set of candidate stable states is a set of time intervals formed by merging and filtering the intervals with continuously increasing similarity and the stable fluctuation intervals. It represents the range of all possible time periods that may meet the condition of stable state.

[0063] In this embodiment, the difference magnitude in the state change detection sequence is a change intensity sequence formed by calculating the dimension-by-dimensional difference of the state vectors of adjacent time slices, which is used to reflect the degree of abrupt or rapid change of the state in continuous time.

[0064] In this embodiment, the continuous low fluctuation interval is the time period in which the change amplitude of multiple consecutive time slices in the difference amplitude sequence remains within a low level range. It is used to represent the time interval in which the state change is weak and the state is in a stable state.

[0065] In this embodiment, the mutation jump point is the time point in the difference amplitude sequence where a significant increase occurs and exceeds the level of continuous historical change. It is used to mark the time boundary point where the state changes significantly.

[0066] In this embodiment, the set of candidate locations for state change triggering is a set of time points composed of boundary points of continuous difference low fluctuation intervals and abrupt jump points, which is used to describe the potential triggering moment when the state changes from stable to changing or from changing to stable.

[0067] In this embodiment, the time alignment matching process is a process of matching and comparing the set of candidate stable intervals and the set of candidate positions that trigger state changes according to a unified time index, so that the end position of the stable interval and the position of the mutation point are analyzed on the same time axis.

[0068] In this embodiment, the effective stable interval is a segment of interval that satisfies the temporal proximity or close proximity between the end point of the stable interval and the abrupt change point in the time alignment matching process. It is used to filter the effective time period that truly corresponds to the transition from a stable state to a changing state.

[0069] In this embodiment, the continuous segment merging process involves connecting and merging temporally adjacent or overlapping effective stable intervals to form a stable interval sequence with a longer continuous time, which is used to eliminate repetitive or fragmented stable interval expressions.

[0070] In this embodiment, the stable state interval sequence is a set of continuous stable time periods after merging, which represents the time structure within which the system continuously believes that the state remains unchanged without significant change.

[0071] In this embodiment, the acquisition time window is a time control parameter that uses the length of time of the stable state interval sequence as the output result. It is used to control the subsequent data acquisition process to only perform effective sampling within the stable state interval.

[0072] In this embodiment, the state update trigger delay sequence is a time series consisting of the time intervals between adjacent abrupt change points. It is used to represent the time delay length between one state update trigger point and the next trigger point, and is used to control the timing of state updates.

[0073] In this embodiment, the state-maintaining tendency value control mapping relationship is a rule relationship that uses the acquisition time window and the state update trigger delay sequence as input control variables to perform time constraint mapping on the second state-maintaining tendency value sequence group. It is used to transform the abstract state-maintaining tendency value into an executable time control mechanism.

[0074] In this embodiment, the equivalent control output of the second state-maintaining tendency value is the output result after constraining the time sampling and update rhythm of the second state-maintaining tendency value sequence by the combined effect of the acquisition time window and the state update trigger delay. In essence, it indirectly adjusts the effect of the state-maintaining tendency value through time dimension control.

[0075] It should be noted that this step uses a joint analysis method of continuous frame similarity change and state difference detection to enable state stability and state change trigger points to be uniformly identified in the same time structure, thereby avoiding the problem of misjudgment of stable intervals caused by relying on only a single similarity index.

[0076] It should be noted that this step maps the stable interval to the acquisition time window and the change trigger point to the update delay time interval, transforming the originally abstract state maintenance tendency value into an executable time control parameter, thereby realizing the operable control of the state maintenance behavior.

[0077] It should be noted that the beneficial effect of this step is that by constructing a time control mechanism based on stable intervals and change trigger points, the state recognition and feedback update during autism sign language training can be adaptively adjusted in the time dimension, thereby improving the stability of state maintenance judgment and the accuracy of change response.

[0078] S4 applies equivalent control of each second state preserving tendency value to the sign language emotion recognition and training feedback process, and performs collection, recognition and feedback and recording.

[0079] In this embodiment, the application of the equivalent control of maintaining the tendency value of each second state to the sign language emotion recognition and training feedback process, performing collection, recognition, feedback and recording, specifically includes: Obtain the sign language training video stream corresponding to the second state preserving tendency value equivalent control output, and perform frame-level denoising and resolution unification processing on the video stream to generate a standardized video frame sequence; Multimodal keypoint detection processing is performed on standardized video frame sequences to extract hand keypoint coordinate sequences, skeleton pose keypoint coordinate sequences and facial keypoint coordinate sequences, and then uniformly transform them to a normalized coordinate system based on the pelvic center. The normalized keypoint sequence is used to construct a frame-level multimodal feature sequence in chronological order, and the continuous frames are segmented based on a sliding window to generate an action segment sequence. Temporal feature extraction processing is performed on the action segment sequence to calculate the hand displacement difference sequence, joint angle change sequence and facial opening and closing amplitude sequence, and then concatenate them into a segment-level multimodal feature vector; The segment-level multimodal feature vectors are input into the temporal coding process to perform temporal correlation modeling on continuous segments and generate a global sign language expression temporal feature matrix. The global sign language expression temporal feature matrix is ​​input into the emotion recognition process, and the emotion category label and emotion intensity value are output, forming an emotion color expression vector; The emotional color expression vector is matched with the training task label to generate a training matching score sequence; Training feedback labels are generated based on the training matching score sequence, and the feedback labels are synchronously mapped to the second state maintain tendency value control output process, recording the corresponding action segment sequence and feedback result data.

[0080] In this embodiment, the equivalent control of the second state retention tendency value is the control output form after the original state retention tendency value sequence is reparameterized by time structure through the acquisition time window and the state update trigger delay. The acquisition time window is used to limit the time range of stable sampling of emotional state, and the state update trigger delay is used to limit the time lag interval of emotional state determination update, thereby realizing the execution control of the second state retention tendency value in the time dimension.

[0081] In this embodiment, the standardized video frame sequence is a continuous image sequence obtained by uniformizing the frame rate, normalizing the resolution, and suppressing the noise of the original sign language training video stream. Its function is to eliminate the influence of differences in acquisition equipment and illumination fluctuations on subsequent key point extraction and temporal modeling, so that each frame of data can participate in subsequent analysis at a uniform scale.

[0082] In this embodiment, the hand key point coordinate sequence is a set of spatial coordinates obtained by detecting the hand joint positions in each video frame. It is used to describe the motion trajectory of the finger joints and palm in the time dimension. After normalization coordinate transformation, it can be used for cross-individual action alignment analysis.

[0083] In this embodiment, the skeletal posture key point coordinate sequence is a set of spatial coordinates formed after detecting key joints of the upper body of the human body. It is used to describe the structural changes of the torso, shoulders and arms. After being transformed by the unified coordinate system of the pelvic center, it is used to eliminate the differences between individual height and shooting angle.

[0084] In this embodiment, the facial key point coordinate sequence is a set of spatial coordinates obtained by detecting the facial contour and key areas of expression. It is used to describe the deformation changes of the eyebrows, eyes and mouth in the time dimension and to assist in the recognition of emotional state.

[0085] In this embodiment, the frame-level multimodal feature sequence is a unified representation data structure formed by splicing and fusing the hand key point coordinate sequence, the skeleton posture key point coordinate sequence, and the facial key point coordinate sequence under the same time index. It is used to characterize the comprehensive action and emotional state at a single frame moment.

[0086] In this embodiment, the action segment sequence is a set of local time segments formed by continuously segmenting the frame-level multimodal feature sequence through a sliding time window. Each segment contains a fixed number of consecutive frames and is used to characterize the local action evolution process.

[0087] In this embodiment, the hand displacement difference sequence is a sequence obtained by performing time-series difference processing on the changes in the coordinates of key hand points in consecutive frames, and is used to describe the trend of hand movement speed changes.

[0088] In this embodiment, the joint angle change sequence is a time series formed by calculating the change in the angle between adjacent bones based on the key points of the skeleton, and is used to describe the intensity of changes in human posture structure.

[0089] In this embodiment, the facial opening and closing amplitude sequence is a time series formed based on the changes in the distance between key points of the mouth and eyes, used to characterize the degree of facial expression opening and closing changes.

[0090] In this embodiment, the segment-level multimodal feature vector is a feature expression formed by splicing and fusing the hand displacement difference sequence, joint angle change sequence and facial opening and closing amplitude sequence extracted from the action segment sequence, and is used as a comprehensive description of emotion and action in a single time segment.

[0091] In this embodiment, the temporal coding process is a process of inputting segment-level multimodal feature vectors in chronological order and performing temporal dependency modeling, which outputs a global sign language expression temporal feature matrix to characterize the action continuity and semantic association across segments.

[0092] In this embodiment, the global sign language expression temporal feature matrix is ​​a two-dimensional temporal feature structure formed by stacking the temporal encoding results of all action segments, which is used to describe the dynamic changes in the complete sign language expression process.

[0093] In this embodiment, the emotion category label is the discrete category result output after classifying the global sign language expression temporal feature matrix into emotions, which is used to represent the emotion state type corresponding to the current sign language expression.

[0094] In this embodiment, the emotional intensity value is a continuous value obtained by normalizing the score of the consistency of emotional category expression in time segment, which is used to characterize the significance of emotional expression.

[0095] In this embodiment, the emotion color expression vector is a vector structure formed by combining and encoding emotion category labels and emotion intensity values, which is used to uniformly describe emotion type and emotion intensity information.

[0096] In this embodiment, the training task label is a combination of a preset standard sign language movement category and its corresponding target emotional state, which is used as a reference benchmark during the training process.

[0097] In this embodiment, the similarity matching process is a process of comparing and analyzing the consistency between the emotional color expression vector and the training task label in the semantic and emotional dimensions, which is used to generate a training matching score sequence.

[0098] In this embodiment, the training matching score sequence is a time series formed by continuously recording the degree of matching between the sentiment expression and the target task label in each time segment, which is used to reflect the changes in training execution quality.

[0099] In this embodiment, the training feedback marker is a state identifier result obtained by dividing the training matching score sequence into different threshold intervals, which is used to indicate whether the current action execution meets the preset standard.

[0100] It should be noted that this embodiment further introduces time synchronization alignment and normalization constraints during data processing, so that multimodal features participate in fusion calculation under the same time base. At the same time, the sliding window segmentation method ensures the structural consistency between local actions and global emotions, thereby improving the stability and executability of subsequent state-maintaining tendency value control.

[0101] It should be noted that this embodiment achieves dynamic adjustment of the state retention strength during training by mapping the emotion recognition results with the training feedback labels in a closed loop, so that the change in the second state retention tendency value can be directly fed back to the control input. This enhances the system's ability to respond to changes in the micro-expressions and movements of autistic users.

[0102] It should be noted that the beneficial effect of this embodiment is that, through multimodal key point fusion, temporal coding and feedback closed-loop mechanism, a consistent mapping relationship is formed between the sign language emotion recognition result and the training control parameters, thereby realizing controllable output and dynamic optimization of the state maintenance tendency value, and improving the stability and adaptability of training feedback.

[0103] S5 extracts features from the application process of each second state hold-up tendency value, iteratively updates it based on stochastic gradient optimization, and generates the corresponding state hold-up reward value through error feedback.

[0104] In this embodiment, the process of applying each second state-preservation tendency value involves feature extraction, iterative updates based on stochastic gradient optimization, and the generation of corresponding state-preservation reward values ​​through error feedback. Specifically: The training feedback result data is reconstructed by time alignment, and the action segment sequence, emotion category label, emotion intensity value and training matching score sequence are uniformly mapped to the same time index to generate a training interaction time series data set. The training interaction time series data set is processed by feature splitting to extract the emotional stability duration sequence, the emotional state jump number sequence, the action feature deviation amplitude sequence and the matching score fluctuation sequence, and the application feature vector group corresponding to the second state maintenance tendency value is constructed. Gradient direction difference processing is performed on the application feature vector group, an error propagation sequence is constructed based on the feature change trend of adjacent time segments, and the error propagation sequence is updated by random gradient iteration to generate a state update error correction sequence. The state update error correction sequence and the training matching score sequence are subjected to time-aligned differential fusion processing to extract the error offset sequence of each second state hold tendency value in the continuous training process. A state-keeping error feedback sequence is constructed based on the error offset sequence, and the error feedback sequence is cumulatively mapped to generate the corresponding state-keeping reward value sequence. The state-maintaining reward value sequence and the second state-maintaining tendency value sequence are bound together on a time-slice basis to form a reward-tendency corresponding sequence group, and the evaluation data set used for subsequent optimal interval selection is output.

[0105] In this embodiment, the application process corresponding to the second state retention tendency value is the set of records of the entire process after the tendency value participates in the control of the sign language emotion recognition and training feedback system. Specifically, it is the combination process of the corresponding action output, emotion recognition result and feedback execution state in each time segment, which is used to reflect the trajectory of the tendency value in the real training environment.

[0106] In this embodiment, the training feedback result data is a set of complete process records output by the system during sign language training, including action segment sequences, emotion category labels, emotion intensity values, and training matching score sequences, which are used as a unified input data source for subsequent feature extraction and error analysis.

[0107] In this embodiment, the time alignment reconstruction process is a method of synchronizing training feedback data from different sources according to a unified time index, so that action, emotion and rating data are aligned under the same time reference, thereby ensuring that the time consistency is not disrupted in the subsequent feature extraction process.

[0108] In this embodiment, the training interaction time series data set is a comprehensive data structure arranged in chronological order after time alignment and reconstruction. It contains the action execution state and emotional feedback state corresponding to each time segment, which is used to characterize the dynamic evolution of the complete training process.

[0109] In this embodiment, the emotional stability duration sequence is a sequence formed by recording the duration during which the emotional category label remains unchanged in a continuous time segment, and is used to reflect the changes in the time span over which emotions remain stable.

[0110] In this embodiment, the emotional state transition sequence is a sequence formed by recording the number of times the emotional category label changes in the time series over time, and is used to reflect the frequency of emotional state switching.

[0111] In this embodiment, the motion feature deviation magnitude sequence is a sequence formed by recording the degree of difference between the current motion segment features and the standard motion features of the training task on a time-slice basis, which is used to reflect the degree of deviation of the motion execution from the standard.

[0112] In this embodiment, the matching score fluctuation sequence is a record sequence of changes in the training matching score over consecutive time segments, used to reflect the stability and fluctuation of the training performance.

[0113] In this embodiment, the applied feature vector group is a set of vectors formed by concatenating the emotional stability duration sequence, the emotional state jump frequency sequence, the action feature deviation amplitude sequence, and the matching score fluctuation sequence under the same time index. It is used to characterize the behavioral response features of the second state maintenance tendency value during the application process.

[0114] In this embodiment, the error propagation sequence is a sequence obtained by performing directional consistency difference processing on the feature change trends between adjacent time segments in the applied feature vector group, which is used to characterize the deviation propagation path between the system output and the target training state.

[0115] In this embodiment, the stochastic gradient iterative update process is a process of gradually correcting the direction of error change according to the time sequence based on the error propagation sequence. The error change trend of random local time segments is adjusted successively to generate the state update error correction sequence.

[0116] In this embodiment, the state update error correction sequence is the error adjustment result sequence after stochastic gradient iterative update, which is used to represent the corrected state change of each time segment after error propagation adjustment.

[0117] In this embodiment, the error offset sequence is a deviation record sequence obtained by aligning the state update error correction sequence and the training matching score sequence on the time axis, which is used to characterize the degree of deviation between the actual training result and the target matching result.

[0118] In this embodiment, the state-maintaining error feedback sequence is a feedback sequence formed by unifying and integrating the deviation direction and magnitude based on the error offset sequence, which is used to reflect the error feedback trend of the state-maintaining tendency value during the training process.

[0119] In this embodiment, the state-holding reward value sequence is a scoring sequence obtained by cumulative mapping the state-holding error feedback sequence. It is used to represent the performance of different second state-holding tendency values ​​during the training process. The time segments with smaller and more stable errors correspond to higher reward values, while the time segments with larger error fluctuations correspond to lower reward values.

[0120] In this embodiment, the reward-propensity correspondence sequence group is a data set formed by binding the state-maintaining reward value sequence and the second state-maintaining tendency value sequence in a one-to-one correspondence according to the time index, which is used to establish the mapping relationship between different tendency values ​​and their corresponding performance effects.

[0121] It should be noted that this embodiment introduces a time alignment mechanism in the error propagation and reward generation process, so that training feedback data at different time scales can be correlated and analyzed on a unified time reference. At the same time, the one-to-one correspondence between reward value and tendency value is guaranteed by the time slice binding method, thereby avoiding the cumulative impact of statistical bias on the evaluation results.

[0122] It should be noted that this embodiment further introduces a stochastic gradient iterative update mechanism, so that the error correction process does not depend on a single global trend, but is gradually adjusted based on the dynamic changes of local time segments, thereby improving the adaptability to the unstable sign language expressions of autistic users.

[0123] It should be noted that the beneficial effect of this embodiment is that by structuring the training feedback data into a time-aligned feature sequence and introducing an error propagation and reward feedback closed-loop mechanism, the control effect of maintaining the tendency value in the second state can be quantitatively evaluated and continuously optimized, thereby improving the stability and adaptability of the sign language emotion training system.

[0124] S6. Match, sort and filter the state maintenance reward value and the second state maintenance tendency value, extract the value corresponding to the optimal stable interval as the optimal state maintenance tendency value and use it for sign language training feedback control.

[0125] In this embodiment, the process of matching, sorting, and filtering the state maintenance reward value and the second state maintenance tendency value, extracting the value corresponding to the optimal stable interval as the optimal state maintenance tendency value, and using it for sign language training feedback control specifically involves: Time index alignment is performed on the reward-propensity corresponding sequence group, and the second state-maintaining propensity value sequence and the state-maintaining reward value sequence are rearranged along a unified time axis to generate an aligned sequence set. The aligned sequence set is segmented by a continuous sliding window to generate a set of candidate evaluation intervals with a fixed time span, and the interval change trajectory of the reward value sequence and the tendency value sequence is extracted within each candidate evaluation interval. The reward value sequence within the candidate evaluation interval is analyzed for interval fluctuation amplitude. Stable reward intervals with continuously converging reward value changes and mean values ​​maintained at the upper quantile position of the interval are identified and marked as the set of stable reward intervals. Synchronous fluctuation detection is performed on the second state maintains the tendency value sequence in the reward stable interval set to screen the tendency stable interval set in which the tendency value change amplitude continues to decrease and the sequence fluctuation converges. The set of reward-stable intervals and the set of tendency-stable intervals are subjected to time overlap matching to extract the set of overlapping intervals that simultaneously satisfy reward convergence and tendency convergence. The overlapping interval set is sorted by interval duration length, and a stable and preferred interval sequence is generated based on the duration length of time. The first optimal interval in the stable preferred interval sequence is processed by time slice extraction. The second state holding tendency value corresponding to this interval is output as the optimal state holding tendency value and used in the sign language training feedback control process.

[0126] In this embodiment, the reward-propensity correspondence sequence group is a data set formed in the preceding steps by binding the state-maintaining reward value sequence and the second state-maintaining propensity value sequence one by one according to the time index. It is used to describe the one-to-one correspondence between control parameters and effect evaluation in each time segment and is the basic input structure for subsequent interval screening and optimal value extraction.

[0127] In this embodiment, the aligned sequence set is a set of sequences formed by rearranging data from different sources in the reward-propensity corresponding sequence group according to a unified time axis. It is used to eliminate the time misalignment problem caused by inconsistent step sizes of different data records, so that reward values ​​and propensity values ​​can participate in subsequent analysis under the same time reference.

[0128] In this embodiment, the candidate evaluation interval set is a set of multiple continuous time segments obtained by segmenting the aligned sequence set through a fixed-length sliding time window. Each interval contains a complete reward value change sequence and a corresponding tendency value change sequence, which are used as local stability analysis units.

[0129] In this embodiment, the interval change trajectory is a continuous change path formed by the change of the reward value sequence and the tendency value sequence over time within a single candidate evaluation interval, which is used to describe the dynamic relationship between the control effect and the system response within this time period.

[0130] In this embodiment, the reward value sequence is a continuous numerical sequence formed by accumulating and mapping the training feedback error in each time segment, which is used to reflect the overall performance level of the current second state hold tendency value during the training process.

[0131] In this embodiment, the interval fluctuation amplitude analysis is a process of continuously comparing the variation amplitude of the reward value sequence within a specified time window to identify whether the reward value is in a stable state.

[0132] In this embodiment, the stable reward interval set is a set of time periods in the candidate evaluation interval where the change in reward value continuously decreases and the value remains within a relatively stable range for a long time. It is used to characterize the control interval where the system feedback effect is better and more stable.

[0133] In this embodiment, the decrease in the magnitude of the change in the tendency value is a state description after detecting the continuous decreasing trend of the fluctuation of the tendency value in the time series in the second state, which is used to reflect the process of the control parameters gradually stabilizing.

[0134] In this embodiment, the set of tending stable intervals is a set of time intervals in which the fluctuation amplitude of the second state tending value continues to decrease and the change tends to converge, which is further filtered on the basis of the reward stable interval. It is used to represent the region where the control parameters and feedback effects are stable at the same time.

[0135] In this embodiment, the time overlap matching process is the process of performing an intersection operation on the time axis between the set of reward stable intervals and the set of tendency stable intervals, which is used to filter out common time periods that simultaneously satisfy both reward stability and tendency stability.

[0136] In this embodiment, the overlapping interval set is a set of time intervals that simultaneously contain stable reward states and stable tendency states, obtained after time overlap matching, and is used as the basis for optimal interval candidates.

[0137] In this embodiment, the interval duration is the time span during which the overlapping intervals exist continuously on the time axis, and is used to measure the sustainability of the stable state in the time dimension.

[0138] In this embodiment, the stable preferred interval sequence is a set of overlapping intervals sorted from longest to shortest according to their duration, used to screen control intervals that are stable in the long term and have better continuous effects.

[0139] In this embodiment, the optimal state retention tendency value is the second state retention tendency value corresponding to the interval with the longest duration and the highest reward value stability selected from the stable preferred interval sequence, which is used as the final control output parameter in the sign language training feedback process.

[0140] It should be noted that this embodiment introduces a time sliding window and an overlapping matching mechanism in the interval screening process, so that reward stability and tendency stability can be jointly evaluated on a unified time scale, thereby avoiding the problem of misselection caused by fluctuation of a single indicator and improving the reliability of the optimal control interval.

[0141] It should be noted that this embodiment further reconstructs the priority of stable intervals through a persistence length sorting mechanism, enabling the system to prioritize control intervals that are stable in the long term and have consistent feedback, thereby enhancing the stable output capability and adaptability of the training process.

[0142] It should be noted that the beneficial effect of this embodiment is that, through the dual-stability constraint screening mechanism of reward value and tendency value, the optimal state of the final output maintains the tendency value while satisfying control stability and feedback consistency, thereby improving the accuracy and robustness of the autism sign language training feedback system.

[0143] Example 2: This invention also includes a sign language public opinion tendency feedback control system based on emotion color recognition, comprising an alignment modeling module, a propagation reconstruction module, an interval control module, a feedback execution module, an error optimization module, and an interval filtering module. The alignment modeling module aligns and structures the multimodal time-series data of autism sign language, constructs a state continuity association structure, and extracts stable state components by combining similarity constraints and frequency domain decomposition. Based on the stable components, it performs cross-scale aggregation and posterior correction to output a standard state maintenance tendency value. The propagation reconstruction module reconstructs the standard state maintenance tendency value through propagation, generating a state maintenance extension sequence and a multi-path propagation chain, and obtains several second state maintenance tendency values ​​through resampling. The interval control module determines the state based on the cosine similarity change of consecutive frames and the difference detection of adjacent time segments. The system establishes a stable state interval and a state change trigger position, mapping the stable state interval to a data acquisition time window and the time interval corresponding to the state change trigger position to a state update trigger delay, thus achieving equivalent control of the state maintenance tendency value. A feedback execution module applies the equivalent control of each second state maintenance tendency value to the sign language emotion recognition and training feedback process, performing data acquisition, recognition, and feedback, and recording the data. An error optimization module extracts features from the application process of each second state maintenance tendency value, iteratively updates it based on stochastic gradient optimization, and generates the corresponding state maintenance reward value through error feedback. An interval filtering module matches and sorts the state maintenance reward value with the second state maintenance tendency value, extracts the value corresponding to the optimal stable interval as the optimal state maintenance tendency value, and uses it for sign language training feedback control.

[0144] The present invention also includes an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a sign language opinion tendency feedback control method based on emotion color recognition.

[0145] The present invention also includes a computer-readable storage medium storing a computer program that, when executed by a processor, implements a sign language public opinion tendency feedback control method based on emotion color recognition.

[0146] In the embodiments provided by this invention, it should be understood that the disclosed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0147] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0148] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0149] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0150] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0151] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

[0152] In the embodiments provided in this disclosure, it should be understood that the disclosed systems and methods can also be implemented in other ways. The system embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0153] It should be noted that, in this disclosure, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element limited by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0154] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A sign language public opinion tendency feedback control method based on emotion color recognition, characterized in that, Includes the following steps: Alignment and structuring of multimodal temporal data of autism sign language are performed to construct a state continuity association structure. Stable state components are extracted by combining similarity constraints and frequency domain decomposition. Based on the stable components, cross-scale aggregation and posterior correction are performed to output standard state preservation propensity values. The standard state-preservation tendency value is propagated and reconstructed to generate a state-preservation extension sequence and a multi-path propagation chain. Several second state-preservation tendency values ​​are obtained by resampling. Based on the detection of cosine similarity changes in consecutive frames and differences between adjacent time segments, the stable state maintenance interval and the state change trigger position are determined. The stable state maintenance interval is then mapped to a data acquisition time window, and the time interval corresponding to the state change trigger position is mapped to a state update trigger delay, thereby achieving equivalent control of the state maintenance tendency value. Specifically: Based on the adjacent difference sequences of the second state-preserving tendency value sequence group on the time axis, a continuous frame cosine similarity sequence is constructed, and the difference amplitude of the state vectors of adjacent time slices is calculated to generate a state change detection sequence. The continuous frame cosine similarity sequence is segmented by a sliding window. Within each window, the intervals of continuous increase in similarity and the intervals of stable fluctuation are extracted to form a set of intervals where the candidate state remains stable. Threshold segmentation is performed on the difference amplitude in the state change detection sequence to extract the continuous difference low fluctuation interval and the position of abrupt jump point, forming a set of candidate positions for state change triggering; The set of candidate stable intervals and the set of candidate positions that trigger state changes are time-aligned and matched to filter out effective stable intervals that satisfy the adjacency relationship between the end point of the stable interval and the mutation point. The effective stable intervals are merged into consecutive segments on the time axis to generate a final state stable interval sequence, and its time length is output as the acquisition time window. The time interval difference processing is performed on the set of candidate locations for triggering state changes, and the time interval sequence between adjacent mutation points is output as the state update trigger delay sequence. A state-maintaining tendency value control mapping relationship is constructed based on the acquisition time window sequence and the state update trigger delay sequence to achieve equivalent control output for the second state-maintaining tendency value. The equivalent control of maintaining the tendency value of each second state is applied to the sign language emotion recognition and training feedback process to perform collection, recognition and feedback and record; Feature extraction is performed on the application process of each second state hold tendency value, and iterative updates are performed based on stochastic gradient optimization. The corresponding state hold reward value is generated through error feedback. The state maintenance reward value and the second state maintenance tendency value are matched, sorted, and filtered by interval. The value corresponding to the optimal stable interval is extracted as the optimal state maintenance tendency value and used for sign language training feedback control.

2. The sign language public opinion tendency feedback control method based on emotion color recognition according to claim 1, characterized in that, The process involves aligning and structuring multimodal temporal data of autism sign language, constructing a state continuity association structure, and extracting stable state components by combining similarity constraints and frequency domain decomposition. Based on these stable components, cross-scale aggregation and posterior correction are performed to output a standard state preservation propensity value. Specifically: The hand trajectory, posture skeleton and facial expression multimodal data in the autism sign language training process are time-stamped and aligned, and uniformly mapped to the same time axis; Spatial normalization and joint topology reconstruction are performed on the aligned multimodal data to generate structured state sequence units; Cosine similarity analysis is performed on the structured state sequence units to construct a continuous correlation structure of states in adjacent time slices; The state continuity correlation structure is subjected to frequency domain decomposition to separate the low-frequency stable component and the high-frequency disturbance component, while retaining the low-frequency stable component. The low-frequency stable components are subjected to multi-timescale sliding aggregation and posterior correction fusion based on different autism sample distributions to obtain the stable state baseline sequence. The steady-state baseline sequence is subjected to time compression mapping to generate a continuous state preservation strength sequence, and the standard state preservation tendency value is output.

3. The sign language public opinion tendency feedback control method based on emotion color recognition according to claim 2, characterized in that, The standard state-preservation tendency value is propagated and reconstructed to generate a state-preservation extension sequence and a multi-path propagation chain. Several second state-preservation tendency values ​​are obtained through resampling. Specifically: The standard state hold-up tendency value sequence is expanded into an initial state hold-up curve in chronological order, and the curve is then segmented and smoothed to eliminate local abrupt change points. A sliding time window coverage sequence is constructed based on the smoothed state preservation curve. The state preservation strength is repeatedly sampled and expanded under different window spans to generate a multi-scale state preservation expansion sequence. The multi-scale state-preserving extended sequence is processed by path unfolding, and the state-preserving change paths within each time window are connected according to the time progression relationship to generate a set of multi-path state propagation chains. The set of multi-path state propagation chains is perturbed and reconstructed. Under the condition of maintaining the original state trend constraints, the time steps are rearranged and combined to generate a set of state-preserving extended sequences. The set of state-preserving extended sequences is resampled at equal intervals to map each extended sequence to a standard time resolution, resulting in several second state-preserving tendency value sequences. Each second-state hold-up tendency sequence is bound to the standard state hold-up tendency sequence to form a propagated and reconstructed state hold-up sequence group.

4. The sign language public opinion tendency feedback control method based on emotion color recognition according to claim 3, characterized in that, The application of the equivalent control of maintaining the tendency value of each second state to the sign language emotion recognition and training feedback process, specifically performing collection, recognition, feedback and recording, is as follows: Obtain the sign language training video stream corresponding to the second state preserving tendency value equivalent control output, and perform frame-level denoising and resolution unification processing on the video stream to generate a standardized video frame sequence; Multimodal keypoint detection processing is performed on standardized video frame sequences to extract hand keypoint coordinate sequences, skeleton pose keypoint coordinate sequences and facial keypoint coordinate sequences, and then uniformly transform them to a normalized coordinate system based on the pelvic center. The normalized keypoint sequence is used to construct a frame-level multimodal feature sequence in chronological order, and the continuous frames are segmented based on a sliding window to generate an action segment sequence. Temporal feature extraction processing is performed on the action segment sequence to calculate the hand displacement difference sequence, joint angle change sequence and facial opening and closing amplitude sequence, and then concatenate them into a segment-level multimodal feature vector; The segment-level multimodal feature vectors are input into the temporal coding process to perform temporal correlation modeling on continuous segments and generate a global sign language expression temporal feature matrix. The global sign language expression temporal feature matrix is ​​input into the emotion recognition process, and the emotion category label and emotion intensity value are output, forming an emotion color expression vector; The emotional color expression vector is matched with the training task label to generate a training matching score sequence; Training feedback labels are generated based on the training matching score sequence, and the feedback labels are synchronously mapped to the second state maintain tendency value control output process, recording the corresponding action segment sequence and feedback result data.

5. The sign language public opinion tendency feedback control method based on emotion color recognition according to claim 4, characterized in that, The process of applying each second state-preservation tendency value involves feature extraction, iterative updates based on stochastic gradient optimization, and the generation of corresponding state-preservation reward values ​​through error feedback. Specifically: The training feedback result data is reconstructed by time alignment, and the action segment sequence, emotion category label, emotion intensity value and training matching score sequence are uniformly mapped to the same time index to generate a training interaction time series data set. The training interaction time series data set is processed by feature splitting to extract the emotional stability duration sequence, the emotional state jump number sequence, the action feature deviation amplitude sequence and the matching score fluctuation sequence, and the application feature vector group corresponding to the second state maintenance tendency value is constructed. Gradient direction difference processing is performed on the application feature vector group, an error propagation sequence is constructed based on the feature change trend of adjacent time segments, and the error propagation sequence is updated by random gradient iteration to generate a state update error correction sequence. The state update error correction sequence and the training matching score sequence are subjected to time-aligned differential fusion processing to extract the error offset sequence of each second state hold tendency value in the continuous training process. A state-keeping error feedback sequence is constructed based on the error offset sequence, and the error feedback sequence is cumulatively mapped to generate the corresponding state-keeping reward value sequence. The state-maintaining reward value sequence and the second state-maintaining tendency value sequence are bound together on a time-slice basis to form a reward-tendency corresponding sequence group, and the evaluation data set used for subsequent optimal interval selection is output.

6. The sign language public opinion tendency feedback control method based on emotion color recognition according to claim 5, characterized in that, The process of matching, sorting, and filtering the state maintenance reward value and the second state maintenance tendency value, extracting the value corresponding to the optimal stable interval as the optimal state maintenance tendency value, and using it for sign language training feedback control specifically involves: Time index alignment is performed on the reward-propensity corresponding sequence group, and the second state-maintaining propensity value sequence and the state-maintaining reward value sequence are rearranged along a unified time axis to generate an aligned sequence set. The aligned sequence set is segmented by a continuous sliding window to generate a set of candidate evaluation intervals with a fixed time span, and the interval change trajectory of the reward value sequence and the tendency value sequence is extracted within each candidate evaluation interval. The reward value sequence within the candidate evaluation interval is analyzed for interval fluctuation amplitude. Stable reward intervals with continuously converging reward value changes and mean values ​​maintained at the upper quantile position of the interval are identified and marked as the set of stable reward intervals. Synchronous fluctuation detection is performed on the second state maintains the tendency value sequence in the reward stable interval set to screen the tendency stable interval set in which the tendency value change amplitude continues to decrease and the sequence fluctuation converges. The set of reward-stable intervals and the set of tendency-stable intervals are subjected to time overlap matching to extract the set of overlapping intervals that simultaneously satisfy reward convergence and tendency convergence. The overlapping interval set is sorted by interval duration length, and a stable and preferred interval sequence is generated based on the duration length of time. The first optimal interval in the stable preferred interval sequence is processed by time slice extraction. The second state holding tendency value corresponding to this interval is output as the optimal state holding tendency value and used in the sign language training feedback control process.

7. A system using the sign language public opinion tendency feedback control method based on emotional color recognition as described in any one of claims 1-6, characterized in that, It includes an alignment modeling module, a propagation reconstruction module, an interval control module, a feedback execution module, an error optimization module, and an interval filtering module; The alignment modeling module is used to align and structure multimodal time-series data of autism sign language, construct a state continuity association structure, extract stable state components by combining similarity constraints and frequency domain decomposition, perform cross-scale aggregation and posterior correction based on stable components, and output standard state preservation propensity values. The propagation reconstruction module is used to propagate and reconstruct the standard state-preservation tendency value, generate the state-preservation extension sequence and multi-path propagation chain, and obtain several second state-preservation tendency values ​​through resampling. The interval control module is used to determine the stable state interval and the state change trigger position based on the cosine similarity change of continuous frames and the difference detection of adjacent time segments. It also maps the stable state interval to the acquisition time window and the time interval corresponding to the state change trigger position to the state update trigger delay, thereby achieving equivalent control of the state maintenance tendency value. The feedback execution module is used to apply equivalent control of each second state to the sign language emotion recognition and training feedback process, and to perform collection, recognition and feedback and record. The error optimization module is used to extract features from the application process of each second state hold tendency value, perform iterative updates based on stochastic gradient optimization, and generate the corresponding state hold reward value through error feedback. The interval filtering module is used to match, sort, and filter the state maintenance reward value and the second state maintenance tendency value, extract the value corresponding to the optimal stable interval as the optimal state maintenance tendency value, and use it for sign language training feedback control.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the sign language opinion tendency feedback control method based on emotional color recognition as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the sign language public opinion tendency feedback control method based on emotional color recognition as described in any one of claims 1 to 6.

Images