A multi-modal perception feedback fusion adjustment method and system

Abstract

The application discloses a multi-modal perception feedback fusion adjustment method and system, belongs to the technical field of intelligent perception and self-adaptive adjustment, and aims to solve the problems of user state perception lag and intervention strategy adaptability deficiency in a complex scene. By collecting scene characteristics, a scene characteristic interaction layer and an interaction matrix are constructed, feature flattening coding and scene adaptability attention mechanism are used to fuse the characteristics, a scene classification model is constructed in combination with a random forest classifier to predict the scene type, a feature check triggers an update mechanism to update the scene type and match the user state parameter baseline, according to the current scene type, a standard state parameter baseline is matched, user types are divided, and state parameter abnormalities are identified through differential monitoring, when the abnormalities are detected, intervention instructions are matched according to the abnormal direction and deviation level, user states are tracked, and intervention effect features are extracted, adjustment feasibility is judged, and the state parameter baseline is updated. The accuracy and real-time performance of user state management in a complex scene are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent sensing and adaptive adjustment technology, and more specifically to an adjustment method and system for multimodal sensing feedback fusion. Background Technology

[0002] In user status monitoring and management, achieving accurate judgment, timely response, and continuous adaptation in complex environments remains a core challenge. Current methods often have limitations in information processing, lacking sufficient integration and correlation analysis of multi-dimensional information, leading to biased judgments of the actual situation, especially when the environment and user status are dynamically changing, making it difficult to maintain accuracy.

[0003] Meanwhile, existing solutions do not adequately consider individual differences, relying heavily on general standards or fixed models, making it difficult to adapt to the actual characteristics of different objects, resulting in insufficient targeting and effectiveness of response measures. Furthermore, the benchmark information used for reference is outdated, failing to reflect long-term changes or short-term adjustments in a timely manner, thus limiting the accuracy and adaptability of overall management and making it difficult to meet the needs of refined and dynamic management in complex scenarios. Therefore, to overcome these limitations, this invention proposes a multimodal perception feedback fusion adjustment method and system. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a multimodal perception feedback fusion adjustment method and system to solve the problems of accurately identifying the user's scene in complex environments, timely and accurately detecting abnormal user states, formulating appropriate intervention plans, and dynamically updating user state benchmarks, thereby improving the accuracy and adaptability of user state management.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A multimodal sensing feedback fusion modulation system, comprising:

[0007] Scene features are collected, and scene feature interaction layers and interaction matrices are constructed. Scene features are fused using feature flattening encoding and scene adaptability attention mechanism. A scene classification model is constructed by combining a random forest classifier to predict the current scene type. Scene feature verification is used to determine whether to trigger the update mechanism to update the scene type and match the user's state parameter baseline.

[0008] Based on the current scenario type, a standard state parameter baseline is matched, and users are segmented based on the user's state parameter baseline to identify key targets. Through differentiated monitoring, it is determined whether the user has abnormal state parameters.

[0009] When a user's state parameters are abnormal, an intervention instruction is matched based on the direction and deviation level of the user's state parameter values ​​from the standard state parameter baseline of the current scenario type. During the implementation of the intervention instruction, the user's state parameters are tracked and the intervention effect features are extracted to determine whether the user can be intervened and adjusted, and the user's state parameter baseline is updated.

[0010] Specifically, the steps for constructing the scene feature interaction layer and interaction matrix include:

[0011] Acquire training samples for each scene type and label the scene categories. The scene features of the training samples include physical features, temporal features, and spatial features.

[0012] The training samples are processed using feature interpolation, and derivative samples are generated by inserting intermediate state features between the training samples to construct an enhanced sample set.

[0013] A feature association rule library is pre-set for each scene category. The feature association rules are generated based on the statistical analysis results of training samples under each scene type. They include the dependency relationships between different scene features in each scene type, including positive correlation, negative correlation and conditional correlation.

[0014] The association rules in the association rule base include scene feature pairs, association strength coefficients, and applicable scene thresholds;

[0015] The applicable scenario threshold is used to define the range of feature values ​​for scenarios in which association rules take effect;

[0016] Based on the feature association rule base, for each scene category, the interaction matrix is ​​formed by taking scene features as the dimension of the interaction matrix and using the association strength coefficient in the feature association rules as the matrix element value. The interaction matrix element values ​​are then dynamically updated by combining scene features from the enhanced sample set.

[0017] Specifically, the steps for constructing a scene classification model include:

[0018] The scene features are concatenated into a feature vector by feature flattening encoding in a preset dimension order, and then the basic fused feature vector is obtained after nonlinear transformation through a fully connected layer.

[0019] Based on the correlation strength coefficient of scene feature pairs in the interaction matrix, and combined with the numerical distribution data of the current scene features and the applicable scene threshold, a dynamic attention weight vector is generated, and the basic fusion feature vector is weighted and adjusted to obtain the weighted fusion feature.

[0020] The weighted fusion features are input into a classifier built on the random forest algorithm. The predicted probability of each scene category is output through a voting mechanism of multiple decision trees. The predicted scene category is obtained based on the predicted probability.

[0021] Specifically, the steps for determining whether a user has abnormal status parameters through differentiated monitoring include:

[0022] Retrieve the standard state parameter baseline corresponding to the current scene type, and obtain the user's state parameter baseline under this scene type;

[0023] Based on the degree of deviation between the user status parameter baseline and the standard status parameter baseline, users are divided into key focus objects and non-key focus objects;

[0024] For non-key targets, anomaly monitoring is conducted using state parameter trend analysis combined with a fluctuation tolerance mechanism. For key targets, anomaly monitoring is conducted using multi-dimensional collaborative monitoring combined with anomaly amplification mechanism to determine whether there are any state parameter anomalies.

[0025] Specifically, anomaly monitoring employs multi-dimensional collaborative monitoring combined with anomaly amplification mechanisms, including:

[0026] Collect user status parameter values ​​under the current scenario type, construct a real-time status parameter monitoring matrix, calculate the deviation of the current status parameter from the standard status parameter baseline in each status parameter dimension, and the deviation magnitude and direction from the user's status parameter baseline in each status parameter dimension;

[0027] Deviation direction refers to the trend of the current state parameter relative to the user's state parameter baseline, including trending toward the standard state parameter baseline and deviating from the standard state parameter baseline;

[0028] Set the anomaly weights for the state parameter dimension, and adjust the anomaly weights proportionally upwards or downwards based on the direction and magnitude of the deviation.

[0029] Based on the adjusted anomaly weights of each state parameter dimension, the deviations of each state parameter dimension are weighted and summed to obtain the comprehensive anomaly index.

[0030] Set an abnormality trigger threshold. When the comprehensive abnormality index is greater than the abnormality trigger threshold, it is determined that the status parameter is abnormal, and the abnormal status parameter is marked according to the deviation amount.

[0031] Specifically, the steps for matching intervention instructions include:

[0032] Obtain the user's real-time status parameter values ​​and the standard status parameter baseline for the current scene type, calculate and determine the abnormal direction, which includes positive deviation and negative deviation, and quantify the deviation level by the ratio of the deviation amount to the standard status parameter baseline;

[0033] The preset intervention instruction library is classified into primary categories according to scenario type. Each scenario type is further divided into secondary sub-libraries according to the direction of the abnormality. The secondary sub-libraries are further subdivided into tertiary sub-libraries according to the level of deviation. Each tertiary sub-library stores intervention instructions corresponding to the scenario type, the direction of the abnormality, and the degree of deviation. The intervention instructions include basic adjustment parameters, the number of adjustments, and the time for a single adjustment.

[0034] Based on the extracted current scene type, anomaly direction, and deviation level, the system locates the corresponding third-level sub-library of the intervention instruction library and retrieves the intervention instructions.

[0035] Specifically, the steps for determining whether a user is capable of intervention and adjusting their behavior, and for updating the user's baseline status parameters, include:

[0036] During the implementation of intervention instructions, user status parameter values ​​are collected at preset time intervals to generate a status parameter change sequence; intervention effect features are extracted based on the status parameter change sequence.

[0037] Set a threshold for intervention effect features, compare the extracted intervention effect features with the threshold for intervention effect features, and determine whether the user can be intervened and adjusted.

[0038] For users determined to be manageable, the state parameter values ​​within a preset stable period after the intervention command is completed are collected to form a stable state parameter set; statistical analysis is performed on the stable state parameter set to obtain the updated state parameter baseline, which is stored as the user's state parameter baseline under the current scenario type for anomaly detection and intervention.

[0039] Specifically, the steps to determine whether the update mechanism has been triggered include:

[0040] Collect the physical features of the user's scene, record the timestamp of the physical feature collection, extract the time features, obtain the spatial coordinates of the physical feature collection points, extract the spatial features, and form a multi-dimensional scene feature set.

[0041] The standardized multi-dimensional scene feature set is input into the scene classification model to obtain the predicted category of the current scene;

[0042] Based on the predicted category of the current scene, the feature association rule library corresponding to that category in the scene classification model is called to extract the applicable scene thresholds for each scene feature dimension. The scene feature in the multi-dimensional scene feature set is verified in real time to determine whether its scene feature value is within the corresponding applicable scene threshold range.

[0043] If a scene feature dimension exceeds the threshold of the corresponding applicable scene, the scene type update mechanism is triggered, and the deviating feature is marked.

[0044] Specifically, the steps for updating the scene type and matching the user's state parameter baseline include:

[0045] Based on the feature distribution range of each scene type in the scene type information database, candidate scene types are selected by calculating the overlap between the current multi-dimensional scene feature set and the feature distribution range of each scene type.

[0046] Call the interaction matrix constructed by the current feature association rule library, calculate the association strength score between each candidate scene type and the current multi-dimensional scene feature set, sort the association strength scores from largest to smallest, and select the temporary update type;

[0047] When the temporary update type is inconsistent with the predicted scenario type, the scenario transition probability is calculated based on the frequency of switching from the predicted scenario type to the temporary update scenario type according to the user's historical scenario switching records. When the scenario transition probability is greater than the preset transition probability threshold, the scenario type is updated to the temporary update type.

[0048] Extract the user's historical state parameters under the current scenario type, construct a state parameter sequence, process the state parameter sequence using a sliding time window, obtain the statistical distribution characteristics of the state parameters under the current scenario type, and generate a state parameter baseline.

[0049] A method for regulating multimodal sensing feedback fusion, comprising:

[0050] Step S1: Collect scene features, including physical features, temporal features, and spatial features;

[0051] Step S2: By constructing a scene feature interaction layer and interaction matrix, scene features are fused using feature flattening encoding and scene adaptability attention mechanism. A scene classification model is constructed by combining a random forest classifier to predict the current scene type. The scene feature is then verified to determine whether to trigger the update mechanism to update the scene type.

[0052] Step S3: Based on the predicted scenario type, extract the user's historical state parameters under that scenario type and generate the user's state parameter baseline;

[0053] Step S4: Based on the current scenario type, match the standard state parameter baseline, and based on the user's state parameter baseline, classify users, identify key objects of concern, and determine whether the user has abnormal state parameters through differentiated monitoring;

[0054] Step S3: In response to the user's abnormal state parameters, based on the abnormal direction and deviation level of the user's state parameter value and the standard state parameter baseline of the current scenario type, an intervention instruction is matched. During the implementation of the intervention instruction, the user's state parameters are tracked and the intervention effect features are extracted to determine whether the user can be intervened and adjusted, and the user's state parameter baseline is updated.

[0055] The beneficial effects of this invention are:

[0056] This application enhances the accuracy and real-time performance of complex scene recognition by constructing enhanced samples based on multimodal scene features and quantifying feature association using dynamic interaction matrices, combined with a scene-adaptive attention mechanism to achieve scene-specific allocation of feature contribution. Through a differentiated monitoring mechanism based on the deviation of user state parameter baselines from standard baselines, targeted monitoring strategies are adopted for different objects, improving the accuracy and precision of state anomaly detection. Furthermore, by matching intervention instructions according to anomaly direction and level, tracking intervention effects in real time, and dynamically updating state parameter baselines, precise adaptation of intervention strategies and dynamic optimization of the baseline are achieved. This effectively solves problems such as scene judgment bias, inaccurate anomaly detection, insufficient intervention adaptability, and lagging baseline updates in complex environments, significantly improving the accuracy, adaptability, and efficiency of user state management. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the structure of a multimodal sensing feedback fusion control system according to the present invention;

[0058] Figure 2 This is a flowchart illustrating the construction of the scene classification model of the present invention;

[0059] Figure 3 This is a flowchart illustrating the specific steps of the present invention to determine whether a user has abnormal status parameters through differential monitoring;

[0060] Figure 4 This is a flowchart illustrating the anomaly monitoring method employed in this invention, which combines multi-dimensional collaborative monitoring with an anomaly amplification mechanism.

[0061] Figure 5 This is a flowchart of a multimodal sensing feedback fusion adjustment method according to the present invention. Detailed Implementation

[0062] Please see Figure 1 This embodiment introduces a multimodal perception feedback fusion regulation system, including: a data perception module, a coupling analysis module, and an intervention coordination module;

[0063] The data perception module is used to collect scene features in real time, expand the sample through feature interpolation to solve the problem of insufficient model learning in small sample scenarios, and build an enhanced sample set covering the full feature distribution of the scene. It constructs a scene feature interaction layer and interaction matrix to accurately record the dependencies between features in different scenes, quantify the strength of feature associations, and provide a structured basis for feature fusion. Scene feature fusion is achieved using feature flattening encoding and a scene-adaptive attention mechanism, unifying multiple types of features into a processable feature vector. Dynamic weight allocation strengthens the synergistic effect of high-contribution feature combinations, achieving scene-specific allocation of feature contributions. A scene classification model is built by combining a random forest classifier and a hierarchical training strategy. Voting among multiple decision trees improves classification stability, and hierarchical training deeply fits feature associations to improve prediction accuracy. This system is used to accurately predict the type of scenario a user is currently in, enabling real-time identification and classification of complex scenarios. Based on the predicted scenario type, it obtains the corresponding scenario parameter range and generates a status parameter baseline based on the user's historical status parameters under the predicted scenario type. This baseline reflects the user's typical behavior patterns in that scenario, forming a referable behavioral standard to measure the deviation between the user's current status parameters and the standard status in that scenario, thus enabling timely monitoring and early warning of abnormal user states. By continuously collecting the scenario features and status parameter values ​​of the user, the system dynamically updates the scenario type and matches it with the status parameter baseline. This ensures that scenario type identification is calibrated in real-time as features change, and that the status parameter baseline matches the user's latest behavioral characteristics, ensuring the module's long-term adaptability to scenario changes and user state evolution.

[0064] Please see Figure 2 Preferably, the specific steps for constructing a scene classification model include:

[0065] Based on the core scene types involved in the system, training samples covering each scene type are obtained. The training samples under each scene category cover the scene features of each scene type, including physical features such as light intensity and noise frequency distribution, time features such as weekday identification and time period division, and spatial features such as distance from landmarks and distribution of surrounding scenes. Scene category labels are then applied to the training samples of each scene type.

[0066] The training samples are processed using feature interpolation. By inserting intermediate state features between known training samples, derived samples are generated to construct an enhanced sample set that covers the full feature distribution of the scene, thereby solving the problem of insufficient model learning in small sample scenarios.

[0067] A scene feature interaction layer is constructed based on an enhanced sample set. A feature association rule library is preset for each scene category. The feature association rules are generated based on the statistical analysis results of the training samples in the enhanced sample set under that scene type. Specifically, it records the dependencies between different scene features in each scene type, including positive correlation, negative correlation, and conditional correlation between features. Each association rule in the association rule library contains a scene feature pair, an association strength coefficient, and an applicable scene threshold. The association strength coefficient is used to quantify the strength of the association between scene feature pairs. The value reflects the degree of synergistic change of two scene features under specific scene conditions. The applicable scene threshold is used to define the range of scene feature values ​​for which the association rule is effective. For example, in the classroom scene, when the light intensity is in the range of 300 to 500 lux, there is a strong positive correlation between the light intensity and the personnel density during the 9 to 11 hours on weekdays, with an association strength coefficient of 0.82. Through this structured rule recording, the inherent association patterns of features within the scene can be accurately invoked during the feature fusion process, strengthening the synergistic effect of high-contribution feature combinations and providing a basis for the subsequent weight allocation of the dynamic interaction matrix.

[0068] An interaction matrix is ​​constructed based on a feature association rule base. For each scene category, the interaction matrix is ​​formed by using scene features as the dimension and the association strength coefficient in the feature association rules as the matrix element value. The initial interaction matrix for each scene category is then formed by combining scene features in the enhanced sample set and dynamically updating the interaction matrix element values ​​using a sliding window statistical method. This ensures that the initial interaction matrix can reflect the changing trend of feature association relationships in real time. For feature combinations that exceed the applicable scene threshold, the corresponding matrix element value is set to zero to shield invalid associations from interference with feature fusion and ensure the quantification accuracy of scene feature associations by the dynamic interaction matrix.

[0069] The scene features are concatenated into a feature vector by feature flattening encoding in a preset dimension order, and then the basic fused feature vector is obtained after nonlinear transformation through a fully connected layer.

[0070] By introducing a scene-adaptive attention mechanism, based on the association strength coefficients of scene feature pairs in the interaction matrix, and combined with the numerical distribution data of the current scene features and the applicable scene threshold, a dynamic attention weight vector is generated, namely:

[0071] Extract the association strength coefficients of all scene feature pairs in the interaction matrix and establish a set of scene feature association strength coefficients, where each element corresponds to the association strength coefficient value between a pair of scene features.

[0072] Obtain the numerical distribution data of the current scene features, including the specific values ​​of each scene feature and the distribution range of the values ​​in the scene feature dimension, and extract the applicable scene thresholds corresponding to each scene feature.

[0073] For each scene feature, the association strength coefficient of the corresponding scene feature pair is extracted from the set of scene feature association strength coefficients and used as the initial weight reference value for that scene feature dimension.

[0074] For each scene feature dimension, determine whether its current scene feature value is within the applicable scene threshold range: if it is within the range, retain the initial weight reference value corresponding to the scene feature dimension; if it exceeds the range, perform attenuation processing on the initial weight reference value, and the attenuation ratio is set according to the degree of exceeding the threshold. The greater the degree of exceeding, the higher the attenuation ratio.

[0075] Based on the weight reference values ​​of each scene feature dimension after the above processing, and combined with the numerical distribution data of the current scene features, the relative importance ratio of each scene feature dimension in the overall scene feature set is calculated. Specifically, this is obtained by dividing the weight reference value of each scene feature dimension by the sum of the weight reference values ​​of all scene feature dimensions.

[0076] The relative importance of each scene feature dimension is used as an element and arranged in order of scene feature dimension to form a dynamic attention weight vector. The value of each element in this vector corresponds to the weight of the corresponding scene feature dimension in the scene feature fusion process.

[0077] We obtain weighted fusion features by weighting and adjusting the feature dimensions of each scene in the basic fusion features. Specifically, we multiply each feature dimension of the scene with the attention weight vector element by element, so that the weight of the feature dimension with high correlation strength is increased according to the correlation strength coefficient, and the weight of the feature dimension with low correlation strength is suppressed according to the coefficient, thereby achieving scene-specific allocation of feature contribution.

[0078] The weighted fusion features are input into a classifier built on the random forest algorithm. The predicted probability of each scene category is output through a voting mechanism of multiple decision trees. The predicted scene category is obtained based on the predicted probability.

[0079] A loss function is constructed by combining the cross-entropy loss function with a regularization term. The difference between the predicted probability and the true label of each scene category is used as the optimization objective, and the parameter norm is limited to suppress overfitting. A hierarchical training strategy is adopted. First, the model is pre-trained based on the training samples, and then fine-tuned and updated with the full set of augmented samples to deeply fit the feature association. After training, the classification index is evaluated through the test sample set. The feature distribution of scene categories with low accuracy is analyzed. The model is deployed to achieve real-time accurate recognition until all scene indexes meet the standards.

[0080] Preferably, the specific steps for dynamically updating the user's scenario type and matching the baseline state parameters include:

[0081] The system collects physical characteristics of the user's environment using an array of environmental sensors, including light intensity, noise frequency distribution, and ambient temperature and humidity. Simultaneously, it records the timestamps of these physical characteristic acquisitions and extracts time features from the system's calendar and clock information based on these timestamps. These features include weekday and holiday identifiers and hourly time divisions within a 24-hour timeframe. Spatial coordinates of the physical characteristic acquisition points are obtained through spatial positioning. Combined with a pre-stored spatial map database, the system calculates the straight-line distance between these coordinates and preset landmarks, as well as the distribution of surrounding scene types within a preset radius centered on these coordinates. This process extracts spatial features and forms a multi-dimensional scene feature set.

[0082] The multi-dimensional scene feature set is standardized to eliminate the differences in the dimensions of different types of features, so that it is consistent with the feature format during model training; the standardized scene features are then input into the scene classification model to obtain the predicted category of the current scene.

[0083] Based on the predicted category of the current scene, the feature association rule library corresponding to that category in the scene classification model is called to extract the applicable scene thresholds for each scene feature dimension; each scene feature in the multi-dimensional scene feature set is verified in real time to determine whether its scene feature value is within the corresponding applicable scene threshold range.

[0084] Determine if any scene feature dimensions exceed the corresponding applicable scene threshold. If so, trigger the scene type update mechanism and mark the scene features that exceed the corresponding applicable scene threshold as deviation features. Extract the type and degree of deviation of the deviation features. Based on the feature distribution range of each scene type in the scene type information database, calculate the overlap between the current multi-dimensional scene feature set and the feature distribution range of each scene type. Select scene types with an overlap greater than the preset overlap threshold as candidate scene types. The scene type information database refers to the data set of all preset scene types and their corresponding physical features, time features, and spatial feature distribution ranges.

[0085] The interaction matrix constructed by the current feature association rule base is called to calculate the association strength score between each candidate scene type and the current multi-dimensional scene feature set. The interaction matrix is ​​a matrix with features as the dimension and association strength coefficient as the element value, which is used to quantify the association relationship between features in different scene categories. The association strength scores are sorted from largest to smallest, and the scene type with the first position in the sort is selected as the temporary update type.

[0086] The temporary update type is compared with the original predicted type. If they are inconsistent, the user's historical scene switching records are extracted, the frequency of switching from the original predicted type to the temporary update scene type is counted, and the scene transition probability is calculated in combination with the total number of switching records. If the scene transition probability reaches the preset transition probability threshold, the scene type is updated to the temporary update type. If the set standard is not met, the manual confirmation process is initiated, the scene type is received from manual feedback and used as the current scene type.

[0087] Once the scenario type is confirmed, the baseline of state parameters is updated synchronously. Historical state parameters of users under this scenario type are extracted and organized into a state parameter sequence in chronological order. These historical state parameters represent the accumulated state parameter data of users under this scenario type. A sliding time window is used to process the state parameter sequence, covering a set time range. Statistical calculations are performed on the state parameters within the time window to obtain the mean, median, and standard deviation of the state parameters. These values ​​reflect the statistical distribution characteristics of users' typical state parameters under this scenario type, thus generating the state parameter baseline.

[0088] The coupling analysis module matches a standard state parameter baseline based on the current scenario type and, based on the user's state parameter baseline, classifies users, identifies key monitoring targets, and determines whether users have abnormal state parameters through differentiated monitoring. Specifically, it classifies users based on the degree of deviation between the user's state parameter baseline and the standard state parameter baseline: those with deviations below a preset threshold are considered non-key monitoring targets, while those reaching or exceeding the threshold are considered key monitoring targets, with higher deviations indicating higher priority. For non-key monitoring targets, the user's state parameter baseline is used as the core monitoring benchmark, employing trend analysis combined with a fluctuation tolerance mechanism for anomaly monitoring, only determining state parameter anomalies when specific anomaly conditions are met. For key monitoring targets, the standard state parameter baseline is used as the core monitoring benchmark, referencing the user's state parameter baseline, employing multi-dimensional collaborative monitoring combined with anomaly amplification mechanisms. By setting anomaly weights and dynamically adjusting them according to the direction and magnitude of deviation, a comprehensive anomaly index is calculated and compared with a tiered trigger threshold corresponding to the priority level; exceeding the threshold indicates an abnormal state parameter.

[0089] Please see Figure 3 Preferably, the specific steps for determining whether a user has abnormal status parameters through differential monitoring include:

[0090] Based on the current scenario type, retrieve the standard state parameter baseline corresponding to the scenario type, and at the same time extract the user's state parameter baseline under the scenario type;

[0091] The deviation between the user's status parameter baseline and the standard status parameter baseline is calculated. Specifically, the absolute value of the difference between the two in each dimension of the status parameter is calculated, and the result is obtained by weighted summation based on the weight value of each status parameter. The weight value of each status parameter is based on the preset importance of the status parameter. The calculated deviation is compared with the preset basic threshold. If the deviation is lower than the basic threshold, the user is classified as a non-key focus object; otherwise, the user is classified as a key focus object.

[0092] For non-key monitoring targets, considering their low deviation and low need for adjustment in this scenario type, the user status parameter baseline is used as the core monitoring benchmark. Anomaly monitoring is conducted by combining status parameter trend analysis with a fluctuation tolerance mechanism. User status parameter values ​​for the current scenario type are collected in real time, and a status parameter change curve is generated according to the collected time series. The slope of the status parameter change curve is calculated. A trend tolerance range is set based on the slope distribution range of the user's historical status parameter change curves for this scenario type. When the slope of the status parameter change curve is within the trend tolerance range, or the user status parameter value is within the user status parameter baseline, it indicates that the user status parameter change is stable and has not deviated from the user's normal state. This is considered normal fluctuation, relying on user self-adjustment without triggering intervention. Only when the slope of the change curve exceeds the trend tolerance range, and the user status parameter value exceeds the user status parameter baseline, and the cumulative number of times the user status parameter value exceeds the user status parameter baseline within the preset monitoring time window reaches a preset number, is the status parameter considered abnormal. The abnormal status parameter is marked, indicating that the user status parameter has shown a continuous trend of deviating from the user's normal state and has exceeded the range that the user can recover from through self-adjustment.

[0093] Please see Figure 4For key monitoring targets, given their high probability of deviation and high demand for control in this scenario, a multi-dimensional collaborative monitoring system combined with an anomaly amplification mechanism is employed, using the standard state parameter baseline as the core monitoring benchmark and simultaneously referencing the user state parameter baseline. This system collects user state parameter values ​​in real-time for the current scenario, constructs a real-time state parameter monitoring matrix, and calculates the deviation of the current state parameter from the standard state parameter baseline in each state parameter dimension, as well as the deviation magnitude and direction from the user's state parameter baseline in each state parameter dimension. Specifically, the deviation amount refers to the difference between the current state parameter value and the corresponding dimension parameter value of the standard state parameter baseline, directly reflecting the static degree of deviation; the deviation magnitude refers to the absolute value of the difference between the current state parameter value and the corresponding dimension parameter value of the user's state parameter baseline, reflecting the degree of deviation of the current state parameter relative to the user's state parameter baseline; and the deviation direction refers to the trend of the current state parameter relative to the user's state parameter baseline, including trending towards the standard state parameter baseline and deviating from the standard state parameter baseline, used to determine whether the change in the state parameter is developing in a direction consistent with the standard.

[0094] Based on the degree of influence of each state parameter dimension on the user's overall state, and the criticality of that dimension in the current scenario, anomaly weights are assigned to each state parameter dimension. Dimensions with a higher degree of influence on the user's overall state and greater criticality in the current scenario receive larger anomaly weights.

[0095] In the dynamic control phase, if the deviation direction is towards the standard state parameter baseline, the abnormal weight is adjusted downward based on the ratio of the deviation magnitude to the preset magnitude threshold: the smaller the deviation magnitude, that is, the closer it is to the standard state parameter baseline, the larger the adjustment ratio, but the weight after adjustment must not be lower than the preset lower limit value; if the deviation direction is away from the standard state parameter baseline, the abnormal weight is adjusted upward based on the ratio of the deviation magnitude to the preset magnitude threshold value: the larger the deviation magnitude, that is, the farther it is from the standard state parameter baseline, the larger the adjustment ratio, and the weight after adjustment must not exceed the preset upper limit value.

[0096] Based on the adjusted anomaly weights for each state parameter dimension, the deviations of each dimension are weighted and summed to obtain a comprehensive anomaly index. Simultaneously, an anomaly trigger threshold matching the standard state parameter baseline is set, determined based on the allowable fluctuation range of the standard state parameter baseline. When the comprehensive anomaly index exceeds the anomaly trigger threshold, the state parameters are deemed abnormal. At this point, the abnormal state parameters are labeled according to the magnitude of the deviation in each dimension, indicating that the user's state has significantly deviated from the standard range, requiring timely intervention and adjustment measures to bring it back to the standard range.

[0097] The intervention coordination module is used when the coupling analysis module determines that a user's state parameters are abnormal. Based on the direction and deviation level of the abnormality between the user's state parameter values ​​and the standard state parameter baseline for the current scenario type, it matches intervention instructions. During the implementation of the intervention instructions, it tracks the changes in the user's state parameters in real time, extracts intervention effect features, and uses these features to determine whether the user is suitable for intervention and adjustment, updating the user's state parameter baseline. Several pre-effect features showing that the state parameters are continuously approaching the normal range and the degree of abnormality is significantly reduced indicate that the user is suitable for intervention and adjustment. Several pre-effect features showing no significant improvement in state parameters or even an increase in the degree of abnormality indicate that the user is not suitable for intervention and adjustment using the current method, and the intervention strategy needs to be adjusted. The user's state parameter baseline is updated based on the judgment results: for users who are suitable for intervention and adjustment, the upper and lower limits of the state parameter baseline are adjusted based on the stable range of the state parameters after intervention, making it more closely match the user's normal state achievable after intervention; for users who are not suitable for the current intervention method, the fluctuation tolerance range of the state parameter baseline is appropriately optimized by comprehensively considering the state parameter features before and after intervention, and effective intervention and adjustment records are incorporated to enhance the adaptability of the state parameter baseline to the individual user's state, providing a more accurate reference for subsequent interventions.

[0098] Preferably, the specific steps for matching intervention instructions include:

[0099] The system obtains the user's real-time status parameter values ​​and the standard status parameter baseline for the current scenario type, calculates and determines the direction of the anomaly, that is, the direction of deviation of the user's status parameter values ​​relative to the standard status parameter baseline, which is divided into positive deviation and negative deviation. The deviation level is quantified by the ratio of the deviation amount to the standard status parameter baseline, for example, divided into three levels: slight deviation, moderate deviation, and severe deviation.

[0100] The preset intervention instruction library is classified into primary categories according to scenario type. Each scenario type is further divided into secondary sub-libraries according to the direction of the anomaly. The secondary sub-libraries are further subdivided into tertiary sub-libraries according to the deviation level. Each tertiary sub-library stores intervention instructions for the corresponding scenario type, anomaly direction, and deviation level. The intervention instructions include basic adjustment parameters, default number of adjustments, and single adjustment time range.

[0101] Based on the extracted current scene type, anomaly direction, and deviation level, the system locates the corresponding third-level sub-library of the intervention instruction library and retrieves the intervention instructions.

[0102] Preferably, the specific steps for determining whether a user can intervene and adjust, and updating the user's status parameter baseline, include:

[0103] During the implementation of intervention instructions, user status parameter values ​​are collected at preset time intervals to generate a status parameter change sequence. Based on the status parameter change sequence, intervention effect characteristics are extracted, including the real-time rate of regression of status parameters to the normal range, i.e., the reduction value of deviation per unit time; the attenuation of abnormality with the number of adjustments, i.e., the degree of reduction of abnormality level after each adjustment; and the fluctuation stability of status parameters, i.e., the fluctuation range of parameter values ​​within the continuous collection period.

[0104] Thresholds for intervention effect characteristics are set, including the lower limit of regression rate, the lower limit of decay magnitude, and the upper limit of fluctuation stability. The extracted intervention effect characteristics are compared with the thresholds. If all intervention effect characteristics meet the threshold requirements, i.e., the state parameters continuously approach the normal range, the degree of abnormality is significantly reduced, and the fluctuation is stable, the user is determined to be suitable for intervention and adjustment. If any intervention effect characteristic does not meet the threshold requirements, i.e., the state parameters do not improve significantly, the degree of abnormality is aggravated, or the fluctuation is too large, the user is determined to be temporarily unsuitable for intervention and adjustment in the current way, and the user is marked as an uninterventional user.

[0105] For users determined to be intervention-adjustable, the state parameter values ​​within the preset stable period after the intervention instruction are collected to form a stable state parameter set; the stable state parameter set is statistically analyzed to calculate the mean, median and reasonable fluctuation range, and the upper and lower limit thresholds of the state parameter baseline are adjusted accordingly so that the new state parameter baseline covers the parameter range of the stable state after intervention; it is stored as the updated state parameter baseline of the user in the current scenario type for reference in subsequent anomaly detection and intervention adjustment.

[0106] Please see Figure 5 This embodiment introduces a method for regulating multimodal sensing feedback fusion, including:

[0107] Step S1: Collect scene features of the user's location in real time, including physical features such as light intensity and noise frequency distribution, time features such as weekday identification and time period division, and spatial features such as distance from landmarks and distribution of surrounding scenes, to form a scene feature set; use feature interpolation to insert intermediate state features between known samples to generate derived samples, and construct an enhanced sample set covering the full feature distribution of the scene to solve the problem of insufficient model learning in small sample scenarios.

[0108] Step S2: Construct a scene feature interaction layer and a dynamic interaction matrix based on the enhanced sample set; the scene feature interaction layer includes a feature association rule base, recording the dependencies and association strength coefficients between features; the dynamic interaction matrix uses the association strength coefficient as elements to quantify feature association; multi-type features are concatenated into feature vectors using feature flattening encoding, and dynamic attention weight vectors are generated by combining scene adaptability attention mechanism to achieve feature fusion; a scene classification model is constructed through a random forest classifier and a hierarchical training strategy, with multiple decision trees voting to improve stability, hierarchical training to fit feature associations, and accurate prediction of the current scene type; if the scene features exceed the corresponding applicable threshold, a scene type update mechanism is triggered, and the scene type is dynamically calibrated by combining historical switching records or manual confirmation.

[0109] Step S3: Based on the predicted scenario type, extract the user's historical state parameters under that scenario type and organize them into a state parameter sequence according to the time series; use a sliding time window to perform statistical operations on the sequence to obtain the mean, median and standard deviation, and generate a state parameter baseline that reflects the user's normal state; continuously collect scenario features and state parameters, and synchronously update the scenario type and the matching state parameter baseline to ensure that the state parameter baseline matches the user's latest behavioral characteristics.

[0110] Step S4: Based on the current scenario type, retrieve the standard state parameter baseline, calculate the deviation between the user state parameter baseline and the standard state parameter baseline, and sum the weighted absolute values ​​of the differences in each dimension to obtain the deviation degree. Compare the deviation degree with a preset basic threshold. Items with a deviation degree below the basic threshold are considered non-key targets, while those with a deviation degree above the basic threshold are considered key targets. The higher the deviation degree, the higher the priority level. For non-key targets, use the user state parameter baseline as the core monitoring benchmark. Determine anomalies by the slope of the state parameter change curve and the cumulative number of times it exceeds the state parameter baseline. When the slope exceeds the trend tolerance range and the cumulative number reaches a preset number, it is determined that the state parameters are abnormal. For key targets, use the standard state parameter baseline as the core monitoring benchmark, and simultaneously refer to the user state parameter baseline. Calculate the deviation amount, deviation magnitude, and deviation direction of the current parameters from the standard state parameter baseline. Dynamically adjust the anomaly weights of each dimension, lowering the weights when they tend towards the standard state parameter baseline and raising them when they deviate. The weighted sum is used to obtain a comprehensive anomaly index. When the comprehensive anomaly index is greater than the trigger threshold, it is determined that the state parameters are abnormal.

[0111] Step S5: When an abnormality is determined in the status parameters, based on the direction of the abnormality and the level of deviation, the direction of the abnormality is a positive or negative deviation of the user's status parameter value relative to the standard status parameter baseline, and the level of deviation is divided into mild deviation, moderate deviation, and severe deviation. An appropriate intervention instruction is matched from the intervention instruction library, which is categorized by scenario, direction of abnormality, and level of deviation. The intervention instruction includes basic adjustment parameters, default number of adjustments, and the time range for a single adjustment. During the intervention, user status parameters are collected at preset intervals to generate a change sequence. Intervention effect characteristics are extracted, including regression rate, decay amplitude, and fluctuation stability. The intervention effect characteristics are compared with preset thresholds. If all characteristics meet the thresholds, the user is deemed suitable for intervention; if any characteristic does not meet the threshold, the user is deemed temporarily unsuitable for the current intervention method and is marked as an untouchable user. For users who are suitable for intervention, post-intervention stabilization period parameters are collected, and the upper and lower limits of the status parameter baseline are adjusted and updated. For users temporarily unsuitable for the current intervention method, the fluctuation tolerance range of the status parameter baseline is optimized by comprehensively considering parameters before and after the intervention, and effective intervention records are incorporated and the status parameter baseline is updated.

[0112] Working principle and its effects:

[0113] This invention achieves multimodal perception feedback fusion and regulation by constructing a closed-loop mechanism of perception, analysis and intervention. The core lies in relying on the collaborative operation of multiple modules to accurately capture scene and user state characteristics, dynamically generate adaptation strategies and continuously optimize the benchmark, thereby improving the accuracy and adaptability of user state management in complex scenarios.

[0114] First, a data perception module collects multi-dimensional scene features, using feature interpolation to expand the sample size and address the challenge of small-sample learning. A scene feature interaction layer, dynamic interaction matrix, and scene-adaptive attention mechanism are combined to achieve feature fusion and dynamic weight allocation. A random forest classifier and hierarchical training strategy are used to accurately identify scene types and dynamically calibrate them, ensuring that scene judgments are updated in real-time as the environment changes. Simultaneously, a baseline of state parameters tailored to user behavior is generated, providing a personalized benchmark for subsequent monitoring. Based on this, a coupling analysis module categorizes attention objects by calculating the deviation between the user baseline and the standard baseline. For non-priority objects, trend analysis combined with a fluctuation tolerance mechanism is used; for priority objects, multi-dimensional collaborative monitoring combined with anomaly amplification is employed to achieve differentiated and accurate identification of abnormal states. For abnormal states, an intervention collaboration module matches intervention instructions based on the direction and level of the anomaly, tracks the intervention effect in real-time, and dynamically updates the baseline—adjusting the upper and lower limits of the baseline for users who can be intervened, and optimizing the fluctuation tolerance range for users who are not suitable for current intervention, ensuring the adaptability of the intervention strategy and the timeliness of the baseline.

[0115] In summary, this invention improves scene recognition accuracy through multimodal feature fusion, enhances anomaly judgment accuracy through differentiated monitoring, and ensures strategy adaptability through dynamic intervention and baseline updates. It effectively solves the problems of scene judgment bias, coarse anomaly detection, and unstable intervention effect of traditional methods in complex environments, and realizes refined and dynamic management of user status, significantly improving management efficiency and adaptability.

[0116] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal sensing feedback fusion control system, characterized in that, include: Collect scene features of the user's location, including physical features, temporal features, and spatial features; Physical characteristics include light intensity and noise frequency distribution; Time characteristics include weekday identification and time period division; Spatial features include distance to landmarks and distribution of surrounding scenes. By constructing a scene feature interaction layer and interaction matrix, scene features are fused using feature flattening encoding and scene-adaptive attention mechanism. A scene classification model is constructed using a random forest classifier to predict the current scene type. Scene feature verification is used to determine whether to trigger the update mechanism to update the scene type and match the user's state parameter baseline. Based on the current scenario type, a standard state parameter baseline is matched, and users are segmented based on the user's state parameter baseline to identify key targets. Through differentiated monitoring, it is determined whether the user has abnormal state parameters. The specific steps for determining whether a user has abnormal status parameters through differential monitoring include: Retrieve the standard state parameter baseline corresponding to the current scenario type and obtain the user's state parameter baseline under this scenario type; based on the degree of deviation between the user's state parameter baseline and the standard state parameter baseline, divide users into key attention objects and non-key attention objects; for non-key attention objects, use state parameter trend analysis combined with fluctuation tolerance mechanism for anomaly monitoring; for key attention objects, use multi-dimensional collaborative monitoring combined with anomaly amplification mechanism for anomaly monitoring to determine whether there are state parameter anomalies. The method of anomaly detection using multi-dimensional collaborative monitoring combined with anomaly amplification mechanism includes: Collect user status parameter values ​​under the current scenario type, construct a real-time status parameter monitoring matrix, calculate the deviation of the current status parameter from the standard status parameter baseline in each status parameter dimension, and the deviation magnitude and direction from the user's status parameter baseline in each status parameter dimension; The deviation direction refers to the trend of the current state parameter relative to the user's state parameter baseline, including trending toward the standard state parameter baseline and deviating from the standard state parameter baseline. Set the anomaly weights for the state parameter dimension, and adjust the anomaly weights proportionally upwards or downwards based on the direction and magnitude of the deviation. Based on the adjusted anomaly weights of each state parameter dimension, the deviations of each state parameter dimension are weighted and summed to obtain the comprehensive anomaly index. Set an abnormality trigger threshold. When the comprehensive abnormality index is greater than the abnormality trigger threshold, it is determined that the status parameter is abnormal, and the abnormal status parameter is marked according to the deviation amount. When a user's state parameters are abnormal, an intervention instruction is matched based on the direction and deviation level of the user's state parameter values ​​from the standard state parameter baseline of the current scenario type. During the implementation of the intervention instruction, the user's state parameters are tracked and the intervention effect features are extracted to determine whether the user can be intervened and adjusted, and the user's state parameter baseline is updated.

2. The multimodal sensing feedback fusion adjustment system as described in claim 1, characterized in that, The specific steps for constructing the scene feature interaction layer and interaction matrix include: Acquire training samples for each scene type and label the scene categories. The scene features of the training samples include physical features, temporal features, and spatial features. The training samples are processed using feature interpolation, and derivative samples are generated by inserting intermediate state features between the training samples to construct an enhanced sample set. A feature association rule library is preset for each scene category. The feature association rules are generated based on the statistical analysis results of training samples under each scene type. They include the dependency relationships between different scene features in each scene type, including positive correlation association, negative correlation association and conditional association. The association rules in the association rule base include scene feature pairs, association strength coefficients, and applicable scene thresholds; The applicable scenario threshold is used to define the range of scenario feature values ​​in which the association rule takes effect; Based on the feature association rule base, for each scene category, the interaction matrix is ​​formed by taking scene features as the dimension of the interaction matrix and using the association strength coefficient in the feature association rules as the matrix element value. The interaction matrix element values ​​are then dynamically updated by combining scene features from the enhanced sample set.

3. The multimodal sensing feedback fusion adjustment system as described in claim 1, characterized in that, The specific steps for constructing the scene classification model include: The scene features are concatenated into a feature vector by feature flattening encoding in a preset dimension order, and then the basic fused feature vector is obtained after nonlinear transformation through a fully connected layer. Based on the correlation strength coefficient of scene feature pairs in the interaction matrix, and combined with the numerical distribution data of the current scene features and the applicable scene threshold, a dynamic attention weight vector is generated, and the basic fusion feature vector is weighted and adjusted to obtain the weighted fusion feature. The weighted fusion features are input into a classifier built on the random forest algorithm. The predicted probability of each scene category is output through a voting mechanism of multiple decision trees. The predicted scene category is obtained based on the predicted probability.

4. The multimodal sensing feedback fusion adjustment system as described in claim 1, characterized in that, The specific steps of the matching intervention instruction include: Obtain the user's real-time status parameter values ​​and the standard status parameter baseline for the current scene type, calculate and determine the abnormal direction, which includes positive deviation and negative deviation, and quantify the deviation level by the ratio of the deviation amount to the standard status parameter baseline; The preset intervention instruction library is classified into primary categories according to scenario type. Each scenario type is further divided into secondary sub-libraries according to the direction of the abnormality. The secondary sub-libraries are further subdivided into tertiary sub-libraries according to the level of deviation. Each tertiary sub-library stores intervention instructions corresponding to the scenario type, the direction of the abnormality, and the degree of deviation. The intervention instructions include basic adjustment parameters, the number of adjustments, and the time for a single adjustment. Based on the extracted current scene type, anomaly direction, and deviation level, the system locates the corresponding third-level sub-library of the intervention instruction library and retrieves the intervention instructions.

5. The multimodal sensing feedback fusion adjustment system as described in claim 1, characterized in that, The specific steps for determining whether a user can intervene and adjust, and updating the user's status parameter baseline, include: During the implementation of intervention instructions, user status parameter values ​​are collected at preset time intervals to generate a status parameter change sequence; intervention effect features are extracted based on the status parameter change sequence. Set a threshold for intervention effect features, compare the extracted intervention effect features with the threshold for intervention effect features, and determine whether the user can be intervened and adjusted. For users determined to be manageable, the state parameter values ​​within a preset stable period after the intervention command is completed are collected to form a stable state parameter set; statistical analysis is performed on the stable state parameter set to obtain the updated state parameter baseline, which is stored as the user's state parameter baseline under the current scenario type for anomaly detection and intervention.

6. The multimodal sensing feedback fusion adjustment system as described in claim 1, characterized in that, The specific steps for determining whether the update mechanism has been triggered include: Collect the physical features of the user's scene, record the timestamp of the physical feature collection, extract the time features, obtain the spatial coordinates of the physical feature collection points, extract the spatial features, and form a multi-dimensional scene feature set. The standardized multi-dimensional scene feature set is input into the scene classification model to obtain the predicted category of the current scene; Based on the predicted category of the current scene, the feature association rule library corresponding to that category in the scene classification model is called to extract the applicable scene thresholds for each scene feature dimension. The scene feature in the multi-dimensional scene feature set is verified in real time to determine whether its scene feature value is within the corresponding applicable scene threshold range. If a scene feature dimension exceeds the threshold of the corresponding applicable scene, the scene type update mechanism is triggered, and the deviating feature is marked.

7. The multimodal sensing feedback fusion control system as described in claim 1, characterized in that, The specific steps for updating the scenario type and matching the user's status parameter baseline include: Based on the feature distribution range of each scene type in the scene type information database, candidate scene types are selected by calculating the overlap between the current multi-dimensional scene feature set and the feature distribution range of each scene type. Call the interaction matrix constructed by the current feature association rule library, calculate the association strength score between each candidate scene type and the current multi-dimensional scene feature set, sort the association strength scores from largest to smallest, and select the temporary update type; When the temporary update type is inconsistent with the predicted scenario type, the scenario transition probability is calculated based on the frequency of switching from the predicted scenario type to the temporary update scenario type according to the user's historical scenario switching records. When the scenario transition probability is greater than the preset transition probability threshold, the scenario type is updated to the temporary update type. Extract the user's historical state parameters under the current scenario type, construct a state parameter sequence, process the state parameter sequence using a sliding time window, obtain the statistical distribution characteristics of the state parameters under the current scenario type, and generate a state parameter baseline.

8. A method for regulating multimodal sensing feedback fusion, implemented based on a regulating system for multimodal sensing feedback fusion as described in any one of claims 1-7, characterized in that, include: Step S1: Collect scene features of the user's location, including physical features, temporal features, and spatial features; Step S2: By constructing a scene feature interaction layer and interaction matrix, scene features are fused using feature flattening encoding and scene adaptability attention mechanism. A scene classification model is constructed by combining a random forest classifier to predict the current scene type. The scene feature is then verified to determine whether to trigger the update mechanism to update the scene type. Step S3: Based on the predicted scenario type, extract the user's historical state parameters under that scenario type and generate the user's state parameter baseline; Step S4: Based on the current scenario type, match the standard state parameter baseline, and based on the user's state parameter baseline, classify users, identify key objects of concern, and determine whether the user has abnormal state parameters through differentiated monitoring; The specific steps for determining whether a user has abnormal status parameters through differential monitoring include: Retrieve the standard state parameter baseline corresponding to the current scenario type and obtain the user's state parameter baseline under this scenario type; based on the degree of deviation between the user's state parameter baseline and the standard state parameter baseline, divide users into key attention objects and non-key attention objects; for non-key attention objects, use state parameter trend analysis combined with fluctuation tolerance mechanism for anomaly monitoring; for key attention objects, use multi-dimensional collaborative monitoring combined with anomaly amplification mechanism for anomaly monitoring to determine whether there are state parameter anomalies. Step S5: In response to the user's abnormal state parameters, based on the abnormal direction and deviation level of the user's state parameter value and the standard state parameter baseline of the current scenario type, an intervention instruction is matched. During the implementation of the intervention instruction, the user's state parameters are tracked and the intervention effect features are extracted to determine whether the user can be intervened and adjusted, and the user's state parameter baseline is updated.

Images