Method for analyzing multi-modal data in a motion scenario, electronic device and storage medium
By generating feature vector sequences of multimodal data and calculating weight distribution, the problem of signal interference from sensors in bumpy or moving scenarios is solved, thus improving the accuracy of health monitoring.
Patent Information
- Application Number
- CN202610291131.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-11
- Publication Date
- 2026-05-19
- Estimated Expiration
- 2046-03-11
AI Technical Summary
In bumpy or moving environments, sensor signals are easily interfered with, leading to low accuracy in health monitoring results.
By acquiring temperature value sequences, heart rate variability feature sequences, and electrical conductance feature sequences, first and second feature vector sequences are generated. The weight distribution is calculated using an anomaly detection algorithm, and multiple methods are combined to determine the classification label of pain level, reducing the impact of interference from single sensor signals.
It improves the accuracy of health monitoring in bumpy or moving environments, reduces the impact of interference from single sensor signals, and is suitable for real-time monitoring.
Smart Images
Figure CN121812195B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of data processing technology, specifically relating to a method for analyzing multimodal data in motion scenarios, an electronic device, and a storage medium. Background Technology
[0002] With the development of IoT and AI technologies, wearable devices are increasingly being used in health monitoring. These devices typically integrate multiple sensors to collect various data in real time, such as body temperature, heart rate, and blood pressure. However, in bumpy or mobile environments, sensor signals are easily interfered with, affecting the accuracy of monitoring results. Summary of the Invention
[0003] One objective of this invention is to provide a method, electronic device, and storage medium for analyzing multimodal data in motion scenarios, which can solve the technical problem of low accuracy of monitoring results due to interference with sensor signals in bumpy or moving scenarios in the prior art.
[0004] According to a first aspect of the present invention, a method for analyzing multimodal data in a motion scene is provided, comprising:
[0005] Acquire first data within a preset time period, wherein the first data includes at least a temperature value sequence, a heart rate variability characteristic sequence, and an electrical conductance characteristic sequence;
[0006] Based on the temperature value sequence and the heart rate variability feature sequence, the first feature vector sequence is obtained;
[0007] Based on the temperature value sequence, heart rate variability feature sequence, and conductance feature sequence, a second feature vector sequence is obtained;
[0008] The first feature vector sequence is fused to obtain the first fused vector;
[0009] The second feature vector sequence is fused to obtain the second fused vector;
[0010] The first weight distribution corresponding to the first fusion vector and the second weight distribution corresponding to the second fusion vector are obtained based on the anomaly detection algorithm.
[0011] The classification label of the state is determined based on the first weight distribution and the second weight distribution;
[0012] The first weight distribution or the second weight distribution includes the category label and the weight corresponding to the category label.
[0013] Optionally, the data acquisition sites for the first data include at least a first region and a second region, and at least one temperature sensor and one photoplethysmography sensor are provided in the first region; at least one temperature sensor and a set of electrodermal sensors are provided in the second region.
[0014] Optionally, the heart rate variability feature sequence includes short-term heart rate variability features, which include pNN50 features.
[0015] Optionally, the process of obtaining the first fusion vector or the second fusion vector includes:
[0016] Determine the sampling coverage ratio corresponding to the preset time window based on the preset time window;
[0017] The first time window with the maximum coverage ratio is obtained based on the sampling coverage ratio;
[0018] The first feature vector sequence is sampled and pooled based on the first time window to obtain the first fused vector;
[0019] The second feature vector sequence is sampled and pooled based on the first time window to obtain the second fused vector.
[0020] Optionally, the process of obtaining the first fusion vector and the second fusion vector includes:
[0021] Based on the first coverage ratio of the data type and the first feature vector sequence, the first effective data sequence within the first feature vector sequence is obtained;
[0022] Based on the second coverage ratio and the second feature vector sequence of the data type, the second effective data sequence within the second feature vector sequence is obtained;
[0023] The first valid data sequence is sampled and pooled to obtain the first fusion vector;
[0024] The second valid data sequence is sampled and pooled to obtain the second fusion vector;
[0025] The first coverage ratio or the second coverage ratio is the ratio of the number of non-empty measurements to the total number of measurements.
[0026] Optionally, the anomaly detection algorithm is a clustering algorithm. When performing clustering, preset data and a first fusion vector or a second fusion vector are used for clustering, and the weights corresponding to the labels are calculated based on the labels corresponding to the vectors contained in the clusters obtained by classification.
[0027] Optionally, the process of determining the classification label of the state includes:
[0028] The weights corresponding to the same state level in the first weight distribution and the second weight distribution are added together to obtain the comprehensive weight distribution;
[0029] Based on the weights in the comprehensive weight distribution, the state level with the highest weight is determined as the classification label for the current state.
[0030] Optionally, the process of determining the classification label of the state includes:
[0031] The corresponding first vector and second vector are obtained based on the first weight distribution and the second weight distribution;
[0032] Obtain the first Hamad product of the first vector and the second vector;
[0033] The classification label of the current state is determined based on the first Hamada product;
[0034] The first vector or the second vector is used to represent the distribution of classification labels in the first weight distribution or the second weight distribution.
[0035] According to a second aspect of the present invention, an electronic device is provided, including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions being executed by the processor to implement the steps of the method for analyzing multimodal data in motion scenarios as described in the first aspect of the present invention.
[0036] According to a third aspect of the present invention, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method for analyzing multimodal data in a motion scene as described in the first aspect of the present invention.
[0037] The beneficial effects of this invention are as follows: By obtaining a first feature vector sequence and a second feature vector sequence, this invention determines the weights corresponding to each pain level through multiple methods, reducing the impact of interference from a single sensor signal. Furthermore, cross-validation using the first weight distribution and the second weight distribution improves accuracy. Attached Figure Description
[0038] Figure 1 This is a flowchart of the method for analyzing multimodal data in motion scenarios in an embodiment of the present invention. Detailed Implementation
[0039] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of the invention.
[0040] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.
[0041] Techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification. In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.
[0042] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.
[0043] In the specification of this invention, the terms "first" and "second" may explicitly or implicitly include one or more of the same feature. In the description of this invention, unless otherwise stated, "multiple" means two or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0044] like Figure 1 As shown in the figure, this embodiment introduces a method for analyzing multimodal data in motion scenarios to monitor a user's pain level in bumpy or moving situations. The method includes steps 1100-1700.
[0045] Step 1100: Obtain first data within a preset time period, the first data including at least a temperature value sequence, a heart rate variability characteristic sequence, and an electrical conductance characteristic sequence.
[0046] The preset time period is the time during which the user exercises, including bumpy and moving scenarios, such as ambulance transport scenarios. The first data is the data during the user's exercise process.
[0047] The temperature value sequence contains multiple temperature values within a preset time period. These temperature values represent the user's body temperature, which can be the temperature of a specific part of the user's body. The temperature value sequence is used to reflect peripheral blood circulation and autonomic nervous system regulation. Temperature values can be acquired at specified time intervals to form the temperature value sequence.
[0048] Heart rate variability (HRV) feature sequences include short-term HRV features, which include the pNN50 feature. pNN50 is a core indicator in HRV analysis, representing the percentage of heartbeats with a difference greater than 50 ms between adjacent sinus beats, primarily reflecting the intensity of parasympathetic activity. The pNN50 feature changes under pain or stress conditions.
[0049] Electrical conductance sequence represents skin electrical activity and is used to reflect sympathetic nerve excitation, emotional arousal, or stress response.
[0050] The first data can be collected by sensors. Specifically, the collection points for the first data include at least a first region and a second region, and at least one temperature sensor and one photoplethysmography sensor are provided in the first region; at least one temperature sensor and a set of electrodermal sensors are provided in the second region.
[0051] The first and second regions are two different areas of the user's body. For example, the first region could be the head area, and the second region could be the wrist area.
[0052] The temperature value sequence includes multiple temperature values detected by a temperature sensor in the first area within a preset time period, and multiple temperature values detected by a temperature sensor in the second area within a preset time period. The temperature value sequence includes temperature values from two different areas of the user's body.
[0053] Photoplethysmography (PPG) sensors are used to acquire heart rate variability characteristic sequences. Electrodermal (EDS) sensors are used to acquire conductance characteristic sequences.
[0054] After acquiring temperature value sequences, heart rate variability characteristic sequences, and conductance characteristic sequences through the aforementioned temperature sensor, volumetric plethysmography sensor, and electrodermal sensor, the raw values acquired by the sensors are normalized for subsequent processing.
[0055] Step 1200: Obtain the first feature vector sequence based on the temperature value sequence and the heart rate variability feature sequence.
[0056] The first feature vector integrates temperature value and heart rate variability features, mainly reflecting the relationship between cardiovascular and temperature regulation, and is relatively resistant to exercise interference. The sequence of first feature vectors contains multiple first feature vectors, and each first feature vector contains at least one temperature value and at least one heart rate variability feature.
[0057] Step 1300: Obtain the second feature vector sequence based on the temperature value sequence, heart rate variability feature sequence, and conductance feature sequence.
[0058] The second feature vector sequence adds a conductance feature sequence, which is obtained through a skin conductance sensor. This conductance feature sequence is very sensitive to pain and pressure, but is easily affected by motion artifacts. Therefore, it needs to be used in conjunction with the temperature value sequence and the heart rate variability feature sequence.
[0059] The second feature vector sequence contains multiple second feature vectors, each of which contains at least one temperature value, at least one heart rate variability feature, and at least one conductance feature.
[0060] Step 1400: Fuse the first feature vector sequence to obtain the first fused vector.
[0061] Step 1500: Fuse the second feature vector sequence to obtain the second fused vector.
[0062] Step 1600: Obtain the first weight distribution corresponding to the first fusion vector and the second weight distribution corresponding to the second fusion vector based on the anomaly detection algorithm.
[0063] The anomaly detection algorithm calculates anomaly scores in the feature space for the first and second fused vectors, respectively, and assigns weights based on these scores. A high anomaly score for a fused vector indicates potential motion noise contamination, and it is assigned a lower weight. Conversely, a low anomaly score indicates a reliable signal, and it is assigned a higher weight.
[0064] The first weight distribution represents the pain level determined based on the first fusion vector and its corresponding weight. The weight corresponding to the pain level indicates the confidence level of that pain level; the higher the weight, the higher the confidence level. For example, in one example, the first weight distribution could be: {L1: 0.1, L2: 0.3, L3: 0.4, L4: 0.2}, meaning that the weight of pain level L1 is 0.1, the weight of pain level L2 is 0.3, the weight of pain level L3 is 0.4, and the weight of pain level L4 is 0.2.
[0065] The second weight distribution represents the pain level and its weight determined based on the second fusion vector.
[0066] Step 1700: Determine the classification label of the state based on the first weight distribution and the second weight distribution. The first weight distribution or the second weight distribution includes the classification label and the weight corresponding to the classification label.
[0067] Category labels are used to indicate pain levels; the category label for a status indicates the user's pain level in the current status.
[0068] This embodiment obtains a first feature vector sequence and a second feature vector sequence, and determines the weights corresponding to each pain level through multiple methods, reducing the impact of interference from a single sensor signal. Furthermore, cross-validation is performed using the first weight distribution and the second weight distribution to improve accuracy.
[0069] In one embodiment of this application, the process of determining the classification label of the state includes: adding the weights of the same state level in the first weight distribution and the second weight distribution to obtain a comprehensive weight distribution; sorting the weights in the comprehensive weight distribution according to their magnitude, and determining the state level with the largest weight as the classification label of the current state.
[0070] The overall weight distribution also includes multiple pain levels and the corresponding weight for each pain level. For each pain level, its weight in the first weight distribution is added to its weight in the second weight distribution to obtain the overall weight for that pain level.
[0071] For example, the first weight distribution is {L1: 0.1, L2: 0.3, L3: 0.4, L4: 0.2}, and the second weight distribution is {L1: 0.2, L2: 0.5, L3: 0.2, L4: 0.1}. Pain level L1 has a weight of 0.1 in the first weight distribution and a weight of 0.2 in the second weight distribution. Adding these two weights together gives a combined weight of 0.3 for pain level L1. The combined weights for other pain levels are calculated in the same way. The final combined weight distribution is {L1: 0.3, L2: 0.8, L3: 0.6, L4: 0.3}. L2 has the largest weight in this distribution, so the final classification label for the current state is pain level L2.
[0072] This embodiment obtains a comprehensive weight distribution by adding the weights of the same state level in the first weight distribution and the second weight distribution, and determines the state level with the largest weight as the classification label of the current state. It does not require a complex model, has high computational efficiency, and is suitable for real-time monitoring.
[0073] In another embodiment of this application, the process of determining the classification label of the state includes: obtaining a corresponding first vector and a second vector according to the first weight distribution and the second weight distribution; obtaining a first Hamad product of the first vector and the second vector; determining the classification label of the current state according to the first Hamad product; the first vector or the second vector is used to represent the distribution of classification labels in the first weight distribution or the second weight distribution.
[0074] The first vector is the weight value vector extracted from the first weight distribution, and the second vector is the weight value vector extracted from the second weight distribution. For example, if the first weight distribution is {L1: 0.1, L2: 0.3, L3: 0.4, L4: 0.2}, and the second weight distribution is {L1: 0.2, L2: 0.5, L3: 0.2, L4: 0.1}, then the first vector is [0.1, 0.3, 0.4, 0.2], and the second vector is [0.2, 0.5, 0.2, 0.1].
[0075] The Hamada product is an element-wise multiplication. Calculating the Hamada product for the first and second vectors yields the first Hamada product as [0.02, 0.15, 0.08, 0.02]. Then, the pain state corresponding to the maximum value in the first Hamada product is found. In the above example, the maximum value in the first Hamada product is 0.15, corresponding to pain level L2. Therefore, the final classification label for the current state is pain level L2.
[0076] Since the Hamad product calculates the element-wise product of two distributions, the size of the product depends on the consistency of the two distributions in that category. A large product only occurs when both modalities are given high weights; giving either modality a low weight significantly reduces the product value. In the application scenario of this application, a high-confidence judgment is only given when both the first and second weighted distributions point to the same pain level, avoiding misjudgments caused by interference from a single modality and effectively improving the accuracy of the judgment results.
[0077] In one embodiment of this application, the process of obtaining the first fusion vector and the second fusion vector includes: determining the sampling coverage ratio corresponding to a preset time window; obtaining a first time window with the maximum coverage ratio based on the sampling coverage ratio; sampling and pooling a first feature vector sequence based on the first time window to obtain a first fusion vector; and sampling and pooling a second feature vector sequence based on the first time window to obtain a second fusion vector.
[0078] A series of candidate time window lengths can be preset, such as 0.2s, 0.5s, 2s, 5s, etc. The effective mode duration of the signal varies under different motion intensities. Presetting multiple time windows can adapt to different actual situations.
[0079] The sampling coverage ratio is the ratio of the number of valid data points to the theoretical total number of data points within a given time window. In bumpy or moving environments, sensors may lose data due to motion artifacts, temporary detachment, and other factors. The sampling coverage ratio reflects the integrity and reliability of the data within that time window.
[0080] The system traces backward from the current time, attempting to capture data within each preset time window, calculating the sampling coverage ratio for each window, and selecting the window with the highest coverage ratio as the first time window. During periods of severe turbulence, the coverage ratio of longer time windows is low, so the system automatically selects shorter, relatively complete time windows for analysis, sacrificing some temporal resolution to ensure input quality. During periods of stability, the system selects longer time windows to obtain more stable and statistically significant data.
[0081] For example, if the preset time windows are 2s, 5s, and 10s, and the sensor sampling frequency is 5 samples per second, then theoretically the sensor should collect 10 data points in 2s, 25 data points in 5s, and 50 data points in 10s. After backtracking, the number of data points per second within 10s is found to be 5, 3, 5, 4, 5, 5, 3, 3, 2, 5. Therefore, there are 8 data points in 2s, 22 data points in 5s, and 40 data points in 10s. The calculated coverage ratio is 0.8 for a 2s time window, 0.88 for a 5s time window, and 0.8 for a 10s time window. The time window with the highest coverage ratio is 5s, so the first time window is 5s.
[0082] Temperature values are acquired using a temperature sensor, containing temperature values collected at multiple different time points. Similarly, the heart rate variability (HRV) feature sequence also contains HRV features at multiple different time points. The first feature vector sequence is generated from the temperature value sequence and the HRV feature sequence, containing multiple first feature vectors, each corresponding to a different time point. The first feature vector sequence is sampled through a first time window, extracting multiple first feature vectors that fall within the first time window. Pooling operations, such as average pooling, are performed on the extracted first feature vectors. Since the first feature vector has n dimensions, the pooling operation calculates the average of these n dimensions within the time window, resulting in an n-dimensional first fusion vector.
[0083] The method for obtaining the second fusion vector is the same as that for the first fusion vector, and will not be repeated here.
[0084] This embodiment determines a first time window with the maximum sampling coverage ratio, filtering out time windows with poor data quality based on the coverage ratio. Then, it samples and pools the first and second feature vector sequences within the first time window. Pooling reduces the impact of outlier data points within the time window and compresses multiple feature vectors within a time period into one, significantly reducing the computational burden and meeting the requirements of real-time monitoring. A fused vector represents the state over a period of time, which is more stable than a single instantaneous point and better reflects a continuous pain state rather than transient interference.
[0085] In another embodiment of this application, the process of obtaining the first fusion vector and the second fusion vector includes: obtaining a first effective data sequence within the first feature vector sequence based on a first coverage ratio of the data type and a first feature vector sequence; obtaining a second effective data sequence within the second feature vector sequence based on a second coverage ratio of the data type and a second feature vector sequence; sampling and pooling the first effective data sequence to obtain the first fusion vector; sampling and pooling the second effective data sequence to obtain the second fusion vector; wherein the first coverage ratio or the second coverage ratio is the ratio of the number of non-empty measurements to the number of measurements.
[0086] For a sequence of first feature vectors, determine whether each first feature vector is valid. Each first feature vector contains multiple data points. For example, if a first feature vector contains 5 data values, and all 5 data values are valid, then the first coverage ratio of the first feature vector is 1.0. If only 4 data values are valid, then the first coverage ratio of the first feature vector is 0.8.
[0087] After calculating the first coverage ratio of each first feature vector in the first feature vector sequence, the first feature vectors with a coverage ratio reaching a certain threshold are selected to form the first effective data sequence.
[0088] The method for obtaining the second valid data sequence is the same as that for the first valid data sequence, and will not be repeated here.
[0089] In this embodiment, the anomaly detection algorithm is a clustering algorithm. When performing clustering, preset data and a first fusion vector or a second fusion vector are used to perform clustering, and the weights corresponding to the labels are calculated based on the labels corresponding to the vectors contained in the clusters obtained by classification.
[0090] The first fusion vector is clustered with the preset data to obtain multiple clusters. The distance from the first fusion vector to the center of each cluster is calculated, and the cluster with the closest distance is the cluster obtained from the above classification. The label set of all preset data in the nearest cluster is examined, and the frequency of each label in the cluster is counted. The frequency is normalized to form a probability distribution, which is the weight distribution.
[0091] For example, if the cluster obtained from classification contains 100 data points and the label distribution is: {L1: 10, L2: 30, L3: 50, L4: 10}, then the weight distribution is: {L1: 0.1, L2: 0.3, L3: 0.5, L4: 0.1}.
[0092] In practical applications, obtaining a large amount of accurate data labeled with different pain levels is very difficult. This method only requires a relatively small amount of labeled data. Simultaneously, it clusters the first or second fusion vector with preset data, obtaining labels through the clusters obtained from the classification, rather than comparing them with individual labeled data, thus improving accuracy.
[0093] This embodiment introduces an electronic device, including a processor and a memory. The memory stores programs or instructions that can run on the processor. When the program or instructions are executed by the processor, they implement the steps of the method for analyzing multimodal data in motion scenarios as described in any embodiment of the present invention.
[0094] This embodiment introduces a readable storage medium storing a program or instructions, which, when executed by a processor, implements the steps of the method for analyzing multimodal data in motion scenarios as described in any embodiment of the present invention.
[0095] While specific embodiments of the present invention have been described in detail by way of examples, those skilled in the art should understand that the above examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of the invention.
[0096] Those skilled in the art will recognize that the modules and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0097] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and equipment can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0098] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.
[0099] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.
[0100] In addition, the functional modules in the embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.
[0101] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.
[0102] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.
[0103] It should be understood that the sequence numbers of the steps in the invention's content and embodiments do not absolutely imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention. The foregoing description of embodiments of this disclosure has been provided for illustrative and descriptive purposes. The foregoing description is not exhaustive and is not intended to limit this disclosure to the exact form disclosed. Various modifications and variations may exist based on the foregoing teachings, or various modifications and variations may be derived from the practice of this disclosure. These embodiments were chosen and described to illustrate the principles of this disclosure and its practical application, so that those skilled in the art can utilize this disclosure in various implementations and modifications suitable for the specific purpose of the concept.
Claims
1. A method for analyzing multimodal data in motion scenarios, characterized in that, include: Acquire first data within a preset time period, wherein the first data includes at least a temperature value sequence, a heart rate variability characteristic sequence, and an electrical conductance characteristic sequence; Based on the temperature value sequence and the heart rate variability feature sequence, the first feature vector sequence is obtained; Based on the temperature value sequence, heart rate variability feature sequence, and conductance feature sequence, a second feature vector sequence is obtained; The first feature vector sequence is fused to obtain the first fused vector; The second feature vector sequence is fused to obtain the second fused vector; The first weight distribution corresponding to the first fusion vector and the second weight distribution corresponding to the second fusion vector are obtained based on the anomaly detection algorithm. The classification label of the state is determined based on the first weight distribution and the second weight distribution; The first weight distribution or the second weight distribution includes the category label and the weight corresponding to the category label; The process of obtaining the first fusion vector and the second fusion vector includes: Based on the first coverage ratio of the data type and the first feature vector sequence, the first effective data sequence within the first feature vector sequence is obtained; Based on the second coverage ratio and the second feature vector sequence of the data type, the second effective data sequence within the second feature vector sequence is obtained; The first valid data sequence is sampled and pooled to obtain the first fusion vector; The second valid data sequence is sampled and pooled to obtain the second fusion vector; The first coverage ratio or the second coverage ratio is the ratio of the number of non-empty measurements to the total number of measurements; The anomaly detection algorithm is a clustering algorithm. When performing clustering, preset data and a first fusion vector or a second fusion vector are used to perform clustering, and the weights corresponding to the labels are calculated based on the labels corresponding to the vectors contained in the clusters obtained by classification. Cluster the first fusion vector with the preset data to obtain multiple clusters. Calculate the distance from the first fusion vector to the center of each cluster and find the cluster with the closest distance, which is the cluster obtained by the above classification. View the label set of all preset data in the cluster with the closest distance, count the frequency of each label in the cluster, and form a probability distribution through normalization, which is the weight distribution.
2. The method for analyzing multimodal data in motion scenarios as described in claim 1, characterized in that, The first data acquisition sites include at least a first region and a second region, and at least one temperature sensor and one photoplethysmography sensor are provided in the first region; at least one temperature sensor and a set of skin conductance sensors are provided in the second region.
3. The method for analyzing multimodal data in motion scenarios as described in claim 1, characterized in that, The heart rate variability feature sequence includes short-term heart rate variability features, which include pNN50 features.
4. The method for analyzing multimodal data in motion scenarios as described in claim 1, characterized in that, The process of obtaining the first fusion vector or the second fusion vector includes: Based on the preset time window, determine the sampling coverage ratio corresponding to the preset time window. The sampling coverage ratio is the ratio of the number of valid data points within the given time window to the theoretical total number of data points. The first time window with the maximum coverage ratio is obtained based on the sampling coverage ratio; The first feature vector sequence is sampled and pooled based on the first time window to obtain the first fused vector; The second feature vector sequence is sampled and pooled based on the first time window to obtain the second fused vector.
5. The method for analyzing multimodal data in motion scenarios as described in claim 1, characterized in that, The process of determining the classification label of the state includes: The weights corresponding to the same state level in the first weight distribution and the second weight distribution are added together to obtain the comprehensive weight distribution; Based on the weights in the comprehensive weight distribution, the state level with the highest weight is determined as the classification label for the current state.
6. The method for analyzing multimodal data in motion scenarios as described in claim 1, characterized in that, The process of determining the classification label of the state includes: The corresponding first vector and second vector are obtained based on the first weight distribution and the second weight distribution; Obtain the first Hamad product of the first vector and the second vector; The classification label of the current state is determined based on the first Hamada product; The first vector or the second vector is used to represent the distribution of classification labels in the first weight distribution or the second weight distribution.
7. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the method for analyzing multimodal data in a motion scene as described in any one of claims 1 to 5.
8. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions, which, when executed by a processor, implement the steps of the method for analyzing multimodal data in a motion scene as described in any one of claims 1 to 5.