Task research and judgment method and terminal based on multi-modal data of near-electricity smart bracelet

Abstract

The application discloses a kind of electric energy and the joint clearing method and terminal of climbing reserve, including the multi-modal data obtained is sequentially carried out feature embedding, feature alignment and feature extraction and obtains multidimensional preset matrix;The preset matrix is converted into one-dimensional matrix, and input full connection network, and output feature matrix;According to the feature matrix, task research is judged.The application is characterized by the feature embedding, alignment and extraction of the multi-modal data obtained, and the feature data under the preset matrix is obtained after the feature processing, the preset matrix is converted into one-dimensional matrix to construct task network layer-full connection network, the feature data in preset matrix is calculated and processed, and finally the event corresponding to different kinds of modal data is predicted according to the processing result, that is, the task research is judged, to predict the potential danger of the wearer of near-electric smart bracelet in real time according to multi-modal data, so as to alarm in time, effectively improve the accuracy of near-electric smart bracelet health prediction.

Description

Technical Field

[0001] This invention relates to the field of power construction safety technology, and in particular to a task assessment method and terminal based on multimodal data from a near-electric smart wristband. Background Technology

[0002] Existing technologies include smart bracelets that integrate proximity alarms, health monitoring, and remote management. However, current industry methods for analyzing proximity alarm data mostly rely on formulas or threshold values, which often have the following drawbacks:

[0003] Disadvantage 1: The numerical formula analysis method is relatively subjective and cannot be dynamically adjusted in response to changes in environmental factors, resulting in inaccurate analysis.

[0004] Disadvantage 2: The interval threshold judgment has a rigid numerical division, which can only judge specific values ​​and cannot perform joint judgment of multiple values, which will also result in a single and inaccurate judgment result. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a task judgment method and terminal based on multimodal data of a near-electric smart bracelet. By constructing a fully connected task network layer, the method outputs task judgment results in real time based on the acquired multimodal data, thereby improving the accuracy of health prediction by the near-electric smart bracelet.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A task assessment method based on multimodal data from a near-electric smart bracelet includes the following steps:

[0008] S1. The acquired multimodal data is sequentially subjected to feature embedding, feature alignment and feature extraction to obtain a multidimensional preset matrix;

[0009] S2. Convert the preset matrix into a one-dimensional matrix, input it into a fully connected network, and output the feature matrix;

[0010] S3. Based on the feature matrix, perform task analysis.

[0011] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows:

[0012] A task analysis terminal based on multimodal data from a near-electric smart bracelet includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor, when executing the computer program, performs the following steps:

[0013] S1. The acquired multimodal data is sequentially subjected to feature embedding, feature alignment and feature extraction to obtain a multidimensional preset matrix;

[0014] S2. Convert the preset matrix into a one-dimensional matrix, input it into a fully connected network, and output the feature matrix;

[0015] S3. Based on the feature matrix, perform task analysis.

[0016] The beneficial effects of this invention are as follows: by performing feature embedding, alignment and extraction on the acquired multimodal data, feature data under a preset matrix is ​​obtained. The preset matrix is ​​then converted into a one-dimensional matrix to construct a task network layer—a fully connected network. The feature data in the preset matrix is ​​calculated and processed. Finally, the events corresponding to different types of modal data are predicted based on the processing results, that is, task judgment is realized. In order to predict the potential dangers of near-field smart bracelet wearers in real time based on multimodal data, timely alarms can be issued, and the accuracy of near-field smart bracelet health prediction can be effectively improved. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a task assessment method based on multimodal data from a near-electric smart bracelet, according to an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of the structure of a task analysis terminal based on multimodal data from a near-electric smart bracelet, according to an embodiment of the present invention.

[0019] Label Explanation:

[0020] 1. A task analysis terminal based on multimodal data from a near-electric smart bracelet; 2. Memory; 3. Processor. Detailed Implementation

[0021] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0022] Please refer to Figure 1 and Figure 2 A task assessment method based on multimodal data from a near-electric smart bracelet, comprising the following steps:

[0023] S1. The acquired multimodal data is sequentially subjected to feature embedding, feature alignment and feature extraction to obtain a multidimensional preset matrix;

[0024] S2. Convert the preset matrix into a one-dimensional matrix, input it into a fully connected network, and output the feature matrix;

[0025] S3. Based on the feature matrix, perform task analysis.

[0026] As can be seen from the above description, the beneficial effects of the present invention are as follows: by performing feature embedding, alignment and extraction on the acquired multimodal data, feature data under a preset matrix is ​​obtained, and the preset matrix is ​​converted into a one-dimensional matrix to construct a task network layer - a fully connected network. The feature data in the preset matrix is ​​calculated and processed, and finally the events corresponding to different types of modal data are predicted according to the processing results, that is, task judgment is realized, so as to predict the potential dangers of near-field smart bracelet wearers in real time based on multimodal data, so as to issue timely warnings and effectively improve the accuracy of near-field smart bracelet health prediction.

[0027] Furthermore, the multimodal data consists of the wearer's heart rate (A), blood oxygen saturation (B), location (P), six-axis posture (S), air pressure (M), near-electric body temperature (T), and blood pressure (Q) collected in real time by the near-electric smart bracelet.

[0028] As described above, the multimodal data collected in real time by the JinDian smart bracelet includes various physiological parameters such as the wearer's heart rate, blood oxygen saturation, location, six-axis posture, air pressure, body temperature, and blood pressure. This enables comprehensive monitoring and management of the entire work process and personal safety of workers in power construction, ensuring their safety.

[0029] Furthermore, the feature embedding specifically refers to:

[0030] S11. Let the feature embeddings of each of the multimodal data be A', B', P', S', M', T', and Q', and the embedding representation be: Y = Embed(X), where X is the original input of the multimodal data, and Y is the representation of the multimodal data after embedding.

[0031]

[0032] Among them, W m,n The weights of the multimodal data are initialized using a random normal distribution, ranging from 0 to 1, X. n The input is the original multimodal data, where n represents the dimension of the multimodal data, N is the dimension of the modality with the largest dimension among all the multimodal data, and 0 is the output in n dimensions after embedding the multimodal data. The dimension of the feature data of different modalities is different. Then the output is:

[0033] Y = [00, 01, 02, ..., 0 m (2);

[0034] In the formula, m∈(0,5), that is, the output matrix is ​​an embedding matrix Y' of size 1*6;

[0035] The feature alignment specifically refers to:

[0036] S12. Map the embedding matrix Y to the same data space:

[0037] Y' = [Y0, Y1, Y2, ..., Y] c (3);

[0038] Where c represents the seven types of the multimodal data A', B', P', S', M', T' and Q', c∈(0,6), that is, the embedding matrix Y' is mapped to a 7*6 mapping matrix;

[0039] Feature cross-validation is performed on the modal data within the mapping matrix, denoted as:

[0040] F = Y' i *Y' i+1 (4);

[0041] In the formula, 0 <= i < 6, resulting in a 6*6 cross matrix F;

[0042] Stacking the individual cross features in the cross matrix F yields the final matrix F':

[0043]

[0044] In the formula, j∈(0,4), that is, the structure of the final matrix F' is 5*6*6;

[0045] The feature extraction specifically involves:

[0046] S13. Let the feature extraction layers, from bottom to top, be the first regular convolutional layer, the first regular activation function layer, the second regular convolutional layer, the second regular activation function layer, the third regular convolutional layer, and the third regular activation function layer, and define them as follows:

[0047] Assume that the first regular convolutional layer and the second regular convolutional layer both have 5-channel input and 5-channel output, the input feature map size is 6*6, and the convolutional kernel is 1*1;

[0048] Assume the third conventional convolutional layer has 5 input channels and 1 output channel, the input feature map size is 6*6, and the convolutional kernel is 1*1;

[0049] Assume that the first, second, and third conventional activation function layers all use the relu(X) function:

[0050]

[0051] The feature extraction layer extracts a preset 6*6 matrix.

[0052] As described above, feature embedding is based on the modal data collected by the near-electric smart bracelet. Since different modal data have different dimensions, for example, blood oxygen A is a single value with a dimension of 1, but position P includes x-axis, y-axis, and z-axis coordinates, with a dimension of 3, and six-axis pose S has six axis parameters, with a dimension of 6. Therefore, it is necessary to unify the multimodal parameters of each dimension to one dimension, that is, to unify them to the dimension 6 of the six-axis pose S, which has the largest dimension among all modal parameters. Thus, feature embedding is used to achieve dimension unification, feature alignment is used to achieve spatial constraints on features, and finally, feature extraction is used to train the modal data through multi-layer feature extraction layers to ensure that each modal data has good fitting characteristics, thereby improving the adaptability of subsequent fully connected networks for task judgment.

[0053] Further, step S2 specifically includes:

[0054] S21. Copy a set of 6 feature values ​​from the 6*6 preset matrix and add them to list K. Repeat the copying of the next set of 6 feature values ​​to list K. After repeating the operation 6 times, a one-dimensional matrix L containing 36 elements is formed.

[0055] S22. Input the one-dimensional matrix L into a 36*7 fully connected matrix network, where 36 is the number of neuron inputs and 7 is the number of neuron outputs, and define the output as a 1*7 feature matrix L'.

[0056] As described above, the 6*6 preset matrix obtained from training is transformed into a one-dimensional matrix containing 36 elements, which facilitates the construction of fully connected network layers.

[0057] Further, step S3 specifically includes:

[0058] S31. Take the maximum output value L' in the feature matrix L'. max =softmaxW(L') is used as the task output, with an output range of 1 to 7 task types, each corresponding to one of the seven types of multimodal data. The function softmaxW() is used to calculate the probability of each task type, and the probability vector of each task is denoted as M:

[0059]

[0060] Among them, M c The probability of the task analysis output for the current category is represented by the subscript c∈(0,6), and its value corresponds to the seven types of multimodal data features A', B', P', S', M', T' and Q', respectively. exp(L' c The normally distributed data is regressed to the natural distribution, and the lower coordinate number of the maximum probability is taken from 0 to 6, which is denoted as max_index;

[0061] S32. When max_index = 0, it indicates that the task type is heart rate A, and the output is M. c <0.5 indicates normal operation, output M c >0.9 indicates an anomaly;

[0062] When max_index = 1, it indicates that the task type is blood oxygen saturation B, and the output is M. c <0.5 indicates normal operation, output M c >0.9 indicates an anomaly;

[0063] When max_index = 2, it indicates that the task type is position P, and the output is M. c <0.5 indicates normal operation, output M c >0.9 indicates an anomaly;

[0064] When max_index = 3, it indicates that the task type is six-axis attitude S, and the output is M. c <0.5 indicates normal operation, output M c >0.9 indicates an anomaly;

[0065] When max_index = 4, it indicates that the task type is air pressure M, and the output is M. c <0.5 indicates normal operation, output M c >0.9 indicates an anomaly;

[0066] When max_index = 5, it indicates that the task type is near-electrical body temperature (T), and the output is M. c <0.5 indicates normal operation, output M c >0.9 indicates an anomaly;

[0067] When max_index = 6, it indicates that the task type is blood pressure Q, and the output is M. c <0.5 indicates normal operation, output M c A value greater than 0.9 indicates an anomaly.

[0068] As described above, since there are seven types of multimodal data acquired, the types of task judgments are based on these seven types of data. Therefore, the output of the fully connected network is defined as 1 to 7 types, that is, based on the probability values ​​of the wearer's heart rate, blood oxygen saturation, location, six-axis posture, air pressure, body temperature and blood pressure at the location, the potential dangers of the wearer are predicted, making the prediction results more accurate.

[0069] Please refer to Figure 2 A task analysis terminal based on multimodal data from a near-electric smart bracelet includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:

[0070] S1. The acquired multimodal data is sequentially subjected to feature embedding, feature alignment and feature extraction to obtain a multidimensional preset matrix;

[0071] S2. Convert the preset matrix into a one-dimensional matrix, input it into a fully connected network, and output the feature matrix;

[0072] S3. Based on the feature matrix, perform task analysis.

[0073] As described above, the beneficial effects of this invention are as follows: Based on the same technical concept, and in conjunction with the aforementioned multimodal data feature method based on a near-field smart bracelet, a task assessment terminal based on multimodal data from a near-field smart bracelet is provided. By performing feature embedding, alignment, and extraction on the acquired multimodal data, feature data under a preset matrix is ​​obtained. The preset matrix is ​​then converted into a one-dimensional matrix to construct a task network layer—a fully connected network. The feature data in the preset matrix is ​​then processed, and finally, based on the processing results, the events corresponding to different types of modal data are predicted, thus achieving task assessment. This allows for real-time prediction of potential dangers to near-field smart bracelet wearers based on multimodal data, enabling timely alerts and effectively improving the accuracy of health predictions by the near-field smart bracelet.

[0074] Furthermore, the multimodal data consists of the wearer's heart rate (A), blood oxygen saturation (B), location (P), six-axis posture (S), air pressure (M), near-electric body temperature (T), and blood pressure (Q) collected in real time by the near-electric smart bracelet.

[0075] As described above, the multimodal data collected in real time by the JinDian smart bracelet includes various physiological parameters such as the wearer's heart rate, blood oxygen saturation, location, six-axis posture, air pressure, body temperature, and blood pressure. This enables comprehensive monitoring and management of the entire work process and personal safety of workers in power construction, ensuring their safety.

[0076] Furthermore, the feature embedding specifically refers to:

[0077] S11. Let the feature embeddings of each of the multimodal data be A', B', P', S', M', T', and Q', and the embedding representation be: Y = Embed(X), where X is the original input of the multimodal data, and Y is the representation of the multimodal data after embedding.

[0078]

[0079] Among them, W m,n The weights of the multimodal data are initialized using a random normal distribution, ranging from 0 to 1, X. nThe input is the original multimodal data, where n represents the dimension of the multimodal data, N is the dimension of the modality with the largest dimension among all the multimodal data, and 0 is the output in n dimensions after embedding the multimodal data. The dimension of the feature data of different modalities is different. Then the output is:

[0080] Y = [00, 01, 02, ..., 0 m (2);

[0081] In the formula, m∈(0,5), that is, the output matrix is ​​an embedding matrix Y' of size 1*6;

[0082] The feature alignment specifically refers to:

[0083] S12. Map the embedding matrix Y to the same data space:

[0084] Y' = [Y0, Y1, Y2, ..., Y] c (3);

[0085] Where c represents the seven types of the multimodal data A', B', P', S', M', T' and Q', c∈(0,6), that is, the embedding matrix Y' is mapped to a 7*6 mapping matrix;

[0086] Feature cross-validation is performed on the modal data within the mapping matrix, denoted as:

[0087] F = Y' i *Y' i+1 (4);

[0088] In the formula, 0 <= i < 6, resulting in a 6*6 cross matrix F;

[0089] Stacking the individual cross features in the cross matrix F yields the final matrix F':

[0090]

[0091] In the formula, j∈(0,4), that is, the structure of the final matrix F' is 5*6*6;

[0092] The feature extraction specifically involves:

[0093] S13. Let the feature extraction layers, from bottom to top, be the first regular convolutional layer, the first regular activation function layer, the second regular convolutional layer, the second regular activation function layer, the third regular convolutional layer, and the third regular activation function layer, and define them as follows:

[0094] Assume that the first regular convolutional layer and the second regular convolutional layer both have 5-channel input and 5-channel output, the input feature map size is 6*6, and the convolutional kernel is 1*1;

[0095] Assume the third conventional convolutional layer has 5 input channels and 1 output channel, the input feature map size is 6*6, and the convolutional kernel is 1*1;

[0096] Assume that the first, second, and third conventional activation function layers all use the relu(X) function:

[0097]

[0098] The feature extraction layer extracts a preset 6*6 matrix.

[0099] As described above, feature embedding is based on the modal data collected by the near-electric smart bracelet. Since different modal data have different dimensions, for example, blood oxygen A is a single value with a dimension of 1, but position P includes x-axis, y-axis, and z-axis coordinates, with a dimension of 3, and six-axis pose S has six axis parameters, with a dimension of 6. Therefore, it is necessary to unify the multimodal parameters of each dimension to one dimension, that is, to unify them to the dimension 6 of the six-axis pose S, which has the largest dimension among all modal parameters. Thus, feature embedding is used to achieve dimension unification, feature alignment is used to achieve spatial constraints on features, and finally, feature extraction is used to train the modal data through multi-layer feature extraction layers to ensure that each modal data has good fitting characteristics, thereby improving the adaptability of subsequent fully connected networks for task judgment.

[0100] Further, step S2 specifically includes:

[0101] S21. Copy a set of 6 feature values ​​from the 6*6 preset matrix and add them to list K. Repeat the copying of the next set of 6 feature values ​​to list K. After repeating the operation 6 times, a one-dimensional matrix L containing 36 elements is formed.

[0102] S22. Input the one-dimensional matrix L into a 36*7 fully connected matrix network, where 36 is the number of neuron inputs and 7 is the number of neuron outputs, and define the output as a 1*7 feature matrix L'.

[0103] As described above, the 6*6 preset matrix obtained from training is transformed into a one-dimensional matrix containing 36 elements, which facilitates the construction of fully connected network layers.

[0104] Further, step S3 specifically includes:

[0105] S31. Take the maximum output value L' in the feature matrix L'. max=softmaxW(L') is used as the task output, with an output range of 1 to 7 task types, each corresponding to one of the seven types of multimodal data. The function softmaxW() is used to calculate the probability of each task type, and the probability vector of each task is denoted as M:

[0106]

[0107] Among them, M c The probability of the task analysis output for the current category is represented by the subscript c∈(0,6), and its value corresponds to the seven types of multimodal data features A', B', P', S', M', T' and Q', respectively. exp(L' c The normally distributed data is regressed to the natural distribution, and the lower coordinate number of the maximum probability is taken from 0 to 6, which is denoted as max_index;

[0108] S32. When max_index = 0, it indicates that the task type is heart rate A, and the output is M. c <0.5 indicates normal operation, output M c >0.9 indicates an anomaly;

[0109] When max_index = 1, it indicates that the task type is blood oxygen saturation B, and the output is M. c <0.5 indicates normal operation, output M c >0.9 indicates an anomaly;

[0110] When max_index = 2, it indicates that the task type is position P, and the output is M. c <0.5 indicates normal operation, output M c >0.9 indicates an anomaly;

[0111] When max_index = 3, it indicates that the task type is six-axis attitude S, and the output is M. c <0.5 indicates normal operation, output M c >0.9 indicates an anomaly;

[0112] When max_index = 4, it indicates that the task type is air pressure M, and the output is M. c <0.5 indicates normal operation, output M c >0.9 indicates an anomaly;

[0113] When max_index = 5, it indicates that the task type is near-electrical body temperature (T), and the output is M. c <0.5 indicates normal operation, output M c >0.9 indicates an anomaly;

[0114] When max_index = 6, it indicates that the task type is blood pressure Q, and the output is M. c<0.5 indicates normal operation, output M c A value greater than 0.9 indicates an anomaly.

[0115] As described above, since there are seven types of multimodal data acquired, the types of task judgments are based on these seven types of data. Therefore, the output of the fully connected network is defined as 1 to 7 types, that is, based on the probability values ​​of the wearer's heart rate, blood oxygen saturation, location, six-axis posture, air pressure, body temperature and blood pressure at the location, the potential dangers of the wearer are predicted, making the prediction results more accurate.

[0116] The inventors provide a method and computing terminal for multimodal data characterization of a proximity smart bracelet, applicable to proximity early warning, health monitoring, and remote management in power construction based on multimodal data collected from the proximity smart bracelet worn by workers. The following detailed description is provided with reference to embodiments.

[0117] Please refer to Figure 1 Embodiment 1 of the present invention is as follows:

[0118] A multimodal data feature generation method for near-electric smart bracelets, such as Figure 1 As shown, the steps include:

[0119] S1. The acquired multimodal data is sequentially subjected to feature embedding, feature alignment and feature extraction to obtain a multidimensional preset matrix;

[0120] S2. Convert the preset matrix into a one-dimensional matrix, input it into the fully connected network, and output the feature matrix;

[0121] S3. Based on the feature matrix, perform task analysis.

[0122] In this embodiment, feature data is obtained by embedding, aligning, and extracting features from the acquired multimodal data. The preset matrix is ​​then converted into a one-dimensional matrix to construct a task network layer—a fully connected network. The feature data in the preset matrix is ​​processed, and the events corresponding to different types of modal data are predicted based on the processing results. This achieves task judgment and enables real-time prediction of potential dangers to wearers of near-field smart bracelets based on multimodal data, so as to provide timely warnings and effectively improve the accuracy of health prediction by near-field smart bracelets.

[0123] Embodiment 2 of the present invention is as follows:

[0124] A multimodal data feature method for a near-field smart bracelet, based on the above embodiment one, wherein the multimodal data in this embodiment are the wearer's heart rate A, blood oxygen saturation B, location P, six-axis posture S, air pressure M, near-field body temperature T, and blood pressure Q collected in real time by the near-field smart bracelet.

[0125] The smart bracelet collects multimodal data in real time, including the wearer's heart rate, blood oxygen saturation, location, six-axis posture, air pressure, body temperature, and blood pressure, enabling comprehensive monitoring and management of workers' safety throughout the entire work process and ensuring their safety.

[0126] In this embodiment, feature embedding is based on modal data collected by the near-field smart bracelet, which is actually obtained from sensors of different modalities on the near-field smart bracelet. Each modal data has unique features, and each feature has its own dimension. For example, heart rate (A), blood oxygen saturation (B), air pressure (M), and near-field body temperature (T) are all single values, i.e., one dimension; blood pressure (Q) includes two values, systolic and diastolic, i.e., two dimensions; position (P) includes coordinate values ​​of three axes, x, y, and z, i.e., three dimensions; and six-axis posture (S) includes posture values ​​of six axes, i.e., six dimensions. Therefore, in order to make each feature comparable across different modalities, it is necessary to first embed the modal data of different dimensions into the same dimension using the Embed() function. That is, the feature embedding in step S1 is specifically as follows:

[0127] S11. Let the feature embeddings of each multimodal data be A', B', P', S', M', T', and Q', and the embedding representation be: Y = Embed(X), where X is the original input of the multimodal data, and Y is the representation of the multimodal data after embedding.

[0128]

[0129] Among them, W m,n The weights for the multimodal data are initialized using a random normal distribution, ranging from 0 to 1, X. n The input is the original multimodal data, where n represents the dimension of the multimodal data, N is the dimension of the largest dimensional modality in the multimodal data, and 0 represents the output in n dimensions after embedding the multimodal data. The dimensions of different modalities are different. Then, the output is:

[0130] Y = [00, 01, 02, ..., 0 m (2);

[0131] In the formula, m∈(0,5), that is, the output matrix is ​​an embedding matrix Y' with a size of 1*6.

[0132] In this embodiment, besides embedding different modal data into embedding matrices of the same dimension, in order to unify and match different modal data to improve recognition accuracy, it is also necessary to align the modal data to map them to the same data space so that they can be uniformly recognized in further processing. That is, the feature alignment in step S1 specifically involves:

[0133] Map the embedding matrix Y' to the same data space:

[0134] Y' = [Y0, Y1, Y2, ..., Y] c (3);

[0135] Where c represents the seven types of multimodal data A', B', P', S', M', T' and Q', c∈(0,6), that is, the embedding matrix Y' is mapped to a 7*6 mapping matrix;

[0136] Feature cross-validation is performed on the modal data within the mapping matrix, denoted as:

[0137] F = Y' i *Y' i+1 (4);

[0138] In the formula, 0 <= i < 6, resulting in a 6*6 cross matrix F;

[0139] Stacking the individual cross features in the cross matrix F yields the final matrix F':

[0140]

[0141] In the formula, j∈(0,4), that is, the structure of the final matrix F' is 5*6*6.

[0142] In the feature alignment step, feature crossing and stacking of single cross features are added to feature data mapped to the same data space. This can achieve spatial constraints on features, that is, similar features of different modalities will be closer in space, while unrelated features will be kept as far apart as possible in terms of spatial distance and distribution.

[0143] Finally, by constructing a multi-layer feature extraction layer and training the modality data, we ensure that each modality data has good fitting characteristics, thereby improving the adaptability of subsequent task evaluation in a fully connected network. Therefore, the feature extraction in step S1 is as follows:

[0144] S13. Let the feature extraction layers, from bottom to top, be the first regular convolutional layer, the first regular activation function layer, the second regular convolutional layer, the second regular activation function layer, the third regular convolutional layer, and the third regular activation function layer, and define them as follows:

[0145] Assume that the first and second regular convolutional layers both have 5-channel input and 5-channel output, the input feature map size is 6*6, and the convolutional kernel is 1*1;

[0146] Assume the third regular convolutional layer has 5 channels for input and 1 channel for output, the input feature map size is 6*6, and the convolutional kernel size is 1*1;

[0147] Suppose that the first, second, and third conventional activation function layers all use the relu(X) function:

[0148]

[0149] The feature extraction layer extracts a preset 6x6 matrix.

[0150] After obtaining the 6*6 preset matrix, in order to construct the fully connected network layer for task analysis, the preset matrix needs to be converted into a one-dimensional matrix. In this embodiment, step S2 specifically involves:

[0151] S21. Copy a set of 6 eigenvalues ​​from the 6*6 preset matrix and add them to list K. Repeat the copying of the next set of 6 eigenvalues ​​to list K. After repeating the operation 6 times, a one-dimensional matrix L containing 36 elements is formed.

[0152] S22. Input a one-dimensional matrix L into a 36*7 fully connected matrix network, where 36 is the number of neuron inputs and 7 is the number of neuron outputs, and define the output as a 1*7 feature matrix L'.

[0153] Finally, since seven types of multimodal data are acquired, the task assessment is based on these seven types of data. Therefore, the output of the fully connected network is defined as 1 to 7 types, that is, based on the probability values ​​of the wearer's heart rate, blood oxygen saturation, location, six-axis posture, air pressure, body temperature, and blood pressure, the potential dangers of the wearer are predicted, making the prediction results more accurate. Specifically, step S3 is as follows:

[0154] S31. Take the maximum output value L' in the feature matrix L'. max =softmaxW(L') is used as the task output, with an output range of 1 to 7 task types, each corresponding to one of the seven types of multimodal data. The function softmaxW() is used to calculate the probability of each task type, and the probability vector of each task is denoted as M:

[0155]

[0156] Among them, M c The probability of the task analysis output for the current category is represented by the subscript c∈(0,6), and its value corresponds to the seven types of multimodal data features A', B', P', S', M', T' and Q', respectively. exp(L' c The normally distributed data is regressed to the natural distribution, and the lower coordinate number of the maximum probability is taken from 0 to 6, which is denoted as max_index;

[0157] S32. When max_index = 0, it indicates that the task type is heart rate A, and the output is M. c <0.5 indicates normal operation, output M c >0.9 indicates an anomaly;

[0158] When max_index = 1, it indicates that the task type is blood oxygen saturation B, and the output is M. c <0.5 indicates normal operation, output M c >0.9 indicates an anomaly;

[0159] When max_index = 2, it indicates that the task type is position P, and the output is M. c <0.5 indicates normal operation, output M c >0.9 indicates an anomaly;

[0160] When max_index = 3, it indicates that the task type is six-axis attitude S, and the output is M. c <0.5 indicates normal operation, output M c >0.9 indicates an anomaly;

[0161] When max_index = 4, it indicates that the task type is air pressure M, and the output is M. c <0.5 indicates normal operation, output M c >0.9 indicates an anomaly;

[0162] When max_index = 5, it indicates that the task type is near-electrical body temperature (T), and the output is M. c <0.5 indicates normal operation, output M c >0.9 indicates an anomaly;

[0163] When max_index = 6, it indicates that the task type is blood pressure Q, and the output is M. c <0.5 indicates normal operation, output M c A value greater than 0.9 indicates an anomaly.

[0164] Please refer to Figure 2 Embodiment 3 of the present invention is as follows:

[0165] A task analysis terminal 1 based on multimodal data from a near-electric smart bracelet includes a memory 2, a processor 3, and a computer program stored in the memory 2 and executable on the processor 3. In this embodiment, when the processor 3 executes the computer program, it implements the steps in Embodiment 1 or Embodiment 2 described above.

[0166] In summary, the task assessment method and terminal based on multimodal data from a near-electric smart bracelet provided by this invention have the following beneficial effects:

[0167] 1. By effectively combining the fusion relationships between multiple modalities through feature embedding, feature alignment, and feature extraction, more accurate monitoring data is obtained, which improves the accuracy of subsequent event judgment predictions, reduces misjudgments, and makes the judgments more consistent with the data distribution.

[0168] 2. Construct a task network layer—a fully connected network—to process the feature data in the preset matrix. Finally, based on the processing results, predict the events corresponding to different types of modal data, thus realizing task judgment. This allows for real-time prediction of potential dangers to near-field smart bracelet wearers based on multimodal data, enabling timely alerts and effectively improving the accuracy of near-field smart bracelet health predictions.

[0169] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A task assessment method based on multimodal data from a near-electric smart bracelet, characterized in that, Including the following steps: S1. The acquired multimodal data is sequentially subjected to feature embedding, feature alignment and feature extraction to obtain a multidimensional preset matrix; the multimodal data is the wearer's heart rate A, blood oxygen saturation B, location P, six-axis posture S, air pressure M, near-electric body temperature T and blood pressure Q collected by the near-electric smart bracelet in real time. The feature embedding specifically refers to: S11. Let the feature embeddings of each of the multimodal data be A', B', P', S', M', T', and Q', and the embeddings be expressed as: Y =Embed( X ),in X This is the raw input for the multimodal data. Y Let the embedded representation of the multimodal data be denoted as: （1）； in, W m,n The weights of the multimodal data are initialized using a random normal distribution, ranging from 0 to 1. X n This is the raw input for the multimodal data. n This represents the dimension of the multimodal data. N The dimension of the modality with the largest dimension among all the multimodal data is 0, and 0 represents the dimension of the multimodal data after embedding. n The output dimensions differ depending on the modality of the feature data. The output is then: （2）； In the formula, m ∈(0,5), that is, the output matrix is ​​an embedding matrix of size 1*6. Y' ; The feature alignment specifically refers to: S12, embedding the matrix Y Mapped to the same data space: （3）； in, c The seven categories A', B', P', S', M', T', and Q' represent the multimodal data. c ∈(0,6), that is, the embedding matrix Y’ The mapping is a 7x6 mapping matrix; Feature cross-validation is performed on the modal data within the mapping matrix, denoted as: （4）； In the formula, 0 <= i <6, resulting in a 6x6 cross matrix. F ; The cross matrix F The individual cross features in the matrix are stacked to obtain the final matrix. F' : （5）； In the formula, j ∈(0,4), that is, the final matrix F' The structure is 5*6*6; The feature extraction specifically involves: S13. Let the feature extraction layers, from bottom to top, be the first regular convolutional layer, the first regular activation function layer, the second regular convolutional layer, the second regular activation function layer, the third regular convolutional layer, and the third regular activation function layer, and define them as follows: Assume that the first regular convolutional layer and the second regular convolutional layer both have 5-channel input and 5-channel output, the input feature map size is 6*6, and the convolutional kernel is 1*1; Assume the third conventional convolutional layer has 5 input channels and 1 output channel, the input feature map size is 6*6, and the convolutional kernel is 1*1; Assume that the first, second, and third conventional activation function layers all use ReLU ( X )function: （6）； The feature extraction layer extracts a preset 6*6 matrix. S2. Convert the preset matrix into a one-dimensional matrix, input it into a fully connected network, and output the feature matrix; S21. Copy a set of 6 eigenvalues ​​from the 6*6 preset matrix and add them to the list. K And repeat the process of copying the next set of 6 feature values ​​to the list. K After repeating the operation 6 times, a one-dimensional matrix containing 36 elements is formed. L ; S22, the one-dimensional matrix L The input is fed into a 36*7 fully connected matrix network, where 36 is the number of neuron inputs and 7 is the number of neuron outputs. The output is defined as a 1*7 feature matrix. L' ; S3. Based on the feature matrix, perform task analysis.

2. The task assessment method based on multimodal data from a near-electric smart bracelet according to claim 1, characterized in that, Step S3 specifically involves: S31. Obtain the feature matrix. L' Maximum output value L' max =softmaxW( L' The output is a task with 1 to 7 task types, each corresponding to one of the seven types of multimodal data. The function softmaxW() is used to calculate the probability of each task type, and the probability vector of each task is denoted as M. （6）； Among them, M c The index represents the probability of the task analysis output for the current type. c ∈(0,6), whose values ​​correspond to the seven types of multimodal data features A', B', P', S', M', T' and Q', respectively, exp( L' c The normally distributed data is regressed to the natural distribution, and the lower coordinate number of the maximum probability [0] is taken from 0 to 6 and denoted as max_index; S32. When max_index=0, it indicates that the task type is heart rate A, and the output is M. c < 0.5 indicates normal operation, output M c > 0.9 indicates an anomaly; When max_index=1, it indicates that the task type is blood oxygen saturation B, and the output is M. c < 0.5 indicates normal operation, output M c > 0.9 indicates an anomaly; When max_index=2, it indicates that the task type is position P, and the output is M. c < 0.5 indicates normal operation, output M c > 0.9 indicates an anomaly; When max_index=3, it indicates that the task type is six-axis attitude S, and the output is M. c < 0.5 indicates normal operation, output M c > 0.9 indicates an anomaly; When max_index=4, it indicates that the task type is air pressure M, and the output is M. c < 0.5 indicates normal operation, output M c > 0.9 indicates an anomaly; When max_index=5, it indicates that the task type is near-electrical body temperature (T), and the output is M. c < 0.5 indicates normal operation, output M c > 0.9 indicates an anomaly; When max_index=6, it indicates that the task type is blood pressure Q, and the output is M. c < 0.5 indicates normal operation, output M c > 0.9 indicates an anomaly.

3. A task analysis terminal based on multimodal data from a near-electric smart bracelet, characterized in that, Includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, performs the following steps: S1. The acquired multimodal data is sequentially subjected to feature embedding, feature alignment and feature extraction to obtain a multidimensional preset matrix; the multimodal data is the wearer's heart rate A, blood oxygen saturation B, location P, six-axis posture S, air pressure M, near-electric body temperature T and blood pressure Q collected by the near-electric smart bracelet in real time. The feature embedding specifically refers to: S11. Let the feature embeddings of each of the multimodal data be A', B', P', S', M', T', and Q', and the embeddings be expressed as: Y =Embed( X ),in X This is the raw input for the multimodal data. Y Let the embedded representation of the multimodal data be denoted as: （1）； in, W m,n The weights of the multimodal data are initialized using a random normal distribution, ranging from 0 to 1. X n This is the raw input for the multimodal data. n This represents the dimension of the multimodal data. N The dimension of the modality with the largest dimension among all the multimodal data is 0, and 0 represents the dimension of the multimodal data after embedding. n The output dimensions differ depending on the modality of the feature data. The output is then: （2）； In the formula, m ∈(0,5), that is, the output matrix is ​​an embedding matrix of size 1*6. Y' ; The feature alignment specifically refers to: S12, embedding the matrix Y Mapped to the same data space: （3）； in, c The seven categories A', B', P', S', M', T', and Q' represent the multimodal data. c ∈(0,6), that is, the embedding matrix Y’ The mapping is a 7x6 mapping matrix; Feature cross-validation is performed on the modal data within the mapping matrix, denoted as: （4）； In the formula, 0 <= i <6, resulting in a 6x6 cross matrix. F ; The cross matrix F The individual cross features in the matrix are stacked to obtain the final matrix. F' : （5）； In the formula, j ∈(0,4), that is, the final matrix F' The structure is 5*6*6; The feature extraction specifically involves: S13. Let the feature extraction layers, from bottom to top, be the first regular convolutional layer, the first regular activation function layer, the second regular convolutional layer, the second regular activation function layer, the third regular convolutional layer, and the third regular activation function layer, and define them as follows: Assume that the first regular convolutional layer and the second regular convolutional layer both have 5-channel input and 5-channel output, the input feature map size is 6*6, and the convolutional kernel is 1*1; Assume the third conventional convolutional layer has 5 input channels and 1 output channel, the input feature map size is 6*6, and the convolutional kernel is 1*1; Assume that the first, second, and third conventional activation function layers all use ReLU ( X )function: （6）； The feature extraction layer extracts a preset 6*6 matrix. S2. Convert the preset matrix into a one-dimensional matrix, input it into a fully connected network, and output the feature matrix; S21. Copy a set of 6 eigenvalues ​​from the 6*6 preset matrix and add them to the list. K And repeat the process of copying the next set of 6 feature values ​​to the list. K After repeating the operation 6 times, a one-dimensional matrix containing 36 elements is formed. L ; S22, the one-dimensional matrix L The input is fed into a 36*7 fully connected matrix network, where 36 is the number of neuron inputs and 7 is the number of neuron outputs. The output is defined as a 1*7 feature matrix. L' ; S3. Based on the feature matrix, perform task analysis.

4. A task analysis terminal based on multimodal data from a near-electric smart bracelet according to claim 3, characterized in that, Step S3 specifically involves: S31. Obtain the feature matrix. L' Maximum output value L' max =softmaxW( L' The output is a task with 1 to 7 task types, each corresponding to one of the seven types of multimodal data. The function softmaxW() is used to calculate the probability of each task type, and the probability vector of each task is denoted as M. （6）； Among them, M c The index represents the probability of the task analysis output for the current type. c ∈(0,6), whose values ​​correspond to the seven types of multimodal data features A', B', P', S', M', T' and Q', respectively, exp( L' c The normally distributed data is regressed to the natural distribution, and the lower coordinate number of the maximum probability [0] is taken from 0 to 6 and denoted as max_index; S32. When max_index=0, it indicates that the task type is heart rate A, and the output is M. c < 0.5 indicates normal operation, output M c > 0.9 indicates an anomaly; When max_index=1, it indicates that the task type is blood oxygen saturation B, and the output is M. c < 0.5 indicates normal operation, output M c > 0.9 indicates an anomaly; When max_index=2, it indicates that the task type is position P, and the output is M. c < 0.5 indicates normal operation, output M c > 0.9 indicates an anomaly; When max_index=3, it indicates that the task type is six-axis attitude S, and the output is M. c < 0.5 indicates normal operation, output M c > 0.9 indicates an anomaly; When max_index=4, it indicates that the task type is air pressure M, and the output is M. c < 0.5 indicates normal operation, output M c > 0.9 indicates an anomaly; When max_index=5, it indicates that the task type is near-electrical body temperature (T), and the output is M. c < 0.5 indicates normal operation, output M c > 0.9 indicates an anomaly; When max_index=6, it indicates that the task type is blood pressure Q, and the output is M. c < 0.5 indicates normal operation, output M c > 0.9 indicates an anomaly.

Images