Physiological motion data analysis method and device and computer readable storage medium

Abstract

The invention provides a physiological movement data analysis method and device and a computer readable storage medium. The method comprises the steps that physiological movement data of a user are obtained, and the physiological movement data comprise electrocardiosignals, respiration signals and movement signals; performing feature extraction processing on the physiological movement data by adopting a neural network model, generating physiological movement features corresponding to the physiological movement data, and determining intra-modal attention features and inter-modal attention features corresponding to the physiological movement features according to the physiological movement features; and performing fusion processing on the intra-modal attention features and the inter-modal attention features to generate advanced fusion features, and determining a health report of the user at least according to the advanced fusion features. According to the method, the key features of all modes are extracted and fused through attention calculation in the modes and among the modes, the robustness and accuracy of multi-mode data representation are improved, and the problem that a health analysis result calculated through a traditional health state analysis method is low in accuracy is solved.

Description

Technical Field

[0001] The present application relates to the technical field of physiological motion data analysis, and in particular, to a physiological motion data analysis method, a physiological motion data analysis device, a computer-readable storage medium, and an electronic device. Background Art

[0002] In the field of digital health monitoring, wearable devices such as smart watches and health trackers have been widely used in daily health management. These devices can monitor physiological parameters such as heart rate, blood pressure, and blood oxygen saturation in real time. For example, smart watches use their built-in sensors to monitor electrocardiograms (ECGs) and heart rate variability (HRV). Heart rate belts can dynamically collect ECG signals during human exercise in real time, and by analyzing ECG signals, heart rate variability-related parameters can be calculated. Through long-term data collection, a personal parameter benchmark can be established, and then the physiological and psychological state of the human body, such as tension and relaxation, can be determined based on real-time data. For most users, similar methods have extremely high requirements for professionalism on the one hand, and lack of adaptability in judging abnormal phenomena on the other hand, resulting in unsatisfactory judgments. Most health status analysis methods rely only on a single ECG signal, ignoring the influence of other physiological signals, resulting in insufficient comprehensiveness and accuracy of the analysis results. Summary of the invention

[0003] The main purpose of the present application is to provide a method for analyzing physiological motion data, an analysis device for physiological motion data, a computer-readable storage medium and an electronic device, so as to at least solve the problem of low accuracy of health analysis results calculated by traditional health status analysis methods.

[0004] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for analyzing physiological motion data is provided, comprising: obtaining physiological motion data of a user, wherein the physiological motion data includes electrocardiogram signals, respiratory signals and motion signals; performing feature extraction processing on the physiological motion data using a neural network model to generate physiological motion features corresponding to the physiological motion data, and determining intra-modal attention features and inter-modal attention features corresponding to the physiological motion features based on the physiological motion features; fusing the intra-modal attention features and the inter-modal attention features to generate high-level fusion features, and determining a health report of the user at least based on the high-level fusion features.

[0005] Optionally, determining the health report of the user at least based on the advanced fusion features includes: obtaining a user health database, extracting health-related features from the user health database to generate a user health portrait, wherein the user health portrait includes user information, lifestyle, previous symptoms and diagnosis; inputting the user health portrait and the advanced fusion features into a personalized analysis model to generate multimodal fusion features, and determining latent variables based on the multimodal fusion features, wherein the personalized analysis model is a parameterized conditional variational autoencoder; constructing a medical health knowledge base, and using a large language model to encode the medical health knowledge base into an embedding vector; performing similarity calculation on the latent variable and the embedding vector to generate a sequence of embedding vectors arranged in descending order of similarity, selecting the first preset number of the embedding vectors in the embedding vector sequence and inputting them into the large language model to generate the health report of the user.

[0006] Optionally, determining, according to the physiological motion feature, an intra-modal attention feature corresponding to the physiological motion feature comprises: according to a first formula: Determine the intra-modality attention feature, where Characterize all the physiological motion features of the i-th physiological motion data, W i represents the weight of the ith type of physiological motion data, b i represents the bias of the physiological motion data of the i-th type, a i represents the eigenvalue within the i-th mode, N i represents the length of the feature vector of the i-th physiological motion data, Characterize the intra-modal attention calculated for the i-th physiological motion data.

[0007] Optionally, determining an inter-modal attention feature corresponding to the physiological motion feature according to the physiological motion feature includes: according to a second formula: Sure

[0008]

[0009] The inter-modal attention feature, where W β represents the weight matrix, b β is the bias vector, β represents the inter-modal eigenvalue, represents the intra-modality attention feature, and z represents the inter-modality attention feature.

[0010] Optionally, before using a neural network model to perform feature extraction processing on the physiological motion data, the method also includes: determining a signal extraction time window for various physiological motion data according to the type of the physiological motion data; extracting and processing various physiological motion data according to the signal extraction time window to obtain target physiological motion data, and determining the similarity between various target physiological motion data; determining a phase difference between various target physiological motion data according to the similarity, and controlling the phase synchronization of various target physiological motion data according to the phase difference.

[0011] Optionally, determining the signal extraction time window of various physiological motion data according to the type of the physiological motion data includes: according to a third formula: Determine the signal extraction time window of various physiological motion data, where T i is the signal extraction time window of the i-th physiological motion data, ω i represents the main frequency of the signal of the i-th physiological motion data, σ i is the signal standard deviation of the ith physiological motion data, A i represents the signal amplitude of the ith physiological motion data, A0 is the custom threshold, and f is the adaptive function.

[0012] Optionally, after acquiring the user's physiological motion data, the method further includes: filtering and denoising the physiological motion data, wherein the filtering and denoising includes retaining the portion of the electrocardiogram signal related to the PQRST waveform and removing the noise of the power frequency interference in the motion data; and standardizing the physiological motion data after the filtering and denoising, wherein the standardization is used to scale the signal amplitude of the physiological motion data to within a preset range.

[0013] According to another aspect of the present application, a device for analyzing physiological motion data is provided, comprising: an acquisition unit, used to acquire the physiological motion data of a user, wherein the physiological motion data includes an electrocardiogram signal, a respiratory signal and a motion signal; a processing unit, used to perform feature extraction processing on the physiological motion data using a neural network model, generate physiological motion features corresponding to the physiological motion data, and determine, based on the physiological motion features, intra-modal attention features and inter-modal attention features corresponding to the physiological motion features; a determination unit, used to perform fusion processing on the intra-modal attention features and the inter-modal attention features, generate high-level fusion features, and determine a health report of the user based at least on the high-level fusion features.

[0014] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods for analyzing physiological motion data.

[0015] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for executing any one of the physiological motion data analysis methods.

[0016] Apply the technical solution of the present application to obtain the user's physiological motion data, wherein the physiological motion data includes electrocardiogram signals, respiratory signals and motion signals; use a neural network model to perform feature extraction processing on the physiological motion data to generate physiological motion features corresponding to the physiological motion data, and determine the intra-modal attention features and inter-modal attention features corresponding to the physiological motion features based on the physiological motion features; fuse the intra-modal attention features and inter-modal attention features to generate high-level fusion features, and determine the user's health report at least based on the high-level fusion features. Through the calculation of intra-modal and inter-modal attention, the key features of each modality are extracted and fused, which improves the robustness and accuracy of the multimodal data representation and solves the problem of low accuracy of health analysis results calculated by traditional health status analysis methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings constituting part of the present application are used to provide a further understanding of the present application. The exemplary embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 A hardware structure block diagram of a mobile terminal for performing a method for analyzing physiological motion data provided in an embodiment of the present application is shown;

[0019] Figure 2 A schematic diagram of a process of analyzing physiological motion data provided according to an embodiment of the present application is shown;

[0020] Figure 3 A schematic diagram showing a flow chart of a specific physiological motion data analysis method provided in an embodiment of the present application;

[0021] Figure 4 A structural block diagram of a physiological motion data analysis device provided according to an embodiment of the present application is shown.

[0022] The above drawings include the following reference numerals:

[0023] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0024] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] As introduced in the background technology, in order to solve the problem that the health analysis results calculated by the traditional health status analysis method have low accuracy, the embodiments of the present application provide a method for analyzing physiological motion data, an analysis device for physiological motion data, a computer-readable storage medium and an electronic device.

[0028] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0029] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal for analyzing physiological motion data according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0030] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the analysis method of physiological motion data in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0031] In this embodiment, a method for analyzing physiological motion data running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] Figure 2 FIG. 1 is a flow chart of a method for analyzing physiological motion data according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0033] Step S201, obtaining physiological motion data of the user, wherein the physiological motion data includes an electrocardiogram signal, a breathing signal and a motion signal;

[0034] Step S202, performing feature extraction processing on the physiological motion data using a neural network model to generate physiological motion features corresponding to the physiological motion data, and determining intra-modal attention features and inter-modal attention features corresponding to the physiological motion features according to the physiological motion features;

[0035] Specifically, the use of the dual attention mechanism extracts and fuses the key features of each modality through intra-modal and inter-modal attention calculations, thereby improving the robustness and accuracy of multimodal data representation.

[0036] Step S203, fusing the above-mentioned intra-modality attention features and the above-mentioned inter-modality attention features to generate high-level fusion features, and determining the health report of the above-mentioned user at least based on the high-level fusion features.

[0037] Specifically, the user's health report is generated by combining multimodal fusion features, which reduces the dependence on professional knowledge, provides users with intuitive and personalized explanations of their health status, and significantly improves user experience and application practicality.

[0038] Through this embodiment, the physiological motion data of the user is obtained, wherein the physiological motion data includes an electrocardiogram signal, a breathing signal, and a motion signal; a neural network model is used to perform feature extraction processing on the physiological motion data to generate physiological motion features corresponding to the physiological motion data, and the intra-modal attention features and inter-modal attention features corresponding to the physiological motion features are determined based on the physiological motion features; the intra-modal attention features and the inter-modal attention features are fused to generate high-level fusion features, and the user's health report is determined at least based on the high-level fusion features. Through the intra-modal and inter-modal attention calculations, the key features of each modality are extracted and fused, which improves the robustness and accuracy of the multimodal data representation, and solves the problem of low accuracy of the health analysis results calculated by the traditional health status analysis method.

[0039] In the specific implementation process, the health report of the above-mentioned user is determined at least based on the advanced fusion features, including: obtaining a user health database, extracting health-related features from the above-mentioned user health database to generate a user health portrait, wherein the above-mentioned user health portrait includes user information, lifestyle, previous symptoms and diagnosis; inputting the above-mentioned user health portrait and the above-mentioned advanced fusion features into a personalized analysis model to generate multimodal fusion features, and determining latent variables based on the above-mentioned multimodal fusion features, wherein the above-mentioned personalized analysis model is a parameterized conditional variational autoencoder; constructing a medical and health knowledge base, and using a large language model to encode the above-mentioned medical and health knowledge base into an embedding vector; performing similarity calculation on the above-mentioned latent variables and the above-mentioned embedding vectors to generate an embedding vector sequence arranged in descending order of similarity, selecting the first preset number of the above-mentioned embedding vectors in the above-mentioned embedding vector sequence and inputting them into the above-mentioned large language model to generate the above-mentioned health report of the above-mentioned user.

[0040] Among them, user information includes user age, gender, weight, height, occupation and other user-related information.

[0041] This method encodes the pre-built medical and health knowledge base (such as disease symptoms, health advice, etc.) into an embedding vector using a large language model, combines multimodal fusion features and health profiles, and uses the large language model to automatically generate a natural language report. By selecting the knowledge base entries with the highest similarity to the latent variables, easy-to-understand health advice and explanation reports are generated.

[0042] Specifically, determining the intra-modal attention feature corresponding to the physiological motion feature according to the physiological motion feature includes: according to the first formula: Determine the above intra-modal attention features, where All the above physiological motion features characterizing the above physiological motion data of the i-th type, W i represents the weight of the above physiological motion data of the ith type, b i represents the bias of the above physiological motion data of the i-th type, a i represents the eigenvalue within the i-th mode, N i represents the length of the feature vector of the i-th physiological motion data, Characterize the intra-modal attention computed over the i-th physiological motion data above.

[0043] This method uses intra-modal attention calculation to simulate the human attention mechanism by using a deep learning model. By weighting different parts of the input data, the model can better focus on important information and improve the performance and accuracy of the model. This solution uses intra-modal attention calculation to achieve the following effects:

[0044] 1. Improve model performance: Intra-modal attention calculation can help the model better capture important information in the input data, thereby improving the performance and accuracy of the model. By weighting the input data, the model can better focus on important features and improve the performance of the model on various tasks.

[0045] 2. Improve the generalization ability of the model: Intra-modal attention calculation can help the model better understand the relationship between input data, thereby improving the generalization ability of the model. By associating and integrating information between different modalities, the model can be better generalized to new data sets, improving the adaptability and robustness of the model.

[0046] 3. Reduce the computational complexity of the model: Intra-modal attention calculation can help the model reduce the computational complexity when processing large-scale data. By weighting the input data, the model can identify important information more quickly, reduce unnecessary calculations, and improve the efficiency and speed of the model.

[0047] In general, intra-modal attention computation is an effective technique that can improve the performance and generalization ability of the model and reduce the computational complexity, which is of great significance for various deep learning tasks.

[0048] More specifically, determining the inter-modal attention feature corresponding to the physiological motion feature according to the physiological motion feature includes: according to the second formula:

[0049]

[0050] Determine the above inter-modal attention features, where W β represents the weight matrix, b β is the bias vector, β represents the inter-modal eigenvalue, represents the above-mentioned intra-modal attention feature, and z represents the above-mentioned inter-modal attention feature.

[0051] This method uses inter-modal attention calculation, which is a technology used in multimodal data processing. It can help computer systems more effectively identify and understand the relevance and importance between different modalities when processing multimodal data. This solution uses inter-modal attention calculation to achieve the following effects:

[0052] 1. Improve the accuracy of data processing: Inter-modal attention calculation can help the system more accurately identify the correlation between different modalities, thereby improving the accuracy and performance of data processing.

[0053] 2. Improve the generalization ability of the model: By learning the correlation between different modalities, inter-modal attention calculation can improve the generalization ability of the model, making the model perform better when processing new data.

[0054] 3. Enhance the effect of feature extraction: Inter-modal attention calculation can help the system extract useful feature information more effectively, thereby improving the performance of the model in multimodal data processing tasks.

[0055] 4. Reduce the complexity of data processing: By learning the correlation between modalities, inter-modal attention calculation can help the system process multimodal data more efficiently and reduce the complexity and computational cost of data processing.

[0056] In general, inter-modal attention calculation technology can help the system better process multimodal data and improve the efficiency and accuracy of data processing.

[0057] Furthermore, before using the neural network model to perform feature extraction processing on the above-mentioned physiological motion data, the above-mentioned method also includes: determining the signal extraction time window of various physiological motion data according to the type of the above-mentioned physiological motion data; extracting and processing various physiological motion data according to the above-mentioned signal extraction time window to obtain target physiological motion data, and determining the similarity between various target physiological motion data; determining the phase difference between various target physiological motion data according to the above-mentioned similarity, and controlling the phase synchronization of various target physiological motion data according to the above-mentioned phase difference.

[0058] Furthermore, determining the signal extraction time window of various physiological motion data according to the type of the physiological motion data includes: according to the third formula: Determine the signal extraction time window of various physiological motion data mentioned above, where T i is the signal extraction time window of the i-th physiological motion data, ω i represents the main frequency of the signal of the i-th physiological motion data, σ i is the signal standard deviation of the above physiological motion data of the i-th type, A i represents the signal amplitude of the i-th physiological motion data, A0 is the custom threshold, and f is the adaptive function.

[0059] This method uses the formula: Calculate the similarity R between physiological motion data ij (τ), x i and x j are two physiological signals, N is the signal length, and τ is the time delay.

[0060] Then use the phase difference formula: The phase difference between physiological motion signals is calculated, and the ECG signal, motion signal and respiratory signal are synchronized according to the phase difference, so that the features of the fused and extracted physiological motion data are more accurate, and the results of model recognition are more accurate.

[0061] Specifically, after obtaining the user's physiological motion data, the method further includes: filtering and denoising the physiological motion data, wherein the filtering and denoising includes retaining the portion of the electrocardiogram signal related to the PQRST waveform and removing the noise of the power frequency interference in the motion data; and standardizing the physiological motion data after the filtering and denoising, wherein the standardization is used to scale the signal amplitude of the physiological motion data to within a preset range.

[0062] This method uses a bandpass filter or other suitable filter to remove various types of noise such as high-frequency noise and low-frequency drift in the signal. For ECG signals, the components related to the PQRST waveform are retained. For respiratory signals, they can be extracted through ECG signals or collected using other proprietary sensors. For motion data, the noise of power frequency interference is mainly removed. Normalization is to scale the amplitude of the signal to a uniform range, such as [-1,1] or [0,1], for subsequent processing.

[0063] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the method for analyzing physiological motion data of the present application will be described in detail below in conjunction with specific embodiments.

[0064] This embodiment relates to a specific method for analyzing physiological motion data, which provides a more comprehensive and accurate analysis by integrating multiple physiological signals such as electrocardiogram (ECG), respiratory rate, and motion data. The adaptive personalized analysis model dynamically adjusts according to personal historical health data and current physiological state to achieve personalized health assessment and prediction. The large language model is used to automatically generate natural language explanation reports to help users understand abnormal events and provide personalized health suggestions, significantly improving ease of use and practicality. Figure 3 As shown, the following steps are included:

[0065] Step S1: Data preprocessing: including filtering, denoising and standardization. Filtering and denoising uses a bandpass filter or other suitable filters to remove various types of noise such as high-frequency noise and low-frequency drift in the signal. For ECG signals, the components related to the PQRST waveform are retained. For respiratory signals, they can be extracted through ECG signals or collected using other proprietary sensors. For motion data, the noise of power frequency interference is mainly removed. Standardization is to scale the amplitude of the signal to a uniform range, such as [-1,1] or [0,1], for subsequent processing.

[0066] Step S2: Feature extraction, select a network structure suitable for time series data, such as 1D convolution layer to capture local time features, LSTM or GRU layer to capture time dependencies. In this part, it is important to note that physiological signals such as heart rate, breathing, and movement have different time scales. Therefore, a dynamic time window mechanism is introduced. This mechanism can adaptively adjust the time window size according to the characteristics of different physiological signals. As shown in the following formula:

[0067]

[0068] Among them, T i represents the time window size of the i-th signal, ω i Indicates the main frequency of the signal, σ i is the standard deviation of the signal, A i represents the amplitude of the signal, and A0 is a custom threshold. f is an adaptive function that dynamically adjusts the time window size according to the input signal characteristics. The above formula is only one form of f. Generally speaking, anything that meets the following three requirements can be used as an implementation of f. The higher the frequency, the smaller the time window; the greater the signal change, the larger the time window; when the signal amplitude exceeds a certain threshold, the time window will be further increased. This design enables the time window to be adaptively adjusted according to the characteristics of different physiological signals, and is particularly suitable for processing physiological signals with different characteristics such as heart rate, breathing, and movement. For example, for a rapidly changing heart rate signal, the function generates a smaller time window to capture instantaneous changes; while for a slowly changing breathing signal, the function generates a larger time window to obtain more stable features. In addition, it is also very necessary to synchronize the phases of the three modal signals. Different physiological signals (such as heart rate and breathing) often have an inherent coupling relationship, but their periods and phases may not be exactly the same. The purpose of phase synchronization is to find the best alignment between these signals so that their interactions can be analyzed more accurately. First, the similarity between the two signals is calculated, and the formula is as follows:

[0069]

[0070] Among them, x i and x j are two physiological signals, N is the signal length, and τ is the time delay. The phase difference is defined as the time delay that maximizes the cross-correlation function. The phase difference formula is:

[0071] Physiological signals such as heart rate, breathing, and movement are synchronously processed based on the phase difference calculated above.

[0072] Step S3: Intra-modality attention calculation. For each modality that has completed phase synchronization, the intra-modality attention is calculated as follows:

[0073]

[0074] a i =softmax(W i X i +b i );

[0075] in, Characterize all the features of the i-th mode. i represents the weight of the i-th mode, b i represents its offset, and N i Represents the length of the eigenvector of the i-th signal. Characterizes the intra-modality attention computed for the i-th modality.

[0076] Step S4: Calculation of inter-modal attention: The inter-modal attention is calculated according to the following formula.

[0077]

[0078] Among them, W β represents a weight matrix, b β is the corresponding bias vector, which is used to perform nonlinear transformation on the fused feature z.

[0079] Step S5: Multimodal fusion. The attention mechanism is used within the modality to obtain the weighted feature representation of each modality, and then the attention mechanism is used between the modalities to fuse different modalities, and finally the high-level fusion features are obtained through nonlinear transformation. This dual attention fusion method can adaptively learn the importance of different features and modalities and obtain a more robust multimodal representation.

[0080] Step S6: Constructing a personal health profile. Extract health-related features from the user's historical data, including demographic information (including age, gender, occupation, etc.), lifestyle, previous symptoms and diagnosis, etc., represented as P = [p1p2…p K ].

[0081] Step S7: Knowledge base embedding: The pre-built medical and health knowledge base (such as disease symptoms, health advice, etc.) is encoded into an embedding vector using a large language model.

[0082] Step S8: Personalized analysis model. The input of the model is the multimodal fused feature h and the personal health profile P; the model is a parameterized conditional variational autoencoder:

[0083] Z~Q(z|h,P);

[0084]

[0085] The latent variable Z integrates multimodal information and personal health status. f represents a nonlinear mapping function with parameter θ, which takes the latent variable Z and personal health profile P as input and outputs the reconstructed multimodal fusion feature.

[0086] Step S9: Health report generation. Calculate the similarity between the latent variable Z and the knowledge base embedding, and select the top-K related items. Concatenate the selected items into a sequence and send them to the large language model to generate a natural language report. We need to choose a large enough language model to ensure that it has a broad knowledge reserve and strong generation capabilities. In this solution, it is recommended to select a model of 7B (7 billion) or more as the base.

[0087] This embodiment uses a dual attention mechanism to analyze the fusion of multiple physiological signals such as electrocardiogram, respiratory rate, and motion data. Through the intra-modal and inter-modal attention calculation, the key features of each modality are extracted and integrated, which improves the robustness and accuracy of multimodal data representation; and uses the user's personal historical data (such as lifestyle, previous symptoms, demographic information, etc.) to build a health profile, and combines the domain knowledge base to perform personalized health analysis through parameterized conditional variational autoencoders, so that the parameters of the analysis model can be dynamically adjusted, and personalized and dynamic health assessment and prediction can be achieved by combining multimodal data and personal historical information; and combined with multimodal fusion features and health profiles, a large language model is used to automatically generate a natural language report. By selecting the knowledge base entry with the highest similarity to the hidden variable, easy-to-understand health advice and explanation reports are generated, which reduces the dependence on professional knowledge, provides users with intuitive and personalized health status explanations, and significantly improves user experience and application practicality. This embodiment also uses an adaptive time window mechanism and phase synchronization to process physiological signals of different time scales, making data analysis more adaptive and accurate, especially suitable for the synchronous analysis of multiple physiological signals.

[0088] The present application embodiment also provides an analysis device for physiological motion data. It should be noted that the analysis device for physiological motion data of the present application embodiment can be used to perform the analysis method for physiological motion data provided by the present application embodiment. The device is used to implement the above-mentioned embodiments and preferred embodiments, and the descriptions have been made and will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0089] The following is an introduction to the physiological motion data analysis device provided in the embodiment of the present application.

[0090] Figure 4 Schematic diagram of a device for analyzing physiological motion data according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0091] An acquisition unit 41 is used to acquire physiological motion data of the user, wherein the physiological motion data includes an electrocardiogram signal, a breathing signal and a motion signal;

[0092] The processing unit 42 is used to perform feature extraction processing on the physiological motion data using a neural network model to generate physiological motion features corresponding to the physiological motion data, and determine, based on the physiological motion features, intra-modality attention features and inter-modality attention features corresponding to the physiological motion features;

[0093] The first determination unit 43 is used to fuse the above-mentioned intra-modality attention features and the above-mentioned inter-modality attention features to generate high-level fusion features, and determine the health report of the above-mentioned user at least based on the high-level fusion features.

[0094] In this embodiment, the acquisition unit is used to acquire the physiological motion data of the user, wherein the physiological motion data includes an electrocardiogram signal, a breathing signal, and a motion signal; the processing unit is used to perform feature extraction processing on the physiological motion data using a neural network model, generate physiological motion features corresponding to the physiological motion data, and determine the intra-modal attention features and inter-modal attention features corresponding to the physiological motion features according to the physiological motion features; the first determination unit is used to perform fusion processing on the intra-modal attention features and the inter-modal attention features to generate high-level fusion features, and determine the health report of the user at least according to the high-level fusion features. Through the intra-modal and inter-modal attention calculations, the key features of each modality are extracted and fused, which improves the robustness and accuracy of the multimodal data representation, and solves the problem of low accuracy of the health analysis results calculated by the traditional health status analysis method.

[0095] As an optional solution, the first determination unit includes a first acquisition module, an input module, a construction module and a similarity calculation module; the first acquisition module is used to acquire a user health database, extract health-related features from the above user health database to generate a user health portrait, wherein the above user health portrait includes user information, lifestyle, previous symptoms and diagnosis; the input module is used to input the above user health portrait and the above advanced fusion features into a personalized analysis model to generate multimodal fusion features, and determine latent variables based on the above multimodal fusion features, wherein the above personalized analysis model is a parameterized conditional variational autoencoder; the construction module is used to construct a medical health knowledge base, and use a large language model to encode the above medical health knowledge base into an embedding vector; the similarity calculation module is used to perform similarity calculation on the above latent variables and the above embedding vectors, generate an embedding vector sequence arranged in descending order of similarity, select the first preset number of the above embedding vectors in the above embedding vector sequence and input them into the above large language model to generate the above health report of the above user.

[0096] In an optional solution, the processing unit includes a first determination module, configured to determine, according to a first formula: Determine the above intra-modal attention features, where All the above physiological motion features characterizing the above physiological motion data of the i-th type, W i represents the weight of the above physiological motion data of the ith type, b i represents the bias of the above physiological motion data of the i-th type, a i represents the eigenvalue within the i-th mode, N i represents the length of the feature vector of the i-th physiological motion data, Characterize the intra-modal attention computed over the i-th physiological motion data above.

[0097] In an optional solution, the processing unit further includes a second determining module, configured to determine the value of the second determination module according to a second formula: Determine the above inter-modal attention features, where W β represents the weight matrix, b β is the bias vector, β represents the inter-modal eigenvalue, represents the above-mentioned intra-modal attention feature, and z represents the above-mentioned inter-modal attention feature.

[0098] An optional solution, the device also includes a second determination unit, an extraction processing unit and a control unit; the second determination unit determines the signal extraction time window of various physiological motion data according to the type of the above-mentioned physiological motion data before using the neural network model to perform feature extraction processing on the above-mentioned physiological motion data; the extraction processing unit is used to extract and process the various physiological motion data according to the above-mentioned signal extraction time window to obtain target physiological motion data, and determine the similarity between the various target physiological motion data; the control unit is used to determine the phase difference between the various target physiological motion data according to the above-mentioned similarity, and control the phase synchronization of the various target physiological motion data according to the above-mentioned phase difference.

[0099] In an optional solution, the second determining unit includes a third determining module, configured to determine the value according to a third formula: Determine the signal extraction time window of various physiological motion data, where T i is the signal extraction time window of the i-th physiological motion data, ω i represents the main frequency of the signal of the i-th physiological motion data, σ i is the signal standard deviation of the above physiological motion data of the i-th type, A i represents the signal amplitude of the i-th physiological motion data, A0 is the custom threshold, and f is the adaptive function.

[0100] In an optional solution, the device also includes a denoising processing unit and a standardization processing unit; the denoising processing unit is used to filter and denoise the physiological motion data after acquiring the user's physiological motion data, wherein the filtering and denoising processing includes retaining the part of the electrocardiogram signal related to the PQRST waveform and removing the noise of the power frequency interference in the motion data; the standardization processing unit is used to standardize the physiological motion data after the filtering and denoising processing, wherein the standardization processing is used to scale the signal amplitude of the physiological motion data to a preset range.

[0101] The above-mentioned physiological motion data analysis device includes a processor and a memory, and the above-mentioned acquisition unit, processing unit, first determination unit, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in any combination.

[0102] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the problem of low accuracy of health analysis results calculated by traditional health status analysis methods can be solved by adjusting kernel parameters.

[0103] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0104] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the physiological motion data analysis method.

[0105] Specifically, the analysis methods of physiological motion data include:

[0106] Step S201, obtaining physiological motion data of the user, wherein the physiological motion data includes an electrocardiogram signal, a breathing signal and a motion signal;

[0107] Step S202, performing feature extraction processing on the physiological motion data using a neural network model to generate physiological motion features corresponding to the physiological motion data, and determining intra-modal attention features and inter-modal attention features corresponding to the physiological motion features according to the physiological motion features;

[0108] Step S203, fusing the above-mentioned intra-modality attention features and the above-mentioned inter-modality attention features to generate high-level fusion features, and determining the health report of the above-mentioned user at least based on the high-level fusion features.

[0109] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the above-mentioned physiological motion data analysis method is executed when the above-mentioned program is run.

[0110] Specifically, the analysis methods of physiological motion data include:

[0111] Step S201, obtaining physiological motion data of the user, wherein the physiological motion data includes an electrocardiogram signal, a breathing signal and a motion signal;

[0112] Step S202, performing feature extraction processing on the physiological motion data using a neural network model to generate physiological motion features corresponding to the physiological motion data, and determining intra-modal attention features and inter-modal attention features corresponding to the physiological motion features according to the physiological motion features;

[0113] Step S203, fusing the above-mentioned intra-modality attention features and the above-mentioned inter-modality attention features to generate high-level fusion features, and determining the health report of the above-mentioned user at least based on the high-level fusion features.

[0114] An embodiment of the present invention provides an electronic device, the device comprising a processor, a memory, and a program stored in the memory and executable on the processor, and when the processor executes the program, at least the following steps are implemented:

[0115] Step S201, obtaining physiological motion data of the user, wherein the physiological motion data includes an electrocardiogram signal, a breathing signal and a motion signal;

[0116] Step S202, performing feature extraction processing on the physiological motion data using a neural network model to generate physiological motion features corresponding to the physiological motion data, and determining intra-modal attention features and inter-modal attention features corresponding to the physiological motion features according to the physiological motion features;

[0117] Step S203, fusing the above-mentioned intra-modality attention features and the above-mentioned inter-modality attention features to generate high-level fusion features, and determining the health report of the above-mentioned user at least based on the high-level fusion features.

[0118] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0119] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:

[0120] Step S201, obtaining physiological motion data of the user, wherein the physiological motion data includes an electrocardiogram signal, a breathing signal and a motion signal;

[0121] Step S202, performing feature extraction processing on the physiological motion data using a neural network model to generate physiological motion features corresponding to the physiological motion data, and determining intra-modal attention features and inter-modal attention features corresponding to the physiological motion features according to the physiological motion features;

[0122] Step S203, fusing the above-mentioned intra-modality attention features and the above-mentioned inter-modality attention features to generate high-level fusion features, and determining the health report of the above-mentioned user at least based on the high-level fusion features.

[0123] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0124] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0125] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0128] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0129] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0130] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0131] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0132] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0133] 1) A method for analyzing physiological motion data of the present application includes: obtaining the physiological motion data of the user, wherein the physiological motion data includes an electrocardiogram signal, a breathing signal, and a motion signal; using a neural network model to perform feature extraction processing on the physiological motion data to generate physiological motion features corresponding to the physiological motion data, and determining the intra-modal attention features and inter-modal attention features corresponding to the physiological motion features based on the physiological motion features; fusing the intra-modal attention features and the inter-modal attention features to generate advanced fusion features, and determining the user's health report at least based on the advanced fusion features. By calculating the intra-modal and inter-modal attention, the key features of each modality are extracted and fused, which improves the robustness and accuracy of the multimodal data representation, and solves the problem of low accuracy of the health analysis results calculated by the traditional health status analysis method.

[0134] 2) A physiological motion data analysis device of the present application includes: an acquisition unit, which is used to acquire the physiological motion data of the user, wherein the physiological motion data includes electrocardiogram signals, respiratory signals and motion signals; a processing unit, which is used to perform feature extraction processing on the physiological motion data using a neural network model, generate physiological motion features corresponding to the physiological motion data, and determine the intra-modal attention features and inter-modal attention features corresponding to the physiological motion features based on the physiological motion features; a first determination unit, which is used to perform fusion processing on the intra-modal attention features and inter-modal attention features to generate high-level fusion features, and at least determine the user's health report based on the high-level fusion features. Through the intra-modal and inter-modal attention calculations, the key features of each modality are extracted and fused, which improves the robustness and accuracy of the multimodal data representation, and solves the problem of low accuracy of the health analysis results calculated by the traditional health status analysis method.

[0135] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for analyzing physiological motion data, characterized in that: include: Acquiring physiological motion data of the user, wherein the physiological motion data includes an electrocardiogram signal, a breathing signal, and a motion signal; Performing feature extraction processing on the physiological motion data using a neural network model to generate physiological motion features corresponding to the physiological motion data, and determining intra-modal attention features and inter-modal attention features corresponding to the physiological motion features according to the physiological motion features; The intra-modality attention feature and the inter-modality attention feature are fused to generate a high-level fusion feature, and the health report of the user is determined at least based on the high-level fusion feature.

2. The method according to claim 1, characterized in that Determining a health report of the user based at least on the high-level fusion features includes: Obtaining a user health database, extracting health-related features from the user health database to generate a user health profile, wherein the user health profile includes user information, lifestyle, past symptoms and diagnosis; Inputting the user health profile and the advanced fusion features into a personalized analysis model to generate multimodal fusion features, and determining latent variables according to the multimodal fusion features, wherein the personalized analysis model is a parameterized conditional variational autoencoder; Building a medical and health knowledge base, and encoding the medical and health knowledge base into an embedding vector using a large language model; The similarity between the latent variable and the embedding vector is calculated to generate a sequence of embedding vectors arranged in descending order of similarity, and the first preset number of the embedding vectors in the sequence of embedding vectors are selected and input into the large language model to generate the health report of the user.

3. The method according to claim 1, characterized in that Determining, according to the physiological motion feature, an intra-modal attention feature corresponding to the physiological motion feature, comprising: According to the first formula: Determine the intra-modality attention feature, where Characterize all the physiological motion features of the i-th physiological motion data, W i represents the weight of the ith type of physiological motion data, b i represents the bias of the physiological motion data of the i-th type, a i represents the eigenvalue within the i-th mode, N i represents the length of the feature vector of the i-th physiological motion data, Characterize the intra-modal attention calculated for the i-th physiological motion data.

4. The method according to claim 1, characterized in that Determining, according to the physiological motion feature, an inter-modal attention feature corresponding to the physiological motion feature, comprising: According to the second formula: Determine the inter-modal attention feature, where W β represents the weight matrix, b β is the bias vector, β represents the inter-modal eigenvalue, represents the intra-modality attention feature, and z represents the inter-modality attention feature.

5. The method according to claim 1, characterized in that Before using the neural network model to perform feature extraction processing on the physiological motion data, the method further includes: Determining a signal extraction time window for various physiological motion data according to the type of the physiological motion data; Extracting and processing various physiological motion data according to the signal extraction time window to obtain target physiological motion data, and determining the similarity between various target physiological motion data; The phase difference between the various target physiological motion data is determined according to the similarity, and the phase synchronization of the various target physiological motion data is controlled according to the phase difference.

6. The method according to claim 5, characterized in that Determining a signal extraction time window of various physiological motion data according to the type of the physiological motion data includes: According to the third formula: Determine the signal extraction time window of various physiological motion data, wherein T i is the signal extraction time window of the i-th physiological motion data, ω i represents the main frequency of the signal of the i-th physiological motion data, σ i is the signal standard deviation of the ith physiological motion data, A i represents the signal amplitude of the ith physiological motion data, A0 is the custom threshold, and f is the adaptive function.

7. The method according to claim 1, characterized in that After obtaining the user's physiological motion data, the method further includes: Performing filtering and denoising processing on the physiological motion data, wherein the filtering and denoising processing includes retaining the portion of the electrocardiogram signal related to the PQRST waveform and removing the noise of power frequency interference in the motion data; The physiological motion data after the filtering and denoising process is subjected to standardization processing, wherein the standardization processing is used to scale the signal amplitude of the physiological motion data to within a preset range.

8. A device for analyzing physiological motion data, characterized in that: include: An acquisition unit, used to acquire physiological motion data of the user, wherein the physiological motion data includes an electrocardiogram signal, a breathing signal and a motion signal; a processing unit, configured to perform feature extraction processing on the physiological motion data using a neural network model, generate physiological motion features corresponding to the physiological motion data, and determine, based on the physiological motion features, intra-modality attention features and inter-modality attention features corresponding to the physiological motion features; The first determination unit is used to fuse the intra-modality attention feature and the inter-modality attention feature to generate a high-level fusion feature, and determine the health report of the user at least based on the high-level fusion feature.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for analyzing physiological motion data according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include a method for analyzing physiological motion data according to any one of claims 1 to 7.

Images