Hidden aspiration monitoring method and device based on multi-source sensing data fusion, electronic equipment and storage medium

Abstract

The invention provides a hidden aspiration monitoring method and device based on multi-source sensing data fusion, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence and computers. According to the implementation scheme, the method comprises the steps that multi-mode swallowing time sequence signals of a first monitoring object are collected, and the multi-mode swallowing time sequence signals comprise a swallowing audio signal, a swallowing throat vibration signal and a swallowing breathing signal; performing multi-modal feature fusion on the multi-modal swallowing time sequence signal to obtain a target feature sequence; swallowing identification, aspiration identification and cough event identification are respectively carried out on each sequence segment feature in the target feature sequence; and under the condition that the target sequence segmentation characteristics which are identified to have swallowing and aspiration and do not have the cough event are obtained, determining that the first monitoring object has hidden aspiration.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and computer technology. Specifically, it relates to a method, apparatus, electronic device, and storage medium for detecting latent aspiration based on multi-source sensor data fusion. Background Technology

[0002] Silent aspiration refers to the phenomenon where a foreign object (such as food or liquid) enters the airway during swallowing without causing an obvious coughing or choking response. It is common in patients with impaired neurological function or weakened swallowing reflexes and is characterized by its high degree of concealment and serious harm. Long-term unrecognized silent aspiration can lead to aspiration pneumonia, chronic lung disease, and even suffocation, seriously threatening the patient's life and health.

[0003] Currently, clinical assessment of occult aspiration primarily relies on invasive imaging examinations (such as X-ray fluoroscopy). These methods are complex, costly, and difficult to implement for high-frequency, dynamic bedside screening and long-term follow-up. Some studies have begun to explore the use of non-invasive sensors (such as sound and surface electromyography) to record physiological signals during swallowing to aid in aspiration detection. However, traditional machine learning methods typically rely on manual feature extraction, heavily depend on expert experience, and struggle to adapt to signal variations across different patient groups and complex scenarios. Furthermore, most existing methods utilize only a single modality of signal, resulting in limited information and an inability to comprehensively characterize abnormal patterns during swallowing, thus limiting detection accuracy. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and storage medium for detecting latent aspiration based on multi-source sensor data fusion.

[0005] According to one aspect of this disclosure, a method for detecting latent aspiration based on multi-source sensor data fusion is provided, comprising: Collect multimodal swallowing timing signals of the first monitoring object, wherein the multimodal swallowing timing signals include swallowing audio signals, swallowing laryngeal vibration signals, and swallowing breathing signals; Multimodal feature fusion is performed on the multimodal swallowing time-series signal to obtain the target feature sequence; Swallowing recognition, aspiration recognition, and cough event recognition are performed on the segmented features of each sequence in the target feature sequence; If the target sequence segmentation features are obtained, indicating that swallowing and aspiration have occurred, but no coughing events are present, it is determined that the first monitored object has latent aspiration.

[0006] According to another aspect of this disclosure, a latent aspiration monitoring device based on multi-source sensor data fusion is provided, comprising: The timing signal acquisition module is used to acquire multimodal swallowing timing signals of the first monitoring object, wherein the multimodal swallowing timing signals include swallowing audio signals, swallowing laryngeal vibration signals, and swallowing breathing signals; The feature fusion module is used to perform multimodal feature fusion on the multimodal swallowing time-series signal to obtain the target feature sequence; The target recognition module is used to perform swallowing recognition, aspiration recognition, and cough event recognition on the segmented features of each sequence in the target feature sequence; The latent aspiration determination module is used to determine that the first monitored object has latent aspiration when the target sequence segmentation features that are identified as having swallowing and aspiration and no coughing events are obtained.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the latent aspiration detection methods based on multi-source sensor data fusion in the embodiments of this disclosure.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the latent aspiration monitoring methods based on multi-source sensor data fusion according to embodiments of this disclosure.

[0009] According to the technology disclosed herein, multimodal swallowing time-series signals of a first monitoring subject are collected, including swallowing audio signals, swallowing laryngeal vibration signals, and swallowing respiratory signals. Multimodal feature fusion is performed on the multimodal swallowing time-series signals to obtain a target feature sequence. Swallowing recognition, aspiration recognition, and cough event recognition are performed on the segmented features of each sequence in the target feature sequence. If the segmented features of the target sequence show that swallowing and aspiration are identified, but no cough event is found, then the first monitoring subject is determined to have latent aspiration. Therefore, using the technology disclosed herein, it is possible to accurately determine whether a monitoring subject exhibits latent aspiration.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart of a method for detecting latent aspiration based on multi-source sensor data fusion according to an embodiment of the present disclosure; Figure 2 This is a flowchart of another embodiment of the implicit aspiration detection method based on multi-source sensor data fusion in this disclosure; Figure 3 This is a structural block diagram of a latent aspiration monitoring device based on multi-source sensor data fusion according to an embodiment of the present disclosure; Figure 4 This is a block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] Figure 1 This is a flowchart of a latent aspiration detection method based on multi-source sensor data fusion, according to an embodiment of the present invention.

[0014] like Figure 1 As shown, this method for detecting latent aspiration based on multi-source sensor data fusion may include: S110, Collect multimodal swallowing timing signals of the first monitoring object, wherein the multimodal swallowing timing signals include swallowing audio signals, swallowing laryngeal vibration signals and swallowing breathing signals; S120, Multimodal feature fusion is performed on the multimodal swallowing time-series signal to obtain the target feature sequence; S130, perform swallowing recognition, aspiration recognition and cough event recognition on the segmented features of each sequence in the target feature sequence; S140, if the target sequence segmentation features are obtained where swallowing and aspiration are identified and there are no coughing events, it is determined that the first monitoring object has latent aspiration.

[0015] For example, the multimodal swallowing timing signal may include multiple modal swallowing timing signals, such as swallowing audio signals, swallowing laryngeal vibration signals, and swallowing breathing signals.

[0016] For example, swallowing audio signals of the first monitored subject are acquired via a microphone. Throat vibration signals during swallowing of the first monitored subject are acquired via a triaxial accelerometer. Swallowing respiratory signals are acquired via a nasal airflow sensor.

[0017] Understandably, the swallowing audio signal, swallowing laryngeal vibration signal, and swallowing breathing signal in the multimodal swallowing timing signal are aligned in time.

[0018] For example, before performing multimodal feature fusion on the multimodal swallowing time-series signal, the acquired multimodal swallowing time-series signal is preprocessed, including filtering, windowing or segmentation by sliding window, and feature extraction.

[0019] For example, the swallowing audio signal is downsampled to reduce the original high-sampling-rate audio signal to a lower sampling rate, while preserving key frequency components. Then, a bandpass filter is used to filter out non-swallowing-related low-frequency noise and high-frequency interference from the downsampled swallowing audio signal. Next, the spectrum of the swallowing audio signal is filtered again using a Mel filter bank. This yields the final swallowing audio signal.

[0020] For example, a median filter is used to smooth the swallowing throat vibration signal to remove pulse noise and abnormal peaks. Then, a low-pass filter is used to remove high-frequency interference from the swallowing throat vibration signal, retaining the low-frequency vibration signal related to swallowing vibration, to obtain the final swallowing throat vibration signal.

[0021] For example, statistical analysis of the duration of zero values ​​in swallowing and breathing signals yields the characteristics of apnea time. Analyzing the temporal relationship between breathing and swallowing actions in the swallowing and breathing signals helps determine the characteristics of respiratory change patterns during swallowing, i.e., the characteristics of respiratory-swallowing coordination patterns.

[0022] For example, a dual attention mechanism can be used to perform inter-modal and intra-modal feature fusion on multimodal swallowing time-series signals to obtain a target feature sequence, thereby achieving feature enhancement of multimodal swallowing time-series signals.

[0023] For example, such as Figure 2 As shown, swallowing recognition, aspiration recognition, and cough event recognition can be performed in parallel for each sequence segment feature. The aforementioned data preprocessing is applied to the sound signal (swallowing audio signal), the laryngeal vibration triaxial signal (swallowing laryngeal vibration signal), and the respiratory signal (swallowing and breathing signal). Then, the preprocessed signals undergo temporal encoding, channel encoding, and respiratory feature extraction to obtain multi-window features (also called temporal features), channel features, and respiratory features. These three features are then fused using dual attention to obtain the target feature sequence. Next, swallowing recognition, aspiration classification, and cough event recognition are performed on each sequence segment feature in the target feature sequence. This allows us to determine whether each sequence segment feature is identified as a swallowing segment (indicating swallowing), aspiration, or a cough event.

[0024] According to the above implementation method, multimodal swallowing time-series signals of the first monitoring object are collected, wherein the multimodal swallowing time-series signals include swallowing audio signals, swallowing laryngeal vibration signals, and swallowing respiratory signals; multimodal feature fusion is performed on the multimodal swallowing time-series signals to obtain a target feature sequence; swallowing recognition, aspiration recognition, and cough event recognition are performed on the segmented features of each sequence in the target feature sequence; if the segmented features of the target sequence show that swallowing and aspiration are identified, but no cough event is found, it is determined that the first monitoring object has latent aspiration. Therefore, using the technology of this disclosure, it is possible to accurately determine whether the monitoring object has latent aspiration.

[0025] In one implementation, multimodal feature fusion is performed on multimodal swallowing time-series signals to obtain a target feature sequence, including: using a time-series encoder to perform temporal attention learning on each modal swallowing time-series signal in the multimodal swallowing time-series signal to obtain a first feature sequence of each modal swallowing time-series signal; using a channel encoder to perform channel feature encoding on each modal swallowing time-series signal to obtain a second feature sequence of each modal swallowing time-series signal; performing cross-modal cross-attention learning on the first feature sequences of each modal swallowing time-series signal to obtain a third feature sequence of each modal swallowing time-series signal; fusing the second and third feature sequences of each modal swallowing time-series signal for each modal swallowing time-series signal to obtain a fourth feature sequence of each modal swallowing time-series signal; and fusing the fourth feature sequences of each modal swallowing time-series signal to obtain the target feature sequence.

[0026] Understandably, the modal swallowing timing signal is input into the timing encoder. The timing encoder uses a global attention mechanism to perform temporal attention learning between each sequence segment in the modal swallowing timing signal, thereby obtaining the first feature sequence of the modal swallowing timing signal, which can also be called the temporal feature sequence, and outputs the first feature sequence.

[0027] Understandably, the modal swallowing timing signal is input into the channel encoder, which performs channel encoding on the modal swallowing timing signal to obtain the second feature sequence of the modal swallowing timing signal, also known as the channel feature sequence, and outputs the second feature sequence.

[0028] Understandably, through a cross-modal cross-attention mechanism, cross-modal cross-attention learning is performed on the first feature sequences of each modal swallowing time-series signal to update the first feature sequences of each modal swallowing time-series signal, thus obtaining the third feature sequences of each modal swallowing time-series signal, which can also be called the updated time-series feature sequences. This achieves feature fusion between modalities.

[0029] Understandably, after completing the intermodal feature fusion, for each modal signal, intramodal feature fusion is performed on the channel feature sequence and the updated temporal feature sequence of that modality, thereby realizing the feature fusion of intramodal channel features and temporal features.

[0030] Understandably, the target feature sequence is obtained by splicing together the intermodal and intramodal fusion features (fourth feature sequence) of each modality and the respiratory features extracted from the swallowing and breathing signals.

[0031] According to the above implementation method, by performing intermodal and intramodal feature fusion on the channel features and timing features of each modal swallowing timing signal, feature enhancement of multimodal swallowing timing signals can be achieved. Furthermore, when using the enhanced features for latent aspiration, the accuracy of latent aspiration detection can be improved.

[0032] In one implementation, feature fusion is performed on the second feature sequence and the third feature sequence of the modal swallowing time-series signal, including: determining a first query vector based on the second feature sequence of the modal swallowing time-series signal; determining a first key vector and a first value vector based on the third feature sequence of the modal swallowing time-series signal; and performing attention interaction on the first query vector, the first key vector, and the first value vector to obtain a fourth feature sequence of the modal swallowing time-series signal.

[0033] Understandably, the channel features extracted from the modal swallowing timing signal are used as the first query vector, that is, the second feature sequence mentioned above is used as the first query vector.

[0034] Understandably, the temporal features updated through intermodal fusion are used as key-value pair vectors, that is, key-value pairs consisting of the first key vector and the first value vector are constructed using the third feature sequence.

[0035] For example, the attention interaction between the first query vector, the first key vector, and the first value vector can be performed using the following formula, as follows:

[0036] in, This indicates the attention learning mechanism. These represent the first query vector, the first key vector, and the first value vector, respectively. This represents the Euclidean distance between the first query vector and the first key vector. This represents the activation function.

[0037] According to the above implementation method, the channel features of the modal swallowing timing signal and the timing features after intermodal fusion can be fused into intramodal features through the attention mechanism to obtain the intermodal and intramodal fusion features of the modal swallowing timing signal, namely the fourth feature sequence.

[0038] In one implementation, feature fusion is performed on the fourth feature sequence of each modal swallowing timing signal to obtain a target feature sequence, including: extracting apnea time features and breathing-swallowing coordination mode features from the swallowing and breathing signals; and performing feature fusion on the fourth feature sequence, apnea time features, and breathing-swallowing coordination mode features of each modal swallowing timing signal to obtain the target feature sequence.

[0039] For example, apnea time characteristics and breathing-swallowing coordination pattern characteristics can be respiratory characteristics of swallowing and breathing signals.

[0040] For example, feature fusion is performed on the fourth feature sequences of each modality of swallowing time-series signal, and the fused features are concatenated with respiratory features to obtain the target feature sequence. Alternatively, the fourth feature sequences of each modality of swallowing time-series signal are concatenated with respiratory features to obtain the target feature sequence.

[0041] According to the above implementation method, the fusion features obtained by fusing intermodal and intramodal features of the multimodal swallowing time-series signal, and the respiratory features from the swallowing and breathing signals in the multimodal swallowing time-series signal are spliced ​​together to obtain the final target feature sequence. Thus, using the target feature sequence to detect latent aspiration can improve the accuracy of latent aspiration detection.

[0042] In one embodiment, swallowing recognition, aspiration recognition, and cough event recognition are performed on each sequence segment feature in the target feature sequence, including: performing swallowing recognition on each sequence segment feature in the target feature sequence to obtain a first sequence segment feature identified as having swallowing; performing aspiration recognition on each first sequence segment feature to obtain a second sequence segment feature identified as having aspiration; and performing cough event recognition on each second sequence segment feature to obtain a target sequence segment feature identified as having cough events.

[0043] For example, a binary classification model is used to classify each sequence segment feature in the target feature sequence into swallowing and non-swallowing categories, thereby determining whether each sequence segment feature is a swallowing segment or a non-swallowing segment. The swallowing segment is the first sequence segment feature identified as having been swallowed.

[0044] For example, after the swallowing segment is segmented, the swallowing segment (i.e., the second sequence segmentation feature) is weighted and fused with the corresponding channel feature to obtain a high-dimensional swallowing feature. This high-dimensional swallowing feature is then input into an aspiration classifier. The aspiration classifier performs aspiration identification on the high-dimensional swallowing feature to obtain a judgment result on whether the high-dimensional swallowing feature is identified as indicating aspiration.

[0045] For example, the high-dimensional swallowing feature identified as an aspiration event is input into a cough event classifier, and the classification result of whether the high-dimensional swallowing feature is identified as a cough event is obtained from the output of the cough event classifier.

[0046] According to the above implementation method, swallowing recognition, aspiration recognition, and cough event recognition can be performed sequentially on the segmented features of each sequence in the target feature sequence, thereby improving recognition efficiency.

[0047] In one implementation, cough event identification is performed on each second sequence segment feature to obtain target sequence segment features identified as having cough events. This includes: performing convolution processing on the energy features and waveform features in the second sequence segment features to obtain a cough probability time series map corresponding to the second sequence segment features; determining an attention mask based on the cough probability time series map; determining a second key vector and a second value vector based on the second sequence segment features; performing attention interaction on the second query vector, second key vector, and second value vector of the cough event classifier and superimposing an attention mask on the attention interaction to obtain a fused and updated second sequence segment feature; inputting the fused and updated second sequence segment feature into the cough event classifier to obtain the cough event identification result of the cough event classifier outputting the second sequence segment features, wherein the cough event identification result includes whether the second sequence segment features have cough events or not.

[0048] For example, the swallowing segment features identified as indicating aspiration are input into a cough-sensing attention module, which aims to identify latent aspiration by determining the presence of a cough event. First, cough probability modeling is performed on the local energy features of the multimodal signal. For each modality, short-time energy and waveform abrupt change features are extracted from its time series, and after processing through a one-dimensional convolutional network, a cough probability heatmap is generated. ,in For time, This is the modal number.

[0049] For example, a learnable cough query vector is used to guide an attention mechanism that focuses on a uniform cough pattern. Specifically, the cough query vector... The bond formed by the temporal features corresponding to the swallowing segment Sum To enhance attention-based interaction and highlight potentially cough-related areas, a cough probability heatmap was used. Constructing attention masks ， This represents the probability that each time window is a cough segment, and is a weighted cough mask based on attention operations. .

[0050] in, This represents the Euclidean distance between the second query vector and the second key vector.

[0051] For example, the fused and updated second sequence segmentation features are input into a cough event classifier to obtain the detection results of cough events. Based on the cough detection results, if a cough event occurs, it is determined to be an overt aspiration; if no cough event occurs, it is determined to be a latent aspiration.

[0052] According to the above implementation method, it is possible to accurately determine whether a cough event exists or does not exist in each multimodal segmentation feature, thereby improving the accuracy of latent aspiration detection.

[0053] In one implementation, the method further includes: determining an energy regularization loss based on the fused and updated features of each second sequence segment; determining a cross-entropy loss based on the cough event identification results and pre-labeled results corresponding to each second sequence segment feature; weighted summing of the energy regularization loss and the cross-entropy loss to obtain a target loss; and updating the model parameters of the cough event classifier and the second query vector based on the target loss.

[0054] For example, the cross-entropy loss function is used for both swallowing segment segmentation and aspiration detection during model training. An energy regularization term is introduced in the latent false absorption detection stage. The loss function in this stage is composed of cross-entropy loss. With energy canonical loss Combined structure.

[0055]

[0056] in, For the first The second sequence segmentation feature, For the first The second sequence segmentation feature, The total number of segmented features for the second sequence. This is the proportionality coefficient.

[0057] Figure 3 This is a structural block diagram of a latent aspiration monitoring device based on multi-source sensor data fusion according to an embodiment of the present disclosure.

[0058] like Figure 3 As shown, the latent aspiration monitoring device based on multi-source sensor data fusion includes: The timing signal acquisition module 410 is used to acquire the multimodal swallowing timing signal of the first monitoring object, wherein the multimodal swallowing timing signal includes swallowing audio signal, swallowing laryngeal vibration signal and swallowing breathing signal; The feature fusion module 420 is used to perform multimodal feature fusion on the multimodal swallowing time-series signal to obtain a target feature sequence; The target recognition module 430 is used to perform swallowing recognition, aspiration recognition, and cough event recognition on the segment features of each sequence in the target feature sequence, respectively. The latent aspiration determination module 440 is used to determine that the first monitoring object has latent aspiration when the target sequence segmentation features that are identified as having swallowing and aspiration and no coughing events are obtained.

[0059] In one embodiment, the feature fusion module 420 includes: The first feature determination unit is used to perform temporal attention learning on each modal swallowing time-series signal in the multimodal swallowing time-series signal using a temporal encoder to obtain the first feature sequence of each modal swallowing time-series signal; The second feature determination unit is used to perform channel feature encoding on each of the modal swallowing timing signals using a channel encoder to obtain a second feature sequence of each of the modal swallowing timing signals; The third feature determination unit is used to perform cross-modal cross-attention learning on the first feature sequence of each of the modal swallowing time-series signals to obtain the third feature sequence of each of the modal swallowing time-series signals; The fourth feature determination unit is used to perform feature fusion on the second feature sequence and the third feature sequence of each modal swallowing time sequence signal to obtain the fourth feature sequence of each modal swallowing time sequence signal. The target feature determination unit is used to perform feature fusion on the fourth feature sequence of each of the said modal swallowing timing signals to obtain the target feature sequence.

[0060] In one implementation, the fourth feature determining unit is specifically used for: Based on the second feature sequence of the modal swallowing timing signal, a first query vector is determined; Based on the third feature sequence of the modal swallowing timing signal, the first key vector and the first value vector are determined; Attention interaction is performed on the first query vector, the first key vector, and the first value vector to obtain the fourth feature sequence of the modal swallowing timing signal.

[0061] In one implementation, the target feature determination unit is specifically used for: Extract apnea time features and respiratory-swallowing coordination pattern features from the swallowing and breathing signals; The fourth feature sequence of each of the said modal swallowing timing signals, the apnea time feature and the breathing-swallowing coordination mode feature are fused to obtain the target feature sequence.

[0062] In one embodiment, the target recognition module 430 includes: The first identification unit is used to perform swallowing identification on each sequence segment feature in the target feature sequence to obtain the first sequence segment feature identified as having been swallowed. The second identification unit is used to identify mis-absorption of each of the first sequence segment features to obtain the second sequence segment features identified as having mis-absorption. The third identification unit is used to identify cough events in each of the second sequence segment features to obtain target sequence segment features that are identified as having cough events.

[0063] In one embodiment, the third identification unit is specifically used for: Convolution processing is performed on the energy features and waveform features in the second sequence segmentation features to obtain the cough probability time series diagram corresponding to the second sequence segmentation features; Based on the cough probability time series diagram, determine the attention mask; Based on the segmentation features of the second sequence, the second key vector and the second value vector are determined; The second query vector, the second key vector, and the second value vector of the cough event classifier are subjected to attention interaction, and the attention mask is superimposed on the attention interaction to obtain the fused and updated second sequence segmentation features. The fused and updated second sequence segment features are input into the cough event classifier to obtain the cough event identification result of the second sequence segment features output by the cough event classifier. The cough event identification result includes whether the second sequence segment features have a cough event or not.

[0064] In one embodiment, the above-described apparatus further includes an updating module, which comprises: The first loss determination unit is used to determine the energy regularization loss based on the fused and updated features of each segment of the second sequence; The second loss determination unit is used to determine the cross-entropy loss based on the cough event identification results and pre-labeling results corresponding to each of the second sequence segment features; The target loss determination unit is used to perform a weighted summation of the energy regularization loss and the cross-entropy loss to obtain the target loss; The update unit is used to update the model parameters of the cough event classifier and the second query vector based on the target loss.

[0065] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0066] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0067] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0068] Figure 4 This is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Figure 4 As shown, the electronic device includes a memory 410 and a processor 420. The memory 410 stores a computer program that can run on the processor 420. There can be one or more memories 410 and processors 420. The memory 410 can store one or more computer programs, which, when executed by the electronic device, cause the electronic device to perform the methods provided in the above-described method embodiments. The electronic device may also include a communication interface 430 for communicating with external devices and performing data exchange and transmission.

[0069] If the memory 410, processor 420, and communication interface 430 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0070] Optionally, in a specific implementation, if the memory 410, processor 420 and communication interface 430 are integrated on a single chip, the memory 410, processor 420 and communication interface 430 can communicate with each other through an internal interface.

[0071] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0072] Further, optionally, the aforementioned memory may include read-only memory and random access memory, and may also include non-volatile random access memory. The memory may be volatile or non-volatile, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct RAMBUS RAM (DR RAM).

[0073] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line, DSL) or wireless (e.g., infrared, Bluetooth, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)). It is worth noting that the computer-readable storage media mentioned in this disclosure may be non-volatile storage media; in other words, they may be non-transient storage media.

[0074] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0075] In the description of the embodiments of this disclosure, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0076] In the description of the embodiments disclosed herein, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0077] In the description of embodiments of this disclosure, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0078] The above description is merely an exemplary embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A method for detecting latent aspiration based on multi-source sensor data fusion, characterized in that, include: Collect multimodal swallowing timing signals of the first monitoring object, wherein the multimodal swallowing timing signals include swallowing audio signals, swallowing laryngeal vibration signals, and swallowing breathing signals; Multimodal feature fusion is performed on the multimodal swallowing time-series signal to obtain the target feature sequence; Swallowing recognition, aspiration recognition, and cough event recognition are performed on the segmented features of each sequence in the target feature sequence; If the target sequence segmentation features are obtained, indicating that swallowing and aspiration have occurred, but no coughing events are present, it is determined that the first monitored object has latent aspiration.

2. The method according to claim 1, characterized in that, The step of performing multimodal feature fusion on the multimodal swallowing time-series signal to obtain the target feature sequence includes: A timing encoder is used to perform timing attention learning on each modal swallowing timing signal in the multimodal swallowing timing signal to obtain the first feature sequence of each modal swallowing timing signal; A channel encoder is used to encode the channel features of each of the said modal swallowing timing signals to obtain the second feature sequence of each of the said modal swallowing timing signals; Cross-modal cross-attention learning is performed on the first feature sequence of each of the said modal swallowing time-series signals to obtain the third feature sequence of each of the said modal swallowing time-series signals; For each of the aforementioned modal swallowing time-series signals, feature fusion is performed on the second and third feature sequences of the aforementioned modal swallowing time-series signals to obtain the fourth feature sequence of each of the aforementioned modal swallowing time-series signals; The fourth feature sequence of each of the aforementioned modal swallowing timing signals is fused to obtain the target feature sequence.

3. The method according to claim 2, characterized in that, The feature fusion of the second and third feature sequences of the modal swallowing time-series signal includes: Based on the second feature sequence of the modal swallowing timing signal, a first query vector is determined; Based on the third feature sequence of the modal swallowing timing signal, the first key vector and the first value vector are determined; Attention interaction is performed on the first query vector, the first key vector, and the first value vector to obtain the fourth feature sequence of the modal swallowing timing signal.

4. The method according to claim 2, characterized in that, The step of fusing the fourth feature sequences of each of the aforementioned modal swallowing time-series signals to obtain the target feature sequence includes: Extract apnea time features and respiratory-swallowing coordination pattern features from the swallowing and breathing signals; The fourth feature sequence of each of the said modal swallowing timing signals, the apnea time feature and the breathing-swallowing coordination mode feature are fused to obtain the target feature sequence.

5. The method according to any one of claims 1-4, characterized in that, The step of performing swallowing recognition, aspiration recognition, and cough event recognition on the segmented features of each sequence in the target feature sequence includes: Swallowing identification is performed on each sequence segment feature in the target feature sequence to obtain the first sequence segment feature identified as having been swallowed; Misabsorption is identified for each of the first sequence segment features to obtain the second sequence segment features that are identified as having misabsorption. Cough event identification is performed on each of the second sequence segment features to obtain the target sequence segment features identified as having cough events.

6. The method according to claim 5, characterized in that, The step of identifying cough events by performing cough event identification on each of the second sequence segment features to obtain target sequence segment features identified as having cough events includes: Convolution processing is performed on the energy features and waveform features in the second sequence segmentation features to obtain the cough probability time series diagram corresponding to the second sequence segmentation features; Based on the cough probability time series diagram, determine the attention mask; Based on the segmentation features of the second sequence, the second key vector and the second value vector are determined; The second query vector, the second key vector, and the second value vector of the cough event classifier are subjected to attention interaction, and the attention mask is superimposed on the attention interaction to obtain the fused and updated second sequence segmentation features. The fused and updated second sequence segment features are input into the cough event classifier to obtain the cough event identification result of the second sequence segment features output by the cough event classifier. The cough event identification result includes whether the second sequence segment features have a cough event or not.

7. The method according to claim 6, characterized in that, Also includes: Based on the fused and updated features of each of the second sequence segments, the energy regularization loss is determined; Based on the cough event identification results and pre-labeling results corresponding to each of the second sequence segment features, the cross-entropy loss is determined; The target loss is obtained by weighted summation of the energy regularization loss and the cross-entropy loss; Based on the target loss, the model parameters of the cough event classifier and the second query vector are updated.

8. A latent aspiration monitoring device based on multi-source sensor data fusion, characterized in that, include: The timing signal acquisition module is used to acquire multimodal swallowing timing signals of the first monitoring object, wherein the multimodal swallowing timing signals include swallowing audio signals, swallowing laryngeal vibration signals, and swallowing breathing signals; The feature fusion module is used to perform multimodal feature fusion on the multimodal swallowing time-series signal to obtain the target feature sequence; The target recognition module is used to perform swallowing recognition, aspiration recognition, and cough event recognition on the segmented features of each sequence in the target feature sequence; The latent aspiration determination module is used to determine that the first monitored object has latent aspiration when the target sequence segmentation features that are identified as having swallowing and aspiration and no coughing events are obtained.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images