Method for recognition based on multi-modal data, electronic device and storage medium

Abstract

The present disclosure provides a method for recognition based on multi-modal data, an electronic device and a computer readable storage medium. The method comprises: obtaining a data string; feeding the data string into a first deep neural network to generate a first feature map; generating multi-dimensional data based on the data string; feeding the multi-dimensional data into a second deep neural network to generate a second feature map; fusing at least the first feature map and the second feature map into a specific feature vector; feeding the specific feature vector into a machine learning model, wherein the machine learning model outputs a recognition result corresponding to the data string in response to the specific feature vector. In this way, the machine learning model can achieve good recognition results based on a small amount of training data, thereby reducing the time and labor costs for labeling data.

Description

Technical Field

[0001] This disclosure relates to a data identification technology, and more particularly to a method, electronic device, and computer-readable storage medium for identification based on multimodal data. Background Technology

[0002] In related technologies, signal classification techniques generally involve using trained classifiers (such as deep neural networks (DNNs), support vector machines (SVMs), and random forests) to classify signals / data strings in a single dimension. However, this classification method may fail to achieve ideal classification results in certain situations.

[0003] For example, when determining whether a human's electrocardiogram (ECG) signal is abnormal, doctors generally make their judgment based on the waveform changes on a two-dimensional ECG, rather than directly interpreting the original one-dimensional ECG values / data.

[0004] Furthermore, when training a classifier capable of making judgments based on one-dimensional electrocardiogram (ECG) signals, since one-dimensional ECG signals record the cardiac electrophysiological activity that varies over time (measured in millivolts), the classifier may only learn the characteristics of voltage changes during training, and may fail to learn some clinically important features, such as the changes in the RR interval (i.e., the duration between the R waves of two adjacent heartbeats). Additionally, if the classifier is allowed to learn solely based on two-dimensional ECGs, it may be unable to learn the characteristics of voltage changes. Summary of the Invention

[0005] In view of this, the present disclosure provides a method, electronic device, and computer-readable storage medium for identification based on multimodal data, in order to at least partially solve the above-mentioned technical problems.

[0006] This disclosure provides a method for identification based on multimodal data, comprising: obtaining a first data string; feeding the first data string into a first deep neural network, wherein the first deep neural network generates a first feature map by extracting features from the first data string; generating multidimensional data based on the first data string, wherein the dimension of the multidimensional data is higher than the dimension of the first data string; feeding the multidimensional data into a second deep neural network, wherein the second deep neural network generates a second feature map by extracting features from the multidimensional data; fusing at least the first feature map and the second feature map into a specific feature vector; and feeding the specific feature vector into a machine learning model, wherein the machine learning model outputs an identification result corresponding to the first data string in response to the specific feature vector.

[0007] An electronic device for identification based on multimodal data includes a storage device and a processor. The storage device stores program code. The processor is coupled to the storage device and accesses the program code to execute: acquiring a first data string; feeding the first data string into a first deep neural network, wherein the first deep neural network generates a first feature map by extracting features from the first data string; generating multidimensional data based on the first data string, wherein the dimension of the multidimensional data is higher than the dimension of the first data string; feeding the multidimensional data into a second deep neural network, wherein the second deep neural network generates a second feature map by extracting features from the multidimensional data; fusing at least the first feature map and the second feature map into a specific feature vector; and feeding the specific feature vector into a machine learning model, wherein the machine learning model outputs an identification result corresponding to the first data string in response to the specific feature vector.

[0008] This disclosure provides a computer-readable storage medium recording an executable computer program, which is loaded by an electronic device to perform the following steps: obtaining a first data string; feeding the first data string into a first deep neural network, wherein the first deep neural network generates a first feature map by extracting features from the first data string; generating multi-dimensional data based on the first data string, wherein the dimension of the multi-dimensional data is higher than the dimension of the first data string; feeding the multi-dimensional data into a second deep neural network, wherein the second deep neural network generates a second feature map by extracting features from the multi-dimensional data; fusing at least the first feature map and the second feature map into a specific feature vector; and feeding the specific feature vector into a machine learning model, wherein the machine learning model outputs an identification result corresponding to the first data string in response to the specific feature vector.

[0009] Based on the above, this disclosure can obtain a corresponding first feature map after obtaining the first data string. Furthermore, this disclosure can also convert the first data string into corresponding multi-dimensional data and then obtain a corresponding second feature map. Subsequently, this disclosure can fuse the first and second feature maps into a specific feature vector through a specific mechanism, and then a trained machine learning model can output the corresponding recognition result based on this specific feature vector. In this way, the machine learning model can achieve good recognition results with only a small amount of training data, thereby correspondingly reducing the time and manpower costs for data annotation. Attached Figure Description

[0010] The accompanying drawings are incorporated in and form a part of this specification, and are used to further understand the embodiments of this disclosure. Hereinafter, embodiments of this disclosure will be described with reference to the accompanying drawings, and the principles of this disclosure will be explained.

[0011] Figure 1A This is a schematic diagram illustrating classification based on a one-dimensional data string according to embodiments of the present disclosure;

[0012] Figure 1B This is a normal electrocardiogram corresponding to a single heartbeat, illustrated according to embodiments of the present disclosure;

[0013] Figure 1C This is an electrocardiogram of a patient with atrial fibrillation (AF) illustrated according to embodiments of this disclosure;

[0014] Figure 2A These are multiple electrocardiograms that do not include signs of atrial fibrillation (AF) illustrated according to embodiments of this disclosure;

[0015] Figure 2B These are multiple electrocardiograms illustrating AF symptoms, drawn according to embodiments of this disclosure;

[0016] Figure 3 This is a schematic diagram of an electronic device for identification based on multimodal data, illustrated according to embodiments of the present disclosure;

[0017] Figure 4 This is a flowchart illustrating a method for identification based on multimodal data according to the first embodiment of this disclosure;

[0018] Figure 5A This is an application scenario diagram illustrated according to the first embodiment of this disclosure;

[0019] Figure 5B This is a schematic diagram of the fusion mechanism illustrated according to the first embodiment of this disclosure;

[0020] Figure 6 This is a flowchart illustrating a method for identification based on multimodal data according to a second embodiment of the present disclosure;

[0021] Figure 7A This is an application scenario diagram illustrated according to the second embodiment of this disclosure;

[0022] Figure 7B This is a schematic diagram of the fusion mechanism illustrated according to the second embodiment of this disclosure. Detailed Implementation

[0023] To make the objectives, technical solutions and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0024] Exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. Wherever possible, the same element symbols are used in the drawings and description to denote the same or similar parts.

[0025] Figure 1A This is a schematic diagram illustrating classification based on a one-dimensional data string according to embodiments of the present disclosure. Figure 1AIn this context, suppose data string 110 is a one-dimensional electrocardiogram signal measured from a patient, and classifier 112 is, for example, a judgment model (e.g., a deep learning network / machine learning model) that can determine whether data string 110 contains symptoms of atrial fibrillation (AF).

[0026] As mentioned above, classifier 112 may fail to achieve good classification results during training because it cannot learn some clinically important features. This will be explained further below.

[0027] Figure 1B This is a normal electrocardiogram corresponding to a single heartbeat, illustrated according to embodiments of this disclosure. Figure 1B As can be seen, the electrocardiogram corresponding to a normal heartbeat will include waveforms such as P, Q, R, S, and T. Among them, R corresponds to the R wave mentioned above, while the P wave is generated, for example, in response to atrial depolarization.

[0028] Figure 1C This is an electrocardiogram of a patient with atrial fibrillation (AF) illustrated according to embodiments of this disclosure. Figure 1C It can be seen that, in addition to the irregular ventricular rhythm, there is also no P wave originating from atrial depolarization. These clinical features cannot be learned by classifier 112 during its training process. Therefore, if classifier 112 is based on the corresponding... Figure 1C Interpreting one-dimensional electrocardiogram signals may result in... Figure 1C This could be a case of misdiagnosis as having no symptoms of atrial fibrillation (AF).

[0029] Figure 2A These are multiple electrocardiograms (ECGs) that do not contain atrial fibrillation (AF) symptoms, illustrated according to embodiments of this disclosure. In this embodiment, due to the regularity of the RR intervals in ECG 211, it should be possible to correctly determine that ECG 211 does not contain AF symptoms. However, if classifier 112 is allowed to interpret based on a one-dimensional ECG signal (e.g., voltage change) corresponding to ECG 211, classifier 112 may misjudge that ECG 211 contains AF symptoms because it has not learned that there can be irregularities in the RR intervals.

[0030] Furthermore, due to the presence of significant noise in ECG 212, the RR intervals may appear irregular. In this case, ECG 212 might be misclassified as containing AF symptoms. However, if classifier 112 interprets the data based on the one-dimensional ECG signal corresponding to ECG 212, it can still correctly determine that ECG 212 does not contain AF symptoms because the features learned by classifier 112 are different.

[0031] Figure 2BThese are multiple electrocardiograms (ECGs) containing atrial fibrillation (AF) symptoms, illustrated according to embodiments of this disclosure. In this embodiment, since the RR intervals of ECG 221 are irregular, it should be possible to correctly identify that ECG 221 contains AF symptoms. However, if classifier 112 is allowed to interpret based on a one-dimensional ECG signal (e.g., voltage change) corresponding to ECG 221, classifier 112 may misjudge that ECG 221 does not contain AF symptoms because it has not learned that the RR intervals can be irregular.

[0032] Furthermore, due to the presence of large T waves in ECG 222, it may be difficult to identify whether ECG 222 contains AF symptoms. However, if classifier 112 interprets the ECG based on the one-dimensional ECG signal corresponding to ECG 222, it can still correctly determine whether ECG 222 contains AF symptoms because the features learned by classifier 112 are different.

[0033] In view of this, this disclosure proposes a method and device for identification based on multimodal data, which can make a more accurate judgment after fusing data from multiple dimensions, and will be further explained below.

[0034] Figure 3 This is a schematic diagram of an electronic device based on multimodal data identification, illustrated according to embodiments of the present disclosure. In some embodiments, the electronic device 300 may be implemented as various computer devices and / or intelligent devices. Figure 3 In this context, electronic device 300 may include storage device 302 and processor 304.

[0035] Storage device 302 may be, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk or other similar device or combination of these devices, and may be used to record multiple program codes or modules.

[0036] Processor 304 is coupled to memory circuit 302 and may be a general purpose processor, special purpose processor, conventional processor, digital signal processor, multiple microprocessors, one or more microprocessors incorporating a digital signal processor core, controller, microcontroller, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), any other type of integrated circuit, state machine, processor based on advanced reduced instruction set machine (ARM), and the like.

[0037] In the embodiments of this disclosure, the processor 304 may access the modules and program code recorded in the storage device 302 to implement the identification method based on multimodal data proposed in this disclosure, the details of which are described below.

[0038] Figure 4 This is a flowchart illustrating a method for identification based on multimodal data according to a first embodiment of this disclosure. The method of this embodiment can be... Figure 3 The electronic device 300 executes, and the following is combined with Figure 3 Component description shown Figure 4 Details of each step. Furthermore, to make the concepts of this disclosure easier to understand, the following also incorporates… Figure 5A To elaborate further, among which, Figure 5A This is an application scenario diagram illustrated according to the first embodiment of this disclosure.

[0039] First, in step S410, the processor 304 obtains a first data string D1. In different embodiments, the first data string D1 is, for example, a data string composed of multiple data with a single dimension. For ease of explanation, it is assumed below that the first data string D1 is the aforementioned one-dimensional electrocardiogram signal measured from a patient, but it is not limited to this.

[0040] In one embodiment, after obtaining the first data string D1, the processor 304 may first perform some signal preprocessing on the first data string D1 to remove baseline drift and / or filter out some noise, thereby improving the signal quality of the first data string D1, but may not be limited to this.

[0041] Subsequently, in step S420, the processor 304 can feed the first data string D1 into the first deep neural network 510, wherein the first deep neural network 510 can generate a first feature map FM1 by extracting features of the first data string D1.

[0042] exist Figure 5AIn this embodiment, the first deep neural network 510 may include a first convolutional neural network (CNN) 512 and a recurrent neural network (RNN) 514 connected in series. In different embodiments, the designer may adjust the parameters and implementation methods of the first convolutional neural network 512 and the recurrent neural network 514 as needed. In one embodiment, the recurrent neural network 514 may be implemented as a long short-term memory (LSTM) model, for example, but is not limited to this.

[0043] In one embodiment, the processor 304 can feed a first data string D1 (after signal preprocessing) into a first convolutional neural network 512, wherein the first convolutional neural network 512 can output a first spatial feature vector SV1 in response to the first data string D1. In one embodiment, the first convolutional neural network 512 can be understood as extracting relevant one-dimensional spatial features from the first data string D1 to generate the first spatial feature vector SV1, but it is not limited to this.

[0044] Furthermore, the processor 304 can feed the first spatial feature vector SV1 into the recurrent neural network 514, wherein the recurrent neural network 514 can output a first temporal feature vector TV1 as a first feature map FM1 in response to the first spatial feature vector SV1. In one embodiment, the recurrent neural network 514 can be understood as extracting relevant temporal features from the first spatial feature vector SV1 to generate the first temporal feature vector TV1 (i.e., the first feature map FM1), but it is not limited to this.

[0045] Furthermore, after obtaining the first data string D1 via step S410, the processor 304 may also execute step S430 to generate multi-dimensional data D2 based on the first data string D1. In different embodiments, the multi-dimensional data D2 is, for example, an image drawn / converted by the processor 304 based on the first data string D1. In some embodiments, the multi-dimensional data D2 is, for example, a waveform image drawn / converted by the processor 304 based on the first data string D1, but is not limited thereto.

[0046] exist Figure 5A In this scenario, since the first data string D1 is assumed to be a one-dimensional electrocardiogram signal, the image drawn by the processor 304 based on the first data string D1 is, for example, Figure 5A The electrocardiogram shown in D2 is a multi-dimensional data set, but it is not limited to this.

[0047] Subsequently, in step S440, the processor 304 can feed the multi-dimensional data D2 into the second deep neural network 520, wherein the second deep neural network 520 can generate a second feature map FM2 by extracting features from the multi-dimensional data D2.

[0048] exist Figure 5A In this embodiment, the second deep neural network 520 may include a second convolutional neural network 522. In this case, the processor 304 may feed multi-dimensional data D2 into the second convolutional neural network 522, wherein the second convolutional neural network 522 may output a second spatial feature vector SV2 as a second feature map FM2 in response to the multi-dimensional data D2. In one embodiment, the second convolutional neural network 522 may be understood as extracting relevant two-dimensional spatial features from the multi-dimensional data D2 to generate the second spatial feature vector SV2, but it is not limited to this.

[0049] After obtaining the first feature map FM1 and the second feature map FM2, the processor 304 may execute step S450 to fuse the first feature map FM1 and the second feature map FM2 into a specific feature vector SS.

[0050] exist Figure 5A In this context, processor 304 can, for example, execute fusion mechanism 530 to fuse the first feature map FM1 and the second feature map FM2 into a specific feature vector SS, wherein the details of fusion mechanism 530 will be supplemented by... Figure 5B Further explanation is needed.

[0051] Figure 5B This is a schematic diagram illustrating the fusion mechanism according to the first embodiment of this disclosure. Figure 5B In this embodiment, the first feature map FM1 may have three dimensions: amplitude, time, and channel. The magnitude of the first feature map FM1 in the amplitude dimension may be 1 (since the first data string D1 has a dimension of 1). Furthermore, the magnitudes of the first feature map FM1 in the time and channel dimensions may be represented as b1 and b2, respectively, but are not limited to these. In some embodiments, the size of the first feature map FM1 may be represented as b1×b2×1, but is not limited to this.

[0052] In other embodiments, when the first data string D1 has other dimensions, the magnitude of the corresponding first feature map FM1 in the amplitude dimension can be any other value other than 1, and is not limited to... Figure 5A As shown in the figure.

[0053] Furthermore, the second feature map FM2 may have three dimensions: amplitude, time, and channel. The size of the second feature map FM2 in these three dimensions may be represented as a1, a2, and a3, respectively, but is not limited to these. In some embodiments, the size of the second feature map FM2 may be represented as a1×a2×a3, but is not limited to these.

[0054] In one embodiment, the processor 304 may feed the first feature map FM1 into the third convolutional neural network 532, wherein the third convolutional neural network 532 may output a first reference feature vector RV1 in response to the first feature map FM1.

[0055] exist Figure 5B In this process, the third convolutional neural network 532 can convert the first feature map FM1 into a first reference feature vector RV1 based on the magnitude of the second feature map FM2 in the channel and time dimensions, respectively. The magnitude of the first reference feature vector RV1 in the channel and time dimensions can be the same as the magnitude of the second feature map FM2 in the channel and time dimensions, respectively. That is, the magnitudes of the first reference feature vector RV1 in the channel and time dimensions can be a1 and a2, respectively. In this case, the size of the first reference feature vector RV1 can be expressed as a1×a2×1, but it is not limited to this.

[0056] Subsequently, the processor 304 can stack multiple first reference feature vectors RV1 into a second reference feature vector RV2 according to the size of the second feature map FM2, wherein the size of the second feature map FM2 is the same as the size of the second reference feature vector RV2.

[0057] exist Figure 5B In this process, processor 304 can vertically stack a3 first reference feature vectors RV1 to generate a second reference feature vector RV2. In this case, the size of the second reference feature vector RV2 can be represented as a1×a2×a3 (which is the same as the size of the second feature map FM2).

[0058] Subsequently, processor 304 can convert the second reference feature vector RV2 into a third reference feature vector RV3, wherein each element in the third reference feature vector RV3 can be between 0 and 1. In one embodiment, processor 304 can input the second reference feature vector RV2 into a Sigmoid function, wherein this Sigmoid function can output the third reference feature vector RV3 in response to the second reference feature vector RV2. In one embodiment, this Sigmoid function can convert / map each element in the second reference feature vector RV2 to a value between 0 and 1, thereby forming the third reference feature vector RV3, but it is not limited to this.

[0059] In addition to the Sigmoid function, the processor 304 may also use other functions to transform the second reference eigenvector RV2 into the third reference eigenvector RV3, such as the Tanh function (which allows each element in the third reference eigenvector RV3 to be between -1 and 1), the Softmax function, the Rectified Linear Unit (ReLU) function, etc., but is not limited to these.

[0060] Next, the processor 304 can generate a specific feature vector SS based on the second feature map FM2 and the third reference feature vector RV3. In one embodiment, since the second feature map FM2 and the third reference feature vector RV3 have the same size, the processor 304 can perform an attention mechanism (e.g., dot multiplication / dot addition / matrix inner product, etc.) based on the second feature map FM2 and the third reference feature vector RV3 to generate the specific feature vector SS, but it is not limited to this.

[0061] Furthermore, although Figure 5B The first feature map FM1 and the second feature map FM2 in the dataset originate from a first data string D1 and a multi-dimensional data string D2, respectively, which have different data types. However, the first data string D1 and the multi-dimensional data string D2 have the same length in time. Based on this, through... Figure 5B The mechanism can fuse the first feature map FM1 and the second feature map FM2 based on the time dimension.

[0062] In this case, since the first deep neural network 510 and the second deep neural network 520 have different designs and process data of different dimensions, the first feature map FM1 (i.e., the first temporal feature vector TV1) generated by the first deep neural network 510 and the second feature map FM2 (i.e., the second spatial feature vector SV2) generated by the second deep neural network 520 have different sizes. Therefore, the first feature map FM1 can be converted into a first reference feature vector RV1 (which has the same size as the second feature map FM2 in the time and channel dimensions) by the third convolutional neural network 532.

[0063] In addition, since the second feature map FM2 has an additional vertical dimension, the processor 304 can repeatedly stack the first reference feature vector RV1 into the second reference feature vector RV2 (which has the same size as the second feature map FM2).

[0064] Next, processor 304 can convert each element of the second reference feature vector RV2 into a value between 0 and 1 using the sigmoid function (which can be understood as a weight value) to generate the third reference feature vector RV3. Then, processor 304 can perform an attention mechanism (e.g., dot product / dot addition / matrix inner product) between the third reference feature vector RV3 and the second feature map FM2. In this way, features corresponding to one-dimensional data and features corresponding to multi-dimensional data can be appropriately fused to generate a specific feature vector SS for subsequent classification operations.

[0065] After obtaining the specific feature vector SS, in step S460, the processor 304 can feed the specific feature vector SS into the machine learning model 540, wherein the machine learning model 540 can output an identification result corresponding to the first data string D1 in response to the specific feature vector SS. In one embodiment, the identification result may indicate, for example, whether the first data string D1 contains symptoms of AF, but is not limited thereto.

[0066] In one embodiment, to enable the machine learning model 540 to possess the aforementioned capabilities, during the training process of the machine learning model 540, the designer can feed specially designed training data into the machine learning model 540 to allow it to learn accordingly. For example, after obtaining a first data string labeled as corresponding to AF (e.g., a one-dimensional electrocardiogram signal measured from a patient with AF), the processor 304 can generate a corresponding specific feature vector according to the scheme described in the first embodiment above, and feed it into the machine learning model 540. This allows the machine learning model 540 to learn relevant features about AF from this specific feature vector. In this case, when the machine learning model 540 later receives a specific feature vector SS corresponding to the first data string D1, the machine learning model 540 can determine whether the first data string D1 contains symptoms of AF, but this is not limited to this.

[0067] Figure 6 This is a flowchart illustrating a method for identification based on multimodal data according to a second embodiment of this disclosure. The method of this embodiment can be... Figure 3 The electronic device 300 executes, and the following will be combined with Figure 3 The components shown are used to illustrate. Figure 6 Details of each step. Furthermore, to make the concepts of this disclosure easier to understand, the following also incorporates… Figure 7A To elaborate further, among which, Figure 7A This is an application scenario diagram illustrated according to the second embodiment of this disclosure.

[0068] First, in step S610, the processor 304 obtains a first data string D1. In different embodiments, the first data string D1 is, for example, a data string composed of multiple data with a single dimension. For ease of explanation, it is assumed below that the first data string D1 is the aforementioned one-dimensional electrocardiogram signal measured from a patient, but it is not limited to this.

[0069] In one embodiment, after obtaining the first data string D1, the processor 304 may perform some signal preprocessing on the first data string D1 to eliminate baseline drift and / or filter out some noise, thereby improving the signal quality of the first data string D1, but may not be limited to this.

[0070] Subsequently, in step S620, the processor 304 can feed the first data string D1 into the first deep neural network 710, wherein the first deep neural network 710 can generate a first feature map FM1 by extracting features of the first data string D1.

[0071] exist Figure 7A In this embodiment, the first deep neural network 710 may include a first convolutional neural network 712. In one embodiment, the processor 304 may feed a first data string D1 into the first convolutional neural network 712, wherein the first convolutional neural network 712 may output a third spatial feature vector SV3 as a first feature map FM1 in response to the first data string D1. In one embodiment, the first convolutional neural network 712 may be understood as extracting relevant (one-dimensional) spatial features from the first data string D1 to generate the third spatial feature vector SV3, but it is not limited to this.

[0072] Furthermore, after obtaining the first data string D1 via step S610, the processor 304 may also execute step S630 to generate multi-dimensional data D2 based on the first data string D1. In different embodiments, the multi-dimensional data D2 is, for example, an image drawn / converted by the processor 304 based on the first data string D1. In some embodiments, the multi-dimensional data D2 is, for example, a waveform image drawn / converted by the processor 304 based on the first data string D1, but is not limited thereto.

[0073] exist Figure 7A In this scenario, since the first data string D1 is assumed to be a one-dimensional electrocardiogram signal, the image drawn by the processor 304 based on the first data string D1 is, for example, Figure 7A The electrocardiogram shown in D2 is a multi-dimensional data set, but it is not limited to this.

[0074] Subsequently, in step S640, the processor 304 can feed the multi-dimensional data D2 into the second deep neural network 520, wherein the second deep neural network 520 can generate a second feature map FM2 by extracting features from the multi-dimensional data D2.

[0075] exist Figure 7A In this embodiment, the second deep neural network 520 may include a second convolutional neural network 522. In this case, the processor 304 may feed multi-dimensional data D2 into the second convolutional neural network 522, wherein the second convolutional neural network 522 may output a second spatial feature vector SV2 as a second feature map FM2 in response to the multi-dimensional data D2. In one embodiment, the second convolutional neural network 522 may be understood as extracting relevant two-dimensional spatial features from the multi-dimensional data D2 to generate the second spatial feature vector SV2, but it is not limited to this.

[0076] In addition, in step S650, the processor 304 may also feed the first data string D3 into the third deep neural network 720, wherein the third deep neural network 720 can generate a third feature map FM3 by extracting features of the first data string D1.

[0077] exist Figure 7A In this embodiment, the third deep neural network 720 may include a recurrent neural network 722. In one embodiment, the recurrent neural network 722 may be implemented as an LSTM model, but is not limited thereto. In this case, the processor 304 may feed the first data string D1 into the recurrent neural network 722, wherein the recurrent neural network 722 may output a second temporal feature vector TV2 as a third feature map FM3 in response to the first data string D1. In one embodiment, the recurrent neural network 722 may be understood as extracting relevant temporal features from the first data string D1 to generate the second temporal feature vector TV2 (i.e., the third feature map FM3), but is not limited thereto.

[0078] After obtaining the first feature map FM1, the second feature map FM2 and the third feature map FM3, the processor 304 may execute step S660 to fuse the first feature map FM1, the second feature map FM2 and the third feature map FM3 into a specific feature vector SS1.

[0079] exist Figure 7A In this context, processor 304 can, for example, execute fusion mechanism 730 to fuse the first feature map FM1, the second feature map FM2, and the third feature map FM3 into a specific feature vector SS1. The details of fusion mechanism 730 will combine... Figure 7B Further explanation is needed.

[0080] Figure 7B This is a schematic diagram illustrating the fusion mechanism according to the second embodiment of this disclosure. Figure 7B In this configuration, the first feature map FM1 and the third feature map FM3 can have three dimensions: amplitude, time, and channel. The amplitude dimension of both the first feature map FM1 and the third feature map FM3 can be 1 (since the first data string D1 has a dimension of 1). Furthermore, the dimensions of the first feature map FM1 in the time and channel dimensions can be represented as b1 and b2, respectively. Similarly, the dimensions of the third feature map FM3 in the time and channel dimensions can be represented as c1 and c2, respectively. In some embodiments, the size of the first feature map FM1 can be represented as b1×b2×1, and the size of the third feature map FM3 can be represented as c1×c2×1, but these are not limited to these dimensions.

[0081] Furthermore, the second feature map FM2 may have three dimensions: amplitude, time, and channel. The size of the second feature map FM2 in these three dimensions may be represented as a1, a2, and a3, respectively, but is not limited to these. In some embodiments, the size of the second feature map FM2 may be represented as a1×a2×a3, but is not limited to these.

[0082] In one embodiment, the processor 304 may feed the first feature map FM1 into the fourth convolutional neural network 542, wherein the fourth convolutional neural network 542 may output a fourth reference feature vector RV4 in response to the first feature map FM1.

[0083] exist Figure 7B In this process, the fourth convolutional neural network 542 can convert the first feature map FM1 into a fourth reference feature vector RV4 based on the magnitudes of the second feature map FM2 in the channel and time dimensions, respectively. The magnitudes of the fourth reference feature vector RV4 in the channel and time dimensions can be the same as those of the second feature map FM2 in the channel and time dimensions, respectively. That is, the magnitudes of the fourth reference feature vector RV4 in the channel and time dimensions can be a1 and a2, respectively. In this case, the size of the fourth reference feature vector RV4 can be represented as a1×a2×1, but it is not limited to this.

[0084] Subsequently, the processor 304 can stack multiple fourth reference feature vectors RV4 into a fifth reference feature vector RV5 according to the size of the second feature map FM2, wherein the size of the second feature map FM2 is the same as the size of the fifth reference feature vector RV5.

[0085] exist Figure 7B In this process, processor 304 can vertically stack a3 fourth reference feature vectors RV4 to generate a fifth reference feature vector RV5. In this case, the size of the fifth reference feature vector RV5 can be represented as a1×a2×a3 (which is the same as the size of the second feature map FM2).

[0086] Subsequently, processor 304 can convert the fifth reference feature vector RV5 into a sixth reference feature vector RV6. In one embodiment, processor 304 can input the fifth reference feature vector RV5 into a Sigmoid function, wherein the Sigmoid function can output the sixth reference feature vector RV6 in response to the fifth reference feature vector RV5. In one embodiment, the Sigmoid function can convert / map each element in the fifth reference feature vector RV5 to a value between 0 and 1, thereby forming the sixth reference feature vector RV6, but it is not limited to this.

[0087] In addition to the Sigmoid function, the processor 304 may also use other functions to transform the fifth reference eigenvector RV5 into the sixth reference eigenvector RV6, such as the Tanh function, Softmax function, ReLU, etc., but it is not limited to these.

[0088] In one embodiment, the processor 304 may feed the third feature map FM3 into the fifth convolutional neural network 552, wherein the fifth convolutional neural network 552 may output a seventh reference feature vector RV7 in response to the third feature map FM3.

[0089] exist Figure 7B In this model, the fifth convolutional neural network 552 can transform the third feature map FM3 into a seventh reference feature vector RV7 based on the magnitudes of the second feature map FM2 in the channel and time dimensions. The magnitudes of the seventh reference feature vector RV7 in the channel and time dimensions can be the same as those of the second feature map FM2 in the channel and time dimensions, respectively. That is, the magnitudes of the seventh reference feature vector RV7 in the channel and time dimensions can be a1 and a2, respectively. In this case, the size of the seventh reference feature vector RV7 can be represented as a1×a2×1, but it is not limited to this.

[0090] Subsequently, the processor 304 can stack multiple seventh reference feature vectors RV7 into an eighth reference feature vector RV8 according to the size of the second feature map FM2, wherein the size of the second feature map FM2 is the same as the size of the eighth reference feature vector RV8.

[0091] exist Figure 7B In this process, processor 304 can vertically stack a3 seventh reference feature vectors RV7 to generate an eighth reference feature vector RV8. In this case, the size of the eighth reference feature vector RV8 can be represented as a1×a2×a3 (which is the same as the size of the second feature map FM2).

[0092] Subsequently, processor 304 can convert the eighth reference feature vector RV8 into a ninth reference feature vector RV9. In one embodiment, processor 304 can input the eighth reference feature vector RV8 into a Sigmoid function, wherein the Sigmoid function can output the ninth reference feature vector RV9 in response to the eighth reference feature vector RV8. In one embodiment, the Sigmoid function can convert / map each element in the eighth reference feature vector RV8 to a value between 0 and 1, thereby forming the ninth reference feature vector RV9, but it is not limited to this.

[0093] In addition to the Sigmoid function, the processor 304 may also use other functions to transform the eight reference eigenvectors RV8 into the ninth reference eigenvectors RV9, such as the Tanh function, Softmax function, ReLU, etc., but it is not limited to these.

[0094] Next, the processor 304 can generate a specific feature vector SS1 based on the second feature map FM2, the sixth reference feature vector RV6, and the ninth feature vector RV9. In one embodiment, since the second feature map FM2, the sixth reference feature vector RV6, and the ninth feature vector RV9 have the same size, the processor 304 can perform an attention mechanism (e.g., dot multiplication / dot addition / matrix inner product, etc.) based on the second feature map FM2, the sixth reference feature vector RV6, and the ninth feature vector RV9 to generate the specific feature vector SS1, but it is not limited to this.

[0095] After obtaining the specific feature vector SS1, in step S670, the processor 304 can feed the specific feature vector SS1 into the machine learning model 740, wherein the machine learning model 740 can output an identification result corresponding to the first data string D1 in response to the specific feature vector SS1. In one embodiment, the identification result may indicate, for example, whether the first data string D1 contains AF symptoms, but is not limited thereto.

[0096] In one embodiment, to enable the machine learning model 740 to possess the aforementioned capabilities, during the training process of the machine learning model 740, the designer can feed specially designed training data into the machine learning model 740 to allow it to learn accordingly. For example, after obtaining a first data string labeled as corresponding to AF (e.g., a one-dimensional electrocardiogram signal measured from a patient with AF), the processor 304 can generate a corresponding specific feature vector according to the scheme described in the second embodiment above, and feed it into the machine learning model 740. This allows the machine learning model 740 to learn relevant features about AF from this specific feature vector. In this case, when the machine learning model 740 later receives a specific feature vector SS1 corresponding to the first data string D1, the machine learning model 740 can determine whether the first data string D1 contains AF symptoms, but this is not limited to this.

[0097] Experiments have shown that the method proposed in this disclosure enables machine learning models 540 and 740 to achieve good recognition results with only a small amount of training data. Furthermore, compared to methods that rely solely on the first data string D1 or multi-dimensional data D2 for recognition, the method of this disclosure also allows machine learning models 540 and 740 to achieve better recognition results.

[0098] To make the above concepts easier to understand, the following explanation is further provided in conjunction with Table 1.

[0099] Table 1

[0100]

[0101] In Table 1, Method 1 can be understood as being based solely on Figure 5A The identification mechanism performed in the upper part (i.e., the mechanism that identifies based solely on the first data string D1) can be understood in Method 2 as being based solely on... Figure 5A The identification mechanism in the lower half (i.e., a mechanism that performs identification based solely on multi-dimensional data D2). Furthermore, Method 3 can be understood as an identification mechanism that, after obtaining the identification results from Methods 1 and 2, fuses these two results to generate a comprehensive identification result. Additionally, Methods 1-4 can be used to train the corresponding machine learning models based on training datasets A or B, respectively, where training dataset B includes only approximately 60% of the training data from training dataset A.

[0102] As shown in Table 1, the accuracy achieved by training the machine learning model corresponding to method 4 using training dataset A is approximately 96.14%, while the accuracy achieved by training the machine learning model corresponding to method 4 using training dataset B is approximately 96.30%. Therefore, the difference in accuracy between training the machine learning model corresponding to method 4 using either training dataset A or B is only about 0.16%. In other words, the machine learning model corresponding to the method disclosed in this paper can achieve good recognition accuracy based on only a small amount of training data.

[0103] In contrast, when training the machine learning model corresponding to Method 1 using training datasets A or B, the accuracy difference is only about 0.29%, but the overall recognition accuracy is significantly lower than that of the machine learning model corresponding to the method disclosed in this paper. Furthermore, when training the machine learning model corresponding to Method 2 using training datasets A or B, the accuracy difference is as high as approximately -3.47%. Therefore, it can be seen that the machine learning model corresponding to Method 2 cannot achieve good recognition accuracy based on only a small amount of training data. Similarly, as shown in Table 1, the machine learning model corresponding to Method 3 also cannot achieve good recognition accuracy based on only a small amount of training data.

[0104] Furthermore, the fields related to sensitivity and specificity also show the same trend as the accuracy field. That is, the machine learning model corresponding to the method disclosed in this paper can achieve good recognition sensitivity and specificity based on only a small amount of training data, while machine learning models trained by other methods cannot achieve good recognition sensitivity and specificity based on only a small amount of training data.

[0105] It should be understood that although the above example uses a one-dimensional electrocardiogram (ECG) signal as the first data string D1, the possible implementations of this disclosure are not limited to this. In other embodiments, any kind of (one-dimensional) signal can be used as a possible implementation of the first data string D1. Accordingly, the multi-dimensional data D2 converted from the first data string D1 is not limited to the ECG of the above embodiment, but can be other forms of data.

[0106] In summary, this disclosure allows for the acquisition of a first feature map after obtaining the first data string. Furthermore, this disclosure can convert the first data string into corresponding multi-dimensional data and then obtain a corresponding second feature map. Subsequently, this disclosure can fuse the first and second feature maps into a specific feature vector using a specific mechanism, and then a trained machine learning model can output the corresponding recognition result based on this specific feature vector.

[0107] By converting the first data string (e.g., an electrocardiogram signal) into multi-dimensional data in different forms (e.g., an electrocardiogram), the diversity of the data can be increased accordingly, thereby achieving feature enrichment. Furthermore, although the first data string and the multi-dimensional data have different forms, they are highly complementary (i.e., one possesses features lacking in the other), allowing the machine learning model to output better decisions, thus improving the classification performance. Moreover, since this disclosure requires only a small amount of training data to achieve good recognition / classification results, the cost of labeling the training data can be reduced.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the above embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A method for identification based on multimodal data, applied to the detection of abnormalities in electrocardiogram signals, the method comprising: Obtain the first data string, which is a one-dimensional electrocardiogram signal; The first data string is fed into a first deep neural network, wherein the first deep neural network generates a first feature map by extracting features from the first data string; Multi-dimensional data is generated based on the first data string, wherein the dimension of the multi-dimensional data is higher than the dimension of the first data string, and the multi-dimensional data is an image obtained by conversion based on the first data string; The multi-dimensional data is fed into a second deep neural network, wherein the second deep neural network generates a second feature map by extracting features from the multi-dimensional data; The method of fusing at least the first feature map and the second feature map into a specific feature vector includes: feeding the first feature map into a third convolutional neural network, wherein the third convolutional neural network outputs a first reference feature vector in response to the first feature map; stacking multiple first reference feature vectors into a second reference feature vector according to the size of the second feature map, wherein the size of the second feature map is the same as the size of the second reference feature vector; converting the second reference feature vector into a third reference feature vector; and generating the specific feature vector based on the second feature map and the third reference feature vector. The specific feature vector is fed into a machine learning model, wherein the machine learning model outputs an identification result corresponding to the first data string in response to the specific feature vector, and the identification result indicates that the first data string is suitable for clinical decision-making.

2. The method according to claim 1, wherein, The first deep neural network includes a first convolutional neural network and a recurrent neural network, and the method includes: The first data string is fed into the first convolutional neural network, wherein the first convolutional neural network outputs a first spatial feature vector in response to the first data string; and The first spatial feature vector is fed into the recurrent neural network, wherein the recurrent neural network outputs a first temporal feature vector as the first feature map in response to the first spatial feature vector.

3. The method according to claim 1, wherein, The second deep neural network includes a second convolutional neural network, and the method includes: The multi-dimensional data is fed into the second convolutional neural network, wherein the second convolutional neural network outputs a second spatial feature vector as the second feature map in response to the multi-dimensional data.

4. The method according to claim 1, wherein, The steps of converting the second reference feature vector into the third reference feature vector include: The second reference feature vector is input into the Sigmoid function, wherein the Sigmoid function outputs the third reference feature vector in response to the second reference feature vector, wherein each element in the third reference feature vector is between 0 and 1.

5. The method according to claim 1, wherein, The step of generating the specific feature vector based on the second feature map and the third reference feature vector includes: An attention mechanism is performed based on the second feature map and the third reference feature vector to generate the specific feature vector.

6. The method according to claim 1, further comprising: The step of feeding the first data string into a third deep neural network, wherein the third deep neural network generates a third feature map by extracting features from the first data string, and at least fusing the first feature map and the second feature map into the specific feature vector includes: The first feature map, the second feature map, and the third feature map are fused into the specific feature vector.

7. The method according to claim 6, wherein, The first deep neural network includes a first convolutional neural network, and the method includes: The first data string is fed into the first convolutional neural network, wherein the first convolutional neural network outputs a third spatial feature vector as the first feature map in response to the first data string.

8. The method according to claim 6, wherein, The third deep neural network includes a recurrent neural network, and the method includes: The first data string is fed into the recurrent neural network, wherein the recurrent neural network outputs a second time feature vector as the third feature map in response to the first data string.

9. The method according to claim 6, wherein, The step of fusing the first feature map, the second feature map, and the third feature map into the specific feature vector includes: The first feature map is fed into a fourth convolutional neural network, wherein the fourth convolutional neural network outputs a fourth reference feature vector in response to the first feature map; The multiple fourth reference feature vectors are stacked into a fifth reference feature vector according to the size of the second feature map, wherein the size of the second feature map is the same as the size of the fifth reference feature vector; Transform the fifth reference feature vector into a sixth reference feature vector; and The third feature map is fed into the fifth convolutional neural network, wherein the fifth convolutional neural network outputs a seventh reference feature vector in response to the third feature map; The multiple seventh reference feature vectors are stacked into an eighth reference feature vector according to the size of the second feature map, wherein the size of the second feature map is the same as the size of the eighth reference feature vector; Transform the eighth reference feature vector into a ninth reference feature vector; and The specific feature vector is generated based on the second feature map, the sixth reference feature vector, and the ninth reference feature vector.

10. The method according to claim 9, wherein, The step of converting the fifth reference feature vector into the sixth reference feature vector includes: The fifth reference feature vector is input into the Sigmoid function, wherein the Sigmoid function outputs the sixth reference feature vector in response to the fifth reference feature vector, wherein each element in the sixth reference feature vector is between 0 and 1; The step of converting the eighth reference feature vector into the ninth reference feature vector includes: The eighth reference feature vector is input into the Sigmoid function, wherein the Sigmoid function outputs the ninth reference feature vector in response to the eighth reference feature vector, wherein each element in the ninth reference feature vector is between 0 and 1.

11. The method according to claim 9, wherein, The step of generating the specific feature vector based on the second feature map, the sixth reference feature vector, and the ninth reference feature vector includes: An attention mechanism is executed based on the second feature map, the sixth reference feature vector, and the ninth reference feature vector to generate the specific feature vector.

12. The method according to claim 1, wherein, The multi-dimensional data report includes waveform images drawn based on the first data string.

13. An electronic device for identification based on multimodal data, applied to the detection of abnormalities in electrocardiogram signals, the electronic device comprising: Storage device, which stores program code; A processor, coupled to the storage device, accesses the program code for execution: Obtain the first data string, which is a one-dimensional electrocardiogram signal; The first data string is fed into a first deep neural network, wherein the first deep neural network generates a first feature map by extracting features from the first data string; Multi-dimensional data is generated based on the first data string, wherein the dimension of the multi-dimensional data is higher than the dimension of the first data string, and the multi-dimensional data is an image obtained by conversion based on the first data string; The multi-dimensional data is fed into a second deep neural network, wherein the second deep neural network generates a second feature map by extracting features from the multi-dimensional data; The method of fusing at least the first feature map and the second feature map into a specific feature vector includes: feeding the first feature map into a third convolutional neural network, wherein the third convolutional neural network outputs a first reference feature vector in response to the first feature map; stacking multiple first reference feature vectors into a second reference feature vector according to the size of the second feature map, wherein the size of the second feature map is the same as the size of the second reference feature vector; converting the second reference feature vector into a third reference feature vector; and generating the specific feature vector based on the second feature map and the third reference feature vector. The specific feature vector is fed into a machine learning model, wherein the machine learning model outputs an identification result corresponding to the first data string in response to the specific feature vector, and the identification result indicates that the first data string is suitable for clinical decision-making.

14. The electronic device according to claim 13, wherein, The first deep neural network includes a first convolutional neural network and a recurrent neural network, and the processor is configured to: The first data string is fed into the first convolutional neural network, wherein the first convolutional neural network outputs a first spatial feature vector in response to the first data string; and The first spatial feature vector is fed into the recurrent neural network, wherein the recurrent neural network outputs a first temporal feature vector as the first feature map in response to the first spatial feature vector.

15. The electronic device according to claim 13, wherein, The second deep neural network includes a second convolutional neural network, and the processor is configured to: The multi-dimensional data is fed into the second convolutional neural network, wherein the second convolutional neural network outputs a second spatial feature vector as the second feature map in response to the multi-dimensional data.

16. The electronic device according to claim 13, wherein, The processor is configured to: The second reference feature vector is input into the Sigmoid function, wherein the Sigmoid function outputs the third reference feature vector in response to the second reference feature vector, wherein each element in the third reference feature vector is between 0 and 1.

17. The electronic device according to claim 13, wherein, The step of generating the specific feature vector based on the second feature map and the third reference feature vector includes: An attention mechanism is performed based on the second feature map and the third reference feature vector to generate the specific feature vector.

18. The electronic device of claim 13, wherein the processor is further configured to: The first data string is fed into a third deep neural network, wherein... The third deep neural network generates a third feature map by extracting features from the first data string, and the step of fusing at least the first feature map and the second feature map into the specific feature vector includes: The first feature map, the second feature map, and the third feature map are fused into the specific feature vector.

19. The electronic device according to claim 18, wherein, The first deep neural network includes a first convolutional neural network, and the processor is configured to: The first data string is fed into the first convolutional neural network, wherein the first convolutional neural network outputs a third spatial feature vector as the first feature map in response to the first data string.

20. The electronic device according to claim 18, wherein, The third deep neural network includes a recurrent neural network, and the processor is configured to: The first data string is fed into the recurrent neural network, wherein the recurrent neural network outputs a second time feature vector as the third feature map in response to the first data string.

21. The electronic device according to claim 18, wherein, The processor is configured to: The first feature map is fed into a fourth convolutional neural network, wherein the fourth convolutional neural network outputs a fourth reference feature vector in response to the first feature map; The multiple fourth reference feature vectors are stacked into a fifth reference feature vector according to the size of the second feature map, wherein the size of the second feature map is the same as the size of the fifth reference feature vector; Transform the fifth reference feature vector into a sixth reference feature vector; and The third feature map is fed into the fifth convolutional neural network, wherein the fifth convolutional neural network outputs a seventh reference feature vector in response to the third feature map; The multiple seventh reference feature vectors are stacked into an eighth reference feature vector according to the size of the second feature map, wherein the size of the second feature map is the same as the size of the eighth reference feature vector; Transform the eighth reference feature vector into a ninth reference feature vector; and The specific feature vector is generated based on the second feature map, the sixth reference feature vector, and the ninth reference feature vector.

22. The electronic device according to claim 21, wherein, The processor is configured to: The fifth reference feature vector is input into the Sigmoid function, wherein the Sigmoid function outputs the sixth reference feature vector in response to the fifth reference feature vector, wherein each element in the sixth reference feature vector is between 0 and 1; The step of converting the eighth reference feature vector into the ninth reference feature vector includes: The eighth reference feature vector is input into the Sigmoid function, wherein the Sigmoid function outputs the ninth reference feature vector in response to the eighth reference feature vector, wherein each element in the ninth reference feature vector is between 0 and 1.

23. The electronic device according to claim 21, wherein, The processor is configured to: An attention mechanism is executed based on the second feature map, the sixth reference feature vector, and the ninth reference feature vector to generate the specific feature vector.

24. The electronic device according to claim 13, wherein, The multi-dimensional data includes waveform images drawn based on the first data string.

25. A computer-readable storage medium having recorded an executable computer program, the executable computer program being used for the detection of abnormalities in electrocardiogram signals, the executable computer program being loaded by an electronic device to perform the following steps: Obtain the first data string, which is a one-dimensional electrocardiogram signal; The first data string is fed into a first deep neural network, wherein... The first deep neural network generates a first feature map by extracting features from the first data string; Multi-dimensional data is generated based on the first data string, wherein the dimension of the multi-dimensional data is higher than the dimension of the first data string, and the multi-dimensional data is an image obtained by conversion based on the first data string; The multi-dimensional data is fed into a second deep neural network, wherein the second deep neural network generates a second feature map by extracting features from the multi-dimensional data; The method of fusing at least the first feature map and the second feature map into a specific feature vector includes: feeding the first feature map into a third convolutional neural network, wherein the third convolutional neural network outputs a first reference feature vector in response to the first feature map; stacking multiple first reference feature vectors into a second reference feature vector according to the size of the second feature map, wherein the size of the second feature map is the same as the size of the second reference feature vector; converting the second reference feature vector into a third reference feature vector; and generating the specific feature vector based on the second feature map and the third reference feature vector. The specific feature vector is fed into a machine learning model, wherein the machine learning model outputs an identification result corresponding to the first data string in response to the specific feature vector, and the identification result indicates that the first data string is suitable for clinical decision-making.

Images