Decision prediction method and device based on multimodal physiological signals

Abstract

The present application provides a decision prediction method and device based on multimodal physiological signals, the method comprising obtaining EEG data, ECG data and skin conduction data of a decision maker when making a decision; inputting the EEG data, ECG data and skin conduction data into a decision prediction model, and outputting decision prediction features, wherein the decision prediction features represent the decision confidence of the decision maker after making a decision, and the decision prediction model is trained based on multimodal physiological signals.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a decision prediction method and a decision prediction device based on multimodal physiological signals. Background Art

[0002] As an important factor affecting decision-making, emotions have received extensive attention in decision-making research in recent years. Lerner et al. pointed out that emotions constitute a powerful, universal, predictable, sometimes harmful, and sometimes beneficial driver of decision-making. Therefore, the introduction of emotions has become crucial in the study of decision prediction models. Summary of the invention

[0003] In view of this, the present application provides a decision prediction method and a decision prediction device based on multimodal physiological signals.

[0004] One aspect of the present application provides a decision prediction method based on multimodal physiological signals, comprising:

[0005] Obtaining the EEG data, ECG data and skin conductance data of the decision maker when making a decision;

[0006] The EEG data, the ECG data and the galvanodermal data are input into a decision prediction model, and a decision prediction feature is output, wherein the decision prediction feature represents the decision confidence of the decision maker after making a decision, and the decision prediction model is trained based on multimodal physiological signals.

[0007] According to an embodiment of the present application, the above-mentioned EEG data, the above-mentioned ECG data and the above-mentioned skin electrical data are input into a decision prediction model, and a decision prediction feature is output, including:

[0008] The above EEG data are processed using a multi-scale graph convolutional network to obtain target EEG features;

[0009] The target neural network is used to process the above-mentioned ECG data and the above-mentioned skin electrical data to obtain target ECG features and target skin electrical features;

[0010] Using a graph residual fusion network, multimodal fusion processing is performed on the target EEG features, the target ECG features, and the target electrodermal features to obtain multimodal physiological features;

[0011] The above-mentioned multimodal physiological characteristics are processed using a multi-task learning network to obtain the above-mentioned decision prediction characteristics.

[0012] According to an embodiment of the present application, a multi-scale graph convolutional network is used to process the above-mentioned EEG acquisition data to obtain target EEG features, including:

[0013] The EEG data are processed using multiple one-dimensional convolutional layers of different scales to obtain multiple multi-scale power features;

[0014] A plurality of the multi-scale power features are fused and spliced ​​to obtain an EEG splicing feature, and a graph structure feature is generated based on the EEG splicing feature;

[0015] The graph structure features are processed by the graph structure filtering layer to generate the target EEG features.

[0016] According to an embodiment of the present application, the above-mentioned graph structure feature includes a plurality of local graph features;

[0017] The above target EEG features are generated by processing a plurality of the above graph structure features using a graph structure filter layer, including:

[0018] For each of the local image features, the local image feature is processed using a local image filter layer to obtain a local filter feature;

[0019] The global graph filter layer is used to process multiple local filter features to obtain the target EEG features.

[0020] According to an embodiment of the present application, the target neural network is used to process the above-mentioned electrocardiogram acquisition data and the above-mentioned skin electrical acquisition data respectively to obtain target electrocardiogram features and target skin electrical features, including:

[0021] For any physiological data among the electrocardiogram data and the electrodermal electrical data, extract recursive graph features, Markov transition field features and Gram matrix angular domain features from the physiological data based on a preset extraction formula;

[0022] Arrange the recursive graph feature, the Markov transition field feature, and the Gram matrix angular domain feature based on multiple color channels to obtain a channel image feature;

[0023] The target residual network is used to process the channel image features to obtain the target electrocardiogram features or the target skin electrical features.

[0024] According to an embodiment of the present application, recursive graph features, Markov transition field features and Gram matrix angular domain features are extracted from the above physiological acquisition data based on preset extraction formulas, including:

[0025] Extracting the recursive graph feature from the physiological acquisition data based on a first preset extraction formula;

[0026] Extracting the Markov transfer field feature from the physiological acquisition data based on a second preset extraction formula;

[0027] The Gramma matrix angular domain features are extracted from the physiological acquisition data based on a third preset extraction formula.

[0028] According to an embodiment of the present application, a graph residual fusion network is used to perform multimodal fusion processing on the target EEG features, the target ECG features, and the target skin electrical features to obtain multimodal physiological features, including:

[0029] Using the first image residual fusion network to perform multimodal fusion processing on the three single-modal features, three bimodal features are obtained, wherein the three single-modal features include the target EEG feature, the target ECG feature and the target skin electrical feature;

[0030] Using the second image residual fusion network, multimodal fusion processing is performed on the target EEG feature, the target ECG feature, the target skin electrical feature and the three bimodal features to obtain six trimodal features;

[0031] The global attention network is used to process the three unimodal features, the three bimodal features and the six trimodal features respectively to obtain unimodal fusion features, bimodal fusion features and trimodal fusion features;

[0032] The single-modal fusion feature, the above-mentioned dual-modal fusion feature and the above-mentioned tri-modal fusion feature are subjected to feature splicing processing to obtain the above-mentioned multi-modal physiological feature.

[0033] According to an embodiment of the present application, the above decision prediction model is trained in the following manner:

[0034] In response to the model training instruction, a multimodal training set is obtained, wherein the multimodal training set includes a plurality of training sample combinations, and the training sample combinations include EEG data samples, ECG data samples, electrodermal conductivity data samples, emotion labels, and decision labels when training users to make decisions;

[0035] For each of the above training sample combinations, the above EEG data samples are processed using a multi-scale initial graph convolutional network to obtain training EEG features;

[0036] Using the initial neural network to process the ECG data samples and the skin electrical data samples respectively, to obtain training ECG features and training skin electrical features;

[0037] Using the initial graph residual fusion network, the training EEG features, the training ECG features and the training skin electrical features are subjected to multimodal fusion processing to obtain multimodal physiological training features;

[0038] Using an initial multi-task learning network to process the multimodal physiological training features to obtain decision features and emotional features, wherein the initial prediction model includes the multi-scale initial graph convolutional network, the initial neural network, the initial graph residual fusion network, and the initial multi-task learning network;

[0039] The network parameters of the initial prediction model are iteratively adjusted according to the decision features, the emotion features, the emotion labels and the decision labels to obtain the decision prediction model.

[0040] According to an embodiment of the present application, the network parameters of the initial prediction model are iteratively adjusted according to the decision features, the emotion features, the emotion labels and the decision labels to obtain the decision prediction model, including:

[0041] According to the above decision features and the above decision labels, a decision cross entropy loss is generated;

[0042] Generate emotional smoothed cross entropy loss based on the emotional features and emotional labels;

[0043] Generate a decision prediction loss value based on the above decision cross entropy loss and the above sentiment smoothed cross entropy loss;

[0044] The network parameters of the initial prediction model are iteratively adjusted according to the decision prediction loss value to obtain the decision prediction model.

[0045] Another aspect of the present application provides a decision prediction device based on multimodal physiological signals, comprising:

[0046] The first acquisition module is used to acquire the EEG data, ECG data and skin conduction data of the decision maker when making a decision;

[0047] A prediction module is used to input the above-mentioned EEG data, the above-mentioned ECG data and the above-mentioned skin conductance data into a decision prediction model, and output decision prediction features, wherein the above-mentioned decision prediction features represent the decision confidence of the above-mentioned decision maker after making a decision, and the above-mentioned decision prediction model is trained based on multimodal physiological signals.

[0048] Another aspect of the present application provides an electronic device, comprising:

[0049] one or more processors;

[0050] a memory for storing one or more programs,

[0051] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0052] Another aspect of the present application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the method described above when executed.

[0053] Another aspect of the present application provides a computer program product, which includes computer executable instructions, and the instructions are used to implement the method as described above when executed.

[0054] According to an embodiment of the present application, the EEG data, the ECG data and the skin electrode data are processed by a multi-scale graph convolution network and a target neural network to obtain target EEG features, target ECG features and target skin electrode features, and a graph residual fusion network is used to perform multimodal fusion processing on the target EEG features, target ECG features and target skin electrode features to obtain multimodal physiological features, thereby using a multi-task learning network to process the multimodal physiological features to obtain decision prediction features. By fully integrating the features of the three modalities of EEG, ECG and skin electrode, complementary information between different modalities is extracted, thereby improving the accuracy of predicting whether the decision maker makes the right decision. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0056] Figure 1 An exemplary system architecture to which a decision prediction method based on multimodal physiological signals can be applied according to an embodiment of the present application is shown;

[0057] Figure 2 A flowchart of a decision prediction method based on multimodal physiological signals according to an embodiment of the present application is shown;

[0058] Figure 3 A flow chart of a method for generating decision prediction features according to a decision prediction model according to an embodiment of the present application is shown;

[0059] Figure 4 A schematic diagram of the model structure of a decision prediction model according to an embodiment of the present application is shown;

[0060] Figure 5 A block diagram of a decision prediction device according to an embodiment of the present application is shown;

[0061] Figure 6 A block diagram of a training device for a decision prediction model according to an embodiment of the present application is shown;

[0062] Figure 7 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0063] Below, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present application.

[0064] The terms used herein are only for describing specific embodiments and are not intended to limit the present application. The terms "include", "comprising", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0065] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0066] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0067] Most current decision prediction models based on physiological signals lack the introduction of emotions and often use single-modal physiological signal data, such as EEG, ECG, and skin conduction signals. Although single-modal decision prediction models can predict human decisions, they cannot fully integrate the complementary information between multimodal physiological signals. In addition, the complexity, effectiveness, and specificity of psychophysiological reasoning have hindered the development of physiological computing. Most physiological signals exhibit complex dynamics, making it challenging to establish a one-to-one correspondence with psychological constructs. Considering these challenges, it is necessary to consider multimodal fusion methods to fully integrate information from each modality and achieve better performance in decision behavior prediction.

[0068] In view of this, an embodiment of the present application provides a decision prediction method and a decision prediction device based on multimodal physiological signals, the method comprising obtaining EEG data, ECG data and galvanic skin data of a decision maker when making a decision; inputting the EEG data, ECG data and galvanic skin data into a decision prediction model, and outputting decision prediction features, wherein the decision prediction features represent the decision confidence of the decision maker after making a decision, and the decision prediction model is trained based on multimodal physiological signals.

[0069] In the embodiments of this application, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information and network security.

[0070] In the embodiments of the present application, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0071] Figure 1 FIG. 1 shows an exemplary system architecture 100 to which a decision prediction method based on multimodal physiological signals can be applied according to an embodiment of the present application. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present application can be applied, in order to help those skilled in the art understand the technical content of the present application, but it does not mean that the embodiments of the present application cannot be used in other devices, systems, environments or scenarios.

[0072] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, a server 105, and a data acquisition device 106. The network 104 is used to provide a medium for communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, the server 105, and the data acquisition device 106. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0073] The user may use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only for example).

[0074] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0075] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0076] The data acquisition device 106 may be an electroencephalogram (EEG) acquisition device, an electrocardiogram (ECG) acquisition device, or an electrical skin acquisition device.

[0077] It should be noted that the decision prediction method provided in the embodiment of the present application can generally be executed by the server 105. Accordingly, the decision prediction device provided in the embodiment of the present application can generally be arranged in the server 105. The decision prediction method provided in the embodiment of the present application can also be executed by a server or server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the decision prediction device provided in the embodiment of the present application can also be arranged in a server or server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Alternatively, the decision prediction method provided in the embodiment of the present application can also be performed by the first terminal device 101, the second terminal device 102 or the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103. Correspondingly, the decision prediction device provided in the embodiment of the present application can also be set in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0078] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only . According to the implementation requirements, there can be any number of terminal devices, networks and servers.

[0079] Figure 2 A flowchart of a decision prediction method based on multimodal physiological signals according to an embodiment of the present application is shown.

[0080] like Figure 2 As shown, the decision prediction method based on multimodal physiological signals includes operations S201-S202.

[0081] In operation S201, the EEG data, the ECG data, and the galvanic skin data of the decision maker when making a decision are obtained;

[0082] In operation S202, the EEG data, the ECG data and the galvanodermal data are input into a decision prediction model, and a decision prediction feature is output, wherein the decision prediction feature represents the decision confidence of the decision maker after making a decision, and the decision prediction model is trained based on multimodal physiological signals.

[0083] According to an embodiment of the present application, the decision confidence may represent the correctness or correctness rate of the decision.

[0084] According to the embodiments of the present application, when the decision maker is making a decision, an EEG acquisition device can be used to collect his EEG data, ECG data and galvanic skin data. Thereafter, the collected EEG data, ECG data and galvanic skin data are input into a trained decision prediction model, and whether the decision made is correct can be predicted based on the decision prediction features.

[0085] According to the embodiments of the present application, the EEG data, ECG data and skin electrical data are processed by a multi-scale graph convolution network and a target neural network to obtain target EEG features, target ECG features and target skin electrical features, and the target EEG features, target ECG features and target skin electrical features are subjected to multimodal fusion processing by a graph residual fusion network to obtain multimodal physiological features, and the multimodal physiological features are processed by a multi-task learning network to obtain decision prediction features. By fully integrating the features of the three modalities of EEG, ECG and skin electrical, complementary information between different modalities is extracted, thereby improving the accuracy of predicting whether the decision maker makes the right decision or not.

[0086] Figure 3 A flow chart of a method for generating decision prediction features according to a decision prediction model according to an embodiment of the present application is shown.

[0087] like Figure 3 As shown, the method for generating decision prediction features based on a decision prediction model of multimodal physiological signals includes operations S301 to S304.

[0088] In operation S301, the EEG data is processed using a multi-scale graph convolutional network to obtain target EEG features;

[0089] In operation S302, the target neural network is used to process the ECG data and the skin electrical data to obtain target ECG features and target skin electrical features.

[0090] In operation S303, a multimodal fusion process is performed on the target EEG feature, the target ECG feature, and the target electrocutaneous feature using a graph residual fusion network to obtain a multimodal physiological feature;

[0091] In operation S304, the multi-modal physiological features are processed using a multi-task learning network to obtain decision prediction features.

[0092] According to the embodiment of the present application, the EEG data is first processed using a multi-scale graph convolutional network to obtain the target EEG feature F EEG Then, the target neural network is used to process the ECG data and skin electrical data respectively to obtain the target ECG feature F ECG and target skin electrical characteristics F EDA . The target EEG feature F is fused using the graph residual network. EEG , target ECG feature F ECG and target skin electrical characteristics F EDA Perform multimodal fusion processing to obtain multimodal physiological features F Multi .

[0093] According to an embodiment of the present application, a multi-task learning network is used to process multi-modal physiological features F Multi , and obtain decision prediction features. The multi-task learning network can be at least two fully connected networks (FCs), or at least two CNNs or graph convolutional networks, etc.

[0094] Figure 4 A schematic diagram of the model structure of a decision prediction model according to an embodiment of the present application is shown.

[0095] According to the embodiments of the present application, Figure 4 As shown in the figure, a multi-scale graph convolutional network is used to process EEG data to obtain target EEG features, including:

[0096] Multiple one-dimensional convolutional layers of different scales are used to process EEG data to obtain multiple multi-scale power features;

[0097] Multiple multi-scale power features are fused and spliced ​​to obtain EEG splicing features, and graph structure features are generated based on the EEG splicing features;

[0098] The graph structure filtering layer is used to process the graph structure features and generate the target EEG features.

[0099] According to the embodiment of the present application, one-dimensional convolution layers of different scales are used to extract multi-scale power features, and then the above features are spliced ​​to obtain spliced ​​features F1. F1 is fused through a kernel-based attention feature fusion module to obtain spliced ​​features F2. F2 is rearranged to obtain EEG spliced ​​features EEG T Then the EEG feature is spliced TThe graph structure is represented to obtain the graph structure feature G1, which is processed by the graph structure filter layer to generate the target EEG feature F EEG .

[0100] According to an embodiment of the present application, the steps included in the above-mentioned kernel-based attention feature fusion module are batch normalization, 1×1 convolution, Leaky ReLU, average pooling and batch normalization in sequence.

[0101] According to an embodiment of the present application, the graph structure feature includes a plurality of local graph features.

[0102] According to the embodiments of the present application, Figure 4 As shown in the figure, the graph structure filtering layer is used to process multiple graph structure features to generate target EEG features, including:

[0103] For each local image feature, the local image feature is processed using a local image filter layer to obtain a local filter feature;

[0104] The global graph filter layer is used to process multiple local filter features to obtain the target EEG features.

[0105] According to an embodiment of the present application, the local image feature is processed using the local image filter layer to obtain the local filter feature G2. The specific calculation is shown in formulas (1) and (2):

[0106] (1)

[0107] (2)

[0108] in, and are the trainable local graph filter matrix and the local graph filter bias vector, ◦ is the Hadamard product, is the ReLU activation function, is an aggregation function used to obtain the hidden embedding of the local graph (i.e., the local graph feature), is a graphical representation of local filtering, Z1,...,Z R is the local graph representation, R is the total number of local graphs, h local is the latent representation of the local graph.

[0109] According to the embodiment of the present application, the global graph filter layer is intended to learn the complex relationship between local graphs, so the global graph filter layer can be used to process multiple local filter features to obtain the target EEG feature F EEG , the specific calculation is shown in formula (3):

[0110] The global graph filter layer aims to learn the complex relationship between local graphs. Its calculation formula is as follows:

[0111] (3)

[0112] in, is the basic adjacency matrix of the global graph, is a trainable attention mask, is the final global adjacency matrix, is the normalized adjacency matrix, where ,yes Finally, the result of global graph filtering is , and this feature is represented as the target EEG feature F EEG .

[0113] According to the embodiments of the present application, Figure 4 As shown, the target neural network is used to process the ECG data and the skin electrical data respectively to obtain the target ECG features and the target skin electrical features, including:

[0114] For any physiological data among the electrocardiogram data and the electrodermal electrodermal data, recursive graph features, Markov transition field features and Gram matrix angular domain features are respectively extracted from the physiological data based on a preset extraction formula;

[0115] Based on multiple color channels, the recursive graph features, the Markov transfer field features and the Gram matrix corner domain features are arranged and processed to obtain channel image features;

[0116] The target residual network is used to process the channel image features to obtain the target electrocardiogram features or target skin electrical features.

[0117] According to an embodiment of the present application, the processing process of the electrocardiogram data sample and the electrodermal conductivity data sample is the same, and the electrocardiogram data sample is used as an example for illustration.

[0118] According to the embodiment of the present application, the recursive graph feature ECG is extracted from the electrocardiogram data based on the preset extraction formula. RP , Markov transition field characteristics ECG MTF and Graham matrix angular domain characteristics ECG GAF .

[0119] According to an embodiment of the present application, based on the above three 2D image features ECG RP , ECG MTF , ECG GAF , and arrange them according to the three channels of the image RGB to obtain the final ECG feature, that is, the channel image feature G ECG , the specific calculation is shown in formula (4):

[0120] G ECG = (ECG RP , ECG MTF , ECG GAF )(4)

[0121] Based on the channel image feature G ECG , and use it as the input of the target residual network ResNet18 to obtain the target ECG feature F ECG .

[0122] According to the embodiment of the present application, similarly, for the skin electrical collection data, the recursive graph feature EDA is first calculated RP , Markov transition field feature EDA MTF and Gram Matrix Corner Domain EDA GAF Based on formula (4), the channel image feature G of the skin electrical acquisition data is obtained EDA Finally, the target residual network ResNet18 is used to process G EDA The target skin electrical feature F can be obtained EDA .

[0123] According to an embodiment of the present application, recursive graph features, Markov transition field features, and Gram matrix angular domain features are extracted from physiological acquisition data based on preset extraction formulas, including:

[0124] Extracting recursive graph features from the physiological acquisition data based on a first preset extraction formula;

[0125] extracting Markov transfer field features from the physiological acquisition data based on a second preset extraction formula;

[0126] The Gram matrix angular domain features are extracted from the physiological acquisition data based on the third preset extraction formula.

[0127] According to an embodiment of the present application, the ECG data is used as an example to illustrate the recursive graph feature ECG of the ECG data ECG2={s1, s2, s3…sN}. RP , can be obtained by using the first preset extraction formula as shown in formula (5):

[0128] [ECG RP ] i, j =F resize (Heaviside(||s i -s j || / eps)), i=1N, j=1,...,N.

[0129] (5)

[0130] Among them, eps is used to scale the distance, steps is the heaviside threshold, for example 10, F resize A function that compresses an image to a specified size.

[0131] According to an embodiment of the present application, the Markov transition field feature ECG of the electrocardiogram acquisition data ECG2 is extracted using the second preset extraction formula shown in formula (6): MTF :

[0132] (6)

[0133] Here, ECG2 is divided into Q quantiles with an interval of qk, and Wlk is the transition frequency of the points between two quantiles.

[0134] According to an embodiment of the present application, the Gram matrix angular domain feature ECG of the ECG2 is extracted. GAF Specifically, first, ECG2 is converted into polar coordinates (R, β) using formula (7):

[0135] (7)

[0136] in Yes k The maximum and minimum normalized results, t k Yes k The timestamp of the polar coordinates, C is the polar coordinate adjustment constant, k=1,...,N. Based on this polar coordinate, the third preset extraction formula shown in formula (8) is used to calculate the Gram matrix angular domain feature ECG GAF :

[0137] (8)

[0138] in, is the result of the maximum and minimum value normalization of ECG2, and I is a unit row vector.

[0139] According to an embodiment of the present application, a graph residual fusion network is used to perform multimodal fusion processing on target EEG features, target ECG features, and target electrodermal features to obtain multimodal physiological features, including:

[0140] The first image residual fusion network is used to perform multimodal fusion processing on the three single-modal features to obtain three bimodal features, wherein the three single-modal features include target EEG features, target ECG features and target skin electrical features;

[0141] The second image residual fusion network is used to perform multimodal fusion processing on the target EEG features, target ECG features, target skin electrical features and three bimodal features to obtain six trimodal features;

[0142] The global attention network is used to process three unimodal features, three bimodal features and six trimodal features respectively to obtain unimodal fusion features, bimodal fusion features and trimodal fusion features;

[0143] The single-modal fusion features, the dual-modal fusion features and the tri-modal fusion features are processed by feature splicing to obtain multi-modal physiological features.

[0144] According to the embodiment of the present application, based on the single modal features extracted from the above three modalities, that is, the target EEG feature F EEG , target ECG feature F ECG , target skin electrical characteristics F EDA , the first image residual fusion network is used to fuse them two by two to obtain the bimodal feature F EEG-ECG 、F EEG-EDA 、F ECG-EDA .

[0145] According to an embodiment of the present application, the second graph residual fusion network is used to transform the bimodal feature F EEG-ECG 、F EEG-EDA 、F ECG-EDA Perform pairwise fusion and transform F EEG-ECG With F EDA 、F EEG-EDA With F ECG 、F ECG-EDA With F EEG Perform pairwise fusion to obtain the trimodal feature F ECED 、F ECD 、F ECCD 、F EDC 、F EDCD 、F CDE .

[0146] According to an embodiment of the present application, a global attention network is used to process three unimodal target EEG features F EEG , target ECG feature F ECG , target skin electrical characteristics F EDA , and obtain the single-modal fusion feature. At the same time, the global attention network is used to process F EEG-ECG 、F EEG-EDA 、F ECG-EDA Obtain bimodal fusion features. Then use the global attention network to process the trimodal features F ECED 、F ECD 、F ECCD 、F EDC 、F EDCD 、F CDE The three-modal fusion features are obtained.

[0147] According to the embodiment of the present application, the single-modal fusion feature, the dual-modal fusion feature and the tri-modal fusion feature are subjected to feature splicing processing to obtain a multi-modal physiological feature F Multi .

[0148] According to an embodiment of the present application, the first graph residual fusion network GRF Layer1 and the second graph residual fusion network GRF Layer2. When processing, the graph residual fusion network GRF Layer first concatenates the input features, and then uses the graph residual fusion network to process them. For example, when processing F EEG and F ECG When fusing, the two are initially spliced, and the spliced ​​features are used as the input of the graph residual fusion network, so that F EEG-ECG .

[0149] According to an embodiment of the present application, a training method for a decision prediction model includes:

[0150] In response to the model training instruction, a multimodal training set is obtained, wherein the multimodal training set includes a plurality of training sample combinations, and the training sample combinations include EEG data samples, ECG data samples, electrodermal conductivity data samples, emotion labels, and decision labels when training users to make decisions;

[0151] For each training sample combination, the EEG data samples are processed using a multi-scale initial graph convolutional network to obtain training EEG features;

[0152] The initial neural network is used to process the ECG data samples and the skin electrical data samples respectively to obtain the training ECG features and the training skin electrical features;

[0153] The initial graph residual fusion network is used to perform multimodal fusion processing on the training EEG features, training ECG features and training skin electrical features to obtain multimodal physiological training features;

[0154] The initial multi-task learning network is used to process multimodal physiological training features to obtain decision features and emotional features, wherein the initial prediction model includes a multi-scale initial graph convolutional network, an initial neural network, an initial graph residual fusion network, and an initial multi-task learning network;

[0155] The network parameters of the initial prediction model are iteratively adjusted according to the decision features, emotion features, emotion labels and decision labels to obtain a decision prediction model.

[0156] According to an embodiment of the present application, a combination of multiple training samples can be physiological signals collected when one or more decision makers make decisions in at least one decision task. Model training instructions can be automatically generated by electronic devices in response to the operation performed by staff on mobile phones, computers and other electronic devices. Emotional labels represent the emotions of decision makers when making decisions, specifically in the form of information such as the category of emotions aroused or the subjective score of emotions. Emotions can be happy, neutral, sad, fearful, angry, etc. The decision label is the decision confidence of information such as the accuracy of a single trial of the decision under each emotion or the options selected in a single trial. For example, the correctness of the decision can be represented by 0 or 1 to indicate whether the decision is correct or wrong, or it can be represented by a percentage.

[0157] It should be noted that the emotion stimulation stage uses stimulation methods such as movies, pictures, and music to stimulate the decision maker to feel emotions such as happiness, neutrality, sadness, fear, and anger. The decision-making task stage can be multi-trial spatial decision-making, perceptual decision-making, and other tasks.

[0158] According to an embodiment of the present application, after obtaining multiple training sample combinations, for each training sample combination, the EEG data samples are first processed using a multi-scale initial graph convolutional network to obtain training EEG features, and then the ECG data samples and the skin electrical data samples are processed using an initial neural network to obtain training ECG features and training skin electrical features. The initial graph residual fusion network is used to perform multimodal fusion processing on the training EEG features, training ECG features, and training skin electrical features to obtain multimodal physiological training features.

[0159] According to an embodiment of the present application, the initial multi-task learning network is used to process the multimodal physiological training features to obtain the decision feature f dec and emotional characteristics emo Therefore, according to the decision feature f dec and emotional characteristics emo The loss values ​​of the emotion label and decision label calculation model are then adjusted based on the loss values ​​to finally obtain a trained decision prediction model. The initial multi-task learning network can be at least two fully connected networks (FCs), or at least two CNNs or graph convolutional networks, etc.

[0160] According to an embodiment of the present application, the network parameters of the initial prediction model are iteratively adjusted according to the decision features, the emotion features, the emotion labels and the decision labels to obtain the decision prediction model, including:

[0161] Generate decision cross entropy loss based on decision features and decision labels;

[0162] Generate emotional smoothed cross entropy loss based on emotional features and emotional labels;

[0163] Generate decision prediction loss value based on decision cross entropy loss and sentiment smooth cross entropy loss;

[0164] The network parameters of the initial prediction model are iteratively adjusted according to the decision prediction loss value to obtain the decision prediction model.

[0165] According to an embodiment of the present application, based on the fused multimodal physiological training features, the decision features f are extracted through the initial multi-task learning network. dec and emotional characteristics emo .f dec and f emo Calculate the decision cross entropy loss L based on the decision label and sentiment label respectively dec and sentiment smoothed cross entropy loss L emo . Calculate the decision prediction loss value L = αL dec +βL emo , the decision prediction is achieved by minimizing the decision prediction loss value L. Smooth cross entropy takes into account the possibility of multiple crossovers in emotions, thereby effectively improving the model performance.

[0166] According to an embodiment of the present application, minimizing the decision prediction loss value L can be determined by setting a threshold.

[0167] According to the embodiments of the present application, EEG data samples, ECG data samples, and electrodermal data samples are processed by a multi-scale graph initial convolutional network and an initial neural network to obtain training EEG features, training ECG features, and training electrodermal features, and the initial graph residual fusion network is used to perform multimodal fusion processing on the training EEG features, training ECG features, and training electrodermal features to obtain multimodal physiological training features, thereby using the initial multi-task learning network to process the multimodal physiological training features to obtain decision features and emotional features, and iterative adjustments are made based on the decision features, emotional features, emotional labels, and decision labels to obtain a decision prediction model. By fully integrating the features of the three modalities of EEG, ECG, and electrodermal, complementary information between different modalities is extracted, thereby improving the accuracy of the decision prediction model.

[0168] In a specific embodiment, the prediction method of the present application is more than 3% higher in accuracy than the single-modal method MeCM-EDNet, as shown in Table 1.

[0169] Table 1

[0170]

[0171] Figure 5 A block diagram of a decision prediction device according to an embodiment of the present application is shown.

[0172] like Figure 5As shown, the decision prediction device 500 based on multimodal physiological signals includes a first acquisition module 510 and a prediction module 520 .

[0173] The first acquisition module 510 is used to acquire the EEG data, ECG data and skin conduction data of the decision maker when making a decision;

[0174] The prediction module 520 is used to input the EEG data, the ECG data and the electrodermal data into the decision prediction model, and output the decision prediction features, wherein the decision prediction features represent the decision confidence of the decision maker after making the decision, and the decision prediction model is trained based on multimodal physiological signals.

[0175] According to the embodiments of the present application, the EEG data, ECG data and skin electrical data are processed by a multi-scale graph convolution network and a target neural network to obtain target EEG features, target ECG features and target skin electrical features, and the target EEG features, target ECG features and target skin electrical features are subjected to multimodal fusion processing by a graph residual fusion network to obtain multimodal physiological features, and the multimodal physiological features are processed by a multi-task learning network to obtain decision prediction features. By fully integrating the features of the three modalities of EEG, ECG and skin electrical, complementary information between different modalities is extracted, thereby improving the accuracy of predicting whether the decision maker makes the right decision or not.

[0176] Figure 6 A block diagram of a training device for a decision prediction model according to an embodiment of the present application is shown.

[0177] like Figure 6 As shown, the training device 600 of the decision prediction model based on multimodal physiological signals includes a second acquisition module 610, a convolution module 620, a first obtaining module 630, a fusion module 640, a second obtaining module 650 and an adjustment module 660.

[0178] A second acquisition module 610 is used to acquire a multimodal training set in response to a model training instruction, wherein the multimodal training set includes a plurality of training sample combinations, and the training sample combinations include EEG data samples, ECG data samples, electrodermal conductivity data samples, emotion labels, and decision labels when training users to make decisions;

[0179] A graph convolution module 620 is used to process the EEG data samples using a multi-scale initial graph convolution network for each training sample combination to obtain training EEG features;

[0180] A first obtaining module 630 is used to process the ECG data samples and the electrodermal conductivity data samples respectively using the initial neural network to obtain training ECG features and training electrodermal conductivity features;

[0181] A fusion module 640 is used to perform multimodal fusion processing on the training EEG features, the training ECG features and the training skin electrical features using the initial graph residual fusion network to obtain multimodal physiological training features;

[0182] A second obtaining module 650 is used to process multimodal physiological training features using an initial multi-task learning network to obtain decision features and emotional features, wherein the initial prediction model includes a multi-scale initial graph convolutional network, an initial neural network, an initial graph residual fusion network, and an initial multi-task learning network;

[0183] The adjustment module 660 is used to iteratively adjust the network parameters of the initial prediction model according to the decision features, emotion features, emotion labels and decision labels to obtain a decision prediction model.

[0184] According to the embodiments of the present application, EEG data samples, ECG data samples, and electrodermal data samples are processed by a multi-scale graph initial convolutional network and an initial neural network to obtain training EEG features, training ECG features, and training electrodermal features, and the initial graph residual fusion network is used to perform multimodal fusion processing on the training EEG features, training ECG features, and training electrodermal features to obtain multimodal physiological training features, thereby using the initial multi-task learning network to process the multimodal physiological training features to obtain decision features and emotional features, and iterative adjustments are made based on the decision features, emotional features, emotional labels, and decision labels to obtain a decision prediction model. By fully integrating the features of the three modalities of EEG, ECG, and electrodermal, complementary information between different modalities is extracted, thereby improving the accuracy of the decision prediction model.

[0185] According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present application, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present application, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be executed.

[0186] For example, any of the first acquisition module 510, the prediction module 520, or the second acquisition module 610, the convolution module 620, the first acquisition module 630, the fusion module 640, the second acquisition module 650, and the adjustment module 660 can be combined in one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present application, at least one of the first acquisition module 510, the prediction module 520, or the second acquisition module 610, the convolution module 620, the first acquisition module 630, the fusion module 640, the second acquisition module 650 and the adjustment module 660 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by an appropriate combination of any of them. Alternatively, at least one of the first acquisition module 510, the prediction module 520, or the second acquisition module 610, the convolution module 620, the first acquisition module 630, the fusion module 640, the second acquisition module 650 and the adjustment module 660 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0187] It should be noted that the training device or the decision prediction device part of the decision prediction model in the embodiments of the present application corresponds to the training method or the decision prediction method part of the decision prediction model in the embodiments of the present application. The description of the training device or the decision prediction device part of the decision prediction model specifically refers to the training method or the decision prediction method part of the decision prediction model, which will not be repeated here.

[0188] Figure 7 A block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present application is shown. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0189] like Figure 7As shown, the electronic device 700 according to an embodiment of the present application includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 to a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present application.

[0190] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present application by executing the program in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present application by executing the program stored in the one or more memories.

[0191] According to an embodiment of the present application, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the input / output (I / O) interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0192] According to an embodiment of the present application, the method flow according to the embodiment of the present application can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above-mentioned functions defined in the system of the embodiment of the present application are executed. According to an embodiment of the present application, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0193] The present application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present application is implemented.

[0194] According to an embodiment of the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0195] For example, according to an embodiment of the present application, the computer-readable storage medium may include the ROM 702 and / or the RAM 703 described above and / or one or more memories other than the ROM 702 and the RAM 703 .

[0196] An embodiment of the present application also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present application. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiment of the present application.

[0197] When the computer program is executed by the processor 701, the above functions defined in the system / device of the embodiment of the present application are executed. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0198] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0199] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages, and specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or completely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0200] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It will be appreciated by those skilled in the art that the features recorded in the various embodiments of the present application can be combined and / or combined in a variety of ways, even if such a combination or combination is not clearly recorded in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments of the present application may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present application.

[0201] The embodiments of the present application are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present application. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present application, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present application.

Claims

1. A decision prediction method based on multimodal physiological signals, characterized in that: include: Obtaining the EEG data, ECG data and skin conductance data of the decision maker when making a decision; Inputting the EEG data, the ECG data and the electrodermal data into a decision prediction model, and outputting a decision prediction feature, wherein the decision prediction feature represents the decision confidence of the decision maker after making a decision, and the decision prediction model is trained based on multimodal physiological signals; The EEG data, the ECG data and the galvanic skin data are input into a decision prediction model, and decision prediction features are output, including: Processing the EEG data using a multi-scale graph convolutional network to obtain target EEG features; For any physiological acquisition data among the electrocardiogram acquisition data and the electrodermal electrodermal acquisition data, recursive graph features, Markov transfer field features and Gram matrix angular domain features are respectively extracted from the physiological acquisition data based on a preset extraction formula; the recursive graph features, the Markov transfer field features and the Gram matrix angular domain features are arranged and processed based on multiple color channels to obtain channel image features; the channel image features are processed using a target residual network to obtain a target electrocardiogram feature or a target electrodermal electrodermal feature; Using a graph residual fusion network to perform multimodal fusion processing on the target EEG feature, the target ECG feature and the target electrodermal feature to obtain a multimodal physiological feature; The multi-modal physiological features are processed using a multi-task learning network to obtain the decision prediction features.

2. The method according to claim 1, characterized in that The EEG data is processed using a multi-scale graph convolutional network to obtain target EEG features, including: Processing the EEG data using multiple one-dimensional convolutional layers of different scales to obtain multiple multi-scale power features; Performing fusion and splicing processing on a plurality of the multi-scale power features to obtain an EEG splicing feature, and generating a graph structure feature based on the EEG splicing feature; The graph structure feature is processed by using a graph structure filter layer to generate the target EEG feature.

3. The method according to claim 2, characterized in that The graph structure feature includes a plurality of local graph features; The process of processing the plurality of graph structure features using a graph structure filter layer to generate the target EEG features includes: For each of the local image features, use a local image filter layer to process the local image feature to obtain a local filter feature; The target EEG feature is obtained by processing the plurality of local filter features using a global graph filter layer.

4. The method according to claim 1, characterized in that: Recursive graph features, Markov transition field features and Gram matrix angular domain features are extracted from the physiological acquisition data based on preset extraction formulas, including: Extracting the recursive graph feature from the physiological acquisition data based on a first preset extraction formula; Extracting the Markov transfer field feature from the physiological acquisition data based on a second preset extraction formula; The Graham matrix angular domain feature is extracted from the physiological acquisition data based on a third preset extraction formula.

5. The method according to claim 1, characterized in that The target EEG feature, the target ECG feature and the target electrocutaneous feature are subjected to multimodal fusion processing by using a graph residual fusion network to obtain multimodal physiological features, including: Using the first image residual fusion network to perform multimodal fusion processing on the three single-modal features to obtain three bimodal features, wherein the three single-modal features include the target EEG feature, the target ECG feature, and the target skin conductance feature; Using the second image residual fusion network, multimodal fusion processing is performed on the target EEG feature, the target ECG feature, the target skin electrical feature and the three bimodal features to obtain six trimodal features; Using a global attention network to process the three unimodal features, the three bimodal features, and the six trimodal features, respectively, to obtain unimodal fusion features, bimodal fusion features, and trimodal fusion features; The single-modal fusion feature, the dual-modal fusion feature and the tri-modal fusion feature are subjected to feature splicing processing to obtain the multi-modal physiological feature.

6. The method according to any one of claims 1 to 5, characterized in that The decision prediction model is trained in the following way: In response to the model training instruction, a multimodal training set is obtained, wherein the multimodal training set includes a plurality of training sample combinations, and the training sample combinations include EEG data samples, ECG data samples, electrodermal conductivity data samples, emotion labels, and decision labels when training a user to make a decision; For each of the training sample combinations, the EEG data samples are processed using a multi-scale initial graph convolutional network to obtain training EEG features; Using an initial neural network to process the electrocardiogram data samples and the electrodermal conductivity data samples respectively, to obtain training electrocardiogram features and training electrodermal conductivity features; Using an initial graph residual fusion network, multimodal fusion processing is performed on the training EEG features, the training ECG features, and the training skin conductance features to obtain multimodal physiological training features; Processing the multimodal physiological training features using an initial multi-task learning network to obtain decision features and emotional features, wherein the initial prediction model includes the multi-scale initial graph convolutional network, the initial neural network, the initial graph residual fusion network, and the initial multi-task learning network; The network parameters of the initial prediction model are iteratively adjusted according to the decision features, the emotion features, the emotion labels and the decision labels to obtain the decision prediction model.

7. The method according to claim 6, characterized in that Iteratively adjusting the network parameters of the initial prediction model according to the decision feature, the emotion feature, the emotion label, and the decision label to obtain the decision prediction model includes: Generating a decision cross entropy loss according to the decision feature and the decision label; Generate an emotion smoothed cross entropy loss according to the emotion feature and the emotion label; Generate a decision prediction loss value according to the decision cross entropy loss and the emotion smoothing cross entropy loss; The network parameters of the initial prediction model are iteratively adjusted according to the decision prediction loss value to obtain the decision prediction model.

8. A decision prediction device based on multimodal physiological signals, characterized in that: include: The first acquisition module is used to acquire the EEG data, ECG data and skin conduction data of the decision maker when making a decision; A prediction module, used for inputting the EEG data, the ECG data and the electrodermal data into a decision prediction model, and outputting a decision prediction feature, wherein the decision prediction feature represents the decision confidence of the decision maker after making a decision, and the decision prediction model is trained based on multimodal physiological signals; The EEG data, the ECG data and the galvanic skin data are input into a decision prediction model, and decision prediction features are output, including: Processing the EEG data using a multi-scale graph convolutional network to obtain target EEG features; For any physiological acquisition data among the electrocardiogram acquisition data and the electrodermal electrodermal acquisition data, recursive graph features, Markov transfer field features and Gram matrix angular domain features are respectively extracted from the physiological acquisition data based on a preset extraction formula; the recursive graph features, the Markov transfer field features and the Gram matrix angular domain features are arranged and processed based on multiple color channels to obtain channel image features; the channel image features are processed using a target residual network to obtain a target electrocardiogram feature or a target electrodermal electrodermal feature; Using a graph residual fusion network to perform multimodal fusion processing on the target EEG feature, the target ECG feature and the target electrodermal feature to obtain a multimodal physiological feature; The multi-modal physiological features are processed using a multi-task learning network to obtain the decision prediction features.

Images