Multi-modal signal analysis method and system, electronic equipment and storage medium

Abstract

The invention provides a multi-modal signal analysis method and system, electronic equipment and a storage medium, and relates to the technical field of signal processing, and the analysis method comprises the steps: obtaining the time domain data information of a to-be-analyzed multi-modal signal; performing division and fast Fourier transform on the time domain data information to obtain frequency domain data information; extracting specific feature data corresponding to a real part and an imaginary part of the frequency domain data information, and constructing a dynamic complex value graph; processing the dynamic complex value graph through a double-flow graph attention network to obtain target data information; and performing aggregation and classification to obtain target output information. According to the method, the time domain data information is divided into segmented signals by adopting an overlapped sliding window method, and time domain and frequency domain conversion is performed, so that the phase shift problem is avoided; the dynamic complex value graph is balanced between local frequency convolution and sampling points, and the recognition characteristic of multiple components in the frequency domain is improved; and solving the dynamic complex value graph based on a double-flow graph attention network to determine the spatial characteristics of the multi-modal signal.

Description

Technical Field

[0001] The present invention relates to the field of signal processing technology, and in particular to a multimodal signal analysis method, system, electronic device and storage medium. Background Art

[0002] Complex networks have been widely used in analysis tasks in various fields due to their excellent representation of graph structured data. Graph neural networks are a deep learning technology used to process graph data. Graph data is a type of structured data that can express complex relationships and logical rules between nodes. Graph neural networks can collect information about nodes and neighboring nodes through operations such as graph convolution, thereby achieving tasks such as node classification, graph classification, and graph generation.

[0003] In related technologies, graph neural networks can process irregular graph topological structures that are difficult for traditional neural networks to solve. Current research on multimodal signals based on graph neural networks focuses on time analysis, directly converting the original time domain signals into graphical representations, resulting in poor accuracy in multimodal signal analysis and inability to reliably classify and identify time domain signals. This makes graph neural networks very limited in performing multimodal signal analysis and limits application scenarios. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a multimodal signal analysis method, system, electronic device and storage medium, which solve the problems faced by neural network technology of inaccurate multimodal signal analysis and poor general applicability.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a multimodal signal analysis method, the method comprising: obtaining time domain data information of a multimodal signal to be analyzed; dividing the time domain data information based on an overlapping sliding window method to obtain a group of segmented signals; performing fast Fourier transform on a group of segmented signals to obtain frequency domain data information; extracting specific feature data corresponding to the real and imaginary parts of the frequency domain data information to construct a dynamic complex-valued graph; performing weight assignment and aggregation processing on each vertex of the dynamic complex-valued graph through a dual-stream graph attention network to obtain target data information; aggregating and classifying the global information in the target data information to obtain target output information; wherein the target output information can characterize the dynamic spatial feature information of the dynamic complex-valued graph and the sequence information of different component vertices.

[0007] According to the first aspect of the embodiment of the present application, the dynamic complex-valued graph includes a real domain graph and an imaginary domain graph and both are in matrix form; the aforementioned extraction of specific feature data corresponding to the real part and imaginary part of the frequency domain data information to construct a dynamic complex-valued graph includes: constructing a real component and an imaginary component based on the real part and imaginary part of the frequency domain data information respectively; performing convolution processing on the real component and the imaginary component respectively based on a one-dimensional convolution operator to obtain a first initial variable and a second initial variable; performing activation processing on the first initial variable and the second initial variable based on a preset nonlinear activation function to obtain a first target variable and a second target variable; arranging the feature sequences in the first target variable and the second target variable in diagonally parallel directions respectively to obtain a real domain graph and an imaginary domain graph.

[0008] According to the first aspect of the embodiment of the present application, the above-mentioned expression for performing convolution processing on the real component and the imaginary component based on the one-dimensional convolution operator is:

[0009]

[0010] in, is the first initial variable, is the second initial variable, R represents the real domain graph, I represents the imaginary domain graph, Rconvl and Iconvl are one-dimensional convolutional layers, l is the length of the convolution kernel and satisfies: l∈[2,h], h is a hyperparameter that constrains the distance between two sample points, And it characterizes the length of the signal to be analyzed.

[0011] According to the first aspect of the embodiment of the present application, the expression for activating the first initial variable and the second initial variable based on the preset nonlinear activation function is:

[0012]

[0013] in, is the first target variable, is the second target variable, ReLU is a nonlinear activation function and satisfies: ReLU = max(0, ·), R represents the real domain graph, I represents the imaginary domain graph, S represents the element in the real domain graph, and M represents the element in the imaginary domain graph. And it characterizes the length of the signal to be analyzed.

[0014] According to the first aspect of the embodiment of the present application, the aforementioned method of allocating weights and aggregating the vertices of the dynamic complex-valued graph through a dual-stream graph attention network to obtain target data information includes: inputting the real-domain graph and the imaginary-domain graph into the dual-stream graph neural attention network layer; based on the sequence information between different component vertices in the dynamic complex-valued graph, assigning different weights to each vertex in the real-domain graph and the imaginary-domain graph through the self-attention mechanism in the dual-stream graph neural attention network layer, to obtain the target vertex; performing aggregation analysis on the target vertices in the real-domain graph and the imaginary-domain graph to obtain the target data information to retain the sequence information between different component vertices; wherein the self-attention mechanism is a signal layer feedforward neural network that maps the features in the real-domain graph and the imaginary domain graph to real numbers.

[0015] According to a first aspect of an embodiment of the present application, the target data information includes first target data and second target data corresponding to a real domain graph and a virtual domain graph, respectively, and the expressions of the two are:

[0016]

[0017] in, is the first target data and is the second target data and R represents the real domain graph, I represents the imaginary domain graph, σ(·) represents the activation function, and It represents the normalized result of the attention coefficient using the k-th order self-attention mechanism, W is the weight matrix, K is the hyperparameter and represents the total order of the self-attention mechanism.

[0018] According to the first aspect of the embodiment of the present application, the above-mentioned global information in the target data information is aggregated and classified to obtain target output information, including: adding a jump connection mechanism to the target data information to obtain a first fused feature vector and a second fused feature vector corresponding to the real domain graph and the imaginary domain graph respectively; inputting the first fused feature vector and the second fused feature vector into different dual-stream graph neural attention network layers to obtain an output array; aggregating the global information of the output array to obtain an aggregated feature vector that can characterize a dynamic complex-valued graph; based on a preset cross entropy function and a preset softmax function, analyzing the aggregated feature vector to obtain target output information corresponding to different categories.

[0019] In the second aspect, an embodiment of the present application provides a multimodal signal analysis system, which includes an acquisition module, a division module, a transformation module, a construction module and a processing module; the acquisition module is used to acquire time domain data information of the multimodal signal to be analyzed; the division module is used to divide the time domain data information based on the overlapping sliding window method to obtain a group of segmented signals; the transformation module is used to perform a fast Fourier transform on a group of segmented signals to obtain frequency domain data information; the construction module is used to extract specific feature data corresponding to the real and imaginary parts of the frequency domain data information to construct a dynamic complex-valued graph; the processing module is used to assign weights and aggregate each vertex of the dynamic complex-valued graph through a dual-stream graph attention network to obtain target data information; the processing module is also used to aggregate and classify the global information in the target data information to obtain target output information; wherein the target output information can characterize the dynamic spatial feature information of the dynamic complex-valued graph and the sequence information of different component vertices.

[0020] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the multimodal signal analysis method in the first aspect is implemented.

[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the multimodal signal analysis method in the aforementioned first aspect is implemented.

[0022] The present invention provides a multimodal signal analysis method, system, electronic device and storage medium. Compared with the prior art, the present invention has the following beneficial effects:

[0023] The present application adopts an overlapping sliding window method to divide the time domain data information of the multimodal signal to be analyzed into segmented signals, and converts the segmented signals between the time domain and the frequency domain, thereby avoiding the phase shift problem of the time domain data information; wherein, the overlapping sliding window method as a data enhancement scheme can also reduce the loss in the signal analysis process; after analyzing the specific feature data corresponding to the real and imaginary parts of the frequency domain data information, a balance is achieved between the local frequency convolution and the sampling points through a dynamic complex-valued graph, thereby improving the recognition characteristics of multiple components in the frequency domain; the present application also uses a dual-stream graph attention network to assign weights and aggregate each vertex of the dynamic complex-valued graph, thereby accurately determining the spatial characteristics of the multimodal signal. The present application can analyze multimodal signals based on different complex-valued networks, and has high general applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 It is a flowchart of a multimodal signal analysis method provided in an embodiment of the present application;

[0026] Figure 2 yes Figure 1 An exemplary flow chart of S140;

[0027] Figure 3 yes Figure 1 An exemplary flow chart of S150;

[0028] Figure 4 yes Figure 1 An exemplary flow chart of S160;

[0029] Figure 5 It is a structural schematic diagram of a multimodal signal analysis system provided in an embodiment of the present application;

[0030] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0033] The embodiments of the present application solve the problems of inaccurate multimodal signal analysis and poor general applicability faced by neural network technology by providing a multimodal signal analysis method, system, electronic device and storage medium, achieve accurate learning of multimodal signal spatial characteristics, can analyze multimodal signals based on different complex-valued networks, and have high general applicability.

[0034] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0035] Complex networks have been widely used in analysis tasks in various fields due to their excellent representation of graph structured data. Graph neural networks are a deep learning technology used to process graph data. Graph data is a structured data that can express complex relationships and logical rules between nodes. Graph neural networks can collect information about nodes and neighboring nodes through operations such as graph convolution, thereby achieving tasks such as node classification, graph classification, and graph generation. Graph neural networks can handle irregular graph topologies that are difficult for traditional neural networks to solve.

[0036] Current research on multimodal signals based on graph neural networks focuses on time analysis, directly converting the original time domain signal into a graphical representation, while ignoring the phase shift problem of the time domain signal, resulting in poor accuracy in multimodal signal analysis and inability to reliably classify and identify time domain signals, which makes graph neural networks very limited in multimodal signal analysis and restricts application scenarios. In addition, current research and analysis on multimodal signals mainly focuses on frequency analysis, while ignoring the arrangement of multiple components and the power of specific waves relative to other waves. If only the amplitude information of the signal is concerned, some specific information in the signal will be lost. Therefore, there is an urgent need to provide a multimodal signal analysis method, system, electronic device and storage medium to solve the aforementioned problems existing in the relevant technology.

[0037] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0038] The following first introduces a multimodal signal analysis method provided in an embodiment of the present application.

[0039] A schematic diagram of a multimodal signal analysis method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the multimodal signal analysis method may include the following steps S110 - S160 .

[0040] S110: Acquire time domain data information of the multimodal signal to be analyzed.

[0041] S120. Divide the time domain data information based on an overlapping sliding window method to obtain a group of segmented signals.

[0042] S130, performing fast Fourier transform on a group of segmented signals to obtain frequency domain data information.

[0043] S140, extracting specific feature data corresponding to the real part and the imaginary part of the frequency domain data information, and constructing a dynamic complex value graph.

[0044] S150, weight allocation and aggregation processing are performed on each vertex of the dynamic complex-valued graph through the dual-stream graph attention network to obtain target data information.

[0045] S160, aggregate and classify the global information in the target data information to obtain target output information; wherein the target output information can represent the dynamic spatial feature information of the dynamic complex-valued graph and the sequence information of different component vertices.

[0046] In an embodiment of the present application, in the process of analyzing the multimodal signal, the time domain data information in the multimodal signal is first extracted; it is understandable that since the collected multimodal signal may have the problem of a small data set size, an overlapping sliding window method can be introduced as a data enhancement scheme to divide the time domain data information into segmented signals for analysis and processing. After obtaining the segmented signal, the segmented signal can be converted from the time domain to the frequency domain through the fast Fourier transform. By analyzing the frequency domain data information, a dynamic complex-valued graph can be constructed, and the dynamic complex-valued graph can be operated based on the dual-stream graph attention network to determine the target data information; further, by aggregating and classifying the target data information, the dynamic spatial feature information that can characterize the dynamic complex-valued graph and the target output information of the sequence information of different component vertices can be obtained, thereby realizing the analytical processing of the multimodal signal.

[0047] The above is a specific implementation method of the multimodal signal analysis method provided in an embodiment of the present application. The present application divides the time domain data information of the multimodal signal to be analyzed into segmented signals, and performs conversion between the time domain and the frequency domain, thereby solving the phase shift problem of the time domain data information; after analyzing the specific feature data corresponding to the real and imaginary parts of the frequency domain data information, a balance is achieved between local frequency convolution and sampling points through a dynamic complex-valued graph, thereby improving the recognition characteristics of multiple components in the frequency domain; and further based on the dual-stream graph attention network, weight allocation and aggregation processing and other operations are performed on each vertex of the dynamic complex-valued graph to accurately determine the spatial characteristics of the multimodal signal.

[0048] In some embodiments, the dynamic complex-valued graph includes a real domain graph and an imaginary domain graph, both of which are in matrix form. The aforementioned extraction of specific feature data corresponding to the real part and the imaginary part of the frequency domain data information to construct the dynamic complex-valued graph, that is, S140, may include the following steps:

[0049] S210 , constructing a real component and an imaginary component based on the real part and the imaginary part of the frequency domain data information respectively.

[0050] S220. Perform convolution processing on the real component and the imaginary component respectively based on a one-dimensional convolution operator to obtain a first initial variable and a second initial variable.

[0051] S230: Based on a preset nonlinear activation function, activate the first initial variable and the second initial variable to obtain a first target variable and a second target variable.

[0052] S240, respectively arrange the feature sequences in the first target variable and the second target variable along diagonal parallel directions to obtain a real domain graph and an imaginary domain graph.

[0053] In an embodiment of the present application, the real part and imaginary part of the frequency domain data information are respectively constructed into real components and imaginary components, and multi-scale real convolution and multi-scale imaginary convolution are used on the real components and imaginary components to obtain a first initial variable and a second initial variable; it can be understood that based on the first initial variable and the second initial variable, through activation processing and feature sequence arrangement, a real domain graph and an imaginary domain graph can be constructed to obtain the interaction between multimodal signals.

[0054] It should be noted that real components and imaginary components can be determined from a multimodal signal. Different forms of real components and imaginary components indicate the encoding of various semantic information. Therefore, real domain diagrams and imaginary domain diagrams that characterize real information and imaginary information help to obtain more accurate results.

[0055] In an example, the frequency signal corresponding to the real component R and the imaginary component I is shown as follows:

[0056]

[0057] Among them, R i and I i Represents the signal value corresponding to the sampling point i in the frequency domain data information. In order to obtain the dynamic complex-valued graph representation of the real component R and the imaginary component I mentioned above and eliminate the phase transformation problem caused by frequency transformation, a local learnable frequency convolution operator is required. To achieve this, the real component R and the imaginary component I are processed by training a one-dimensional convolution operator.

[0058] Specifically, the expressions for convolution processing of the real component R and the imaginary component I respectively based on the one-dimensional convolution operator satisfy:

[0059]

[0060] in, is the first initial variable, is the second initial variable, R represents the real domain graph, I represents the imaginary domain graph, Rconvl and Iconvl are one-dimensional convolutional layers, l is the length of the convolution kernel and satisfies: l∈[2,h], It represents the length of the signal to be analyzed, and h is a hyperparameter that constrains the distance between two sample points and can be used to extract local information.

[0061] Furthermore, the present application processes the first initial variable based on the nonlinear activation function ReLU and the second initial variable The expression satisfies:

[0062]

[0063] in, is the first target variable, is the second target variable, ReLU is a nonlinear activation function and satisfies: ReLU = max(0, ·), R represents the real domain graph, I represents the imaginary domain graph, S represents the element in the real domain graph, and M represents the element in the imaginary domain graph. And it characterizes the length of the signal to be analyzed.

[0064] Then, the first target variable and the second target variable The characteristic sequences in are arranged in the diagonal parallel direction, and the real domain graph and the imaginary domain graph corresponding to the real component and the imaginary component are obtained respectively. The form of the real domain graph and the imaginary domain graph is the characteristic matrix, as shown below:

[0065]

[0066]

[0067] Among them, A R is the characteristic matrix corresponding to the real domain graph, A Iis the characteristic matrix corresponding to the virtual domain graph; A R and A I The dynamic complex-valued graph representation of the frequency domain sequence is described, which can flexibly explore the specific interactions between multiple components of multimodal signals by extracting specific features between signal electrode sample points.

[0068] In some embodiments, the aforementioned weight allocation and aggregation processing of each vertex of the dynamic complex-valued graph through the dual-stream graph attention network to obtain target data information, that is, S150 may include the following steps:

[0069] S310, input the real domain graph and the imaginary domain graph into the dual-stream graph neural attention network layer.

[0070] S320. Based on the sequence information between different component vertices in the dynamic complex-valued graph, different weights are assigned to each vertex in the real domain graph and the virtual domain graph through the self-attention mechanism in the dual-stream graph neural attention network layer to obtain the target vertex.

[0071] S330 , performing aggregation analysis on target vertices in the real domain graph and the virtual domain graph to obtain target data information to retain sequence information between different component vertices.

[0072] In an embodiment of the present application, the dual-stream graph neural attention network layer integrates a self-attention mechanism. It can be understood that the real domain graph and the virtual domain graph serve as the input of the dual-stream graph neural attention network layer. The dual-stream graph neural attention network layer can not only retain the sequence information between different component vertices in the real domain graph and the virtual domain graph, but also highly aggregate the dynamic spatial feature information in the graph.

[0073] It should be emphasized that the dual-stream graph neural attention network layer in the present application includes two parallel branches, and the two parallel branches simultaneously learn corresponding spatial features from the real domain graph and the imaginary domain graph.

[0074] Specifically, in order to obtain the target vertex, in the process of assigning different weights to each vertex in the real domain graph and the virtual domain graph through the self-attention mechanism, the influence of the adjacent node features on node i can be expressed as:

[0075]

[0076] Among them, the degree of impact Used to characterize the influence between vertex i and vertex j, and They correspond to the real domain graph and the imaginary domain graph respectively; and is the weight matrix of the dynamic complex-valued graph. By transforming the features of the nodes, the mapping relationship between the input features and the output features can be obtained. The relationship mapping features in the real domain graph and the virtual domain graph can be expressed as: and

[0077] Second, the self-attention mechanism and It is a signal layer feedforward neural network that maps the features of real domain graphs and imaginary domain graphs to real numbers. They are represented by weight vectors and Parameterization. By regularizing the attention using the softmax function, the attention correlation coefficient can be obtained and

[0078]

[0079] The self-attention mechanism calculates and It can be further expressed as:

[0080]

[0081] Where [·||·] represents the connection relationship between two nodes i and j, (·) T represents transposition, and LeakyReLU represents the activation function.

[0082] By using multiple attention mechanisms to calculate the attention coefficients of surrounding nodes, the learning process of the model can be made more stable. After the information of adjacent nodes is fused, the first target data and the second target data in the target data information can be expressed as:

[0083]

[0084] in, is the first target data and is the second target data and R represents the real domain graph, I represents the imaginary domain graph, σ(·) represents the activation function, and It represents the normalized result of the attention coefficient using the k-th order self-attention mechanism, W is the weight matrix, K is the hyperparameter and represents the total order of the self-attention mechanism.

[0085] In addition, the eigenvectors of all vertices in the real domain graph and the imaginary domain graph are and As shown below:

[0086]

[0087] In some embodiments, the above-mentioned aggregation and classification of the global information in the target data information to obtain the target output information, that is, S160 may include the following steps:

[0088] S410 , adding a skip connection mechanism to the target data information to obtain a first fused feature vector and a second fused feature vector corresponding to the real domain graph and the imaginary domain graph respectively.

[0089] S420, input the first fused feature vector and the second fused feature vector into different dual-stream graph neural attention network layers to obtain an output array.

[0090] S430 , aggregate the global information of the output array to obtain an aggregated feature vector capable of characterizing the dynamic complex-valued graph.

[0091] S440, based on a preset cross entropy function and a preset softmax function, the aggregated feature vector is analyzed to obtain target output information corresponding to different categories.

[0092] In the embodiments of the present application, it can be understood that after aggregating the target vertices in the real domain graph and the imaginary domain graph, adding a skip connection mechanism can better utilize the node features to obtain a first fused feature vector and a second fused feature vector; the first fused feature vector and the second fused feature vector can be input into different dual-stream graph neural attention network layers, and the outputs of multiple dual-stream graph neural attention network layers can be processed into a series representation and further processed by a classification model using a feedforward network with adaptive readout and connection to perform sequence learning and recognition.

[0093] The present application also provides a multimodal signal analysis system 500, such as Figure 5 As shown, the multimodal signal analysis system 500 may include the following modules:

[0094] The acquisition module 510 is used to acquire time domain data information of the multimodal signal to be analyzed.

[0095] The division module 520 is used to divide the time domain data information based on the overlapping sliding window method to obtain a group of segmented signals.

[0096] The transformation module 530 is used to perform fast Fourier transform on a group of segmented signals to obtain frequency domain data information.

[0097] The construction module 540 is used to extract specific feature data corresponding to the real part and the imaginary part of the frequency domain data information and construct a dynamic complex value graph.

[0098] Processing module 550 is used to assign weights and aggregate each vertex of the dynamic complex-valued graph through a dual-stream graph attention network to obtain target data information; it is also used to aggregate and classify the global information in the target data information to obtain target output information; wherein the target output information can represent the dynamic spatial feature information of the dynamic complex-valued graph and the sequence information of different component vertices.

[0099] According to an embodiment of the present application, any multiple modules among the acquisition module 510, the division module 520, the transformation module 530, the construction module 540 and the processing module 550 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0100] In some embodiments, the construction module 540 may specifically include:

[0101] The construction unit 541 is used to construct a real component and an imaginary component based on the real part and the imaginary part of the frequency domain data information respectively.

[0102] The convolution unit 542 is used to perform convolution processing on the real component and the imaginary component respectively based on a one-dimensional convolution operator to obtain a first initial variable and a second initial variable.

[0103] The activation unit 543 is used to activate the first initial variable and the second initial variable based on a preset nonlinear activation function to obtain the first target variable and the second target variable.

[0104] The arrangement unit 544 is used to arrange the feature sequences in the first target variable and the second target variable along diagonal parallel directions to obtain a real domain graph and an imaginary domain graph.

[0105] In some embodiments, the processing module 550 may specifically include:

[0106] The first input unit 551 is used to input the real domain graph and the imaginary domain graph into the dual-stream graph neural attention network layer.

[0107] The allocation unit 552 is used to allocate different weights to each vertex in the real domain graph and the virtual domain graph based on the sequence information between different component vertices in the dynamic complex-valued graph through the self-attention mechanism in the dual-stream graph neural attention network layer to obtain the target vertex.

[0108] The first aggregation unit 553 is used to perform aggregation analysis on the target vertices in the real domain graph and the imaginary domain graph to obtain target data information so as to retain sequence information between different component vertices.

[0109] In some embodiments, the processing module 550 may further include:

[0110] The adding unit 554 is used to add a skip connection mechanism to the target data information to obtain a first fused feature vector and a second fused feature vector corresponding to the real domain graph and the imaginary domain graph respectively.

[0111] The second input unit 555 is used to input the first fused feature vector and the second fused feature vector into different dual-stream graph neural attention network layers to obtain an output array.

[0112] The second aggregation unit 556 is used to aggregate the global information of the output array to obtain an aggregated feature vector that can characterize the dynamic complex-valued graph.

[0113] The vector analysis unit 557 is used to analyze the aggregated feature vector based on a preset cross entropy function and a preset softmax function to obtain target output information corresponding to different categories.

[0114] In some embodiments, the present application provides an electronic device, the structure diagram of the electronic device is as follows Figure 6 shown.

[0115] The electronic device may include a processor 610 and a memory 620 storing computer program instructions.

[0116] Specifically, the processor 610 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0117] The memory 620 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 620 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 620 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 620 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 620 is a non-volatile solid-state memory.

[0118] The memory 620 may include a read-only memory (ROM), a random access memory (RAM), a disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, typically, the memory 620 includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer executable instructions, and when the software is executed (e.g., by one or more processors), it can perform the operations described in any of the multimodal signal analysis methods in the above-mentioned embodiments.

[0119] The processor 610 implements any one of the multimodal signal analysis methods in the above embodiments by reading and executing computer program instructions stored in the memory 620 .

[0120] In one example, the electronic device may further include a communication interface 630 and a bus 600. Figure 6 As shown, the processor 610, the memory 620, and the communication interface 630 are connected via a bus 600 and communicate with each other.

[0121] The communication interface 630 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0122] Bus 600 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 600 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0123] In addition, in combination with the multimodal signal analysis method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any multimodal signal analysis method in the above embodiment is implemented.

[0124] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0125] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0126] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0127] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0128] In summary, compared with the prior art, this application has the following beneficial effects:

[0129] 1. This application adopts an overlapping sliding window method as a data enhancement solution, which can reduce the loss in the signal analysis process. It solves the phase shift problem of the time domain data information by dividing the time domain data information of the multimodal signal to be analyzed into segmented signals and converting the segmented signals between the time domain and the frequency domain.

[0130] 2. This application adopts a dynamic complex-valued graph as a new frequency signal representation method. After analyzing the specific feature data corresponding to the real and imaginary parts of the frequency domain data information, the dynamic complex-valued graph is used to achieve a balance between local frequency convolution and sampling points, thereby improving the recognition characteristics of multiple components in the frequency domain. The dynamic complex-valued graph has linear time and space complexity.

[0131] 3. This application proposes a new dual-stream graph attention framework. The dual-stream graph neural attention network layer includes two parallel branches. The two parallel branches can simultaneously learn the corresponding spatial features from the real domain graph and the imaginary domain graph. The dual-stream graph attention network is used to assign weights and aggregate the vertices of the dynamic complex-valued graph, thereby accurately determining the spatial features of the multimodal signal.

[0132] 4. The dual-stream graph neural attention network layer in this application is universal and can be applied to various complex-valued networks and learn the spatial features of various multimodal signals. This application can analyze multimodal signals based on different complex-valued networks and has high universal applicability.

[0133] It should also be noted that the multimodal signal analysis method, system, electronic device and storage medium provided by this application can be applied to multiple fields to solve corresponding practical problems; specifically, the multimodal signal analysis method in this application is applicable to various types of signals, and can process various signals such as electroencephalogram signals, electrocardiogram signals and functional magnetic resonance imaging signals. This application provides a new signal analysis tool that can be used in interdisciplinary applied research and fields such as biomedical engineering. In fields such as engineering, psychology and neuroscience, it can provide a new perspective for the analysis and interpretation of various signal data, and promote interdisciplinary cooperation and technological innovation.

[0134] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal signal analysis method, characterized in that: include: Obtaining time domain data information of the multimodal signal to be analyzed; Based on the overlapping sliding window method, the time domain data information is divided to obtain a group of segmented signals; Performing fast Fourier transform on the group of segmented signals to obtain frequency domain data information; Extracting specific characteristic data corresponding to the real part and the imaginary part of the frequency domain data information, and constructing a dynamic complex value graph; By using a dual-stream graph attention network, weights are assigned and aggregated to each vertex of the dynamic complex-valued graph to obtain target data information; The global information in the target data information is aggregated and classified to obtain target output information; wherein the target output information can represent the dynamic spatial feature information of the dynamic complex-valued graph and the sequence information of different component vertices.

2. The multimodal signal analysis method according to claim 1, characterized in that: The dynamic complex-valued graph includes a real domain graph and an imaginary domain graph, both of which are in matrix form; extracting specific feature data corresponding to the real part and the imaginary part of the frequency domain data information to construct a dynamic complex-valued graph includes: Constructing a real component and an imaginary component based on the real part and the imaginary part of the frequency domain data information respectively; Performing convolution processing on the real component and the imaginary component respectively based on a one-dimensional convolution operator to obtain a first initial variable and a second initial variable; Based on a preset nonlinear activation function, the first initial variable and the second initial variable are activated to obtain a first target variable and a second target variable; The feature sequences in the first target variable and the second target variable are arranged along diagonal parallel directions respectively to obtain the real domain graph and the imaginary domain graph.

3. The multimodal signal analysis method according to claim 2, characterized in that: The expression for performing convolution processing on the real component and the imaginary component based on the one-dimensional convolution operator is: in, is the first initial variable, is the second initial variable, R represents the real domain graph, I represents the imaginary domain graph, Rconvl and Iconvl are one-dimensional convolutional layers, l is the convolution kernel length and satisfies: l∈[2,h], h is a hyperparameter constraining the distance between two sample points, And characterizes the length of the signal to be analyzed.

4. The multimodal signal analysis method according to claim 2 or 3, characterized in that: The expression for activating the first initial variable and the second initial variable based on the preset nonlinear activation function is: in, is the first target variable, is the second target variable, ReLU is a nonlinear activation function and satisfies: ReLU=max(0,·), R represents the real domain graph, I represents the imaginary domain graph, S represents an element in the real domain graph, and M represents an element in the imaginary domain graph, And characterizes the length of the signal to be analyzed.

5. The multimodal signal analysis method according to claim 2, characterized in that: The dual-stream graph attention network is used to perform weight allocation and aggregation processing on each vertex of the dynamic complex-valued graph to obtain target data information, including: Inputting the real domain graph and the imaginary domain graph into a dual-stream graph neural attention network layer; Based on the sequence information between different component vertices in the dynamic complex-valued graph, different weights are assigned to each vertex in the real domain graph and the imaginary domain graph through the self-attention mechanism in the dual-stream graph neural attention network layer to obtain a target vertex; Performing aggregation analysis on the target vertices in the real domain graph and the imaginary domain graph to obtain target data information to retain sequence information between different component vertices; Among them, the self-attention mechanism is a signal layer feedforward neural network that maps the features in the real domain graph and the imaginary domain graph to real numbers.

6. The multimodal signal analysis method according to claim 5, characterized in that: The target data information includes first target data and second target data corresponding to the real domain graph and the virtual domain graph respectively, and the expressions of the two are: in, is the first target data and is the second target data and R represents the real domain graph, I represents the imaginary domain graph, σ(·) represents the activation function, and represents the normalized result of the attention coefficient using the k-th order self-attention mechanism, W is the weight matrix, K is a hyperparameter and represents the total order of the self-attention mechanism.

7. The multimodal signal analysis method according to claim 2, characterized in that: The aggregating and classifying the global information in the target data information to obtain target output information includes: Adding a skip connection mechanism to the target data information to obtain a first fused feature vector and a second fused feature vector corresponding to the real domain graph and the imaginary domain graph respectively; Inputting the first fused feature vector and the second fused feature vector into different dual-stream graph neural attention network layers to obtain an output array; Aggregating the global information of the output array to obtain an aggregated feature vector capable of characterizing the dynamic complex-valued graph; Based on a preset cross entropy function and a preset softmax function, the aggregated feature vector is analyzed to obtain target output information corresponding to different categories.

8. A multimodal signal analysis system, characterized in that: include: An acquisition module, used to acquire time domain data information of the multimodal signal to be analyzed; A division module, used for dividing the time domain data information based on an overlapping sliding window method to obtain a group of segmented signals; A transformation module, used for performing a fast Fourier transform on the group of segmented signals to obtain frequency domain data information; A construction module is used to extract specific feature data corresponding to the real part and the imaginary part of the frequency domain data information and construct a dynamic complex value graph; A processing module, used for performing weight assignment and aggregation processing on each vertex of the dynamic complex-valued graph through a dual-stream graph attention network to obtain target data information; The processing module is also used to aggregate and classify the global information in the target data information to obtain target output information; wherein the target output information can represent the dynamic spatial feature information of the dynamic complex-valued graph and the sequence information of different component vertices.

9. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the multimodal signal analysis method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the multimodal signal analysis method according to any one of claims 1 to 7 is implemented.

Images