Multi-source physiological information processing method and device, electronic equipment and storage medium

By constructing the topological graph and graph coding prompt module of multi-source physiological data, a graph embedding prompt is generated, which solves the problem of insufficient data processing capabilities of medical large models in complex physiological states, and improves the accuracy and correlation of data processing.

CN119008012APending Publication Date: 2024-11-22SHENZHEN TECH UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411212779.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

When processing multi-source physiological information, existing medical models lack generalization ability and robustness, and fail to fully capture the data relationships in complex physiological states.

Method used

By entering multi-source physiological data into the first large language model to obtain text embedding, and combining the topology diagram building module to build functional topology diagrams and spatial topology diagrams, use the graph encoding prompt module to generate graph embedding prompts, and enter the second largest language model to improve data Processing capability.

Benefits of technology

It improves the data processing capabilities of medical big models in complex physiological states, can better capture the nonlinear correlation between multi-source physiological data dimensions, and enhances the accuracy in tasks such as health monitoring.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119008012A_ABST
    Figure CN119008012A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-source physiological information processing method and device, electronic equipment and a storage medium, and can be applied to the technical field of physiological information processing. According to the method, after the multi-source physiological data are input into the first large language model to obtain the first text embedding corresponding to the multi-source physiological data, the multi-source physiological data and the first text embedding are input into the large multi-source physiological information fusion model together, so that the functional topological graph and the spatial topological graph of the multi-source physiological data are constructed through the topological graph construction module; a spatial topological relation of human body signs can be captured in multiple levels, and then a graph embedding prompt is constructed through a graph coding prompt module according to first text embedding, a functional topological graph and a spatial topological graph, so that the graph embedding prompt contains nonlinear relevance among multi-source physiological data dimensions; and then the task prompt and the graph embedding prompt are input into the second large language model, so that the data processing capability of the second large language model in a complex physiological state can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of physiological information processing, and in particular to a method and device for processing multi-source physiological information, an electronic device, and a storage medium. Background Art

[0002] In related technologies, there are significant differences in the morphological structure of physiological information data between different individuals. This difference will lead to the problem of poor generalization and robustness of multi-source physiological information data fusion models when applied to downstream tasks such as health monitoring. In addition, there are complex interrelated relationships between the dimensions of multi-source physiological information data, and the current medical big model mainly processes this type of information through manual prompt words, which fails to fully capture the complex relationship between physiological information, resulting in poor data processing capabilities of the medical big model under complex physiological conditions.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a method and device for processing multi-source physiological information, an electronic device and a storage medium, which can improve the data processing capability of a large medical model under complex physiological conditions.

[0005] To achieve the above objective, an embodiment of the present application provides a method for processing multi-source physiological information, the method comprising the following steps:

[0006] Acquire multi-source physiological data;

[0007] Inputting the multi-source physiological data into a first large language model to obtain a first text embedding corresponding to the multi-source physiological data;

[0008] embedding the multi-source physiological data and the first text into a multi-source physiological information fusion model to obtain a graph embedding prompt;

[0009] Obtaining task prompts corresponding to the multi-source physiological data;

[0010] Inputting the task prompt and the graph embedding prompt into a second language model to obtain a processing result of the multi-source physiological data;

[0011] The multi-source physiological information fusion model includes a topological map construction module and a map coding prompt module;

[0012] The topology map construction module is used to construct a functional topology map of the multi-source physiological data and a spatial topology map of the multi-source physiological data;

[0013] The graph encoding hint module is used to construct the graph embedding hint according to the first text embedding, the functional topology map and the spatial topology map.

[0014] In some embodiments, constructing the functional topology map of the multi-source physiological data includes:

[0015] Acquire features between different physiological signal channels in the multi-source physiological data;

[0016] The functional topology map is constructed according to the characteristics between the different physiological signal channels, and the functional topology map is used to capture the functional connections between various regions in the brain.

[0017] In some embodiments, constructing the spatial topological map of the multi-source physiological data includes:

[0018] Acquire the physical distances between brain regions corresponding to the multi-source physiological data;

[0019] The spatial topological map is constructed according to the physical distance, and the spatial topological map is used to describe the spatial relationship between various regions inside the brain.

[0020] In some embodiments, inputting the multi-source physiological data into a first large language model to obtain a first text embedding corresponding to the multi-source physiological data includes:

[0021] Obtaining task description and category information corresponding to the multi-source physiological data;

[0022] The task description and the category information are input into the first large language model to obtain the first text embedding corresponding to the multi-source physiological data.

[0023] In some embodiments, the graph encoding hint module includes a graph neural network encoder, a cross-modal pooling unit, a data domain mapping unit, and a self-supervised edge link prediction unit;

[0024] The graph neural network encoder is used to perform encoding node embedding on the functional topology graph and the spatial topology graph;

[0025] The cross-modal pooling unit is used to transform the first text embedding to obtain a second text embedding, and combine the second text embedding with the node embedding to obtain a graph node embedding;

[0026] The data domain mapping unit is used to align the graphic information and text information in the graph node embedding to obtain a graph embedding prompt;

[0027] The self-supervised edge link prediction unit is used to predict the hidden edges in the node embedding for self-learning.

[0028] In some embodiments, the process of transforming the first text embedding to obtain the second text embedding, and combining the second text embedding with the node embedding to obtain the graph node embedding is as follows:

[0029] T ′ =FFN1(T);

[0030] H3=Softmax(H2,T ′ );

[0031] Where FFN1 represents a fully connected network; T represents the first text embedding of each token in the input text; T ′ It represents the second text embedding after transformation; H3 represents the set of graph node embeddings after cross-modal pooling.

[0032] In some embodiments, the second largest language model is fine-tuned in parameters using the LoRA algorithm.

[0033] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a multi-source physiological information processing device, the device comprising:

[0034] The first module is used to obtain multi-source physiological data;

[0035] A second module is used to input the multi-source physiological data into a first large language model to obtain a first text embedding corresponding to the multi-source physiological data;

[0036] A third module is used to embed the multi-source physiological data and the first text into a multi-source physiological information fusion model to obtain a graph embedding prompt;

[0037] A fourth module is used to obtain task prompts corresponding to the multi-source physiological data;

[0038] A fifth module is used to input the task prompt and the graph embedding prompt into a second language model to obtain a processing result of the multi-source physiological data;

[0039] The multi-source physiological information fusion model includes a topological map construction module and a map coding prompt module;

[0040] The topology map construction module is used to construct a functional topology map of the multi-source physiological data and a spatial topology map of the multi-source physiological data;

[0041] The graph encoding hint module is used to construct the graph embedding hint according to the first text embedding, the functional topology map and the spatial topology map.

[0042] To achieve the above object, another aspect of an embodiment of the present application provides an electronic device, including:

[0043] at least one processor;

[0044] at least one memory for storing at least one program;

[0045] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0046] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0047] The embodiments of the present application include at least the following beneficial effects: The present application provides a method and device for processing multi-source physiological information, an electronic device and a storage medium. The scheme inputs the multi-source physiological data into a first large language model to obtain a first text embedding corresponding to the multi-source physiological data, and then inputs the multi-source physiological data and the first text embedding together into a multi-source physiological information fusion model, so as to construct a functional topological map and a spatial topological map of the multi-source physiological data through a topological map construction module in the multi-source physiological information fusion model, so as to capture the spatial topological relationship of human body signs at multiple levels, and then construct a graph embedding prompt according to the first text embedding, the functional topological map and the spatial topological map through a graph coding prompt module in the multi-source physiological information fusion model, so that the graph embedding prompt contains the nonlinear correlation between the dimensions of the multi-source physiological data, and after obtaining the task prompt corresponding to the multi-source physiological data, the task prompt and the graph embedding prompt are input into the second large language model, thereby improving the data processing capability of the second large language model under complex physiological conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of a method for processing multi-source physiological information provided by an embodiment of the present application;

[0049] Figure 2 It is a schematic diagram of the network architecture of the multi-source physiological information fusion model provided in the embodiment of the present application;

[0050] Figure 3 is a schematic diagram of the structure of a device for processing multi-source physiological information provided by an embodiment of the present application;

[0051] Figure 4 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application, they are only examples of devices and methods consistent with some aspects of the embodiments of the present application.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0054] In the related technology, the two main challenges faced when fusion modeling of multi-source physiological information data are: the diversity and difference of individual physiological information and the high correlation between the dimensions of multi-source physiological information data. First, there are significant differences in the morphological structure of physiological information data between different individuals. For example, when using a heart rate monitoring device, the sensor may record inaccurate heart rate readings due to the difference in reflectance spectra of people with different skin colors. This situation is particularly prominent in the multi-source physiological information data fusion model, which leads to the problem of weak generalization ability and insufficient robustness when the model is applied to downstream tasks such as health monitoring; secondly, there are complex interrelationships between the dimensions of multi-source physiological information data, such as the manifestation of myocardial infarction on the electrocardiogram: the appearance of ST segment elevation and pathological Q waves in the V1-V3 leads indicates the occurrence of anterior septal myocardial infarction. However, the current large medical models mainly process this type of information through manual prompt words, which fails to fully capture the complex relationship between physiological information. This simplified processing method ignores the rich correlation inherent in physiological signals and limits the model's ability to understand and analyze complex physiological states.

[0055] In view of this, an embodiment of the present application provides a method and device for processing multi-source physiological information, an electronic device and a storage medium. After inputting multi-source physiological data into a first large language model to obtain a first text embedding corresponding to the multi-source physiological data, the multi-source physiological data and the first text embedding are input together into a multi-source physiological information fusion model, so as to construct a functional topological map and a spatial topological map of the multi-source physiological data through a topological map construction module in the multi-source physiological information fusion model, so as to capture the spatial topological relationship of human body signs at multiple levels, and then construct a graph embedding prompt according to the first text embedding, the functional topological map and the spatial topological map through a graph coding prompt module in the multi-source physiological information fusion model, so that the graph embedding prompt contains the nonlinear correlation between the dimensions of the multi-source physiological data, and after obtaining the task prompt corresponding to the multi-source physiological data, the task prompt and the graph embedding prompt are input into a second large language model, thereby improving the data processing capability of the second large language model under complex physiological conditions.

[0056] The processing method of multi-source physiological information provided in the embodiment of the present application relates to the field of physiological information processing technology. The processing method of multi-source physiological information provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in a blockchain network; the software can be an application that implements the processing method of multi-source physiological information, etc., but is not limited to the above forms.

[0057] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0058] Figure 1 is an optional flow chart of the method for processing multi-source physiological information provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0059] Step S110, acquiring multi-source physiological data;

[0060] Step S120, inputting the multi-source physiological data into the first large language model to obtain a first text embedding corresponding to the multi-source physiological data;

[0061] Step S130, embedding the multi-source physiological data and the first text into a multi-source physiological information fusion model to obtain a graph embedding prompt;

[0062] Step S140, obtaining task prompts corresponding to multi-source physiological data;

[0063] Step S150: Input the task prompt and the image embedding prompt into the second largest language model to obtain the processing result of the multi-source physiological data.

[0064] In the embodiments of the present application, Figure 2As shown in the figure, the multi-source physiological information fusion model (LLM-PhysioFM) includes a topological graph construction module and a graph encoding prompt module. Among them, the graph construction module is used to construct the functional topological graph and spatial topological graph of multi-source physiological data, so that the subsequent modules can deeply capture the functional and spatial associations between different physiological signals. It can be understood that the functional topological graph (Adaptive-functional Graph, AG) is generated by a data-driven method, and is constructed based on the characteristic correlation between different physiological signal channels, thereby revealing the functional connection between each interval in the brain; the spatial topological graph (Distance-based Graph, DG) is constructed based on the physical distance between brain regions, which is used to describe the spatial relationship between various regions in the brain, and then reflects the importance of spatial proximity on brain function.

[0065] In an embodiment of the present application, in the process of developing LLM-PhysioFM, the use of a GNN encoder to learn the functions and spatial topological graphs of physiological information may introduce a large amount of redundant information, which may reduce the quality of the graph embedding representation or even completely invalidate it. Therefore, the present embodiment introduces physiological data category information and task description as core elements. Specifically, by obtaining the task description and category information corresponding to the multi-source physiological data as core elements before processing by the graph encoding prompt module, and inputting these core elements into the first large language model (LLM) to obtain the first text embedding, and using the first text embedding as the query variable of the cross-modal pooling unit, the learning process of graph node embedding is optimized.

[0066] In the embodiment of the present application, in order to better capture the functional and spatial associations between different physiological data, such as Figure 2 As shown, the graph encoding prompt module designed in this embodiment includes a graph neural network encoder, a cross-modal pooling unit, a data domain mapping unit, and a self-supervised edge link prediction unit to further learn and refine the embedded representation of these topological graphs. Through this design, this embodiment can not only effectively understand and process the differences in physiological information between individuals, but also accurately capture the complex relationships between physiological signals, thereby providing strong support and high accuracy for downstream health monitoring tasks.

[0067] It can be understood that the graph neural network (GNN) encoder is used to encode node embeddings of functional topology maps and spatial topology maps to extract structural features in physiological data. Among them, the GNN encoder converts these structural information into an embedded vector space by focusing on the entities and relationships in the functional topology maps and spatial topology maps through a standard graph attention network, identifying and emphasizing the features most relevant to the current analysis task. The encoding process is expressed as the following formula:

[0068] H1=f GNN (G ′ );

[0069] In the formula, H1 represents the set of node embeddings learned by the GNN encoder.

[0070] The cross-modal pooling unit uses a self-attention layer to dynamically adjust the importance weights of node embeddings to highlight the nodes that are most important for the current task. The first text embedding generated by the first language model is combined with the node embedding, and the cross-modal attention mechanism is used to evaluate the relevance of these nodes. Specifically, the self-attention mechanism function is as follows:

[0071] H2=f self-attn (H1);

[0072] Among them, H2 represents the node embedding set after self-attention.

[0073] In order to further calculate the importance of each node in the graph, this embodiment adopts the concept of text prompts. Specifically, after extracting the first text embedding from the dictionary of the first large language model, the first text embedding is transformed to obtain a second text embedding to match the dimension of the node embedding. The second text embedding and the node embedding are then combined to obtain a graph node embedding to evaluate the importance of the node using a cross-modal attention mechanism. Specifically, the processing process of this embodiment includes transforming the text embeddings and using these embeddings to calculate cross-modal attention, as shown in the following formula:

[0074] T ′ =FFN1(T);

[0075] H3=Softmax(H2,T ′ );

[0076] In the formula, FFN1 is a fully connected network, T represents the first text embedding of each token in the input text, and T ′ It represents the transformed second text embedding, and H3 represents the set of graph node embeddings after cross-modal pooling.

[0077] The data domain mapping unit is used to align the graphic information and text information in the graph node embedding to obtain the graph embedding prompt, so that a mapping can be established between the graph-level embedding and the text domain, which facilitates the understanding of the second largest language model (LLM) and bridges the inherent differences between graphic information and text information to achieve a more seamless integration. In addition, in order to ensure compatibility and consistency when interacting with the inherent structure of the second largest language model, this embodiment also maps the graph-level embedding to the same dimension as the second largest language model. Specifically, the implementation process of this embodiment is as follows:

[0078] Z = FFN (H3);

[0079] In the formula, FFN is a fully connected network, and Z is a prompt word embedded in the graph node based on the correlation relationship of multimodal physiological signals.

[0080] This embodiment is to improve Figure 2 The model shown understands the relationship between entities and captures the knowledge in the graph in a self-supervised manner, using a self-supervised edge link prediction unit as an auxiliary task. In this task, the model self-learns by predicting the hidden edges in the graph, thereby promoting the mastery of graph structure knowledge and improving the understanding of physiological information networks without additional annotation information.

[0081] In the embodiment of the present application, the second largest language model is fine-tuned through the LoRA algorithm, and the default prompt encoder is used to generate the corresponding segmentation mask. The LoRA algorithm is an efficient fine-tuning method designed specifically for large pre-trained models. It introduces a low-rank linear adjustment layer in a specific layer of the model to achieve fine adjustment of model parameters. This embodiment fixes the main weights W of the pre-trained model and introduces a trainable LoRA layer ΔW to perform targeted updates on specific parameters while maintaining model stability. At the same time, this technology uses two linear transformations and Its internal dimension r< <min{C in ,C out}, which greatly reduces the computational burden of model training. The parameters of the update layer are defined as follows:

[0082]

[0083] In the formula, Indicates the updated parameters.

[0084] After the parameters of the second language model of this embodiment are updated, it can be processed based on the input task prompts and graph embedding prompts, thereby providing more effective data support for downstream tasks (such as sleep monitoring and depression analysis).

[0085] In summary, the method of the embodiment of the present application can capture the spatial topological relationship of human signs at multiple levels by constructing a functional topological map and a physical space topological map of multi-source physiological information data; based on the functional topological map and the physical space topological map, the processing process of the coding prompt module is optimized, so that the graph node embedding contains the nonlinear correlation between the dimensions of the multi-source physiological information data, and solves the problem of insufficient connectivity between the prompt words of the existing large model; through the large model, the corresponding representation space of the multi-source physiological information data and the text embedding space mapping relationship are established, so that the large model can fully understand the morphological structure information of the physiological information; the large model parameter fine-tuning technology is studied to facilitate its application in different downstream health monitoring tasks.

[0086] Reference Figure 3 , an embodiment of the present application provides a multi-source physiological information processing device, the device comprising:

[0087] A first module 310 is used to obtain multi-source physiological data;

[0088] The second module 320 is used to input the multi-source physiological data into the first large language model to obtain a first text embedding corresponding to the multi-source physiological data;

[0089] The third module 330 is used to embed the multi-source physiological data and the first text into a multi-source physiological information fusion model to obtain a graph embedding prompt;

[0090] The fourth module 340 is used to obtain task prompts corresponding to multi-source physiological data;

[0091] A fifth module 350 is used to input the task prompt and the graph embedding prompt into the second largest language model to obtain a processing result of the multi-source physiological data;

[0092] Among them, the multi-source physiological information fusion model includes a topological map construction module and a map coding prompt module;

[0093] The topology map construction module is used to construct a functional topology map of multi-source physiological data and a spatial topology map of multi-source physiological data;

[0094] The graph encoding hint module is used to construct a graph embedding hint according to the first text embedding, the functional topology map and the spatial topology map.

[0095] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0096] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned multi-source physiological information processing method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0097] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0098] See also Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0099] The processor 410 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0100] The memory 420 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 420 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 420, and the processor 410 calls and executes the multi-source physiological information processing method of the embodiment of this application;

[0101] Input / output interface 430, used to implement information input and output;

[0102] Communication interface 440, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0103] bus 450 , which transmits information between the various components of the device (e.g., processor 410 , memory 420 , input / output interface 430 , and communication interface 440 );

[0104] The processor 410 , the memory 420 , the input / output interface 430 , and the communication interface 440 are connected to each other in communication within the device via the bus 450 .

[0105] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned method for processing multi-source physiological information is implemented.

[0106] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0107] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0108] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0109] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0110] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0111] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0112] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0113] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0114] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0115] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0116] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0117] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0118] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for processing multi-source physiological information, It is characterized in that The method comprises the following steps: Acquire multi-source physiological data; Inputting the multi-source physiological data into a first large language model to obtain a first text embedding corresponding to the multi-source physiological data; embedding the multi-source physiological data and the first text into a multi-source physiological information fusion model to obtain a graph embedding prompt; Obtaining task prompts corresponding to the multi-source physiological data; Inputting the task prompt and the graph embedding prompt into a second language model to obtain a processing result of the multi-source physiological data; The multi-source physiological information fusion model includes a topological map construction module and a map coding prompt module; The topology map construction module is used to construct a functional topology map of the multi-source physiological data and a spatial topology map of the multi-source physiological data; The graph encoding hint module is used to construct the graph embedding hint according to the first text embedding, the functional topology map and the spatial topology map.

2. The method according to claim 1, It is characterized in that The constructing of the functional topology map of the multi-source physiological data includes: Acquire features between different physiological signal channels in the multi-source physiological data; The functional topology map is constructed according to the characteristics between the different physiological signal channels, and the functional topology map is used to capture the functional connections between various regions in the brain.

3. The method according to claim 1, It is characterized in that The step of constructing the spatial topological map of the multi-source physiological data includes: Acquire the physical distances between brain regions corresponding to the multi-source physiological data; The spatial topological map is constructed according to the physical distance, and the spatial topological map is used to describe the spatial relationship between various regions inside the brain.

4. The method according to claim 1, It is characterized in that The step of inputting the multi-source physiological data into a first large language model to obtain a first text embedding corresponding to the multi-source physiological data includes: Obtaining task description and category information corresponding to the multi-source physiological data; The task description and the category information are input into the first large language model to obtain the first text embedding corresponding to the multi-source physiological data.

5. The method according to claim 1, It is characterized in that The graph encoding prompt module includes a graph neural network encoder, a cross-modal pooling unit, a data domain mapping unit and a self-supervised edge link prediction unit; The graph neural network encoder is used to perform encoding node embedding on the functional topology graph and the spatial topology graph; The cross-modal pooling unit is used to transform the first text embedding to obtain a second text embedding, and combine the second text embedding with the node embedding to obtain a graph node embedding; The data domain mapping unit is used to align the graphic information and text information in the graph node embedding to obtain a graph embedding prompt; The self-supervised edge link prediction unit is used to predict the hidden edges in the node embedding for self-learning.

6. The method according to claim 5, It is characterized in that The process of transforming the first text embedding to obtain the second text embedding, and combining the second text embedding with the node embedding to obtain the graph node embedding is as follows: T ′ =FFN 1 (T); H 3 =Softmax(H 2 ,T ′ ); Among them, FFN 1 represents a fully connected network; T represents the first text embedding of each token in the input text; T ′ H represents the second text embedding after transformation; 3 Represented as the set of graph node embeddings after cross-modal pooling.

7. The method according to claim 1, It is characterized in that The second largest language model is fine-tuned using the LoRA algorithm.

8. A device for processing multi-source physiological information, It is characterized in that The device comprises: The first module is used to obtain multi-source physiological data; A second module is used to input the multi-source physiological data into a first large language model to obtain a first text embedding corresponding to the multi-source physiological data; A third module is used to embed the multi-source physiological data and the first text into a multi-source physiological information fusion model to obtain a graph embedding prompt; A fourth module is used to obtain task prompts corresponding to the multi-source physiological data; A fifth module is used to input the task prompt and the graph embedding prompt into a second language model to obtain a processing result of the multi-source physiological data; The multi-source physiological information fusion model includes a topological map construction module and a map coding prompt module; The topology map construction module is used to construct a functional topology map of the multi-source physiological data and a spatial topology map of the multi-source physiological data; The graph encoding hint module is used to construct the graph embedding hint according to the first text embedding, the functional topology map and the spatial topology map.

9. An electronic device, It is characterized in that include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.