Swallowing function evaluation system and method based on multi-modal fusion, equipment and medium

Abstract

The invention provides a swallowing function evaluation system and method based on multi-modal fusion, equipment and a medium. A swallowing radiography device, a multi-channel array flexible myoelectricity acquisition sensor and a near-infrared brain function imaging device of the system are respectively used for synchronously acquiring swallowing image information, swallowing muscle group myoelectricity array information and near-infrared brain function imaging information when a patient executes a swallowing mode; and the main controller processes the synchronously acquired information to realize swallowing function evaluation. According to the method, sEMG array image information of sEMG and near-infrared brain function imaging information are combined with swallowing image information to construct a human body swallowing evaluation model, and a self-developed multi-channel array flexible myoelectricity acquisition sensor is adopted to construct a correlation mapping model from a swallowing contrast internal structure to external muscle point information; and by utilizing the priori of the myoelectricity information and the near-infrared brain imaging information, the evaluation and prediction of the dysphagia are realized by adopting a classifier algorithm, and technical support is provided for the application of the near-infrared technology in the swallowing evaluation.

Description

Technical Field

[0001] The present invention relates to the technical field of swallowing function evaluation, and in particular to a swallowing function evaluation system and method, device and medium based on multimodal fusion. Background Art

[0002] Swallowing activity is completed by the orderly coordination of multiple muscles through neural regulation. The muscles related to swallowing function are concentrated in the neck and face. Simply using a few surface electrodes cannot collect the electrophysiological activities of various muscles during the complete swallowing process. The present invention uses independently designed flexible surface electrodes combined with near-infrared brain function imaging devices to collect key muscle activity information and brain area activation information during the complete swallowing process, and evaluates swallowing activities and swallowing dysfunction based on multimodal data. It studies the electrophysiological functions of the neck and facial muscle groups during swallowing and the coordinated working mechanism of the internal structure in swallowing angiography, and establishes a multimodal swallowing function evaluation system, which provides more accurate, non-invasive and objective new theories and methods for detection and evaluation for the clinical diagnosis of patients with swallowing disorders. Summary of the invention

[0003] In order to achieve the above-mentioned purpose and other advantages according to the present invention, the first purpose of the present invention is to provide a swallowing function evaluation system based on multimodal fusion, including a swallowing imaging device, a multi-channel array flexible electromyography acquisition sensor, a near-infrared brain function imaging device, and a main controller; wherein, The swallowing imaging device is used to perform swallowing imaging on the patient's swallowing pattern to collect swallowing image information; The multi-channel array flexible electromyography acquisition sensor is used to synchronously acquire the electromyographic signals of the swallowing muscle group during the swallowing angiography process to obtain the electromyographic array information of the swallowing muscle group; The near-infrared brain function imaging device is used to synchronously collect near-infrared brain function imaging information during swallowing angiography; The main controller is used to construct a relationship mapping model from the swallowing image information to the swallowing muscle group electromyographic array information, and to implement pattern recognition based on the near-infrared brain functional imaging information and the swallowing muscle group electromyographic array information using a classification algorithm. The relationship mapping model and the recognized pattern are combined through classifier decision to obtain a swallowing function evaluation result.

[0004] Furthermore, the multi-channel array flexible electromyography acquisition sensor includes a transparent PET layer, a silver chloride circuit, a carbon paste circuit, insulating oil, a hydrogel, a medical tape, and a release film; the silver chloride circuit is laid on the transparent PET layer, the carbon paste circuit is electrically connected to the silver chloride circuit, the silver chloride circuit is connected to the hydrogel, insulating oil is arranged around the hydrogel, the medical tape is used to fix the multi-channel array flexible electromyography acquisition sensor at the position to be measured, and the release film covers the hydrogel and the medical tape.

[0005] Furthermore, the swallowing mode includes a standard group, swallowing of different amounts, swallowing of different concentrations, and swallowing of different states. The standard group is configured as no swallowing action, the swallowing of different amounts is configured as empty swallowing, single swallowing, single repeated swallowing, and continuous swallowing, the swallowing of different concentrations is configured as low concentration swallowing and high concentration swallowing, and the swallowing of different states is configured as semi-solid swallowing, liquid swallowing, and solid swallowing.

[0006] Furthermore, the near-infrared brain functional imaging information is configured as information obtained by detecting preliminary motor and sensory cortical areas.

[0007] Furthermore, the classifier adopts a multi-weak classifier signal information fusion classification method based on a parallel multi-source information fusion algorithm. Based on the parallel multi-source information fusion algorithm structure, a multi-weak classifier signal information fusion classification system is designed according to the number of weak classifiers and input data, and a multimodal information fusion method is used to fuse the weak classifier results.

[0008] The second object of the present invention is to provide a swallowing function evaluation method based on multimodal fusion, based on the above system, comprising the following steps: Acquire swallowing image information of the patient's swallowing pattern, swallowing muscle group electromyography array information, and near-infrared brain functional imaging information collected synchronously; Constructing a relationship mapping model from the swallowing image information to the swallowing muscle group electromyographic array information; Based on the near-infrared brain function imaging information and the swallowing muscle group electromyographic array information, a classification algorithm is used to realize pattern recognition, and the swallowing function evaluation result is obtained by combining the relationship mapping model and the recognized pattern through classifier decision.

[0009] Furthermore, the construction of the relationship mapping model includes the following steps: Preprocessing and feature extraction of the swallowing image information and the swallowing muscle group electromyographic array information; The extracted swallowing image information features are integrated with the swallowing muscle group electromyography array information features; A relationship mapping model is established through the fused features.

[0010] Furthermore, the classifier adopts a multi-weak classifier signal information fusion classification method based on a parallel multi-source information fusion algorithm. Based on the parallel multi-source information fusion algorithm structure, a multi-weak classifier signal information fusion classification system is designed according to the number of weak classifiers and input data, and a multimodal information fusion method is used to fuse the weak classifier results.

[0011] A third object of the present invention is to provide a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0012] A fourth object of the present invention is to provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0013] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a swallowing function evaluation system and method, device and medium based on multimodal fusion. A human swallowing evaluation model is constructed by using swallowing muscle group sEMG array image information, near-infrared brain function imaging information and swallowing image information. A self-developed multi-channel array flexible electromyography acquisition sensor is used to construct an association mapping model from the internal structure of swallowing angiography to the external muscle point information. The a priori nature of electromyography information and near-infrared brain imaging information is utilized, and a classifier algorithm is used to realize the evaluation and prediction of swallowing disorders, providing technical support for the application of near-infrared technology in swallowing evaluation.

[0014] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. The specific implementation of the present invention is given in detail by the following embodiments and their accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 Schematic diagram of a swallowing function evaluation system based on multimodal fusion provided in an embodiment; Figure 2 A schematic diagram of an 8-channel array flexible electromyography acquisition sensor provided in an embodiment; Figure 3 A structural diagram of an 8-channel array flexible electromyography acquisition sensor provided in an embodiment; Figure 4 A schematic diagram of the attachment position of the swallowing muscle group provided in the embodiment; Figure 5 A schematic diagram of the near-infrared brain region of interest provided in the embodiment; Figure 6 A schematic diagram of the swallowing experiment provided in the embodiment; Figure 7 It is a flow chart of the swallowing function evaluation method based on multimodal fusion provided in the embodiment; Figure 8 A schematic diagram of a computer device provided in an embodiment; Fig. 9 A schematic diagram of a computer-readable storage medium provided in an embodiment. DETAILED DESCRIPTION

[0016] The present invention is further described below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. It should be noted that, under the premise of no conflict, the embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0017] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0018] The figure numbers in this application are only used to distinguish the various steps in the scheme, and are not used to limit the execution order of the various steps. The specific execution order is subject to the description in the specification.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0020] Example 1 A swallowing function evaluation system based on multimodal fusion, such as Figure 1 As shown, it includes swallowing imaging equipment, multi-channel array flexible electromyography acquisition sensor, near-infrared brain function imaging equipment, and main controller; wherein, The swallowing imaging device is used to perform swallowing imaging on the patient's swallowing pattern to collect swallowing image information; The multi-channel array flexible electromyography acquisition sensor is used to synchronously acquire the electromyographic signals of the swallowing muscle group during the swallowing angiography process to obtain the electromyographic array information of the swallowing muscle group; The near-infrared brain function imaging device is used to synchronously collect near-infrared brain function imaging information during swallowing angiography; The main controller is used to construct a relationship mapping model from the swallowing image information to the swallowing muscle group electromyographic array information, and to realize swallowing pattern recognition based on the near-infrared brain functional imaging information and the swallowing muscle group electromyographic array information by using a classification algorithm. The relationship mapping model and the identified swallowing pattern are combined with the classifier decision to obtain a swallowing function evaluation result.

[0021] Since it is impossible to collect the electrophysiological activities of various muscles during the complete swallowing process simply by using a few surface electrodes, in order to collect the electromyographic signals of the neck, combined with the position arrangement of the suprahyoid and infrahyoid muscles in the neck, this embodiment designs an 8-channel array flexible electromyographic acquisition sensor suitable for swallowing muscle group detection, such as Figure 2 , Figure 3 shown.

[0022] Specifically, the multi-channel array flexible electromyography acquisition sensor includes a transparent PET layer, a silver chloride circuit, a carbon paste circuit, insulating oil, a hydrogel, a medical tape, and a release film; the silver chloride circuit is laid on the transparent PET layer, the carbon paste circuit is electrically connected to the silver chloride circuit, the silver chloride circuit is connected to the hydrogel, insulating oil is arranged around the hydrogel, the medical tape is used to fix the multi-channel array flexible electromyography acquisition sensor at the position to be measured, and the release film covers the hydrogel and the medical tape.

[0023] Due to the differences in height and body shape of users, the muscle group positions of different users will not be exactly the same. The multi-channel array flexible electromyographic acquisition sensor provided in this embodiment is deformable and can be deformed within a certain range, with better adhesion and biocompatibility. For the muscle groups active during swallowing, the muscle group acquisition positions selected in this embodiment are as follows: Figure 4 shown.

[0024] Conventional flexible electromyography sensors are usually made of magnetic materials, which have limitations in use under CT environments. The 8-channel array flexible electromyography sensor provided in this embodiment can be used normally during swallowing angiography.

[0025] Swallowing radiography is the gold standard for swallowing disorder assessment, but there is a risk of excessive radiation dose. In order to reduce the radiation dose and enhance the accuracy of myoelectric assessment, the 8-channel array flexible myoelectric acquisition sensor provided in this embodiment is used to complete the relevant swallowing tasks during swallowing radiography, realize the synchronous acquisition of swallowing myoelectric information and swallowing image information, and map the swallowing image information to the swallowing myoelectric array information, constructing a correlation relationship from the outside to the inside, thereby providing a better explanation of the myoelectric information.

[0026] Specifically, the construction of the relationship mapping model includes the following steps: Preprocessing and feature extraction of the swallowing image information and the swallowing muscle group electromyographic array information; The extracted swallowing image information features are integrated with the swallowing muscle group electromyography array information features; A relationship mapping model is established through the fused features.

[0027] In some embodiments, Figure 6 As shown, the swallowing patterns include a standard group, swallowing of different amounts, swallowing of different concentrations, and swallowing of different states. The standard group is configured as no swallowing action, the swallowing of different amounts is configured as empty swallowing, single swallowing, single repeated swallowing, and continuous swallowing, the swallowing of different concentrations is configured as low concentration swallowing and high concentration swallowing, and the swallowing of different states is configured as semi-solid swallowing, liquid swallowing, and solid swallowing.

[0028] Optionally, experimental setup: In order to study the effects of different volume, viscosity, head posture and other factors on the swallowing process, the swallowing experiment is divided into the following different combinations: Head position: Keep the head in a straight forward position; (1) Rest: Do not make any movements, keep your head still, for 5 seconds; (2) Swallowing of different amounts: empty swallowing (swallowing saliva - dry swallowing), single swallowing (clear water - 5ml / 10ml / 15ml), single repeated swallowing (clear water - 10ml, repeated 10 times), continuous swallowing (clear water - 50ml / 100ml); (3) Swallowing at different concentrations: low concentration (100 ml water + 20 g powder) (5 ml / 10 ml), high concentration (100 ml water + 35 g powder) (5 ml / 10 ml); (4) Swallowing in different states: semi-solid (similar to pudding) (3-5 spoons), liquid (water) (5 ml), solid (a small piece of dry bread or biscuits, etc.).

[0029] Since the swallowing process may involve the activation of multiple brain regions, this embodiment selects the preliminary motor and sensory cortical regions for detection, as shown in the following example. Figure 5 Specifically, the regions of interest were located in the middle and lower parts of the left and right precentral and postcentral gyri, which were active during swallowing but inactive during finger tapping. The mean, peak, and root mean square values ​​of hemoglobin (HbO) and their combinations were calculated as classification features.

[0030] This embodiment is mainly divided into two parts for experimental setting. First, based on swallowing imaging technology and surface electromyography signal experimental data collection, by implementing Figure 6The experimental content is to construct the association mapping from swallowing angiography to surface array electromyography information, and realize the construction of digital models of the two; secondly, collect near-infrared brain functional imaging information and swallowing muscle group electromyography information, and provide better interpretability for near-infrared brain functional imaging information through the established association mapping model from swallowing angiography to surface array electromyography information. The classifier model is used to perform multi-modal data fusion of near-infrared brain functional imaging information and swallowing muscle group electromyography information, and the classifier is used to classify swallowing assessment.

[0031] In order to address the problem of low swallowing function evaluation rate of a single classifier, the classifier in this embodiment adopts a multi-weak classifier signal information fusion classification method based on a parallel multi-source information fusion algorithm. Based on the parallel multi-source information fusion algorithm structure and according to the number of weak classifiers and input data, a multi-weak classifier signal information fusion classification system is designed, and the weak classifier results are fused using a multimodal information fusion method.

[0032] This embodiment aims at swallowing dysfunction, and uses high-density sEMG array information at the neck and masseter muscle, near-infrared brain functional imaging signals of the head and swallowing angiography technology to achieve swallowing disorder assessment, and realizes swallowing assessment through a multimodal data fusion algorithm.

[0033] Example 2 A swallowing function evaluation method based on multimodal fusion is based on the swallowing function evaluation system based on multimodal fusion provided in Example 1. For a detailed description of the system, please refer to the corresponding description in the above system embodiment, which will not be repeated here. Figure 7 As shown, the method comprises the following steps: Acquire swallowing image information of the patient's swallowing pattern, swallowing muscle group electromyography array information, and near-infrared brain functional imaging information collected synchronously; Among them, swallowing imaging information is obtained by swallowing imaging by using a swallowing imaging device to perform swallowing imaging on the patient's swallowing pattern; The swallowing muscle group electromyographic array information is obtained by synchronously collecting the electromyographic signals of the swallowing muscle group during the swallowing angiography process through a multi-channel array flexible electromyographic acquisition sensor; The specific structure and principle of the multi-channel array flexible electromyographic acquisition sensor refer to the relevant description in the swallowing function evaluation system based on multimodal fusion provided in Example 1, and will not be repeated here.

[0034] Near-infrared brain functional imaging information is obtained by synchronously collecting near-infrared brain functional imaging during swallowing angiography using near-infrared brain functional imaging equipment.

[0035] Since the swallowing process may involve the activation of multiple brain regions, this embodiment selects the preliminary motor and sensory cortical regions for detection, as shown in the following example. Figure 5Specifically, the regions of interest were located in the middle and lower parts of the left and right precentral and postcentral gyri, which were active during swallowing but inactive during finger tapping. The mean, peak, and root mean square values ​​of hemoglobin (HbO) and their combinations were calculated as classification features.

[0036] The specific setting of the swallowing mode refers to the relevant description in the swallowing function evaluation system based on multimodal fusion provided in Example 1, which will not be repeated here.

[0037] Constructing a relationship mapping model from the swallowing image information to the swallowing muscle group electromyographic array information; Based on the near-infrared brain function imaging information and the swallowing muscle group electromyographic array information, a classification algorithm is used to realize swallowing pattern recognition, and the swallowing function evaluation result is obtained by combining the relationship mapping model and the recognized swallowing pattern through classifier decision.

[0038] This embodiment is mainly divided into two parts for experimental setting. First, based on swallowing imaging technology and surface electromyography signal experimental data collection, by implementing Figure 6 The experimental content is to construct the association mapping from swallowing angiography to surface array electromyography information, and realize the construction of digital models of the two; secondly, collect near-infrared brain functional imaging information and swallowing muscle group electromyography information, and provide better interpretability for near-infrared brain functional imaging information through the established association mapping model from swallowing angiography to surface array electromyography information. The classifier model is used to perform multi-modal data fusion of near-infrared brain functional imaging information and swallowing muscle group electromyography information, and the classifier is used to classify swallowing assessment.

[0039] In some embodiments, the construction of the relationship mapping model includes the following steps: Preprocessing and feature extraction of the swallowing image information and the swallowing muscle group electromyographic array information; The extracted swallowing image information features are integrated with the swallowing muscle group electromyography array information features; A relationship mapping model is established through the fused features.

[0040] In order to address the problem of low swallowing function evaluation rate of a single classifier, the classifier in this embodiment adopts a multi-weak classifier signal information fusion classification method based on a parallel multi-source information fusion algorithm. Based on the parallel multi-source information fusion algorithm structure and according to the number of weak classifiers and input data, a multi-weak classifier signal information fusion classification system is designed, and the weak classifier results are fused using a multimodal information fusion method.

[0041] This embodiment aims at swallowing dysfunction, and uses high-density sEMG array information at the neck and masseter muscle, near-infrared brain functional imaging signals of the head and swallowing angiography technology to achieve swallowing disorder assessment, and realizes swallowing assessment through a multimodal data fusion algorithm.

[0042] Example 3 A computer device such as Figure 8 As shown, it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, a swallowing function evaluation method based on multimodal fusion is implemented. For a detailed description of the method, reference may be made to the corresponding description in the above method embodiment, which will not be repeated here.

[0043] Example 4 A computer readable storage medium such as Fig. 9 As shown, the computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, a swallowing function evaluation method based on multimodal fusion is implemented. For a detailed description of the method, reference may be made to the corresponding description in the above method embodiment, which will not be repeated here.

[0044] The number of devices and processing scales described here are used to simplify the description of the present invention. Applications, modifications and variations of the present invention will be obvious to those skilled in the art.

[0045] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and implementation modes. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

[0046] The apparatus, electronic device, non-volatile computer storage medium and method provided in the embodiments of this specification correspond to each other, and therefore, the apparatus, electronic device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, electronic device and non-volatile computer storage medium will not be repeated here.

[0047] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software units for implementing the method and structures within the hardware component.

[0048] The systems, devices or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions in various units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or more software and / or hardware.

[0049] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0050] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0051] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0052] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0053] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0054] The specification may be described in the general context of computer-executable instructions executed by a computer, such as program units. Generally, program units include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program units may be located in local and remote computer storage media, including storage devices.

[0055] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0056] The above description is only an embodiment of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of one or more embodiments of this specification.

Claims

1. A swallowing function evaluation system based on multimodal fusion, characterized by: It includes swallowing imaging equipment, multi-channel array flexible electromyography acquisition sensor, near-infrared brain function imaging equipment, and main controller; among them, The swallowing imaging device is used to perform swallowing imaging on the patient's swallowing pattern to collect swallowing image information; The multi-channel array flexible electromyography acquisition sensor is used to synchronously acquire the electromyographic signals of the swallowing muscle group during the swallowing angiography process to obtain the electromyographic array information of the swallowing muscle group; The near-infrared brain function imaging device is used to synchronously collect near-infrared brain function imaging information during swallowing angiography; The main controller is used to construct a relationship mapping model from the swallowing image information to the swallowing muscle group electromyographic array information, and to implement pattern recognition based on the near-infrared brain functional imaging information and the swallowing muscle group electromyographic array information using a classification algorithm. The relationship mapping model and the recognized pattern are combined through classifier decision to obtain a swallowing function evaluation result.

2. A swallowing function evaluation system based on multimodal fusion as claimed in claim 1, characterized in that: The multi-channel array flexible electromyography acquisition sensor includes a transparent PET layer, a silver chloride circuit, a carbon paste circuit, insulating oil, a hydrogel, a medical tape, and a release film; the silver chloride circuit is laid on the transparent PET layer, the carbon paste circuit is electrically connected to the silver chloride circuit, the silver chloride circuit is connected to the hydrogel, insulating oil is arranged around the hydrogel, the medical tape is used to fix the multi-channel array flexible electromyography acquisition sensor at a position to be measured, and the release film covers the hydrogel and the medical tape.

3. The swallowing function evaluation system based on multimodal fusion according to claim 1, characterized in that: The swallowing patterns include a standard group, swallowing of different amounts, swallowing of different concentrations, and swallowing of different states. The standard group is configured as no swallowing action, the swallowing of different amounts is configured as empty swallowing, single swallowing, single repeated swallowing, and continuous swallowing, the swallowing of different concentrations is configured as low concentration swallowing and high concentration swallowing, and the swallowing of different states is configured as semi-solid swallowing, liquid swallowing, and solid swallowing.

4. The swallowing function evaluation system based on multimodal fusion according to claim 1, characterized in that: The near-infrared brain functional imaging information is configured as information obtained by detecting preliminary motor and sensory cortical areas.

5. The swallowing function evaluation system based on multimodal fusion according to claim 1, characterized in that: The classifier adopts a multi-weak classifier signal information fusion classification method based on a parallel multi-source information fusion algorithm. Based on the parallel multi-source information fusion algorithm structure, a multi-weak classifier signal information fusion classification system is designed according to the number of weak classifiers and input data, and a multimodal information fusion method is used to fuse the weak classifier results.

6. A swallowing function evaluation method based on multimodal fusion, based on the system according to any one of claims 1 to 5, characterized in that: The following steps are involved: Acquire swallowing image information of the patient's swallowing pattern, swallowing muscle group electromyography array information, and near-infrared brain functional imaging information collected synchronously; Constructing a relationship mapping model from the swallowing image information to the swallowing muscle group electromyographic array information; Based on the near-infrared brain function imaging information and the swallowing muscle group electromyographic array information, a classification algorithm is used to realize pattern recognition, and the swallowing function evaluation result is obtained by combining the relationship mapping model and the recognized pattern through classifier decision.

7. The swallowing function evaluation method based on multimodal fusion according to claim 6, characterized in that: The construction of the relationship mapping model includes the following steps: Preprocessing and feature extraction of the swallowing image information and the swallowing muscle group electromyographic array information; The extracted swallowing image information features are integrated with the swallowing muscle group electromyography array information features; A relationship mapping model is established through the fused features.

8. The swallowing function evaluation method based on multimodal fusion according to claim 7, characterized in that: The classifier adopts a multi-weak classifier signal information fusion classification method based on a parallel multi-source information fusion algorithm. Based on the parallel multi-source information fusion algorithm structure, a multi-weak classifier signal information fusion classification system is designed according to the number of weak classifiers and input data, and a multimodal information fusion method is used to fuse the weak classifier results.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 6 to 8 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 6 to 8 is implemented.

Images