Scoliosis evaluation system and device based on multi-modal information fusion

Abstract

The invention discloses a scoliosis evaluation system and device based on multi-modal information fusion, and belongs to the field of spine health. A back pressure electromyographic signal acquisition module acquires stress electromyographic information of a back muscle group; the thermal imaging module obtains a thermal imaging information graph of a human body; the plantar pressure acquisition module acquires plantar pressure information; the multi-mode data fusion module inputs the stress myoelectricity information, the thermal imaging information graph and the plantar pressure information into the encoder layer, a cross attention mechanism is adopted in the feature matching layer to fuse features from different modules, a classifier model is used for scoliosis evaluation, and classification is conducted according to the scoliosis degree. According to the application, the collaborative mechanism of the stress myoelectricity information, the thermal imaging information graph and the plantar pressure information of the back during scoliosis is utilized, multi-modal data fusion is performed on the stress myoelectricity information, the thermal imaging information graph and the plantar pressure information of the back, full-automatic evaluation is realized through a multi-modal data fusion model, the evaluation accuracy is high, and no radiation is generated.

Description

Technical Field

[0001] The present invention relates to the field of spinal health, and in particular to a scoliosis assessment system and device based on multimodal information fusion. Background Art

[0002] With the increase in the number of office workers, the number of people with scoliosis is increasing, mostly in adolescents, which seriously affects physical development and health. Medical imaging is the most commonly used method to evaluate and treat scoliosis. However, the imaging method of medical imaging is insufficient in providing real-time feedback and there are radiation dose issues. Summary of the invention

[0003] In order to overcome the deficiencies of the prior art, one of the objectives of the present invention is to provide a scoliosis assessment system based on multimodal information fusion that can accurately assess scoliosis.

[0004] In order to overcome the deficiencies of the prior art, a second object of the present invention is to provide a scoliosis assessment device based on multimodal information fusion that can accurately assess scoliosis.

[0005] One of the purposes of the present invention is achieved by the following technical solution:

[0006] A scoliosis assessment system based on multimodal information fusion, including

[0007] Back pressure electromyographic signal acquisition module, used to collect stress electromyographic information of back muscles;

[0008] Thermal imaging module, which obtains thermal imaging information of the human body surface;

[0009] A plantar pressure collection module collects plantar pressure information;

[0010] The multimodal data fusion module constructs a multimodal data fusion network model including an encoder layer, a feature matching layer and a prediction layer. The stress electromyography information, the thermal imaging information map and the plantar pressure information are input into the encoder layer. The cross-attention mechanism is used in the feature matching layer to fuse the features from different modules. In the prediction layer, the vectors of each modality are converted into the same fixed length using a weighted pool. A fusion gate is constructed using MLP and they are combined with weights. Finally, a classifier model is used to evaluate scoliosis and classify it according to the degree of scoliosis.

[0011] Furthermore, the back pressure electromyographic signal acquisition module includes a multi-channel array flexible electromyographic acquisition sensor, and the stress electromyographic information of the back muscle group is collected by the multi-channel array flexible electromyographic acquisition sensor.

[0012] Furthermore, the thermal imaging information captured by the thermal imaging module is a back thermal imaging information map.

[0013] Furthermore, the plantar pressure information is array information.

[0014] Furthermore, the encoder layer includes a back electromyographic pressure signal encoder, a thermal imaging image encoder and a plantar pressure data encoder. The stress electromyographic information is input into the back electromyographic pressure signal encoder, the thermal imaging information map is input into the thermal imaging image encoder, and the plantar pressure information is input into the plantar pressure data encoder.

[0015] Furthermore, the feature matching layer includes three matching blocks, and the three matching blocks iteratively learn the attention-aware representation of each modality.

[0016] Furthermore, the feature matching layer also includes an extraction module, which extracts useful features of the stress electromyography information, the thermal imaging information map and the plantar pressure information.

[0017] The second object of the present invention is achieved by adopting the following technical solution:

[0018] A scoliosis assessment device based on multimodal information fusion includes an electromyography acquisition sensor, a thermal imaging camera, a plantar array pressure sensor and a processor. The processor is communicatively connected with the electromyography acquisition sensor, the thermal imaging camera and the plantar array pressure sensor. The electromyography acquisition sensor acquires stress electromyography information of back muscles, the thermal imaging camera obtains thermal imaging information of the human body surface, and the plantar array pressure sensor acquires plantar pressure information. A multimodal data fusion model is provided in the processor. The multimodal data fusion model extracts useful features of sensor data, and uses a cross-attention mechanism to fuse features from different sensors, uses a weighted pool to convert the vector of each modality into the same fixed length, uses a classifier model to assess scoliosis, and classifies according to the degree of scoliosis.

[0019] Furthermore, the electromyography acquisition sensor is a flexible sensor.

[0020] Compared with the existing technology, the scoliosis assessment system based on multimodal information fusion of the present invention utilizes the synergistic mechanism of the stress electromyographic information, thermal imaging information map and plantar pressure information of the back during scoliosis, fuses the multimodal data of the stress electromyographic information, thermal imaging information map and plantar pressure information of the back, and realizes fully automated assessment through the multimodal data fusion model, with high assessment accuracy and no radiation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a schematic diagram of the attachment position of the back pressure electromyographic sensor when the spine is not scoliotic;

[0022] Figure 2 This is a schematic diagram of the back pressure electromyographic sensor deforming during scoliosis;

[0023] Figure 3 This is a schematic diagram of thermal imaging of the human back;

[0024] Figure 4 This is a schematic diagram of plantar pressure array information collection;

[0025] Figure 5 Schematic diagram of the scoliosis network model. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] It should be noted that when a component is referred to as being "fixed to" another component, it may be directly on the other component or there may be another intermediate component through which it is fixed. When a component is considered to be "connected to" another component, it may be directly connected to the other component or there may be another intermediate component at the same time. When a component is considered to be "set on" another component, it may be directly set on the other component or there may be another intermediate component at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0029] The present invention provides a scoliosis assessment system based on multimodal information fusion, comprising

[0030] Back pressure electromyographic signal acquisition module, used to collect stress electromyographic information of back muscles;

[0031] Thermal imaging module, which obtains thermal imaging information of the human body surface;

[0032] A plantar pressure collection module collects plantar pressure information;

[0033] The multimodal data fusion module constructs a multimodal data fusion network model including an encoder layer, a feature matching layer and a prediction layer. The stress electromyography information, the thermal imaging information map and the plantar pressure information are input into the encoder layer. The cross-attention mechanism is used in the feature matching layer to fuse the features from different modules. In the prediction layer, the vectors of each modality are converted into the same fixed length using a weighted pool. A fusion gate is constructed using MLP and they are combined with weights. Finally, a classifier model is used to evaluate scoliosis and classify it according to the degree of scoliosis.

[0034] Specifically, the back pressure EMG signal acquisition module uses an EMG acquisition sensor to collect stress EMG information of the back muscles. Since it is impossible to collect the complete electrophysiological activity of the back simply by using a few surface electrodes, a multi-channel array flexible EMG acquisition sensor suitable for back EMG and pressure signal detection is designed in combination with the position arrangement of the back muscles for EMG signal acquisition of the back.

[0035] Due to the differences in height and body shape of users, the muscle group positions of different users cannot be exactly the same. The electromyographic acquisition sensor in this patent is a flexible sensor with deformability. It can be deformed within a certain range and has better adhesion and biocompatibility. When the spine is not scoliotic, the muscle group acquisition position is as follows: Figure 1 When the spine is scoliotic, the muscle groups are located as shown in Figure 2 As shown in the figure, the electromyographic signal diagram and the back stress change diagram of the back can be obtained through the electromyographic acquisition sensor.

[0036] The thermal imaging module uses a thermal imaging camera to obtain thermal images of the body surface of the human body. One of the characteristics that reflects the correct body structure and posture is the thermal symmetry of the body surface, that is, the skin temperature, and asymmetry may be a symptom of different background diseases. Thermal imaging images can be used as a method for evaluating musculoskeletal dysfunction, especially in scoliosis. Thermal imaging is a useful and non-invasive method to evaluate muscle tension imbalance during scoliosis. In the case of scoliosis, there are areas of obvious thermal asymmetry on the body surface of the upper back, and the spinal rotation angle is highly positively correlated with the degree of thermal asymmetry. Therefore, when using thermal vision to assist in the diagnosis of scoliosis, detailed thermal assessments of the upper back, chest, thighs, and posterior calves can be performed. Thermal imaging cameras can capture thermal images of the body surface of the human body, such as Figure 3 shown.

[0037] The plantar pressure collection module collects plantar pressure information through a plantar array pressure sensor. Under normal circumstances, the plantar pressure information of people with normal spines conforms to a certain distribution, and the information of the left and right feet should be symmetrical. Poor plantar force will also have a negative impact on the spine. This application installs a plantar array pressure sensor on the sole of the subject's foot to detect the subject's foot pressure information during exercise, and collects plantar pressure array images such as Figure 4 shown.

[0038] In order to effectively utilize multimodal data and improve the evaluation accuracy, this application adopts a network model of multimodal data fusion. The network model is as follows Figure 5 As shown in the figure, the network model mainly consists of three parts, including encoder layer, feature matching layer and prediction layer. The encoder layer is divided into back electromyography pressure signal encoder, thermal imaging image encoder and plantar pressure data encoder, which input preprocessed data respectively. In the feature matching layer, a matching layer consisting of 3 matching blocks is used to iteratively learn the attention perception representation of each modality. In the feature extraction layer, the data of each sensor passes through the corresponding feature extraction module to extract useful features of the sensor data. In the feature fusion layer, the cross attention mechanism is used to fuse the features from different sensors to obtain more comprehensive and consistent environmental perception information. In the prediction layer, the vectors of each modality are converted to the same fixed length using weighted pooling, a fusion gate is constructed using MLP, and they are combined with the weighted sum. Finally, the classifier model is used to evaluate scoliosis and classify it according to its scoliosis degree.

[0039] The present application also discloses a scoliosis assessment device based on multimodal information fusion, including an electromyography acquisition sensor, a thermal imaging camera, a plantar array pressure sensor and a processor, wherein the processor is communicatively connected with the electromyography acquisition sensor, the thermal imaging camera and the plantar array pressure sensor, wherein the electromyography acquisition sensor acquires stress electromyography information of the back muscle group, the thermal imaging camera obtains a thermal imaging information map of the human body surface, and the plantar array pressure sensor acquires plantar pressure information. A multimodal data fusion model is provided in the processor, wherein the multimodal data fusion model extracts useful features of sensor data, and adopts a cross-attention mechanism to fuse features from different sensors, and uses a weighted pool to convert the vector of each modality into the same fixed length, and uses a classifier model to perform scoliosis assessment and classify according to the degree of scoliosis.

[0040] Furthermore, the electromyography acquisition sensor is a flexible sensor.

[0041] Compared with the existing technology, the scoliosis assessment system based on multimodal information fusion of the present invention utilizes the synergistic mechanism of the stress electromyographic information, thermal imaging information map and plantar pressure information of the back during scoliosis, fuses the multimodal data of the stress electromyographic information, thermal imaging information map and plantar pressure information of the back, and realizes fully automated assessment through the multimodal data fusion model, with high assessment accuracy and no radiation.

[0042] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which are equivalent modifications and improvements made to the above embodiments based on the essential technology of the present invention, and all of them belong to the protection scope of the present invention.

Claims

1. A scoliosis assessment system based on multimodal information fusion, characterized by: include Back pressure electromyographic signal acquisition module, used to collect stress electromyographic information of back muscles; Thermal imaging module, which obtains thermal imaging information of the human body surface; A plantar pressure collection module collects plantar pressure information; The multimodal data fusion module constructs a multimodal data fusion network model including an encoder layer, a feature matching layer and a prediction layer. The stress electromyography information, the thermal imaging information map and the plantar pressure information are input into the encoder layer. The cross-attention mechanism is used in the feature matching layer to fuse the features from different modules. In the prediction layer, the vectors of each modality are converted into the same fixed length using a weighted pool. A fusion gate is constructed using MLP and they are combined with weights. Finally, a classifier model is used to evaluate scoliosis and classify it according to the degree of scoliosis.

2. The scoliosis assessment system based on multimodal information fusion according to claim 1, characterized in that: The back pressure electromyographic signal acquisition module includes a multi-channel array flexible electromyographic acquisition sensor, and the stress electromyographic information of the back muscle group is collected through the multi-channel array flexible electromyographic acquisition sensor.

3. The scoliosis assessment system based on multimodal information fusion according to claim 1, characterized in that: The thermal imaging information captured by the thermal imaging module is a back thermal imaging information map.

4. The scoliosis assessment system based on multimodal information fusion according to claim 1, characterized in that: The plantar pressure information is array information.

5. The scoliosis assessment system based on multimodal information fusion according to claim 1, characterized in that: The encoder layer includes a back electromyographic pressure signal encoder, a thermal imaging image encoder and a plantar pressure data encoder. The stress electromyographic information is input into the back electromyographic pressure signal encoder, the thermal imaging information map is input into the thermal imaging image encoder, and the plantar pressure information is input into the plantar pressure data encoder.

6. The scoliosis assessment system based on multimodal information fusion according to claim 1, characterized in that: The feature matching layer includes three matching blocks, which iteratively learn the attention-aware representation of each modality.

7. The scoliosis assessment system based on multimodal information fusion according to claim 6, characterized in that: The feature matching layer also includes an extraction module, which extracts useful features of the stress electromyography information, the thermal imaging information map and the plantar pressure information.

8. A scoliosis assessment device based on multimodal information fusion, characterized in that: It includes an electromyography acquisition sensor, a thermal imaging camera, a plantar array pressure sensor and a processor. The processor is communicatively connected with the electromyography acquisition sensor, the thermal imaging camera and the plantar array pressure sensor. The electromyography acquisition sensor acquires stress electromyography information of the back muscles, the thermal imaging camera obtains a thermal imaging information map of the human body surface, and the plantar array pressure sensor acquires plantar pressure information. A multi-modal data fusion model is provided in the processor. The multi-modal data fusion model extracts useful features of sensor data, and uses a cross-attention mechanism to fuse features from different sensors, and uses a weighted pool to convert the vector of each mode into the same fixed length. A classifier model is used to evaluate scoliosis and classify it according to the degree of scoliosis.

9. The scoliosis assessment device based on multimodal information fusion according to claim 8, characterized in that: The electromyography acquisition sensor is a flexible sensor.

Images