Bone parameter estimation method and device based on multi-modal information fusion

Abstract

The invention provides a skeleton parameter estimation method and device based on multi-modal information fusion, and relates to the technical field of medical image analysis, and the method comprises the steps: obtaining the X-ray image data, depth image data, inertial measurement unit sensor data and electromyogram signal data of a human skeleton; performing three-dimensional reconstruction on the human skeleton according to the X-ray image data and the depth image data to obtain three-dimensional reconstruction data; performing data fusion on the X-ray image data, the inertial measurement unit sensor data, the electromyogram signal data and the three-dimensional reconstruction data to obtain multi-modal fusion data; establishing a skeleton parameter estimation model based on the multi-modal fusion data and a mapping relation between skeleton structure features and skeleton parameters; and performing skeleton parameter estimation by using the skeleton parameter estimation model. According to the method, accurate skeleton parameter estimation can be provided in multiple application scenes such as dynamic motion analysis, motion rehabilitation and intelligent wearable equipment.

Description

Background Art

[0002] With the rapid development of medical imaging technology and computer vision, bone analysis plays an increasingly important role in medical diagnosis, sports rehabilitation and health monitoring.

[0003] Traditional bone analysis bone parameter estimation methods based on multimodal information fusion usually rely on a single medical imaging modality such as X-ray images and computed tomography (CT) for bone parameter estimation. Although these bone parameter estimation methods based on multimodal information fusion can provide certain bone structure information, due to the limited two-dimensional or three-dimensional viewing angles of X-ray images and CT images, they cannot fully reflect the dynamic changes of the human skeleton and its spatial structure.

[0004] In addition, although modern technologies such as magnetic resonance imaging (MRI) can provide higher-resolution three-dimensional bone images, these devices are usually expensive and complicated to operate, and there are still significant limitations in the analysis of bones in dynamic movements or complex movement postures. For example, they are not applicable to high-frequency movement monitoring, real-time feedback, and personalized health management, and are difficult to apply to dynamic monitoring or sports rehabilitation therapy in daily life.

[0005] On the other hand, although sensor technologies such as inertial measurement units (IMUs) have been widely used in recent years, these sensors can capture the dynamic changes of bones in real-time motion, but existing technologies often use them alone and fail to fully combine medical imaging data for multimodal data fusion. Bone analysis based on single sensor data cannot obtain comprehensive and accurate bone parameters. Especially in complex motion states, IMU data may be affected by posture changes and noise, resulting in low accuracy of the results. Summary of the invention

[0006] In order to overcome the problems existing in the related art, the present invention provides a bone parameter estimation method and device based on multimodal information fusion.

[0007] According to a first aspect of an embodiment of the present invention, a method for estimating bone parameters based on multimodal information fusion is provided, and the method for estimating bone parameters based on multimodal information fusion comprises:

[0008] Obtaining X-ray image data, depth image data, inertial measurement unit sensor data, and electromyography signal data of human skeleton;

[0009] Performing three-dimensional reconstruction of the human skeleton according to the X-ray image data and the depth image data to obtain three-dimensional reconstruction data;

[0010] fusing the X-ray image data, the inertial measurement unit sensor data, the electromyography signal data and the three-dimensional reconstruction data to obtain multimodal fusion data;

[0011] Based on the multimodal fusion data and the mapping relationship between the bone structure features and the bone parameters, a bone parameter estimation model is established;

[0012] The bone parameter estimation model is used to estimate bone parameters.

[0013] In some example embodiments of the present invention, based on the above scheme, the X-ray image data, the depth image data, the inertial measurement unit sensor data, the electromyography signal data and the three-dimensional reconstruction data are fused to obtain multimodal fusion data, including:

[0014] Preprocessing the X-ray image data, the inertial measurement unit sensor data, the electromyography signal data, and the three-dimensional reconstruction data respectively to obtain preprocessed X-ray image data, preprocessed inertial measurement unit sensor data, preprocessed electromyography signal data, and preprocessed three-dimensional reconstruction data;

[0015] Extracting bone structure features from the preprocessed radiographic image data, dynamic motion features from the preprocessed inertial measurement unit sensor data, muscle activity features from the preprocessed electromyography signal data, and bone spatial structure features from the preprocessed three-dimensional reconstruction data;

[0016] Linearly mapping the skeletal structure feature, the dynamic motion feature, the muscle activity feature, and the skeletal spatial structure feature respectively to obtain a query, a key, and a value corresponding to each feature;

[0017] Calculate the attention score of each feature based on the query, key, and value corresponding to each feature;

[0018] Based on the attention score of each feature, all features are adaptively weighted to obtain the multimodal fusion data.

[0019] In some exemplary embodiments of the present invention, based on the above scheme, the bone parameter estimation model includes:

[0020] The bone structure analysis part and the bone parameter regression part, the bone structure analysis part is responsible for the bone structure analysis task, including multiple first fully connected layers and ReLU activation function layers set in succession; the bone parameter regression part includes a second fully connected layer, which is used to estimate bone density, bone strength and osteoporosis degree according to the analysis results of the bone structure.

[0021] In some exemplary embodiments of the present invention, based on the aforementioned scheme, the analysis results of the skeletal structure include joint angles, bone lengths, and joint range of motion;

[0022] The joint angle analysis process includes: calculating the minimum extension angle and the maximum bending angle of the joint angle according to the direction vectors of the two bones connecting the joint;

[0023] The analysis process of the joint range of motion includes: calculating the joint range of motion according to the minimum extension angle and the maximum bending angle;

[0024] The bone length analysis process includes: calculating the bone length according to the coordinates of the two endpoints of the bone using the Euclidean distance formula.

[0025] In some exemplary embodiments of the present invention, based on the above scheme, calculating the joint angle θ according to the direction vectors of the two bones connecting the joint includes:

[0026]

[0027] Among them, v1 and v2 are the direction vectors of the two bones connecting the joint.

[0028] In some exemplary embodiments of the present invention, based on the above scheme, according to the coordinates of the two endpoints of the bone, using the Euclidean distance formula, calculating the bone length includes:

[0029]

[0030] Among them, L represents the bone length, (x1, y1, z1) and (x2, y2, z2) represent the coordinates of the two endpoints of the bone respectively.

[0031] In some exemplary embodiments of the present invention, based on the above scheme, the loss function of the bone parameter estimation model is:

[0032] L=λ1·L s +λ2·L h

[0033] Among them, λ1 and λ2 are the weights of the bone structure analysis task and the bone parameter regression task respectively, L s represents the loss function of the bone structure analysis task and represents the predicted bone structure parameters, represents the true bone structure parameters, N represents the number of samples, i represents the i-th sample, and L h represents the loss function of the skeleton parameter regression task and represents the predicted bone parameters, Represents the actual bone parameters.

[0034] According to a second aspect of an embodiment of the present invention, a device based on the above-mentioned method for estimating skeleton parameters based on multimodal information fusion is provided, comprising:

[0035] A data acquisition module, used to acquire X-ray image data, depth image data, inertial measurement unit sensor data and electromyography signal data of human skeleton;

[0036] A three-dimensional reconstruction module, used to perform three-dimensional reconstruction of the human skeleton according to the X-ray image data and the depth image data to obtain three-dimensional reconstruction data;

[0037] A data fusion module, used for fusing the X-ray image data, the inertial measurement unit sensor data, the electromyography signal data and the three-dimensional reconstruction data to obtain multimodal fusion data;

[0038] A model building module, used to build a bone parameter estimation model based on the multimodal fusion data and the mapping relationship between the bone structure features and the bone parameters;

[0039] The parameter estimation module is used to perform bone parameter estimation using the bone parameter estimation model.

[0040] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the bone parameter estimation method based on multimodal information fusion in the first aspect is implemented.

[0041] According to a fourth aspect of an embodiment of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the bone parameter estimation method based on multimodal information fusion in the first aspect is implemented.

[0042] The technical solution provided by the embodiments of the present invention may have the following beneficial effects:

[0043] Compared with the traditional X-Ray single-modality solution, the present invention combines X-ray image data, depth image data, inertial measurement unit sensor data and electromyography signal data, and can integrate multiple information sources to make up for the shortcomings of single-modality data, thereby effectively improving the accuracy and precision of bone parameter estimation; three-dimensional reconstruction of bones based on X-ray image data and depth image data can accurately restore the three-dimensional morphology, joint position and bone density distribution of bones, thereby solving the limitations of traditional two-dimensional images, so that bone analysis is not limited to surface structure, but can deeply understand the spatial structure and morphology of bones; by fusing three-dimensional reconstruction data, IMU data, EMG signal data and X-ray image data, the structural characteristics and motion characteristics of bones can be captured, and a bone parameter estimation model can be established in combination with this information to enhance the connection between bone health assessment and structural analysis, while taking into account the dynamic movement and health status of bones, and improving the overall assessment effect; through the bone parameter estimation model, the bone structure analysis and bone parameter regression tasks are processed at the same time, which can improve the accuracy of bone parameter estimation, especially in complex postures and motion states. Therefore, the present invention has strong versatility and expansibility, and is suitable for multiple fields such as medical image analysis, sports rehabilitation, personalized medicine, etc., thereby providing precise support for bone health testing, disease diagnosis and sports evaluation.

[0044] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0046] Figure 1 A schematic diagram showing a system architecture of an exemplary application environment of a method and device for estimating bone parameters based on multimodal information fusion to which an embodiment of the present invention can be applied;

[0047] Figure 2 A schematic diagram of a flow chart of a method for estimating bone parameters based on multimodal information fusion according to some embodiments of the present invention is shown;

[0048] Figure 3 The figure schematically shows a data processing process of a method for estimating bone parameters based on multimodal information fusion according to some embodiments of the present invention, and includes a structure of a bone parameter estimation model;

[0049] Figure 4 A schematic diagram of a bone parameter estimation device based on multimodal information fusion according to some embodiments of the present invention is schematically shown;

[0050] Figure 5 A schematic diagram schematically shows a structure of a computer system of an electronic device according to some embodiments of the present invention;

[0051] Figure 6 A schematic diagram of a computer-readable storage medium according to some embodiments of the present invention is schematically shown. DETAILED DESCRIPTION

[0052] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description relates to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. On the contrary, they are only examples of devices consistent with some aspects of the present invention as detailed in the attached claims and bone parameter estimation methods based on multimodal information fusion.

[0053] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0054] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0055] Figure 1 A schematic diagram of a system architecture of an exemplary application environment of a method and apparatus for estimating bone parameters based on multimodal information fusion according to an embodiment of the present invention is shown.

[0056] like Figure 1As shown, the system architecture 100 may include one or more terminal devices such as a desktop computer 101, a portable computer 102, a smart phone 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the terminal device and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables, etc. The terminal device may be any electronic device with data processing capabilities, which has a display screen for displaying bone parameter estimation results to the user, including but not limited to the above-mentioned desktop computers, portable computers, smart phones, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a sub-server cluster composed of multiple sub-servers.

[0057] The skeletal parameter estimation method based on multimodal information fusion provided in the embodiment of the present invention can generally be executed by a terminal device, and accordingly, the skeletal parameter estimation device based on multimodal information fusion is generally arranged in the terminal device. However, it is easy for a person skilled in the art to understand that the skeletal parameter estimation method based on multimodal information fusion provided in the embodiment of the present invention can also be executed by the server 105, and accordingly, the skeletal parameter estimation device based on multimodal information fusion can also be arranged in the server 105, and this is not particularly limited in the present exemplary embodiment.

[0058] In addition, it should be understood that the skeleton parameter estimation method based on multimodal information fusion of the embodiment of the present invention can be configured as a software module. In some implementation scenarios, the skeleton parameter estimation scheme based on multimodal information fusion of the present invention can be deployed separately, to generate the skeleton parameter estimation result of different task requirements. In other implementation scenarios, the skeleton parameter estimation scheme based on multimodal information fusion of the present invention can be deployed in other software, as a functional module of this software, such as in the analysis software of the deployment skeleton parameters, the present invention is not particularly limited for the application mode of the skeleton parameter estimation method based on multimodal information fusion.

[0059] Next, the embodiments of the present invention are described in detail.

[0060] like Figure 2 As shown, Figure 2 1 is a flowchart of a method for estimating bone parameters based on multimodal information fusion according to an exemplary embodiment of the present invention, comprising the following steps:

[0061] S210: Acquire X-ray image data, depth image data, inertial measurement unit sensor data, and electromyography signal data of human skeleton;

[0062] S220: Performing three-dimensional reconstruction of the human skeleton according to the X-ray image data and the depth image data to obtain three-dimensional reconstruction data;

[0063] S230: fusing the X-ray image data, the inertial measurement unit sensor data, the electromyography signal data, and the three-dimensional reconstruction data to obtain multimodal fusion data;

[0064] S240: Establishing a bone parameter estimation model based on the multimodal fusion data and a mapping relationship between bone structure features and bone parameters;

[0065] S250: Estimating bone parameters using the bone parameter estimation model.

[0066] In S210, X-ray image data, depth image data, inertial measurement unit sensor data and electromyography signal data of the human skeleton are obtained.

[0067] X-ray imaging data is an image generated by X-ray radiation penetrating the human body and using the absorption and penetration of X-rays by different tissues. For bone imaging, since bones contain a high amount of calcium, X-rays can show their structure through differences in bone density, presenting a clear image difference from soft tissue.

[0068] X-ray images only provide two-dimensional images, so when viewing complex bone structures or multiple bones overlapping, it is impossible to directly obtain three-dimensional information. In addition, X-rays have certain radiation risks, so long-term or frequent use is generally avoided.

[0069] Based on this, the present invention also obtains depth image data. Depth image data can be obtained through a depth camera (such as a somatosensory peripheral (Kinect for Xbox 360, Kinect), a laser radar (Light Detection and Ranging, LiDAR), etc.) or a stereoscopic vision system. These technologies generate a depth image of the surface of an object by measuring the distance between the object and the sensor. Depth image data can accurately capture the spatial position, morphology and relative position of bones and joints, and thus combined with X-ray image data, it can effectively overcome the limitation of X-ray image data lacking depth information, and provide stereoscopic spatial data for three-dimensional analysis.

[0070] An inertial measurement unit (IMU) sensor is a device composed of an accelerometer, a gyroscope, and sometimes a magnetometer, which can measure the acceleration, angular velocity, and direction of an object in real time. The present invention uses the inertial measurement unit sensor data acquired by the IMU sensor to provide dynamic data on the motion state of the human body, especially in terms of joint and bone activity. When acquiring the inertial measurement unit sensor data, the IMU sensor can be placed in different parts (such as joints, limbs, etc.), and the motion characteristics such as the angle change, acceleration, angular velocity, etc. of the joint can be tracked in real time to provide dynamic data for bone and joint movement. IMU data is particularly suitable for dynamic monitoring, and can make up for the deficiency that X-ray image data and depth image data cannot provide motion status.

[0071] Electromyography (EMG) signal data is an electrical signal generated by measuring muscle fiber activity through electrodes. It can capture information such as the strength, frequency, and duration of muscle contraction, and help analyze the relationship between muscle activity and skeletal movement. Through the analysis of EMG signals, it is possible to determine muscle fatigue, load changes, and even monitor muscle recovery during rehabilitation.

[0072] In S220, the human skeleton is three-dimensionally reconstructed according to the X-ray image data and the depth image data to obtain three-dimensional reconstruction data.

[0073] The purpose of three-dimensional reconstruction is to convert two-dimensional X-ray image data and depth image data into a three-dimensional bone structure model, so as to facilitate the subsequent accurate capture of key information such as the spatial shape of the bones, joint position, bone density distribution, etc.

[0074] However, since the X-ray image data and the depth image data come from different measuring devices and have different coordinate systems, when using the X-ray image data and the depth image data, the X-ray image data and the depth image data are firstly image-aligned, and the alignment method can be a feature-based alignment method (for example, Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF)), a region-based alignment method (such as the mutual information method) or an iterative optimization algorithm-based alignment method (for example, the iterative closest point algorithm (ICP)). The present invention does not make any limitation.

[0075] On this basis, since the depth value of each pixel in the depth image data corresponds to a three-dimensional point in space, the registered depth image data can also be converted into point cloud data, which represents the spatial position and shape of the human skeleton.

[0076] Finally, the bone edges and structural features are extracted from the registered X-ray image data and used together with the point cloud data as input data for the 3D reconstruction algorithm to construct a bone model.

[0077] Here, the three-dimensional reconstruction algorithm may be a surface reconstruction algorithm, a stereo matching algorithm, etc., and the present invention does not impose any specific limitation thereto.

[0078] In addition, in order to improve the accuracy of reconstruction, deep learning can also be used to optimize the skeleton model to make it more accurate in spatial position, shape and details. In order to remove possible noise points and optimize the surface smoothness of the skeleton model, post-processing operations such as mesh smoothing and simplification can also be performed, which are not limited by the present invention.

[0079] After three-dimensional reconstruction, the obtained three-dimensional reconstruction data includes the three-dimensional geometric shape of the bones, which can accurately describe the shape, size and position of the bones in space.

[0080] In S230, data fusion is performed on the X-ray image data, the inertial measurement unit sensor data, the electromyography signal data and the three-dimensional reconstruction data to obtain multimodal fusion data.

[0081] In some embodiments, this process may include:

[0082] Preprocessing the X-ray image data, the inertial measurement unit sensor data, the electromyography signal data, and the three-dimensional reconstruction data respectively to obtain preprocessed X-ray image data, preprocessed inertial measurement unit sensor data, preprocessed electromyography signal data, and preprocessed three-dimensional reconstruction data;

[0083] Extracting bone structure features from the preprocessed radiographic image data, dynamic motion features from the preprocessed inertial measurement unit sensor data, muscle activity features from the preprocessed electromyography signal data, and bone spatial structure features from the preprocessed three-dimensional reconstruction data;

[0084] Linearly mapping the skeletal structure feature, the dynamic motion feature, the muscle activity feature, and the skeletal spatial structure feature respectively to obtain a query, a key, and a value corresponding to each feature;

[0085] Calculate the attention score of each feature based on the query, key, and value corresponding to each feature;

[0086] Based on the attention score of each feature, all features are adaptively weighted to obtain the multimodal fusion data.

[0087] In some embodiments, a convolutional neural network (CNN) can be used to preprocess the X-ray image data, including denoising and standardization, and then a residual network (ResNet) is used to extract the bone structure features in the preprocessed X-ray image data. The bone structure features include features such as bone key points, bone edges, and joint positions.

[0088] In some embodiments, the IMU sensor data can be filtered and denoised using a low-pass filter or a Kalman filter, and then the dynamic motion features in the pre-processed inertial measurement unit sensor data are extracted through a sliding window or a temporal convolutional neural network (TCNs). Dynamic motion features include joint angle changes, acceleration, rotational angular velocity, etc., which are used to describe the dynamic motion information of the skeleton.

[0089] In some embodiments, wavelet transform can be used to extract frequency domain features of EMG signals, remove low-frequency noise in frequency domain features and extract time-frequency features of muscle activity, and use the time-frequency features as preprocessed EMG signal data. Neural networks can also be used to extract features such as muscle activity intensity and contraction frequency from time-frequency features.

[0090] In some embodiments, the three-dimensional reconstruction data can be denoised and spatially optimized using algorithms such as the voxel grid method or Poisson surface reconstruction to obtain preprocessed three-dimensional reconstruction data, and then a 3D convolutional neural network is used to extract spatial structural features such as the three-dimensional morphology, joint position, and bone density distribution of the bones.

[0091] Afterwards, the self-attention mechanism is used to fuse the skeletal structure features, dynamic motion features, muscle activity features, and skeletal spatial structure features. First, the feature vector of each modality is linearly mapped to obtain the query, key, and value of each modality. These mappings are implemented through a simple fully connected layer (FC).

[0092] Calculate the attention score of each feature based on the query, key, and value corresponding to each feature:

[0093]

[0094] Where Q is the query vector, K is the key vector, V is the value vector, and d k is the dimension of the key vector.

[0095] Based on the attention score of each feature, all features are adaptively weighted to obtain the multimodal fusion data.

[0096] Unlike traditional weighted summation or concatenation methods, the self-attention mechanism can not only eliminate the need to manually set weights or concatenation through intrinsic weight learning, but also has relatively low computational complexity. It can automatically learn the importance of each modality without the need for additional resources to perform complex operations and flexibly adapt to the relative contributions of different modalities.

[0097] In S240, a mapping relationship between bone structure features and bone parameters is established based on the multimodal fusion data.

[0098] Here, in some embodiments, the multimodal fusion data can be input into a deep neural network (DNN), that is, a bone parameter estimation model is generated by using the deep neural network and the mapping relationship between bone structure features and bone parameters.

[0099] In other embodiments, the multimodal fusion data may be input into a multi-task learning module, and a bone parameter estimation model may be established by combining the mapping relationship between bone structure features and bone parameters.

[0100] In an embodiment of the present invention, bone parameters include joint angle, bone density, bone length, range of joint motion, bone strength and degree of osteoporosis, etc.

[0101] In the embodiment of the present invention, reference Figure 3 As shown, the bone parameter estimation model includes at least a bone structure analysis part and a bone parameter regression part. The bone structure analysis part is responsible for the bone structure analysis task, including multiple first fully connected layers and ReLU activation function layers set continuously, and the ReLU activation function layer is used to perform nonlinear transformation and output the analysis results of the bone structure (including joint angles, bone lengths, and joint motion ranges, etc.); the bone parameter regression part includes a second fully connected layer, which is used to estimate bone health-related parameters such as bone density, bone strength, and osteoporosis degree based on the analysis results of the bone structure.

[0102] The calculation process of the joint angle θ can be expressed as:

[0103]

[0104] Among them, v1 and v2 are the direction vectors of the two bones connecting the joint.

[0105] Assuming the two endpoints of the bone are (x1, y1, z1) and (x2, y2, z2), the bone length L can be calculated using the Euclidean distance formula:

[0106]

[0107] The range of joint motion is predicted based on the joint angle, that is, assuming that the range of joint angles is [a, b] (minimum extension angle a and maximum bending angle b), the range of motion is ba.

[0108] Bone density, bone strength and osteoporosis degree are obtained by regression according to the bone parameter regression part of the present invention.

[0109] The loss function of the bone parameter estimation model of the present invention is:

[0110] L=λ1·L s +λ2·L h

[0111] Among them, λ1 and λ2 are the weights of the bone structure analysis task and the bone parameter regression task respectively, L s represents the loss function of the bone structure analysis task and represents the predicted bone structure parameters, represents the true bone structure parameters, N represents the number of samples, i represents the i-th sample, and L h represents the loss function of the skeleton parameter regression task and represents the predicted bone parameters, Represents the actual bone parameters.

[0112] Those skilled in the art may select appropriate weight coefficients λ1 and λ2 according to task requirements and the performance of the model during training, and the present invention does not impose any specific limitation thereto.

[0113] In S250, the bone parameter estimation model is used to perform bone parameter estimation.

[0114] Since the bone parameter estimation model has been trained through multimodal fusion data and the mapping relationship between bone structure features and bone parameters, it can predict bone parameters based on input data (such as X-ray image data, IMU sensor data, electromyography signal data, and 3D reconstruction data). Therefore, the data can be directly input for bone parameter estimation.

[0115] According to a second aspect of an embodiment of the present invention, a skeleton parameter estimation device 400 based on multimodal information fusion is further provided. Figure 4 As shown, the bone parameter estimation device based on multimodal information fusion includes:

[0116] The data acquisition module 410 is used to acquire X-ray image data, depth image data, inertial measurement unit sensor data and electromyography signal data of human skeleton;

[0117] A three-dimensional reconstruction module 420 is used to perform three-dimensional reconstruction of the human skeleton according to the X-ray image data and the depth image data to obtain three-dimensional reconstruction data;

[0118] A data fusion module 430 is used to fuse the X-ray image data, the inertial measurement unit sensor data, the electromyography signal data and the three-dimensional reconstruction data to obtain multi-modal fusion data;

[0119] A model building module 440 is used to build a bone parameter estimation model based on the multimodal fusion data and the mapping relationship between the bone structure features and the bone parameters;

[0120] The parameter estimation module 450 is used to perform bone parameter estimation using the bone parameter estimation model.

[0121] It should be noted that, although several modules of the skeletal parameter estimation device based on multimodal information fusion are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be concretized in one module or unit. On the contrary, the features and functions of a module described above can be further divided into and concretized by multiple modules or submodules.

[0122] In addition, in an exemplary embodiment of the present invention, an electronic device capable of implementing the above-mentioned bone parameter estimation method based on multimodal information fusion is also provided.

[0123] Those skilled in the art will appreciate that various aspects of the present invention can be implemented as a system, a skeletal parameter estimation method or a program product based on multimodal information fusion. Therefore, various aspects of the present invention can be specifically implemented as the following forms, namely: a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment of a combination of hardware and software, which can be collectively referred to as "circuit", "module" or "system" here.

[0124] Refer to the following Figure 5 An electronic device 500 according to such an embodiment of the present invention is described. Figure 5 The electronic device 500 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0125] like Figure 5As shown, the electronic device 500 is in the form of a general computing device. The components of the electronic device 500 may include but are not limited to: at least one processing unit 510, at least one storage unit 520, a bus 530 connecting different system components (including the storage unit 520 and the processing unit 510), and a display unit 540.

[0126] The storage unit stores a program code, which can be executed by the processing unit 510, so that the processing unit 510 performs the steps of various exemplary embodiments of the present invention described in the above-mentioned "Exemplary method for estimating bone parameters based on multimodal information fusion" section. For example, the processing unit 510 can perform the following steps: Figure 2 S210 shown in: acquiring X-ray image data, depth image data, inertial measurement unit sensor data and electromyography signal data of a human skeleton; S220: performing three-dimensional reconstruction of the human skeleton according to the X-ray image data and the depth image data to obtain three-dimensional reconstruction data; S230: performing data fusion on the X-ray image data, the inertial measurement unit sensor data, the electromyography signal data and the three-dimensional reconstruction data to obtain multimodal data; S240: establishing a mapping relationship between bone structure features and bone parameters based on the multimodal data; S250: establishing a bone parameter estimation model according to the mapping relationship; S260: performing bone parameter estimation using the bone parameter estimation model.

[0127] The storage unit 520 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 521 and / or a cache memory unit 522 , and may further include a read-only memory unit (ROM) 523 .

[0128] The storage unit 520 may also include a program / utility 524 having a set (at least one) of program modules 525, such program modules 525 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0129] Bus 530 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0130] The electronic device 500 may also communicate with one or more external devices 570 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 500, and / or any device that enables the electronic device 500 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 550. Furthermore, the electronic device 500 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 560. Figure 5 As shown, the network adapter 560 communicates with other modules of the electronic device 500 via the bus 530. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0131] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, and can also be implemented by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions so that a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) executes the skeletal parameter estimation method based on multimodal information fusion according to the embodiment of the present invention.

[0132] In an exemplary embodiment of the present invention, a computer-readable storage medium is also provided, on which a program product capable of implementing the above-mentioned method for estimating bone parameters based on multimodal information fusion of the present invention is stored. In some possible embodiments, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above-mentioned "Exemplary method for estimating bone parameters based on multimodal information fusion" section of the present invention.

[0133] refer to Figure 6As shown, a program product 600 for implementing the above-mentioned method for estimating bone parameters based on multimodal information fusion according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In the present invention, the readable storage medium can be any tangible medium containing or storing a program, and the program can be used by or in combination with an instruction execution system, an apparatus or a device.

[0134] The program product may employ any combination of one or more readable storage media. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0135] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0136] In addition, the above-mentioned accompanying drawings are only schematic illustrations of the processing included by the skeletal parameter estimation method based on multimodal information fusion according to an exemplary embodiment of the present invention, rather than limiting the purpose. It is easy to understand that the processing shown in the above-mentioned accompanying drawings does not indicate or limit the time sequence of these processing. In addition, it is also easy to understand that these processing can be, for example, synchronous or asynchronous execution in multiple modules.

[0137] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, and can also be implemented by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions so that a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) executes the skeletal parameter estimation method based on multimodal information fusion according to the embodiment of the present invention.

[0138] Other embodiments of the invention will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. The invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0139] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for estimating bone parameters based on multimodal information fusion, characterized in that: include: Obtaining X-ray image data, depth image data, inertial measurement unit sensor data, and electromyography signal data of human skeleton; Performing three-dimensional reconstruction of the human skeleton according to the X-ray image data and the depth image data to obtain three-dimensional reconstruction data; fusing the X-ray image data, the inertial measurement unit sensor data, the electromyography signal data and the three-dimensional reconstruction data to obtain multimodal fusion data; Based on the multimodal fusion data and the mapping relationship between the bone structure features and the bone parameters, a bone parameter estimation model is established; The bone parameter estimation model is used to estimate bone parameters.

2. The method for estimating bone parameters based on multimodal information fusion according to claim 1, characterized in that: The X-ray image data, the depth image data, the inertial measurement unit sensor data, the electromyography signal data and the three-dimensional reconstruction data are fused to obtain multimodal fusion data including: Preprocessing the X-ray image data, the inertial measurement unit sensor data, the electromyography signal data, and the three-dimensional reconstruction data respectively to obtain preprocessed X-ray image data, preprocessed inertial measurement unit sensor data, preprocessed electromyography signal data, and preprocessed three-dimensional reconstruction data; Extracting bone structure features from the preprocessed radiographic image data, dynamic motion features from the preprocessed inertial measurement unit sensor data, muscle activity features from the preprocessed electromyography signal data, and bone spatial structure features from the preprocessed three-dimensional reconstruction data; Linearly mapping the skeletal structure feature, the dynamic motion feature, the muscle activity feature, and the skeletal spatial structure feature respectively to obtain a query, a key, and a value corresponding to each feature; Calculate the attention score of each feature based on the query, key, and value corresponding to each feature; Based on the attention score of each feature, all features are adaptively weighted to obtain the multimodal fusion data.

3. The method for estimating bone parameters based on multimodal information fusion according to claim 1, characterized in that: The bone parameter estimation model includes: The bone structure analysis part and the bone parameter regression part, the bone structure analysis part is responsible for the bone structure analysis task, including multiple first fully connected layers and ReLU activation function layers set in succession; the bone parameter regression part includes a second fully connected layer, which is used to estimate bone density, bone strength and osteoporosis degree according to the analysis results of the bone structure.

4. The method for estimating bone parameters based on multimodal information fusion according to claim 3, characterized in that: The analysis results of the bone structure include joint angles, bone lengths and range of joint motion; The joint angle analysis process includes: calculating the minimum extension angle and the maximum bending angle of the joint angle according to the direction vectors of the two bones connecting the joint; The analysis process of the joint range of motion includes: calculating the joint range of motion according to the minimum extension angle and the maximum bending angle; The bone length analysis process includes: calculating the bone length according to the coordinates of the two endpoints of the bone using the Euclidean distance formula.

5. The method for estimating bone parameters based on multimodal information fusion according to claim 4, characterized in that: Calculating the joint angle θ based on the direction vectors of the two bones connecting the joint includes: Among them, v1 and v2 are the direction vectors of the two bones connecting the joint.

6. The method for estimating bone parameters based on multimodal information fusion according to claim 4, characterized in that: According to the coordinates of the two endpoints of the bone, using the Euclidean distance formula, the bone length is calculated including: Among them, L represents the bone length, (x1, y1, z1) and (x2, y2, z2) represent the coordinates of the two endpoints of the bone respectively.

7. The method for estimating bone parameters based on multimodal information fusion according to any one of claims 1 to 6, characterized in that: The loss function of the bone parameter estimation model is: L=λ1·L s +λ2·L h Among them, λ1 and λ2 are the weights of the bone structure analysis task and the bone parameter regression task respectively, L s represents the loss function of the bone structure analysis task and represents the predicted bone structure parameters, represents the true bone structure parameters, N represents the number of samples, i represents the i-th sample, and L h represents the loss function of the skeleton parameter regression task and represents the predicted bone parameters, Represents the actual bone parameters.

8. A device for estimating bone parameters based on the multimodal information fusion method according to any one of claims 1 to 7, characterized in that: include: A data acquisition module, used to acquire X-ray image data, depth image data, inertial measurement unit sensor data and electromyography signal data of human skeleton; A three-dimensional reconstruction module, used to perform three-dimensional reconstruction of the human skeleton according to the X-ray image data and the depth image data to obtain three-dimensional reconstruction data; A data fusion module, used for fusing the X-ray image data, the inertial measurement unit sensor data, the electromyography signal data and the three-dimensional reconstruction data to obtain multimodal fusion data; A model building module, used to build a bone parameter estimation model based on the multimodal fusion data and the mapping relationship between the bone structure features and the bone parameters; The parameter estimation module is used to perform bone parameter estimation using the bone parameter estimation model.

9. An electronic device, characterized in that: include: processor; as well as A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the bone parameter estimation method based on multimodal information fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the bone parameter estimation method based on multimodal information fusion as described in any one of claims 1 to 7 is implemented.

Images