Bone joint injury detection method and electronic equipment

Abstract

The invention relates to a bone joint injury detection method and electronic equipment, and relates to the technical field of bone joint detection, and the method comprises the steps: collecting bone joint data of a human body in a motion state and a static state through multiple sensors; inputting the bone joint data into a joint injury prediction model to obtain an injury grade corresponding to the bone joint injury of the human body; wherein the higher the injury level corresponding to the bone joint injury is, the higher the severity of the bone joint injury is, the joint injury prediction model is obtained by training according to a first error and a second error, the first error is the error between bone joint data input into the joint injury prediction model and reconstructed bone joint data, and the second error is the error between the reconstructed bone joint data and the bone joint data input into the joint injury prediction model. The second error is the error between the injury grade corresponding to the bone joint injury predicted by the joint injury prediction model and the actual injury grade. By using the bone joint injury detection method provided by the invention, an accurate and more refined injury grade can be obtained.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of bone joint detection, in particular, to a bone joint injury detection method and an electronic device. BACKGROUND

[0002] Bone joint injury, such as knee osteoarthritis (KOA), is a common joint disease that affects the patient's mobility.

[0003] In related technologies, instruments can be used to detect whether the human body has inflammation in the bone joint, thereby outputting two detection results of arthritis and no arthritis, but the bone joint detection result obtained is relatively single. SUMMARY

[0004] The purpose of the present disclosure is to provide a bone joint injury detection method and an electronic device to solve the above technical problems.

[0005] To achieve the above purpose, the present disclosure provides a bone joint injury detection method, comprising: acquiring bone joint data of a human body in a motion state and a static state through a plurality of sensors; inputting the bone joint data into a joint injury prediction model to obtain an injury grade corresponding to the bone joint injury of the human body; wherein the higher the injury grade corresponding to the bone joint injury, the higher the severity of the bone joint injury, and the joint injury prediction model is trained according to a first error and a second error, the first error being an error between the bone joint data input into the joint injury prediction model and reconstructed bone joint data, and the second error being an error between the injury grade corresponding to the bone joint injury predicted by the joint injury prediction model and an actual injury grade.

[0006] Optionally, the joint injury prediction model comprises a first network layer, a second network layer and a third network layer; and the inputting the bone joint data into the joint injury prediction model to obtain the injury grade corresponding to the bone joint injury of the human body comprises: inputting the bone joint data into the first network layer for decomposition to obtain a low-frequency component and a high-frequency component; the low-frequency component indicating the motion ability of the bone joint of the human body, and the high-frequency component indicating the local pathological features of the bone joint of the human body; inputting the low-frequency component and the high-frequency component into the second network layer for fusion to obtain fused bone joint data; inputting the fused bone joint data into the third network layer for classification to obtain the injury grade corresponding to the bone joint injury of the human body.

[0007] Optionally, the joint damage prediction model further comprises a fourth network layer; and the method further comprises: inputting the bone joint data into the fourth network layer for reconstruction to obtain reconstructed bone joint data; determining a first error and a second error; the first error is a reconstruction error between the reconstructed bone joint data and the bone joint data, and the second error is an error between the predicted damage level output by the third network layer and the actual damage level; updating network parameters in the joint damage prediction model according to the first error and the second error; the network parameters in the joint damage prediction model include network parameters in the first network layer, the second network layer, the third network layer and the fourth network layer.

[0008] Optionally, the bone joint data includes bone joint data of a human body with different physical characteristics, and the different physical characteristics include age, gender, weight and height.

[0009] Optionally, the fourth network layer comprises an encoder and a decoder, the encoder is associated with the first network layer and the second network layer, and the decoder is associated with the third network layer; and the determining the first error comprises: inputting the bone joint data into the encoder for encoding to obtain encoded bone joint data; inputting the encoded bone joint data into the decoder for decoding to obtain decoded bone joint data; obtaining the first error according to the bone joint data and the decoded bone joint data.

[0010] Optionally, the first network layer comprises a wavelet transform layer, an inverse wavelet transform layer and a first full connection layer; and the inputting the bone joint data into the first network layer for decomposition to obtain low-frequency components and high-frequency components comprises: inputting the bone joint data into the wavelet transform layer for convolution to obtain low-frequency components and high-frequency components; inputting the low-frequency components and the high-frequency components into the inverse wavelet transform layer to obtain aggregated low-frequency components and high-frequency components; inputting the aggregated low-frequency components and high-frequency components into the first full connection layer to obtain aggregated low-frequency components and high-frequency components in a high-dimensional space.

[0011] Optionally, the second network layer comprises a double-flow backbone network and a residual gate fusion module; and the inputting the low-frequency components and the high-frequency components into the second network layer for fusion to obtain fused bone joint data comprises: input the low-frequency component and the high-frequency component into the double-flow backbone network to obtain low-frequency features and high-frequency features; input the low-frequency features and the high-frequency features into the residual gate fusion module to obtain fused joint data.

[0012] Optionally, the third network layer comprises a second fully connected layer; and the inputting the fused joint data into the third network layer for classification to obtain the damage grade corresponding to the joint damage of the human body comprises: inputting the fused joint data into the second fully connected layer to obtain the damage grade.

[0013] Optionally, the method further comprises: correcting the damage grade according to human body information to obtain a corrected damage grade; the human body information comprises description information of the human body on the joint, age information of the human body, weight information of the human body, and hobby information of the human body; displaying the corrected damage grade.

[0014] To achieve the above object, the present disclosure provides an electronic device comprising: a memory having a computer program stored thereon; a processor configured to execute the computer program in the memory to implement the steps of the joint damage detection method proposed by the present disclosure.

[0015] Through the above technical solution, in a first aspect, since the multi-sensor can detect joint data in multiple aspects such as the static state and the motion state of the joint, the joint damage prediction model can also obtain a more refined joint damage grade, for example, at least three damage grades, after inputting the joint data in multiple aspects into the joint damage prediction model, and the inflammation degree corresponding to different damage grades is different; instead of a single damage grade, for example, no longer a single inflammation or no inflammation. In a second aspect, the joint data of the human body in the motion state and the static state can more comprehensively reflect the pathological state of the joint of the human body, so that the predicted damage grade of the joint is closer to the actual situation of the user. In a third aspect, after training the joint damage prediction model based on the first error and the second error, the joint damage prediction model can extract the noise-removed joint data from the joint data, thereby reducing the noise influence brought by the multi-sensor and the motion state; and the joint damage prediction model can also learn the gap between the predicted damage grade and the actual damage grade, and predict a damage grade that is more in line with the actual situation.

[0016] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and serve to explain the principles of the present disclosure, but are not intended to limit the present disclosure. In the drawings: Figure 1 is a step flow chart of a detection method of a bone joint according to an exemplary embodiment.

[0018] Figure 2 is a schematic diagram of a test binding of a bone joint injury according to an exemplary embodiment.

[0019] Figure 3 is a schematic diagram of a detection device of a bone joint injury according to an exemplary embodiment.

[0020] Figure 4 is a step flow chart of a detection method of a bone joint injury according to an exemplary embodiment.

[0021] Figure 5 is a structural block diagram of a bone joint injury detection model according to an exemplary embodiment.

[0022] Figure 6 is a step flow chart of a detection method of a bone joint injury according to an exemplary embodiment.

[0023] Figure 7 is a step flow chart of a detection method of a bone joint injury according to an exemplary embodiment.

[0024] Figure 8 is a step flow chart of a detection method of a bone joint injury according to an exemplary embodiment.

[0025] Figure 9 is a schematic diagram of a detection device of a bone joint injury according to an exemplary embodiment.

[0026] Figure 10 is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0027] The specific embodiments of the present disclosure will be described below in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended to illustrate and explain the present disclosure, and are not intended to limit the present disclosure.

[0028] Figure 1 is a step flow chart of a detection method of a bone joint injury according to an exemplary embodiment, which can be a detection method of a joint, the detection method comprising the steps of: In step S10, bone and joint data of the human body in motion and at rest are collected by multiple sensors.

[0029] Among them, bone and joint data is used to reflect the pathological / health status of human bones and joints. Bone and joint data includes joint motion data and biomechanical signal data. Bone and joint data includes bone and joint data of people with different body characteristics, such as age, sex, weight, and height.

[0030] This application primarily collects information on forces, vibrations, and acoustic emissions generated by joints during movement, while also collecting electromyographic signals from muscles within the joint. See also... Figure 9 As shown, a multi-sensor system comprises various types of sensors used to capture data on the human body's bones and joints. The multi-sensor system includes a triaxial accelerometer, a first-contact acoustic sensor, and other sensors. Figure 9 The first contact acoustic sensor and the second contact acoustic sensor are described in detail. Figure 9 The system includes contact acoustic sensors (2), surface electromyography (EMG) sensors, and small inertial sensors. These sensors are placed at key locations in and around the joints to collect joint motion and biomechanical data. The collected joint data is then transmitted wirelessly to wireless communication module 1 in the motherboard hardware module, and further transmitted via biosignal conditioning module to data acquisition card. Data acquisition card receives the joint data and sends it to central microprocessor for processing. Central microprocessor then transmits the joint data to satellite processor and forwards it to data processing module and KL standard classification module for damage assessment based on the joint data. In addition, the motherboard hardware module includes a status monitoring module, display module, storage module, voice module, and wireless communication module, which enables communication with mobile phones and other terminals.

[0031] One method is to use a load measurement approach to measure joint data in both active and passive states. In the no-load measurement mode, the subject is seated in a relaxed, static position, and joint data is collected during this static state. The subject then performs small-range flexion and extension movements, allowing for joint data collection during this active, relaxed, static state. Please refer to [link to relevant documentation]. Figure 2 and Figure 3 As shown, in the load measurement mode, the test subject stands still and the body remains stationary to collect bone and joint data in the standing and static state; the test subject then performs squatting movements to collect bone and joint data in the squatting movement.

[0032] Please see Figure 2 and Figure 3As shown, when testing, the leg position is a knee guard, the multi-sensor is arranged in the knee guard to collect bone joint data; in addition, the leg joint, hip joint and shoulder joint can also adopt the form of a patch; the hip and shoulder joint can fix the multi-sensor through a bandage, and three sensors are arranged on the bandage, wherein the shoulder joint is tested by passing through the bandage from the armpit, of course, a patch can also be used to directly paste the structure on the user, which is not limited in the present application.

[0033] It can be understood that the reason for measuring bone joint data in a load measurement mode rather than in a load-free measurement mode is that in the load-free measurement mode, the human body is in a relaxed state, at which time the human bone joint is not subjected to a large force, thereby greatly reducing noise interference, but since the human body is in a relaxed state, the friction of the bone joint is low, and the collected bone joint data is insufficient to fully reflect the state of the bone joint, and it is difficult to capture the subtle pathological patterns of the bone joint. Therefore, the load measurement mode can be used, in which the human body is in a standing state and performs a squatting motion, in which mode the bone joint will bear the pressure of the upper body, increasing the tactile force and friction of the bone joint, so that the measured bone joint data can better reflect the sensitivity to pathological changes of the bone joint, such as enhancing the sensitivity to bone spurs and cartilage deformation, thereby providing more rich and effective pathological information, but in the load measurement mode, the pressure applied by the upper body will bring noise to the detected bone joint data, resulting in certain errors in the detected bone joint data.

[0034] It can be understood that the reason for collecting bone joint data in a static state and a motion state is that when the human body is in a static state, the bone joint friction is low, so the noise of the collected bone joint data is less, but the bone joint data in a static state is insufficient to fully reflect the state of the human bone joint; when the human body is in a motion state, the bone joint friction is large, which can more fully reflect the state of the human bone joint, but the noise of the bone joint data in a motion state is large. Therefore, the bone joint data in a static state and a motion state can be used at the same time, the bone joint data in a static state is used as the basis, and the bone joint data in a dynamic state is used to compensate the bone joint data in a static state, the bone joint data in the two states is mutually compensated, so that the bone joint data input into the joint damage prediction model later more fully reflects the state of the user's bone joint.

[0035] In step S20, the bone joint data is input into the joint damage prediction model to obtain the damage grade corresponding to the bone joint damage of the human body.

[0036] The damage grade includes at least three damage grades. The higher the damage grade corresponding to the bone joint damage is, the higher the severity of the bone joint damage is indicated.

[0037] Taking five damage grades of damage grade 0, damage grade 1, damage grade 2, damage grade 3 and damage grade 4 as examples, the damage grades gradually increase from damage grade 0 to damage grade 4. Damage grade 0 indicates that the human bone joint is normal, without joint space narrowing, osteophyte or cartilage damage; damage grade 1 indicates possible joint space narrowing with suspicious osteophyte formation, but no clear signs of arthritis; damage grade 2 indicates clear osteophytes, with mild joint space narrowing, early cartilage degeneration; damage grade 3 indicates moderate osteophytes, with obvious joint space narrowing, subchondral bone sclerosis and potential joint deformity; and damage grade 4 indicates larger osteophytes, with severe joint space narrowing, extensive subchondral bone sclerosis and obvious joint deformity.

[0038] The joint damage prediction model is used to obtain the damage grade corresponding to the bone joint damage of the human body based on the input bone joint data, and the joint damage prediction model is trained according to the first error and the second error.

[0039] For the first error, the first error is the error between the bone joint data input into the joint damage prediction model and the reconstructed bone joint data. The bone joint data input into the joint damage prediction model is the original bone joint data, and the reconstructed bone joint data is the bone joint data after denoising and restoration.

[0040] For the second error, the second error is the error between the damage grade corresponding to the bone joint damage predicted by the joint damage prediction model and the actual damage grade.

[0041] It can be understood that when the bone joint data in the running state is used, noise will be brought by the human bone joint in the motion state. And with the increase of the bone joint friction force, the noise level in the bone joint motion signal will also increase; and when the human bone joint is measured by the load measurement method, noise will also be brought due to the influence of the acting force on the human body; and when the bone joint data of the human body is measured by multiple sensors, the more the number of sensors, the more interference will be brought, and therefore the more noise will be brought. It can be seen that although the load measurement method is used, the bone joint data of the human body is collected by multiple sensors, which can more comprehensively reflect the state of the human bone joint, but due to the influence of noise from multiple aspects, the measured bone joint data will not be accurate.

[0042] Based on this, the first error can be used to denoise the bone joint data, and the second error can be used to obtain an accurate damage level. After updating the joint damage prediction model using the first error, the joint damage prediction model can learn the difference between the original bone joint data input into the joint damage prediction model and the denoised bone joint data after reconstruction, and minimize this difference, so that the subsequent joint damage prediction model can extract the denoised bone joint data from the original bone joint data; after updating the joint damage prediction model using the second error, the joint damage prediction model can learn the difference between the predicted damage level and the actual damage level, and minimize this difference, so that the subsequent joint damage prediction model can predict a more realistic damage level.

[0043] In some scenarios, please refer to Figure 8 As shown in the figure, the captured bone joint data includes bone joint vibration data, first bone joint acoustic emission data (bone joint acoustic emission 1 data in Figure 8 ), bone joint surface electromyography data, second bone joint acoustic emission data (bone joint acoustic emission 2 data in Figure 8 ), etc. Among them, the bone joint vibration data reflects the low-frequency mechanical vibration when the bone joint moves; the first bone joint acoustic emission data and the second bone joint acoustic emission data reflect the ultrasonic signals generated by the friction or rupture of the cartilage of the bone joint, and the collection frequency bands of the first bone joint acoustic emission data and the second bone joint acoustic emission data are different; the bone joint surface electromyography data reflects the muscle activation pattern.

[0044] The captured bone joint data is filtered through a filter and then normalized, so that the normalized bone joint data and the bone joint angle flexion data are in a unified dimension; then the angle data is used to divide the normalized bone joint data and the bone joint angle flexion data into their respective period segment data. For example, the first bone joint acoustic emission data is divided into period segment data under the first period, the second bone joint acoustic emission data is divided into period segment data under the second period, and the bone joint surface electromyography data is divided into period segment data under the third period, etc. Then, using a double-density dual-tree complex wavelet decomposition layer, the period segment data is decomposed.

[0045] Then determine the decomposition coefficients of each period segment data, for example, determine the wavelet decomposition coefficients of the bone joint vibration data, the first bone joint acoustic emission data, the bone joint surface electromyography data, the second bone joint acoustic emission data, and the bone joint angle flexion data.

[0046] The multi-scale coefficient matrix of each cycle segment data is respectively constructed by using the multi-scale wavelet decomposition coefficients and scale coefficients of each cycle segment data under the multi-scale wavelet decomposition. Different types of multi-scale coefficient matrices are stacked according to their corresponding cycles to form a multi-channel multi-scale information coefficient matrix, and a multi-channel multi-scale information coefficient matrix dataset is constructed. The dataset label is determined, the training set and the test set are divided, and finally a multi-convolution classification model such as a classifier and a K-L hierarchical evaluation model is selected. The selected model is trained using a center ordinal loss function to obtain the damage degree at different levels such as level 0 to level IV.

[0047] Through the above technical solution, in a first aspect, since the multi-sensor can detect bone joint data in multiple aspects such as the static state and the motion state of the bone joint, the bone joint in multiple aspects can be input into the joint damage prediction model, and the damage level of the bone joint can also be more diversified. In a second aspect, the bone joint data of the human body in the motion state and the static state can more comprehensively reflect the pathological state of the human body bone joint, so that the predicted damage level of the bone joint is closer to the actual situation of the user. In a third aspect, after training the joint damage prediction model based on the first error and the second error, the joint damage prediction model can extract the noise-removed bone joint data from the bone joint data, thereby reducing the noise influence of the multi-sensor and the motion state. The joint damage prediction model can also learn the gap between the predicted damage level and the actual damage level, and minimize the gap between the predicted damage level and the actual damage level, so as to predict a damage level that is more in line with the actual situation. In a fourth aspect, after inputting the more comprehensive bone joint data detected by the multi-sensor into the joint damage prediction model, the joint damage prediction model can predict the damage level of at least three types of arthritis, rather than simply determining whether the human body has bone joint damage. It can obtain more comprehensive and more detailed pathological characteristics to assist the user in understanding the current pathological condition of the bone joint. In a fifth aspect, after training the joint damage prediction model using bone joint data of different ages, genders, weights, and heights, the joint damage prediction model can also predict damage levels that are adapted to different populations, thereby improving the generalization ability of the joint damage prediction model.

[0048] Figure 4 is an exemplary embodiment related to the above step S20, which is used to interpret the exemplary scheme of the damage level obtained by the joint damage prediction model. Please refer to Figure 5 Since the joint damage prediction model includes a first network layer, a second network layer, and a third network layer, the step S20 includes the following steps: In step S21, the bone-joint data is input into the first network layer for decomposition to obtain a low-frequency component and a high-frequency component.

[0049] The first network layer is configured to process non-stationary bone-joint data and decompose the bone-joint data into a low-frequency component and a high-frequency component. The low-frequency component reflects the overall pathological degree / severity of the bone joint, and the high-frequency component reflects the local pathological characteristics of the human body under a specific movement mode. The first network layer can be a Wavelet-Connected Flow Layer (WCFL).

[0050] The low-frequency component indicates the movement ability of the bone joint of the human body. For example, the low-frequency component reflects the long-term trend and stable movement mode in the bone joint movement process, which reflects the overall stability and movement ability of the bone joint.

[0051] For the stability of the bone joint, the higher the energy of the low-frequency component, the lower the indicated damage level, and the better the stability of the bone joint movement, indicating that the consistency of the bone joint movement is good. For example, in the early damage level 0-damage level 2, the bone joint movement is relatively stable, the energy of the low-frequency feature is high, and the consistency of the bone joint movement execution is good.

[0052] For the movement ability of the bone joint, the movement ability of the bone joint reflects the range of motion and the fluency of movement, and the lower the low-frequency component, the higher the indicated damage level. For example, in the late damage level 4, the movement ability of the bone joint decreases, and the energy of the low-frequency component decreases significantly, indicating that the movement amplitude decreases and the irregular shape increases.

[0053] The high-frequency component indicates the local pathological characteristics of the bone joint of the human body. For example, the high-frequency component reflects the instantaneous dynamics and local changes in the bone joint movement process, which reflects the local friction, cartilage roughness, and bone spur in the bone joint.

[0054] For local friction, the high-frequency component can reflect the local friction in the bone joint, such as bone spur and cartilage damage, which can increase the friction in the bone joint, and the energy of the high-frequency component increases accordingly.

[0055] For cartilage roughness, the high-frequency component can also reflect the cartilage roughness. In the early damage level stage, the cartilage may be slightly degraded, and the energy of the high-frequency component is low; in the late damage level stage, the cartilage degradation is more serious, and the energy of the high-frequency component increases significantly.

[0056] For bone spur, the high-frequency component can reflect the local dynamic changes caused by bone spur, which can cause instantaneous impact during bone joint movement, resulting in an increase in the energy of the high-frequency component.

[0057] In a possible implementation, the first network layer comprises a wavelet transform layer (DWT), an inverse wavelet transform layer (inverse DWT), and a first fully connected layer (FC), and the step S21 comprises the following sub-steps: Sub-step A1, inputting the bone joint data into the wavelet transform layer for convolution to obtain a low-frequency component and a high-frequency component.

[0058] The wavelet transform layer can be a multi-level discrete wavelet transform layer, which can decompose the bone joint data collected by multiple sensors at multiple time steps into multiple frequency components level by level, thereby obtaining a single low-frequency component and multiple high-frequency components.

[0059] For example, the bone joint data can be first input into a low-pass filter (g) and a high-pass filter (h), the bone joint data is convolved by the low-pass filter to obtain low-pass filtered bone joint data, and the bone joint data is convolved by the high-pass filter to obtain high-pass filtered bone joint data. The low-pass filtered bone joint data and the high-pass filtered bone joint data are then down-sampled to obtain a low-frequency component (X k , g) and a high-frequency component (X k , h), respectively. The above steps are repeated, the low-frequency component is continuously input into the low-pass filter and the high-pass filter to obtain new low-frequency components and high-frequency components, and the new low-frequency component is input into the low-pass filter and the high-pass filter to obtain new low-frequency components and high-frequency components, until a predetermined decomposition level K is reached. After K-level decomposition, a single low-frequency component and K high-frequency components are obtained, and the length of the low-frequency component obtained after decomposition is halved compared with the length of the low-frequency component before decomposition. The low-frequency component represents the global trend in the bone joint data, and the high-frequency component represents the local dynamic change in the bone joint data.

[0060] It can be understood that the reason why the bone joint data is convolved by the wavelet transform layer in sub-step A1 to obtain the low-frequency component and the high-frequency component is that the bone joint data can be divided into the overall feature of the low-frequency component and the local feature of the high-frequency component.

[0061] Sub-step A2, inputting the low-frequency component and the high-frequency component into the inverse wavelet transform layer to obtain aggregated low-frequency components and high-frequency components.

[0062] The inverse wavelet transform layer has the following functions: First, the length of the low-frequency component and the high-frequency component after wavelet transform is restored to the length of the original bone joint data, because the signal of the low-frequency component is halved each time the low-frequency component is decomposed during wavelet transform, and therefore the length of the low-frequency component needs to be restored to the same length as the initial low-frequency component by the inverse wavelet transform layer.

[0063] Secondly, the components of the K different frequencies are recombined and converted back to the time domain for subsequent processing.

[0064] For the low-frequency component, the low-frequency component can be subjected to a one-level upsampling operation to double the signal length of the low-frequency component; the upsampled low-frequency component is smoothed by a low-pass filter to restore an approximation of the original bone joint data.

[0065] For the high-frequency component, the high-frequency component can be subjected to multiple upsampling and inverse wavelet transform to preserve the detail information in the high-frequency component. The high-frequency component is decomposed into multiple layers of high-frequency components during multiple wavelet transforms, each layer of high-frequency component corresponding to detail information of different frequencies. Through multiple upsampling and inverse wavelet transform, these detail information can be recovered layer by layer, and the high-frequency details of each layer can be aggregated.

[0066] It can be understood that the reason for sub-step A2 inputting the low-frequency component and the high-frequency component into the inverse wavelet transform layer to obtain the aggregated low-frequency component and high-frequency component is that the low-frequency component as a whole feature and the high-frequency component as a local feature are recombined to obtain the complete bone joint feature of the human body.

[0067] Sub-step A3 inputs the aggregated low-frequency component and high-frequency component into the first fully connected layer to obtain the aggregated low-frequency component and high-frequency component in the high-dimensional space.

[0068] The aggregated low-frequency component and high-frequency component can be mapped to a high-dimensional space to enhance the ability of the joint prediction model to capture high-frequency features of different frequencies, and the calculation formula is as follows: X l = FC(X low ) X h = FC(X high ) (1) In formula (1), X l is the low-frequency component mapped to the high-dimensional space; X h is the high-frequency component mapped to the high-dimensional space; FC(*) represents the expression function of the first fully connected layer; X low is the aggregated low-frequency component; and X high is the aggregated high-frequency component.

[0069] It can be understood that the reason for inputting the aggregated low-frequency component and the high-frequency component into the first fully connected layer in sub-step A3 is that the combined low-frequency component and the high-frequency component output by the inverse wavelet transform layer are relatively original and contain all frequency components, but these components are not well organized together. Therefore, the combined low-frequency component and the high-frequency component can be further processed by the first fully connected layer to enhance the expression ability of each feature after combination.

[0070] In step S22, the low-frequency component and the high-frequency component are input into the second network layer for fusion to obtain fused bone joint data.

[0071] In a possible implementation, the aggregated low-frequency component and the high-frequency component in the high-dimensional space can be input into the second network layer for fusion to obtain fused bone joint data.

[0072] The second network layer is used to capture the change rule of the low-frequency component and the high-frequency component over time, so as to facilitate the joint damage prediction model to understand the dynamically changing bone joint data. The second network layer can be a dual-stream adaptive temporal fusion network (DATFN). The fused bone joint data is fused bone joint features.

[0073] In a possible implementation, the second network layer includes a dual-stream backbone network and a residual gate fusion module, and the step S22 includes the following sub-steps: Sub-step B1: inputting the low-frequency component and the high-frequency component into the dual-stream backbone network to obtain low-frequency features and high-frequency features.

[0074] The dual-stream backbone network is used to process the low-frequency component and the high-frequency component in the fused bone joint data respectively, and capture different frequency dynamics. The dual-stream backbone network includes the following multiple sub-networks. First, the low-frequency stream and the high-frequency stream: the fused low-frequency component and the high-frequency component are input into two independent streams of the low-frequency stream and the high-frequency stream respectively, and each stream in the low-frequency stream and the high-frequency stream uses a residual block of a stacked modern temporal convolutional network (Modern TCN) as a time encoder.

[0075] Second, the temporal convolutional network, each temporal convolutional network includes a deep convolutional layer (DWConv) with a large kernel and a convolutional feedforward network (ConvFFN), which can effectively capture long-distance time dependence while maintaining computational efficiency.

[0076] Thirdly, the time encoder, the time encoder of the low-frequency stream and the high-frequency stream works independently, and extracts the low-frequency feature and the high-frequency feature respectively, and the calculation formula is as follows: H l =Encoder(X l ) H h =Encoder(X h ) (2) In formula (2), H l is the learned low-frequency feature; H h is the learned high-frequency feature; Encoder(*) is the function of the encoder; X l is the aggregated low-frequency component in the high-dimensional space; X h is the aggregated high-frequency component in the high-dimensional space.

[0077] It can be understood that by inputting the low-frequency component and the high-frequency component to the dual-stream backbone network through sub-step B1 to obtain the low-frequency feature and the high-frequency feature, the key features in the low-frequency component can be independently processed by the low-frequency stream in the dual-stream backbone network, such as processing the stability and consistency of the bone joint movement, so as to obtain the low-frequency feature; the key features in the high-frequency component can be independently processed by the high-frequency stream in the dual-stream backbone network, such as the subtle abnormalities and friction details in the bone joint movement, so as to obtain the high-frequency feature.

[0078] Sub-step B2, input the low-frequency feature and the high-frequency feature into the residual gating fusion module to obtain the fused bone joint data.

[0079] Wherein, the residual gating fusion module is used to fuse the extracted low-frequency feature and high-frequency feature, so as to form a comprehensive bone joint feature / bone joint data, so as to facilitate subsequent classification, the residual gating fusion module can be an adaptive residual gating fusion module (Adaptive Residual Gating Fusion Module, ARGFM), and the residual gating fusion module includes the following multiple networks: Firstly, the feature enhancement module uses a multi-layer perceptron (MLP) to project the low-frequency feature and the high-frequency feature into a rich semantic space, so that the low-frequency feature and the high-frequency feature are more rich and easy to distinguish, and the calculation formula is as follows: H~ l =MLP(H l ) H~ h =MLP(H h ) (3) In formula (3), H~ l is the low-frequency feature projected into the semantic space; H~h is the high-frequency feature projected into the semantic space; H l is the learned low-frequency feature; H h is the learned high-frequency feature; MLP (*) is the expression function of the multi-layer perceptron.

[0080] Secondly, the gating weighted fusion module adaptively adjusts the weight coefficients of the low-frequency feature and the high-frequency feature by learning the gating coefficient, so as to emphasize the important low-frequency feature and suppress the unimportant high-frequency feature, or emphasize the important high-frequency feature and suppress the unimportant low-frequency feature, and the calculation formula is as follows: η = σ (W l *H l +W h *H h +b) (4) In formula (4), η is the gating coefficient; σ is the Sigmoid (S-shaped activation) function; W l and W h are weight coefficients; b is a constant; H l is the low-frequency feature projected into the semantic space; H h is the high-frequency feature projected into the semantic space.

[0081] As can be seen from formula (5), if the weight coefficient W l is greater than the weight coefficient W h , the low-frequency feature can be emphasized and the high-frequency feature can be suppressed; if the weight coefficient W l is less than the weight coefficient W h , the high-frequency feature can be emphasized and the low-frequency feature can be suppressed.

[0082] Thirdly, the residual refinement module applies weighted fusion and residual set formation to form the fused bone joint data / bone joint feature, combines the weighted fused feature with the original low-frequency feature, so that the key information in the fused bone joint data is retained, and the feature discrimination ability is enhanced, and the calculation formula is as follows: H = η · H l + (1-η) · H h +H l (5) In formula (5), H is the fused bone joint data; η is the gating coefficient; H l is the low-frequency feature projected into the semantic space; H h is the high-frequency feature projected into the semantic space.

[0083] In step S23, the fused bone joint data is input into the third network layer for classification, and the damage grade corresponding to the bone joint damage of the human body is obtained.

[0084] In a possible implementation, the third network layer comprises a second fully connected layer, and the fused bone joint data can be input into the second fully connected layer to obtain the injury level, and the calculation formula is as follows: y = FC(H) (6) In formula (6), y is the probability of the predicted injury level, FC(*) is the expression function of the second fully connected layer, and H is the fused bone joint data.

[0085] Through the above technical solution, the input bone joint data can be subjected to feature extraction and feature fusion through the first network layer and the second network layer, and the fused bone joint data can be input into the third network layer for classification, so as to obtain the injury level indicated by the bone joint data. In this process, the bone joint data is divided into low-frequency features and high-frequency features, the stability and consistency of the human bone joint as a whole are determined through the low-frequency features, and the local details of the human bone joint are determined through the high-frequency features, so that a more accurate injury level is obtained.

[0086] Figure 6 is an example embodiment related to the present disclosure, which is used to explain an example scheme for updating network parameters in a joint injury prediction model. Please refer to Figure 5 The arthritic prediction model also includes a fourth network layer, and the method comprises the following steps: In step S30, the bone joint data is input into the fourth network layer for reconstruction to obtain reconstructed bone joint data.

[0087] The fourth network layer is used to reconstruct the input bone joint data, so as to output bone joint data with less noise. The fourth network layer can be a Variational Autoencoder (VAE).

[0088] In step S40, the first error and the second error are determined.

[0089] The first error is the reconstruction error of the fourth network layer, and is the reconstruction error between the reconstructed bone joint data and the bone joint data.

[0090] In a possible implementation, the fourth network layer comprises an encoder and a decoder. The bone joint data can be input into the encoder for encoding to obtain encoded bone joint data, and the encoded bone joint data can be input into the decoder to obtain decoded bone joint data. Then, the first error is obtained according to the decoded bone joint data and the bone joint data input into the encoder.

[0091] The encoder is associated with the first network layer and the second network layer, and the encoder enhances the capturing ability of key features of the bone joint data by using a self-attention mechanism. The encoder is used to map the input bone joint data to a latent space to obtain a probability distribution, which is referred to as a latent distribution, and the calculation formula is as follows: q(z|X) = N(z; μ(X), σ 2 (X)) (7) In formula (7), q(z|X) represents a latent distribution of the input bone joint data X in the latent space; μ(X) and σ 2 (X) are the mean and variance of the latent distribution, respectively; and Z is a latent variable.

[0092] It can be understood that formula (7) describes a latent distribution of generating a latent variable Z given the bone joint data X, and the goal of the encoder is to learn the mean and variance of the latent distribution, so as to be able to generate the latent variable Z, which essentially learns the main features in the bone joint data and ignores some unimportant noise. In this process, the encoder compresses high-dimensional bone joint data into a low-dimensional latent space, thereby filtering out non-important information such as noise in the data compression process.

[0093] The decoder is associated with the third network layer, and the decoder is fused with the third network layer to improve the accuracy of the damage grade prediction through the sequence generation capability, for example, the decoder is fused with the Transformer architecture in the third network layer, the low-frequency component and the high-frequency component are input into the decoder under the Transformer architecture, the fused bone joint data is obtained, and the fused bone joint data is input into another decoder under the Transformer architecture for classification to obtain the damage grade corresponding to the bone joint damage. The decoder is used to generate a latent distribution of data X given a latent variable Z, and the calculation formula is as follows: p(X|z) = N(X; μ(z), σ 2 (z)) (8) In formula (8), p(X|z) represents a latent distribution of the latent variable Z in the latent space, and represents a probability of generating data X from the latent variable; μ(z) and σ 2 (z) are the mean and variance that the decoder needs to learn, respectively, μ(z) represents an expected value of generating data X, and σ 2 (z) represents the uncertainty of generating data X; and X is the finally reconstructed bone joint data.

[0094] It can be understood that formula (8) describes generating reconstructed bone joint data under the condition of a given latent variable Z. In the training stage of the decoder, the decoder learns the mean and variance to reconstruct the bone joint data as much as possible. In the application stage of the decoder, the decoder generates new bone joint data by using the learned mean and variance. The purpose of the decoder is to learn how to restore the bone joint data as close to the real bone joint data as possible according to the latent variable Z. Since the latent variable X has removed most of the noise information, the bone joint data reconstructed by the decoder will also be cleaner.

[0095] wherein the first error is calculated according to the following formula: L VAE =L recon +βL KL (9) In formula (9), L VAE is a first loss function of the first error; L recon is a reconstruction loss, which is a difference between the reconstructed bone joint data and the bone joint data input into the fourth network layer; L KL is a KL (Kullback-Leibler) divergence, which is used for regularizing the latent space to obtain better generalization ability; and β is a weight, which is used for measuring the importance of the reconstruction error and the KL divergence.

[0096] wherein the second error is an error between the predicted damage level output by the third network layer and the actual damage level.

[0097] For the predicted damage level, the bone joint data is input into the first network layer and the second network layer to obtain fused bone joint data, and then the fused bone joint data is input into the third network layer to obtain a predicted damage level.

[0098] For the actual damage level, different bone joint data corresponding to the actual damage level can be established in the database in advance. The mapping relationship between the bone joint data and the actual damage level can be obtained by combining different human bodies and clinical diagnosis results.

[0099] In step S50, the network parameters in the joint damage prediction model are updated according to the first error and the second error.

[0100] In a possible implementation, a target error can be obtained according to the first error and the second error, and then the network parameters in the joint damage prediction model are updated according to the target error.

[0101] wherein obtaining the target error according to the first error and the second error includes: taking a sum of the first error and the second error as the target error, and the calculation formula is as follows: L=L cls +λLVAE (10) In formula (10), L is a target error; L cls is a second error; L VAE is a first error; λ is a weight, used to measure the importance of the first error and the second error.

[0102] As can be seen from formula (10), if the weight is large, it means that the second error occupies a larger weight, and at this time the error between the predicted damage level of the joint damage prediction model and the actual damage level is higher; if the weight is small, it means that the first error occupies a larger weight, and at this time the error between the reconstructed joint data and the original joint data is higher.

[0103] The network parameters in the joint damage prediction model include network parameters in the first network layer, the second network layer, the third network layer and the fourth network layer.

[0104] It can be understood that the fourth network layer and the third network layer share the same joint input data, but the fourth network layer and the third network layer are jointly processing two different tasks, the third network layer is used to process a classification task, and is used to divide at least three damage levels according to the joint data; the fourth network layer is a task of assisting the classification task to denoise, and is used to denoise the joint data to reduce the influence of noise on the accuracy of the output damage level.

[0105] The fourth network layer and the third network layer have a common first network layer and a second network layer, so the network parameters in the first network layer and the second network layer can be updated according to the first error and the second error, so that the noise in the joint data processed by the first network layer and the second network layer after feature extraction and feature fusion is less, thereby reducing the influence of noise on the final predicted damage level. For example, the wavelet transform parameters in the first network layer, the weights and biases of the first fully connected layer, and the like can be updated according to the first error and the second error, so that the noise of the aggregated low-frequency components and high-frequency components in the high-dimensional space output by the first network layer is less. For example, the weights of the multilayer perceptron and the gating coefficient in the second network layer can also be updated according to the first error and the second error, so that the noise of the fused joint data output by the second network layer is less.

[0106] The third network layer is a classification layer, which is used to obtain a damage level based on joint data, and the weights and biases of the second fully connected layer in the third network layer can be updated according to the first error and the second error, so that the predicted damage level obtained by the second network layer is closer to the actual damage level, and a more accurate predicted damage level is obtained.

[0107] The fourth network layer is a reconstruction layer, which is configured to reconstruct the input bone joint data to obtain reconstructed bone joint data. The network parameters of the encoder and the decoder in the fourth network layer can be updated according to the first error and the second error, so that the reconstructed bone joint data output by the decoder is as close as possible to the bone joint data before reconstruction, and the noise is removed by the encoder while the original bone joint data is retained.

[0108] According to the technical solution, the target error can be obtained according to the first error and the second error, and the network parameters of each layer from the first network layer to the fourth network layer can be updated layer by layer based on the target error.

[0109] In the process of updating the third network layer, the second error between the predicted damage level output by the third network layer and the actual damage level gradually decreases, and the target loss value composed of the first error and the second error is also updated, so that the network parameters of the third network layer are adaptively updated, and the updated third network layer can output more accurate damage levels.

[0110] In the process of updating the encoder and the decoder in the fourth network layer, the first error between the reconstructed bone joint data of the fourth network layer and the input bone joint data gradually decreases, so that the decoder can output reconstructed bone joint data with less noise. At the same time, due to the decrease of the first error, the target loss value composed of the first error and the second error is also updated, so that the first network layer and the second network layer are adaptively updated. After receiving the bone joint data, the first network layer and the second network layer also extract bone joint data with less noise, thereby reducing the influence of noise.

[0111] Figure 7 The present disclosure relates to an example scheme for correcting the damage level, which includes the following steps: In step S60, the damage level is corrected according to the human information to obtain a corrected damage level.

[0112] In a possible implementation, a correction coefficient can be obtained according to the human information, and the damage level is corrected according to the correction coefficient to obtain a corrected damage level. The product of the correction coefficient and the damage level can be taken as the corrected damage level.

[0113] The human information includes description information of the human body to the bone joint, age information of the human body, weight information of the human body, and hobby information of the human body.

[0114] For example, the correction coefficient is larger when the description information of the human body on the bone joint indicates that the bone joint injury is more serious; the correction coefficient is larger when the age information of the human body indicates that the human body is older; the correction coefficient is larger when the weight information of the human body indicates that the human body is heavier; and the correction coefficient is larger when the hobby information of the human body indicates that the human body prefers sports.

[0115] It can be understood that the more serious the description information of the human body on the bone joint, the higher the degree of bone joint degeneration perceived by the human body; the older the human body, the higher the degree of bone joint degeneration; the heavier the human body, the greater the load on the bone joint; and the more the human body likes sports, the greater the load on the bone joint. Therefore, these factors can be considered to obtain the corrected injury level.

[0116] In step S70, the corrected injury level is displayed.

[0117] Among them, the corrected injury level and the corresponding disease description can be displayed on the device.

[0118] Through the above technical solution, the injury level can be corrected based on the description information of the human body on the bone joint, the age information of the human body, the weight information of the human body, and the hobby information of the human body. The corrected injury level can consider the physiological information of different populations, and the obtained injury level can be adaptively adjusted according to different populations, so that a more accurate injury level can be obtained.

[0119] Figure 10 is a block diagram of an electronic device according to an example embodiment. As shown in Figure 10 The electronic device 800 can include a processor 801 and a memory 802. The electronic device 800 can further include one or more of a multimedia component 803, an input / output (I / O) interface 804, and a communication component 805.

[0120] The processor 801 is configured to control overall operations of the electronic device 800 to complete all or part of the steps of the bone joint injury detection method described above. The memory 802 is configured to store various types of data to support operations of the electronic device 800, which can include, for example, instructions for any application or method operating on the electronic device 800, and application-related data, such as contact data, transmitted and received messages, pictures, audio, video, and the like. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. The multimedia component 803 can include a screen and an audio component. The screen can be, for example, a touch screen, and the audio component is configured to output and / or input audio signals. For example, the audio component can include a microphone configured to receive external audio signals. The received audio signals can be further stored in the memory 802 or transmitted through the communication component 805. The audio component also includes at least one speaker configured to output audio signals. The I / O interface 804 provides an interface between the processor 801 and other interface modules, which can be a keyboard, a mouse, a button, and the like. The buttons can be virtual buttons or physical buttons. The communication component 805 is configured to perform wired or wireless communication between the electronic device 800 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, near field communication (NFC), 2G, 3G, or 4G, or a combination of one or more of them, so the corresponding communication component 805 can include a Wi-Fi module, a Bluetooth module, and an NFC module.

[0121] In an exemplary embodiment, the electronic device 800 can be implemented by one or more Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor or other electronic elements for executing the above-mentioned method for detecting bone joint injury.

[0122] In another exemplary embodiment, a computer readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the above-mentioned method for detecting bone joint injury. For example, the computer readable storage medium can be the above-mentioned memory 802 including program instructions, which can be executed by the processor 801 of the electronic device 800 to complete the above-mentioned method for detecting bone joint injury.

[0123] In another exemplary embodiment, a computer program product is also provided, which contains a computer program capable of being executed by a processor, and when the computer program is executed by the processor, the steps of the above-mentioned method for detecting bone joint injury are implemented.

[0124] The preferred embodiments of the present disclosure are described in detail above with reference to the accompanying drawings, but the present disclosure is not limited to the specific details in the above-described embodiments. Within the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all belong to the protection scope of the present disclosure.

[0125] In addition, it should be noted that each specific technical feature described in the above-described specific embodiments can be combined in any appropriate manner without contradiction, and in order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0126] In addition, any combination of various different embodiments of the present disclosure can also be made, as long as it does not deviate from the idea of the present disclosure, and it should also be considered as the disclosed content of the present disclosure.

Claims

1. A method of detecting a bone joint injury, characterized by, The method comprises: acquiring bone joint data of a human body in a motion state and a static state through a plurality of sensors; inputting the bone joint data into a joint injury prediction model to obtain an injury grade corresponding to a bone joint injury of the human body; wherein the higher the injury grade corresponding to the bone joint injury is, the higher the severity of the bone joint injury is, the joint injury prediction model is trained according to a first error and a second error, the first error is an error between the bone joint data input into the joint injury prediction model and reconstructed bone joint data, and the second error is an error between an injury grade corresponding to a bone joint injury predicted by the joint injury prediction model and an actual injury grade.

2. The method of claim 1, wherein, The joint injury prediction model comprises a first network layer, a second network layer and a third network layer; the inputting of the bone joint data into the joint injury prediction model to obtain the injury grade corresponding to the bone joint injury of the human body comprises: inputting the bone joint data into the first network layer for decomposition to obtain a low-frequency component and a high-frequency component; the low-frequency component indicates a motion ability of the bone joint of the human body, and the high-frequency component indicates a local pathological feature of the bone joint of the human body; inputting the low-frequency component and the high-frequency component into the second network layer for fusion to obtain fused bone joint data; inputting the fused bone joint data into the third network layer for classification to obtain the injury grade corresponding to the bone joint injury of the human body.

3. The method of claim 2, wherein, The joint injury prediction model further comprises a fourth network layer; the method further comprises: inputting the bone joint data into the fourth network layer for reconstruction to obtain reconstructed bone joint data; determining a first error and a second error; the first error is a reconstruction error between the reconstructed bone joint data and the bone joint data, and the second error is an error between a predicted injury grade output by the third network layer and an actual injury grade; updating network parameters in the joint injury prediction model according to the first error and the second error; the network parameters in the joint injury prediction model comprise network parameters in the first network layer, the second network layer, the third network layer and the fourth network layer.

4. The method of claim 3, wherein, The bone joint data comprises bone joint data of human bodies with different physical characteristics, and the different physical characteristics comprise age, gender, weight and height.

5. The method of claim 3, wherein, The fourth network layer comprises an encoder and a decoder, the encoder is associated with the first network layer and the second network layer, and the decoder is associated with the third network layer; the determination of the first error comprises: inputting the bone joint data into the encoder for encoding to obtain encoded bone joint data; inputting the encoded bone joint data into the decoder for decoding to obtain decoded bone joint data; obtaining the first error according to the bone joint data and the decoded bone joint data.

6. The method of claim 2, wherein, The first network layer comprises a wavelet transform layer, an inverse wavelet transform layer and a first full connection layer; the inputting of the bone joint data into the first network layer for decomposition to obtain the low-frequency component and the high-frequency component comprises: The bone joint data is input into the wavelet transform layer for convolution to obtain low-frequency components and high-frequency components; The low-frequency components and the high-frequency components are input into the inverse wavelet transform layer to obtain aggregated low-frequency components and high-frequency components; The aggregated low-frequency components and the high-frequency components are input into the first fully connected layer to obtain aggregated low-frequency components and high-frequency components in a high-dimensional space.

7. The method of claim 2, wherein, The second network layer includes a double-flow backbone network and a residual gate fusion module; the low-frequency components and the high-frequency components are input into the second network layer for fusion to obtain fused bone joint data, including: The low-frequency components and the high-frequency components are input into the double-flow backbone network to obtain low-frequency features and high-frequency features; The low-frequency features and the high-frequency features are input into the residual gate fusion module to obtain the fused bone joint data.

8. The method of claim 2, wherein, The third network layer includes a second fully connected layer; the fused bone joint data is input into the third network layer for classification to obtain an injury grade corresponding to the bone joint injury of the human body, including: The fused bone joint data is input into the second fully connected layer to obtain the injury grade.

9. The method of claim 1, wherein, The method further includes: The injury grade is corrected according to human body information to obtain a corrected injury grade; the human body information includes description information of the human body on the bone joint, age information of the human body, weight information of the human body, and hobby information of the human body; The corrected injury grade is displayed.

10. An electronic device, comprising: It includes: A memory having a computer program stored thereon; A processor configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1-9.

Images