Bone joint motion analysis method and system

Abstract

The present invention relates to the technical field of bone joint motion analysis, and provides a bone joint motion analysis method, comprising the following steps: S1: establishing a bone joint motion analysis depth model; S2: setting at least two image shooting devices at different viewing angles based on the bone joint, and acquiring at least two images at different viewing angles at a first moment and a second moment to generate a first image set and a second image set; S3: inputting the first image set into the bone joint motion analysis model, outputting first three-dimensional key point position information, and converting it into first coordinate information; inputting the second image set into the bone joint motion analysis model, outputting second three-dimensional key point position information, and converting it into second coordinate information; S4: calculating the relative rotation and displacement at the second moment relative to the first moment based on the first coordinate information and the second coordinate information. Utilizing multi-view images and based on a deep learning method, the method can quickly realize the recognition of bone joint feature points, and then establish a coordinate system to solve the joint motion information.

Description

Technical Field

[0001] The present invention relates to the technical field of bone and joint motion analysis, and in particular to a bone and joint motion analysis method and system. Background Art

[0002] In order to achieve rapid analysis of human joint movement, current methods often use RGB-D cameras (such as Kinect), reflective markers (Vicon motion capture system), inertial measurement units, etc.

[0003] Patent application CN112989996A uses motion capture equipment and infrared light stereo tracking technology to achieve dynamic tracking of hip and knee joints. After establishing a human skeleton model, it calculates the six-degree-of-freedom kinematic data of the bones and joints in real time and quickly estimates human motion.

[0004] Patent application CN114931743A uses continuous camera images to identify key point information of the human body and completes the calculation of the flexion perspective of the main joints of the human body.

[0005] Continuously exposed X-ray images can accurately and dynamically display bone and joint movements without being restricted by soft tissue. Patent application CN112184782A uses X-ray images for motion analysis. After pre-loading three-dimensional volume data, it uses the method of generating virtual X-ray images to perform three-dimensional to two-dimensional alignment. After alignment, six-degree-of-freedom motion analysis based on the coordinate system is performed.

[0006] Although the above method can achieve a certain degree of motion analysis, it is limited by skin and soft tissue, and its accuracy error can reach the centimeter level, seriously affecting the motion assessment of bone and joints in actual clinical measurements.

[0007] Patent application CN112989996A describes a skeleton-based model for evaluating knee joint movement. This method requires pre-attaching reflective markers to the human body to evaluate joint movement, and cannot avoid errors in skin and soft tissue.

[0008] Patent application CN114931743A describes a human motion assessment method based on an RGB camera. Although key points are used to locate human joints, it is limited by viewing angles, key point positioning accuracy, and soft tissue errors, and cannot achieve accurate six-degree-of-freedom motion assessment of bone joints. It can only complete the flexion perspective measurement of the main joints, and its functions are very simple.

[0009] Patent application CN112184782A uses X-ray images for motion analysis, but the registration-based method is inefficient. Although the results are accurate, it is difficult to achieve rapid motion assessment and is not practical in rapid clinical diagnosis. Summary of the Invention

[0010] In response to the above problems, the purpose of the present invention is to provide a bone joint motion analysis method and system, which uses multi-view X-ray images and a deep learning method to quickly identify bone joint feature points and then establish a coordinate system to solve joint motion information.

[0011] The above-mentioned object of the present invention is achieved through the following technical solutions:

[0012] A method for analyzing bone joint motion comprises the following steps:

[0013] S1: Establish a deep model for bone and joint motion analysis;

[0014] S2: at least two image capturing devices at different viewing angles are provided based on the bone joint, and at a first moment and a second moment during the movement of the bone joint, at least two images captured by the image capturing devices at different viewing angles are respectively acquired to generate a first image set corresponding to the first moment and a second image set corresponding to the second moment;

[0015] S3: Inputting the first image set into the bone joint motion analysis model, outputting first three-dimensional key point position information at the first moment, and converting the information into first coordinate information; inputting the second image set into the bone joint motion analysis model, outputting second three-dimensional key point position information at the second moment, and converting the information into second coordinate information;

[0016] S4: Calculate the relative rotation and displacement of the bone joint at the second moment relative to the first moment based on the first coordinate information and the second coordinate information to obtain six-degree-of-freedom data.

[0017] Furthermore, in step S1, the bone joint motion analysis depth model is established, specifically:

[0018] Pre-collecting a training set for establishing the bone joint motion analysis depth model, the training set at least including images captured at the first moment and the second moment by at least two image capture devices located at different viewing angles;

[0019] Establishing a feature extraction module to extract multiple feature maps of different resolutions for each image in the training set, and upsampling the multiple feature maps of different resolutions to the same resolution and then splicing them to form a feature map after feature extraction;

[0020] Establishing a multi-view feature fusion module, for at least two images captured at different viewpoints at each moment, fusing the feature maps after feature extraction by directly adding all feature maps, directly concatenating all feature maps, or performing addition, concatenation, and convolution operations on the feature maps to obtain an updated feature map, which is then concatenated with the original feature map to output a new feature map;

[0021] Establishing a multi-task processing module to perform loss function settings for each image during the training of the bone joint motion analysis deep model, including bone segmentation, bone target detection, bone center point detection based on feature maps, and bone feature point recognition;

[0022] Establish a triangulation module, use the triangulation method to solve the current 3D key point position information based on the 2D key point position information of different perspectives predicted at the current moment, and establish the loss function solved by the triangulation method;

[0023] A training module is established to solve five loss functions for skeleton segmentation, skeleton target detection, skeleton center point detection based on feature map, skeleton feature point recognition, and triangulation method to obtain the total loss. Each task shares the weight of the feature extraction module and uses stochastic gradient descent to train the task.

[0024] Furthermore, each image in the training set collected in advance for establishing the bone joint motion analysis deep model is processed as follows:

[0025] Given a mask of bones in each image, the target bone area is marked as non-zero and the non-target bone area is marked as zero;

[0026] Define the skeleton's detection anchor box [x, y, w, h], where x and y are the top left corners of the detection anchor box, and w and h are the width and height of the detection anchor box.

[0027] Define the bone center point [x center ,y center ], the bone center point is the midpoint of the inner and outer malleolus or the midpoint of the inner and outer platforms of the bone;

[0028] Define the bone feature points, and use the following methods to extract the three-dimensional key points of the bones from the CT volume data using the three-dimensional Canny operator, manually select the bone feature points required to establish the bone anatomical coordinate system, and randomly select the three-dimensional key points of the bones to obtain the bone feature points, which are recorded as P = [x 1bone ,y 1bone ,z 1bone ,x 2bone ,y 2bone ,z 2bone,...], and based on each of the image capturing devices, project the bone feature points onto the image capturing device to obtain the pixel coordinates [x 1bone ,y 1bone ,x 2bone ,y 2bone ...], where x and y are the corresponding bone center points [x center ,y center ]Pixel coordinates along the X and Y directions of the image.

[0029] Furthermore, in the feature extraction module, the specific method of extracting the feature map is:

[0030] For each image in the training set, the backbone network HRnetV2 is used to obtain feature maps of different resolutions. And upsample multiple feature maps of different resolutions to After the same resolution is obtained, the feature map H after feature extraction is obtained. n The size of the feature map after feature extraction is W*H*C, where W and H are the width and height of the image, C is the number of feature map channels, and n is the sequence number of the image at each moment.

[0031] Furthermore, in the multi-view feature fusion module, for at least two images captured at different viewpoints at each moment, the feature map H after feature extraction is processed. 1 ...H n 、H n+ 1 ...H N Fusion is performed, in which the feature maps are added, concatenated, and convolved to obtain an updated feature map, which is then spliced with the original feature map to output a new feature map. Specifically:

[0032] For H n With H n+1 Splicing to obtain CH n , its dimensions are W*H*C*2;

[0033] Use 1*1 convolution to restore the number of channels to CCH n , its dimensions are W*H*C;

[0034] Using the element-wise addition method with H n Add together to get the feature map FH at this perspective n .

[0035] Furthermore, in the multi-task processing module, a loss function loss setting including skeleton segmentation, skeleton target detection, skeleton center point detection based on feature map, and skeleton feature point recognition is performed on each image, specifically:

[0036] Skeleton segmentation loss:

[0037] loss1=-y bone log(y pred )-(1-y bone )log(1-y pred )

[0038] Among them, y pred is the probability of predicting it as bone, y bone The probability of being a real bone;

[0039] Loss of skeleton target detection:

[0040]

[0041]

[0042]

[0043] Among them, IOU is the intersection-over-union ratio of the predicted skeleton and the skeleton annotation, γ is the aspect ratio 2 parameter, and w gt 、h gt is the width and height of the real annotation anchor box, w and h are the predicted width and height of the anchor box, α is the weight parameter, c is the predicted center, and c gt is the true annotation center, p is the Euclidean distance between the two points, and d is the diagonal distance between the intersection of the true and predicted boxes of the two points;

[0044] Skeleton center point detection loss:

[0045] loss3=-(1-p t ) γ log(p t )

[0046] Among them, p t is the predicted probability of the bone center point;

[0047] Skeleton feature point loss:

[0048]

[0049] Among them, N is the number of key points P defined, is the position of the key point marked, is the predicted key point position.

[0050] Furthermore, in the triangulation module, the three-dimensional key point position information at the current moment is solved using a triangulation method based on the two-dimensional key point position information of different perspectives predicted at the current moment, and a loss function loss solved by the triangulation method is established, specifically:

[0051] The two-dimensional positions of a series of key points at different perspectives obtained according to the prediction Using triangulation method to solve P pred , the loss is:

[0052]

[0053] Among them, N is the number of key points to be constructed, is the three-dimensional position of the real point, is the three-dimensional position of the predicted point.

[0054] A bone joint motion analysis system for executing the above-mentioned bone joint motion analysis method, the bone joint motion analysis system specifically comprising:

[0055] A feature extraction module is used to extract multiple feature maps of different resolutions from each image in the training set, and to upsample the multiple feature maps of different resolutions to the same resolution and then splice them to form a feature map after feature extraction;

[0056] a multi-view feature fusion module for fusing, for each moment, at least two images captured by the image capture device at different viewpoints, the feature maps extracted from the images by directly adding all feature maps, directly concatenating all feature maps, or performing addition, concatenation, and convolution operations on the feature maps to obtain an updated feature map, which is then concatenated with the original feature map to output a new feature map;

[0057] A multi-task processing module is used to set a loss function for each image during the training of the bone joint motion analysis deep model, including bone segmentation, bone target detection, bone center point detection based on feature maps, and bone feature point recognition;

[0058] The triangulation module is used to solve the 3D key point position information at the current moment based on the 2D key point position information of different perspectives predicted at the current moment using the triangulation method, and establish the loss function solved by the triangulation method;

[0059] The training module is used to solve the five loss functions for skeleton segmentation, skeleton target detection, skeleton center point detection based on feature maps, skeleton feature point recognition, and triangulation method to obtain the total loss. Each task shares the weight of the feature extraction module and uses stochastic gradient descent to train the task.

[0060] A computer device includes a memory and one or more processors, wherein the memory stores computer code, and when the computer code is executed by the one or more processors, the one or more processors execute the above method.

[0061] A computer-readable storage medium stores computer code. When the computer code is executed, the above method is performed.

[0062] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0063] (1) Based on dynamic X-ray motion calculation, there is no soft tissue movement error;

[0064] (2) It has multi-view depth information, which can provide six-degree-of-freedom information instead of two-dimensional plane view information. It can reconstruct three-dimensional points through multi-view X-ray images, establish bone joint coordinate systems, and calculate six-degree-of-freedom kinematics. The calculation does not require iterative registration operations.

[0065] (3) No registration is required, which reduces analysis time and improves clinical usability;

[0066] (4) Design of deep models for multi-task training such as target positioning, segmentation, and key point recognition to achieve more stable bone and joint feature recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is an overall flow chart of a bone joint motion analysis method in the first embodiment of the present invention;

[0068] Figure 2 Schematic diagram of establishing a depth model for bone joint motion analysis in the first embodiment of the present invention;

[0069] Figure 3 This is a flow chart of establishing a depth model for bone joint motion analysis in the first embodiment of the present invention;

[0070] Figure 4 Schematic diagram of an image capturing device for three-viewing angles in a first embodiment of the present invention;

[0071] Figure 5 Schematic diagram of multi-view feature fusion in the first embodiment of the present invention;

[0072] Figure 6 This is a structural diagram of the depth model for bone joint motion analysis in the second embodiment of the present invention. DETAILED DESCRIPTION

[0073] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0074] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0075] First embodiment

[0076] like Figure 1 As shown, this embodiment provides a bone joint motion analysis method, comprising the following steps:

[0077] S1: Establish a deep model for bone and joint motion analysis.

[0078] Specifically, in the present invention, to carry out motion analysis of bone joints, it is first necessary to establish a bone joint motion analysis depth model. After the bone joint motion analysis model is established, the bone joint motion data is analyzed based on the bone joint motion analysis model. It should be noted that establishing the bone joint motion analysis depth model is a prior step to bone joint motion analysis, but it does not mean that the bone joint motion analysis depth model needs to be re-established each time the bone joint motion analysis is carried out. After the bone joint motion analysis model is established, the bone joint motion analysis can be performed multiple times. The bone joint motion analysis depth model can be permanently used after it has been established, or the bone joint motion analysis depth model can be continuously optimized based on new bone joint motion data and images.

[0079] In the present invention, bone joints may refer to any bone joints, such as knee joints, hip joints, elbow joints, glenohumeral joints, etc. The present invention does not impose any restrictions on the specific types of bone joints, and the technical content of the present invention may be applicable to any bone joints.

[0080] The following is a deep model for bone and joint motion analysis (such as Figure 2 ) method, and explain it in detail, such as Figure 3 As shown, the specific steps include:

[0081] S11: Pre-collect a training set for establishing a depth model for bone joint motion analysis, where the training set at least includes images captured at a first moment and a second moment by at least two image capturing devices located at different viewing angles.

[0082] Concrete, owing to carry out a motion analysis, generally need at least in conjunction with the image data of two different moments just can analyze the bone joint motion situation between two moments, therefore, in the present embodiment, the image obtained need at least comprise first moment and second moment two moments.But, in the process of bone joint motion analysis, for the comprehensiveness to bone joint motion analysis, generally can not only analyze the motion of two two moment points, most of the cases need to carry out coherent analysis to the motion of continuous multiple moments.Therefore, in the process of bone joint motion, need to obtain the image of multiple different moments as much as possible, cover the motion moment that may be analyzed when following up motion analysis.In the present embodiment, be recorded as K different moments, namely carry out K times shooting.

[0083] At the same time, since the two-dimensional projection points on the images at two different perspectives (the shooting planes need to have a certain angle, cannot be on the same line, and cannot be parallel to each other) can be restored to three-dimensional points by triangulation, in this embodiment, at least two image shooting devices are required for multi-perspective shooting. However, in actual shooting, the more perspectives there are, the more accurate the subsequent analysis will be. It is possible to set as many perspectives as possible (such as Figure 4 That is, N images with different perspectives are taken at the same time.

[0084] In step S11, each image in the pre-collected training set for establishing the bone joint motion analysis depth model is processed, specifically:

[0085] (1) Given each image The mask of the lower skeleton (size is WxH, where W is width and H is height), where the target skeleton area is marked as non-zero and the non-target skeleton area is marked as zero;

[0086] (2) Define the skeleton detection anchor box [x, y, w, h], where x and y are the top left corner vertices of the detection anchor box, and w and h are the width and height of the detection anchor box;

[0087] (3) Define the center point of the skeleton [x center ,y center ], the center point of the bone is the midpoint of the inner and outer malleolus or the midpoint of the inner and outer platforms;

[0088] (3) Define the bone feature points. The bone feature points are obtained by using the three-dimensional Canny operator to extract the three-dimensional key points of the bone from the CT volume data, manually selecting the bone feature points required to establish the bone anatomical coordinate system, and randomly selecting the three-dimensional key points of the bone. The bone feature points are recorded as P = [x 1bone ,y 1bone ,z 1bone ,x 2bone ,y 2bone ,z 2bone ,...], and based on each image capturing device, the bone feature points are projected onto the image capturing device to obtain the pixel coordinates [x 1bone ,y 1bone ,x 2bone ,y 2bone ...], where x and y are the corresponding bone center points [x center ,y center ]Pixel coordinates along the X and Y directions of the image.

[0089] S12: Establish a feature extraction module to extract multiple feature maps of different resolutions for each image in the training set, and upsample the multiple feature maps of different resolutions to the same resolution and then splice them to form a feature map after feature extraction.

[0090] Specifically, the specific method of feature map extraction is: for each image in the training set, the backbone network HRnetV2 is used to obtain feature maps of different resolutions And upsample multiple feature maps of different resolutions to After the same resolution is stitched together, the feature map H after feature extraction is obtained. n The size of the feature map after feature extraction is W*H*C, where W and H are the width and height of the image, C is the number of feature map channels, and n is the sequence number of the image at each moment.

[0091] It should be noted that four feature maps with different resolutions are obtained using the backbone network HRnetV2 This is just a specific example. In actual applications, there can be more feature maps with different resolutions, or simply two or three. The more maps with different resolutions there are, the more accurate the subsequent analysis will be.

[0092] S13: Establish a multi-view feature fusion module. For at least two images captured at different viewpoints at each moment, the feature maps after feature extraction are fused using the following operations: (1) directly adding all feature maps, (2) directly splicing all feature maps, (3) adding, concatenating, and convolving the feature maps to obtain an updated feature map, which is then spliced with the original feature map to output a new feature map.

[0093] Specifically, for at least two images captured at different viewing angles at each moment, the feature map H after feature extraction is extracted. 1 ...H n 、H n+1 ...H N To merge, such as Figure 5 As shown, in this embodiment, method (3) is used as a specific example for explanation, specifically:

[0094] For H n With H n+1 Splicing to obtain CH n , its dimensions are W*H*C*2;

[0095] Use 1*1 convolution to restore the number of channels to CCH n , its dimensions are W*H*C;

[0096] Using the element-wise addition method with H n Add together to get the feature map FH at this perspective n .

[0097] S14: Establish a multi-task processing module. During the training of the deep model for bone and joint motion analysis, set the loss function for each image, including bone segmentation, bone target detection, bone center point detection based on feature maps, and bone feature point recognition. Specifically:

[0098] The loss of bone segmentation is used to improve the network's ability to predict the classification of bone regions in images during training:

[0099] loss1=-y bone log(y pred )-(1-y bone )log(1-y pred )

[0100] Among them, y pred is the probability of predicting it as bone, y bone The probability of being a real bone;

[0101] The loss of bone target detection is used to achieve the network's predictive ability for detecting bone regions in images during training:

[0102]

[0103]

[0104]

[0105] Among them, IOU is the intersection-over-union ratio of the predicted skeleton and the skeleton annotation, γ is the aspect ratio 2 parameter, and w gt 、h gt is the width and height of the real annotation anchor box, w and h are the predicted width and height of the anchor box, α is the weight parameter, c is the predicted center, and c gt is the true annotation center, p is the Euclidean distance between the two points, and d is the diagonal distance between the intersection of the true and predicted boxes of the two points;

[0106] Skeleton center point detection loss is used to implement the network's predictive ability for detecting key points of the skeleton center in images during training:

[0107] loss3=-(1-p t ) γ log(p t )

[0108] Among them, p t is the predicted probability of the bone center point;

[0109] Skeletal feature point loss is used to improve the network's ability to predict key bone feature point detection in images during training:

[0110]

[0111] Among them, N is the number of key points P defined, is the position of the key point marked, is the predicted key point position.

[0112] S15: Establish a triangulation module, use the triangulation method to solve the three-dimensional key point position information at the current moment based on the two-dimensional key point position information of different perspectives predicted at the current moment, and establish the loss function loss solved by the triangulation method, specifically:

[0113] According to the skeleton feature point and skeleton center point subtasks in the S14 multi-task processing module, the feature map FH extracted from the image is obtained. n After that, the skeleton center point is predicted to obtain the probability of different pixel positions in the image being the center point, and the point with the maximum probability is taken as the skeleton center point C bone ; The skeleton feature point subtask is based on FH n Output the distance C between each key point bone The regression results of the deviations are the series of deviation points J is the number of key points, and the position of the predicted point is finally obtained C in bones bone Corresponding addition x offset 、y offset . According to the real two-dimensional key point position Triangulate and reconstruct the 3D positions of key real points The three-dimensional position of the key prediction points is reconstructed by triangulation, and the final triangulation loss is:

[0114]

[0115] Among them, N is the number of constructed key points. This loss realizes the network's ability to locate the final three-dimensional key point position during training to serve the subsequent coordinate system establishment.

[0116] S16: Establish a training module to solve the five loss functions of bone segmentation, bone target detection, bone center point detection based on feature map, bone feature point recognition, and triangulation method to obtain the total loss s; each task shares the weight of the feature extraction module to achieve stable extraction of joint image features; use stochastic gradient descent training tasks to realize the network's prediction ability for different tasks.

[0117] S2: At least two image capturing devices at different perspectives are set based on the bone joints, and at the first moment and the second moment during the movement of the bone joints, at least two images at different perspectives captured by the image capturing devices are obtained respectively, to generate a first image set corresponding to the first moment and a second image set corresponding to the second moment.

[0118] Specifically, steps S2-S4 are specific steps for performing bone joint motion analysis based on the bone joint motion analysis model trained in step S1.

[0119] Similarly, in step S2, although only at least two image capturing devices at different viewing angles are set up to capture images at the first moment and the second moment, in the actual motion analysis process, as long as conditions permit, image capturing devices will be set up at as many different viewing angles as possible (e.g., Figure 4 Three imaging devices are set up, and multiple sets of images are captured at different times. For any bone joint (this could be the knee, hip, elbow, glenohumeral joint, etc.), multiple images are captured using N dynamic imaging devices (including but not limited to fluoroscopic C-arm X-ray machines, dynamic DR equipment, and standard static DR equipment). During the capture process, the bone joint completes a full range of motion (such as knee flexion). Each capture of K complete images (K different moments) will produce N images from different perspectives.

[0120] S3: Input the first image set into the bone joint motion analysis model, output the first three-dimensional key point position information at the first moment, and convert it into first coordinate information; input the second image set into the bone joint motion analysis model, output the second three-dimensional key point position information at the second moment, and convert it into second coordinate information.

[0121] Specifically, although each motion process includes multiple different moments, in the motion analysis process, it is generally necessary to analyze each two moments in time in pairs to analyze the motion changes between the two moments. It should be noted that the first moment and the second moment mentioned in the present invention do not refer to the first and second moments in the actual shooting process, but generally refer to the previous moment and the next moment in the process of analyzing each two moments in time in pairs.

[0122] In this embodiment, the first and second images are taken as examples to analyze the two time points in pairs, and the subsequent analysis of other time points is carried out using the same method. Specifically:

[0123] After obtaining K series of N images, for the first shot It is used as the input image of the bone joint motion analysis model to obtain J1 key point positions, which are converted into coordinate system C1. The above operation is repeated for the next image to obtain J2 key point positions, and the coordinate system C2 is updated.

[0124] For each key point and key point coordinate, in the process of obtaining key points, the bone joint motion analysis model is used to predict each three-dimensional key point P pred , establish a coordinate system based on key points to calculate kinematics; or according to different moments P pred The iterative closest point algorithm is used to obtain the rotation and displacement matrix relative to the initial position, and the rotation and displacement matrix is used to update the coordinate system position at this moment to obtain P at this moment. pred Corresponding coordinate system.

[0125] S4: Calculate the relative rotation and displacement of the bone joint at the second moment relative to the first moment based on the first coordinate information C1 and the second coordinate information C2 to obtain six-degree-of-freedom data.

[0126] Second embodiment

[0127] like Figure 6 This embodiment provides a bone joint motion analysis system for executing the bone joint motion analysis method of the first embodiment. The bone joint motion analysis system specifically includes:

[0128] The feature extraction module 11 is used to extract multiple feature maps of different resolutions for each image in the training set, and upsample the multiple feature maps of different resolutions to the same resolution and then splice them to form a feature map after feature extraction;

[0129] The multi-view feature fusion module 12 is configured to fuse the feature maps extracted from at least two images captured at different viewpoints at each moment by directly adding all feature maps, directly concatenating all feature maps, or performing addition, concatenation, and convolution operations on the feature maps to obtain an updated feature map, which is then concatenated with the original feature map to output a new feature map.

[0130] The multi-task processing module 13 is used to set a loss function for each image during the training of the deep model for bone joint motion analysis, including bone segmentation, bone target detection, bone center point detection based on feature maps, and bone feature point recognition;

[0131] The triangulation module 14 is used to solve the three-dimensional key point position information at the current moment using a triangulation method based on the two-dimensional key point position information of different perspectives predicted at the current moment, and establish a loss function loss solved by the triangulation method;

[0132] Training module 15 is used to solve the five loss functions of skeleton segmentation, skeleton target detection, skeleton center point detection based on feature map, skeleton feature point recognition, and triangulation method to obtain the total loss. Each task shares the weight of the feature extraction module and uses stochastic gradient descent to train the task.

[0133] A computer-readable storage medium stores computer code. When the computer code is executed, the above-described method is performed. A person skilled in the art will appreciate that all or part of the steps in the various methods of the above-described embodiments can be performed by a program instructing related hardware. The program can be stored in a computer-readable storage medium. The storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0134] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

[0135] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] It should be noted that the above embodiments can be freely combined as needed. The above description is only a preferred embodiment of the present invention. It should be pointed out that those skilled in the art can make several improvements and modifications without departing from the principles of the present invention, and such improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for analyzing bone and joint motion, characterized in that: The following steps are involved: S1: Establish a deep model for bone and joint motion analysis; S2: at least two image capturing devices at different viewing angles are provided based on the bone joint, and at a first moment and a second moment during the movement of the bone joint, at least two images captured by the image capturing devices at different viewing angles are respectively acquired to generate a first image set corresponding to the first moment and a second image set corresponding to the second moment; S3: Inputting the first image set into the bone joint motion analysis depth model, outputting first three-dimensional key point position information at the first moment, and converting the information into first coordinate information; inputting the second image set into the bone joint motion analysis depth model, outputting second three-dimensional key point position information at the second moment, and converting the information into second coordinate information; S4: Calculating the relative rotation and displacement of the bone joint at the second moment relative to the first moment based on the first coordinate information and the second coordinate information to obtain six-degree-of-freedom data; In step S1, the bone joint motion analysis depth model is established, specifically: Pre-collecting a training set for establishing the bone joint motion analysis depth model, the training set at least including images captured at the first moment and the second moment by at least two image capture devices located at different viewing angles; Establishing a feature extraction module to extract multiple feature maps of different resolutions for each image in the training set, and upsampling the multiple feature maps of different resolutions to the same resolution and then splicing them to form a feature map after feature extraction; Establishing a multi-view feature fusion module, for at least two images captured at different viewpoints at each moment, performing operations including adding, concatenating, and convolving the feature maps after feature extraction to obtain an updated feature map, then splicing it with the original feature map, and outputting a new feature map for fusion; Establishing a multi-task processing module to perform loss function settings for each image during the training of the bone joint motion analysis deep model, including bone segmentation, bone target detection, bone center point detection based on feature maps, and bone feature point recognition; Establish a triangulation module, use the triangulation method to solve the current 3D key point position information based on the 2D key point position information of different perspectives predicted at the current moment, and establish the loss function solved by the triangulation method; A training module is established to solve five loss functions for skeleton segmentation, skeleton target detection, skeleton center point detection based on feature map, skeleton feature point recognition, and triangulation method to obtain the total loss. Each task shares the weight of the feature extraction module and uses stochastic gradient descent to train the task.

2. The bone joint motion analysis method according to claim 1, characterized in that: Also includes: Each image in the training set collected in advance for establishing the bone joint motion analysis deep model is processed, specifically: Given a mask of bones in each image, the target bone area is marked as non-zero and the non-target bone area is marked as zero; Define the skeleton's detection anchor box [x, y, w, h], where x and y are the top left corners of the detection anchor box, and w and h are the width and height of the detection anchor box. Define the bone center point [x center ,y center ], the bone center point is the midpoint of the inner and outer malleolus or the midpoint of the inner and outer platforms of the bone; Define the bone feature points, and use the following methods to extract the three-dimensional key points of the bones from the CT volume data using the three-dimensional Canny operator, manually select the bone feature points required to establish the bone anatomical coordinate system, and randomly select the three-dimensional key points of the bones to obtain the bone feature points, which are recorded as P = [x 1bone ,y 1bone ,z 1bone ,x 2bone ,y 2bone ,z 2bone ,...], and based on each of the image capturing devices, project the bone feature points onto the image capturing device to obtain the pixel coordinates [x 1bone ',y 1bone ',x 2bone ',y 2bone '...], where x 1bone ',y 1bone ',x 2bone ',y 2bone '... respectively correspond to the center point of the bone [x center ,y center ] The pixel coordinates along the X and Y directions of the image.

3. The bone joint motion analysis method according to claim 2, characterized in that: In the feature extraction module, the specific method of extracting the feature map is: For each image in the training set, the backbone network HRnetV2 is used to obtain feature maps of different resolutions. And upsample multiple feature maps of different resolutions to After the same resolution is obtained, the feature map H after feature extraction is obtained. n The size of the feature map after feature extraction is W*H*C, where W and H are the width and height of the image, C is the number of feature map channels, and n is the sequence number of the image at each moment.

4. The bone joint motion analysis method according to claim 3, characterized in that: In the multi-view feature fusion module, for at least two images captured at different viewpoints at each moment, the feature map H after feature extraction is processed. 1 ...H n 、H n+1 ...H M Fusion is performed, in which the feature maps are added, concatenated, and convolved to obtain an updated feature map, which is then concatenated with the original feature map to output a new feature map. Specifically: For H n With H n+1 Splicing to obtain CH n , its dimensions are W*H*C*2; Use 1*1 convolution to restore the number of channels to CCH n , its dimensions are W*H*C; Using element-wise addition method with H n Add together to get the feature map FH at the corresponding perspective n .

5. The bone joint motion analysis method according to claim 4, characterized in that: In the multi-task processing module, the loss function loss setting including skeleton segmentation, skeleton target detection, skeleton center point detection based on feature map, and skeleton feature point recognition is performed on each image, specifically: Skeleton segmentation loss: loss1=-y bone log(y pred )-(1-and bone )log(1-y pred ) Among them, y pred is the probability of predicting it as bone, y bone The probability of being a real bone; Loss of skeleton target detection: Among them, IOU is the intersection-over-union ratio of the predicted skeleton and the skeleton annotation, γ is the aspect ratio of 2, and w gt 、h gt is the width and height of the real annotation anchor box, w and h are the predicted width and height of the anchor box, α is the weight parameter, c is the predicted center, and c gt is the true annotation center, p is the Euclidean distance between the two points, and d is the diagonal distance between the intersection of the true and predicted boxes of the two points; Skeleton center point detection loss: loss3=-(1-p t ) γ log(p t ) Among them, p t is the predicted probability of the bone center point; Skeleton feature point loss: Among them, N is the number of key points P defined, is the position of the key point marked, is the predicted key point position.

6. The bone joint motion analysis method according to claim 5, characterized in that: In the triangulation module, the three-dimensional key point position information at the current moment is solved using a triangulation method based on the two-dimensional key point position information of different perspectives predicted at the current moment, and a loss function loss solved by the triangulation method is established, specifically: The two-dimensional positions of a series of key points at different perspectives obtained according to the prediction Using triangulation method to solve P pred , the loss is: Among them, N is the number of key points to be constructed, is the three-dimensional position of the real point, is the three-dimensional position of the predicted point.

7. A bone joint motion analysis system for executing the bone joint motion analysis method according to any one of claims 1 to 6, characterized in that: The bone joint motion analysis system specifically includes: The feature extraction module is used to extract multiple feature maps of different resolutions for each image in the training set, and then upsample the multiple feature maps of different resolutions to the same resolution and splice them to form a feature map after feature extraction; a multi-view feature fusion module for fusing at least two images captured at different viewpoints at each moment, performing operations including adding, concatenating, and convolving the feature maps after feature extraction to obtain an updated feature map, then splicing it with the original feature map, and outputting a new feature map for fusing; A multi-task processing module is used to set a loss function for each image during the training of the bone joint motion analysis deep model, including bone segmentation, bone target detection, bone center point detection based on feature maps, and bone feature point recognition; The triangulation module is used to solve the 3D key point position information at the current moment based on the 2D key point position information of different perspectives predicted at the current moment using the triangulation method, and establish the loss function solved by the triangulation method; The training module is used to solve the five loss functions for skeleton segmentation, skeleton target detection, skeleton center point detection based on feature maps, skeleton feature point recognition, and triangulation method to obtain the total loss. Each task shares the weight of the feature extraction module and uses stochastic gradient descent to train the task.

8. A computer device comprising a memory and one or more processors, wherein the memory stores computer code, and when the computer code is executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 6. 9 . A computer-readable storage medium storing computer code, wherein when the computer code is executed, the method according to claim 1 is performed.

Images