A magnetic stimulation system and method based on computer vision
By combining computer vision and triangulation technology with robotic arm control, precise magnetic stimulation of the patient's entire muscle groups is achieved, solving the problems of insufficient coverage of fixed-position stimulation and low efficiency of manual stimulation in existing technologies, thus improving treatment efficiency and accuracy.
Patent Information
- Application Number
- CN202410464183.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-17
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-04-17
AI Technical Summary
Existing conventional transcranial magnetic stimulation is a localized stimulation at a fixed location, which cannot completely cover the entire muscle group treatment target area. In addition, manual stimulation methods are inefficient and have positioning errors.
A computer vision-based magnetic stimulation system is used to capture human images in real time from different angles using multiple cameras. Combined with triangulation, the three-dimensional position of key skeletal points is reconstructed, and a robotic arm is controlled to drive the magnetic stimulation therapy device for precise stimulation.
It enables real-time marking and autonomous treatment of key points throughout the patient's body, improving diagnostic efficiency and accuracy. The robotic arm stimulates along the direction of muscle fibers, improving the patient's motor control ability.
Smart Images

Figure CN118286603B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of magnetic stimulation, and in particular relates to a magnetic stimulation system and method based on computer vision. Background Art
[0002] Transcranial magnetic stimulation is a non-invasive neuromodulation technology that generates a changing electric field through a changing magnetic field in a coil, thereby regulating the plasticity of nerve cells / fibers at the stimulation site, etc., to achieve the purpose of improving patient function. According to different classification methods, it can involve single pulse transcranial magnetic stimulation, paired pulse transcranial magnetic stimulation, transcranial repetitive magnetic stimulation, etc. Among them, transcranial repetitive magnetic stimulation is the most widely used in clinical treatment, and its indications include various diseases with neurological function damage and structural damage.
[0003] When transcranial repetitive magnetic stimulation is applied to peripheral nerve plexuses, nerve trunks, and neuromuscular junctions (muscle groups), it is called repetitive peripheral magnetic stimulation. Repetitive peripheral magnetic stimulation has gradually become a hot topic in clinical research in recent years, and the stimulation targets are no longer limited to the central nervous system.
[0004] In addition: the remodeling of patients after nerve injury is usually closely related to functional improvement, that is, the functional improvement of patients is not only the contraction of local muscle fibers, but also the restoration of the original function of the muscle group. For example, due to radial nerve injury, the patient cannot perform wrist dorsiflexion. At this time, the function of wrist dorsiflexion should be improved, rather than being limited to the goal of contracting a certain muscle. By stimulating along the direction of the target muscle fibers and functional contraction, the control of the injured nerve to the periphery can be better promoted, thereby bringing more favorable rehabilitation results to patients.
[0005] However, the projection range of peripheral muscle groups or nerve plexuses on the body surface is larger than the magnetic field enrichment area of the stimulation coil. The existing conventional transcranial magnetic line stimulation is local stimulation at a fixed position, and it is still impossible to fully cover the treatment target area of the entire muscle group in one treatment. In addition, the current method of manually stimulating multiple treatment target areas of patients by manpower greatly reduces the efficiency of clinical treatment, and there may also be manual positioning errors of stimulation due to the change of treatment personnel. Summary of the invention
[0006] In view of this, the present invention provides a magnetic stimulation system and method based on computer vision to solve the problem that conventional transcranial magnetic stimulation is local stimulation at a fixed position. In addition, the current method of manually stimulating multiple treatment target areas of a patient using manpower has low clinical treatment efficiency, and there may also be errors in the positioning of the treatment area due to different treatment personnel.
[0007] The technical solution adopted by the present invention is as follows:
[0008] A computer vision-based magnetic stimulation system, comprising:
[0009] 1. Image acquisition module: including multiple cameras, which respectively collect two-dimensional images of the human body from different angles in real time;
[0010] The magnetic stimulation therapeutic device is arranged on one side of the treatment chair, wherein the treatment chair is taken as the zero point, the direction from the treatment chair to the magnetic stimulation therapeutic device is taken as the x-axis, the horizontal axis perpendicular to the x-axis is taken as the y-axis, and the vertical axis is taken as the z-axis to establish a three-dimensional space coordinate system;
[0011] There are three cameras, namely, Camera A, Camera B and Camera C, wherein Camera A is fixed on the magnetic stimulation therapy device, and Camera A is 1m away from the zero point in the z direction and 2m away from the zero point in the x direction; Camera B is 1.5m away from the zero point in the z direction and about 2m away from the zero point in the y direction; Camera C is 2m away from the zero point in the z direction, 3m away from the zero point in the x direction, and 1m away from the zero point in the y direction. Under this placement condition, the three cameras have enough overlapping fields of view, focusing on the relative zero point where the seat is located. At the same time, the three cameras can collect the key points of the patient from three directions, ensuring angle diversity. At the same time, attention should be paid to clear the debris in the main body of the field of view and ensure sufficient lighting to obtain good image quality. If more than three cameras are set, the considerations for their placement are the same as described above.
[0012] The layout of cameras should take the following factors into consideration:
[0013] 1.1 Field of view overlap:
[0014] Basic overlap: The cameras should be placed so that their fields of view overlap significantly in the area that the point of interest is expected to move in. This overlap is critical for triangulation, ensuring that at any given time the point can be seen from at least two different angles.
[0015] Complete coverage: Ideally, the entire region of interest should be covered by the overlapping fields of view of the cameras to ensure that the point can be tracked continuously throughout its motion.
[0016] 1.2. Baseline distance:
[0017] Adequate spacing: Cameras should not be placed too close relative to the size of the room and treatment area. A larger baseline (the distance between any two camera locations) can improve triangulation accuracy by providing a wider difference in viewing angles, which is critical for depth estimation.
[0018] Avoid too large a distance: On the other hand, if the cameras are too far apart, the keypoints in the image may appear very small, or the parallax may be too large, making it difficult to match corresponding points in the image.
[0019] 1.3. Angle Diversity:
[0020] Different angles: Place cameras to capture the scene from different angles. This diversity helps reduce ambiguity in matching points across different camera views and improves the robustness of 3D reconstruction.
[0021] Avoid parallel alignment: Cameras should not be aligned in parallel or capture the scene from too similar angles, which will reduce the effectiveness of triangulated depth estimates.
[0022] 1.4. Minimize occlusion:
[0023] Place the cameras to minimize occlusion of key points, especially by permanent fixtures or furniture in the room. Occlusion can cause tracking to be lost or the reconstructed path to be inaccurate.
[0024] 1.5. Height and elevation:
[0025] Height considerations: The camera should be placed at a height that provides a clear view of the area of interest, avoiding obstacles and reducing the possibility of occlusion.
[0026] Angle considerations: The choice of elevation angle should balance the need for a good view of the region of interest with the need to minimize perspective distortion and ensure valid point detection.
[0027] 1.6. Lighting and background:
[0028] Lighting conditions: Ensure that the camera position does not cause direct glare from light sources and that the area of interest is well lit to optimize image quality.
[0029] Contrast and background: Keep the background of the treatment area clean during treatment to ensure that the deep learning model has enough contrast for accurate detection and tracking.
[0030] Each camera is calibrated using the following method:
[0031] First, each of the three cameras (located at positions A, B, and C) needs to be calibrated to determine its intrinsic parameters (such as focal length, principal point, and lens distortion coefficients) and extrinsic parameters (the position and orientation of each camera). Calibration can be done using calibration patterns (such as a chessboard) placed at different positions and orientations in the room. Here’s how it works:
[0032] The chessboard pattern provides a clear, high-contrast grid that the software can easily detect. The grid of black and white squares provides precise tracking points for the calibration algorithm. The calibration software first analyzes the captured image to detect the corners of the chessboard squares. These detected corners are used to calculate the camera's intrinsic parameters (such as focal length, principal point, and distortion coefficients) and extrinsic parameters (the position and orientation of the camera relative to the chessboard). Parameter estimation: Using the detected corners and their known real-world coordinates on the chessboard, the software uses a mathematical model to estimate the camera's parameters. This process typically involves solving a series of equations derived from the pinhole camera model and the lens distortion model. This task can be accomplished by functions such as findChessboardCorners and calibrateCamera provided by opencv (computer open source vision library) to achieve the purpose of detecting chessboard patterns and calibrating the camera.
[0033] 2. Image processing module: Process the collected two-dimensional images based on computer vision to mark the key points of bones on the two-dimensional images, and connect the key points according to the predefined limb connection structure to construct a visual image of the human skeleton with labels;
[0034] The image processing module comprises:
[0035] Preprocessing module: used to normalize the size of the real-time acquired two-dimensional image; that is, to adjust the image size to a fixed size without distorting its aspect ratio;
[0036] Processing module: Input the two-dimensional image processed by the preprocessing module into the confidence score acquisition model, output the key points of different parts of the human body, and connect the key points according to the predefined limb connection structure to construct a visual image of the human skeleton.
[0037] The processing module outputs the key points of different parts of the human body and, through pre-training and learning, outputs the confidence scores corresponding to the key points. The processing module sets a threshold to filter out the key points with low confidence.
[0038] The threshold is PCK@0.2;
[0039] The method of outputting the confidence scores corresponding to the key points through pre-training and learning specifically includes the following steps:
[0040] Step S1: Collect video information of N subjects through a camera, manually mark key points of different parts of each subject, and form a data set, which is divided into a training set and a test set in a ratio of 7:3;
[0041] Step S2: importing the training set into the confidence score acquisition model for training to obtain the confidence scores corresponding to the key points;
[0042] Step S3: testing the confidence score acquisition model based on the test set and the confidence scores in S2;
[0043] Step S4: Obtain video information collected by the camera in real time, input the video information into the confidence score acquisition model, and obtain the confidence score corresponding to the current human body key point.
[0044] In step S2, the confidence score acquisition model includes:
[0045] 2.1 Sitting posture feature adaptation layer: by modifying or adding layers, or adjusting the convolution layer, more weight is given to the sitting posture area to better capture the features related to the lower body; wherein, the adjustment convolution layer is implemented by the following formula:
[0046]
[0047] Where Y is the feature matrix output after the convolution layer is optimized, X is the feature matrix of the original image after processing and input to the model, i and j are the indexes of the feature matrix elements, K is the convolution layer filter, a and b are the length and width of the convolution kernel respectively, and w is the weight function of K. By adjusting the value of the weight function w matrix, more weight can be given to the sitting area to optimize the extraction and prediction of sitting features;
[0048] 2.2 Depth estimation adjustment layer: used to distinguish limbs that may overlap in a sitting position; the depth estimation adjustment layer includes:
[0049] Introduce the overlap estimation step: Separation(r, s) = σ(∑ k W k ·H k (r, s) + bias) where H is the feature matrix of the extracted edge and texture of the subject, r, s are the one-dimensional index and two-dimensional index of the feature matrix, W is the weight, bias is the bias term, σ is the sigmoid function, and Separation is the overlap probability function;
[0050] Introduce the depth correction term: C(r, s) = Separation(r, ,s)(D est (r, s)-D gt (r,s)) 2 Where D est is the depth estimate of the limb in the input image (on a two-dimensional plane), D gt It is based on the groundtruth of the training set, and the depth correction term is minimally optimized to obtain the model's perception of deep features (limb overlap);
[0051] 2.3 Dataset and training adjustment layer: used to introduce posture variability, so that the training data contains various sitting postures, and introduce training samples of the lower body that may be occluded, so that the confidence score acquisition model can make correct estimates for the occluded data;
[0052] The custom loss function layer described in 2.4 is constructed by the following steps:
[0053] Step A1: Construct a key point accuracy loss function to optimize the accuracy of the key points, as shown in the following formula:
[0054]
[0055] Where P m is the position value of the predicted joint key point in the sitting posture, m represents the index of the visible key point, is the ground truth of the training set, w i To optimize the weight function for sitting posture prediction, M is the total number of visible key points;
[0056] Step A2: Construct a key point visibility loss function to optimize the confidence score when the key point is invisible to obtain the accuracy of the model, as shown in the following formula:
[0057]
[0058] where v n Indicates the weight of whether the nth key point is visible, n represents the index of the key point, N represents the total number of key points, y n is an indicator function indicating whether the key point is visible. represents the probability predicted by the model that the keypoint is visible;
[0059] Step A3: Construct a pose consistency loss function to ensure that the output of the confidence score acquisition model follows certain anatomical rules, as shown in the following formula:
[0060]
[0061] Where S represents the key point pairs that are connected to each other in an anatomical sense, Z is the number of point pairs, and P x Represents the actual point, represents the predicted point, x represents the number of a key point (or index, such as the index of the knee joint key point), and y represents the corresponding neighboring joint point number (or index, such as in the knee-ankle joint point pair, x represents the knee joint key point, and y represents the ankle joint key point);
[0062] Step A4: Determine a custom loss function layer based on the position of the camera and the key point accuracy loss function, key point visibility loss function, and pose consistency loss function constructed in steps A1-A3, as shown in the following formula:
[0063]
[0064] Where p is the camera number, q is the number of the key point accuracy loss function, key point visibility loss function and pose consistency loss function. By optimizing the hybrid loss function, the model can achieve better prediction results in terms of key point prediction, key point visibility, pose consistency, etc.
[0065] 2.5 The context perception layer is as follows:
[0066] C perception =softmax(W c ·F+b c )
[0067] Among them C perception is the output of the context-aware layer, representing the probability distribution of different contexts, W c is the weight, b c is a bias, the context-aware layer is a fully connected layer that maps the feature F to the context output, and softmax normalizes the output into an activation function of the probability distribution.
[0068] 3. Multi-image information fusion module: reconstructs the 3D position of each skeleton key point in each time frame based on triangulation;
[0069] The multi-image information fusion module comprises:
[0070] Conversion module: used to convert the 2D pixel coordinates of key points on the visual image into the corresponding 3D rays in the camera coordinate system;
[0071] Triangulation module: based on the obtained 3D ray, obtain the corresponding intersecting or closest intersecting spatial 3D point;
[0072] Coordinate acquisition module: obtains the coordinates of the 3D points of each camera in the time series, and the obtained spatial 3D point coordinates are fed back to the robotic arm system in real time;
[0073] 4. Input unit, robotic arm, magnetic stimulation therapy device and controller: The input unit is used to input the path of the human body to be stimulated. The controller controls the robotic arm to drive the magnetic stimulation therapy device to move according to the input stimulation path and based on the visual image of the marked human skeleton and the 3D position of each key point of the human skeleton in each time frame.
[0074] A computer vision-based magnetic stimulation method, comprising:
[0075] Step 1: Use multiple cameras to collect two-dimensional images of the human body from different angles in real time;
[0076] Step 2: Processing the acquired two-dimensional image based on computer vision to mark the key points of the skeleton on the two-dimensional image, and connecting the key points according to the predefined limb connection structure to construct a visual image of the human skeleton with the mark;
[0077] Step 3: Reconstruct the 3D position of each skeleton keypoint at each time frame based on triangulation;
[0078] Step 4: Input the part of the human body to be stimulated through the input unit. The controller controls the robotic arm to move the magnetic stimulation therapeutic device according to the input stimulation task, based on the visualized image of the human skeleton with labels and the 3D position of each key point of the human skeleton in each time frame.
[0079] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:
[0080] 1. In the present invention, a camera is added to the overall device, and a computer vision algorithm is used to process images from different cameras in real time, and multi-angle information fusion is performed in the software to obtain the three-dimensional position information of the key points of the patient's joints. The patient can mark the key points of the patient's body in real time, and only needs to input the target area and stimulation area to be treated in the software of the magnetic stimulator, and then feedback can be given to the mechanical arm to complete the treatment course autonomously, greatly improving the efficiency and accuracy of diagnosis and treatment.
[0081] 2. In the present invention, when performing peripheral magnetic stimulation on patients with motor dysfunction, the robotic arm stimulates the coil along the direction of muscle fiber travel, and at the same time leads the patient to actively cooperate in making corresponding movements, which can better improve the patient's motor control ability. BRIEF DESCRIPTION OF THE DRAWINGS
[0082] The present invention will now be described by way of example with reference to the accompanying drawings, in which:
[0083] Figure 1 It is a schematic diagram of the process structure of the present invention;
[0084] Figure 2 It is a schematic diagram of key point markings of the human body of the present invention;
[0085] Figure 3 It is a schematic diagram of the three-dimensional coordinate structure of the present invention. DETAILED DESCRIPTION
[0086] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0087] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0088] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0089] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0090] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.
[0091] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0092] Example 1
[0093] like Figure 1-Figure 3 As shown, an embodiment of the present invention discloses a magnetic stimulation system based on computer vision, including:
[0094] 1. Image acquisition module: including multiple cameras, which respectively collect two-dimensional images of the human body from different angles in real time;
[0095] The magnetic stimulation therapeutic device is arranged on one side of the treatment chair, wherein the treatment chair is taken as the zero point, the direction from the treatment chair to the magnetic stimulation therapeutic device is taken as the x-axis, the horizontal axis perpendicular to the x-axis is taken as the y-axis, and the vertical axis is taken as the z-axis to establish a three-dimensional space coordinate system;
[0096] There are three cameras, namely, Camera A, Camera B and Camera C, wherein Camera A is fixed on the magnetic stimulation therapy device, and Camera A is 1m away from the zero point in the z direction and 2m away from the zero point in the x direction; Camera B is 1.5m away from the zero point in the z direction and about 2m away from the zero point in the y direction; Camera C is 2m away from the zero point in the z direction, 3m away from the zero point in the x direction, and 1m away from the zero point in the y direction. Under this placement condition, the three cameras have enough overlapping fields of view, focusing on the relative zero point where the seat is located. At the same time, the three cameras can collect the key points of the patient from three directions, ensuring angle diversity. At the same time, attention should be paid to clear the debris in the main body of the field of view and ensure sufficient lighting to obtain good image quality. If more than three cameras are set, the considerations for their placement are the same as described above.
[0097] The layout of cameras should take the following factors into consideration:
[0098] 1.1 Field of view overlap:
[0099] Basic overlap: The cameras should be placed so that their fields of view overlap significantly in the area that the point of interest is expected to move in. This overlap is critical for triangulation, ensuring that at any given time the point can be seen from at least two different angles.
[0100] Complete coverage: Ideally, the entire region of interest should be covered by the overlapping fields of view of the cameras to ensure that the point can be tracked continuously throughout its motion.
[0101] 1.2. Baseline distance:
[0102] Adequate spacing: Cameras should not be placed too close relative to the size of the room and treatment area. A larger baseline (the distance between any two camera locations) can improve triangulation accuracy by providing a wider difference in viewing angles, which is critical for depth estimation.
[0103] Avoid too large a distance: On the other hand, if the cameras are too far apart, the keypoints in the image may appear very small, or the parallax may be too large, making it difficult to match corresponding points in the image.
[0104] 1.3. Angle Diversity:
[0105] Different angles: Place cameras to capture the scene from different angles. This diversity helps reduce ambiguity in matching points across different camera views and improves the robustness of 3D reconstruction.
[0106] Avoid parallel alignment: Cameras should not be aligned in parallel or capture the scene from too similar angles, which will reduce the effectiveness of triangulated depth estimates.
[0107] 1.4. Minimize occlusion:
[0108] Place the cameras to minimize occlusion of key points, especially by permanent fixtures or furniture in the room. Occlusion can cause tracking to be lost or the reconstructed path to be inaccurate.
[0109] 1.5. Height and elevation:
[0110] Height considerations: The camera should be placed at a height that provides a clear view of the area of interest, avoiding obstacles and reducing the possibility of occlusion.
[0111] Angle considerations: The choice of elevation angle should balance the need for a good view of the region of interest with the need to minimize perspective distortion and ensure valid point detection.
[0112] 1.6. Lighting and background:
[0113] Lighting conditions: Ensure that the camera position does not cause direct glare from light sources and that the area of interest is well lit to optimize image quality.
[0114] Contrast and background: Keep the background of the treatment area clean during treatment to ensure that the deep learning model has enough contrast for accurate detection and tracking.
[0115] Each camera is calibrated using the following method:
[0116] First, each of the three cameras (located at positions A, B, and C) needs to be calibrated to determine its intrinsic parameters (such as focal length, principal point, and lens distortion coefficients) and extrinsic parameters (the position and orientation of each camera). Calibration can be done using calibration patterns (such as a chessboard) placed at different positions and orientations in the room. Here’s how it works:
[0117] The chessboard pattern provides a clear, high-contrast grid that the software can easily detect. The grid of black and white squares provides precise tracking points for the calibration algorithm. The calibration software first analyzes the captured image to detect the corners of the chessboard squares. These detected corners are used to calculate the camera's intrinsic parameters (such as focal length, principal point, and distortion coefficients) and extrinsic parameters (the position and orientation of the camera relative to the chessboard). Parameter estimation: Using the detected corners and their known real-world coordinates on the chessboard, the software uses a mathematical model to estimate the camera's parameters. This process typically involves solving a series of equations derived from the pinhole camera model and the lens distortion model. This task can be accomplished by functions such as findChessboardCorners and calibrateCamera provided by opencv (computer open source vision library) to achieve the purpose of detecting chessboard patterns and calibrating the camera.
[0118] 2. Image processing module: Process the collected two-dimensional images based on computer vision to mark the key points of bones on the two-dimensional images, and connect the key points according to the predefined limb connection structure to construct a visual image of the human skeleton with labels;
[0119] The image processing module comprises:
[0120] Preprocessing module: used to normalize the size of the real-time acquired two-dimensional image; that is, to adjust the image size to a fixed size without distorting its aspect ratio;
[0121] Processing module: Input the two-dimensional image processed by the preprocessing module into the confidence score acquisition model, output the key points of different parts of the human body, and connect the key points according to the predefined limb connection structure to construct a visual image of the human skeleton.
[0122] The processing module outputs the key points of different parts of the human body and, through pre-training and learning, outputs the confidence scores corresponding to the key points. The processing module sets a threshold to filter out the key points with low confidence.
[0123] The threshold is PCK@0.2;
[0124] The method of outputting the confidence scores corresponding to the key points through pre-training and learning specifically includes the following steps:
[0125] Step S1: Collect video information of N subjects through a camera, manually mark key points of different parts of each subject, and form a data set, which is divided into a training set and a test set in a ratio of 7:3;
[0126] Step S2: importing the training set into the confidence score acquisition model for training to obtain the confidence scores corresponding to the key points;
[0127] Step S3: testing the confidence score acquisition model based on the test set and the confidence scores in S2;
[0128] Step S4: Obtain video information collected by the camera in real time, input the video information into the confidence score acquisition model, and obtain the confidence score corresponding to the current human body key point.
[0129] In step S2, the confidence score acquisition model includes:
[0130] 2.1 Sitting posture feature adaptation layer: by modifying or adding layers, or adjusting the convolution layer, more weight is given to the sitting posture area to better capture the features related to the lower body; wherein, the adjustment convolution layer is implemented by the following formula:
[0131]
[0132] Where Y is the feature matrix output after the convolution layer is optimized, X is the feature matrix of the original image after processing and input to the model, i and j are the indexes of the feature matrix elements, K is the convolution layer filter, a and b are the length and width of the convolution kernel respectively, and w is the weight function of K. By adjusting the value of the weight function w matrix, more weight can be given to the sitting area to optimize the extraction and prediction of sitting features;
[0133] 2.2 Depth estimation adjustment layer: used to distinguish limbs that may overlap in a sitting position; the depth estimation adjustment layer includes:
[0134] Introduce the overlap estimation step: Separation(r,s)=σ(∑ k W k ·H k (r, s) + bias) where H is the feature matrix of the extracted edge and texture of the subject, r, s are the one-dimensional index and two-dimensional index of the feature matrix, W is the weight, bias is the bias term, σ is the sigmoid function, and Separation is the overlap probability function;
[0135] Introduce depth correction term: C(r, s) = Separation(r, s)(D est (r, s)-D gt (r,s)) 2 Where D est is the depth estimate of the limb in the input image (on a two-dimensional plane), D gt It is based on the groundtruth of the training set, and the depth correction term is minimally optimized to obtain the model's perception of deep features (limb overlap);
[0136] 2.3 Dataset and training adjustment layer: used to introduce posture variability, so that the training data contains various sitting postures, and introduce training samples of the lower body that may be occluded, so that the confidence score acquisition model can make correct estimates for the occluded data;
[0137] The custom loss function layer described in 2.4 is constructed by the following steps:
[0138] Step A1: Construct a key point accuracy loss function to optimize the accuracy of the key points, as shown in the following formula:
[0139]
[0140] Where P m is the position value of the predicted joint key point in the sitting posture, m represents the index of the visible key point, is the ground truth of the training set, w i To optimize the weight function for sitting posture prediction, M is the total number of visible key points;
[0141] Step A2: Construct a key point visibility loss function to optimize the confidence score when the key point is invisible to obtain the accuracy of the model, as shown in the following formula:
[0142]
[0143] where v n Indicates the weight of whether the nth key point is visible, n represents the index of the key point, N represents the total number of key points, y n is an indicator function indicating whether the key point is visible. represents the probability predicted by the model that the keypoint is visible;
[0144] Step A3: Construct a pose consistency loss function to ensure that the output of the confidence score acquisition model follows certain anatomical rules, as shown in the following formula:
[0145]
[0146] Where S represents the key point pairs that are connected to each other in an anatomical sense, Z is the number of point pairs, and P x Represents the actual point, represents the predicted point, x represents the number of a key point (or index, such as the index of the knee joint key point), and y represents the corresponding neighboring joint point number (or index, such as in the knee-ankle joint point pair, x represents the knee joint key point, and y represents the ankle joint key point);
[0147] Step A4: Determine a custom loss function layer based on the position of the camera and the key point accuracy loss function, key point visibility loss function, and pose consistency loss function constructed in steps A1-A3, as shown in the following formula:
[0148]
[0149] Where p is the camera number, q is the number of the key point accuracy loss function, key point visibility loss function and pose consistency loss function. By optimizing the hybrid loss function, the model can achieve better prediction results in terms of key point prediction, key point visibility, pose consistency, etc.
[0150] 2.5 The context perception layer is as follows:
[0151] C perception =softmax(W c ·F+b c )
[0152] Among them C perception is the output of the context-aware layer, representing the probability distribution of different contexts, W c is the weight, b c is a bias, the context-aware layer is a fully connected layer that maps the feature F to the context output, and softmax normalizes the output into an activation function of the probability distribution.
[0153] 3. Multi-image information fusion module: reconstructs the 3D position of each skeleton key point in each time frame based on triangulation;
[0154] The multi-image information fusion module comprises:
[0155] Conversion module: used to convert the 2D pixel coordinates of key points on the visual image into the corresponding 3D rays in the camera coordinate system;
[0156] Triangulation module: based on the obtained 3D ray, obtain the corresponding intersecting or closest intersecting spatial 3D point;
[0157] Coordinate acquisition module: obtains the coordinates of the 3D points of each camera in the time series, and the obtained spatial 3D point coordinates are fed back to the robotic arm system in real time;
[0158] 4. Input unit, robotic arm, magnetic stimulation therapy device and controller: The input unit is used to input the path of the human body to be stimulated. The controller controls the robotic arm to drive the magnetic stimulation therapy device to move according to the input stimulation path and based on the visual image of the marked human skeleton and the 3D position of each key point of the human skeleton in each time frame.
[0159] Example 2
[0160] like Figure 1 As shown, this embodiment proposes a magnetic stimulation method based on computer vision, including:
[0161] Step 1: Use multiple cameras to collect two-dimensional images of the human body from different angles in real time;
[0162] Step 2: Processing the acquired two-dimensional image based on computer vision to mark the key points of the skeleton on the two-dimensional image, and connecting the key points according to the predefined limb connection structure to construct a visual image of the human skeleton with the mark;
[0163] Step 3: Reconstruct the 3D position of each skeleton keypoint at each time frame based on triangulation;
[0164] Step 4: Input the part of the human body to be stimulated through the input unit. The controller controls the robotic arm to move the magnetic stimulation therapeutic device according to the input stimulation task, based on the visualized image of the human skeleton with labels and the 3D position of each key point of the human skeleton in each time frame.
[0165] The circuits, electronic components and modules involved are all prior art and can be fully implemented by those skilled in the art. Needless to say, the content protected by the present invention does not involve improvements to software and methods.
[0166] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0167] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A magnetic stimulation system based on computer vision, characterized in that: include: Image acquisition module: including multiple cameras, which respectively acquire two-dimensional images of the human body from different angles in real time; Image processing module: processes the collected two-dimensional images based on computer vision to mark the key points of bones on the two-dimensional images, and connects the key points according to the predefined limb connection structure to construct a visual image of the human skeleton with labels; Multi-image information fusion module: reconstructs the 3D position of each skeleton key point in each time frame based on triangulation; Input unit, robotic arm, magnetic stimulation therapeutic device and controller: The input unit is used to input the part of the human body to be stimulated. The controller controls the mechanical arm to drive the magnetic stimulation therapeutic device to move according to the input stimulation task, based on the visualized image of the human skeleton with the mark and the 3D position of each key point of the human skeleton in each time frame; the image processing module includes: Preprocessing module: used to standardize the size of the two-dimensional images collected in real time; Processing module: input the two-dimensional image processed by the preprocessing module into the confidence score acquisition model, output the key points of different parts of the human body, and connect the key points according to the predefined limb connection structure to construct a visual image of the human skeleton; The processing module outputs the key points of different parts of the human body and, through pre-training and learning, outputs the confidence scores corresponding to the key points. The processing module sets a threshold to filter out the key points with low confidence. The method of outputting the confidence scores corresponding to the key points through pre-training and learning specifically includes the following steps: Step S1: Collect video information of N subjects through a camera, mark key points of different parts of each subject, and form a data set, which is divided into a training set and a test set in a ratio of 7:3; Step S2: importing the training set into the confidence score acquisition model for training to obtain the confidence scores corresponding to the key points; Step S3: testing the confidence score acquisition model based on the test set and the confidence scores in S2; Step S4: obtaining video information collected by the camera in real time, inputting the video information into the confidence score acquisition model, and obtaining the confidence score corresponding to the current human body key point; In step S2, the confidence score acquisition model includes: Sitting posture feature adaptation layer: by modifying or adding layers, or adjusting the convolutional layer, more weight is given to the sitting area to better capture the features related to the lower body; Depth estimation adjustment layer: used to distinguish limbs that may overlap in a sitting position; Dataset and training adjustment layer: used to introduce posture variability, so that the training data contains various sitting postures, and introduce training samples of the lower body that may be occluded, so that the confidence score acquisition model can make correct estimates for the occluded data; Customized loss function layer: By introducing a loss function, more emphasis is placed on accurately predicting the joint positions related to the sitting posture; Context-aware layer: used to understand whether the person is sitting or standing and adjust the prediction accordingly; The custom loss function layer is constructed by the following steps: Step A1: Construct a key point accuracy loss function to optimize the accuracy of the key points, as shown in the following formula: Where P m is the position value of the predicted joint key point in the sitting posture, m represents the index of the visible key point, is the ground truth of the training set, w i To optimize the weight function for sitting posture prediction, M is the total number of visible key points; Step A2: Construct a key point visibility loss function to optimize the confidence score when the key point is invisible to obtain the accuracy of the model, as shown in the following formula: where v n Indicates the weight of whether the nth key point is visible, n represents the index of the key point, N represents the total number of key points, y n is an indicator function indicating whether the key point is visible. represents the probability predicted by the model that the key point is visible; Step A3: Construct a pose consistency loss function to ensure that the output of the confidence score acquisition model follows anatomical rules, as shown in the following formula: Where S represents the key point pairs that are connected to each other in an anatomical sense, Z is the number of point pairs, and P x Represents the actual point, represents the predicted point, x represents the number of a key point, and y represents the number of the corresponding neighboring joint point; Step A4: Determine a custom loss function layer based on the position of the camera and the key point accuracy loss function, key point visibility loss function, and pose consistency loss function constructed in steps A1-A3, as shown in the following formula: Where p is the camera number, q is the number of the key point accuracy loss function, the key point visibility loss function and the pose consistency loss function. By optimizing the hybrid loss function, the model can achieve better prediction results in terms of key point prediction, key point visibility, pose consistency and other aspects.
2. A computer vision-based magnetic stimulation system according to claim 1, characterized in that: The multi-image information fusion module comprises: Conversion module: used to convert the 2D pixel coordinates of key points on the visual image into the corresponding 3D rays in the camera coordinate system; Triangulation module: based on the obtained 3D ray, obtain the corresponding intersecting or closest intersecting spatial 3D point; Coordinate acquisition module: obtains the coordinates of the 3D points of each camera in the time series, and feeds the obtained spatial 3D point coordinates back to the robotic arm system in real time.
3. A computer vision-based magnetic stimulation system according to claim 1, characterized in that: It also includes a treatment chair, the magnetic stimulation therapeutic device is arranged on one side of the treatment chair, and a three-dimensional space coordinate system is established with the treatment chair as the zero point, the direction from the treatment chair to the magnetic stimulation therapeutic device as the x-axis, the horizontal axis perpendicular to the x-axis as the y-axis, and the vertical axis as the z-axis; There are three cameras, namely camera A, camera B and camera C, wherein camera A is fixed on the magnetic stimulation therapy device, the distance between camera A in the z direction and the zero point is 1m, the distance between camera A in the x direction and the zero point is 2m, and the distance between camera A in the y direction and the zero point is 0m; the distance between camera B in the z direction and the zero point is 1.5m, the distance between camera B in the y direction and the zero point is 2m, and the distance between camera C in the z direction and the zero point is 2m, the distance between camera C in the x direction and the zero point is 3m, and the distance between camera C in the y direction and the zero point is 1m.
4. A computer vision-based magnetic stimulation system according to claim 1, characterized in that: The threshold is PCK@0.
2.
5. A magnetic stimulation method based on computer vision, implemented by the system according to any one of claims 1 to 4, characterized in that: include: Step 1: Use multiple cameras to collect two-dimensional images of the human body from different angles in real time; Step 2: Processing the acquired two-dimensional image based on computer vision to mark the key points of the skeleton on the two-dimensional image, and connecting the key points according to the predefined limb connection structure to construct a visual image of the human skeleton with the mark; Step 3: Reconstruct the 3D position of each skeleton keypoint at each time frame based on triangulation; Step 4: Input the part of the human body to be stimulated through the input unit. The controller controls the robotic arm to move the magnetic stimulation therapeutic device according to the input stimulation task, based on the visualized image of the human skeleton with labels and the 3D position of each key point of the human skeleton in each time frame.
