A method for overcoming mechanical arm blocking to achieve human acupoint tracking

Abstract

The application discloses a method for realizing human acupoint tracking by overcoming mechanical arm shielding, and the method comprises the following steps: acquiring a real-time image of a target; processing a current frame image in the real-time image to acquire verification data; processing other images in the real-time image to acquire basic data; comparing the predicted data with the verification data; when the data do not correspond to each other, adding the verification data to the basic data through a Kalman filter to update the basic data, and re-comparing; and when the data correspond to each other, calculating and outputting predicted data of a next frame. The application divides the real-time image into a current frame image and a basic image, forms predicted data for matching, automatically switches different processing modes under different conditions, realizes continuous iteration of detection, prediction, matching and updating, provides stable tracking performance, improves prediction accuracy while reducing performance requirements, enhances the robustness of the algorithm in a complex scene, and effectively solves the problems of performance decline and inaccuracy of a traditional method in a shielding scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of physiotherapy robot control, and in particular to a method for overcoming mechanical arm shielding to achieve human acupoint tracking. BACKGROUND

[0002] With the rapid development of robot technology, physiotherapy robots have shown great application potential in the field of traditional Chinese medicine physiotherapy. Traditional Chinese medicine physiotherapy emphasizes the accuracy of acupoint massage, and more and more users are eager to achieve accurate massage of specific acupoints through intelligent means. This demand has driven the development of key point detection and tracking technology, making it an important support for accurate positioning and dynamic control. However, in complex dynamic environments, the application of physiotherapy robots still faces many challenges. Especially during the operation of the mechanical arm, its movement will cause shielding to the target area, resulting in a significant decrease in the detection and tracking accuracy of human key points such as the back and hands.

[0003] In the prior art, human key point tracking is mainly achieved by calculating the motion of pixels in image sequences. This tracking method has obvious limitations in practical applications, such as relying on open scene datasets, making it difficult to be directly applied to the shielding scene of physiotherapy robots; in the presence of mechanical arm shielding, existing algorithms are prone to losing the target, resulting in an inability to accurately predict the key point position of the shielded part, and difficulty in reassociating the original trajectory after the shielding is restored.

[0004] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0005] In order to solve the problem in the prior art that physiotherapy robot tracking of human key points is mainly achieved by calculating the motion of pixels in image sequences, such tracking method relies on open scene datasets, making it difficult to be applied to the shielding scene of physiotherapy robots, in the presence of mechanical arm shielding, existing algorithms are prone to losing the target, resulting in an inability to accurately predict the key point position of the shielded part, and difficulty in reassociating the original trajectory after the shielding is restored, the present application provides a method for overcoming mechanical arm shielding to achieve human acupoint tracking.

[0006] The present application is realized by the following technical solutions:

[0007] A method for overcoming mechanical arm shielding to achieve human acupoint tracking, wherein the method for overcoming mechanical arm shielding to achieve human acupoint tracking comprises:

[0008] acquiring a real-time image of a target;

[0009] processing a current frame image in the real-time image to obtain verification data, and processing images other than the current frame in the real-time image to obtain basic data;

[0010] The basic data is processed to obtain predicted data, and the predicted data is compared with the verification data.

[0011] When the predicted data does not correspond to the verification data, the verification data is added to the basic data for updating using a Kalman filter, and the predicted data and verification data are compared again; when the predicted data corresponds to the verification data, the predicted data for the next frame is calculated and output.

[0012] The method for overcoming robotic arm obstruction to achieve human acupoint tracking includes processing the current frame image in the real-time image to obtain verification data; and processing images other than the current frame in the real-time image to obtain basic data, including:

[0013] Surface features are extracted from a single frame image, and the surface features are set as targets, including several targets;

[0014] Detection boxes are set for several targets, the detection boxes are labeled, and detection box data is generated based on the detection boxes and labeling information;

[0015] Based on the detection box, obtain several key points corresponding to the detection box, and generate key point coordinates and occlusion probabilities based on the several key points;

[0016] The detection box data corresponding to the current frame image, the key point coordinates, and the occlusion probability are defined as verification data; the set of detection box data excluding the current frame image and the set of key point coordinates are defined as basic data.

[0017] The method for overcoming obstruction by a robotic arm to achieve human acupoint tracking, wherein the detection frame data is represented as: D t ={d1,d2,...,d m}; A single detection box in the detection box data is represented as d i =[x i ,y i ,w i ,h i ], where x i ,y i ,w i ,h i Let x, y, width, and height represent the x-coordinate, y-coordinate, width, and height of the i-th detection box, respectively.

[0018] The coordinates of the key points are represented as: P i ={p i1 ,p i2 ,...,p iN};p i1Xi represents the position coordinates of the first key point corresponding to the ith detection frame, and N is the total number of key points;

[0019] The occlusion probability is represented as: V i i1 i2 iN i1 Xi represents the visibility probability of the first key point corresponding to the ith detection frame, and N is the total number of key points.

[0020] The method for overcoming the occlusion of the mechanical arm to realize the tracking of the human body acupoints, wherein the setting of the detection frame for the plurality of targets further comprises: assigning the same ID identification to the same target in different frames of images, and assigning different ID identifications to different targets in different frames of images;

[0021] When the prediction data does not correspond to the verification data, adding the verification data to the basic data for updating through the Kalman filter further comprises:

[0022] When the verification data does not match the prediction data, assigning a new ID identification to the target in the verification data that does not match the prediction data; and / or, when the target in the prediction data does not appear in the threshold number of continuous frames of the basic data, deleting the ID identification of the corresponding target.

[0023] The method for overcoming the occlusion of the mechanical arm to realize the tracking of the human body acupoints, wherein the processing of the basic data to obtain prediction data comprises:

[0024] Processing the set of detection frame data corresponding to the basic data and the set of key point coordinates;

[0025] Generating prediction frame data according to the set of detection frame data in the basic data;

[0026] Generating prediction key point coordinate data according to the set of key point coordinates in the basic data;

[0027] The prediction data is the prediction frame data and the prediction key point coordinate data.

[0028] The method for overcoming the occlusion of the mechanical arm to realize the tracking of the human body acupoints, wherein, when the prediction data does not correspond to the verification data, adding the verification data to the basic data for updating through the Kalman filter, and re-comparing the prediction data and the verification data comprises:

[0029] Setting a total cost matrix to calculate the matching degree of the prediction data and the verification data;

[0030] ​​​​The total cost matrix comprises:

[0031] An IOU cost matrix for representing an intersection over union of the detection frame data of the current frame image and the prediction frame data;

[0032] A key point position cost matrix for representing a position cost value between key point coordinates of the current frame image and prediction key point coordinate data;

[0033] A key point posture change cost matrix for representing a relative relationship between key point coordinates of the current frame image and prediction key point coordinate data, the relative relationship being an angle and distance ratio.

[0034] The method for overcoming mechanical arm occlusion to realize human body acupoint tracking, wherein

[0035] The calculation formula of the IOU cost matrix is: Wherein C IOU [i,j] represents an IOU cost value between the i-th detection frame and the j-th prediction frame in the cost matrix, and the greater the value, the lower the overlapping degree of the detection frame and the trajectory prediction frame;

[0036] The calculation formula of the key point position cost matrix is: Wherein C keypoints [i,j] represents a position cost value of the key point between the i-th detection frame and the j-th prediction frame in the cost matrix, P ik represents the coordinate position of the k-th key point of the i-th detection frame, represents the predicted coordinate of the k-th key point of the j-th prediction frame, and N represents the total number of key points, represents the Euclidean distance between the key point coordinates;

[0037] The calculation formula of the key point posture change cost matrix is:

[0038] Wherein M represents the number of all possible pair combinations, u and v represent a pair of paired key points in each frame, and represents the Euclidean distance between the two key points in the detection frame, and the closer the relative position ratio is to 1, the more similar the detection value and the prediction are.

[0039] The method for overcoming mechanical arm occlusion to realize human body acupoint tracking, wherein the setting of the total cost matrix and the calculation of the matching degree of the prediction data and the verification data further comprises:

[0040] Respectively setting weights for the IOU cost matrix, the key point position cost matrix and the key point posture change cost matrix to form the total cost matrix.

[0041] The total cost matrix is: C total = aC IOU + bC keypoints + gC pose ; a, b, g are used to control the ratio of different cost matrixes.

[0042] The method for overcoming the mechanical arm shielding to realize human acupoint tracking, wherein when the prediction data corresponds to the verification data, the prediction data of the next frame is calculated and outputted, including:

[0043] The verification data is added to the basic data, and the prediction data of the target in the next frame image is calculated according to the updated detection box data set and the key point coordinate set of the basic data; the prediction data of the target in the next frame image also includes a state vector of the target;

[0044] The calculation formula of the state vector is: X t = FX t-1 + u; wherein X t-1 is the state vector of the last frame, including the position and speed of the key point, F is a state transition matrix, defining the motion model of the state, and u represents random noise in the process, describing random changes.

[0045] The method for overcoming the mechanical arm shielding to realize human acupoint tracking, wherein the verification data is added to the basic data by the Kalman filter for updating, further including:

[0046] The state vector is updated, and the calculation formula of the state vector during the updating is:

[0047] X t = X t + K(Z t -HX t ); wherein Z t is the observation value of the current frame, that is, the detected detection box and the position of the key point, H is an observation matrix, mapping the state space to the observation space, K is a Kalman gain, controlling the fusion weight of the predicted value and the observation value;

[0048] wherein the calculation formula of the Kalman gain is: K = P t|t-1 H T (HP t|t-1 H T + R) -1 ; wherein P t|t-1 is the error covariance matrix in the prediction stage, H is the observation matrix, R is the observation noise covariance matrix, K is the Kalman gain, controlling the fusion weight of the predicted value and the observation value;

[0049] When V < 0.8, K = K*V, and R a = R x (1 + 1-V);

[0050] When V > 0.8, K = K.

[0051] The beneficial effects of the present application are that: the present application divides the real-time image into the current frame image and the image other than the current frame, and respectively processes to form verification data and basic data, forms the prediction data corresponding to the current frame through the basic data, when the verification data corresponds to the prediction data, calculates and outputs the prediction data of the next frame, when it does not correspond, then updates and processes through the Kalman filter to re-predict, realizes continuous iteration of detection, prediction, matching and updating, provides stable tracking performance while improving prediction accuracy and reducing performance requirements, can enhance the robustness of the algorithm in complex scenes, and effectively solves the problems of performance decline and inaccuracy of traditional methods in occlusion scenes. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is a flow chart of the method for overcoming the occlusion of the mechanical arm to realize the tracking of the acupoints of the human body of the present application;

[0053] Figure 2 is a logic diagram of the method for overcoming the occlusion of the mechanical arm to realize the tracking of the acupoints of the human body of the present application;

[0054] Figure 3 is a scene diagram of the user's back being occluded in the running state of the mechanical arm;

[0055] Figure 4 is a scene diagram of the user's hand not being occluded in the running state of the mechanical arm;

[0056] Figure 5 is a scene diagram of the user's hand being partially occluded by the mechanical arm in the running state of the mechanical arm;

[0057] Figure 6 is an image processing diagram corresponding to Figure 3 of the present application;

[0058] Figure 7 is an image processing diagram corresponding to Figure 4 of the present application;

[0059] Figure 8 is an image processing diagram corresponding to Figure 5 of the present application. DETAILED DESCRIPTION

[0060] To make the purpose, technical scheme and effect of the present application more clear and definite, the present application is further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0061] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, motion condition, etc. between components in a certain specific posture (as shown in the drawings), and if the specific posture changes, the directional indications will also change accordingly.

[0062] In addition, if the embodiments of the present application involve descriptions such as "first", "second", etc., the descriptions of "first", "second", etc. are only for description purposes and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the fact that a person skilled in the art can realize it, and when the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the protection scope required by the present application.

[0063] In the prior art, the mechanical arm of the massage robot realizes the tracking of the key points of the human body mainly by calculating the motion of pixels in the image sequence. This tracking method has obvious limitations in actual application, such as dependence on open scene data sets, difficulty in direct application to the occlusion scene of the physiotherapy robot, loss of the target by the existing algorithm in the scene with mechanical arm occlusion, resulting in inability to accurately predict the key point position of the occluded part, and difficulty in reassociating the original trajectory after the occlusion is recovered.

[0064] Based on the above problems in the prior art, the present application provides a method for overcoming mechanical arm occlusion to realize human acupoint tracking, as shown in Figure 1 The method for overcoming mechanical arm occlusion to realize human acupoint tracking comprises:

[0065] S100, acquiring a real-time image of a target;

[0066] S200, processing a current frame image in the real-time image to obtain verification data, and processing images other than the current frame in the real-time image to obtain basic data;

[0067] S300, processing the basic data to obtain prediction data, and comparing the prediction data with the verification data;

[0068] S400, when the prediction data and the verification data do not correspond, adding the verification data to the basic data through a Kalman filter to update, and re-comparing the prediction data and the verification data; when the prediction data and the verification data correspond, calculating and outputting the prediction data of the next frame.

[0069] In the embodiment, the application provides a method for overcoming the mechanical arm shielding to track the human body acupoints. The method is applied to a massage robot with a mechanical arm. Specifically, the method collects the image of human body movement through a depth camera arranged on the mechanical arm, tracks the shielding part by processing the image, and determines the human body acupoints to be massaged through the autonomous judgment of the massage robot in the case of shielding, thereby avoiding the problem of key point tracking failure caused by the mechanical arm shielding in the massage process.

[0070] Specifically, in the embodiment, the real-time image of the target is first acquired. The real-time image is the human body movement data in the process of using the mechanical arm of the massage robot to massage the human body. After the real-time image of the target is acquired, the real-time image is processed, and the real-time image is divided into a current frame and a plurality of continuous frame images other than the current frame. After processing the current frame image, verification data is formed. After processing the other continuous frame images other than the current frame image, the basic data is formed. The basic data is used to generate the prediction data corresponding to the current frame. The verification data is used to compare with the verification data, so as to determine whether the prediction data formed under the current image resource is accurate.

[0071] More specifically, after the prediction data and the verification data are acquired, the prediction data and the verification data are compared. When the prediction data corresponds to the verification data, it indicates that the prediction data formed by the current algorithm is accurate. At this time, the verification data can be added to the basic data, and the prediction data of the next frame after the current frame is formed by prediction, which is used as the basis for the massage robot to track the human body acupoints.

[0072] When the prediction data does not correspond to the verification data, it indicates that the prediction data formed by the current algorithm is not accurate. Obviously, the above-mentioned prediction and verification process fails to work for this case. In order to realize accurate prediction, the Kalman filter is introduced to further process the above-mentioned data, that is, the verification data of the current frame is added to the basic data other than the current frame for updating. A prediction data for the next frame is regenerated through the Kalman filter. After the verification data of the next frame is acquired, it is corresponded with the prediction data of the next frame. Since the basic data has undergone an iteration process in this process, a new trend judgment is formed, so that the prediction data can be more accurately formed for matching. After several iterations, the prediction data corresponding to the corresponding frame can be formed. When the corresponding frame is formed, it indicates that the prediction formed by the algorithm at this time is accurate, and the prediction data of the next frame can be processed for the massage robot as the basis for tracking the human body acupoints.

[0073] Specifically, in one implementation of the present application, the above-mentioned processing of the current frame image in the real-time image to obtain verification data, and processing of images other than the current frame in the real-time image to obtain basic data further comprises:

[0074] S110, extracting surface features in a single frame image, setting the surface features as targets, the targets comprising a plurality of targets;

[0075] S120, setting detection boxes for the plurality of targets, labeling the detection boxes, and generating detection box data according to the detection boxes and the labeling information;

[0076] S130, obtaining corresponding key points in the detection boxes according to the detection boxes, and generating key point coordinates and occlusion probabilities according to the plurality of key points;

[0077] S140, defining the detection box data corresponding to the current frame image, the key point coordinates and the occlusion probabilities as verification data, and defining the collective of the detection box data corresponding to the current frame image and the collective of the key point coordinates as basic data.

[0078] In the present embodiment, how to accurately determine the region of the target is an important basis for subsequent judgment of the moving track of the target. The present embodiment determines the range of the target by extracting surface features in a single frame image. Specifically, the surface features include texture, edge, color and the like, as shown in FIG. 1. Since there is an obvious boundary difference between the occlusion object and the human body, the shape of the range of the occlusion object can be accurately recognized by determining the surface features. In the present embodiment, since there can be multiple occlusion objects, the target can contain a plurality of targets. The above-mentioned image corresponding process is performed simultaneously and separately for the targets. Figure 4

[0079] Further, to realize comparison of the images of the targets, the present application sets a detection box for the target in the process of processing the images. The detection box is a rectangular shape surrounding the target. After setting the detection box, the detection box is labeled to facilitate distinguishing the moving track of the target in the case of multiple targets. At the same time, detection box data is generated according to the coordinates of the detection box in the image, so as to realize the correspondence of the labeling information and the detection box data, and facilitate further processing.

[0080] After generating the detection box, the key points in the detection box are detected and labeled, as shown in FIG. 2. Figure 4 ​As shown, the selection of the key points of the target is the sharp corner part of the occluded object, so as to facilitate subsequent comparison. The key point coordinates and the occlusion probability are generated according to the key points, and are used for subsequent calculation of overcoming occlusion. In the embodiment, the detection box data corresponding to the current frame image, the key point coordinates and the occlusion probability are defined as the verification data, and the set of the detection box data corresponding to the current frame image and the set of the key point coordinates are defined as the basic data. It can be seen that there is a corresponding relationship between the detection box and the key point between the verification data and the basic data, so that the generation of the prediction data and the comparison between the prediction data and the verification data can be realized by calculation.

[0081] In one specific embodiment of the application, the detection box data is represented as D t = {d1, d2,..., di,..., dN} in operation. m The single detection box in the detection box data is represented as d i = [x i , y i , w i , h i ], wherein x i , y i , w i , h i respectively represent the horizontal coordinate, the vertical coordinate, the width and the height of the i-th detection box.

[0082] The key point coordinates are represented as P i = {p i1 , p i2 ,..., p iN} ; p i1 represents the position coordinates of the first key point corresponding to the i-th detection box, and N is the total number of key points.

[0083] The occlusion probability is represented as V i = {v i1 , v i2 ,..., v iN} ; v i1 represents the probability of visibility of the first key point corresponding to the i-th detection box, and N is the total number of key points.

[0084] On the other hand, in one specific embodiment of the application, the target detection box is set for the target, and the labeling of the target detection box further comprises:

[0085] S121, the same target in different frames of images is assigned the same ID identification, and different targets in different frames of images are assigned different ID identifications.

[0086] By assigning the same ID to the same target in different frames of images and assigning different IDs to different targets in different frames of images, on the one hand, the tracking of a specific target in different frames of images can be realized, so as to facilitate the tracking of the moving track of the target, and on the other hand, the effect of simultaneously tracking the moving track of the target can be realized in the case that there are multiple occluded targets, thereby guaranteeing the accuracy of the judgment.

[0087] Further, when the prediction data does not correspond to the verification data, the adding of the verification data to the basic data by the Kalman filter for updating further includes:

[0088] S410, when the verification data does not match the prediction data, a new ID is assigned to the target in the verification data that does not match the prediction data; and / or, when the target in the prediction data does not appear in the threshold number of consecutive frames of the basic data, the ID of the corresponding target is deleted.

[0089] In this embodiment, for the unmatched detection result, the system will initialize a new track for it; for the track that fails to match for consecutive multiple frames, the system will mark it as invalid and delete it in time. This design not only improves the accuracy of matching, but also effectively handles complex situations such as target occlusion, appearance and disappearance.

[0090] Further, the processing of the basic data to obtain the prediction data includes:

[0091] S210, processing the set of detection box data corresponding to the basic data and the set of key point coordinates;

[0092] S220, generating prediction box data according to the set of detection box data in the basic data;

[0093] S230, generating prediction key point coordinate data according to the set of key point coordinates in the basic data;

[0094] S240, the prediction data is the prediction box data and the prediction key point coordinate data.

[0095] In this embodiment, the acquisition of the prediction data is realized by processing the detection box data and the key point coordinates in several frames of images. Since the existence of the occluded target is not instantaneous, its track forming occlusion must leave corresponding images in several frames of images. Therefore, the moving track of the target is obtained by processing the set of detection box data and the set of key point coordinates, so as to predict the future track image, that is, the prediction box data is generated according to the set of detection box data in the basic data, and the prediction key point coordinate data is generated according to the set of key point coordinates in the basic data.

[0096] In the present application, when the prediction frame data and the key point coordinate data are obtained, they can be compared with the verification data formed by the current frame. Since the current frame image is a continuation of other frame images in real-time images except the current frame image, the current frame image must record the latest motion trajectory of the occluded target. By comparing the predicted data with the data of the current frame image, the system can automatically determine whether the predicted result is accurate.

[0097] That is, when the prediction data corresponds to the verification data, it means that the prediction process is accurate. At this time, the verification data can be added to the basic data, and the next frame of image prediction is performed based on the updated basic data, thereby providing a data basis for the next control of the mechanical arm and forming the effect of overcoming the occlusion of human acupoint tracking.

[0098] Meanwhile, the prediction data and the verification data do not correspond, which means that the prediction process is not accurate. In this case, a more accurate prediction method needs to be used, that is, the Kalman filter is used for further processing. When the prediction data does not correspond to the verification data, the verification data is added to the basic data for updating by the Kalman filter, and the prediction data and the verification data are compared again, including:

[0099] S420, setting a total cost matrix to calculate the matching degree of the prediction data and the verification data;

[0100] The total cost matrix includes:

[0101] An IOU cost matrix, which is used to represent the intersection over union of the detection frame data of the current frame image and the prediction frame data;

[0102] A key point position cost matrix, which is used to represent the position cost value between the key point coordinates of the current frame image and the prediction key point coordinate data;

[0103] A key point posture change cost matrix, which is used to represent the relative relationship between the key point coordinates of the current frame image and the prediction key point coordinate data, and the relative relationship is the ratio of angle and distance.

[0104] Specifically, in the process of processing the image, the present application calculates the detection frame data, the key point coordinates and the occlusion probability. The corresponding prediction frame and the position prediction of the key point are obtained by the Kalman filter according to the last frame trajectory, and the cost matrix is calculated based on the detection result of the current frame. The three cost matrices include the IOU cost matrix C IOU , the key point position cost matrix C keypoints , and the key point relative position change matrix C pose .

[0105] The calculation formula of the IOU cost matrix is: C IOU [i,j] represents the IOU cost value between the ith detection box and the jth prediction box in the cost matrix, and the greater the value, the lower the degree of overlap between the detection box and the trajectory prediction box.

[0106] The calculation formula of the key point position cost matrix is: C keypoints [i,j] represents the key point position cost value between the ith detection box and the jth prediction box in the cost matrix, P ik represents the coordinate position of the kth key point of the ith detection box, represents the predicted coordinate of the kth key point of the jth prediction box, and N represents the total number of key points, represents the Euclidean distance between the key point coordinates;

[0107] The calculation formula of the key point posture change cost matrix is:

[0108] M represents the number of all possible pairs, u and v represent a pair of key points in each box, represents the Euclidean distance between the two key points in the detection box, and the closer the ratio of the relative position is to 1, the closer the detection value and the prediction are.

[0109] Based on the above embodiment, in order to achieve the effect of further accurately comparing the prediction data and the verification data, in the present embodiment, the IOU cost matrix, the key point position cost matrix and the key point posture change cost matrix are respectively set weights to form the total cost matrix.

[0110] The formula of the total cost matrix is: total C IOU = αC keypoints + βC pose ; and the α, β, γ are used to control the ratio of different cost matrices.

[0111] According to the comprehensive cost matrix, the Hungarian matching algorithm is used for state management of the trajectory. The optimization goal is to minimize the total cost, and the best matching between the detection box and the prediction box obtained according to the trajectory is found to improve the prediction accuracy.

[0112] In the above embodiment, by introducing the occlusion probability into the Kalman filter, the prediction accuracy of the key points under the occlusion condition is effectively improved.

[0113] Based on the above embodiments, in another embodiment of the present application, when the prediction data and the verification data correspond through the above two comparison methods, the prediction data of the next frame is calculated and output; the prediction data of the next frame is the image of the next frame of the current frame image, and through repeated prediction of the target, the mechanical arm of the massage robot can overcome the occlusion and form tracking of the human body acupoints.

[0114] Specifically, in the above first case, that is, only through the comparison of the prediction data and the verification data to form the corresponding case, when the prediction data and the verification data correspond, the calculation and output of the prediction data of the next frame include:

[0115] S430, the verification data is added to the basis data, and the prediction data of the target in the next frame image is calculated according to the updated detection frame data set and the key point coordinate set of the basis data; the prediction data of the target in the next frame image also includes a state vector of the target;

[0116] The calculation formula of the state vector is X t = FX t-1 + u; wherein, X t-1 is the state vector of the last frame, including the position and speed of the key point, F is a state transition matrix, defining the motion model of the state, u represents the random noise in the process, describing the random change.

[0117] By outputting the prediction data containing the state vector, the motion state information of the target key point can be represented, and the pre-planning and control of the moving state of the mechanical arm through the robot can be realized.

[0118] In the above second case, that is, the verification data is added to the basis data through the Kalman filter for updating, and the prediction data and the verification data are compared again, the above adding the verification data to the basis data through the Kalman filter for updating further includes:

[0119] The state vector is updated, and the calculation formula of the state vector when updating is:

[0120] X t = X t + K(Z t -HX t ); wherein, Z t is the observation value of the current frame, that is, the position of the detected detection frame and key point, H is an observation matrix, mapping the state space to the observation space, K is a Kalman gain, controlling the fusion weight of the prediction value and the observation value;

[0121] The calculation formula of the Kalman gain is K = Pt|t-1 H T (HP t|t-1 H T +R) -1 ; wherein, P t|t-1 is the error covariance matrix of the prediction stage, H is the observation matrix, and R is the observation noise covariance matrix;

[0122] When V < 0.8, K = K*V, and R a = R x (1+1-V);

[0123] When V > 0.8, K = K.

[0124] In the embodiment, to realize the control of the predicted value and the observed value to realize more accurate state vector output, the fusion weight of the predicted value and the observed value is improved by the Kalman gain, and the reliability of the observed value is dynamically adjusted in two cases. Specifically, when V > 0.8, it is indicated that the above shielding condition is slightly shielded or not shielded, and the state can be updated using the observed value, that is, K = K; when V < 0.8, it is indicated that the above shielding condition is serious, and in this case, the influence of the observed value on the state is reduced.

[0125] In addition, in the case of the above serious shielding condition, since the observed value is not reliable, the Kalman filter relies more on the prediction result of the motion model, and therefore the observation noise covariance R is adjusted, that is, R a = R x (1+1-V), and the lower V is, the more observation noise there is, and the influence of the observed value on the state update can be reduced.

[0126] Based on the above embodiment, in actual application, the real-time image of the user is acquired, such as Figure 3 and Figure 6 , Figure 3 is an image in which the part of the back of the human body is shielded by the mechanical arm, and the specific content is that the Shenshu and Yaoshu acupoints are shielded by the mechanical arm, Figure 6 is an image after processing the label according to the shielding condition in Figure 3 . In this state, the detection region of the corresponding back of the user is acquired by the depth camera, and the back contains some acupoints, such as Dazhui, Zhisang, Shenshu, and Yaoshu, and two points for marking the shoulder width in the detection region. These acupoints are used to continuously calibrate the acupoints that the user wants to massage;

[0127] When no shielding occurs, for example Figure 4 is shown, Figure 4 based on the key point detection of the hand, in Figure 4 , the detection of the hand key point is mainly through the hand to mark the acupoint that wants to be massaged, and the image after processing by the massage robot is as shown in Figure 7As shown, the hand is labeled with ID and the detection box and the key points of the hand are labeled, and in the case shown in Figure 7 The hand does not have a serious impact on the working part of the massage robot arm, so the trajectory prediction of the hand can be realized by directly matching the basic data and the predicted data.

[0128] In the case of serious occlusion, as shown in Figure 5 and Figure 8 The mechanical arm forms an occlusion to the user's hand, and has a serious impact on the working part of the massage robot arm, so the Kalman filter is further processed to accurately predict the trajectory of the hand to control the mechanical arm to determine the acupoint part of the user under the occlusion.

[0129] In the above embodiment, the network model of the method for overcoming the occlusion of the mechanical arm to track the acupoint of the human body provided by the present application is trained to be preset, and a Deptrum Aurora 930 series RGBD depth camera (resolution 640*400, frame rate 15fps) is used for image acquisition. The camera is installed on the mechanical arm and used to collect human motion data during the massage process of the mechanical arm. Specifically, 20 different subjects are collected under the operation of the mechanical arm, and each video is about 6 seconds long. Finally, 4000 RGB images are obtained, including 2000 human motion and 2000 hand motion scenes, and ensuring that 1600 of the images contain partial or complete occlusion to enhance the adaptability of the model to the occlusion scene.

[0130] For models that have not been trained on the occlusion scene dataset, the back or part of the area cannot be detected, and the key points cannot be predicted. After training, the model learns the features in the occlusion scene and has strong robustness, and can accurately detect the back or hand area. Before using the improved Kalman filter, the model cannot balance the weight well, and too many occluded key points rely on inaccurate detection models, resulting in poor results. The improved Kalman filter can dynamically adjust the weight when occlusion occurs, and accurately predict the occluded key points according to the motion state.

[0131] In summary, as shown in Figure 2As shown, the above scheme of the present application has an adaptive Kalman filtering mechanism, which dynamically adjusts the Kalman gain through the key point occlusion probability, and adaptively updates the observation noise covariance matrix. This mechanism can intelligently balance the weight of the predicted value and the observed value in the occlusion scene, realize the optimal compromise of real-time and accuracy, and effectively solve the problem of performance degradation of the traditional method in the occlusion scene. At the system implementation level, the present application designs a complete data acquisition and processing process, realizes the seamless integration of detection, estimation and tracking. This system-level innovation not only provides a complete solution for practical applications, but also ensures the reliability and practicality of the algorithm in practical scenarios, providing a new technical approach for human key point tracking in the robot scene.

[0132] In addition, in the technical scheme adopted in the present application, the feature fusion method can be changed, such as using a nonlinear combination method instead of linear weighting; different cost matrix calculation methods can be used, such as using other distance metrics or similarity calculation methods; the trajectory management strategy can be modified, such as using different trajectory initialization and deletion conditions; other types of motion prediction models can be used, such as using a sequence prediction model based on deep learning. In terms of matching strategy, algorithms such as greedy matching and graph matching can be used instead of the Hungarian algorithm, or a multi-level matching strategy can be used, which first performs coarse matching and then performs fine matching, and a trajectory quality evaluation mechanism can be introduced to prioritize matching high-quality trajectories, etc. It should be noted that for those skilled in the art, the above-mentioned improvements or transformations can be made, and all these improvements and transformations should be within the scope of protection of the appended claims of the present application.

[0133] In addition, the practical application scenarios of the present application are not limited to the human key point tracking of the physiotherapy robot as described above, but also in other scenarios involving robot occlusion, such as industrial robot cooperation, rehabilitation training, motion posture analysis, etc. By adjusting the definition and feature extraction method of the key points, it can also be flexibly adapted to different tracking needs; in the subsequent optimization process, an abnormality detection and processing mechanism can be added to evaluate the credibility of the tracking results and make timely corrections if necessary. At the same time, a confidence evaluation system for tracking results can be established to provide reliable reference for subsequent control decisions; it can also be combined with deep reinforcement learning to realize autonomous optimization of the tracking strategy; multi-modal sensor information fusion can be introduced to improve the robustness of the system under extreme conditions; and distributed processing schemes based on edge computing can be researched to realize a more efficient real-time tracking system.

[0134] It should be understood that the application of the present application is not limited to the above examples, and those skilled in the art can make improvements or transformations according to the above description, and all these improvements and transformations should be within the scope of protection of the appended claims of the present application.

Claims

1. A method for tracking acupoints on the human body by overcoming obstruction by a robotic arm, characterized in that, The method for overcoming the obstruction of the robotic arm to achieve human acupoint tracking includes: Acquire real-time images of the target; The process involves processing the current frame image in the real-time image to obtain verification data; and processing images other than the current frame in the real-time image to obtain basic data. This includes: extracting surface features from a single frame image and setting the surface features as targets, where the targets include several; setting detection boxes for the targets, labeling the detection boxes, and generating detection box data based on the detection boxes and labeling information; obtaining several key points corresponding to the detection boxes, and generating key point coordinates and occlusion probabilities based on the key points; defining the detection box data corresponding to the current frame image, the key point coordinates, and the occlusion probability as verification data; and defining the set of detection box data excluding the current frame image and the set of key point coordinates as basic data. The basic data is processed to obtain predicted data, and the predicted data is compared with the verification data. When the predicted data does not correspond to the verification data, the verification data is added to the basic data for updating using a Kalman filter, and the predicted data and verification data are compared again; when the predicted data corresponds to the verification data, the predicted data for the next frame is calculated and output. The step of setting detection boxes for the plurality of targets and annotating the detection boxes further includes: assigning the same ID to the same target in images of different frames, and assigning different ID to different targets in images of different frames; When the predicted data does not correspond to the verified data, updating the base data by adding the verified data to the base data using a Kalman filter also includes: When the verification data does not match the prediction data, a new ID is assigned to the target in the verification data that does not match the prediction data; and, or, when the target in the prediction data does not appear in a consecutive number of frames of the threshold number of the basic data, the ID of the corresponding target is deleted. When the predicted data does not correspond to the verification data, the verification data is added to the base data for updating using a Kalman filter, and the predicted data and verification data are re-compared, including: Set the total cost matrix and calculate the matching degree between the predicted data and the verification data; The total cost matrix includes: The IOU cost matrix is ​​used to represent the intersection-union ratio of the detection box data and the prediction box data in the current frame image; The key point location cost matrix is ​​used to represent the location cost between the key point coordinates of the current frame image and the predicted key point coordinate data. A keypoint pose change cost matrix, which represents the relative relationship between the keypoint coordinates of the current frame image and the predicted keypoint coordinates, wherein the relative relationship is the ratio of angle to distance; when the predicted data corresponds to the verification data, the predicted data for the next frame is calculated and output, including: The verification data is added to the base data, and the predicted data of the target in the next frame image is calculated based on the set of detection box data of the updated base data and the set of key point coordinates; the predicted data of the target in the next frame image also includes the target's state vector; The formula for calculating the state vector is: ;in, It is the state vector of the previous frame, including the position and velocity of key points. It is the state transition matrix, which defines the motion model of the state. It represents random noise in the process and describes random changes.

2. The method for overcoming the obstruction of the robotic arm to achieve human acupoint tracking according to claim 1, characterized in that, The detection frame data is represented as follows: A single detection box in the detection box data is represented as: ,in Let x, y, width, and height represent the x-coordinate, y-coordinate, width, and height of the i-th detection box, respectively. The coordinates of the key points are represented as follows: ; This represents the position coordinates of the first keypoint corresponding to the i-th detection box. The total number of key points; The occlusion probability is expressed as: ; Let N represent the probability of visibility of the first keypoint corresponding to the i-th detection box, where N is the total number of keypoints.

3. The method for overcoming the obstruction of the robotic arm to achieve human acupoint tracking according to claim 1, characterized in that, The process of processing the basic data to obtain the prediction data includes: The set of detection box data corresponding to the basic data and the set of key point coordinates are processed. Predictive box data is generated based on the collection of detection box data in the basic data; Predicted keypoint coordinate data is generated based on the collection of keypoint coordinates in the basic data; The prediction data consists of the prediction frame data and the prediction key point coordinate data.

4. The method for overcoming the obstruction of the robotic arm to achieve human acupoint tracking according to claim 1, characterized in that, The formula for calculating the IOU cost matrix is ​​as follows: ;in This represents the IOU cost between the i-th detection box and the j-th prediction box in the cost matrix. The larger the value, the lower the overlap between the detection box and the trajectory prediction box. The formula for calculating the cost matrix of the key point location is as follows: ;in This represents the cost value of the location of the keypoint between the i-th detection box and the j-th prediction box in the cost matrix. This represents the coordinate position of the k-th keypoint of the i-th detection box. This represents the predicted coordinates of the k-th keypoint in the j-th prediction bounding box, where N represents the total number of keypoints. This represents the Euclidean distance between the coordinates of key points; The formula for calculating the cost matrix of the pose change at key points is as follows: Where M represents the number of all possible pair combinations, and This represents a pair of key points in each box. This represents the Euclidean distance between two key points within the detection box. The closer the ratio of their relative positions is to 1, the closer the detected value is to the predicted value.

5. The method for overcoming the obstruction of the robotic arm to achieve human acupoint tracking according to claim 4, characterized in that, The step of setting the total cost matrix and calculating the matching degree between the predicted data and the validation data further includes: Weights are assigned to the IOU cost matrix, the key point position cost matrix, and the key point pose change cost matrix to form the total cost matrix; The total cost matrix is: The Used to control the ratio of different cost matrices.

6. The method for overcoming the obstruction of the robotic arm to achieve human acupoint tracking according to claim 1, characterized in that, The step of updating the base data by adding the verification data using a Kalman filter also includes: The state vector is updated using the following formula: ;in, It represents the observations of the current frame, specifically the locations of the detected bounding boxes and keypoints. It is the observation matrix, which maps the state space to the observation space. It is the Kalman gain, which controls the fusion weights of the predicted and observed values; The formula for calculating the Kalman gain is as follows: ;in, It is the error covariance matrix during the prediction phase. It is the observation matrix. It is the observation noise covariance matrix. It is the Kalman gain, which controls the fusion weights of the predicted and observed values; When V < 0.8, K = K * V, and ; When V>0.8, K=K.

Images