Data processing method and device and computing equipment

Abstract

The embodiment of the invention provides a data processing method and device and computing equipment. When the method is executed, the computing device obtains acupuncture video data of an acupuncture part, forearm myoelectricity data of an acupuncture object and finger piezoelectric data of the acupuncture object in the acupuncture operation process, and the acupuncture video data of the acupuncture part, the forearm myoelectricity data and the finger piezoelectric data are acquired according to the acupuncture video data, the forearm myoelectricity data and the finger piezoelectric data. And identifying the acupuncture manipulation of the acupuncture object in the acupuncture operation process. The acupuncture video data, the forearm myoelectricity data and the finger piezoelectric data can comprehensively represent the acupuncture manipulation, and the acupuncture video data can restore the actual acupuncture scene of the acupuncture object, so that the acupuncture manipulation is recognized based on the comprehensive data of the acupuncture video data, the forearm myoelectricity data and the finger piezoelectric data, the recognition precision is higher compared with clinical observation, and the recognition accuracy is higher. And a learner can learn the acupuncture manipulation more conveniently.

Description

Technical Field

[0001] This application relates to the field of video technology, and in particular to a data processing method, apparatus, and computing device. Background Technology

[0002] Acupuncture is an important component of Traditional Chinese Medicine (TCM) theory. Acupuncture involves inserting needles into the acupuncture points (also known as acupoints) and applying techniques such as tonifying (rotating), reducing (rotating), and lifting / thrusting (lifting and thrusting). Currently, learners often study acupuncture techniques by observing the acupuncturist's (the practitioner's) clinical procedures. However, the rapid speed at which the practitioner manipulates the needles makes it difficult for learners to accurately identify the techniques applied to the acupuncture points through clinical observation. Therefore, improving learners' accuracy in identifying acupuncture techniques has become a pressing technical problem. Summary of the Invention

[0003] This application provides a data processing method, apparatus, and computing device that can improve learners' ability to identify acupuncture techniques applied to the acupuncture site by the acupuncture subject.

[0004] In a first aspect, embodiments of this application provide a data processing method applied to a computing device, the method comprising:

[0005] During the acupuncture procedure, acupuncture video data of the acupuncture site, electromyographic data of the forearm of the acupunctured subject, and piezoelectric data of the fingers of the acupunctured subject are acquired.

[0006] The forearm electromyography data is used to describe the muscle activity state of the forearm of the acupuncture subject, the finger piezoelectric data indicates the pressure applied by the acupuncture subject to the acupuncture site, and the acupuncture video data is used to describe the operation process of the acupuncture subject performing acupuncture on the acupuncture site.

[0007] Based on the forearm electromyography data, the finger piezoelectric data, and the acupuncture video data, the acupuncture techniques used by the acupuncture subject during the acupuncture operation are identified.

[0008] Optionally, the computing device includes a first camera and a second camera, wherein the first camera and the second camera have different shooting angles;

[0009] Acquiring acupuncture video data of the acupuncture site includes:

[0010] The acupuncture video data is acquired using the first camera and the second camera.

[0011] Optionally, the first camera has a variable shooting angle, and the second camera has a fixed shooting angle. Before acquiring the acupuncture video data using the first camera and the second camera, the method further includes:

[0012] Obtain the marked area of ​​the acupuncture site;

[0013] The step of acquiring the acupuncture video data using the first camera and the second camera includes:

[0014] The acupuncture video data is acquired simultaneously using the first camera and the second camera; wherein, during the acquisition process, the marked area is always located within the target field of view of the first camera.

[0015] Optionally, the method further includes:

[0016] During the preparation stage of acupuncture, a target viewpoint is acquired; wherein, the target viewpoint includes multiple key points of the hand of the acupuncture object, and the initial posture of the target viewpoint is the posture corresponding to the marked area being located in the target field of view area of ​​the first camera before the preparation stage of acupuncture.

[0017] Based on the target viewpoint, the homography matrix of the target viewpoint is obtained; the homography matrix is ​​related to the pixel coordinates of the hand key point in the first camera and the pixel coordinates in the second camera;

[0018] Based on the homography matrix, the pose change of the target viewpoint relative to the initial pose during the preparation stage of needle application is obtained;

[0019] Based on the change in posture, the shooting angle of the first camera is adjusted so that the adjusted marked area is located in the target field of view of the first camera.

[0020] Optionally, the method further includes:

[0021] If there are any suspicious obstructions at the beginning of the acupuncture treatment, a safety margin should be obtained.

[0022] Wherein, the safety margin is positively correlated with the target angle and the target distance; the target angle is the angle between the target point of the suspected obstruction and the viewing cone of the first camera, and the target distance is the distance between the target point and the viewing cone of the first camera; the target point is the spatial location point of the suspected obstruction that is closest to the viewing cone of the first camera.

[0023] If the safety margin is less than or equal to the safety threshold, adjust the first camera so that the adjusted safety margin is greater than or equal to the safety threshold.

[0024] Optionally, identifying the acupuncture techniques used by the acupuncture recipient during the acupuncture procedure based on the forearm electromyography data, the finger piezoelectric data, and the acupuncture video data includes:

[0025] The acupuncture video data is segmented to obtain m sub-video data. Each sub-video data corresponds to one acupuncture technique, and any two sub-video data correspond to different acupuncture techniques. Here, m is a positive integer.

[0026] Based on the acquisition time window of each of the m sub-video data, the forearm electromyography data and the finger piezoelectric data are segmented to obtain m sub-electromyography data and m sub-piezoelectric data; wherein, the acquisition timestamp of 1 sub-video data corresponds to 1 sub-electromyography data and 1 sub-piezoelectric data.

[0027] The system integrates sub-video data, sub-piezoelectric data, and sub-electromyographic data corresponding to the same acquisition time window; based on the fused information, it identifies the acupuncture techniques used by the acupuncture subject during the acupuncture operation.

[0028] Optionally, segmenting the acupuncture video data to obtain m sub-video data includes:

[0029] The acupuncture video data is segmented using a preset segmentation algorithm to obtain m sub-video data. The preset segmentation algorithm is a segmentation algorithm trained based on pre-constructed training samples. The training samples include the start frame and end frame corresponding to the acupuncture technique, as well as the label corresponding to the acupuncture technique.

[0030] Optionally, the computing device further includes an electromyography (EMG) sensor and a piezoelectric film finger sleeve, wherein the EMG sensor is deployed on the forearm used by the acupuncture subject during acupuncture, and the piezoelectric film finger sleeve is deployed on the finger used by the acupuncture subject during acupuncture; the method further includes:

[0031] The electromyographic data is acquired through the electromyographic sensor; the piezoelectric data is acquired through the piezoelectric film finger sleeve.

[0032] Secondly, embodiments of this application provide a data processing apparatus applied to a computing device, the apparatus comprising:

[0033] The acquisition unit is used to acquire forearm electromyography data, finger piezoelectric data, and acupuncture video data of the acupuncture site during the acupuncture operation.

[0034] The forearm electromyography data is used to describe the muscle activity state of the forearm of the acupuncture subject, the finger piezoelectric data indicates the pressure applied by the acupuncture subject to the acupuncture site, and the acupuncture video data is used to describe the operation process of the acupuncture subject performing acupuncture on the acupuncture site.

[0035] The identification unit is used to identify the acupuncture technique used by the acupuncture subject during the acupuncture operation based on the forearm electromyography data, the finger piezoelectric data, and the acupuncture video data.

[0036] Thirdly, embodiments of this application provide a computing device, which includes a processor, a video acquisition device, an electromyography sensor, and a piezoelectric thin film finger sleeve, all of which are connected to the processor.

[0037] The video acquisition device is used to acquire acupuncture video data of the acupuncture site during the acupuncture operation and send the acupuncture video data to the processor;

[0038] The electromyography sensor is used to acquire electromyography data of the forearm of the acupuncture subject during the acupuncture operation and send the forearm electromyography data to the processor;

[0039] The piezoelectric film finger sleeve is used to acquire the piezoelectric data of the acupuncture object's fingers during the acupuncture operation and send the piezoelectric data of the fingers to the processor;

[0040] The forearm electromyography data is used to describe the muscle activity state of the forearm of the acupuncture subject, the finger piezoelectric data indicates the pressure applied by the acupuncture subject to the acupuncture site, and the acupuncture video data is used to describe the operation process of the acupuncture subject performing acupuncture on the acupuncture site.

[0041] The processor is used to: identify the acupuncture techniques used by the acupuncture subject during the acupuncture operation based on the forearm electromyography data, the finger piezoelectric data, and the acupuncture video data.

[0042] This application provides a data processing method, apparatus, and computing device. When executing the method, the computing device acquires acupuncture video data of the acupuncture site, forearm electromyography (EMG) data of the acupuncturist, and finger piezoelectric data of the acupuncturist during an acupuncture operation. Based on the acupuncture video data, forearm EMG data, and finger piezoelectric data, the acupuncture technique used by the acupuncturist during the acupuncture operation is identified. Since the acupuncture video data, forearm EMG data, and finger piezoelectric data can comprehensively characterize the acupuncture technique, and the acupuncture video data can recreate the actual acupuncture scene of the acupuncturist, the identification of acupuncture techniques based on the comprehensive data of the acupuncture video data, forearm EMG data, and finger piezoelectric data is more accurate than clinical observation and is more conducive to learners learning acupuncture techniques. Attached Figure Description

[0043] Figure 1A This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0044] Figure 1B This is a schematic diagram of the structure of a piezoelectric thin film finger sleeve provided in an embodiment of this application;

[0045] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0046] Figure 3 A schematic diagram of the structure of key points of the hand provided in an embodiment of this application;

[0047] Figure 4 A schematic diagram illustrating a target perspective provided in an embodiment of this application;

[0048] Figure 5 This is a schematic diagram of a safety margin provided in the embodiments of this application;

[0049] Figure 6 This is a schematic diagram of a multimodal data fusion structure provided in an embodiment of this application;

[0050] Figure 7 A flowchart of another data processing method provided in this application embodiment;

[0051] Figure 8 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Detailed Implementation

[0052] The data processing method provided in this application is applied to a computing device. During acupuncture, the computing device acquires acupuncture video data of the acupuncture site, electromyographic data of the forearm of the acupuncturist, and piezoelectric data of the fingers of the acupuncturist. Utilizing the comprehensive acupuncture video data, forearm electromyographic data, and finger piezoelectric data characterizing the acupuncture techniques, the device identifies the acupuncture techniques used by the acupuncturist during the acupuncture operation, thereby improving the accuracy of acupuncture technique recognition for learners when learning acupuncture techniques.

[0053] First, let me introduce the technical terms used in this application.

[0054] Acupuncture Techniques: Acupuncture techniques include the lifting and thrusting method and the twisting method. The lifting and thrusting method involves inserting the needle vertically into the acupoint, and then the patient uses their fingers or wrist to lift and thrust the needle up and down within the acupoint. The twisting method involves the patient's thumb pointing forward and index finger pointing backward, using wrist or finger strength to twist the needle back and forth evenly from side to side.

[0055] The lifting and thrusting technique includes the lifting and thrusting supplementation technique and the lifting and thrusting purging technique. The lifting and thrusting supplementation technique involves starting shallow and then deepening, with light insertion and heavy lifting, small amplitude, slow frequency, and small stimulation. The lifting and thrusting purging technique involves starting deep and then shallow, with heavy insertion and light lifting, large amplitude, fast frequency, and large stimulation.

[0056] The twisting method includes the tonifying twisting method and the purging twisting method. The tonifying twisting method involves a small twisting angle, a slow frequency, and light force; the purging twisting method involves a large twisting angle, a fast frequency, and heavy force.

[0057] Acupuncture video data is used to describe the acupuncture procedure performed on the acupuncture site (the acupoint to be needled). In this embodiment, the acupuncture video data includes the acupuncture site, the tip of the needle (the end inserted into the acupuncture site), and the acupuncture process. Different acupuncture techniques will present different operational movements and needle insertion methods in the acupuncture video data, and the external manifestations of acupuncture techniques can be directly obtained using the acupuncture video data.

[0058] For example, the lifting and inserting method in acupuncture video data is as follows: the acupuncturist holds the needle handle with their right thumb, index finger, and middle finger, while the ring finger and little finger are naturally bent. After vertically inserting the needle into the acupoint, the acupuncturist uses their fingers or wrist to lift and insert the needle up and down within the acupoint. Lifting refers to lifting the needle away from the acupuncture site, while inserting it towards the acupuncture site is called inserting. The twisting method in acupuncture video data is as follows: the needle is inserted into the acupuncture site, and the acupuncturist holds the needle handle with their right thumb, index finger, and middle finger, while the ring finger and little finger are naturally bent; the thumb points forward and the index finger points backward, using wrist or finger strength to twist the needle back and forth evenly; the twisting is even, 60-100 times per minute.

[0059] Forearm electromyography (EMG) data refers to the electrical signals generated by forearm muscle contractions during acupuncture. Forearm EMG data describes the muscle activity state of the forearm in the patient being treated. Different acupuncture techniques affect forearm muscle activity differently, resulting in different forearm EMG data. For example, the lifting and thrusting technique and the twisting technique differ in the intensity, frequency, and pattern of forearm muscle contraction during the procedure.

[0060] Finger piezoelectric data refers to the pressure signal data applied by the acupuncturist to the acupuncture site. Different acupuncture techniques result in different pressures applied to the acupuncture site. For example, in the lifting and thrusting technique, the acupuncturist's index finger moves up and down with the needle, applying a regular up-and-down pressure. In this technique, the finger pressure exhibits periodic changes, alternating between increasing and decreasing during lifting and thrusting, and the finger piezoelectric data also reflects these periodic changes. In the twisting technique, the acupuncturist twists the needle with their fingers, applying continuous and regular torsional pressure, and the finger piezoelectric data will show different characteristics compared to the lifting and thrusting technique.

[0061] The technical terms involved in the embodiments of this application have been explained above. The application scenarios of the data processing method provided in the embodiments of this application are described below.

[0062] For example, Appendix Figure 1A This is a schematic diagram of a computing device provided in an embodiment of this application. The computing device 100 includes a processor 101, a video acquisition device 102, an electromyography (EMG) sensor 103, and a piezoelectric film finger sleeve 104. The video acquisition device 102, the EMG sensor 103, and the piezoelectric film finger sleeve 104 are all connected to the processor 101.

[0063] In this embodiment, the video acquisition device 102 is used to acquire acupuncture video data of the acupuncture site during the acupuncture operation and send the acupuncture video data to the processor 104. The video acquisition device 102 is used to acquire acupuncture video data. Considering that a single video acquisition device 102 may have issues such as occlusion or coverage, this embodiment considers using n video acquisition devices to acquire acupuncture video data. Here, n is an integer greater than or equal to 2.

[0064] In one example, considering that having too many video acquisition devices would lead to inconvenience in data acquisition, this embodiment of the application can use two video acquisition devices to acquire acupuncture video data. Specifically, the two video acquisition devices are a first camera and a second camera, with different shooting angles.

[0065] Furthermore, the shooting angle of the first camera is variable, while the shooting angle of the second camera is fixed. For example, the first camera is connected to a robotic arm (the robotic arm and the first camera form a hand-eye system), and the angle of the first camera is adjusted by the robotic arm, so that the shooting angle of the first camera is variable.

[0066] It should be noted that the robotic arm provided in this application embodiment can be a seven-degree-of-freedom robotic arm or a robotic arm with other degrees of freedom; this application embodiment does not specifically limit it.

[0067] In this embodiment, the electromyography (EMG) sensor 103 is used to acquire the forearm EMG data of the acupuncture subject during the acupuncture operation and send the forearm EMG data to the processor 101.

[0068] In one implementation, the electromyography (EMG) sensor 103 is specifically a wireless EMG sensor. The EMG sensor 103 is deployed on the forearm of the acupuncture recipient during acupuncture.

[0069] The piezoelectric film finger sleeve 104 is used to acquire the piezoelectric data of the acupuncture subject's fingers during the acupuncture operation and send the finger piezoelectric data to the processor 101.

[0070] In this embodiment, the piezoelectric film finger sleeve 104 can be a thin film finger sleeve composed of a polyvinylidene fluoride (PVDF) array, deployed on the finger used by the acupuncture recipient during acupuncture, for collecting piezoelectric data of the finger. Exemplarily, see attached... Figure 1B This is a schematic diagram of the structure of a piezoelectric thin film finger sleeve provided in an embodiment of this application. The piezoelectric thin film finger sleeve 104 includes three layers: the first layer is a polytetrafluoroethylene (PTFE) layer, the second layer is a filler layer, and the third layer is a PVDF array.

[0071] It should be noted that in this embodiment of the application, the processor 101 can synchronously control the piezoelectric film finger sleeve 104, the electromyography sensor 103 and the video sampling device 102 to perform sampling operations, thereby ensuring that the time of the acquired acupuncture video data, forearm electromyography data and finger piezoelectric data is synchronized.

[0072] The processor 101 is used to identify the acupuncture techniques used by the acupuncture subject during the acupuncture operation based on forearm electromyography data, finger piezoelectric data and acupuncture video data.

[0073] It should be noted that the above-described computing device is merely illustrative, and its structure can be expanded and adjusted as needed in actual use. Those skilled in the art should understand that such expansions and adjustments should not be considered as limitations on the scope of protection of this application, but rather as equivalent implementations. Regardless of the adjustment, the core purpose is to achieve accurate identification of acupuncture techniques, thereby better assisting acupuncture teaching and research, and all such adjustments should be included within the scope of protection of this application.

[0074] The preceding text describes a computing device for implementing a data processing method. Based on the aforementioned computing device, the following describes the data processing method provided in the embodiments of this application.

[0075] Appendix Figure 2 A flowchart of a data processing method provided in this application embodiment, the method including the following:

[0076] S210. During the acupuncture operation, acquire acupuncture video data of the acupuncture site, electromyographic data of the forearm of the acupuncture subject, and piezoelectric data of the finger of the acupuncture subject.

[0077] Among them, the video acquisition equipment can collect acupuncture video data.

[0078] For example, embodiments of this application can acquire acupuncture video data using two cameras, a first camera and a second camera. Both the first and second cameras are depth cameras. The first camera and the robotic arm form a hand-eye system, with a variable shooting angle. The second camera has a fixed shooting angle.

[0079] To ensure that the acquired acupuncture video data includes the acupuncture site, the tip of the needle (the end inserted into the acupuncture site), and the acupuncture process, the computing device can first acquire a marked area of ​​the acupuncture site. For example, the first camera adds a red ring mark to the acupuncture point to be needled; the area marked by this red ring is the marked area. Alternatively, the first camera can also thicken the acupuncture point to be needled; this thickened area is the marked area.

[0080] To avoid ineffective acquisition of acupuncture video data, the marked area should be kept within the target field of view of the first camera during the acquisition process using both the first and second cameras. For example, the target field of view is the center of the first camera's field of view.

[0081] In one example, the computing device can adjust the hand-eye system based on the visual servoing of the image so that the marked area is always within the target field of view of the first camera.

[0082] In this embodiment of the application, before acquisition, the hand-eye system is first adjusted so that the marked area is located in the target field of view of the first camera. At this time, the posture of the marked area is also called the initial posture of the target view.

[0083] Among them, the target geometric parameter selected for image-based visual servoing is a circle, and its projection on the plane is an ellipse. The equation describing the ellipse is designed as shown in formula (1):

[0084] u 2 +E1ν 2 -2E2uν+2E3u+2E4ν+E5=0 (1)

[0085] Where Ei Here are the ellipse parameters, u and v are the pixel points corresponding to points on the ellipse. To obtain the camera motion, the ellipse equation is combined with the features of the target point to obtain the camera motion speed as shown in formula (2), where the left side of the equal sign in formula (2) is the camera motion speed.

[0086]

[0087] Where u1 and v1 are the pixel positions of the target point in the ellipse, J e and J p Let be the Jacobian matrices of the ellipse and the target point, respectively. The motion speed of the target camera is designed to be proportional to the difference between the current feature parameters and the desired feature parameters, as shown in formula (3):

[0088]

[0089] Wherein, the current feature parameters are The expected feature parameters are (E1, E2, ..., E5, u1, v1). T λ is a direct proportional value.

[0090] Then the first and second cameras began collecting data.

[0091] The acquisition process includes a preparation stage and an application stage. In the preparation stage, the processor first acquires the target viewpoint and, based on the target viewpoint, acquires the homography matrix of the target viewpoint. Based on the homography matrix, it acquires the attitude change of the target viewpoint in the preparation stage relative to the initial attitude. Based on this attitude change, it adjusts the hand-eye system so that the adjusted and marked area is located in the target field of view of the first camera.

[0092] In one example, the processor can detect key hand points using a mediapipe-based hand detection scheme. For example, see attached... Figure 3 This application provides a schematic diagram of the structure of key hand points, including key hand points 1 to 20. For ease of implementation, the processor identifies the key hand points. For example, numerical identifiers can be used; see [link to relevant documentation]. Figure 3 As shown. It can also be represented in other ways, such as color combined with numerical identifiers, such as red 1, blue 1, etc., but this application does not specifically limit the embodiments.

[0093] Next, the processor selects the target viewpoint, that is, chooses at least four points from the acquired hand keypoints as the target viewpoint. For example, the target viewpoint selection includes... Figure 3 The key hand points 2, 3, 4, 6, 7, and 8 shown are used as the target viewpoint, i.e., attached. Figure 4This is a schematic diagram of a target viewpoint provided in an embodiment of this application. The circled points are the key hand points corresponding to the target viewpoint.

[0094] Since the needle-holding posture will be maintained throughout the acupuncture procedure, it is assumed that the aforementioned key points are located on the same plane. The fixed-viewpoint second camera and the variable-viewpoint hand-eye system will synchronously detect the key points of the hand and calculate the corresponding homography matrix based on these key points. The homography matrix contains the posture changes of the fixed-viewpoint camera and the hand-eye system. The method for calculating the homography matrix H is shown in formula (4):

[0095]

[0096] Where h represents its row and column values ​​in the homography matrix H, and h33 is defined as 1. The homography matrix can be obtained using at least four points. In this embodiment, six hand key points can be used to calculate the homography matrix H using singular value decomposition. That is, the singularity decomposition of the coefficient matrix A is performed first. The form of A is a matrix composed of row vectors Ai. The value of Ai is shown in formula (5).

[0097]

[0098] xi, yi, and their superscript asterisks represent the normalized coordinates of the corresponding feature points.

[0099] Singular value decomposition of A yields the result shown in formula (6):

[0100] A = UEV T (6)

[0101] The homography matrix H is composed of the values ​​of the last column of V.

[0102] In summary, in the embodiments of this application, the processor can obtain the homography matrix based on the pixel coordinates of the video frames captured by the first camera and the second camera, and obtain the attitude change of the target viewpoint relative to the initial attitude during the preparation stage of needle application through the decomposition of the homography matrix.

[0103] Based on this attitude change, the processor can adjust the shooting angle of the first camera so that the adjusted marked area is always within the target field of view of the first camera.

[0104] It should be noted that the embodiments of this application define the target viewpoint in the form of an image, which can reduce complex calculations and improve adjustment efficiency.

[0105] The adjusted first and second cameras can simultaneously capture acupuncture video data.

[0106] In another example, considering the potential for obstruction during acupuncture, the processor first determines whether there is a suspicious obstruction. In this embodiment, the distance between the suspicious obstruction and the camera is less than the distance between the acupuncture site and the camera.

[0107] In one implementation, the processor can determine whether an occlusion exists based on changes in image frames (depth maps) acquired by the target camera. If the depth information of an object is less than the depth information of the acupuncture site, then the object is considered a suspected occlusion.

[0108] It should be noted that in the embodiments of this application, the target camera can be either a first camera or a second camera, and the embodiments of this application are not specifically limited to that.

[0109] If a suspicious obstruction is found for the target camera, a safety margin should be established.

[0110] The safety margin Q is positively correlated with the target angle α and the target distance d. See formula (7) for details.

[0111] Q=k1α+k2d (7)

[0112] See appendix Figure 5 This is a schematic diagram of a safety margin provided in an embodiment of this application. The target angle α is the angle between the target point of the suspected obstruction (hereinafter referred to as the obstruction) and the viewing cone of the target camera, and the target distance d is the distance between the target point and the viewing cone of the target camera. The target point is the spatial location point within the obstruction that is closest to the viewing cone of the target camera. k1 and k2 are both numbers greater than 0.

[0113] A safety threshold is set. The processor compares the safety margin with the safety threshold. If the safety margin is less than or equal to the safety threshold, the target camera is adjusted so that the adjusted target camera's safety margin is greater than or equal to the safety threshold. In one example, the target camera moves in an arc; see appendix for details. Figure 5 As shown, the center of the circle is the center of the sphere in the hand.

[0114] For ease of processing, the hand receiving the acupuncture is abstracted as a sphere. The radius of the sphere should be larger than the actual sphere enveloping the hand. For example, the radius of the sphere is r = 7cm, and the coordinates of its center are obtained by averaging the three-dimensional coordinates of the target viewpoint in the target camera.

[0115] It should be noted that in this embodiment, the second camera may be located above the acupuncture site or in other locations; this embodiment does not specifically limit the location.

[0116] S220. Based on the forearm electromyography data, the finger piezoelectric data, and the acupuncture video data, identify the acupuncture technique used by the acupuncture subject during the acupuncture operation.

[0117] In one example, the processor can input the acquired forearm electromyography data, finger piezoelectric data, and acupuncture video data into a multimodal deep neural network. Through processing by the multimodal deep neural network, the processor outputs the acupuncture techniques used by the acupuncture subject during the acupuncture operation, thereby achieving the purpose of recognizing acupuncture techniques.

[0118] Among them, the multimodal deep neural network is a model trained based on sample data. The sample data consists of historical forearm electromyography data, historical finger piezoelectric data, and historical acupuncture video data, as well as the corresponding labels for the acupuncture techniques.

[0119] In another example, the processor can identify acupuncture techniques through segmentation and fusion.

[0120] Step 1: The processor segments the acupuncture video data, obtaining m sub-video data. Each of the m sub-video data corresponds to one acupuncture technique, and any two sub-video data correspond to different acupuncture techniques. m is a positive integer. For example, the acupuncture video data may include three sub-video data: sub-video data A, sub-video data B, and sub-video data C. Sub-video data A corresponds to the tonifying technique (rotating and twisting), sub-video data B corresponds to the reducing technique (rotating and twisting), and sub-video data C corresponds to the tonifying technique (lifting and thrusting). To better help learners understand the acupuncture techniques, the sub-video data should include the start and end frames of each technique. That is, the sub-video data includes the start frame of the acupuncture technique and the end frame of the acupuncture technique.

[0121] In this embodiment, the processor can segment acupuncture video data based on a preset segmentation algorithm. The preset segmentation algorithm is a segmentation algorithm trained on pre-constructed training samples. The training samples include the start and end frames corresponding to acupuncture techniques, as well as the labels corresponding to the acupuncture techniques.

[0122] In one specific implementation, the training method for the preset segmentation algorithm can be achieved using a 70% training set, a 20% test set, and a 10% validation set. The training set should include, as far as possible, the start and end frames corresponding to all acupuncture methods.

[0123] Considering the large data volume of acupuncture video data, and the fact that background information can interfere with the analysis of acupuncture techniques, optical flow is used to process the acupuncture video data, taking into account the small range of motion in acupuncture techniques and the minimal changes in the video background, to obtain finger contour information and needle contour information. The processor inputs the finger contour information and needle contour information into a preset segmentation algorithm to obtain feature vectors corresponding to m sub-video information.

[0124] It should be noted that, in the embodiments of this application, the preset segmentation algorithm can be a convolutional neural network or other algorithms, and the embodiments of this application are not specifically limited.

[0125] Step 2: Based on the acquisition time window of each of the m sub-video data, the processor uses a sliding window method to segment the forearm electromyography data and finger piezoelectric data to obtain m sub-electromyography data and m sub-piezoelectric data; wherein, the acquisition timestamp of 1 sub-video data corresponds to 1 sub-electromyography data and 1 sub-piezoelectric data.

[0126] For example, the acupuncture video data includes three sub-video data: sub-video data A, sub-video data B, and sub-video data C. The acquisition time window corresponding to sub-video data A is from T1 to T2, the acquisition time window corresponding to sub-video data B is from T2 to T3, and the acquisition time window corresponding to sub-video data A is from T3 to T4. The processor can segment the forearm electromyography (EMG) data and finger piezoelectric data based on the acquisition time windows T1 to T2, T2 to T3, and T3 to T4 to obtain sub-EMG data A, sub-EMG data B, and sub-EMG data C, as well as sub-piezoelectric data A, sub-piezoelectric data B, and sub-piezoelectric data C. Sub-EMG data A, sub-piezoelectric data A, and sub-video data A all correspond to the T1 to T2 time window; sub-EMG data B, sub-piezoelectric data B, and sub-video data B all correspond to the T2 to T3 time window; and sub-EMG data C, sub-piezoelectric data C, and sub-video data C all correspond to the T3 to T4 time window.

[0127] Furthermore, if the forearm electromyography (EMG) data is EMG voltage and the finger piezoelectric data is piezoelectric voltage, in this embodiment, a voltage threshold V0 can be set within the sampling time window, and voltages greater than the threshold can be defined as peak values. For the same sampling time window, the piezoelectric data of acupuncture manipulation and the peak duration T1, peak integral H, peak interval T2, and peak average voltage value V under the EMG signal are recorded as sub-data.

[0128] The system integrates sub-video data, sub-piezoelectric data, and sub-electromyographic data corresponding to the same acquisition time window; based on the fused information, it identifies the acupuncture techniques used by the acupuncture subject during the acupuncture operation.

[0129] In this embodiment, the forearm electromyography data, finger piezoelectric data, and acupuncture video data are synchronized on the time axis. Therefore, the fusion of sub-video data, sub-piezoelectric data, and sub-electromyography data within the same acquisition time window constitutes the fusion of three data sets for the same acupuncture technique. The processor identifies the acupuncture technique based on the fused information.

[0130] Specifically, the processor concatenates the sub-video data, sub-piezoelectric data, and sub-electromyographic data within the same acquisition time window into a feature vector, and inputs this feature vector into the neural network for processing to obtain the final recognition result.

[0131] Exemplary illustration, attached Figure 6 This is a schematic diagram of a multimodal data fusion structure provided in an embodiment of this application. Specifically, the processor first processes the acupuncture video data using optical flow, then inputs the processed data into a convolutional neural network (CNN) module to obtain the video feature vector corresponding to the video data. The finger piezoelectric data is segmented using a sliding window (i.e., processed using a sampling time window) to obtain the piezoelectric feature vector, and the forearm electromyography data is segmented using a sliding window to obtain the electromyography feature vector.

[0132] Next, the processor performs multimodal fusion of video feature vectors, piezoelectric feature vectors, and electromyographic feature vectors. Specifically, it fuses vector data from the same time window to obtain a fused vector. This fused vector is then subjected to convolution and pooling processes, and the processed vector is input into a fully connected layer for further processing. The processed information is then output through an output layer to produce the corresponding acupuncture technique. The output layer includes a softmax activation function.

[0133] In summary, this application provides a data processing method that, during acupuncture, acquires acupuncture video data of the acupuncture site, forearm electromyography (EMG) data of the acupuncturist, and finger piezoelectric data of the acupuncturist. Based on these data, the acupuncture techniques employed by the acupuncturist during the acupuncture operation are identified. Since acupuncture video data, forearm EMG data, and finger piezoelectric data can comprehensively characterize acupuncture techniques, and the acupuncture video data can recreate the actual acupuncture scenario, identifying acupuncture techniques based on this comprehensive data (including video data, forearm EMG data, and finger piezoelectric data) is more accurate than clinical observation and is more beneficial for learners to learn acupuncture techniques.

[0134] The data processing method provided in the embodiments of this application has been introduced above. The data processing method provided in the embodiments of this application will be further explained below in conjunction with specific applications.

[0135] Appendix Figure 7 This application provides another data processing method flowchart. The acquisition device of the computing device used in this method specifically includes a robotic arm and an adjustable camera, a fixed camera, a wireless electromyography sensor, and a piezoelectric film finger sleeve. The method includes the following:

[0136] S710, robotic arm navigation, adjustable camera to target acupoint.

[0137] The target acupoints can be marked with a circle.

[0138] S720, before needle administration, the processor obtains and adjusts the viewing angle based on the needle administration operator.

[0139] S730, begin needle application, the processor calculates the safety margin, and adjusts the adjustable camera based on the safety margin.

[0140] Steps S720 to S730 correspond to the specific content of the preparation and acupuncture stages described above, and will not be repeated here.

[0141] The S740 processor triggers both the adjustable and fixed cameras to capture images, obtaining dual-view video information.

[0142] The S750 wireless electromyography sensor synchronously acquires electromyographic signals.

[0143] S760 and piezoelectric film finger sleeves simultaneously acquire fingertip piezoelectric data.

[0144] It should be noted that S740 to S760 execute synchronously.

[0145] The S770 processor inputs dual-view video information, electromyography signals, and fingertip piezoelectric data into a multimodal deep neural network.

[0146] The multimodal deep neural network is described in step S220 and will not be repeated here.

[0147] The S780 multimodal deep neural network outputs fused recognition results.

[0148] This data processing method integrates hardware devices such as robotic arms, adjustable cameras, fixed cameras, wireless electromyography (EMG) sensors, and piezoelectric film finger sleeves to achieve multimodal information fusion for high-definition image acquisition, real-time muscle electrical activity monitoring, and fingertip pressure monitoring during acupuncture. The robotic arm precisely navigates the adjustable camera to the target acupoint, ensuring accurate shooting angles; the adjustable and fixed cameras acquire video information from different perspectives, providing rich visual data; the wireless EMG sensor monitors the EMG signals of the acupuncturist's hand in real time, helping to assess the tension and coordination of hand muscles during acupuncture; and the piezoelectric film finger sleeve measures fingertip pressure, ensuring the accuracy of acupuncture force. The processor receives this multimodal data and fuses it through a multimodal deep neural network, ultimately outputting accurate acupuncture recognition results. These devices and data processing workflows together enable more accurate and optimized acupuncture training, improving the learning quality for learners.

[0149] In addition, this application also provides a data processing apparatus.

[0150] Appendix Figure 8 This is a schematic diagram of a data processing apparatus provided in an embodiment of this application. The apparatus 800 includes:

[0151] The acquisition unit 801 is used to acquire forearm electromyography data, finger piezoelectric data and acupuncture video data of the acupuncture site during the acupuncture operation.

[0152] The forearm electromyography data is used to describe the muscle activity state of the forearm of the acupuncture subject, the finger piezoelectric data indicates the pressure applied by the acupuncture subject to the acupuncture site, and the acupuncture video data is used to describe the operation process of the acupuncture subject performing acupuncture on the acupuncture site.

[0153] The identification unit 802 is used to identify the acupuncture technique used by the acupuncture subject during the acupuncture operation based on the forearm electromyography data, the finger piezoelectric data, and the acupuncture video data.

[0154] Optionally, the computing device includes a first camera and a second camera, the first camera and the second camera having different shooting angles; the acquisition unit 801 is used for:

[0155] The acupuncture video data is acquired using the first camera and the second camera.

[0156] The first camera has a variable shooting angle, while the second camera has a fixed shooting angle. Using both the first and second cameras, the acquisition unit 801 is further configured to:

[0157] The marked area of ​​the acupuncture site is obtained; the acupuncture video data is acquired simultaneously using the first camera and the second camera; wherein, during the acquisition process, the marked area is always located within the target field of view of the first camera.

[0158] Optionally, the acquisition unit 801 is also used for:

[0159] During the preparation stage of acupuncture, a target viewpoint is acquired; wherein, the target viewpoint includes multiple key points of the hand of the acupuncture object, and the initial posture of the target viewpoint is the posture corresponding to the marked area being located in the target field of view area of ​​the first camera before the preparation stage of acupuncture.

[0160] Based on the target viewpoint, the homography matrix of the target viewpoint is obtained; the homography matrix is ​​related to the pixel coordinates of the target viewpoint in the video frame captured by the first camera and the pixel coordinates of the video frame captured by the second camera.

[0161] Based on the homography matrix, the pose change of the target viewpoint relative to the initial pose during the preparation stage of needle application is obtained;

[0162] Based on the change in posture, the shooting angle of the first camera is adjusted so that the adjusted marked area is located in the target field of view of the first camera.

[0163] Optionally, the acquisition unit 801 is also used for:

[0164] If there are any suspicious obstructions at the beginning of the acupuncture treatment, a safety margin should be obtained.

[0165] Wherein, the safety margin is positively correlated with the target angle and the target distance; the target angle is the angle between the target point of the suspected obstruction and the viewing cone of the target camera; the target distance is the distance between the target point and the viewing cone of the target camera; the target point is the spatial location point in the suspected obstruction that is closest to the viewing cone of the target camera; the target camera is the first camera and / or the second camera;

[0166] If the safety margin is less than or equal to the safety threshold, adjust the target camera so that the safety margin of the adjusted target camera is greater than or equal to the safety threshold.

[0167] Optionally, the identification unit 802 is used for:

[0168] The acupuncture video data is segmented to obtain m sub-video data. Each sub-video data corresponds to one acupuncture technique, and any two sub-video data correspond to different acupuncture techniques. Here, m is a positive integer.

[0169] Based on the acquisition time window of each of the m sub-video data, the forearm electromyography data and the finger piezoelectric data are segmented to obtain m sub-electromyography data and m sub-piezoelectric data; wherein, the acquisition timestamp of 1 sub-video data corresponds to 1 sub-electromyography data and 1 sub-piezoelectric data.

[0170] The system integrates sub-video data, sub-piezoelectric data, and sub-electromyographic data corresponding to the same acquisition time window; based on the fused information, it identifies the acupuncture techniques used by the acupuncture subject during the acupuncture operation.

[0171] Optionally, segmenting the acupuncture video data to obtain m sub-video data includes:

[0172] The acupuncture video data is segmented using a preset segmentation algorithm to obtain m sub-video data. The preset segmentation algorithm is a segmentation algorithm trained based on pre-constructed training samples. The training samples include the start frame and end frame corresponding to the acupuncture technique, as well as the label corresponding to the acupuncture technique.

[0173] In summary, this application provides a data processing device that, during acupuncture operations, acquires acupuncture video data of the acupuncture site, forearm electromyography (EMG) data of the acupuncturist, and finger piezoelectric data of the acupuncturist. Based on the acupuncture video data, forearm EMG data, and finger piezoelectric data, it identifies the acupuncture techniques used by the acupuncturist during the acupuncture operation. Since the acupuncture video data, forearm EMG data, and finger piezoelectric data can comprehensively characterize acupuncture techniques, and the acupuncture video data can recreate the actual acupuncture scenario for the acupuncturist, identifying acupuncture techniques based on the comprehensive data from these three sources is more accurate than clinical observation and is more conducive to learners mastering acupuncture techniques.

[0174] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute the various steps or processes executed by the network device or terminal device in any of the foregoing method embodiments.

[0175] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when run on a computer, causes the computer to execute the various steps or processes executed by the network device or terminal device in any of the foregoing method embodiments.

[0176] The computer-readable storage medium may be the aforementioned volatile memory or non-volatile memory, or it may include both volatile memory and non-volatile memory.

[0177] In the embodiments of this application, the terms and English abbreviations are exemplary examples given for ease of description and should not be construed as limiting the application in any way. This application does not preclude the possibility of defining other terms that can achieve the same or similar functions in existing or future agreements.

[0178] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0180] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0181] In summary, the above description is merely a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, Applied to a computing device, the method includes: During the acupuncture procedure, acupuncture video data of the acupuncture site, electromyographic data of the forearm of the acupunctured subject, and piezoelectric data of the fingers of the acupunctured subject are acquired. The forearm electromyography data is used to describe the muscle activity state of the forearm of the acupuncture subject, the finger piezoelectric data indicates the pressure applied by the acupuncture subject to the acupuncture site, and the acupuncture video data is used to describe the operation process of the acupuncture subject performing acupuncture on the acupuncture site. Based on the forearm electromyography data, the finger piezoelectric data, and the acupuncture video data, the acupuncture techniques used by the acupuncture subject during the acupuncture operation are identified.

2. The processing method according to claim 1, characterized in that, The computing device includes a first camera and a second camera, which have different shooting angles. Acquiring acupuncture video data of the acupuncture site includes: The acupuncture video data is acquired using the first camera and the second camera.

3. The processing method according to claim 2, characterized in that, The first camera has a variable shooting angle, while the second camera has a fixed shooting angle. Before acquiring the acupuncture video data using the first and second cameras, the method further includes: Obtain the marked area of ​​the acupuncture site; The step of acquiring the acupuncture video data using the first camera and the second camera includes: The acupuncture video data is acquired simultaneously using the first camera and the second camera; wherein, during the acquisition process, the marked area is always located within the target field of view of the first camera.

4. The processing method according to claim 3, characterized in that, The method further includes: During the preparation stage of acupuncture, a target viewpoint is acquired; wherein, the target viewpoint includes multiple key points of the hand of the acupuncture object, and the initial posture of the target viewpoint is the posture corresponding to the marked area being located in the target field of view area of ​​the first camera before the preparation stage of acupuncture. Based on the target viewpoint, the homography matrix of the target viewpoint is obtained; the homography matrix is ​​related to the pixel coordinates of the target viewpoint in the video frame captured by the first camera and the pixel coordinates of the video frame captured by the second camera. Based on the homography matrix, the pose change of the target viewpoint relative to the initial pose during the preparation stage of needle application is obtained; Based on the change in posture, the shooting angle of the first camera is adjusted so that the adjusted marked area is located in the target field of view of the first camera.

5. The processing method according to claim 4, characterized in that, The method further includes: If there are any suspicious obstructions at the beginning of the acupuncture treatment, a safety margin should be obtained. Wherein, the safety margin is positively correlated with the target angle and the target distance; the target angle is the angle between the target point of the suspected obstruction and the viewing cone of the target camera; the target distance is the distance between the target point and the viewing cone of the target camera; the target point is the spatial location point in the suspected obstruction that is closest to the viewing cone of the target camera; the target camera is the first camera and / or the second camera; If the safety margin is less than or equal to the safety threshold, adjust the target camera so that the safety margin of the adjusted target camera is greater than or equal to the safety threshold.

6. The processing method according to claim 1, characterized in that, The step of identifying the acupuncture techniques used by the acupuncture recipient during the acupuncture procedure based on the forearm electromyography data, the finger piezoelectric data, and the acupuncture video data includes: The acupuncture video data is segmented to obtain m sub-video data. Each sub-video data corresponds to one acupuncture technique, and any two sub-video data correspond to different acupuncture techniques. Here, m is a positive integer. Based on the acquisition time window of each of the m sub-video data, the forearm electromyography data and the finger piezoelectric data are segmented to obtain m sub-electromyography data and m sub-piezoelectric data; wherein, the acquisition timestamp of 1 sub-video data corresponds to 1 sub-electromyography data and 1 sub-piezoelectric data. The system integrates sub-video data, sub-piezoelectric data, and sub-electromyographic data corresponding to the same acquisition time window; based on the fused information, it identifies the acupuncture techniques used by the acupuncture subject during the acupuncture operation.

7. The processing method according to claim 6, characterized in that, The process of segmenting the acupuncture video data to obtain m sub-video data includes: The acupuncture video data is segmented using a preset segmentation algorithm to obtain m sub-video data. The preset segmentation algorithm is a segmentation algorithm trained based on pre-constructed training samples. The training samples include the start frame and end frame corresponding to the acupuncture technique, as well as the label corresponding to the acupuncture technique.

8. The method according to any one of claims 1-7, characterized in that, The computing device further includes an electromyography (EMG) sensor and a piezoelectric film finger sleeve. The EMG sensor is deployed on the forearm used by the acupuncture subject during acupuncture, and the piezoelectric film finger sleeve is deployed on the finger used by the acupuncture subject during acupuncture. The method further includes: The electromyographic data is acquired through the electromyographic sensor; the piezoelectric data is acquired through the piezoelectric film finger sleeve.

9. A data processing apparatus, characterized in that, Applied to a computing device, the apparatus includes: The acquisition unit is used to acquire forearm electromyography data, finger piezoelectric data, and acupuncture video data of the acupuncture site during the acupuncture operation. The forearm electromyography data is used to describe the muscle activity state of the forearm of the acupuncture subject, the finger piezoelectric data indicates the pressure applied by the acupuncture subject to the acupuncture site, and the acupuncture video data is used to describe the operation process of the acupuncture subject performing acupuncture on the acupuncture site. The identification unit is used to identify the acupuncture technique used by the acupuncture subject during the acupuncture operation based on the forearm electromyography data, the finger piezoelectric data, and the acupuncture video data.

10. A computing device, characterized in that, The computing device includes a processor, a video acquisition device, an electromyography sensor, and a piezoelectric film finger sleeve, all of which are connected to the processor. The video acquisition device is used to acquire acupuncture video data of the acupuncture site during the acupuncture operation and send the acupuncture video data to the processor; The electromyography sensor is used to acquire electromyography data of the forearm of the acupuncture subject during the acupuncture operation and send the forearm electromyography data to the processor; The piezoelectric film finger sleeve is used to acquire the piezoelectric data of the acupuncture object's fingers during the acupuncture operation and send the piezoelectric data of the fingers to the processor; The forearm electromyography data is used to describe the muscle activity state of the forearm of the acupuncture subject, the finger piezoelectric data indicates the pressure applied by the acupuncture subject to the acupuncture site, and the acupuncture video data is used to describe the operation process of the acupuncture subject performing acupuncture on the acupuncture site. The processor is used to: identify the acupuncture techniques used by the acupuncture subject during the acupuncture operation based on the forearm electromyography data, the finger piezoelectric data, and the acupuncture video data.

Images