Method and system for determining finger joint angle

Abstract

The embodiment of the invention provides a method and system for determining the angle of a finger joint, and the method comprises the steps: obtaining first sensor data related to at least two target finger joints of a user, and enabling the first sensor data to be collected through at least two strain sensors, each strain sensor is arranged in the glove body worn by the user and located at one of at least two target finger joints, and the at least two target finger joints comprise at least two adjacent metacarpophalangeal joints of the user; obtaining a first mapping relation, wherein the first mapping relation reflects the relation between the sensor data corresponding to the at least two target finger joints and the joint angles; and determining a first joint angle of each target finger joint based on the first sensor data and the first mapping relation.

Description

Cross-references

[0001] This specification claims the priority of the international application with application number PCT / CN2023 / 124295 filed on October 12, 2023, and the entire contents are incorporated herein by reference. Technical Field

[0002] The present invention relates to the field of data processing, and in particular to a method and system for determining finger joint angles. Background Art

[0003] With the gradual maturity of AR / VR technology and the rise of the concept of the metaverse, smart electronic devices have put forward higher requirements for human-computer interaction technology. Flexible sensors can be easily integrated into wearable devices such as smart clothing and smart gloves to achieve accurate recognition and restoration of human movements. They are one of the important underlying technologies of the metaverse and have received extensive attention and research. In order to achieve accurate recognition and restoration of hand movements, it is necessary to use sensors to collect the movements of the hands, especially the fingers, in multiple degrees of freedom, and then determine the angles of the finger joints (bending and / or swinging angles) based on the collected sensor data.

[0004] If you want to collect multi-dimensional finger movements, you generally need to place inertial sensors (such as nine-axis sensors, gyroscopes) at non-joint positions on the fingers, and infer the bending of the fingers by measuring the shapes of various parts (such as spatial positions). This method requires the use of multiple gyroscopes to measure the shapes of various parts of the hand, which is costly. You can also place single-degree-of-freedom strain sensors at the joints and gyroscopes at other locations, and combine the two to calculate the bending of the fingers. However, since the fingers may have multiple degrees of freedom (for example, the metacarpophalangeal joint can have at least two degrees of freedom of bending and swinging), the measurement and calculation results of only placing single-degree-of-freedom strain sensors at the joints are not accurate enough. Summary of the invention

[0005] One of the embodiments of the present specification provides a method for determining the angle of a finger joint, including: obtaining first sensor data related to at least two target finger joints of a user, the first sensor data being collected using at least two strain sensors, each of the strain sensors being arranged in a glove body worn by the user and being located at one of the at least two target finger joints, wherein the at least two target finger joints include at least two adjacent metacarpophalangeal joints of the user; obtaining a first mapping relationship, the first mapping relationship reflecting the relationship between the sensor data corresponding to the at least two target finger joints and the joint angle; and determining the first joint angle of each of the target finger joints based on the first sensor data and the first mapping relationship. Considering the coupling relationship between at least two metacarpophalangeal joints when determining the finger joint angle can improve the accuracy of the obtained first joint angle.

[0006] In some embodiments, the at least two target finger joints include some of the user's finger joints. By acquiring sensor data corresponding to only some of the finger joints, the calculation process is made more targeted and the amount of data in the calculation process is reduced.

[0007] In some embodiments, the at least two target finger joints include all the metacarpophalangeal joints and all the proximal interphalangeal joints of the user. By taking all the metacarpophalangeal joints and all the proximal interphalangeal joints as the target finger joints, it is not necessary to arrange sensors at the distal interphalangeal joints, which simplifies the structure of the glove system and saves costs; at the same time, the sensor data corresponding to the target finger joints does not need to be processed such as data screening, thereby simplifying the processing steps. In addition, by taking all the metacarpophalangeal joints and all the proximal interphalangeal joints as the target finger joints, the processing process can take into account the coupling relationship between all the metacarpophalangeal joints and all the proximal interphalangeal joints, so that the first angle of the target finger joint is more accurate.

[0008] In some embodiments, the at least two strain sensors include at least two first strain sensors located at the at least two metacarpophalangeal joints, each of the first strain sensors being configured to measure the deformation of the corresponding metacarpophalangeal joint in two degrees of freedom. The first strain sensors can simultaneously detect the motion information of the user's joints in multiple degrees of freedom, have a simple structure, are easy to manufacture, and can reduce the overall size of the sensor.

[0009] In some embodiments, the at least two target finger joints further include at least two proximal interphalangeal joints corresponding to the at least two metacarpophalangeal joints, and the at least two strain sensors include at least two second strain sensors located at the at least two proximal interphalangeal joints, and each second strain sensor is configured to measure the deformation of the corresponding proximal interphalangeal joint in a single degree of freedom. Considering that the proximal interphalangeal joint only corresponds to one bending degree of freedom, using a single-axis sensor to measure the sensor data of the proximal interphalangeal joint when bending has a simple structure and low cost.

[0010] In some embodiments, the method further includes: for each of the proximal interphalangeal joints, determining the second joint angle of the distal interphalangeal joint corresponding to the proximal interphalangeal joint based on a second mapping relationship and the first joint angle of the proximal interphalangeal joint, wherein the second mapping relationship reflects the relationship between the joint angles of the proximal interphalangeal joint and the distal interphalangeal joint.

[0011] In some embodiments, the first sensor data includes decoupled data of the at least two target finger joints, and the decoupled data is determined based on the raw data collected by the at least two strain sensors. By decoupling the raw data to obtain decoupled data, the composite deformation of the first strain sensor can be decoupled into separate deformations of each degree of freedom, and the decoupled data corresponding to each degree of freedom can maintain a high degree of independence and stability, thereby improving the accuracy and sensitivity of the deformation on the at least two degrees of freedom obtained.

[0012] In some embodiments, determining the first joint angle of each target finger joint based on the first sensor data and the first mapping relationship includes: obtaining reference sensor data related to at least two target finger joints of the user, the reference sensor data being collected by using the at least two strain sensors when the user makes a preset gesture; based on the reference sensor data, correcting the first mapping relationship to obtain a corrected mapping relationship corresponding to the user; and determining the first joint angle of each target finger joint based on the corrected mapping relationship and the first sensor data.

[0013] In some embodiments, the at least two target finger joints only include at least two adjacent metacarpophalangeal joints of the user, and the method further includes: obtaining second sensor data related to at least two proximal interphalangeal joints corresponding to the at least two metacarpophalangeal joints, the second sensor data being acquired using at least two second strain sensors, each of the second strain sensors being arranged in the glove body worn by the user and located at one of the at least two proximal interphalangeal joints; and determining a third joint angle of each of the proximal interphalangeal joints based on the second sensor data, the first joint angles of the at least two metacarpophalangeal joints and a third mapping relationship, the third mapping relationship reflecting the relationship between the joint angles of the at least two metacarpophalangeal joints, the sensor data of the at least two proximal interphalangeal joints and the joint angles of the at least two proximal interphalangeal joints.

[0014] In some embodiments, the glove body includes fabric wrapped around each finger of the user and fabric located between adjacent fingers, wherein the elastic modulus of the fabric between adjacent fingers is less than the elastic modulus of the fabric on each finger, and the method further includes obtaining the value of a characteristic parameter of the fabric between the adjacent fingers, the characteristic parameter including at least one of an elastic coefficient, a lateral dimension, and a rebound coefficient, the first mapping relationship further reflects the relationship between the characteristic parameter and the sensor data corresponding to the at least two target finger joints and the joint angles of the at least two target finger joints, and the first joint angle of each of the target finger joints is further determined based on the value of the characteristic parameter. Considering the influence of the fabric characteristics between adjacent fingers on the coupling relationship between the finger joints, the first mapping relationship can be made more in line with the actual situation, and the determined first joint angle can be made more accurate.

[0015] In some embodiments, the glove body includes a position sensor arranged at each of the target finger joints, and the method further includes determining the distance information between the at least two strain sensors based on the position data collected by the position sensor at each of the target finger joints, and the first mapping relationship further reflects the relationship between the distance between the at least two strain sensors and the sensor data corresponding to the at least two target finger joints and the joint angles of the at least two target finger joints, and the first joint angle of each of the target finger joints is further determined based on the distance information. Considering the distance information between at least two strain sensors can make the first mapping relationship more consistent with the actual situation, and make the determined first joint angle more accurate.

[0016] In some embodiments, the first mapping relationship is determined by: acquiring multiple data samples, each of the data samples corresponds to a sample user performing a sample gesture, and includes sample sensor data related to at least two sample finger joints of the sample user and sample joint angles of the at least two sample finger joints, the sample sensor data is collected by using at least two sample sensors arranged in a sample glove body worn by the sample user, and the at least two sample finger joints are of the same type as the at least two target finger joints; based on the multiple data samples, determining the first mapping relationship by a data fitting algorithm or a machine learning algorithm.

[0017] In some embodiments, at least one of the data samples is obtained based on the following method: for each data sample in the at least one data sample, sensor data collected by the at least two sample sensors when the sample user performs the sample gesture is obtained as sample sensor data of the data sample; an optical image of the hand taken when the sample user performs the sample gesture is obtained; and based on the optical image of the hand, a sample joint angle of the data sample is determined.

[0018] In some embodiments, at least one of the data samples is obtained based on the following method: for each data sample in the at least one data sample, the terminal device is controlled to display the hand model corresponding to the sample gesture to the sample user corresponding to the data sample; the sensor data collected by the at least two sample sensors when the sample user imitates the sample gesture is obtained as the sample sensor data of the data sample; and the sample joint angle of the data sample is determined based on the hand model.

[0019] In some embodiments, the obtaining of sensor data collected by the at least two sample sensors when the sample user imitates the sample gesture as the sample sensor data of the data sample includes: obtaining an optical image of the hand taken when the sample user imitates the sample gesture; determining whether the gesture of the sample user is the sample gesture based on the optical image of the hand; in response to determining that the gesture of the sample user is the sample gesture, obtaining the sensor data currently collected by the at least two sample sensors as the sample sensor data; or in response to determining that the gesture of the sample user is not the sample gesture, controlling the terminal device to display prompt information for adjusting the gesture to the sample user.

[0020] In some embodiments, the first mapping relationship is represented by a mapping function, the glove body includes a position sensor arranged at each of the target finger joints, and the determination of the first joint angle of each of the target finger joints based on the first sensor data and the first mapping relationship includes: determining the distance information between the at least two strain sensors based on the position data collected by the position sensor at each of the target finger joints; correcting the mapping function based on the distance information to obtain a corrected mapping function corresponding to the user; and determining the first joint angle of each of the target finger joints based on the corrected mapping function and the first sensor data. Correcting the first mapping relationship according to the distance information can enable the first mapping relationship to be combined with the hand features of different users, thereby making the determined first joint angle more targeted and more accurate.

[0021] One of the embodiments of the present specification provides a system for determining the angle of finger joints, comprising at least one storage medium, comprising a set of instructions; and at least one processor communicating with the at least one storage medium, wherein when executing the instruction set, the at least one processor causes the system to execute the method for determining the angle of finger joints described in the present specification.

[0022] One of the embodiments of the present specification provides a non-transitory computer-readable medium, comprising executable instructions, which, when executed by at least one processor, cause the at least one processor to execute the method for determining the finger joint angle described in the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0024] Figure 1 is a schematic diagram of an exemplary glove system according to some embodiments of the present specification;

[0025] Figure 2 is a schematic diagram of the joint positions of a user's hand according to some embodiments of this specification;

[0026] Figure 3 is a three-dimensional coordinate system of a user's hand according to some embodiments of this specification;

[0027] Figure 4 is a schematic diagram of application scenarios of the system shown in some embodiments of this specification;

[0028] Figure 5A is an exemplary block diagram of a processing device 410A according to some embodiments of the present specification;

[0029] Figure 5B is an exemplary block diagram of a processing device 410B according to some embodiments of the present specification;

[0030] Figure 6 is a flow chart of a method for determining finger joint angles according to some embodiments of this specification;

[0031] Figure 7 is an exemplary schematic diagram of the structure of a first strain sensor according to some embodiments of this specification

[0032] Figure 8 is an exemplary schematic diagram of the structure of another first strain sensor according to some embodiments of this specification;

[0033] Fig.9A is a schematic diagram of the distribution of four capacitor structures on a flexible substrate according to some embodiments of this specification;

[0034] Fig. 9B yes Fig.9A The schematic diagram of the cross-sectional structure of the four capacitor structures shown along the Y-axis direction;

[0035] Fig. 10A is a schematic diagram of a second sensor without deformation according to some embodiments of this specification;

[0036] Fig. 10B is based on Fig. 10A Schematic diagram of deformation of the second sensor after being stretched or compressed along the long axis direction

[0037] Fig.11 is based on Fig. 10A A side view schematic diagram of the first strain sensor after bending and deformation around an axis parallel to the short axis direction;

[0038] Fig.12 is based on Fig. 10A A schematic top view of the first strain sensor after bending and deformation around an axis parallel to the thickness direction;

[0039] Fig.13 is a schematic diagram of wrinkles on the flexible sensor according to some embodiments of this specification;

[0040] Fig.14 is based on Fig.13 The schematic diagram of the curve obtained by the actual measurement of the sensor shown;

[0041] Fig.15 is a cross-sectional schematic diagram of a first strain sensor according to some embodiments of this specification;

[0042] Fig.16 is a flow chart of another method for determining finger joint angles according to some embodiments of this specification;

[0043] Fig.17 is a flow chart of another method for determining finger joint angles according to some embodiments of this specification;

[0044] Fig.18 is an exemplary flow chart of determining the third joint angle according to some embodiments of this specification;

[0045] Fig.19 is a flow chart of a method for determining a correspondence between sensor data and finger joint angles according to some embodiments of this specification;

[0046] Fig. 20 is an exemplary flow chart of a method for determining a hand model according to some embodiments of this specification;

[0047] Fig.21 is an exemplary flow chart of a method for determining a sample joint angle according to some embodiments of the present specification;

[0048] Fig. 22 is an exemplary flow chart of another method for determining a sample joint angle according to some embodiments of the present specification. DETAILED DESCRIPTION

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0050] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0051] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprises" and "includes" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0052] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, the various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0053] In order to accurately identify and restore hand movements, it is necessary to collect the movements of the hands, especially the fingers, in multiple degrees of freedom. For example, the user's hand movements (including finger movements) can be captured through a glove system. Figure 1 is a schematic diagram of an exemplary glove system according to some embodiments of the present specification. Figure 1 As shown, the glove system 100 includes a glove body 110, a plurality of strain sensors (eg, first strain sensors 131-135 and second strain sensors 121-125), an elastic fabric structure 170, and a processor 140. The plurality of strain sensors may be distributed at different locations on the glove body 110.

[0054] The glove system 100 refers to a system for capturing the user's hand movements (including finger movements). The glove system 100 can recognize hand movements through one or more devices, such as motion capture gloves, sign language gloves, VR gloves, force feedback gloves, tactile feedback gloves, etc.

[0055] The glove body 110 refers to a wearable item for fitting with the hand. In some embodiments, in order to facilitate the glove body 110 to fit with the hand joints (for example, including the interphalangeal joints, metacarpophalangeal joints, wrist joints, etc.), the glove body 110 has a structure that adapts to the shape of the hand and extends along the hand joints. In some embodiments, the glove body 110 can be a flexible fabric that is easy to deform (such as bending deformation) along with the fingers when subjected to external forces. In some embodiments, the glove body 110 can provide support for the strain sensor to facilitate the arrangement of the strain sensor. For example, the glove body 110 can include one or more layers of flexible structure (for example, flexible fabric). The strain sensor can be fixed to the surface of a layer of flexible structure by means of gluing, pressing or sewing.

[0056] In some embodiments, the strain sensor can be configured to collect sensor data of the finger. When the user wears the glove body 110 and generates finger movement, the finger area of ​​the glove body 110 is deformed, and the strain sensor can convert the deformation information (e.g., deformation direction, deformation size, etc.) at the finger joint into an electrical signal, and the electrical signal generated by the strain sensor can reflect the posture data at the corresponding area of ​​the finger. In some embodiments, the strain sensor may include but is not limited to a capacitive strain sensor, an inductive sensor, a resistive sensor, an optical fiber sensor, etc.

[0057] In some embodiments, strain sensors may be disposed at the knuckles of the glove body 110 . Figure 2 It is a schematic diagram of the joint position of the user's hand according to some embodiments of this specification. The finger joints of the hand may include the metacarpophalangeal joints at positions 0, 2, 5, 8, and 11 and the interphalangeal joints at positions 1, 3, 6, 9, 12, 4, 7, 10, and 13. A plurality of strain sensors may include a first strain sensor (e.g., a first strain sensor 131, 132, 133, 134, and 135) disposed at the metacarpophalangeal joints 0, 2, 5, 8, and 11, and the first strain sensor is configured to measure the deformation of the metacarpophalangeal joint in two degrees of freedom. A plurality of strain sensors may also include a second strain sensor (e.g., a second strain sensor 121, 122, 123, 124, and 125) disposed at the interphalangeal joints 1, 3, 6, 9, 12, 4, 7, 10, and 13, and the second strain sensor is configured to measure the deformation of the interphalangeal joint in a single degree of freedom. In some embodiments, the second strain sensor includes a single degree of freedom sensor, such as a uniaxial sensor (e.g., an inductive, resistive bending sensor, or a capacitive stretching sensor). The first strain sensor includes a multi-degree-of-freedom sensor, such as a multi-axis sensor (eg, a capacitive multi-axis motion sensor).

[0058] The degrees of freedom described in this specification refer to the dimensions in which an object can move. To facilitate the description of the degrees of freedom of the hand's movement, a three-dimensional coordinate system is established for the hand's movement, for example, Figure 3 The three-dimensional coordinate system shown in FIG. In this coordinate system, the origin is the metacarpophalangeal joint, the finger extension direction is the Y axis, and the direction perpendicular to the hand plane is the Z axis. The hand plane can refer to the fitting plane of the palm part. For example, when the palm of the hand is placed on the table, the table or a plane passing through the hand and parallel to the table can be used as the hand plane. This three-dimensional coordinate system can be used to describe the degrees of freedom of hand movement. Specifically, in Figure 3 In the three-dimensional coordinate system shown, the rotation of the hand around the X-axis is defined as bending, the rotation of the hand around the Z-axis is defined as swinging, and the rotation of the hand around the Y-axis is defined as rotation. The interphalangeal joints can only bend and rotate. In other words, the deformation of the interphalangeal joints only corresponds to the one degree of freedom of bending. The metacarpophalangeal joints can bend, swing, and rotate. In other words, the deformation of the metacarpophalangeal joints corresponds to the two degrees of freedom of bending and swinging. Therefore, when any interphalangeal joint performs bending movement, the second strain sensor can measure the deformation of the glove body 110 at the corresponding interphalangeal joint. When any metacarpophalangeal joint performs bending or swinging movement, the first strain sensor can measure the deformation of the glove body 110 at the corresponding metacarpophalangeal joint.

[0059] Continue to see Figure 2, interphalangeal joints 4, 7, 10, and 13 are distal interphalangeal joints relatively far from the palm, and interphalangeal joints 1, 3, 6, 9, and 12 are proximal interphalangeal joints relatively close to the palm. In some embodiments, the movement of the distal interphalangeal joint is manifested as the following movement of the previous joint (proximal interphalangeal joint), and it is usually not bent independently, but bent together with the proximal interphalangeal joint. Therefore, the second strain sensor may not be arranged separately at the distal interphalangeal joint, and the second strain sensor may be arranged only at the proximal interphalangeal joint. In some embodiments, the deformation data at the distal interphalangeal joint may be generated by mapping the deformation data of the corresponding proximal interphalangeal joint. Therefore, the glove system 100 can collect the deformation data of at least five proximal interphalangeal joints in a single degree of freedom through multiple second strain sensors, and collect the deformation data of five metacarpophalangeal joints in two degrees of freedom through multiple first strain sensors. In total, the multiple second strain sensors and the multiple first strain sensors jointly provide at least 15 degrees of freedom deformation data on at least the user's fingers, thereby determining the posture data of the fingers.

[0060] Normally, in order to collect multi-degree-of-freedom movements of fingers, it is necessary to place a gyroscope at a non-joint position on the finger, and infer the bending of the finger by measuring the shape of each part of the finger. Alternatively, a single-degree-of-freedom strain sensor (such as a second strain sensor) is placed at the finger joint, and a gyroscope is placed at a non-joint position on the finger, and the measurement results of the two are combined to calculate the bending of the finger. Some embodiments in this specification control the cost of the product, reduce the subsequent data processing cost, and ensure the accuracy of the measurement by combining a dual-degree-of-freedom strain sensor (such as a first strain sensor) with a single-degree-of-freedom strain sensor (such as a second strain sensor). For more information about the first strain sensor and the second strain sensor, see Figure 7 Related description.

[0061] In some embodiments, the glove system 100 further includes an elastic fabric structure 170, which is disposed between the bases of adjacent fingers (near the metacarpophalangeal joints) to reduce or avoid the pulling of fabric at the bases of adjacent fingers when the fingers move. Since the flexible fabrics between the bases of adjacent fingers may be pulled against each other under the action of external forces, the data collected by the strain sensors at the adjacent metacarpophalangeal joints may be interfered with each other. In some embodiments, the elastic fabric structure 170 may be a slit structure or a pleated structure, and a structure for reducing or avoiding pulling between fabrics on adjacent fingers is provided inside, so that the elastic modulus of the fabric between adjacent fingers is smaller than the elastic modulus of the fabric on each finger, thereby reducing the influence of mutual interference between the first strain sensors disposed at the bases of different fingers, and ensuring that the data collected by each first strain sensor can accurately reflect the deformation of the corresponding position.

[0062] The glove system 100 may also include a processor 140, which is configured to receive and process data collected by multiple strain sensors (including multiple first strain sensors and multiple second strain sensors). For example, the processor 140 can obtain deformation data from the strain sensor and analyze and process it, for example, by calculating the deformation size and deformation angle of each joint of the finger through a mapping relationship (for example, a mapping function or a trained machine learning model). In some embodiments, the processor 140 can directly process the data collected by multiple strain sensors, or receive the data collected by multiple strain sensors, and transmit the data to an external device for data processing. In some embodiments, the processor 140 can perform corresponding functions based on the processing results of the data, such as gesture reproduction, voice broadcast, gesture control, etc. In some embodiments, the processor 140 can interact with an external processing device based on the processing results of the data. Among them, the external device includes a host computer such as a computer, a mobile phone, and a VR device. The interaction may include, for example, controlling the switch of the external device, adjusting the volume, the program process of the external device (such as controlling the movement of the game character), fitness feedback, etc.

[0063] When using a dual-degree-of-freedom strain sensor to collect hand movements, there is coupling between multiple finger joints. If the movement or deformation (such as bending or swinging) of a finger joint causes the sensor data corresponding to another finger joint to change in a coordinated manner, the two finger joints may be coupled. For example, when a finger joint bends, it pulls the fabric of the adjacent finger joint, causing the fabric of the adjacent finger joint to deform, causing the sensor data corresponding to the adjacent finger joint to change (for example, the sensor data corresponding to the swinging degree of freedom changes, or the swing change further causes the sensor data on the bending degree of freedom to change). Therefore, the coupling between different finger joints needs to be considered when determining the finger joint angles. For more information on determining finger joint angles, see Figure 6 and its description.

[0064] Figure 4 It is a schematic diagram of an application scenario of the system shown in some embodiments of this specification. The system 400 shown in the embodiment of this specification can be applied in various software, systems, platforms, and devices. In some embodiments, the system 400 can be used to determine the correspondence between sample sensor data and sample finger joint angles. In some embodiments, the system 400 can be used to determine the joint angle of a target finger joint. For example, the system can determine the mapping relationship between sensor data and finger joint angles based on the correspondence between sample sensor data and sample finger joint angles, and determine the joint angle of the target finger joint based on the mapping relationship.

[0065] like Figure 4As shown, the system 400 may include a processing device 410 , a collection device 420 , a terminal 430 , a storage device 440 , and a network 450 .

[0066] In some embodiments, the processing device 410 can process data and / or information obtained from other devices or system components. The processing device 410 can execute program instructions based on these data, information and / or processing results to perform one or more functions described in this specification. For example, the processing device 410 can obtain first sensor data related to at least two target finger joints of the user from the acquisition device 420, wherein the first sensor data is acquired using at least two strain sensors; for another example, the processing device 410 can obtain a first mapping relationship, the first mapping relationship reflects the relationship between the sensor data corresponding to at least two target finger joints and the joint angle, and then determine the first joint angle of each target finger joint based on the first sensor data and the first mapping relationship. For another example, the processing device 410 can obtain an optical image of the user's hand from the acquisition device 420, and determine the joint angles of the two finger joints of the user based at least on the optical image.

[0067] In some embodiments, the processing device 410 may be a single processing device or a processing device group, such as a server or a server group. The processing device group may be centralized or distributed (e.g., the processing device 410 may be a distributed system). In some embodiments, the processing device 410 may be local or remote. For example, the processing device 410 may be a processor 140 on a glove system. For another example, the processing device 410 may access information and / or data in the acquisition device 420, the terminal 430, and the storage device 440 through the network 450. For another example, the processing device 410 may be directly connected to the acquisition device 420, the terminal 430, and the storage device 440 to access the stored information and / or data. In some embodiments, the processing device 410 may be implemented on a cloud platform.

[0068] In some embodiments, the processing device 410 may include a processing engine 412. The processing engine 412 may process data and / or information related to the sensor data to perform one or more methods or functions described in this specification. For example, the processing engine 412 may process multiple sets of sample sensor data and corresponding hand optical images, and determine the correspondence between the sample sensor data and the sample joint angles corresponding thereto. For another example, the processing engine 412 may obtain and process first sensor data related to at least two target finger joints of the user to determine a first joint angle of each target finger joint.

[0069] In some embodiments, the processing engine 412 may include one or more processing engines (eg, a single-chip processing engine or a multi-chip processor). In some embodiments, the processing engine 412 may be integrated into the acquisition device 420 or the terminal 430 .

[0070] In some embodiments, the acquisition device 410 can be used to collect posture data of the user's fingers and / or deformation information for characterizing the movement at the finger joints. For example, the acquisition device 420 may include a first acquisition device 420-1 for collecting first sensor data related to at least two target finger joints of the user. In some embodiments, the first acquisition device 420-1 can be one or a combination of acquisition devices such as motion capture gloves, sign language gloves, VR gloves, force feedback gloves, tactile feedback gloves, etc. In some embodiments, the first acquisition device 420-1 can be the glove system 100 or a part of the glove system 100. For related content about the glove system 100, see Figure 1 For another example, the acquisition device 420 may include a second acquisition device 420-2 for acquiring an optical image of a user's hand. In some embodiments, the second acquisition device 420-2 may be a device for acquiring optical images, such as a camera, a video camera, etc.

[0071] In some embodiments, the acquisition device 420 can transmit information and / or data with the processing device 410, the terminal 430, and the storage device 440 through the network 450. In some embodiments, the acquisition device 420 can be directly connected to the processing device 410 or the storage device 440 to transmit information and / or data. For example, the acquisition device 420 (e.g., the first acquisition device 420-1) and the processing device 410 can be different parts of the same electronic device (e.g., a glove system) and are connected by metal wires.

[0072] In some embodiments, terminal 430 may be a terminal used by a user or other entity. For example, terminal 430 may be a terminal carrying the above-mentioned acquisition device 420. For another example, terminal 430 may be a terminal that communicates with acquisition device 420 or any one or more components in system 400 through network 450. In some embodiments, acquisition device 420 may be a part of terminal device 430.

[0073] In some embodiments, the terminal 430 may include a mobile device 430-1, a tablet computer 430-2, a laptop computer 430-3, etc. or any combination thereof. In some embodiments, the mobile device 430-1 may include a smart home device, a wearable device, a smart mobile device, a virtual reality device, an augmented reality device, etc. or any combination thereof. In some embodiments, the smart home device may include a smart lighting device, a smart appliance control device, a smart monitoring device, a smart TV, a smart camera, an intercom, etc. or any combination thereof. In some embodiments, the wearable device may include a smart bracelet, smart shoes and socks, smart glasses, a smart helmet, a smart watch, a smart headset, smart clothing, a smart backpack, a smart accessory, etc. or any combination thereof. In some embodiments, the smart mobile device may include a smart phone, a personal digital assistant (PDA), a gaming device, a navigation device, a point of sale (POS), etc. or any combination thereof. In some embodiments, the virtual reality device and / or the augmented reality device may include a virtual reality helmet, virtual reality glasses, virtual reality goggles, an enhanced virtual reality helmet, augmented reality glasses, augmented reality goggles, etc. or any combination thereof.

[0074] In some embodiments, the terminal 430 may acquire / receive data related to other components in the system 400. For example, the terminal 430 may acquire first sensor data related to at least two target finger joints of the user acquired by the acquisition device 420. For another example, the terminal 430 may acquire / receive a first joint angle of each target finger joint determined by the processing device 410 based on the first sensor data. For another example, the terminal device may be used to display information and / or data related to the system 400 (e.g., hand optical images, hand models, etc.) to the user and receive feedback from the user.

[0075] The storage device 440 may store data and / or information obtained from other devices or system components. In some embodiments, the storage device 440 may store data obtained from the acquisition device 420 or data processed by the processing device 410. For example, the storage device 440 may store the correspondence and / or mapping relationship between sensor data and joint angles. For another example, the storage device 440 may store the first sensor data collected by the acquisition device 420, and may also store the first joint angle of the target finger joint obtained after processing by the processing device 410. In some embodiments, the storage device 440 may also store the data and / or instructions used by the processing device 410 to execute or use to complete the exemplary method described in this specification. In some embodiments, the storage device 440 may include a large capacity memory, a removable memory, a volatile read-write memory, a read-only memory (ROM), etc., or any combination thereof. In some embodiments, the storage device 440 may be implemented on a cloud platform.

[0076] In some embodiments, storage device 440 may be connected to network 450 to communicate with one or more components (e.g., processing device 410, acquisition device 420, terminal 430) in system 400. One or more components in system 400 may access data or instructions stored in storage device 440 through network 450.

[0077] The network 450 can facilitate the exchange of information and / or data. In some embodiments, one or more components in the system 400 (e.g., the processing device 410, the acquisition device 420, the terminal 430, and the storage device 440) can send / receive information and / or data to / from other components in the system 400 through the network 450. For example, the processing device 410 can obtain the first sensor data or the deformation information of the movement at the finger joint from the acquisition device 420 or the storage device 440 through the network 450, and the terminal 430 can obtain any one or more of the first sensor data, the first joint angle of the target finger joint, etc. from the processing device 410 or the storage device 440 through the network 450. In some embodiments, the network 450 can be any form of wired or wireless network or any combination thereof. In some embodiments, the system 400 can include one or more network access points. For example, the system 400 can include a wired or wireless network access point, such as a base station and / or a wireless access point 450-1, 450-2, ..., through which one or more components of the system 400 can be connected to the network 450 to exchange data and / or information.

[0078] The above description is for illustrative purposes only, and actual application scenarios may have various changes. It should be noted that the injection system 400 is provided for illustrative purposes only and is not intended to limit the scope of the present application. For those of ordinary skill in the art, various modifications or changes can be made according to the description of this specification. However, these changes and modifications will not deviate from the scope of the present application.

[0079] Figure 5A and Figure 5B is an exemplary block diagram of processing devices 410A and 410B according to some embodiments of the present specification. Processing devices 410A and 410B may be Figure 4 The processing device 410 described. In some embodiments, the processing device 410A can be configured to determine the joint angle of the finger joint based on the sensor data and the mapping relationship. The processing device 410B can be configured to determine the mapping relationship between the sensor data and the finger joint angle. In some embodiments, the processing devices 410A and 410B can be implemented on the same or different devices.

[0080] like Figure 5A As shown, the processing device 410A may include an acquisition module 510 and a determination module 520 .

[0081] The acquisition module 510 can be used to acquire first sensor data related to at least two target finger joints of the user, wherein the first sensor data is collected using at least two strain sensors, each of which is arranged in the glove body worn by the user and located at one of the at least two target finger joints. The at least two target finger joints may include at least two adjacent metacarpophalangeal joints of the user. In some embodiments, the acquisition module 510 can acquire the first sensor data from a glove system or a storage device.

[0082] The acquisition module 510 can also be used to acquire a first mapping relationship, which reflects the relationship between the sensor data corresponding to the at least two target finger joints and the joint angles. The first mapping relationship can be represented by a mapping function, a machine learning model, etc. The first mapping relationship can be determined by the processing device 410A or the processing device 410B based on multiple data samples. For more information about generating the first mapping relationship, see Fig.19 and its related description.

[0083] The determination module 520 can be used to determine the first joint angle of each target finger joint based on the first sensor data and the first mapping relationship. In some embodiments, the determination module 520 can input the first sensor data into the first mapping relationship, and output the first joint angle of the target finger joint after processing the first mapping relationship.

[0084] like Figure 5B As shown, the processing device 410B may include an acquisition module 540 , a processing module 550 , and a generation module 560 .

[0085] The acquisition module 540 can be used to acquire multiple groups of sample sensor data, each group of sample sensor data is related to at least two sample finger joints of a sample user, and is collected at a target time using at least two sample strain sensors, each sample strain sensor is arranged in a sample glove body worn by the sample user and is located at one of the at least two sample finger joints, wherein the at least two sample finger joints include at least two adjacent metacarpophalangeal joints of the sample user.

[0086] The processing module 550 may be used to determine the sample joint angles of at least two sample finger joints of the corresponding sample user for each set of sample sensor data. In some embodiments, the processing module 550 may obtain the hand optical image of the sample user corresponding to each set of sample sensor data taken at the target time, and determine the sample joint angles of at least two sample finger joints of the corresponding sample user based on the hand optical image. For example, the processing module 550 may determine a hand model based on the hand optical image, and determine the sample joint angles based on the hand model.

[0087] The generation module 560 may be used to determine the correspondence between each group of sample sensor data and the sample joint angles corresponding thereto. In some embodiments, the generation module 560 may also be used to determine the mapping relationship (i.e., the first mapping relationship) between the sensor data corresponding to at least two sample finger joints and the joint angles based on the correspondence. For example, the generation module 560 may determine the mapping relationship based on the correspondence using a data fitting algorithm or a machine learning algorithm.

[0088] In some embodiments, processing device 410A and / or processing device 410B may share two or more modules, and any one of the modules may be divided into two or more units. For example, processing devices 410A and 410B may share the same acquisition module; that is, acquisition module 510 and acquisition module 540 are the same module. In some embodiments, processing device 410A and / or processing device 410B may include one or more additional modules, such as a storage module (not shown) for storing data. In some embodiments, processing device 410A and processing device 410B may be integrated into one processing device 410.

[0089] The methods disclosed in this specification may be stored in a storage device (e.g., a storage unit of the storage device 440 or the processing device 410) in the form of a program or instruction. Figure 5A When the modules shown in FIG. 5B execute programs or instructions, these methods can be implemented. In some embodiments, the methods disclosed in this specification may include one or more additional operations / steps, and / or omit one or more operations / steps. In addition, the order of the operations / steps in the method is not restrictive.

[0090] Figure 6 It is a flow chart of a method for determining finger joint angles according to some embodiments of this specification.

[0091] In some embodiments, method 600 may be performed by processing device 410A. Figure 6 As shown, method 600 may include:

[0092] In step 610 , the processing device 410A obtains first sensor data related to at least two target finger joints of the user.

[0093] The first sensor data can be used to characterize the posture and / or motion information of the target finger joint of the user, which can be collected by using at least two strain sensors, each of which is arranged in the glove body worn by the user and is located at one of the at least two target finger joints. Figure 1The glove system 100 (hereinafter referred to as the glove system) collects and obtains first sensor data. The glove system includes a plurality of strain sensors, and the plurality of strain sensors may include a first strain sensor and a second strain sensor arranged in the glove body 110. For more information about the glove system 100, the glove body 110, the first strain sensor and the second strain sensor, see Figure 1 and its description.

[0094] In some embodiments, the processing device 410A may be independent of the glove system, and may obtain the first sensor data from the glove system. For example, the processing device 410A may obtain the first sensor data from the glove system via a network. For another example, the processing device 410A may be the processor 140 on the glove system and connected to a plurality of strain sensors located on the glove body, thereby obtaining the first sensor data. In some embodiments, the processing device 410A may obtain the first sensor data from a storage device (such as the storage device 440).

[0095] Finger joints may be joints in the user's fingers (such as the thumb, index finger, middle finger, ring finger and pinky finger), such as interphalangeal joints (such as proximal interphalangeal joints, distal interphalangeal joints), metacarpophalangeal joints, etc. The target finger joint may be one or a combination of the above finger joints.

[0096] In some embodiments, at least two target finger joints include partial finger joints of the user, wherein the partial finger joints refer to partial finger joints that are easier to couple or have a greater coupling relationship.

[0097] If the movement or deformation (such as bending or swinging) of a finger joint causes the sensor data corresponding to another finger joint to change in a coordinated manner, there is a coupling relationship between the two finger joints. For example, when a finger joint bends, it affects the fabric of the adjacent finger joint, causing the fabric of the adjacent finger joint to deform, resulting in changes in the sensor data corresponding to the adjacent finger joint (for example, the sensor data corresponding to the swinging degree of freedom changes, or the swinging change further causes changes in the sensor data on the bending degree of freedom). The coupling relationship can include the coupling relationship between the finger joints of the same finger and the coupling relationship between the finger joints of different fingers (such as adjacent fingers). As an example only, when a finger (such as the index finger) bends, it may affect the bending of the adjacent finger (such as the middle finger), and the index finger and the middle finger and their finger joints have a linkage / coupling relationship.

[0098] In some embodiments, the strength of the coupling relationship between the finger joints can be measured by the degree of coupling between the finger joints. The degree of coupling can be expressed in the form of a numerical value (such as a numerical value in the range of [0-1]) or in other forms (such as a level). For example, when a slight deformation (such as bending, swinging, etc.) of a finger joint can trigger a linkage change in the sensor data of another finger joint, it indicates that the coupling degree between the two is high (having a strong coupling relationship).

[0099] In some embodiments, the coupling relationship between the finger joints may be pre-set. For example, a coupling relationship table may be pre-set, which may include the coupling relationship between different finger joints and their coupling degrees. The processing device 410A may determine the target finger joints based on the coupling relationship between the finger joints. For example, the processing device 410A may use some finger joints with a higher coupling degree (greater than a preset threshold) as target finger joints. In some embodiments of the present specification, some finger joints are determined based on the coupling relationship between different finger joints and their coupling degrees, and then, the processing device 410A may only obtain sensor data corresponding to some finger joints, so that the calculation process is more targeted and the amount of data in the calculation process is reduced.

[0100] In some embodiments, the at least two target finger joints include at least two adjacent metacarpophalangeal joints of the user. It is understandable that the coupling degree of adjacent metacarpophalangeal joints is relatively high. The processing device 410A may determine at least two adjacent metacarpophalangeal joints of the user as target finger joints. In some embodiments, the at least two target finger joints include all metacarpophalangeal joints of the user.

[0101] In some embodiments, the at least two metacarpophalangeal joints include the metacarpophalangeal joints of at least one of the user's index finger, middle finger, and ring finger and the two adjacent fingers on both sides thereof. For example, for the index finger, the two adjacent fingers on both sides thereof are the thumb and the middle finger, and the at least two metacarpophalangeal joints include the metacarpophalangeal joint of the index finger and the metacarpophalangeal joints of the thumb and the middle finger.

[0102] In some embodiments, the at least two metacarpophalangeal joints include the metacarpophalangeal joints of at least one of the user's thumb and pinky finger and an adjacent finger thereof. For example, for a thumb, an adjacent finger thereof is an index finger, and the at least two metacarpophalangeal joints include the metacarpophalangeal joint of the thumb and the metacarpophalangeal joint of the index finger.

[0103] In some embodiments, at least two target finger joints include at least two adjacent metacarpophalangeal joints of the user, and proximal interphalangeal joints corresponding to at least two metacarpophalangeal joints. For example, at least two target fingers include the metacarpophalangeal joints and proximal interphalangeal joints of the index finger and the middle finger. In some embodiments, at least two target finger joints include all metacarpophalangeal joints and all proximal interphalangeal joints of the user. When the metacarpophalangeal joints move or deform, it may also cause a linkage effect on the sensor data of the proximal interphalangeal joints. Therefore, the processing device 410A also considers the coupling relationship between the metacarpophalangeal joints and the proximal interphalangeal joints when determining the finger joint angles.

[0104] It should be noted that the interphalangeal joints include the proximal interphalangeal joints and the distal interphalangeal joints, but the distal interphalangeal joints are generally difficult to move completely independently (such as bending), and they are manifested as following the proximal interphalangeal joints. Therefore, the target finger joints may only include all the metacarpophalangeal joints and all the proximal interphalangeal joints of the user, but not the distal interphalangeal joints.

[0105] In some embodiments of the present specification, all metacarpophalangeal joints and all proximal interphalangeal joints are used as target finger joints, and sensors do not need to be arranged at distal interphalangeal joints, which simplifies the structure of the glove system and saves costs; at the same time, data screening and other processing are not required for sensor data corresponding to the target finger joints, thereby simplifying the processing steps. In addition, by using all metacarpophalangeal joints and all proximal interphalangeal joints as target finger joints, the processing process can take into account the coupling relationship between all metacarpophalangeal joints and all proximal interphalangeal joints, so that the first angle of the target finger joint is more accurate.

[0106] In some embodiments, the at least two strain sensors include at least two first strain sensors located at at least two metacarpophalangeal joints, and each first strain sensor is configured to measure deformation of its corresponding metacarpophalangeal joint in two degrees of freedom.

[0107] Degree of freedom refers to the dimension in which an object can move, which can be used to reflect the posture or deformation information of an object in a certain dimension. Degree of freedom may include but is not limited to bending and swinging. The degrees of freedom of different finger joints may be the same or different. More information about degrees of freedom can be found elsewhere in this manual (e.g. Figure 1 ).

[0108] The first strain sensor refers to a strain sensor arranged at the metacarpophalangeal joint, which is used to collect the deformation of the metacarpophalangeal joint in two degrees of freedom (such as bending and swinging). In some embodiments, the first strain sensor can be a multi-axis sensor, which can include various flexible bending sensors, such as inductive sensors, resistive sensors, capacitive sensors, optical fiber sensors, etc. As an example only, the first strain sensor includes multiple capacitor structures, and the capacitance of the multiple capacitor structures can change respectively with the changes of the metacarpophalangeal joint in each degree of freedom.

[0109] In some embodiments of the present specification, the first strain sensor can simultaneously realize the detection of motion information of user joints in multiple degrees of freedom (for example, three dimensions of X-axis bending, Z-axis bending, and Y-axis stretching), and its structure is simple and easy to manufacture; there is no need to set a sensitive unit (such as a capacitor structure) on its side (the side walls adjacent to the two surfaces in the thickness direction), which can reduce the overall size of the sensor (especially the size in the thickness direction), and can make the sensor thinner, so as to facilitate its application in smart wearable devices and improve the user's wearing experience; in addition, the electrical quantity of the capacitor structure is proportional to its area or parameters related to the affected area, and has high sensitivity and stability.

[0110] Figure 7 is an exemplary schematic diagram of the structure of a first strain sensor according to some embodiments of this specification, Figure 8 is an exemplary schematic diagram of the structure of another first strain sensor according to some embodiments of the present specification.

[0111] See also Figure 7 and Figure 8 , the first strain sensor 700 includes a flexible substrate 710, a first capacitor structure 720 and a second capacitor structure 730. Figure 7 As shown, the first capacitor structure 720 and the second capacitor structure 730 are distributed on both sides of the flexible substrate 710 along the thickness direction, and the projection of the first capacitor structure 720 on the flexible substrate 710 at least partially overlaps with the projection of the second capacitor structure 730 on the flexible substrate 710. In some embodiments, as Figure 8 As shown, the first capacitor structure 720 and the second capacitor structure 730 are distributed on the same side of the hand-flexible substrate 710 along the thickness direction.

[0112] The flexible substrate 710 has a flexible property and is easily deformed (such as bending) when subjected to an external force. In some embodiments, in order to facilitate fitting with human joints, the flexible substrate 710 may have a flat structure. In this case, the flexible substrate 710 has a thickness direction, which may be as follows: Figure 7 The Z-axis direction is shown.

[0113] The capacitor structure (including the first capacitor structure 720 and the second capacitor structure 730) is a structure for measuring the bending deformation of the flexible substrate 710. Specifically, the capacitor structure has a corresponding area, and the electrical quantity of the capacitor structure (for example, capacitance) is proportional to its area (or a dimensional parameter related to the area, such as the length, width of the capacitor structure, or the bending angle of the first strain sensor, etc.), that is, the electrical quantity of the capacitor structure in different directions is proportional to the deformation of the first strain sensor in different degrees of freedom, thereby reflecting the deformation (such as bending angle, bending direction). It should be understood that the first strain sensor may not be limited to a capacitor structure, for example, it may also be a resistor structure or a capacitor-resistor composite structure. In some embodiments, the first capacitor structure 720 and the second capacitor structure 730 may include a structure arranged in the thickness direction of the flexible substrate 710 (such as Figure 7 or Figure 8 The multilayer structure on the same side surface of the flexible substrate 710 (Z-axis direction shown in the figure), each layer of each multilayer structure is stacked along the thickness direction. Wherein, the multilayer structure is a structure formed by stacking layer structures. The layers of each multilayer structure stacked along the thickness direction can be understood as: each layer in the multiple layer structures of the first capacitor structure 720 and the second capacitor structure 730 is arranged along the thickness direction of the flexible substrate 710 and stacked together. In some embodiments, the multilayer structure includes a second conductive layer, an intermediate layer and a first conductive layer. In some embodiments, the multilayer structure includes a second conductive layer, a second intermediate layer, a third conductive layer, a first intermediate layer and a first conductive layer. The specific layered structure included in the multilayer structure can be found below.

[0114] When different capacitor structures are arranged at different positions of the first strain sensor 700, these capacitor structures will produce differentiated responses to the deformation of different dimensions of the flexible substrate 710. When the flexible substrate 710 is deformed in a certain dimension, the detection parameters generated by these capacitor structures have characteristics corresponding to the deformation of this dimension; and when the flexible substrate 710 is deformed in another dimension, the detection parameters generated by these capacitor structures will have corresponding characteristics. It can be understood that the characteristics of the detection parameters generated by these capacitor structures will correspond to the deformation dimensions of the flexible substrate 710 one by one, and on this basis, the deformation of the flexible substrate 710 can be identified according to the detection parameters of each capacitor structure. In some embodiments, the processor (e.g., processor 140 or external computing device (e.g., processor 410A)) can determine the deformation of the flexible substrate 710 in at least two dimensions through a specific algorithm based on the parameters related to capacitance of each capacitor structure (e.g., first capacitor structure 720, second capacitor structure 730). For example, the deformation of the flexible substrate 710 in at least two dimensions is determined by a machine learning model, a mapping relationship or a functional relationship. The description of deformation in at least two dimensions can be found in the following.

[0115] When the first strain sensor 700 (flexible substrate 710) is bent and deformed, the physical shape of at least part of the capacitor structure (e.g., the first capacitor structure 720 and the second capacitor structure 730) located on the flexible substrate 710 will change accordingly, thereby causing the corresponding capacitance to change. For example, the bending of the first strain sensor 700 will cause the area of ​​the capacitor structure to change, thereby changing the capacitance of the capacitor structure. By collecting the capacitance of the capacitor structure or parameters related to the capacitance, and analyzing the changes in the capacitance of the capacitor structure or parameters related to the capacitance, the bending condition (such as bending angle, bending direction) of the first strain sensor 700 can be sensed more accurately.

[0116] like Figure 7 As shown, in some embodiments, the first capacitor structure 720 and the second capacitor structure 730 are respectively distributed on both sides of the flexible substrate 710 in the thickness direction. In some embodiments, the first capacitor structure 720 and the second capacitor structure 730 are symmetrically arranged along the flexible substrate 710 in the thickness direction, and the projection of the first capacitor structure 720 on the flexible substrate 710 and the projection of the second capacitor structure 730 on the flexible substrate 710 completely overlap. When the first capacitor structure 720 and the second capacitor structure 730 are symmetrically arranged about the flexible substrate 710 in the thickness direction, the two capacitor structures have the same response to some external interference (for example, the first strain sensor is stretched or compressed as a whole along a certain direction). At this time, according to the parameters related to the capacitance of the capacitor structure, the influence of the external interference on the first strain sensor 700 can be eliminated through the differential processing algorithm, thereby improving the sensitivity of determining the deformation of the flexible substrate 710 in at least two degrees of freedom.

[0117] In some embodiments, the flexible substrate 710 includes a short axis direction and a long axis direction perpendicular to the thickness direction. Figure 7 The deformations in multiple degrees of freedom generated by the first strain sensor 700 (flexible substrate 710) shown in the figure include at least bending deformation around an axis parallel to the short axis direction and stretching or compressing deformation along the long axis direction. Figure 7 The X-axis direction is shown; the long axis direction can be Figure 7 The Y-axis direction is shown.

[0118] like Figure 7As shown, when the first strain sensor 700 is bent and deformed around an axis parallel to the short axis direction (i.e., the X-axis direction), the first capacitor structure 720 and the second capacitor structure 730 will undergo tensile deformation and compressive deformation respectively. For example, the first capacitor structure 720 will be bent and elongated, and the second capacitor structure 730 will be bent and compressed. At this time, the capacitances of the first capacitor structure 720 and the second capacitor structure 730 will undergo opposite changes. In this case, the difference in capacitance between the first capacitor structure 720 and the second capacitor structure 730 can reflect the bending direction and degree of bending of the first strain sensor 700 around the axis parallel to the X-axis direction, and since the first capacitor structure 720 and the second capacitor structure 730 are stretched and compressed to substantially the same extent, the sum of their capacitances can remain substantially unchanged.

[0119] When the first strain sensor 700 is stretched or compressed along the long axis direction, the first capacitor structure 720 and the second capacitor structure 730 are stretched or compressed synchronously, and the capacitance of the first capacitor structure 720 and the second capacitor structure 730 changes synchronously. In this case, the difference between the capacitance of the first capacitor structure 720 and the second capacitor structure 730 remains substantially unchanged (or close to 0), and the sum of the two can reflect the degree of stretching or compression of the first strain sensor 700 along the long axis direction.

[0120] In some embodiments, by comparing the characteristics of the capacitance of the first capacitor structure 720 and the second capacitor structure 730 under the above two degrees of freedom, for example, the relationship between the sum (difference) of the capacitance and the deformation of the flexible substrate 710, the mutual interference between the deformations of the two degrees of freedom can be avoided, and the deformation of the flexible substrate 710 in the two degrees of freedom can be effectively distinguished. For example, when the first strain sensor 700 is stretched or compressed as a whole, the first capacitor structure 720 and the second capacitor structure 730 will be stretched or compressed synchronously, and the synchronous stretching or compression of the first capacitor structure 720 and the second capacitor structure 730 is regarded as common mode interference. At this time, differential processing of the signal (such as capacitance) can eliminate this common mode interference, so that the first strain sensor 700 is insensitive to its own stretching or compression deformation, and is only sensitive to the bending deformation around the axis parallel to the short axis direction (i.e., the X-axis direction), so that the first strain sensor 700 can accurately detect the bending deformation of this degree of freedom. That is, the difference between the capacitance of the first capacitor structure 720 and the second capacitor structure 730 mainly reflects the bending deformation of the flexible substrate 710 around an axis parallel to the short axis direction (i.e., the X-axis direction). Similarly, the sum of the capacitance of the first capacitor structure 720 and the second capacitor structure 730 mainly reflects the tensile or compressive deformation of the flexible substrate 710 along the long axis direction. More about the detection principle of the tensile or compressive deformation in the long axis direction is similar to the detection principle of the first strain sensor 700 having four capacitor structures. For details, please refer to FIG. 10A to FIG. 12 and its related description.

[0121] like Figure 8 As shown, in some embodiments, the flexible substrate 710 includes a long axis direction perpendicular to the thickness direction, the first capacitor structure 720 and the second capacitor structure 730 are distributed on the same side of the flexible substrate 710 in the thickness direction, and the first capacitor structure 720 and the second capacitor structure 730 are arranged side by side and both extend along the long axis direction of the flexible substrate 710.

[0122] In some embodiments, the first capacitor structure 720 and the second capacitor structure 730 are symmetrically arranged along a mid-section parallel to a plane formed by the thickness direction and the long axis direction, and the mid-section is located on the flexible substrate 710. When the first capacitor structure 720 and the second capacitor structure 730 are symmetrically arranged about the flexible substrate 710 on the mid-section, the two capacitor structures have the same response to partial external interference (for example, the first strain sensor 700 is stretched or compressed as a whole along a certain direction). At this time, the processor, for example, the processor 140 or an external computing device (for example, the processor 410A)) can eliminate the influence of external interference on the first strain sensor 700 through a differential processing algorithm based on the parameters related to the capacitance of the capacitor structure (for example, the first capacitor structure 720 and the second capacitor structure 730), thereby improving the sensitivity of determining the deformation of at least two degrees of freedom of the flexible substrate 710. For more information about the mid-section, see Fig.9A and Fig. 9B and its related description.

[0123] In some embodiments, Figure 8 The degree of freedom deformation generated by the first strain sensor 700 (flexible substrate 710) shown includes at least bending deformation around an axis parallel to the thickness direction and tensile or compressive deformation along the long axis direction.

[0124] like Figure 8 As shown, when the first strain sensor 700 is bent and deformed along an axis parallel to the thickness direction (i.e., the Z-axis direction), the first capacitor structure 720 and the second capacitor structure 730 will undergo tensile deformation and compressive deformation respectively. For example, the first capacitor structure 720 will be bent and elongated, and the second capacitor structure 730 will be bent and compressed. At this time, the capacitances of the first capacitor structure 720 and the second capacitor structure 730 will undergo opposite changes. In this case, the difference in capacitance between the first capacitor structure 720 and the second capacitor structure 730 can reflect the bending direction and degree of bending of the first strain sensor 700 around the axis parallel to the Z-axis direction, and since the first capacitor structure 720 and the second capacitor structure 730 are stretched and compressed to substantially the same extent, the sum of their capacitances can remain substantially unchanged.

[0125] when Figure 8When the first strain sensor 700 is stretched or compressed along the long axis direction, the first capacitor structure 720 and the second capacitor structure 730 are stretched or compressed synchronously, and the capacitance of the first capacitor structure 720 and the second capacitor structure 730 changes synchronously. In this case, the difference between the capacitance of the first capacitor structure 720 and the second capacitor structure 730 remains substantially unchanged (or close to 0), and the sum of the two can reflect the degree of stretching or compression of the first strain sensor 700 along the long axis direction.

[0126] In some embodiments, by comparing the characteristics of the capacitance of the first capacitor structure 720 and the second capacitor structure 730 under the above two degrees of freedom, for example, the relationship between the sum (difference) of the capacitance and the deformation of the flexible substrate 710, the mutual interference between the deformations of the two degrees of freedom can be avoided, and the deformation of the flexible substrate 710 in the two degrees of freedom can be effectively distinguished. For example, when the first strain sensor 700 is stretched or compressed as a whole, the first capacitor structure 720 and the second capacitor structure 730 will be stretched or compressed synchronously, and the synchronous stretching or compression of the first capacitor structure 720 and the second capacitor structure 730 is regarded as common mode interference. At this time, differential processing of the signal (such as capacitance) can eliminate this common mode interference, so that the first strain sensor 700 is insensitive to its own stretching or compression deformation, and is only sensitive to the bending deformation along the axis parallel to the thickness direction (i.e., the Z-axis direction), so that the first strain sensor 700 can accurately detect the bending deformation of this degree of freedom. That is, the difference between the capacitance of the first capacitor structure 720 and the second capacitor structure 730 mainly reflects the bending deformation of the flexible substrate 710 around an axis parallel to the thickness direction (i.e., the Z-axis direction). Similarly, the sum of the capacitance of the first capacitor structure 720 and the second capacitor structure 730 mainly reflects the tensile or compressive deformation of the flexible substrate 710 along the long axis direction. More about the detection principle of the tensile or compressive deformation in the long axis direction is similar to the detection principle of the first strain sensor 700 having four capacitor structures. For details, please refer to FIG. 10A to FIG. 12 and its related description.

[0127] In some embodiments, the first sensor data acquired by the processing device 410A may include raw data collected by at least two strain sensors. As an example only, when the strain sensor includes a capacitor structure, the raw data may include a capacitance value of the capacitor structure; when the strain sensor includes an inductor structure, the raw data may include an inductor value of the inductor structure. Directly using the raw data as the first sensor data can simplify the data processing process.

[0128] In some embodiments, the processing device 410A can determine the decoupling data of at least two target finger joints based on the raw data. The first sensor data acquired by the processing device 410A may include the decoupling data. The decoupling data may include parameters that can reflect the deformation of the first strain sensor in different dimensions. For example, the raw data can be decoupled into some parameters (for example, Ctotal, dCx, dCz, etc. obtained by the differential processing algorithm below), which can reflect the bending deformation of the strain sensor around an axis parallel to the thickness direction, the bending deformation around an axis parallel to the short axis direction, and the tensile or compressive deformation along the long axis direction, thereby restoring the actual bending movement of the sensor. For more information on data decoupling, see Figure 9A-14 and its description.

[0129] Fig.9A is a schematic diagram of the distribution of four capacitor structures on a flexible substrate according to some embodiments of this specification; Fig. 9B yes Fig.9A The cross-sectional structure diagram of the four capacitor structures shown is along the Y-axis direction.

[0130] like Fig.9A and Fig. 9B As shown, in some embodiments, compared to Figure 8 The first strain sensor 700 further includes a third capacitor structure 740 and a fourth capacitor structure 750. The first capacitor structure 720 and the second capacitor structure 730 are arranged side by side on one side surface of the flexible substrate 710 along the thickness direction, and both extend along the long axis direction; the third capacitor structure 740 and the fourth capacitor structure 750 are arranged side by side on the other side surface of the flexible substrate 710 along the thickness direction, and both extend along the long axis direction.

[0131] In some embodiments, the processor reads capacitance-related parameters of the first capacitor structure 720 , the second capacitor structure 730 , the third capacitor structure 740 , and the fourth capacitor structure 750 , respectively, and determines deformation of the flexible substrate 710 in at least two degrees of freedom based on the parameters.

[0132] In some embodiments, Fig.9A The degree of freedom deformation generated by the first strain sensor 700 (flexible substrate 710) shown in the figure includes at least bending deformation around an axis parallel to the thickness direction, bending deformation around an axis parallel to the short axis direction, and tensile or compressive deformation along the long axis direction of the flexible substrate 710. For the purpose of illustration, the following is combined with Fig. 10A , Fig. 10B , Fig.11 and Fig.12 The capacitance-related parameters of each capacitor structure under deformation with different degrees of freedom are described respectively.

[0133] In some embodiments, the parameters related to capacitance include: C1, C2, C3, C4, Ctotal, dCx, dCz; and Ctotal = C1 + C2 + C3 + C4, dCx after composite differential operation = (C1 + C2) - (C3 + C4), dCz after composite differential operation = (C1 + C3) - (C2 + C4). Wherein, C1, C2, C3, C4 represent the capacitance of the first capacitor structure 720, the second capacitor structure 730, the third capacitor structure 740 and the fourth capacitor structure 750 respectively.

[0134] In some embodiments, the first capacitor structure 720 and the second capacitor structure 730 are symmetrically arranged about the first middle section S1, and the third capacitor structure 740 and the fourth capacitor structure 750 are symmetrically arranged about the first middle section S1. The first middle section S1 represents: a section parallel to the plane formed by the thickness direction and the long axis direction in the flexible substrate 710; the first capacitor structure 720 and the third capacitor structure 740 are symmetrically arranged about the second middle section S2 of the flexible substrate 710, and the second capacitor structure 730 and the fourth capacitor structure 750 are symmetrically arranged about the second middle section S2. The second middle section S2 represents: a mid-plane parallel to the plane formed by the long axis direction and the short axis direction in the flexible substrate 710.

[0135] By setting four symmetrical capacitor structures on the first strain sensor 700 and performing a composite differential operation, common mode interference can be effectively eliminated in the process of identifying the bending deformation of the sensor. As an example only, when the first strain sensor 700 is stretched or compressed as a whole, the first capacitor structure 720, the second capacitor structure 730, the third capacitor structure 740 and the fourth capacitor structure 750 will be stretched or compressed synchronously, and the stretching or compression of the four capacitor structures is regarded as common mode interference. The dCx and dCz obtained by setting four capacitor structures and performing composite differential processing on the signal remain unchanged, effectively eliminating such common mode interference, so that the flexible first strain sensor 700 is insensitive to its own stretching or compression deformation, and is only sensitive to its own bending deformation, thereby increasing the accuracy of the first strain sensor 700.

[0136] Fig. 10A is a schematic diagram of a second sensor without deformation according to some embodiments of this specification; Fig. 10B yes Fig. 10A The diagram shows the deformation of the second sensor after being stretched or compressed along the long axis direction.

[0137] In some embodiments, when the first capacitor structure 720 , the second capacitor structure 730 , the third capacitor structure 740 and the fourth capacitor structure 750 are all capacitor structures, the first strain sensor 700 can identify tensile or compressive deformation along the long axis direction of the flexible substrate 11 according to parameters related to capacitance.

[0138] In some embodiments, Fig. 10A When the first strain sensor 700 shown is not deformed, the capacitance of the four capacitor structures can be obtained by the following formula (1). For the purpose of illustration only, the four capacitor structures of the first strain sensor 700 are considered to be completely consistent and symmetrical about the flexible substrate 710 (i.e., the first middle section S1 and the second middle section S2). Among them, ε 0 represents the dielectric constant of vacuum; ε represents the relative dielectric constant of each of the four capacitor structures (i.e., the middle layer, see below); d represents the thickness of the dielectric layer of each of the four capacitor structures; A represents the area of ​​each of the four capacitor structures on the plane formed by the long axis direction and the short axis direction; L0 represents the initial length of each of the four capacitor structures in the long axis direction of the flexible substrate 710; w represents the width of each of the four capacitor structures in the short axis direction of the flexible substrate 710.

[0139] In some embodiments, Fig. 10B As shown, when the first strain sensor 700 is stretched or compressed along the long axis direction (i.e., the Y axis direction), the initial length L0 of each of the four capacitor structures is stretched or compressed to Lx. Considering that the thickness of each of the four capacitor structures changes very little, for the convenience of explanation, it is approximately assumed that the thickness of each of the four capacitor structures remains unchanged. According to formula (1), the capacitance change of the four capacitor structures can be obtained as follows: formula (2).

[0140] At this time, Ctotal, dCx, and dCz are determined by the following formulas (3), (4), and (5), respectively. dCx= (C1+C2)-(C3+C4)=0 (4) dCz= (C1+C3)-(C2+C4)=0 (5)

[0141] When the first strain sensor 700 is stretched or compressed along the long axis, the four capacitor structures will be stretched or compressed synchronously, and the capacitance of the four capacitor structures will change synchronously. In this case, the sum of the capacitances of the four capacitor structures can reflect the degree of stretching or compression of the first strain sensor 700 (flexible substrate 710) along the long axis. According to formulas (2) and (3), Ctotal is only related to the length Lx after the capacitor structure changes, and Lx can be used to represent the stretching or compression deformation of the first strain sensor 700 along the long axis, so Ctotal can be used to reflect the stretching or compression deformation of the first strain sensor 700 (flexible substrate 710) along the long axis.

[0142] According to formula (4), dCx reflects the difference between the sum of the capacitance C1 of the first capacitance structure 720 and the capacitance C2 of the second capacitance structure 730 and the sum of the capacitance C3 of the third capacitance structure 740 and the capacitance C4 of the fourth capacitance structure 750. Fig. 10B From the structural distribution of the four capacitor structures, it can be seen that the sum of the capacitance C1 of the first capacitor structure 720 and the capacitance C2 of the second capacitor structure 730, that is, C1+C2, can reflect the bending deformation of the upper side of the flexible substrate 710 in the thickness direction; the sum of the capacitance C3 of the third capacitor structure 740 and the capacitance C4 of the fourth capacitor structure 750, that is, C3+C4, can reflect the bending deformation of the lower side of the flexible substrate 710 in the thickness direction. Therefore, dCx is used to reflect the difference in bending deformation generated by the upper and lower sides (i.e., the upper and lower sides along the thickness direction) of the flexible substrate 710 when it is deformed, which is related to the bending deformation around the axis parallel to the short axis direction (i.e., the X-axis direction), while the deformation of stretching or compression along the long axis direction (i.e., the Y-axis direction) does not cause the deformation difference between the upper and lower sides of the flexible substrate 710, so dCx is a constant of 0.

[0143] Similarly, according to formula (5), dCz is used to reflect the difference in bending deformation of the two side surfaces along the short axis direction of the flexible substrate 710 when the flexible substrate 710 is deformed. dCz is only related to the bending deformation around the axis parallel to the thickness direction (i.e., the Z-axis direction), and the deformation caused by stretching or compressing along the long axis direction (i.e., the Y-axis direction) will not cause the deformation difference of the two side surfaces of the flexible substrate 710 along the short axis direction, so dCz is also a constant 0.

[0144] In some embodiments, the output parameter Ctotal of the first strain sensor 700 stretched or compressed along the Y axis changes linearly in proportion to the stretched length or compressed length Lx, and has extremely high linearity, while dCz and dCx are both 0, and are not affected by the stretched length or compressed length Lx. Therefore, the first strain sensor 700 can identify the tensile or compressive deformation along the long axis direction of the flexible substrate 710 according to the parameter Ctotal related to capacitance.

[0145] Fig.11 yes Fig. 10A The schematic side view of the first strain sensor shown is a schematic diagram of a first strain sensor after being bent and deformed around an axis parallel to the short axis direction.

[0146] In some embodiments, when the first capacitor structure 720, the second capacitor structure 730, the third capacitor structure 740 and the fourth capacitor structure 750 are all capacitor structures, the first strain sensor 700 can identify the bending deformation along the flexible substrate 710 around an axis parallel to the short axis direction according to the parameter dCx related to the capacitance.

[0147] like Fig. 10A and Fig.11As shown, when the first strain sensor 700 is bent and deformed along an axis parallel to the short axis direction (i.e., the X-axis direction), for the convenience of explanation, it is approximately considered that the first strain sensor 700 after bending and deformation is a circular arc, the first capacitor structure 720 and the second capacitor structure 730 are bent and elongated to L1, and the third capacitor structure 740 and the fourth capacitor structure 750 are bent and compressed to L3. Considering that the thickness of each of the four capacitor structures changes very little, for the convenience of explanation, it is approximately considered that the thickness of each of the four capacitor structures remains unchanged. According to formula (1), it can be known that the capacitance C1 of the first capacitor structure 720 and the capacitance C2 of the second capacitor structure 730 change to the following formula (6), and the capacitance C3 of the third capacitor structure 740 and the capacitance C4 of the fourth capacitor structure 750 change to the following formula (7).

[0148] At this time, Ctotal, dCx, and dCz are determined by the following formulas (8), (9), and (10), respectively. dCz= (C1+C3)-(C2+C4)=0 (10) Wherein, R0 represents the radius of the arc of the bending deformation of the flexible substrate 710; R1 represents the radius of the arc of the bending deformation of the first capacitor structure 720 and the second capacitor structure 730; R2 represents the radius of the arc of the bending deformation of the third capacitor structure 740 and the fourth capacitor structure 750; t represents Fig. 10A The thickness of the first strain sensor 700 is shown; L1 represents the length of the arc of the first capacitor structure 720 and the second capacitor structure 730 bending and deforming; L3 represents the length of the arc of the third capacitor structure 740 and the fourth capacitor structure 750 bending and deforming.

[0149] When the first strain sensor 700 is bent and deformed along an axis parallel to the short axis direction, for example, the first capacitor structure 720 and the second capacitor structure 730 are bent and stretched, and the third capacitor structure 740 and the fourth capacitor structure 750 are bent and compressed. At this time, the sum of the capacitance C1 of the first capacitor structure 720 and the capacitance C2 of the second capacitor structure 730, i.e., C1+C2, and the sum of the capacitance C3 of the third capacitor structure 740 and the capacitance C4 of the fourth capacitor structure 750, i.e., C3+C4, have an opposite change relationship. At this time, Ctotal obtained by adding the capacitances of the four capacitor structures can offset the change trends of C1+C2 and C3+C4, that is, Ctotal is the constant shown in formula (8) at this time, and Ctotal cannot be used to reflect the bending deformation of the first strain sensor 700 along the axis parallel to the short axis direction.

[0150] In some embodiments, the output parameter dCx of the first strain sensor 700 after bending deformation around an axis parallel to the short axis direction (i.e., the X-axis direction) varies linearly in direct proportion to the bending angle β, and has extremely high linearity, while dCz and Ctotal are both constants and are not affected by the bending angle β. Therefore, the first strain sensor 700 can identify bending deformation along an axis parallel to the short axis direction based on the parameter dCx related to capacitance.

[0151] It should be noted that, in addition to the case of the four capacitor structures described above, the first strain sensor 700 may have three or more capacitor structures distributed on the two side surfaces of the flexible substrate 710 along the thickness direction, and all extend along the long axis direction. As long as the deformations of different dimensions can be characterized by the capacitance-related parameters of these capacitor structures, deformations of multiple dimensions can be identified. As an example only, there are three capacitor structures on the first strain sensor 700, two of which are distributed on the upper side of the flexible substrate, and the other is distributed on the lower side of the flexible substrate. At this time, the capacitance-related parameters of the three capacitor structures are also related to the bending deformation around the X-axis (or Z-axis). At this time, the bending deformation of the first strain sensor 700 around the X-axis (or Z-axis) can also be calculated by combining the capacitance-related parameters generated by the three sensor structures.

[0152] Fig.12 yes Fig. 10A The schematic top view of the first strain sensor shown is after being bent and deformed around an axis parallel to the thickness direction.

[0153] In some embodiments, when the first capacitor structure 720, the second capacitor structure 730, the third capacitor structure 740 and the fourth capacitor structure 750 are all capacitor structures, the first strain sensor 700 can identify the bending deformation along the flexible substrate 710 around an axis parallel to the thickness direction according to the parameter dCz related to the capacitance.

[0154] like Fig. 10A and Fig.12 As shown, when the first strain sensor 700 is bent and deformed along an axis parallel to the thickness direction (i.e., the Z-axis direction), for the convenience of description, it is approximately considered that the first strain sensor 700 after bending and deformation can be an arc, and the first capacitor structure 720 and the third capacitor structure 740 ( Fig.12 ) is bent and stretched to L4, the second capacitor structure 730 and the fourth capacitor structure 750 ( Fig.12(not shown) is bent and compressed to L2. Considering that the thickness of each of the four capacitor structures changes very little, for the convenience of explanation, it is approximately assumed that the thickness of each of the four capacitor structures remains unchanged. According to formula (1), the capacitance C1 of the first capacitor structure 720 and the capacitance C3 of the third capacitor structure 740 are expressed by the following formula (11), and the capacitance C2 of the second capacitor structure 730 and the capacitance C4 of the fourth capacitor structure 750 change to the following formula (12).

[0155] At this time, Ctotal, dCx, and dCz are determined by the following formulas (13), (14), and (15), respectively. dCx= (C1+C2)-(C3+C4)=0 (14) Among them, r0 represents the radius of the arc of the bending deformation of the flexible substrate 710; r1 represents the radius of the arc of the bending deformation of the first capacitor structure 720 and the third capacitor structure 740; r2 represents the radius of the arc of the bending deformation of the second capacitor structure 730 and the fourth capacitor structure 750; L2 represents the length of the arc of the bending deformation of the first capacitor structure 720 and the third capacitor structure 740; L4 represents the length of the arc of the bending deformation of the second capacitor structure 730 and the fourth capacitor structure 750.

[0156] In some embodiments, the output parameter dCz of the first strain sensor 700 after bending deformation around an axis parallel to the thickness direction (i.e., the Z-axis direction) varies linearly in proportion to the bending angle α, and has extremely high linearity, while dCx and Ctotal are both constants and are not affected by the bending angle α. Therefore, the first strain sensor 700 can identify bending deformation along an axis parallel to the thickness direction based on the parameter dCz related to capacitance.

[0157] In some embodiments, when the first capacitor structure 720, the second capacitor structure 730, the third capacitor structure 740 and the fourth capacitor structure 750 are all resistor structures, the processor can determine the deformation of the flexible substrate 710 in at least two dimensions according to the parameter resistance related to the four capacitor structures. For example, the resistances of the four capacitor structures are: Substituting the area of ​​each capacitor structure after the deformation of the first strain sensor 700 into it and performing a differential operation, a result similar to the parameters related to the capacitance can be obtained, so that the processor can determine the bending deformation of the first strain sensor 700 around an axis parallel to the thickness direction, the bending deformation around an axis parallel to the short axis direction, and the tensile or compressive deformation along the long axis direction of the flexible substrate. Wherein, p represents the resistivity of each of the four capacitor structures (i.e., the middle layer, see below).

[0158] In some embodiments, the processor can identify a composite deformation in which deformations of any dimension coexist among the bending deformation of the first strain sensor 700 about an axis parallel to the thickness direction, the bending deformation about an axis parallel to the short axis direction, and the tensile or compressive deformation along the long axis direction of the flexible substrate 710. For example, the processor can simultaneously calculate the changes in the parameters Ctotal, dCx, and dCz related to the capacitance, thereby simultaneously restoring the components of the deformations of different dimensions to restore the actual bending deformation of the first strain sensor 700.

[0159] Fig.13 It is a schematic diagram of wrinkles on a flexible sensor according to some embodiments of the present specification.

[0160] The flexible sensor may be a first strain sensor, such as Fig.13 As shown in Figure 2, when the sensor has multiple dimensions of composite deformation at the same time, the sensor will appear wrinkled ( Fig.13 It shows that wrinkles appear in the length direction L), which is equivalent to multiple bends (bending angles may include α1, α2, and α3). After geometric integration, it can be seen that α1, α2, and α3 do not need to be considered, and only the angle between the head and tail of the sensor needs to be considered. In other words, no matter how many times the sensor is bent in the middle, the length change of the upper and lower surfaces of the sensor along the thickness direction L = L0 + αx / 2 and L down = L0-αx / 2 still hold, and the length of the neutral line L0 remains unchanged, where α is the angle between the head and tail of sensor 1, and x represents the width (thickness) of sensor 1 in the bending direction, indicating that the four sensing structures of sensor 1 can effectively prevent the influence of wrinkles.

[0161] Fig.14 is based on Fig.13 The graph shown is a schematic diagram of the curve obtained by actual measurement of the sensor.

[0162] like Fig.14 As shown in the figure, when the sensor is bent around an axis parallel to the short axis direction (i.e., the X-axis direction), dCx will change significantly with the bending angle, while Ctotal and dCz remain basically stable; when the sensor is bent around an axis parallel to the thickness direction (i.e., the Z-axis direction), dCz will change significantly with the bending angle, while Ctotal and dCx remain basically stable; when the sensor is stretched or compressed along the long axis direction (i.e., the Y-axis direction) of the flexible substrate, Ctotal will change significantly with the stretched length or compressed length, while dCx and dCz remain basically stable. Therefore, when the sensor is bent, the output parameters Ctotal, dCx, and dCz related to capacitance in the sensor maintain a high degree of independence and stability, so that the processing equipment can identify deformations of different dimensions based on Ctotal, dCx, and dCz respectively.

[0163] In some embodiments, when the sensor is freely bending (i.e., the sensor has complex deformation), dCx, dCz and Ctotal will all change. At this time, the free bending motion of the sensor can be decoupled into bending deformation around an axis parallel to the thickness direction, bending deformation around an axis parallel to the short axis direction, and tensile or compressive deformation along the long axis direction, thereby restoring the actual bending motion of the sensor.

[0164] By decoupling the original data to obtain decoupled data, the composite deformation of the first strain sensor can be decoupled into separate deformations of each degree of freedom, and the decoupled data corresponding to each degree of freedom can maintain a high degree of independence and stability, thereby improving the accuracy and sensitivity of the deformations acquired in at least two degrees of freedom.

[0165] Therefore, the embodiments of the present specification can measure deformations in multiple degrees of freedom (for example, including bending around the X-axis, bending around the Z-axis, and stretching along the Y-axis) through a single sensor, thereby realizing detection of posture data of fingers in multiple dimensions, which is beneficial for simplifying the preparation process and miniaturization design while improving detection accuracy.

[0166] Fig.15 is a schematic cross-sectional view of a first strain sensor according to some embodiments of the present specification.

[0167] like Fig.15 As shown, in some embodiments, each capacitor structure includes a second conductive layer, an intermediate layer, and a first conductive layer stacked sequentially away from the flexible substrate 710 in the thickness direction. For example, the first capacitor structure 720 includes a second conductive layer 722, an intermediate layer 72-53, and a first conductive layer 721 stacked sequentially away from the flexible substrate 710; the second capacitor structure 730 includes a second conductive layer 732, an intermediate layer 72-53, and a first conductive layer 731 stacked sequentially away from the flexible substrate 710; the third capacitor structure 740 includes a second conductive layer 742, an intermediate layer 74-55, and a first conductive layer 741 stacked sequentially away from the flexible substrate 710; the fourth capacitor structure 750 includes a second conductive layer 752, an intermediate layer 74-55, and a first conductive layer 751 stacked sequentially away from the flexible substrate 710. In some embodiments, the intermediate layer is between the first conductive layer and the second conductive layer.

[0168] In some embodiments, the first conductive layer and the second conductive layer of each capacitor structure include an elastic conductive material, which can make the first conductive layer and the second conductive layer conductive and can return to their original shape when the external force disappears. In some embodiments, the elastic conductive material includes but is not limited to conductive film, conductive ink, conductive polymer material, conductive gel, liquid metal, etc. In some embodiments, the first conductive layer and the second conductive layer include conductive film, which is made by mixing conductive particles with polymer materials. In some embodiments, polymer materials include but are not limited to silicone, rubber or resin, etc. In some embodiments, the first conductive layer and the second conductive layer include conductive ink, which is made by mixing conductive particles with ink materials, and the conductive ink can be prepared by printing to obtain a conductive pattern. In some embodiments, the first conductive layer and the second conductive layer include conductive polymer materials, conductive gel, liquid metal, etc. In some embodiments, conductive polymer materials include but are not limited to polypyrrole, poly (3,4-ethylenedioxythiophene) - polystyrene sulfonate (PEDOT:PSS, Poly (3,4-ethylenedioxythiophene) polystyrene sulfonate), etc.

[0169] In some embodiments, the elastic conductive material includes an elastic material filled with conductive particles, and the conductivity of the first conductive layer and the second conductive layer can be adjusted by filling the conductive particles in the elastic material. In some embodiments, the elastic material includes silicone, rubber, resin, polydimethylsiloxane (PDMS), polyurethane, styrene-butadiene-styrene (SBS), etc. In some embodiments, the conductive particles include metal powder and carbon powder, carbon nanotubes, silver nanowires, carbon black, graphite powder, graphene, etc.

[0170] In some embodiments, the middle layer includes an elastic insulating material, and each capacitor structure is a capacitor structure. For example, the elastic insulating material includes silicone, rubber, polydimethylsiloxane (PDMS), thermoplastic polyurethane (TPU), etc. In some embodiments, the middle layer includes a high-resistance elastic material, and the resistivity of the middle layer is more than 1000 times greater than the resistivity of the first conductive layer and the second conductive layer. In this case, each capacitor structure is a capacitor-resistor composite capacitor structure.

[0171] In some embodiments, the thickness of the first conductive layer and the second conductive layer of each capacitor structure is in the range of 1 to 100 um. In some embodiments, the width of the first conductive layer and the second conductive layer of each capacitor structure is in the range of 0.5 to 10 mm. In some embodiments, the interval between the first conductive layer 721 of the first capacitor structure 720 and the first conductive layer 731 of the second capacitor structure 730 is in the range of 0.02 to 2 mm. In some embodiments, the interval between the second conductive layer 721 of the first capacitor structure 720 and the second conductive layer 732 of the second capacitor structure 730 is in the range of 0.02 to 2 mm. In some embodiments, the interval between the first conductive layer 741 of the third capacitor structure 740 and the first conductive layer 751 of the fourth capacitor structure 750 is in the range of 0.02 to 2 mm. In some embodiments, the interval between the second conductive layer 742 of the third capacitor structure 740 and the second conductive layer 752 of the fourth capacitor structure 750 is in the range of 0.02 to 2 mm.

[0172] In some embodiments, the conductivity of the first conductive layer is greater than the conductivity of the intermediate layer.

[0173] Conductivity is a parameter used to describe the ease of charge flow in a substance. Since the conductivity of the first conductive layer is greater than that of the second conductive layer, the first conductive layer has better conductivity. Therefore, the resistance of the first conductive layer is less than that of the second conductive layer. It should be noted that in this specification, the resistance of a certain component refers to the resistance between two surfaces of the component spaced along the thickness direction of the flexible substrate 710. The resistance of the first conductive layer refers to the resistance between two surfaces of the first conductive layer spaced along the thickness direction of the flexible substrate 710, and the resistance of the second conductive layer may refer to the resistance between two surfaces of the second conductive layer spaced along the thickness direction of the flexible substrate 710. In some embodiments, the conductivity of the first conductive layer may be more than 100 times that of the second conductive layer. In some embodiments, by setting the density of the conductive particles filled in the elastic material of the first conductive layer to be greater than the density of the conductive particles filled in the elastic material of the second conductive layer, the conductivity of the first conductive layer can be greater than that of the second conductive layer.

[0174] When the capacitor structure (e.g., the first capacitor structure 720, the second capacitor structure 730, the third capacitor structure 740, and the fourth capacitor structure 750) is a capacitor-resistance composite capacitor structure or a single capacitor structure, the resistance between the two surfaces of the intermediate layer spaced along the thickness direction of the flexible substrate 710 should not be too small, so as to facilitate the subsequent measurement of the resistance and the analysis of the change in resistance. In some embodiments, the resistance between the two surfaces of the intermediate layer spaced along the thickness direction of the flexible substrate 710 is greater than 0.8MΩ. For example, when the capacitor structure (e.g., the first capacitor structure 720, the second capacitor structure 730, the third capacitor structure 740, and the fourth capacitor structure 750) is a capacitor-resistance composite capacitor structure, the resistance between the two surfaces of the intermediate layer spaced along the thickness direction of the flexible substrate 710 may also be 0.8MΩ to 1150GΩ. For another example, when the capacitor structure (e.g., the first capacitor structure 720, the second capacitor structure 730, the third capacitor structure 740, and the fourth capacitor structure 750) is a single capacitor structure, the resistance between the two surfaces of the intermediate layer spaced along the thickness direction of the flexible substrate 710 can be larger, for example, the resistance can be greater than 1150 GΩ.

[0175] By setting the resistance between the two surfaces of the intermediate layer along the thickness direction of the flexible substrate 710 to be greater than 0.8 MΩ, the intermediate layer can be made conductive while having a sufficiently large resistance, thereby ensuring the measurement of the resistance of the intermediate layer and the analysis of the change in resistance, which can accurately reflect the bending condition of the first strain sensor 700.

[0176] In some embodiments, the relative dielectric constant of the intermediate layer is greater than 2. The relative dielectric constant is a physical parameter that characterizes the dielectric properties or polarization properties of a dielectric material, and its value is equal to the ratio of the capacitance of a capacitor of the same size made with the corresponding material as the medium to the capacitance of a capacitor of the same size made with a vacuum as the medium. The value of the relative dielectric constant is also a characterization of the material's ability to store electricity, also known as relative permittivity. A substance with a relative dielectric constant greater than 2 is a polar substance, that is, an intermediate layer with a relative dielectric constant greater than 2 has a certain ability to store electricity. Therefore, the intermediate layer can be equivalent to a parallel combination of a resistor element and a capacitor element.

[0177] In some embodiments, the relative dielectric constant of the intermediate layer may also be greater than or equal to 4. Preferably, the relative dielectric constant of the intermediate layer is greater than 5. In some embodiments, the relative dielectric constant of the intermediate layer is greater than 10.

[0178] The relative dielectric constant of the middle layer is set to be greater than 2, so that the middle layer can be both a resistor and a capacitor. During the bending process of the first strain sensor 700, the capacitance and resistance of the middle layer will change with the deformation of the flexible substrate 710. Therefore, the parameters related to the capacitance and resistance of the middle layer can reflect the bending of the first strain sensor 700. In this embodiment, the capacitance and resistance exist at the same time to reflect the bending of the first strain sensor 700, which can significantly improve the sensitivity and accuracy of the first strain sensor 700.

[0179] In addition, by setting the resistance between the two surfaces of the intermediate layer along the thickness direction of the flexible substrate 710 to be greater than 0.8 MΩ, it can also be ensured that the intermediate layer can exhibit both better resistance performance and better capacitance performance, thereby ensuring the sensitivity and accuracy of the first strain sensor 700.

[0180] In some embodiments, two capacitor structures on the same side surface of the flexible substrate 710 can share a first conductive layer, and the second conductive layers of the two capacitor structures are spaced apart along the short axis direction of the flexible substrate 710. In some embodiments, more conductive layers (such as a third conductive layer, etc.) can be arranged in a stacked manner along the thickness direction of each inductive structure, which can effectively improve the sensitivity of the sensor to detect deformation of each degree of freedom. For example, each capacitor structure includes a second conductive layer, a second intermediate layer, a third conductive layer, a first intermediate layer, and a first conductive layer stacked in sequence away from the flexible substrate 710 along the thickness direction. Among them, the parameters of the third conductive layer, the first conductive layer, and the second conductive layer are consistent; the parameters of the second intermediate layer, the first intermediate layer, and the intermediate layer described above are consistent.

[0181] In some embodiments, the at least two target finger joints further include at least two proximal interphalangeal joints corresponding to the at least two metacarpophalangeal joints. The finger joint angles can be jointly determined based on the sensor data corresponding to the at least two metacarpophalangeal joints and their corresponding at least two proximal interphalangeal joints. At this time, the at least two strain sensors include at least two second strain sensors located at the at least two proximal interphalangeal joints, and each second strain sensor is configured to measure the deformation of the corresponding proximal interphalangeal joint in a single degree of freedom. The deformation of the proximal interphalangeal joint in a single degree of freedom may refer to the deformation information of the proximal interphalangeal joint when it is bent.

[0182] The second strain sensor refers to a strain sensor arranged at the proximal interphalangeal joint, which is used to collect the deformation (such as bending) of the proximal interphalangeal joint in a single degree of freedom. In some embodiments, the second strain sensor can be a uniaxial sensor, which can include various flexible bending sensors, such as inductive sensors, resistive sensors, capacitive sensors, optical fiber sensors, etc. As an example only, the second strain sensor includes an inductive sensor, and the inductance of the inductive sensor can change with the deformation of the interphalangeal joint (such as the proximal interphalangeal joint). Specifically, the inductive sensor deforms with the movement of the interphalangeal joint, and its inductance responds to the deformation of the inductive sensor. By collecting the parameters related to the inductance of the inductive sensor and analyzing its changes, the bending condition (such as bending angle, bending direction, etc.) of the inductive sensor can be calculated more accurately. For example, the electrical quantity of the inductance of the inductive sensor is proportional to the deformation of the inductive sensor, and thus, the change in the electrical quantity of the inductance of the inductive sensor can characterize the deformation or bending degree of the interphalangeal joint. Since the interphalangeal joint corresponds to only one bending degree of freedom, a single-axis sensor (such as an inductive sensor) is used to measure the movement of the interphalangeal joint, which has a simple structure and low product cost. In addition, a second strain sensor can be arranged only at the interphalangeal joint closest to the metacarpophalangeal joint of each finger of the user, thereby simplifying the structure, reducing product cost, and ensuring the accuracy of deformation data collection.

[0183] In some embodiments of the present specification, considering that the proximal interphalangeal joint only corresponds to one bending degree of freedom, a single-axis sensor is used to measure the sensor data of the proximal interphalangeal joint when it is bent, which has a simple structure and low cost.

[0184] In step 620, the processing device 410A may obtain a first mapping relationship, where the first mapping relationship reflects the relationship between the sensor data corresponding to at least two target finger joints and the joint angles, wherein the at least two target finger joints include at least two adjacent metacarpophalangeal joints of the user.

[0185] The first mapping relationship can be represented by a mapping function, a machine learning model, etc. For example, the first mapping relationship can be a mapping function (e.g., a polynomial) with sensor data as an independent variable and joint angle as a dependent variable. For another example, the first mapping relationship can be a machine learning model, which takes sensor data as input and joint angle as output. For example, the machine learning model can be a recurrent neural network (RNN) or other deep neural network models.

[0186] In some embodiments, the first mapping relationship includes a mapping function or a machine learning model corresponding to each target finger joint. For example, for a specific target finger joint, the corresponding mapping function takes the sensor data of all target finger joints as independent variables and the joint angle of the specific target finger joint as dependent variable. For another example, for a specific target finger joint, the input of the corresponding machine learning model is the sensor data of all target finger joints, and the output is the joint angle of the specific target finger joint.

[0187] In some embodiments, the first mapping relationship may be determined by the processing device 410A or the processing device 410B based on a plurality of data samples. For more information on generating the first mapping relationship, see Fig.19 and its related description.

[0188] In step 630 , the processing device 410A may determine a first joint angle of each target finger joint based on the first sensor data and the first mapping relationship.

[0189] The first joint angle of the target finger joint may include its angle at any one or more degrees of freedom. For example, the first joint angle includes a bending angle and / or a swing angle. The bending angle refers to the angle at which the target finger joint rotates around Figure 3 The swing angle refers to the rotation angle of the target finger joint around Figure 3 The angle of Z-axis rotation shown. In some embodiments, when the target finger joint is a metacarpophalangeal joint, the first joint angle includes both the flexion angle and the swing angle. When the target finger joint is a proximal interphalangeal joint, the first joint angle includes only the flexion angle.

[0190] In some embodiments, the processing device 410A may input the first sensor data into the first mapping relationship, and after processing the first mapping relationship, output the first joint angle of the target finger joint. For example, when the first mapping relationship is represented by a mapping function, the first sensor data may be substituted into the mapping function to obtain the first joint angle. For another example, when the first mapping relationship is represented by a trained machine learning model, the first sensor data may be input into the machine learning model to obtain the first joint angle.

[0191] In some embodiments, as described in step 610, the target finger joints may include some of the user's finger joints or all of the user's metacarpophalangeal joints and all of the proximal interphalangeal joints. The processing device 410A may determine the first joint angle of each of all the target finger joints based on the first mapping relationship. In some embodiments, the processing device 410A may determine the first joint angle of some of the target finger joints based on the first mapping relationship. For example, the first joint angle determined by the processing device 410A may be the angle of some of the metacarpophalangeal joints, and the sensor data related to other metacarpophalangeal joints and / or proximal interphalangeal joints in the first sensor data may be used as auxiliary information to improve the accuracy of the first joint angle obtained. As an example only, the first sensor data related to multiple finger joints may be used as inputs of the first mapping relationship in sequence, and the mapping relationship may be set to output only the first joint angle corresponding to the first input finger joint. As another example, the processing device 410A may provide a selection window to the user through an interactive interface, and the user may select one or more target finger joints in the window. The processing device 410A may output the first joint angle of the target finger joint selected by the user based on the received feedback information.

[0192] In some embodiments, the glove body includes a fabric wrapped around each finger of the user and a fabric located between adjacent fingers, wherein the elastic modulus of the fabric between adjacent fingers is less than the elastic modulus of the fabric on each finger. The fabric between adjacent fingers can be referred to as an elastic fabric structure (e.g., Figure 1 For more information on the elastic fabric structure, see Figure 1 and description thereof. In some embodiments, characteristic parameters of the fabric (i.e., elastic fabric structure) between adjacent fingers may affect the strength of the coupling relationship between the finger joints of adjacent fingers. For example, the smaller the elastic modulus of the fabric between adjacent fingers, the less likely the fabric of the adjacent fingers is to be pulled, and thus the smaller the coupling relationship between the finger joints of the adjacent fingers. The processing device 410A may further determine the first joint angle based on the characteristic parameters.

[0193] In some embodiments, the processing device 410A may obtain the value of the characteristic parameter of the fabric between adjacent fingers, and the characteristic parameter includes at least one of the elastic coefficient, the lateral dimension, and the rebound coefficient. The first mapping relationship may further reflect the relationship between the characteristic parameter and the sensor data corresponding to at least two target finger joints and the joint angles of at least two target finger joints. Thus, the first joint angle of each target finger joint may be further determined based on the value of the characteristic parameter of the fabric between adjacent fingers. For example, the first mapping relationship may be represented by a machine learning model, the training data of which includes sample sensor data of sample finger joints and sample values ​​of characteristic parameters of sample fabric between adjacent sample fingers. Accordingly, when determining the first joint angle using the machine learning model, the processing device 410A may input the first sensor data and the value of the characteristic parameter of the fabric between adjacent fingers (such as the elastic coefficient, the lateral dimension, the rebound coefficient, etc.) into the model, and the model outputs the first joint angle of the target finger joint.

[0194] In some embodiments of the present specification, the influence of fabric characteristics between adjacent fingers on the coupling relationship between finger joints is taken into consideration, so that the first mapping relationship can be more in line with the actual situation and the determined first joint angle can be more accurate.

[0195] In some embodiments, the coupling relationship between the target finger joints is related to the distance between the target finger joints. As an example only, different users have different palm sizes, different distances between the metacarpophalangeal joints and the interphalangeal joints, and different gestures or hand movements that can be made (such as different degrees of bending of the finger joints). For another example, the closer the distance between the finger joints, the stronger the coupling. Thus, the first joint angle can be further determined based on the distance information.

[0196] In some embodiments, the glove body may include a position sensor arranged at each target finger joint, and the processing device 410A may further determine the distance information between at least two strain sensors based on the position data collected by the position sensor at each target finger joint. The first mapping relationship may further reflect the relationship between the distance between at least two strain sensors and the sensor data corresponding to at least two target finger joints and the joint angles of at least two target finger joints. Thus, the first joint angle of each target finger joint may be further determined based on the above distance information. For example, the first mapping relationship may be represented by a machine learning model, the training data of which includes sample sensor data of a sample finger joint and sample distance information between at least two sample strain sensors. Accordingly, when determining the first joint angle using the machine learning model, the processing device 410A may input the distance information between the at least two strain sensors into the model, and the model outputs the first joint angle of the target finger joint.

[0197] In some embodiments, the first mapping relationship may reflect a mapping relationship between a characteristic parameter of the elastic fabric structure, at least a distance between strain sensors, sensor data corresponding to at least two target finger joints, and joint angles of at least two target finger joints. The processing device 410A may further determine the first joint angle based on the value of the characteristic parameter and the distance information.

[0198] It should be noted that the above description of method 600 is for illustrative purposes only and is not intended to limit the scope of this specification. For those of ordinary skill in the art, various changes and modifications can be made under the guidance of this specification. However, these changes and modifications do not depart from the scope of this specification. In some embodiments, method 600 can be completed with one or more additional steps not described and / or without one or more steps discussed above. In addition, the order of method 600 is not limiting. For example, method 600 may include a step for generating a mapping relationship. For another example, method 600 may include a step for correcting a mapping relationship. For more description of correcting a mapping relationship, see Fig.16 and / or Fig.17 For another example, method 600 may further include a step of determining a second joint angle of the distal interphalangeal joint based on the second mapping relationship.

[0199] In some embodiments, when the target joint angle includes the proximal interphalangeal joint, for each proximal interphalangeal joint, the processing device 410A can determine the second joint angle of the distal interphalangeal joint corresponding to the proximal interphalangeal joint based on the second mapping relationship and the first joint angle of the proximal interphalangeal joint. Wherein, the second mapping relationship reflects the relationship between the joint angles of the proximal interphalangeal joint and the distal interphalangeal joint. Since the distal interphalangeal joint behaves as a follow-up to the proximal interphalangeal joint, the second joint angle of the distal interphalangeal joint can be determined according to the first joint angle of the proximal interphalangeal joint corresponding to it. The second mapping relationship can be represented by a mapping function or a trained machine learning model. For example, the second mapping relationship can be a mapping function representing the relationship between the first joint angle and the second joint angle, and the processing device 410A can input the first joint angle into the mapping function to calculate the second joint angle. For another example, the second mapping relationship can be a trained machine learning model, and the processing device can input the first joint angle and / or the first sensor data into the model to obtain the second joint angle. In some embodiments, the second mapping relationship can be obtained by a data fitting algorithm or a machine learning algorithm. For example, the second mapping relationship can be represented by a machine learning model, the training data of the model includes a sample first joint angle and / or a sample first sensor data of a sample proximal interphalangeal joint, and the training label includes a sample second joint angle of a sample distal interphalangeal joint corresponding to the sample proximal interphalangeal joint.

[0200] In some embodiments of the present specification, considering the linkage between the proximal interphalangeal joint and the distal interphalangeal joint, the second joint angle of the distal interphalangeal joint can be determined by the first joint angle of the proximal interphalangeal joint, thereby avoiding the need to arrange sensors at the distal interphalangeal joint, thus saving costs. In addition, through the second mapping relationship, after determining the first joint angle of the proximal interphalangeal joint, the second joint angle of the distal interphalangeal joint corresponding to the proximal interphalangeal joint can be quickly and accurately determined.

[0201] In some embodiments, since the first mapping relationship is determined based on a large number of data samples, it reflects the relationship between the sensor data and the joint angle of the target finger joint under general circumstances. However, due to individual differences, the first sensor data may be different when different users wear gloves to perform the same action. For example, the first mapping relationship can be used to determine the relationship between the first sensor data and the joint angle of the target finger joint within a preset data range, and the first sensor data of some users may exceed the preset data range. Thus, the processing device 410A can correct the first mapping relationship for a specific user, and use the corrected first mapping relationship to determine the first joint angle of each target finger joint of the specific user.

[0202] Fig.16 1 is a flowchart of another method for determining finger joint angles according to some embodiments of the present specification. In some embodiments, method 1600 may be performed by processing device 410A. In some embodiments, step 630 in method 600 may be implemented by method 1600. Fig.16 As shown, method 1600 may include the following steps.

[0203] In step 1610 , the processing device 410A may obtain reference sensor data related to at least two target finger joints of the user, where the reference sensor data is collected by using at least two strain sensors when the user makes a preset gesture.

[0204] The preset gestures may be various preset hand movements that reflect finger postures or deformations. In some embodiments, the preset gestures may be used to reflect the upper and lower limits of the deformation (such as bending, swinging) amplitude of the fingers (such as the maximum bending angle of the fingers, the maximum angle between two adjacent fingers, etc.). Exemplarily, the preset gestures may be the palm's close-to-close and stretch-out movements, open movements, fist-making movements, etc. In some embodiments, deformations in different degrees of freedom may correspond to different preset gestures. For example, for deformations in the bending degree of freedom, the preset gestures may include a fist-making movement and an open movement (or a close-to-close and stretch-out movement), wherein the fist-making movement corresponds to the maximum joint angle of the bending of the finger joints, and the open movement (or a close-to-close and stretch-out movement) corresponds to the minimum joint angle of the bending of the finger joints. For another example, for deformations in the swinging degree of freedom, the preset fingers may include a close-to-close and stretch-out movement and an open movement, wherein the close-to-close and stretch-out movement corresponds to the minimum joint angle of the swing of the finger joints, and the open movement corresponds to the maximum joint angle of the swing of the finger joints. The preset gestures may be displayed to the user in the form of images, videos, or other forms, and the user may make the preset gestures while wearing the glove body.

[0205] In step 1620 , the processing device 410A may calibrate the first mapping relationship based on the reference sensor data to obtain a corrected mapping relationship corresponding to the user.

[0206] In some embodiments, when the user makes different preset gestures, the sensor data collected by the strain sensors (such as the first strain sensor and the second strain sensor) arranged at different target finger joints can be used as reference sensor data for correcting the first mapping relationship. For example, the preset gesture can be a fisting action, which corresponds to the maximum joint angle of one or more target finger joints (or all finger joints). The processing device 410A can obtain reference sensor data related to at least two target finger joints during the fisting action, and determine whether the reference sensor data is outside the preset data range. In response to the reference sensor data being outside the preset data range, the processing device 410A can update the preset data range. As an example only, when the target finger joint is at the maximum joint angle, the deformation degree (such as bending degree) of the strain sensor at the target finger joint is the largest, and the processing device 410A can replace the upper limit of the preset data range with the reference sensor data at this time. For another example, the preset gesture can be an opening action, which corresponds to the minimum joint angle of one or more target finger joints (or all finger joints). The processing device 410A may obtain reference sensor data related to at least two target finger joints during the opening action, and determine whether the reference sensor data is outside the preset data range. In response to the reference sensor data being outside the preset data range, the processing device 410A may update the preset data range. For example, when the target finger joint is at the minimum joint angle, the deformation degree (e.g., bending degree) of the strain sensor at the target finger joint is the smallest, and the processing device 410A may replace the lower limit of the preset data range with the reference sensor data at this time.

[0207] After updating the preset data range, the processing device 410A may determine the correction mapping relationship based on the updated preset data range. As an example only, the first mapping relationship is represented by a mapping function table, and the processing device 410A may perform data fitting calculation based on the updated preset data range to obtain an updated mapping function, which may be used as the correction mapping relationship.

[0208] In some embodiments, the first mapping relationship can be represented by a trained machine learning model, and the correction of the first mapping relationship can be implemented based on a transfer learning algorithm. The trained machine learning model corresponding to the current first mapping relationship can be referred to as a general model.

[0209] In some embodiments, the processing device 410A may use the general model as the initial pre-trained model and further train the initial pre-trained model. Among them, the training sample can be determined based on the reference sensor data, and the training label can be the reference first joint angle corresponding to the reference sensor data (such as the angle of the finger joint under a certain gesture). During training, the value of the loss function can be determined based on the difference between the output of the initial pre-trained model and the training label. The parameters of the initial pre-trained model can be iteratively updated based on the value of the loss function until the training termination condition is met (such as the loss function converges, a specific number of iterations are performed, etc.), and an updated initial pre-trained model is obtained. The processing device 410A can use the updated initial pre-trained model as the corrected first mapping relationship, thereby realizing the correction of the first mapping relationship using the transfer learning method.

[0210] In some embodiments, the processing device 410A can train the universal model for different users or user groups (such as male / female, elderly / children, etc.) to obtain the corrected first mapping relationship (i.e., the corrected universal model) corresponding to the different users or user groups. In actual applications, corresponding models (such as the universal model or the corrected universal model) can be used for different users to determine the first joint angle of the target finger joint according to the first sensor data.

[0211] In step 1630, the processing device may determine the first joint angle of each target finger joint based on the correction mapping relationship and the first sensor data. For example, the processing device 410A may input the first sensor data into the correction mapping relationship, and output the first joint angle of the target finger joint after processing the correction mapping relationship.

[0212] In some implementations of this specification, the first mapping relationship is corrected based on the reference sensor data to obtain a corrected mapping relationship that better matches the current user, thereby making the determined first joint angle more accurate.

[0213] In some embodiments, process 1600 may be executed when the user first wears the glove system, and after the correction mapping relationship corresponding to the user is determined, it may be stored for use each time the user wears the glove system.

[0214] In some embodiments, the first mapping relationship can be represented by a mapping function, and the processing device 410A can correct the mapping function based on the distance information between at least two strain sensors, and use the corrected mapping function to determine the first joint angle of each of the target finger joints.

[0215] Fig.171700 is a flowchart of another method for determining finger joint angles according to some embodiments of the present specification. In some embodiments, method 1700 may be performed by processing device 410A. In some embodiments, step 630 in method 600 may be implemented by method 1700. Fig.17 As shown, method 1700 may include the following steps.

[0216] In step 1710 , the processing device 410A determines distance information between at least two strain sensors based on position data collected by the position sensor at each target finger joint.

[0217] The linkage or coupling relationship between at least two target finger joints is related to the distance between the target finger joints. Just as an example, different users have different palm sizes, different distances between the metacarpophalangeal joints and the interphalangeal joints, and different gestures or hand movements (such as different degrees of bending of the finger joints). For another example, the closer the distance between the finger joints, the stronger the coupling. Therefore, the mapping function can be corrected based on the distance information between the target finger joints.

[0218] In some embodiments, a position sensor and a strain sensor corresponding to each target finger joint are provided in the glove body, and the relative position relationship between the position sensor and the strain sensor is known. For any two strain sensors, the distance between the two strain sensors can be determined based on the position data collected by their corresponding position sensors and the relative position relationship between them and the corresponding position sensors. The distance between the two strain sensors can be used to represent the distance between the two corresponding target finger joints.

[0219] Step 1720: The processing device 410A may correct the mapping function based on the distance information to obtain a corrected mapping function corresponding to the user.

[0220] In some embodiments, the mapping function includes relationship coefficients reflecting the strength of the coupling relationship (such as coupling degree) between different target finger joints, and these relationship coefficients can be adjusted based on the distance information to obtain a corrected mapping function. For example, the mapping function corresponding to the first target joint takes the sensor data corresponding to all target joints as the independent variable, takes the first joint angle of the first target joint as the dependent variable, and includes the relationship coefficients of each other target joint and the first target joint. When the distance between the strain sensor corresponding to a certain target joint and the first target joint is less than the first distance threshold, it means that the coupling relationship between them is strong, and the value of the relationship coefficient between the target joint and the first target joint can be increased; when the distance is greater than the second distance threshold, it means that the coupling relationship between them is weak, and the value of the relationship coefficient between the target joint and the first target joint can be lowered. Among them, the first distance threshold is less than the second distance threshold. In some embodiments, the first distance threshold and the second distance threshold can be set by the user, or can be determined based on statistical data. For example, the distribution of the distances between the sample strain sensors corresponding to the data samples used to determine the mapping function can be analyzed to determine the first distance threshold and the second distance threshold.

[0221] In step 1730, the processing device 410A may determine the first joint angle of each target finger joint based on the correction mapping function and the first sensor data. For example, the processing device 410A may input the first sensor data into the correction mapping function, and output the first joint angle of the target finger joint after being processed by the correction mapping function.

[0222] Some embodiments of the present specification take into account the effect of the distance between different finger joints (or finger joints of different users' hands) on the coupling relationship between the finger joints, and correct the first mapping relationship according to the distance between the target finger joints of different users, so that the first mapping relationship can be combined with the hand features of different users, thereby making the determined first joint angle more targeted and more accurate.

[0223] In some embodiments, the at least two target finger joints may only include at least two adjacent metacarpophalangeal joints of the user, and the processing device 410A may select the target finger joints based on the above Figure 6 The method 600 determines a first joint angle of each metacarpophalangeal joint. Further, the processing device 410A may determine a third joint angle of the proximal interphalangeal joint based on a third mapping relationship.

[0224] Fig.18is an exemplary flow chart of determining the third joint angle according to some embodiments of the present specification. In some embodiments, method 1800 may be performed by processing device 410A. In some embodiments, method 1800 may utilize one or more additional operations / steps not described below, and / or may not be completed by one or more operations / steps discussed below. In addition, as Fig.18 The order of the operations / steps shown is not limiting. Fig.18 As shown, method 1800 may include the following steps.

[0225] In step 1810, the processing device 410A may obtain second sensor data related to at least two proximal interphalangeal joints corresponding to at least two metacarpophalangeal joints, wherein the second sensor data is collected using at least two second strain sensors, each of which is arranged in the glove body worn by the user and is located at one of the at least two proximal interphalangeal joints.

[0226] In some embodiments, each second strain sensor may be configured to measure the deformation of its corresponding proximal interphalangeal joint in a single degree of freedom. The deformation of the proximal interphalangeal joint in a single degree of freedom may refer to the deformation information of the proximal interphalangeal joint when it is bent. Figure 1 and its description.

[0227] In step 1820, the processing device 410A may determine a third joint angle of each proximal interphalangeal joint based on the second sensor data, the first joint angles of at least two metacarpophalangeal joints, and a third mapping relationship, wherein the third mapping relationship reflects the relationship between the joint angles of at least two metacarpophalangeal joints, the sensor data of at least two proximal interphalangeal joints, and the joint angles of at least two proximal interphalangeal joints.

[0228] The third joint angle refers to the bending angle of the proximal interphalangeal joint, for example, the bending angle of the proximal interphalangeal joint of the index finger, middle finger, ring finger and / or little finger.

[0229] In some embodiments, there is a coupling relationship between the metacarpophalangeal joints (hereinafter referred to as the first coupling relationship), and there is also a coupling relationship between the proximal interphalangeal joints and the metacarpophalangeal joints (hereinafter referred to as the second coupling relationship). For example, the deformation (such as bending) of a proximal interphalangeal joint will have different degrees of linkage effects on the metacarpophalangeal joint corresponding to the proximal interphalangeal joint and its adjacent metacarpophalangeal joint. The third mapping relationship can be used to characterize the coupling relationship between the joint angles of the proximal interphalangeal joint and the metacarpophalangeal joint.

[0230] In some embodiments, the third mapping relationship can be obtained by a method similar to the first mapping relationship. In some embodiments, the third mapping relationship can be represented by a mapping function or a trained machine learning model. Taking the representation of the third mapping relationship by a machine learning model as an example, the processing device 410A can input the joint angles of at least two metacarpophalangeal joints and the sensor data of at least two proximal interphalangeal joints into the trained machine learning model corresponding to the third mapping relationship, and output the third joint angle corresponding to each proximal interphalangeal joint through the model. In some embodiments, the at least two proximal interphalangeal joints here can be any at least two proximal interphalangeal joints, and the at least two metacarpophalangeal joints can be part or all of the metacarpophalangeal joints. For example, at least two metacarpophalangeal joints can be the metacarpophalangeal joints corresponding to the proximal interphalangeal joints and one or two metacarpophalangeal joints of adjacent fingers.

[0231] In some embodiments of the present specification, the coupling relationship between the proximal interphalangeal joint and one or more metacarpophalangeal joints is considered, and when determining the third joint angle of the proximal interphalangeal joint, the influence of the metacarpophalangeal joint angle is considered, so that the third joint angle is more accurate. Figure 6 The method 600 described in the embodiment of this specification determines the joint angles corresponding to different types of finger joints (mecarpophalangeal joints and proximal interphalangeal joints) based on different mapping relationships (first mapping relationship and third mapping relationship), and can respectively combine different coupling relationships (first coupling relationship and second coupling relationship) of different types of finger joints, so that the obtained joint angles are more accurate.

[0232] Fig.19 It is a flowchart of a method for determining the correspondence between sensor data and finger joint angles according to some embodiments of this specification.

[0233] In some embodiments, method 1900 may be performed by processing device 410B. Fig.19 As shown, method 1900 may include the following steps.

[0234] In step 1910, the processing device 410B may acquire multiple groups of sample sensor data, each group of sample sensor data being related to at least two sample finger joints of a sample user and being collected at a target time using at least two sample strain sensors, each sample strain sensor being arranged in a sample glove body worn by a sample user and being located at one of the at least two sample finger joints, wherein the at least two sample finger joints include at least two adjacent metacarpophalangeal joints of the sample user.

[0235] In this specification, "sample" is used to indicate that the concept it describes is used to determine the mapping relationship (such as the first mapping relationship, the second mapping relationship and the third mapping relationship described above). Figure 6The target finger joints are similar in meaning, but the sample finger joints are concepts involved in the process of determining the first mapping relationship, while the target finger joints are concepts involved in the process of applying the first mapping relationship.

[0236] The sample sensor data can be used to characterize the posture and / or motion information of the sample finger joints of the sample user, which can be collected at the target time using at least two sample strain sensors, each of which is arranged in the sample glove body worn by the sample user and is located at one of the at least two sample finger joints. Figure 1 The sample glove system 100 is the same as or similar to the sample glove system 100 to collect and obtain sample sensor data. The sample glove system includes a plurality of sample strain sensors, and the plurality of sample strain sensors may include a first sample strain sensor and a second sample strain sensor arranged in the sample glove body. For more information about the sample glove system, the glove body, the first strain sensor and the second strain sensor, see Figure 1 and description thereof. In some embodiments, the sample sensor data may be historical data.

[0237] In some embodiments, the sample sensor data may be data collected by the sample strain sensor when the sample user performs hand movements (such as when the sample user imitates a preset gesture). The target moment refers to the moment when the sample user makes / completes a specific gesture, which may be in the form of a timestamp. When the sample user completes different gestures, different target moments correspond. The processing device 410B may record and / or store the sample sensor data collected by the sample strain sensor at each target moment to obtain a set of sample sensor data corresponding to each target moment.

[0238] The sample user may be any user. In some embodiments, the sample user includes users with different hand features (such as palm size, finger length, etc.) and / or user groups (such as male / female, elderly / children, etc.).

[0239] The sample finger joints may be one or a combination of the joints in the sample user's fingers (such as the thumb, index finger, middle finger, ring finger and little finger), such as interphalangeal joints (such as proximal interphalangeal joints, proximal interphalangeal joints), metacarpophalangeal joints, etc.

[0240] In some embodiments, at least two sample finger joints include some finger joints of the sample user. Among them, some finger joints may include some finger joints with a coupling relationship. In some embodiments, some finger joints include some finger joints that are easier to couple or have a greater degree of coupling. For more information about coupling relationships and coupling degrees, see Figure 6 And its description will not be repeated here.

[0241] In some embodiments, the at least two sample finger joints include at least two adjacent metacarpophalangeal joints of the sample user. It is understandable that the coupling degree of adjacent metacarpophalangeal joints is relatively high. The processing device 410B may determine at least two adjacent metacarpophalangeal joints of the user as sample finger joints. In some embodiments, the at least two sample finger joints include all metacarpophalangeal joints of the sample user.

[0242] In some embodiments, the at least two metacarpophalangeal joints include the metacarpophalangeal joints of at least one of the index finger, middle finger, and ring finger of the sample user and the two adjacent fingers on both sides thereof. For example, for the index finger, the two adjacent fingers on both sides thereof are the thumb and the middle finger, and the at least two metacarpophalangeal joints include the metacarpophalangeal joint of the index finger of the sample user and the metacarpophalangeal joints of the thumb and the middle finger.

[0243] In some embodiments, the at least two metacarpophalangeal joints include the metacarpophalangeal joints of at least one of the thumb and the pinky finger of the sample user and one of its adjacent fingers. For example, for the thumb, the adjacent finger is the index finger, and the at least two metacarpophalangeal joints include the metacarpophalangeal joint of the thumb and the metacarpophalangeal joint of the index finger of the sample user.

[0244] In some embodiments, the at least two sample finger joints further include at least two proximal interphalangeal joints corresponding to at least two metacarpophalangeal joints of the sample user. For example, the at least two sample fingers include the metacarpophalangeal joints and proximal interphalangeal joints of the index finger and the middle finger.

[0245] In some embodiments of the present specification, only sample sensor data corresponding to some finger joints may be obtained to make the calculation process more targeted and reduce the amount of data in the calculation process.

[0246] In some embodiments, the at least two sample finger joints include all the metacarpophalangeal joints and all the proximal interphalangeal joints of the sample user. When the metacarpophalangeal joints move or deform, they may also cause linkage effects on the sample sensor data of the proximal interphalangeal joints. Therefore, the processing device 410B also considers the coupling relationship between the metacarpophalangeal joints and the proximal interphalangeal joints when acquiring the sample sensor data. It can be understood that the at least two sample finger joints can be connected with the sample sensor data. Figure 6 In some embodiments, at least two target finger joints are similar to each other. Figure 6 At least two of the target finger joints described are of the same type. For more information about target finger joints, see Figure 6 and its description.

[0247] In some embodiments of the present specification, all metacarpophalangeal joints and all proximal interphalangeal joints are used as sample finger joints, so that sample sensors do not need to be arranged at distal interphalangeal joints, simplifying the structure of the sample glove system and saving costs; at the same time, the sample sensor data corresponding to the sample finger joints do not need to be processed by data screening, thereby simplifying the processing steps. In addition, by using all metacarpophalangeal joints and all proximal interphalangeal joints as sample finger joints, the processing process can take into account the coupling relationship between all metacarpophalangeal joints and all proximal interphalangeal joints, making the obtained sample sensor data more complete, and using the mapping relationship (e.g., the first mapping relationship) obtained based on the sample sensor data, a more accurate finger joint angle (e.g., the first joint angle) can be obtained.

[0248] The sample strain sensor includes at least two strain sensors arranged at the at least two sample finger joints.

[0249] In some embodiments, the at least two sample strain sensors include at least two first sample strain sensors located at at least two metacarpophalangeal joints of the sample user, and each first sample strain sensor is configured to measure deformation of its corresponding metacarpophalangeal joint in two degrees of freedom. The first sample strain sensor is Figure 6 The first strain sensor described is the same or similar.

[0250] In some embodiments, the at least two sample strain sensors include at least two second sample strain sensors located at at least two proximal interphalangeal joints of the sample user, each second sample strain sensor being configured to measure the deformation of its corresponding proximal interphalangeal joint in a single degree of freedom. Figure 6 The second strain sensor described is the same or similar.

[0251] More information about the first strain sensor, the second strain sensor, and the degrees of freedom can be found elsewhere in this specification (e.g. Figure 6 and its description).

[0252] In some embodiments, the processing device 410B may obtain sample sensor data corresponding to the sample user in a manner similar to step 610. For example, sample raw data collected by the sample strain sensor may be used as sample sensor data; for another example, sample decoupling data determined based on the sample raw data may be used as sample sensor data.

[0253] Step 1920 , for each set of sample sensor data, the processing device 410B may obtain the hand optical image taken by the corresponding sample user at the target time.

[0254] The hand optical image refers to the image data captured when the sample user's hand is wearing gloves. For example, it can be a hand image captured when the sample user makes (completes) a preset gesture (such as an opening palm movement, a fist movement, etc.) at the target moment. The hand optical image can be acquired at the target moment using various image acquisition devices (such as a depth camera, etc.). It can be understood that the acquisition time of the hand optical image and the sample sensor data is the same. For example, the hand optical image may include a timestamp corresponding to the target moment, and the sample sensor data and the hand optical image acquired at the same target moment can be matched based on the timestamp.

[0255] In some embodiments, the hand optical image is a 2D plane image. In some embodiments, the hand optical image contains depth information. Among them, the depth information can reflect the three-dimensional spatial position of different position points or feature points (such as finger joints) of the sample user's hand in the hand optical image. In some embodiments, the depth information can be used to determine the distance information of different feature points in the hand optical image (for example, the distance between different finger joints, the distance between the finger joints and the camera position, etc.). In some embodiments, the depth information can be used to construct a hand model corresponding to the hand optical image. In some embodiments of the present specification, the use of a hand optical image containing depth information can make the hand model determined based on the hand optical image more realistic, thereby making the sample joint angles of the sample finger joints determined based on the hand optical image and / or the hand model more accurate.

[0256] In some embodiments, the hand optical image includes a single image. In some embodiments, the hand optical image includes multiple hand optical images at different shooting angles. The different shooting angles may be multiple preset angles. For example, relative to the palm plane (such as the X plane) of the sample user, the different shooting angles may be n (such as 2, 4, 8) angles within the circumferential space of the palm plane. Exemplarily, multiple image acquisition devices may be arranged on both sides of the horizontal plane of the palm plane, on both sides of the vertical palm plane, or at other positions to acquire multiple hand optical images of the sample user's hand at multiple shooting angles. For another example, an image acquisition device may be used to rotate around the sample user's hand to acquire multiple hand optical images of the sample user's hand at multiple shooting angles. It should be noted that when an image acquisition device is used to rotate around the sample user's hand to acquire multiple hand optical images, the sample user's gesture remains unchanged, and accordingly, the above-mentioned target moment may refer to the time period during which the sample user maintains the gesture.

[0257] It should be noted that, for the same action performed by the sample user, the time corresponding to the multiple hand optical images taken at different shooting angles is the same.

[0258] In some embodiments of this specification, multiple hand optical images taken at different shooting angles are used to make the hand model determined based on the hand optical images better reflect the hand movements or gestures of the sample user, thereby making the sample joint angles of the sample finger joints determined based on the hand optical images and / or the hand model more accurate. In some embodiments, multiple hand optical images taken at different shooting angles can also be scored to screen out hand optical images that meet the requirements. For more descriptions on scoring hand optical images, see Fig.21 and its description.

[0259] In step 1930 , the processing device 410B may determine sample joint angles of at least two sample finger joints of the corresponding sample user based on the hand optical image.

[0260] In some embodiments, the processing device 410B may determine sample joint angles of at least two sample finger joints based on deformation information of the sample finger joints in the hand optical image.

[0261] The sample joint angle refers to the joint angle of the sample finger joint at the target moment. The sample joint angle is similar to the first joint angle described in step 630, and may also include angles in one or more degrees of freedom. Exemplarily, the processing device 410B may identify the position, angle and other posture information of the fingers and / or finger joints corresponding to the sample finger joints in the hand optical image through various image analysis algorithms such as image recognition, and determine the sample joint angle of the sample finger joint accordingly.

[0262] In some embodiments, the processing device 410B may determine a hand model based on the hand optical image, and determine sample joint angles of at least two sample finger joints of the sample user based on the hand model.

[0263] The hand model is used to characterize the hand in the hand optical image, which can be in the form of an image model (such as a 2D or 3D image). The hand model can reflect the real posture information of the sample user's hand, for example, the position, bending, swing, etc. of the fingers and / or finger joints. In some embodiments, the hand model is a 3D model. In some embodiments, the processing device 410B can generate a hand model based on the hand optical image using a modeling technique (such as a 3D image modeling algorithm). For example, the processing device 410B can determine the spatial information of the sample user's hand (such as the position and distance of the boundaries and feature points, etc.) based on the depth information in the hand optical image, and generate a 3D hand model corresponding to the hand optical image through a 3D image modeling algorithm. In some embodiments, the processing device 410B can adjust the initial hand model based on the hand optical image to obtain a hand model corresponding to the hand optical image. Among them, the initial hand model can be a model that is pre-modeled or obtained from an external platform. For more information about determining the hand model based on the initial hand model, see Fig. 20 and its description.

[0264] In some embodiments, the processing device 410B may determine the joint angles of the finger joints in the hand model as the sample joint angles. For example, the joint angles of the metacarpophalangeal joints (such as the metacarpophalangeal joints of the index finger) in the hand model may be determined as the sample joint angles of the corresponding sample metacarpophalangeal joints (i.e., the sample metacarpophalangeal joints corresponding to the sample index finger).

[0265] In some embodiments of the present specification, the hand model may be a three-dimensional model, which can better reflect the real posture information of the sample user's hand in three-dimensional space compared to a two-dimensional hand optical image. Therefore, the sample joint angles determined by the hand model are more in line with the actual situation, thereby making the obtained sample joint angles more accurate. For example, when the sample finger joints in the hand optical image are blocked, a more accurate sample joint angle can be obtained through the hand model. In some embodiments of the present specification, the hand model is generated by adjusting the initial hand model obtained by pre-modeling. Since the angles, positions, and other information of each finger joint in the pre-modeled initial hand model are known, the angles, positions, and other information of the finger joints of the hand model can be quickly and accurately determined based on this known information, thereby improving the efficiency and accuracy of determining the sample joint angles.

[0266] In some embodiments, for each group of sample sensor data in at least one group of sample sensor data, the processing device 410B can also control the terminal device to display the hand model corresponding to the sample gesture to be imitated to the sample user before the target moment, and obtain sensor data collected by at least two sample sensors when the sample user imitates the sample gesture as the group of sample sensor data, wherein the sample joint angles corresponding to the group of sample sensor data are further determined based on the hand model.

[0267] The sample gestures to be imitated can be various preset gestures or hand movements (such as fisting, opening movements, etc.), which can be displayed to sample users through terminal devices (such as display devices), and sample users can make hand movements corresponding to the sample gestures to be imitated at the target time.

[0268] When the gesture made by the sample user is consistent with (or similar to) the sample gesture to be imitated, it means that the sample user has completed the sample gesture. At this time, the sensor data collected by at least two sample sensors can be used as the sample sensor data collected at the target moment. Further, the processing device 410B can determine the sample joint angle corresponding to the sample sensor data based on the hand model corresponding to the sample gesture.

[0269] In some embodiments, the processing device 410B may obtain sensor data collected by at least two sample sensors as sample sensor data based on user feedback information. For example, the sample user may determine whether the gesture made by the sample user is consistent with the sample gesture to be imitated and input user feedback information through the terminal device (such as in the form of voice, etc.). In response to receiving the user feedback information, the processing device 410B may obtain sensor data collected by at least two sample sensors as sample sensor data. In some embodiments, the processing device 410B may obtain an optical image of the hand of the sample user, and determine whether the gesture made by the sample user is consistent with the sample gesture to be imitated based on the optical image of the hand. For example, the processing device 410B may determine whether the difference between the optical image of the hand and the sample gesture to be imitated (for example, the average difference, overall difference, etc. of the distance between the optical image of the hand and each hand joint and / or finger joint in the sample gesture to be imitated) is less than a difference threshold, and in response to the difference being less than the difference threshold, it may be determined that the gesture made by the sample user is consistent with the sample gesture to be imitated. Further, the processing device 410B may obtain sensor data collected by at least two sample sensors as sample sensor data. In some embodiments, in response to determining that the gesture made by the sample user is inconsistent with the sample gesture to be imitated (for example, the difference is greater than a difference threshold), the processing device 410B may send a prompt signal to the sample user through the terminal device to remind and / or guide the sample user to adjust his gesture to make it consistent with the sample gesture.

[0270] In some embodiments of the present specification, by instructing the sample user to imitate the sample gesture before the target moment, the sample sensor data can be collected only after the sample user imitates the sample gesture, and then the sample joint angle can be determined directly based on the gesture model corresponding to the sample gesture, which simplifies the complexity of determining the sample joint angle and improves the efficiency of obtaining the sample joint angle. In addition, the above method is simple and easy to implement, and a large number of sample gestures (hand models) can be provided for the sample user to imitate, so that a large amount of sample sensor data and the corresponding sample joint angles can be obtained, thereby improving the accuracy and practicality of the determined mapping relationship.

[0271] In some embodiments, the processing device 410B may identify feature points corresponding to at least two sample finger joints from the hand optical image, and determine sample joint angles of the at least two sample finger joints based on the feature points.

[0272] The feature points corresponding to the sample finger joints refer to multiple points located on and near the sample finger joints identified from the hand optical image. For example, the feature points may include multiple points on the fingers where the sample finger joints are located. In some embodiments, the processing device 410B may identify the hand optical image based on various algorithms (such as image recognition algorithms, machine vision algorithms, etc.) to determine the feature points corresponding to the sample finger joints, and then determine the sample joint angles of at least two sample finger joints based on the feature points. Exemplarily, the processing device 410B may determine the bending direction and bending amplitude of the sample finger joints based on the orientation relationship or geometric relationship (such as relative position, direction, etc.) of the feature points corresponding to each sample finger joint, thereby obtaining the sample joint angle of the sample finger joint in the bending degree of freedom.

[0273] In some embodiments of the present specification, by identifying feature points corresponding to sample finger joints from a hand optical image to determine sample joint angles of at least two sample finger joints, the processing flow can be simplified and the processing efficiency can be improved.

[0274] It should be noted that the above method for determining the sample joint angle is only an exemplary description. In some embodiments, the sample joint angle can also be obtained by other methods. For example, the sample joint angle can be directly measured by a measuring tool (e.g., a joint ruler) and stored in a storage device (e.g., Figure 4 The processing device 410B may obtain the sample joint angles from the storage device 440 .

[0275] In step 1940 , the processing device 410B may determine a correspondence between each set of sample sensor data and its corresponding sample joint angle.

[0276] In this specification, there is a corresponding relationship between a set of sample sensor data and a hand optical image collected at the same target time, and there is a corresponding relationship between the set of sample sensor data and a sample joint angle determined based on the hand optical image.

[0277] In some embodiments, the process 1900 further includes step 1950. In step 1950, the processing device 410B may determine a mapping relationship between sensor data corresponding to at least two sample finger joints and joint angles based on the corresponding relationship.

[0278] Since at least two sample finger joints are Figure 6 The at least two target finger joints are of the same type, and the mapping relationship between the sensor data corresponding to the at least two sample finger joints and the joint angles can also be referred to as the first mapping relationship between the sensor data corresponding to the at least two target finger joints and the joint angles. For the convenience of description, the mapping relationship determined in step 1950 is referred to as the first mapping relationship below. The first mapping relationship can be represented in the form of a mapping function, a machine learning model, etc. For more description of the first mapping relationship, please refer to step 620.

[0279] In some embodiments, a set of sample sensor data and sample joint angles having a corresponding relationship can form a data sample. The first mapping relationship can be determined based on multiple data samples. For example, when the first mapping relationship is represented by a mapping function, it can be obtained based on multiple data samples through a data fitting algorithm. For another example, when the first mapping relationship is represented by a machine learning model, it can be obtained through model training based on multiple data samples. During the training process, the training input includes multiple sets of sample sensor data, and the training labels include sample joint angles corresponding to each set of sample sensor data.

[0280] In some embodiments, the processing device 410B may further determine a second mapping relationship (eg, Figure 6 The second mapping relationship can reflect the relationship between the joint angles of the proximal interphalangeal joint and the corresponding distal interphalangeal joint. For example, the processing device 410B can obtain the sample joint angles of the proximal interphalangeal joint and the sample joint angles of the corresponding distal interphalangeal joint, and determine the second mapping relationship through a data fitting algorithm or a machine learning algorithm. The second mapping relationship can be represented by a mapping function or a trained machine learning model.

[0281] In some embodiments, the processing device 410B may also determine a third mapping relationship (eg, Figure 6The third mapping relationship) can reflect the relationship between the joint angles of at least two metacarpophalangeal joints, the sensor data of at least two proximal interphalangeal joints, and the joint angles of at least two proximal interphalangeal joints. For example, the processing device 410B can obtain sample joint angles of at least two sample metacarpophalangeal joints, sample sensor data of at least two proximal interphalangeal joints, and sample joint angles of at least two proximal interphalangeal joints, and determine the third mapping relationship through a data fitting algorithm or a machine learning algorithm. The third mapping relationship can be represented by a mapping function or a trained machine learning model.

[0282] For more information about the first mapping relationship, the second mapping relationship, and the third mapping relationship, see Figure 6 and its description.

[0283] In some embodiments, the sample glove body includes a fabric wrapped around each sample finger of the sample user and a sample fabric located between adjacent sample fingers, wherein the elastic modulus of the sample fabric between adjacent sample fingers is less than the elastic modulus of the fabric on each sample finger. The characteristic parameters of the sample fabric between adjacent sample fingers can affect the strength of the coupling relationship between the finger joints of adjacent fingers. Thus, the processing device 410B can obtain sample values ​​of the characteristic parameters of the sample fabric between adjacent sample fingers, the characteristic parameters including at least one of the elastic coefficient, the lateral dimension, and the rebound coefficient, and further determine a first mapping relationship based on the sample values ​​of the characteristic parameters. The determined first mapping relationship can reflect the relationship between the characteristic parameters and the sensor data corresponding to at least two sample finger joints and the joint angles of at least two sample finger joints.

[0284] For more information about the fabric between adjacent fingers and its characteristic parameters, see Figure 6 and its description.

[0285] As an example only, when the first mapping relationship is represented by a machine learning model. The training input may include sensor data corresponding to at least two sample finger joints and sample values ​​of characteristic parameters of sample fabrics between adjacent sample fingers (such as elastic coefficient, lateral dimension, etc.), and the training label may include sample joint angles corresponding to the sample sensor data. During training, the value of the loss function can be determined based on the difference between the output of the initial machine learning model and the training label. The parameters of the initial machine learning model can be iteratively updated based on the value of the loss function until the training termination condition is met (such as the loss function converges, a specific number of iterations are performed, etc.), and an updated initial machine learning model (i.e., a trained machine learning model) is obtained. The trained machine learning model can be used as the first mapping relationship.

[0286] In some embodiments of the present specification, the influence of sample fabric features between adjacent sample fingers on the coupling relationship between finger joints is considered when determining the mapping relationship, so that the mapping relationship can be more in line with the actual situation and the joint angle determined using the mapping relationship can be more accurate.

[0287] In some embodiments, the coupling relationship between the sample finger joints is related to the distance between the sample finger joints. Thus, the mapping relationship can be further determined based on the sample distance information.

[0288] In some embodiments, the sample glove body may include a sample position sensor arranged at each sample finger joint, and the processing device 410B may determine the sample distance information between at least two sample strain sensors based on the position data collected by the sample position sensor at each sample finger joint, and further determine the first mapping relationship based on the sample distance information. The determined first mapping relationship may reflect the relationship between the sample distance between at least two sample strain sensors and the sensor data corresponding to at least two sample finger joints and the joint angles of at least two sample finger joints.

[0289] The sample position sensor refers to a position sensor arranged at the sample finger joint, which can be used to collect position data corresponding to each sample finger joint (ie, sample position data), and the sample distance represents the distance between two sample finger joints.

[0290] As an example only, when the first mapping relationship is represented by a machine learning model, the training input may include sample distances between at least two sample strain sensors and sensor data corresponding to at least two sample finger joints, and the training label may include sample joint angles corresponding to the sample sensor data.

[0291] In some embodiments of the present specification, considering the influence of the distance between finger joints on the coupling relationship between finger joints when determining the mapping relationship can make the mapping relationship more consistent with the actual situation and make the joint angle determined using the mapping relationship more accurate.

[0292] It should be noted that the above description of method 1900 is for illustrative purposes only and is not intended to limit the scope of this specification. For those of ordinary skill in the art, various changes and modifications can be made under the guidance of this specification. However, these changes and modifications do not depart from the scope of this specification. In some embodiments, method 1900 may be completed with one or more additional steps not described and / or without one or more steps discussed above. In addition, the order of method 1900 is not limiting. For example, method 1900 may include a step for correcting a mapping relationship. For another example, method 1900 may include a step for determining a joint angle using a mapping relationship. For another example, method 1900 may include a step for determining a hand model. For more description on determining a hand model, see Fig. 20 For another example, method 1900 may also include the step of displaying the hand model and receiving feedback information of the sample user on the hand model. For more descriptions on displaying the hand model and receiving feedback information of the sample user on the hand model, see Fig. 22 For another example, method 1900 may also include the step of scoring and screening the hand optical image. For more descriptions on scoring and screening the hand optical image, see Fig.21 The related descriptions will not be repeated here.

[0293] Fig. 20 is an exemplary flow chart of a method for determining a hand model according to some embodiments of the present specification. In some embodiments, method 2000 may be performed by processing device 410B. In some embodiments, at least a portion of step 1930 in method 1900 (e.g., determining a hand model based on an optical image of a hand) may be implemented by method 2000. Fig. 20 As shown, method 2000 may include the following steps.

[0294] In step 2010 , the processing device 410B may obtain an initial hand model.

[0295] The initial hand model may be a hand model that is pre-modeled or obtained from an external platform. For example, the processing device 410B may generate an initial hand model based on pre-collected hand data and / or images (such as hand optical images) of multiple arbitrary users using various image modeling techniques. For example, the processing device 410B may obtain an initial hand model from an external platform, and the initial hand model may be obtained based on hand data and / or image modeling of multiple arbitrary users. In some embodiments, the initial hand model may be a three-dimensional model (such as a 3D image model), or a hand frame model composed of a number of points and / or lines, which includes general features of a person's hand (such as the size of the palm, the length of the fingers, the texture, etc.) and hand posture information (such as the bending angle of the fingers and / or finger joints, etc.). In some embodiments, the initial hand model may be presented on a terminal device and may be adjusted (such as the position and shape of the fingers and / or finger joints may be adjusted) to generate a hand model consistent with the gestures made by the sample user.

[0296] In some embodiments, the initial hand model may include a universal hand model. The universal hand model refers to a pre-set basic hand model, which may be a hand model generated based on a hand action (such as any hand action). The processing device 410B may obtain the universal hand model from the storage device.

[0297] In some embodiments, the initial hand model may further include a dedicated hand model. The dedicated hand model refers to a hand model pre-set according to different preset gestures or actions. For example, the dedicated hand model may include a clenched fist hand model corresponding to a clenched fist action, a stretched hand model corresponding to a close and stretched action, an open hand model corresponding to an open palm action, etc.

[0298] In some embodiments, the dedicated hand model may also include a plurality of hand models pre-set in different preset gestures or actions for different users and / or user groups, for example, a male / female fist hand model, an elderly / child's outstretched hand model, etc.

[0299] In some embodiments, the processing device 410B may identify the hand optical image and retrieve or match a target dedicated hand model from multiple dedicated hand models as an initial hand model, wherein the hand movement corresponding to the target dedicated hand model has the highest similarity to the gesture of the sample user in the hand optical image.

[0300] In some embodiments of the present specification, a dedicated hand model can be used to reduce the workload of updating the initial hand model, thereby improving the efficiency of adjusting or updating the initial hand model through the hand optical image and reducing the loss of computing resources.

[0301] In some embodiments, the initial hand model can be generated using a generator in a trained generative adversarial network (GAN). For example, the processing device 410B can input optical images of the hand at multiple angles into a trained GAN model, and the generator of the trained GAN model outputs the initial hand model.

[0302] In some embodiments, the initial GAN ​​model can be trained based on multiple sets of training samples to obtain a trained generator and a trained discriminator. The multiple sets of training samples can include multiple sets of sample hand optical images and sample hand models corresponding to the sample hand optical images. Each set of sample hand optical images and its corresponding sample hand model can be obtained when the sample user makes the same gesture. Multiple sample hand models (or real samples) can be constructed as a real data set.

[0303] During training, multiple groups of sample hand optical images can be input into the initial generator respectively, and the initial generator outputs multiple intermediate hand models, and the initial discriminator distinguishes the authenticity of the multiple intermediate hand models and their corresponding sample hand models. The value of the loss function can be determined based on the difference between the intermediate hand model and the sample hand model and / or the discrimination result of the initial discriminator, and the parameters of the initial generator and / or the initial discriminator are iteratively updated accordingly, so that the initial discriminator can generate a hand model that is as close as possible to the sample hand model and sufficient to confuse the initial discriminator, and the initial discriminator can accurately distinguish the authenticity. The initial generator and the initial discriminator can be trained alternately or together.

[0304] The trained generator can be used to output a corresponding hand model based on the input hand optical image. The hand model output by the trained generator can be used as an initial hand model.

[0305] In step 2020, the processing device 410B may determine, based on the hand optical image and the initial hand model, whether a first deviation between the sample user's gesture and the sample gesture corresponding to the initial hand model is less than a deviation threshold.

[0306] The sample gesture refers to the hand movement corresponding to the initial hand model. The gesture of the sample user refers to the current hand movement of the sample user of the hand optical image (or called the current gesture or user gesture). In some embodiments, gestures (such as sample gestures, user gestures) can be represented in the form of gesture vectors. The gesture vector can be generated based on multiple hand features. Among them, the hand features include the features of the fingers in the gesture (such as length, width, thickness), the features of the finger joints (such as spatial position distribution, bending angle), and the features of the hand feature points (such as position, distance between multiple hand feature points, etc.). In some embodiments, gestures can be represented by hand feature points and their connecting lines. For example, the hand feature points may include multiple points corresponding to the positions of the wrist joints, finger joints, fingertips, etc. The processing device 410B can respectively determine the positions of the sample gesture and the user gesture in the same coordinate system (for example, a three-dimensional coordinate system) and determine the connecting lines of adjacent feature points, thereby obtaining the gesture representations of the sample gesture and the user gesture in the coordinate system respectively.

[0307] The first deviation refers to the difference between the sample user's gesture (i.e., the user gesture) and the sample gesture, and may also be referred to as the overall deviation between the user gesture and the sample gesture. For example, the first deviation may be the difference between gesture vectors (e.g., vector distance), the difference (distance) between feature points, the difference between joint angles, etc. The first deviation may be represented in various forms, such as a numerical value, where a larger value indicates a larger difference.

[0308] In some embodiments, the processing device 410B may determine the vector distance between the gesture vector corresponding to the sample gesture and the gesture vector corresponding to the user gesture based on various vector distance algorithms (such as Euclidean distance, cosine distance, etc.), and determine the value of the first deviation based on the vector distance. The smaller the value of the first deviation, the more similar the sample gesture is to the user gesture, that is, the smaller the first deviation. In some embodiments, the processing device 410B may determine the first deviation based on the distance between the corresponding hand feature points in the sample gesture and the user gesture. The corresponding hand feature points may refer to feature points in the sample gesture and the user gesture corresponding to the same hand position (e.g., the fingertip of the index finger).

[0309] In some embodiments, the value of the first deviation may be a statistical value (such as a sum, average, etc.) of the differences of the plurality of hand features. For example, the first deviation may be the average or sum of the distances between each hand feature point of the sample gesture and the hand feature point of its corresponding user gesture.

[0310] The processing device 410B may further determine whether a first deviation between the sample gesture and the user gesture is less than a deviation threshold, wherein the deviation threshold may be preset.

[0311] Step 2030, in response to determining that the first deviation is greater than the deviation threshold, the processing device 410B may update the initial hand model to obtain a hand model, wherein the second deviation between the sample gesture corresponding to the hand model and the gesture of the sample user is less than the deviation threshold.

[0312] In some embodiments, the processing device 410B may perform a model update process on the initial hand model. Model update process refers to performing one or more rounds of adjustment or fitting on the initial hand model so that the sample gesture is gradually adjusted to be close to (or similar to) the user gesture, thereby obtaining an updated hand model. The deviation (or second deviation) between the sample gesture and the user gesture in the updated hand model may be less than the deviation threshold.

[0313] In some embodiments, the model update processing includes adjusting one or more hand features (or feature points) corresponding to the sample gesture. For example, the processing device 410B can adjust the position and / or angle of the finger joints in the sample gesture by translating, rotating, or other operations on the positions of part or multiple voxel points of the initial hand model, so that the position and / or angle of the finger joints of the sample gesture corresponding to the updated initial hand model (i.e., the updated hand model) is close to the position and / or angle of the finger joints in the user gesture.

[0314] In some embodiments, in order to update the initial hand model, the processing device 410B may sequentially adjust the joint angles of the thumb, index finger, middle finger, ring finger, and little finger in the initial hand model. After each adjustment of the joint angle, it is determined whether the deviation (or third deviation) between the sample gesture corresponding to the adjusted hand model and the gesture of the sample user is less than the deviation threshold; in response to determining that the third deviation is less than or equal to the deviation threshold, the adjustment of other fingers is stopped, and the adjusted hand model is used as the updated hand model; or in response to determining that the third deviation is greater than the deviation threshold, the joint angles of other fingers continue to be adjusted.

[0315] In some embodiments of the present specification, by adjusting the fingers in the initial hand model in sequence and stopping adjusting other fingers after determining that the third deviation is less than or equal to the deviation threshold, the adjustment steps can be simplified while ensuring that the sample gestures in the updated hand model are consistent with the user gestures, thereby improving the adjustment efficiency.

[0316] In some embodiments, in order to update the initial hand model, for each finger of the sample user, the processing device 410B may determine whether the deviation between the finger and its corresponding finger in the initial hand model (or referred to as the fourth deviation or local deviation) is less than a second deviation threshold; in response to determining that the fourth deviation is greater than the second deviation threshold, the finger is adjusted; or in response to determining that the fourth deviation is less than or equal to the second deviation threshold, the processing device 410B may determine that there is no need to adjust the finger.

[0317] The fourth deviation refers to the difference between a certain finger of the sample gesture and the user gesture, for example, the difference between the gesture vectors corresponding to the finger (for example, vector distance), the difference (distance) between the feature points on the finger, the difference between the joint angles on the finger, etc. For example, taking the index finger as an example, the value of the fourth deviation is used to reflect the difference between the angle of the metacarpophalangeal joint of the index finger of the sample gesture and the angle of the metacarpophalangeal joint of the index finger of the user gesture.

[0318] The second deviation threshold refers to a threshold value regarding the difference between the sample gesture and each finger in the user gesture. In some embodiments, the second deviation thresholds corresponding to different fingers may be the same, thereby simplifying the calculation process. In some embodiments, the second deviation thresholds corresponding to different fingers may be different, thereby enabling the corresponding deviation thresholds to be set according to the characteristics of different fingers, making the adjustment process more targeted and the adjustment result more accurate. In some embodiments, the second deviation threshold is less than the first deviation threshold.

[0319] In some embodiments of the present specification, by updating each finger and / or finger joint, a comprehensive update of each finger can be achieved, so that the posture between the sample gesture and each finger of the user gesture remains consistent (less than a second deviation threshold), which can make the adjustment result more accurate.

[0320] Some embodiments of the present specification implement dual-path optical recognition and adjustment of the hand model by using a hand model and an optical image. On the one hand, sample joint angles can be obtained from a hand model that contains more posture information, so that the obtained sample joint angles are more accurate; on the other hand, the initial hand model is generated in advance and only needs to be appropriately adjusted according to the optical image. In this way, a large number of corresponding data sets of sensors and joint angles can be quickly provided, thereby improving the accuracy and practicality of the determined mapping relationship.

[0321] Fig.21 is an exemplary flow chart of a method for determining a sample joint angle according to some embodiments of the present specification. In some embodiments, method 2100 may be performed by processing device 410B. In some embodiments, step 1930 in method 1900 may be implemented by method 2100. Fig.21As shown, method 2100 may include the following steps.

[0322] In some embodiments, the acquired hand optical image may include multiple hand optical images taken at different shooting angles. For example, the hand optical image may be multiple hand optical images taken at different shooting angles at the target time.

[0323] In step 2110 , the processing device 410B may determine an evaluation score for each of the plurality of optical hand images.

[0324] When the sample users make different gestures, the postures of their hands and the positions of different fingers may be different. The hand optical image taken at a certain shooting angle may have some finger joints blocked. Therefore, multiple hand optical images can be taken at different shooting angles and the evaluation score of each hand optical image can be determined to select the hand optical image that meets the requirements. The evaluation score can be used to evaluate the quality of the hand optical image (for example, the amount of hand information contained), which can be expressed in the form of a numerical value (such as a numerical value in the interval [0,10]). The larger the value, the better it is and the more it can be used as the target hand optical image.

[0325] In some embodiments, the evaluation score is related to the degree of occlusion of one or more sample finger joints in the hand optical image, wherein the degree of occlusion of a certain sample finger joint can reflect whether the sample finger joint is completely visible. For example, at a certain shooting angle, if a certain sample finger joint (such as the metacarpophalangeal joint or interphalangeal joint corresponding to the sample middle finger) in a certain hand optical image is not visible, the higher the degree of occlusion, and accordingly, the lower the evaluation score of the hand optical image.

[0326] In some embodiments, the processing device 410B may acquire multiple hand optical images taken at different shooting angles, and evaluate each hand optical image separately to determine the degree of occlusion of one or more sample finger joints, thereby obtaining an evaluation score for each hand optical image.

[0327] In some embodiments, the evaluation score may also be related to other evaluation indicators. For example, exemplary evaluation indicators may include but are not limited to the signal-to-noise ratio and clarity of the hand optical image. For example, the greater the signal-to-noise ratio and the higher the clarity of the hand optical image, the higher the evaluation score.

[0328] In some embodiments, the processing device 410B may determine the evaluation score of each hand optical image based on a trained scoring model. The scoring model may be a convolutional neural network (CNN) model or other deep neural network model. The input of the scoring model includes the hand optical image, and the output includes the evaluation score corresponding to the hand optical image. In some embodiments, supervised learning may be used to iteratively train the initial scoring model to obtain a trained scoring model. The training samples of the initial scoring model may include sample hand optical images collected historically (such as multiple sample hand optical images at different sample shooting angles), and the label of each training sample is the evaluation score corresponding to the sample hand optical image, which may be annotated manually or in other feasible ways.

[0329] In step 2120 , the processing device 410B may select a target hand optical image from the plurality of hand optical images based on the evaluation score of each hand optical image.

[0330] The target hand optical image refers to a hand optical image selected from multiple hand optical images for subsequent processing. In some embodiments, the processing device 410B can determine the evaluation score of each hand optical image respectively, and use the hand optical image with the highest evaluation score (such as the highest evaluation score) as the target hand optical image. In some embodiments, the target hand optical image can be one or more.

[0331] In step 2130 , the processing device 410B may determine sample joint angles of at least two sample finger joints of a corresponding sample user based on the target hand optical image.

[0332] The method of determining the sample joint angle based on the target hand optical image is similar to the process of determining the sample joint angle of the sample finger joint based on the hand optical image described in step 1930, and will not be repeated here.

[0333] In some embodiments of the present specification, by selecting a suitable (such as the one with the highest evaluation score) hand optical image to determine the sample joint angles of the sample finger joints, the subsequent determination of the sample joint angles can be made more accurate, while also improving the efficiency of the calculation.

[0334] Fig. 22 is an exemplary flow chart of another method for determining a sample joint angle according to some embodiments of the present specification. In some embodiments, method 2200 may be performed by processing device 410B. In some embodiments, step 1930 in method 1900 may be implemented by method 2200. Fig. 22 As shown, method 2200 may include the following steps.

[0335] In step 2210 , the processing device 410B may determine a hand model based on the hand optical image.

[0336] For example, the processing device 410B may obtain an initial hand model, and based on the hand optical image and the initial hand model, determine whether a first deviation between a sample user's gesture and a sample gesture corresponding to the initial hand model is less than a deviation threshold, and update the initial hand model based on the determination result, thereby determining the hand model. For relevant contents about the hand model and determining the hand model based on the hand optical image, see Fig. 20 And its description will not be repeated here.

[0337] In step 2220, the processing device 410B may control the terminal device to display the hand model to the sample user.

[0338] For example, the processing device 410B may control the terminal device (eg, Figure 1 The terminal 430 shown in the figure displays the hand model. The sample user can judge whether her / his own gesture is consistent with the sample gesture corresponding to the hand model based on the displayed hand model. In some embodiments, the processing device 410B can also control the terminal device to display the sample user's gesture to the sample user. For example, the sample user's gesture and the hand model can be represented in the form of an image or feature points and their connecting lines. The processing device 410B can control the terminal device to simultaneously display the sample user's gesture and the representation corresponding to the hand model. The sample user can judge whether the sample user's gesture is consistent with the sample gesture corresponding to the hand model based on the displayed hand model and the sample user's gesture.

[0339] In step 2230, the processing device 410B may receive feedback information input by the sample user via the terminal device, where the feedback information indicates whether the gesture of the sample user is consistent with the sample gesture corresponding to the hand model.

[0340] The feedback information includes feedback information on the consistency of the gesture made by the sample user (i.e., the user gesture) with the sample gesture, which may include but is not limited to one or a combination of various forms such as voice and text. For example, the feedback information may indicate that the sample user's gesture is consistent (or inconsistent) with the sample gesture corresponding to the hand model. In some embodiments, the feedback information may also include information related to the consistency of all or part of the fingers and / or finger joints in the user gesture. For example, when the sample user's gesture is inconsistent with the sample gesture corresponding to the hand model, the feedback information may specifically indicate information such as the inconsistent bending angle of the index finger, the inconsistent angle between the thumb and the index finger, and / or adjustment instructions for adjusting the hand model.

[0341] In step 2240 , the processing device 410B may determine the sample joint angles based on the hand model based on the feedback information, or update the hand model and determine the sample joint angles based on the updated hand model.

[0342] For example, if the feedback information indicates that the sample user's gesture and the sample gesture corresponding to the hand model are consistent, the processing device 410B can directly determine the sample joint angle based on the hand model; if the feedback information indicates that the sample user's gesture and the sample gesture corresponding to the hand model are inconsistent, the processing device 410B can update the hand model and determine the sample joint angle based on the updated hand model.

[0343] In some embodiments, the terminal device may include an interactive interface for adjusting the sample gesture. For example, the sample user (or other user) may adjust the bending angle of one or more fingers (such as the index finger, middle finger, etc.) through the interactive interface so that the sample gesture displayed by the terminal device is consistent with the posture (such as position, bending angle, etc.) of the fingers in the user's gesture.

[0344] In some embodiments, the processing device 410B can also automatically update the hand model displayed by the terminal device based on the feedback information to obtain a hand model consistent with the user's gesture. For example, the processing device 410B can automatically update the hand model based on the method described in steps 2020-2030 in method 2000. For another example, the processing device 410B can process the feedback information based on technologies such as speech recognition and text recognition to determine adjustment instructions for updating the hand model. The adjustment instructions can be used to adjust the hand features of the sample gesture. For example, the position adjustment (such as translation) and posture adjustment (such as bending amplitude, direction, etc.) of the fingers and / or finger joints in the hand model. The processing device 410B can further determine the sample joint angle based on the automatically updated hand model.

[0345] In some embodiments of the present specification, adjustments to the hand model based on feedback from sample users can further improve the conformity of the hand model and thereby improve the accuracy of the determined sample joint angles.

[0346] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0347] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0348] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0349] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0350] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of this specification are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

[0351] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification are hereby incorporated by reference in their entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this specification (currently or later attached to this specification) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.

[0352] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A method for determining a finger joint angle, comprising: Acquire first sensor data related to at least two target finger joints of a user, wherein the first sensor data is acquired by using at least two strain sensors, each of the strain sensors being arranged in a glove body worn by the user and located at one of the at least two target finger joints, wherein the at least two target finger joints include at least two adjacent metacarpophalangeal joints of the user; Acquire a first mapping relationship, where the first mapping relationship reflects the relationship between the sensor data corresponding to the at least two target finger joints and the joint angles; and Based on the first sensor data and the first mapping relationship, a first joint angle of each target finger joint is determined. 2 . The method according to claim 1 , wherein the at least two target finger joints include partial finger joints of the user. 3 . The method according to claim 1 , wherein the at least two target finger joints include all metacarpophalangeal joints and all proximal interphalangeal joints of the user.

4. The method according to claim 1, wherein the at least two strain sensors include at least two first strain sensors located at the at least two metacarpophalangeal joints, and each of the first strain sensors is configured to measure deformation of its corresponding metacarpophalangeal joint in two degrees of freedom.

5. The method according to claim 1, characterized in that: The at least two target finger joints further include at least two proximal interphalangeal joints corresponding to the at least two metacarpophalangeal joints, and the at least two strain sensors include at least two second strain sensors located at the at least two proximal interphalangeal joints, each of the second strain sensors being configured to measure the deformation of the corresponding proximal interphalangeal joint in a single degree of freedom.

6. The method according to claim 5, characterized in that Further including: For each of the proximal interphalangeal joints, the second joint angle of the distal interphalangeal joint corresponding to the proximal interphalangeal joint is determined based on the second mapping relationship and the first joint angle of the proximal interphalangeal joint, and the second mapping relationship reflects the relationship between the joint angles of the proximal interphalangeal joint and the distal interphalangeal joint.

7. The method according to claim 1, characterized in that The first sensor data includes decoupling data of the at least two target finger joints, and the decoupling data is determined based on raw data collected by the at least two strain sensors.

8. The method according to claim 1, characterized in that: The determining the first joint angle of each target finger joint based on the first sensor data and the first mapping relationship comprises: Acquire reference sensor data related to at least two target finger joints of the user, where the reference sensor data is collected by using the at least two strain sensors when the user makes a preset gesture; Based on the reference sensor data, correcting the first mapping relationship to obtain a corrected mapping relationship corresponding to the user; and Based on the correction mapping relationship and the first sensor data, a first joint angle of each target finger joint is determined.

9. The method according to claim 1, characterized in that: The at least two target finger joints only include at least two adjacent metacarpophalangeal joints of the user, and the method further includes: Acquire second sensor data related to at least two proximal interphalangeal joints corresponding to the at least two metacarpophalangeal joints, the second sensor data being acquired by using at least two second strain sensors, each of the second strain sensors being arranged in the glove body worn by the user and located at one of the at least two proximal interphalangeal joints; and Based on the second sensor data, the first joint angles of the at least two metacarpophalangeal joints and the third mapping relationship, the third joint angle of each of the proximal interphalangeal joints is determined, and the third mapping relationship reflects the relationship between the joint angles of the at least two metacarpophalangeal joints, the sensor data of the at least two proximal interphalangeal joints and the joint angles of the at least two proximal interphalangeal joints.

10. The method according to claim 1, characterized in that The glove body includes a cloth wrapped around each finger of the user and a cloth located between adjacent fingers, wherein the elastic modulus of the cloth between adjacent fingers is smaller than the elastic modulus of the cloth on each finger, The method further includes obtaining a value of a characteristic parameter of the cloth between the adjacent fingers, wherein the characteristic parameter includes at least one of an elastic coefficient, a lateral dimension, and a rebound coefficient. The first mapping relationship further reflects the relationship between the characteristic parameter and the sensor data corresponding to the at least two target finger joints and the joint angles of the at least two target finger joints. The first joint angle of each of the target finger joints is further determined based on the value of the feature parameter.

11. The method according to claim 1, characterized in that: The glove body includes a position sensor arranged at each of the target finger joints, The method further comprises determining distance information between the at least two strain sensors based on position data collected by a position sensor at each of the target finger joints, The first mapping relationship further reflects the relationship between the distance between the at least two strain sensors and the sensor data corresponding to the at least two target finger joints and the joint angles of the at least two target finger joints. The first joint angle of each of the target finger joints is further determined based on the distance information.

12. The method according to claim 1, characterized in that The first mapping relationship is determined in the following manner: Acquire multiple data samples, each of which corresponds to a sample user performing a sample gesture and includes sample sensor data related to at least two sample finger joints of the sample user and sample joint angles of the at least two sample finger joints, wherein the sample sensor data is acquired by using at least two sample sensors arranged in a sample glove body worn by the sample user, and the at least two sample finger joints are of the same type as the at least two target finger joints; Based on the multiple data samples, the first mapping relationship is determined through a data fitting algorithm or a machine learning algorithm.

13. The method according to claim 12, characterized in that At least one of the data samples is obtained based on: For each data sample of the at least one data sample, Acquire sensor data collected by the at least two sample sensors when the sample user performs the sample gesture as sample sensor data of the data sample; Acquire a hand optical image captured when the sample user performs the sample gesture; Based on the hand optical image, sample joint angles of the data samples are determined.

14. The method according to claim 12, characterized in that At least one of the data samples is obtained based on: For each data sample of the at least one data sample, Controlling the terminal device to display the hand model corresponding to the sample gesture to the sample user corresponding to the data sample; acquiring sensor data collected by the at least two sample sensors when the sample user imitates the sample gesture as sample sensor data of the data sample; Based on the hand model, sample joint angles of the data samples are determined.

15. The method according to claim 14, characterized in that The acquiring of sensor data collected by the at least two sample sensors when the sample user imitates the sample gesture, as the sample sensor data of the data sample, includes: Acquire an optical hand image of the sample user taken when the sample user imitates the sample gesture; Based on the hand optical image, determining whether the gesture of the sample user is the sample gesture; In response to determining that the gesture of the sample user is the sample gesture, the sensor data currently collected by the at least two sample sensors is obtained as sample sensor data; or in response to determining that the gesture of the sample user is not the sample gesture, the terminal device is controlled to display prompt information for adjusting the gesture to the sample user.

16. The method according to claim 1, characterized in that The first mapping relationship is represented by a mapping function, the glove body includes a position sensor arranged at each of the target finger joints, The determining the first joint angle of each target finger joint based on the first sensor data and the first mapping relationship comprises: Determining distance information between the at least two strain sensors based on position data collected by a position sensor at each of the target finger joints; Correcting the mapping function based on the distance information to obtain a corrected mapping function corresponding to the user; and Based on the correction mapping function and the first sensor data, a first joint angle of each of the target finger joints is determined.

17. A system for determining finger joint angles, comprising: at least one storage medium including a set of instructions; as well as At least one processor in communication with the at least one storage medium, wherein when executing the set of instructions, the at least one processor causes the system to perform the method of any one of claims 1 to 16.

18. A non-transitory computer-readable medium comprising executable instructions, which when executed by at least one processor cause the at least one processor to perform the method of any one of claims 1 to 16.

Images