Action data reconstruction method and system, device and medium

Abstract

The present application belongs to the technical field of action data reconstruction, and provides an action data reconstruction method and system, a device and a medium. The method comprises: acquiring initial action data of a target subject, the initial action data having a first degree of freedom, and the first degree of freedom being less than the actual degree of freedom of the target subject; acquiring a preset target measurement matrix, the target measurement matrix being obtained by iteratively updating residuals calculated in advance on the basis of reference action data and sample action data, the reference action data having a second degree of freedom, the second degree of freedom being determined on the basis of the actual degree of freedom of the target subject, and the second degree of freedom being greater than the first degree of freedom; and, on the basis of the target measurement matrix, performing data reconstruction on the initial action data, so as to obtain target reconstructed data of the target subject, the target reconstructed data having a third degree of freedom, and the third degree of freedom corresponding to the second degree of freedom. By means of compressed sensing, the present application reconstructs initial action data to data having a high degree of freedom, thus improving the accuracy of action data.

Description

A motion data reconstruction method, system, device and medium Technical Field

[0001] The present application relates to the field of motion data reconstruction technology, and in particular to a motion data reconstruction method, system, device and medium. Background Art

[0002] Action reconstruction usually refers to a task in the field of computer vision and motion capture. Its goal is to reconstruct the human body's motion trajectory from video or other sensor data. It has a wide range of applications, including posture analysis, interactive games, motion control and other fields.

[0003] Sensors are needed to collect data from human limbs and reconstruct movements. However, in related technologies, the space available for sensor placement is limited, making it impossible to install sensors on every joint of the user. Consequently, the number of sensors is limited, resulting in the sensors' detection freedom being lower than the user's actual freedom of movement, making it impossible to obtain accurate movement data.

[0004] Summary of the Invention

[0005] The main purpose of the embodiments of the present application is to provide a motion data reconstruction method, system, device and medium, which can improve the accuracy of motion data.

[0006] To achieve the above-mentioned purpose, the first aspect of an embodiment of the present application proposes a method for reconstructing motion data, the method comprising: obtaining initial motion data of a target object, wherein the initial motion data has a first degree of freedom, which is smaller than the actual degree of freedom of the target object; obtaining a preset target measurement matrix, wherein the target measurement matrix is ​​obtained after iterative updating of residuals calculated in advance based on reference motion data and sample motion data, and the reference motion data has a second degree of freedom, which is determined based on the actual degree of freedom of the target object, and the second degree of freedom is greater than the first degree of freedom; reconstructing the initial motion data based on the target measurement matrix to obtain target reconstructed data of the target object, wherein the target reconstruction data has a third degree of freedom, which corresponds to the second degree of freedom.

[0007] In some embodiments, obtaining a preset target measurement matrix includes: obtaining the reference action data and a plurality of the sample action data, and establishing an initial measurement matrix; selecting any one of the sample action data in a current iteration cycle, inputting the sample action data into the initial measurement matrix for signal reconstruction, and obtaining sample reconstruction data; obtaining the residual corresponding to the current sample action data based on the difference between the reference action data and the sample reconstruction data; adjusting the current initial measurement matrix based on the residual, and continuing to calculate the residual corresponding to the next sample action data in a next iteration cycle, iteratively updating the measurement matrix based on the next residual until the number of iterations reaches an iteration threshold, and obtaining the target measurement matrix.

[0008] In some embodiments, the steps for obtaining the sample reconstruction data in any iterative cycle are as follows, including: obtaining the residual obtained in the previous iterative cycle and the measurement matrix in the current iterative cycle; under preset relaxation parameters, combining the residual of the previous iterative cycle and the measurement matrix in the current iterative cycle, projecting the sample action data in the current iterative cycle to obtain projection data; optimizing the projection data through a preset iterative threshold function to obtain optimized new sample action data; inputting the optimized new sample action data into the measurement matrix in the current iterative cycle for signal reconstruction to obtain sample reconstruction data.

[0009] In some embodiments, the projection data is optimized by a preset iterative threshold function to obtain the optimized new sample action data, including: if the iterative threshold function is a hard threshold function, the projection data is filtered according to the size relationship between the projection data and the hard threshold corresponding to the hard threshold function to obtain the optimized new sample action data; if the iterative threshold function is a soft threshold function, the sparse solution of the projection data is calculated by the soft threshold function to obtain the optimized new sample action data.

[0010] In some embodiments, obtaining the initial motion data of the target object includes: obtaining the initial motion data sent by a detection device, wherein the detection device is provided with multiple sensors, the detection device is used to be worn on the target object, and the sensors are used to obtain motion data around the joints corresponding to the target joint detection positions on the target object, and use the data as the initial motion data; or, determining multiple target joint detection positions on the target object, and collecting motion data around the corresponding joints based on the multiple target joint detection positions, and use the data as the initial motion data.

[0011] In some embodiments, if the target object is a hand, determining multiple target joint detection positions on the target object includes: determining multiple joint detection positions on the target object; determining a first joint detection position for detecting the opening and closing of the joint between the distal phalanx and the proximal phalanx of the thumb, the opening and closing of the joint between the proximal phalanx and the metacarpal bone of the thumb, and / or the abduction and adduction of the joint between the metacarpal bone and the trapezium of the thumb from the multiple joint detection positions, and also determining a second joint detection position for detecting the opening and closing of the joint between the proximal phalanx and the metacarpal bones of other fingers other than the thumb, a third joint detection position for detecting the abduction and adduction of the joint between the proximal phalanx and the metacarpal bones of other fingers other than the thumb, and a fourth joint detection position for detecting the opening and closing of the joint between the proximal phalanx and the middle phalanx of other fingers other than the thumb; and using multiple of the first joint detection position, the second joint detection position, the third joint detection position, and the fourth joint detection position as target joint detection positions.

[0012] In some embodiments, the reference motion data is obtained by the following steps, including: obtaining reference motion data sent by a detection device, wherein the detection device is provided with multiple sensors, and the detection device is used to be worn on a target object, and the sensors are used to obtain motion data around the joints corresponding to the target joint reference positions on the target object, and use them as the reference motion data; or, determining multiple target joint reference positions on the target object, and collecting motion data around the corresponding joints based on the multiple target joint reference positions, and use them as reference motion data; wherein the number of target joint reference positions is greater than the number of target joint detection positions.

[0013] In some embodiments, if the target object is a hand, determining multiple target joint reference positions on the target object includes: determining multiple joint reference positions on the target object; determining a first joint reference position for detecting the opening and closing between the distal phalanx and the proximal phalanx on the thumb, a second joint reference position for opening and closing between the proximal phalanx and the metacarpal, a third joint reference position for abduction and adduction between the proximal phalanx and the metacarpal, a fourth joint reference position for opening and closing between the metacarpal and the trapezium, and a fifth joint reference position for abduction and adduction between the metacarpal and the trapezium from the multiple joint detection positions; determining a joint reference position for detecting the opening and closing between the distal phalanx and the proximal phalanx on the thumb from the multiple joint detection positions. Measure the sixth joint reference position of opening and closing between the distal phalanx and the middle phalanx, the seventh joint reference position of opening and closing between the middle phalanx and the proximal phalanx, the eighth joint reference position of opening and closing between the proximal phalanx and the metacarpal, and the ninth joint reference position of abduction and adduction between the proximal phalanx and the metacarpal on each of the other fingers; and use the first joint reference position, the second joint reference position, the third joint reference position, the fourth joint reference position, the fifth joint reference position, the sixth joint reference position, the seventh joint reference position, the eighth joint reference position and the ninth joint reference position as target joint detection positions.

[0014] In some embodiments, obtaining the initial motion data of the target object includes: obtaining the original initial motion data of the target object, wherein the original initial motion data is a dimensionless value used to characterize the motion of the target object; calibrating the original initial motion data to obtain the initial motion data, wherein the initial motion data is a dimensioned value used to characterize the motion of the target object.

[0015] In some embodiments, reconstructing the initial action data based on the target measurement matrix to obtain target reconstructed data of the target object includes: reconstructing the initial action data based on the target measurement matrix to obtain initial reconstructed data; and filtering the initial reconstructed data to obtain target reconstructed data of the target object.

[0016] In some embodiments, the target reconstruction data includes multiple degree-of-freedom data under the third degree of freedom; after obtaining the target reconstruction data of the target object, it also includes: obtaining a preset kinematic model, wherein the kinematic model has multiple nodes, and each node corresponds to one degree-of-freedom data; inputting the target reconstruction data into the kinematic model, and binding each degree-of-freedom data in the target reconstruction data to the corresponding node on the kinematic model, and mapping the motion image of the target object through the kinematic model after binding the data.

[0017] In some embodiments, after mapping the motion image of the target object through the kinematic model after binding the data, it also includes: generating a virtual image of the target object based on the motion image, and displaying the virtual image; in response to the interactive operation on the virtual image, displaying the corresponding interactive display interface.

[0018] To achieve the above-mentioned purpose, the second aspect of an embodiment of the present application proposes an action data reconstruction system, which includes: an action data acquisition module, used to obtain initial action data of a target object, wherein the initial action data has a first degree of freedom, which is smaller than the actual degree of freedom of the target object; a measurement matrix acquisition module, used to obtain a preset target measurement matrix, wherein the target measurement matrix is ​​obtained after iterative updating of residuals calculated in advance based on reference action data and sample action data, and the reference action data has a second degree of freedom, which is determined based on the actual degree of freedom of the target object, and the second degree of freedom is greater than the first degree of freedom; an action data reconstruction module, used to reconstruct the initial action data based on the target measurement matrix to obtain target reconstructed data of the target object, wherein the target reconstruction data has a third degree of freedom, which corresponds to the second degree of freedom.

[0019] To achieve the above-mentioned purpose, the third aspect of the embodiments of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the action data reconstruction method described in the first aspect embodiment when executing the computer program.

[0020] To achieve the above-mentioned purpose, the fourth aspect of the embodiment of the present application proposes a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by the processor, it implements the action data reconstruction method described in the embodiment of the first aspect above.

[0021] The embodiments of the present application include at least the following beneficial effects: the initial motion data is processed by a preset target measurement matrix, wherein the first degree of freedom of the initial motion data is low and lower than the actual degree of freedom, and the target measurement matrix is ​​obtained after iterative updating of the residuals calculated in advance based on the reference motion data and the sample motion data, the reference motion data has a second degree of freedom, and the second degree of freedom is determined based on the actual degree of freedom of the target object, through continuous iterative updating, the sample motion data is reconstructed and close to the reference motion data, and the established target measurement matrix can reconstruct the input low-degree-of-freedom motion data into high-degree-of-freedom data, and finally obtain target reconstructed data with a third degree of freedom, and the third degree of freedom corresponds to the second degree of freedom, so that the initial motion data is reconstructed into high-degree-of-freedom data through compressed sensing, which can more accurately reflect the user's actual motion, thereby improving the accuracy of the motion data. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1 is a schematic diagram of an implementation environment of a motion data reconstruction system provided in an embodiment of the present application;

[0023] FIG2 is a flow chart of a method for reconstructing motion data according to an embodiment of the present application;

[0024] FIG3 is a schematic flow chart of step S102 in FIG2 ;

[0025] FIG4 is a schematic diagram of a process for obtaining sample reconstruction data within any iterative cycle provided by an embodiment of the present application;

[0026] FIG5 is a schematic flow chart of step S303 in FIG4 ;

[0027] FIG6 is a schematic flow chart of step S101 in FIG2 ;

[0028] FIG7 is a schematic diagram of a process of determining a target joint detection position in step S502 in FIG6 ;

[0029] FIG8 is a schematic diagram of various joint detection positions set on a motion capture glove provided in an embodiment of the present application;

[0030] FIG9 is a flow chart of a process for obtaining reference action data according to an embodiment of the present application;

[0031] FIG10 is a schematic diagram of a process of determining a target joint reference position in step S702 in FIG9 ;

[0032] FIG11 is a schematic diagram of hand bones provided in an embodiment of the present application;

[0033] FIG12 is a schematic diagram of a flow chart of the calibration process included in step S101 in FIG2 ;

[0034] FIG13 is a schematic diagram of a flow chart of filtering processing included in step S103 in FIG2 ;

[0035] FIG14 is a schematic diagram of the process after step S103 in FIG2 ;

[0036] FIG15 is a schematic diagram of the process after step S1102 in FIG14 ;

[0037] FIG16 is a schematic diagram of functional modules of a motion data reconstruction system according to an embodiment of the present application;

[0038] FIG17 is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0040] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0042] First, let’s analyze some of the terms used in this application:

[0043] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0044] Compressed sensing (CS), also known as compressive sampling or sparse sampling, is a technique for finding sparse solutions to underdetermined linear systems. CS is used in electronic engineering, particularly signal processing, to acquire and reconstruct sparse or compressible signals.

[0045] Before realizing motion reconstruction, it is necessary to obtain the user's motion data. The acquisition of the user's motion data needs to be collected through sensors. The usual processing method is to use the data of a single sensor to correspond one-to-one to the angle of a single joint / contraction and relaxation of a single muscle. Specifically, taking the collection of hand motion data as an example, the existing technology usually sets multiple sensors to correspond to the multiple hand joints to be detected. For example, when it is necessary to detect the movements of the fingertip joints and metacarpophalangeal joints, a sensor is set at each interphalangeal joint and metacarpophalangeal joint. This brings about the problems of complex hardware structure and high cost. Therefore, in order to control costs, it is necessary to reduce the number of sensors.

[0046] Moreover, in related technologies, the space where sensors can be set is limited, and it is impossible to set sensors on every joint of the user. Therefore, the number of sensors is limited, resulting in the detection freedom that the sensors can achieve being lower than the actual movement freedom of the user, and thus it is impossible to obtain accurate movement data.

[0047] Based on this, embodiments of the present application provide a method, system, device, and medium for reconstructing motion data, which can improve the accuracy of motion data.

[0048] Refer to Figure 1, which is a schematic diagram of an optional implementation environment provided by an embodiment of the present application. The implementation environment includes a data detection device and a processing device. The processing device can be a terminal and a server, wherein the detection device and the processing device are communicatively connected, and the processing device can receive motion data collected by the detection device. The detection device and the processing device can be directly or indirectly connected via wired or wireless communication, and the embodiment of the present application does not limit this.

[0049] Furthermore, the processing device in FIG1 may also be connected to a display device to display the processed motion image.

[0050] A server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Furthermore, a server can be a node server in a blockchain network.

[0051] The terminal can be a virtual reality device, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, etc., but is not limited to these.

[0052] The detection device can be equipped with any type of sensor to collect user motion data. In one embodiment, the detection device can be equipped with an accelerometer, a gyroscope, a magnetometer, a flexible sensor, a pressure sensor, a camera, and an ultrasonic sensor. In the embodiment of the present application, a flexible strain sensor made of a stretchable conductor is used to make the detection device. The principle is to set a flexible strain sensor on an elastic fabric. When the user wears the above-mentioned motion capture device and makes a corresponding movement, the flexible strain sensor is pulled and detects the direction and degree of strain. Based on the data of a single or multiple flexible strain sensors, the movement of one or more parts of the human body can be mapped.

[0053] Exemplarily, the processing device can obtain motion data collected by the detection device, such as initial motion data, and the processing device processes the initial motion data through a preset target measurement matrix, wherein the first degree of freedom of the initial motion data is low and lower than the actual degree of freedom, and the target measurement matrix is ​​obtained after iterative updating of the residuals calculated in advance based on the reference motion data and the sample motion data. The reference motion data has a second degree of freedom, and the second degree of freedom is determined based on the actual degree of freedom of the target object. Through continuous iterative updating, the sample motion data is reconstructed and close to the reference motion data. The established target measurement matrix can reconstruct the input low-degree-of-freedom motion data into high-degree-of-freedom data, and finally obtain target reconstructed data with a third degree of freedom. The third degree of freedom corresponds to the second degree of freedom, so that the initial motion data is reconstructed into high-degree-of-freedom data through compressed sensing, which can more accurately reflect the user's actual motion, thereby improving the accuracy of the motion data.

[0054] The method provided in the embodiments of the present application can be applied to different technical fields, including but not limited to cloud technology, artificial intelligence, motion data reconstruction and other technical fields.

[0055] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. For example, when obtaining data stored by the user, the user's permission or consent will be obtained first; when obtaining action data, the embodiment of the present application will first obtain the user's permission or consent. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the embodiment of the present application to operate normally will be obtained.

[0056] FIG2 is an optional flowchart of the motion data reconstruction method provided in an embodiment of the present application. The method in FIG2 may include but is not limited to steps S101 to S103.

[0057] Step S101, obtaining initial motion data of a target object, wherein the initial motion data has a first degree of freedom, which is smaller than an actual degree of freedom of the target object;

[0058] Exemplarily, the target object can be any object that can perform an action, for example, a user, a robot, or an animal. During the movement of a user or animal, the main body or limbs can perform corresponding movements, and the motion data generated by these movements can be used as initial motion data. During the movement of a robot, the bionic robotic arm can perform corresponding movements, and the motion data generated by these movements can be used as initial motion data. While the target object in the embodiments of the present application is described as a user, this does not constitute a limitation on the embodiments of the present application.

[0059] Exemplarily, the initial motion data is the motion data that needs to be processed. The initial motion data may be motion data detected by a sensor. The motion data may represent the motion situation. For example, it may be motion data such as the angle, position, height, etc. of the motion. No specific limitation is made here.

[0060] Exemplarily, the first degree of freedom is the number of degrees of freedom of the initial motion data, and the degree of freedom refers to the number of directions in which an object can freely change in a certain motion mode. For example, in 3D space, the degrees of freedom of an object usually include three directions, namely the degrees of freedom along the X-axis, Y-axis, and Z-axis. In robotics, the degrees of freedom of a robot determine the type and difficulty of tasks it can perform; in control engineering, the degrees of freedom can be used to describe the state space of the system and the design requirements of the controller. In short, the degree of freedom refers to the number of directions in which an object can freely change in a certain motion mode, and is an important concept for describing the posture and motion characteristics of an object in space. The first degree of freedom is the number of degrees of freedom contained in the initial motion data. In the embodiments of the present application, for the convenience of description, when comparing the degrees of freedom, the default comparison is the number of degrees of freedom.

[0061] For example, the initial motion data is the motion data collected by the sensors set up by the detection device. Since the layout space of the detection device on the user is limited, the degree of freedom of the collected initial motion data is lower than the actual degree of freedom of the user. Therefore, the initial motion data cannot fully express the actual motion of the user, and therefore requires subsequent reconstruction processing.

[0062] Step S102: Obtain a preset target measurement matrix, wherein the target measurement matrix is ​​obtained by iteratively updating a residual calculated in advance based on reference motion data and sample motion data, wherein the reference motion data has a second degree of freedom, which is determined based on an actual degree of freedom of the target object and is greater than the first degree of freedom;

[0063] Exemplarily, the target measurement matrix is ​​a measurement matrix of an application process. The measurement matrix can be understood as a matrix consisting of a set of bases, vectors or functions that projects the original signal into a low-dimensional space. Under the action of the measurement matrix, the original signal will be compressed into a lower-dimensional representation. Conversely, the signal can also be restored to a higher-dimensional representation.

[0064] Exemplarily, the reference motion data is motion data with a higher degree of freedom, having a second degree of freedom. The definition of the second degree of freedom is similar to that of the first degree of freedom and is not further described here. The second degree of freedom of the reference motion is determined based on the actual degree of freedom of the target object. In one embodiment, the second degree of freedom is equal to the actual degree of freedom of the target object, or in other words, the second degree of freedom is close to the actual degree of freedom of the target object. For example, if the actual degree of freedom of the target object is 21, the second degree of freedom can be 21, or close to 21. Therefore, the second degree of freedom is greater than the first degree of freedom.

[0065] Exemplarily, the sample motion data is the motion data used as input data in the process of constructing the target measurement matrix. In one embodiment, the degree of freedom of the sample motion data is also lower than the second degree of freedom, or in other words, the degree of freedom of the sample motion data is equal to the first degree of freedom.

[0066] For example, during the pre-construction process of the target measurement matrix, the input sample motion data can be reconstructed, and the residuals can be calculated between the reconstruction situation and the reference motion data. Since the reference motion data represents the actual degrees of freedom of the target object, the obtained residuals can indicate the reconstruction level of the measurement matrix. Therefore, based on the residuals, the measurement matrix can be iterated multiple times and the measurement matrix can be continuously updated to obtain the required target measurement matrix. Therefore, the target measurement matrix is ​​obtained through iterative training. In this way, the target measurement matrix can accurately reconstruct the input data in practical applications.

[0067] Step S103 , reconstructing the initial motion data based on the target measurement matrix to obtain target reconstructed data of the target object, wherein the target reconstructed data has a third degree of freedom, which corresponds to the second degree of freedom.

[0068] Exemplarily, the target reconstructed data is motion data obtained after a reconstruction operation of the target measurement matrix. The target reconstructed data has a third degree of freedom, which is defined similarly to the first degree of freedom and will not be further elaborated here. The target reconstructed data can represent the actual motion data of the target object, or be infinitely close to the actual motion data of the target object. Therefore, its third degree of freedom can be the same as the second degree of freedom, or be infinitely close to the second degree of freedom, which is determined by the performance of the target measurement matrix.

[0069] For example, after the reconstruction of the target measurement matrix, the initial motion data can be converted from data with lower degrees of freedom, which is a compressed sensing processing method, also called a compressed sensing algorithm, to data with higher degrees of freedom. Since the target measurement matrix is ​​established based on the reference motion data based on the actual degrees of freedom of the target object, the degrees of freedom of the obtained target reconstructed data are also equal to or close to the actual degrees of freedom of the target object.

[0070] The embodiment of the present application processes the initial motion data through a preset target measurement matrix, wherein the first degree of freedom of the initial motion data is low and lower than the actual degree of freedom, and the target measurement matrix is ​​obtained after iterative updating of the residual calculated in advance based on the reference motion data and the sample motion data. The reference motion data has a second degree of freedom, and the second degree of freedom is determined based on the actual degree of freedom of the target object. Through continuous iterative updating, the sample motion data is reconstructed and close to the reference motion data. The established target measurement matrix can reconstruct the input low-degree-of-freedom motion data into high-degree-of-freedom data, and finally obtain target reconstructed data with a third degree of freedom, and the third degree of freedom corresponds to the second degree of freedom. Therefore, the initial motion data is reconstructed into high-degree-of-freedom data through compressed sensing, which can more accurately reflect the user's actual motion, thereby improving the accuracy of the motion data.

[0071] Referring to FIG. 3 , in some embodiments, step S102 may include steps S201 to S204 :

[0072] Step S201: obtaining reference motion data and a plurality of sample motion data, and establishing an initial measurement matrix;

[0073] Step S202: selecting any sample motion data in the current iteration cycle, inputting the sample motion data into the initial measurement matrix for signal reconstruction, and obtaining sample reconstructed data;

[0074] Step S203, obtaining a residual corresponding to the current sample motion data according to the difference between the reference motion data and the sample reconstructed data;

[0075] Step S204: adjust the current initial measurement matrix according to the residual, and continue to calculate the residual corresponding to the next sample action data in the next iteration cycle, iteratively update the measurement matrix according to the next residual, until the number of iterations reaches the iteration threshold, and obtain the target measurement matrix.

[0076] For example, in the compressed sensing algorithm, the target measurement matrix in the embodiment of the present application is established after multiple iterative training. For the convenience of description, the compressed sensing algorithm is described as a compressed sensing operation in the above embodiment.

[0077] For example, in the process of establishing the target measurement matrix, it is first necessary to obtain reference motion data and multiple sample motion data. The reference motion data and sample motion data have been described in the above embodiment and will not be repeated here. In the embodiment of the present application, in order to perform multiple iterations, a sample data set can be established. In the sample data set, multiple sample motion data are present for subsequent use in multiple iterations.

[0078] For example, in embodiments of the present application, an initial measurement matrix needs to be established. This initial measurement matrix needs to undergo multiple iterations before it can be used as the target measurement matrix. The initial measurement matrix can be directly acquired and is a measurement matrix previously established for motion data reconstruction. However, this matrix may not be fully applicable to embodiments of the present application and may require adjustment to meet the data reconstruction requirements of embodiments of the present application. The process of constructing the initial measurement matrix in embodiments of the present application is not specifically limited.

[0079] For example, in the embodiment of the present application, multiple iteration cycles are set up, and after the iteration starts, any sample action data is selected as the input data in the current iteration cycle. In the process of training the compressed sensing algorithm to obtain the target measurement matrix, the input data needs to be solved, or the data is reconstructed. In the process of signal reconstruction, the compressed sensing algorithm will utilize the sparsity of the signal, use basis functions to describe the signal, and continuously correct the coefficients to approximate the real signal model, so that the reconstructed data can be obtained.

[0080] For example, in an iterative process, embodiments of the present application reconstruct sample motion data based on an initial measurement matrix to obtain sample reconstructed data. The sample reconstructed data has a high degree of freedom. As iterations proceed, the degree of freedom of the sample reconstructed data approaches the second degree of freedom of the reference motion data, which is determined by the performance of the iterative measurement matrix.

[0081] Exemplarily, during the iteration process, the embodiment of the present application obtains the residual corresponding to the current sample action data based on the difference between the reference action data and the sample reconstruction data, so as to perform the next round of iteration more accurately. The algorithm determines whether the current reconstructed signal is close enough to the true signal by calculating the residual. If the residual is greater than the residual threshold, the measurement matrix in the current iteration cycle is adjusted according to the residual.

[0082] For example, in an embodiment of the present application, after completing the initial measurement matrix update, the residual corresponding to the next sample action data is continuously calculated in the next iteration cycle, and the measurement matrix is ​​iteratively updated based on the next residual until the number of iterations reaches the iteration threshold, thereby obtaining the target measurement matrix. In addition, the completion of the iteration can be determined based on the size of the residual during the iteration process. In an embodiment of the present application, a residual threshold can be set, and the residual during the iteration process can be compared with the residual threshold. If the residual is less than the residual threshold, indicating that the residual is very small, the iteration is stopped.

[0083] Exemplarily, there are many types of compressed sensing algorithms, including but not limited to the following subcategories: basis pursuit algorithm, matching pursuit algorithm, orthogonal matching pursuit algorithm, and iterative threshold algorithm. Among them:

[0084] The basis pursuit algorithm is a commonly used compressed sensing signal reconstruction algorithm. Its basic idea is to use L1 regularized linear programming to solve the minimum L1 norm reconstruction problem of the signal. The algorithm calculates coefficients by tracking the atoms with the largest correlation in the sparse representation model, and then brings the obtained coefficients into the real model for reconstruction.

[0085] The matching pursuit algorithm is also a commonly used compressed sensing signal reconstruction algorithm. It approximates the true coefficients by iteratively adding atoms with the highest correlation. These atoms are usually obtained by calculating the correlation coefficients between the signal and the basis function. In each iteration, the matching pursuit algorithm selects the atom with the current highest correlation and modifies the coefficients to minimize the reconstruction error.

[0086] Orthogonal Matching Pursuit: This algorithm is an improved basis pursuit algorithm that performs orthogonal preprocessing on the basis vectors, improving the accuracy of signal reconstruction. Compared to the basis pursuit algorithm, it considers not only the influence of atoms on the target in each iteration, but also their influence on the residual of the previous iteration, thereby reducing the reconstruction error.

[0087] Iterative Thresholding Algorithm: This algorithm iteratively solves the sparse representation and reconstruction of compressed sensing signals. Its basic concept is to gradually sparsify the signal through the application of a series of threshold functions. These functions can be hard or soft threshold functions. In each iteration, the algorithm determines thresholds based on the current sparse model, applies these thresholds to the signal model, and calculates new coefficients. The next iteration is then performed based on these new coefficients.

[0088] In the embodiments of the present application, any of the above-mentioned compressed sensing algorithms can be used to realize signal reconstruction. Specifically, taking the fast iterative shrinkage-thresholding algorithm (FISTA) in the iterative threshold algorithm as an example, the process of obtaining sample reconstructed data in any of the above-mentioned iterative cycles is explained. The operation process in other iterative cycles is the same as that in this iterative cycle, and will not be repeated here.

[0089] Referring to FIG. 4 , in some embodiments, the steps for obtaining sample reconstruction data in any iteration cycle are as follows, and may include steps S301 to S304 :

[0090] Step S301, obtaining the residual obtained in the previous iteration cycle and the measurement matrix in the current iteration cycle;

[0091] Step S302: Projecting the sample motion data in the current iteration cycle based on the residual of the previous iteration cycle and the measurement matrix in the current iteration cycle under the preset relaxation parameters to obtain projection data;

[0092] Step S303, optimizing the projection data using a preset iterative threshold function to obtain optimized new sample motion data;

[0093] Step S304: inputting the optimized new sample motion data into the measurement matrix in the current iteration cycle to perform signal reconstruction to obtain sample reconstructed data.

[0094] For example, in the embodiment of the present application, a fast iterative shrinkage threshold algorithm is used, so the problem is defined as: min||x||1subject to Y=AX

[0095] Where X is the input data, Y is the reconstructed signal, and A is the measurement matrix. In the process of iteratively constructing the target measurement matrix, X is the sample action data and Y is the sample reconstruction data. In the application process, X is the initial action data and Y is the target reconstruction data. From the above formula, we can see that the reconstructed data needs to be obtained by multiplying the measurement matrix and the action data.

[0096] For example, for the fast iterative shrinkage threshold algorithm, it is also necessary to find a new solution for the input data and update the residual in each iteration cycle. The process of finding a new solution for the input data is to use the projection method to find the updated vector direction, and then optimize it by taking the threshold method. The formula is as follows: t+1 =η thr (x t +κ×(A T r t ))

[0097] Among them, x t is the sample action data in the current iteration cycle, κ is the preset relaxation parameter, and κ∈(0,1), t is the number of iterations, the current cycle is the t+1th iteration, r is the residual, so r t is the residual of the previous iteration cycle, A T is the measurement matrix adjusted in the previous iteration cycle, that is, the measurement matrix used in the current iteration cycle. In the embodiment of the present application, under the preset relaxation parameters, the residual of the previous iteration cycle and the measurement matrix in the current iteration cycle are combined to project the sample action data in the current iteration cycle to obtain the projection data x t +κ×(A T r t ).

[0098] For example, after the projection data is obtained, it is necessary to optimize the projection data using a preset iteration threshold. thr It is an iterative threshold function, that is, a preset threshold function (thresholding function). In the embodiment of the present application, the projection data is input into the preset threshold function for processing. Through the processing of the function, the projection data can gradually approach the solution with sparse properties during the iterative process, and finally the optimized new sample action data x can be obtained. t+1 , which is the new solution for the input data.

[0099] For example, in the embodiment of the present application, the new solution of the input data is used to update the residual. The updating residual requires the sample reconstruction data obtained in the current iteration cycle. Therefore, it is necessary to input the optimized new sample action data into the measurement matrix in the current iteration cycle for signal reconstruction to obtain the sample reconstruction data. Finally, the difference between the sample reconstruction data and the reference action data is used to obtain the new residual in the current iteration cycle. The calculation process of the new residual is shown in the following formula: t+1 =y-Ax t+1

[0100] Among them, r t+1 is the new residual, that is, the residual calculated in the current iteration cycle, y is the reference action data, Ax t+1 It is the sample reconstruction data. Since the measurement matrix is ​​not accurate enough, there is still a gap between the sample reconstruction data and the more accurate and standard reference action data. The quality of the current measurement matrix can be judged by the new residual. If the new residual is greater than or equal to the preset residual threshold, it is necessary to continue iteration. If the new residual is less than the preset residual threshold, or the number of iterations reaches the iteration threshold, the measurement matrix in the current iteration cycle can be used as the target measurement matrix.

[0101] Please refer to FIG. 5 . In some embodiments, step S303 may include steps S401 to S402 .

[0102] Step S401: If the iterative threshold function is a hard threshold function, the projection data is filtered according to the magnitude relationship between the projection data and the hard threshold corresponding to the hard threshold function to obtain optimized new sample motion data;

[0103] Step S402: If the iterative threshold function is a soft threshold function, a sparse solution of the projection data is calculated using the soft threshold function to obtain optimized new sample motion data.

[0104] For example, the iterative threshold function is mainly used to gradually approach a solution with sparse properties during the iteration process. In the embodiments of the present application, there are two types of iterative threshold functions, which can realize two major threshold-taking methods. The threshold functions are hard threshold function (Hard Thresholding Function) and soft threshold function (Soft Thresholding Function).

[0105] When applying the hard threshold function, the iterative threshold to be optimized is the hard threshold. It is necessary to filter the projection data according to the size relationship between the projection data and the hard threshold to obtain the optimized new sample action data. The hard threshold function is as follows:

[0106] in, is a hard threshold function, x is the projection data, and thrH is the hard threshold. From the above formula, it can be seen that if the absolute value of the projection data is greater than the hard threshold, the hard threshold function retains the size of the projection data unchanged. If the absolute value of the projection data is less than or equal to the hard threshold, the hard threshold function filters the projection data to 0. Therefore, in the embodiment of the present application, the solution where the projection data is less than the hard threshold thrH is filtered to 0.

[0107] When applying the soft threshold function, the iterative threshold to be optimized is the soft threshold. It is necessary to calculate the sparse solution of the projection data through the soft threshold to obtain the optimized new sample action data. The soft threshold function is as follows:

[0108] in, is the soft threshold function, x is the projected data, thrS is the hard threshold; Sgn() is the sign function, which determines the positive or negative value of a number. If it is positive, it returns 1, if it is zero, it returns 0, and if it is negative, it returns -1; Where() is used to indicate the location or index of the data, z is the sparse matrix, and II() is the indicator function. As can be seen from the above formula, the soft threshold function uses a relatively smooth method to approximate the value to the sparsity solution, which is used as the new sample action data after optimization.

[0109] It should be noted that a sparse matrix generally refers to a coefficient matrix under a certain basis, where each column represents the projection value of a basis vector in the original signal, and the soft threshold function performs a sparse processing on this coefficient matrix. Sparse matrices can be obtained through various methods, such as wavelet transform, dictionary learning, etc., and the embodiments of this application do not impose specific limitations.

[0110] Referring to FIG. 6 , in some embodiments, step S101 may include steps S501 to S502 :

[0111] Step S501, obtaining initial motion data sent by a detection device, wherein the detection device is provided with multiple sensors, and the detection device is used to be worn on a target object, and the sensors are used to obtain motion data around a joint corresponding to a target joint detection position on the target object, and use the data as the initial motion data;

[0112] Step S502 , alternatively, multiple target joint detection positions on the target object are determined, and motion data around corresponding joints are collected based on the multiple target joint detection positions, and used as initial motion data.

[0113] Exemplarily, the detection device and the processing device in the embodiment of the present application may not be integrated together, so the processing device can serve as the executor of the motion data reconstruction method. Therefore, in order to realize motion reconstruction, when acquiring motion data, an external device is required to collect and send it. In the embodiment of the present application, various motion data, including initial motion data, are acquired through an external detection device, and after acquiring the initial motion data, the initial motion data is sent to the processing device for processing.

[0114] Exemplarily, the detection device and processing device in the embodiments of the present application can be integrated into a terminal system (referred to as the system for short), which can serve as the executor of the motion data reconstruction method. Therefore, the motion data can be directly obtained in the system. However, since the sensor is not directly set on the target joint, when performing data collection, or when arranging the sensor detection position, it is necessary to determine multiple target joint detection positions on the target object and arrange the sensor at the target joint detection position, for example, around the target joint, to realize the collection of motion data around the corresponding joint based on multiple target joint detection positions, and use it as initial motion data.

[0115] Illustratively, the embodiment of the present application uses a wearable device to obtain initial motion data, but is also applicable to using an optical solution to collect initial motion data, so as to improve the degrees of freedom and data accuracy represented by the reconstructed data when the number of optical marker points is insufficient. When the optical solution collects initial motion data, multiple sensors can still be set to detect motion data at corresponding positions, and no specific restrictions are made here.

[0116] It should be noted that the detection device is equipped with multiple sensors, which can be flexible strain sensors made of stretchable conductors. When the target object wears the detection device, the sensors are used to obtain motion data around the joint corresponding to the target joint detection position on the target object, such as initial motion data.

[0117] For example, if the target object is a user, the target joint detection location is the location of sensors placed around a moving or bending joint on the user's body. This joint is the target joint, and the target joint detection location is the sensor detection location, with the sensors placed around the target joint. The target joint can be any joint on the user's body. By reconstructing the initial motion data, the actual motion data of the target joint can be obtained, thereby better matching the actual human body movements.

[0118] For example, the limitations of the related art in obtaining motion data are explained by taking a wearable motion capture device as an example:

[0119] When detecting motion data, sensors in related technologies should theoretically be placed where the elastic fabric produces the greatest strain, specifically where the user's joints are located. This results in greater signal variation, helping to improve detection accuracy. However, in the actual design of wearable motion capture devices, it has been found that, due to factors such as wearing comfort, sensor durability, and insufficient sensor placement space, the location of flexible strain sensors may not necessarily correspond directly to the object being detected. For example, when detecting knee joint angle, the sensor is typically placed above or below the knee joint, rather than directly at the knee joint, to prevent it from experiencing excessive pressure and strain, thereby increasing its service life. Therefore, the sensor location may not correspond to the location of the object being detected. Furthermore, because the elastic fabric is simultaneously deformed by the pull of multiple joints and muscles, the signals collected by a single or multiple sensors may also correspond to multiple joints or muscles, which is greater than the number of sensors. Therefore, the problem of accurately detecting motion when the sensor location and / or number do not correspond to the location and / or number of the object being detected remains an urgent issue.

[0120] The detection device in the embodiment of the present application obtains the motion data of the target joint detection position on the target object. Therefore, the motion data obtained, including the initial motion data, does not correspond to the position and / or number of the target object joint. For example, sensors are arranged on both sides of a target joint to obtain the motion data on both sides of the target joint, or sensors are arranged around the target joint to detect more complete motion data. When facing multiple target joints, based on comfort, stretchability and layout space considerations, the positions where sensors can be placed are relatively few. For example, only one sensor can be arranged between two joints. Therefore, the degree of freedom of the detected motion data is lower than the actual degree of freedom of the target object. However, through the compressed sensing algorithm in the embodiment of the present application, the initial motion data with lower degrees of freedom can be reconstructed into target reconstructed data with higher degrees of freedom, and motion detection can be achieved when the position and / or number of sensors do not correspond to the position and / or number of the target joint.

[0121] It should be noted that when constructing the target measurement matrix, the sample motion data obtained is also obtained by sensors around the target joint, just like the initial motion data. Therefore, the constructed target measurement matrix can reconstruct the motion data collected around the target joint, thereby reconstructing the actual motion data of the target joint, that is, the target reconstruction data.

[0122] If the target object is the user's hand, referring to FIG. 7 , in some embodiments, the process of determining the target joint detection position in step S502 may include steps S601 to S603:

[0123] Step S601, determining multiple joint detection positions on the target object;

[0124] Step S602: Determine, from the plurality of joint detection positions, a first joint detection position for detecting joint opening and closing between the distal and proximal phalanges of the thumb, joint opening and closing between the proximal and metacarpal bones of the thumb, and / or joint abduction and adduction between the metacarpal bones of the thumb and the trapezium. Also determine a second joint detection position for detecting joint opening and closing between the proximal and metacarpal bones of fingers other than the thumb, a third joint detection position for detecting joint abduction and adduction between the proximal and metacarpal bones of fingers other than the thumb, and a fourth joint detection position for detecting joint opening and closing between the proximal and middle phalanges of fingers other than the thumb.

[0125] Step S603 : taking multiple of the first joint detection position, the second joint detection position, the third joint detection position, and the fourth joint detection position as target joint detection positions.

[0126] For example, since the target object is the user's hand, the detection device is a wearable motion capture glove, which is equipped with multiple sensors. However, the space on the hand where the sensors can be arranged is limited, and the arrangement of the sensors must also take into account the comfort and stretchability of the wearable motion capture gloves. Therefore, the target joint detection position needs to be configured to obtain the required motion data, so that the actual motion data of the hand can be reconstructed with the greatest probability during reconstruction.

[0127] For example, the target joint detection position must be able to meet the minimum degree of freedom requirement of the actual movement of the target object. For example, the target joint detection position can reflect the movement of each finger and the movement between each finger and the palm.

[0128] For example, before determining the target joint detection position, it is first necessary to determine multiple joint detection positions on the target object. Taking the joint detection position as an example, which is the detection position around the joint for placing sensors, the joint detection position can be a position around any joint on the target object, and can detect relatively complete motion data of the corresponding joint. When the target object is a hand, these joint detection positions are on the hand.

[0129] Exemplarily, when the target object is the user's hand, the minimum target joint detection positions that need to be determined in the embodiment of the present application include multiple of the first joint detection position, the second joint detection position, the third joint detection position and the fourth joint detection position.

[0130] As shown in Figure 8:

[0131] Exemplarily, the first joint detection position is a joint detection position for detecting the opening and closing of the joint between the distal phalanx and the proximal phalanx of the thumb, the opening and closing of the joint between the proximal phalanx and the metacarpal of the thumb, and / or the abduction and adduction of the joint between the metacarpal bone and the trapezium of the thumb. The first joint detection position can be at least one of the following multiple positions, for example: including being set on the distal phalanx of the thumb and between the proximal phalanx of the thumb, and the opening and closing between the distal phalanx and the proximal phalanx of the thumb can be detected around this joint; also including being set between the metacarpal bone corresponding to the thumb and the proximal phalanx of the thumb, and the opening and closing between the proximal phalanx and the metacarpal of the thumb can be detected around this joint; in addition, it can also be set between the trapezium corresponding to the thumb and the metacarpal of the thumb, and the abduction and adduction of the joint between the metacarpal bone and the trapezium of the thumb can be detected around this joint. In summary, in one embodiment, there are three first joint detection positions, which can achieve detection of three degrees of freedom.

[0132] Exemplarily, the second joint detection position is a joint detection position for detecting the opening and closing of the joints between the proximal phalanges and metacarpal bones of other fingers except the thumb. There can be multiple second joint detection positions, and the metacarpophalangeal opening and closing here refers to the opening and closing between the metacarpal bones and the index finger, middle finger, ring finger or little finger. For example: the second joint detection position includes a position between the metacarpal bones near the proximal phalanges of the index finger and the proximal phalanges of the index finger, which can detect the opening and closing between the proximal phalanges of the index finger and the metacarpal bones; and also includes a position between the metacarpal bones near the proximal phalanges of the middle finger and the proximal phalanges of the middle finger, which can detect the opening and closing between the proximal phalanges of the middle finger and the metacarpal bones. In the embodiment of the present application, the second joint detection position is set at the detection position corresponding to the index finger and the middle finger as an example because, in general, the range of motion of the index finger and the middle finger is larger. Under the condition that the requirements of the embodiment of the present application are met, the second joint detection position can also be set for the ring finger or the little finger. In summary, there are two second joint detection positions in the embodiment of the present application, which can realize detection of two degrees of freedom.

[0133] Exemplarily, the third joint detection position is a joint detection position for detecting the abduction and adduction of the joints between the proximal phalanges and metacarpal bones of other fingers except the thumb. There can be multiple third joint detection positions, and the abduction and adduction of the inter-finger joints here refers to the abduction and adduction between the index finger and the middle finger, between the middle finger and the ring finger, or between the ring finger and the little finger. For example: the third joint detection position includes a position set between the proximal phalange of the index finger and the proximal phalange of the middle finger, which can detect the abduction and adduction between the index finger and the middle finger; also includes a position set between the proximal phalange of the middle finger and the proximal phalange of the ring finger, which can detect the abduction and adduction between the middle finger and the ring finger; also includes a position set between the proximal phalange of the ring finger and the proximal phalange of the little finger, which can detect the abduction and adduction between the ring finger and the little finger. In summary, there are 3 third joint detection positions in the embodiment of the present application, which can achieve detection of 3 degrees of freedom.

[0134] Exemplarily, the fourth joint detection position is a joint detection position for detecting the opening and closing of the joints between the proximal phalanges and the middle phalanges of the fingers other than the thumb. There can be multiple fourth joint detection positions, which refers to the opening and closing of each finger on the index finger, middle finger, ring finger or little finger. For example, the fourth joint detection position includes a position set between the proximal phalanges and the corresponding middle phalanges of the index finger, middle finger, ring finger or little finger, which can detect the corresponding bending changes of the index finger, middle finger, ring finger or little finger, and the opening and closing of the index finger, middle finger, ring finger or little finger can be obtained. In summary, there can be multiple fourth joint detection positions in the embodiment of the present application. If it is set between the proximal phalanges and the middle phalanges, there are 4 fourth joint detection positions, which can achieve detection of 4 degrees of freedom.

[0135] In summary, the target joint detection positions in the embodiment of the present application can be 12, which can make the first degree of freedom of the initial motion data 12 degrees of freedom. The setting of these detection positions can largely reflect the movement of the hand. If the requirements of the embodiment of the present application are met, a detection position can also be set between the hand and the wrist to increase the degrees of freedom to 13 or more to detect the motion data between the hand and the wrist.

[0136] For example, in the embodiment of the present application, the number of target joint detection positions is selected according to the specific hand movements that need to be fitted. The above embodiment is only illustrated by an example of 12 degrees of freedom, which does not represent a limitation of the embodiment of the present application.

[0137] For example, in the embodiment of the present application, taking the setting of joint detection positions around corresponding joints as an example, on the premise of meeting the requirements of the embodiment of the present application, a joint detection position can also be set on both sides of the corresponding joint, so that the joint detection positions are distributed on both sides of the joint, further reducing the impact on the joint movement. At this time, the motion data around the corresponding joint can also be obtained through the motion data on both sides of the joint. Therefore, the above embodiment description does not represent a limitation on the embodiment of the present application.

[0138] Referring to FIG. 9 , in some embodiments, the reference motion data is obtained through the following steps, which may include steps S701 to S702:

[0139] Step S701: Acquire reference motion data sent by a detection device, wherein the detection device is provided with a plurality of sensors, and the detection device is used to be worn on a target object, and the sensors are used to acquire motion data around a joint corresponding to a reference position of a target joint on the target object, and use the data as the reference motion data;

[0140] Step S702, alternatively, determining multiple target joint reference positions on the target object, and collecting motion data around corresponding joints based on the multiple target joint reference positions, and using the data as reference motion data;

[0141] Among them, the number of target joint reference positions is greater than the number of target joint detection positions.

[0142] Exemplarily, the detection device and the processing device in the embodiment of the present application may not be integrated together, so the processing device can serve as the executor of the motion data reconstruction method. Therefore, in order to realize motion reconstruction, when acquiring motion data, an external device is required to collect and send it. In the embodiment of the present application, various motion data, including reference motion data, are acquired through an external detection device, and after acquiring the reference motion data, the reference motion data is sent to the processing device for processing.

[0143] Exemplarily, the detection device and processing device in the embodiments of the present application can be integrated into a terminal system (which can be referred to as the system for short), which can serve as the executor of the motion data reconstruction method. Therefore, the motion data can be directly obtained in the system. However, since the sensor for collecting the initial motion data is not directly set on the target joint, when constructing the target measurement matrix, more accurate motion data is required for residual calculation. Therefore, when collecting reference motion data, or when arranging the sensor detection position for collecting reference motion data, it is necessary to determine multiple target joint reference positions on the target object and arrange the sensor at the target joint reference position, for example, around the target joint, to realize the collection of motion data around the corresponding joint based on multiple target joint reference positions, and use it as reference motion data.

[0144] It should be noted that the detection device is equipped with multiple sensors, which can be flexible strain sensors made of stretchable conductors. When the target object wears the detection device, the sensor is used to obtain the motion data of the reference position of the target joint on the target object and use it as reference motion data.

[0145] For example, if the target object is a user, the target joint reference positions are the locations of sensors placed around the user's moving or bending joints. This joint is the target joint, meaning the target joint reference positions are the sensor detection positions, with the sensors placed around the target joint. The target joint can be any joint on the user's body. Since the number of target joint reference positions is greater than the number of target joint detection positions, the acquired reference motion data is more accurate, facilitating residual calculation and thus better constructing the target measurement matrix.

[0146] Exemplarily, taking the detection device as a wearable motion capture device as an example, there are limitations in obtaining motion data in the related art. The reference motion data in the embodiment of the present application is the motion data around the joint corresponding to the target joint reference position on the target object. Therefore, the reference motion data does not correspond to the position and / or number of the target object joints. However, the number of target joint reference positions is greater than the number of target joint detection positions. In particular, the target joint reference positions are set on both sides of the specified target joint, so the degree of freedom of the detected reference motion data is equal to or close to the actual degree of freedom of the target object. After better constructing the target measurement matrix, the compressed sensing algorithm in the embodiment of the present application can be used to reconstruct the initial motion data with lower degrees of freedom into target reconstructed data with higher degrees of freedom, and motion detection can be achieved when the sensor position and / or number do not correspond to the position and / or number of the target joints during application.

[0147] Exemplarily, the embodiments of the present application adopt a more precise method to obtain reference motion data, for example: using a device based on an optical motion capture solution (such as OptiTrack) to obtain reference motion data, multiple optical markers (such as passive reflective balls) can be set on the target object, and the images of the optical markers captured by multiple optical cameras and a preset kinematic model are used to solve the motion data of each joint of the target object as reference motion data. In addition, the wearable device described in the above embodiment can also be used to collect reference motion data, such as wearing a motion capture device based on an inertial sensor acquisition solution (such as Noitom) to collect reference motion data, provided that the accuracy and degree of freedom of the reference motion data meet the requirements. The above-mentioned use of sensors to obtain reference motion data is only one of the embodiments on the wearable device and does not represent a limitation on the embodiments of the present application.

[0148] On the premise of meeting the requirements of the embodiments of this application, the target joint reference position can also be set on the target joint. This is determined based on the kinematic model of the subsequent processing target reconstruction data. The kinematic model library fits the actual action according to the action data on the joint, and can also fit the actual action according to the action data between the joints. The embodiments of this application do not impose specific restrictions.

[0149] If the target object is the user's hand, referring to FIG. 10 , in some embodiments, the process of determining the target joint reference position in step S702 may include steps S801 to S804:

[0150] Step S801, determining multiple joint reference positions on the target object;

[0151] Step S802: Determine, from the plurality of joint detection positions, a first joint reference position for detecting the opening and closing of the distal and proximal phalanges of the thumb, a second joint reference position for detecting the opening and closing of the proximal phalanges and metacarpals, a third joint reference position for detecting abduction and adduction of the proximal phalanges and metacarpals, a fourth joint reference position for detecting the opening and closing of the metacarpals and trapezium, and a fifth joint reference position for detecting abduction and adduction of the metacarpals and trapezium.

[0152] Step S803, determining, from the plurality of joint detection positions, a sixth joint reference position for detecting the opening and closing of the distal phalanx and the middle phalanx, a seventh joint reference position for detecting the opening and closing of the middle phalanx and the proximal phalanx, an eighth joint reference position for detecting the opening and closing of the proximal phalanx and the metacarpal, and a ninth joint reference position for detecting the abduction and adduction of the proximal phalanx and the metacarpal on each of the other fingers;

[0153] In step S804, each of the first joint reference position, the second joint reference position, the third joint reference position, the fourth joint reference position, the fifth joint reference position, the sixth joint reference position, the seventh joint reference position, the eighth joint reference position and the ninth joint reference position is used as a target joint detection position.

[0154] For example, since the target object is the user's hand, the detection device is a wearable motion capture glove with multiple sensors. When obtaining reference motion data, in order to better construct the target measurement matrix, the target joint detection position is set on the hand as much as possible. Therefore, the target joint reference position needs to be configured to obtain the required reference motion data, so that the actual motion data of the hand can be reconstructed with the greatest probability during reconstruction.

[0155] For example, the target joint reference position must be able to meet the highest degree of freedom requirement of the actual movement of the target object. For example, the target joint reference position can reflect in detail the movement of each finger and the movement between each finger and the palm.

[0156] For example, before determining the target joint reference position, it is first necessary to determine multiple joint reference positions on the target object. The joint reference position is the detection position around the joint for placing sensors; similarly, the joint reference position can also be the detection position on both sides of the joint for placing sensors. When the target object is a hand, the joint reference position can be the middle position of each hand bone, or it can be the position at both ends of each hand bone close to the left and right joints respectively. The embodiments of the present application do not make specific restrictions on this, and only take the joint reference position as the detection position around the joint for placing sensors as an example.

[0157] For example, when the target object is a user's hand, the target joint reference positions that need to be determined in the embodiment of the present application include a first joint reference position, a second joint reference position, a third joint reference position, a fourth joint reference position, a fifth joint reference position, a sixth joint reference position, a seventh joint reference position, an eighth joint reference position, and a ninth joint reference position. Among them, the first joint reference position, the second joint reference position, the third joint reference position, the fourth joint reference position, and the fifth joint reference position correspond to the thumb. The sixth joint reference position, the seventh joint reference position, the eighth joint reference position, and the ninth joint reference position correspond to the index finger, the middle finger, the ring finger, or the little finger.

[0158] As shown in Figure 11:

[0159] For example, on the thumb, the joint reference position between the distal phalanx and the proximal phalanx is the first joint reference position, which is used to detect the opening and closing between the distal phalanx and the proximal phalanx on the thumb; the joint reference position between the proximal phalanx and the metacarpal is the second joint reference position, which is used to detect the opening and closing between the proximal phalanx and the metacarpal on the thumb; another joint reference position between the proximal phalanx and the metacarpal is the third joint reference position, which is used to detect the abduction and adduction between the proximal phalanx and the metacarpal on the thumb. It can be understood that the third joint reference position can be different from the position of the second joint reference position, or it can be different from the position of the second joint reference position. In the same way, if the positions are the same, the sensor obtains motion data in different directions here; the joint reference position between the metacarpal bone and the trapezium is the fourth joint reference position, which is used to detect the opening and closing between the metacarpal bone and the trapezium of the thumb; another joint reference position between the metacarpal bone and the trapezium is the fifth joint reference position, which is used to detect the abduction and adduction between the metacarpal bone and the trapezium of the thumb. It can be understood that the fifth joint reference position can be different from or the same as the second joint reference position. If the positions are the same, the sensor obtains motion data in different directions here.

[0160] In summary, some joint reference positions in the embodiments of the present application may overlap, and the names of the first to fifth are only for distinguishing the detection between different joints. The detection of 5 degrees of freedom can be achieved through the first joint reference position, the second joint reference position, the third joint reference position, the fourth joint reference position, and the fifth joint reference position.

[0161] Exemplarily, the sixth joint reference position, the seventh joint reference position, the eighth joint reference position and the ninth joint reference position are all set on the index finger, the middle finger, the ring finger and the little finger. Taking the index finger as an example, the joint reference position between the distal phalanx and the middle phalanx corresponding to the index finger is the sixth joint reference position, which is used to detect the opening and closing between the distal phalanx and the middle phalanx on the index finger; the joint reference position between the middle phalanx and the proximal phalanx corresponding to the index finger is the seventh joint reference position, which is used to detect the opening and closing between the middle phalanx and the proximal phalanx on the index finger; the joint reference position between the proximal phalanx and the metacarpal bone corresponding to the index finger is the eighth joint reference position, which is used to detect the opening and closing between the proximal phalanx and the metacarpal bone on the index finger; another joint reference position between the proximal phalanx and the metacarpal bone corresponding to the index finger is the ninth joint reference position, which is used to detect the abduction and adduction between the proximal phalanx and the metacarpal bone on the index finger. It can be understood that the ninth joint reference position can be different from or the same as the eighth joint reference position. If the positions are the same, the sensor obtains motion data in different directions. The other fingers are similar to the index finger and will not be described here.

[0162] In summary, in the embodiment of the present application, the index finger, middle finger, ring finger and little finger all have the sixth joint reference position, the seventh joint reference position, the eighth joint reference position and the ninth joint reference position. Therefore, the index finger, middle finger, ring finger and little finger each have 4 degrees of freedom, and overall 20 degrees of freedom detection can be achieved.

[0163] In summary, the second degree of freedom of the reference motion data in the embodiment of the present application is 21 degrees of freedom. The setting of these reference positions can fully reflect the movement of the hand. If the requirements of the embodiment of the present application are met, a reference position can also be set between the hand and the wrist to increase the degrees of freedom to 27 or more to detect the motion data between the hand and the wrist.

[0164] Please refer to FIG. 12 . In some embodiments, step S101 may further include steps S901 to S902 .

[0165] Step S901, obtaining original initial motion data of the target object, wherein the original initial motion data is a dimensionless value used to characterize the motion of the target object;

[0166] Step S902 : Calibrate the original initial motion data to obtain initial motion data, wherein the initial motion data is a dimensioned value used to characterize the motion of the target object.

[0167] For example, the detection device, acting as a slave, outputs initial motion data to the processing device of the master for processing. This initial motion data is a dimensionless value representing the strain generated by the sensor (flexible strain sensor), that is, a dimensionless value representing the motion of the target object. Under normal circumstances, when the target object is a user, human motion is within a certain range. Therefore, the initial motion data needs to be calibrated. The initial motion data is calibrated at the endpoints of the interval in which the human body can perform the action to obtain data for subsequent calculations, so that the interval endpoints of the initial motion data start from 0.

[0168] For example, in the embodiment of the present application, the initial motion data obtained at the beginning is original and dimensionless, which will cause trouble in subsequent calculations. Therefore, the original initial motion data is calibrated, including subtracting the minimum value of the dimensionless value interval from each value in the original initial motion data to obtain the initial motion data, wherein the initial motion data is a dimensional value used to characterize the motion of the target object. After the calibration process, the dimensional value is convenient for the iterative calculation of the subsequent measurement matrix. On the other hand, the result finally output by the target measurement matrix is ​​also an interval value starting from 0, which is convenient for binding the joint angle in the kinematic model (such as 3D engines such as Unity) to start directly from 0 without the need for reconversion.

[0169] For example, using a motion capture glove as a detection device, the original initial motion data sent by the motion capture glove to the host computer has a dimensionless value range of 200 to 300 for representing the angle of the index finger metacarpophalangeal joint. When the dimensionless value is 200, it indicates that the index finger metacarpophalangeal joint is fully extended, and when the dimensionless value is 300, it indicates that the index finger metacarpophalangeal joint is bent to the maximum extent. In other words, the dimensionless value range corresponds to the range of angles that the index finger metacarpophalangeal joint can move. After the original initial motion data of the index finger metacarpophalangeal joint is calibrated, the dimensionless value range of the initial motion data of the index finger metacarpophalangeal joint becomes 0 to 100. In other words, the initial motion data is obtained by subtracting the minimum value of the dimensionless value range under the original initial motion data (200-200=0).

[0170] Referring to FIG. 13 , in some embodiments, step S103 may include steps S1001 to S1002:

[0171] Step S1001, reconstructing the initial motion data based on the target measurement matrix to obtain initial reconstructed data;

[0172] Step S1002 : performing filtering processing on the initial reconstructed data to obtain target reconstructed data of the target object.

[0173] For example, in the embodiment of the present application, since the reconstructed data obtained may have certain mutations, which will be reflected in the kinematic model as unnatural shaking of human limbs, it is also necessary to filter the reconstructed data calculated based on the target measurement matrix to eliminate the shaking.

[0174] For example, in an embodiment of the present application, the initial motion data is reconstructed based on the target measurement matrix to obtain initial reconstructed data. The initial reconstructed data may have certain mutations, so the initial reconstructed data is filtered to obtain target reconstructed data of the target object, thereby eliminating jitter.

[0175] For example, the present embodiment employs a Kalman filter to eliminate jitter. This is a state estimation algorithm used to estimate the unknown state of a discrete-time dynamic system. The Kalman filter uses the dynamic system model, observation data, and its errors to calculate the optimal estimate of the state and the error covariance, thereby achieving accurate state tracking.

[0176] Specifically, the specific process of Kalman filtering is as follows: first, the initial reconstruction data is initialized, and the initial state and its error covariance matrix are set; then prediction is performed, and the state and its error covariance matrix at the next moment are predicted based on the known model and control quantity; then update is performed, and the predicted state and its error covariance matrix are corrected by the observation data to obtain the optimal estimate and error covariance matrix at the current moment; finally, the steps of prediction and update are repeated until the state estimation at all moments is completed. The key to the Kalman filter in the embodiment of the present application is to design the model of the dynamic system and the error covariance matrix of the observation data, and to use the recursive equation to calculate the optimal estimate and error covariance matrix of the state. Among them, the model usually adopts a linear Gaussian model, and the observation data usually contains noise. Kalman filtering can effectively suppress the influence of noise. In addition, Kalman filtering has many variants, such as extended Kalman filtering, unscented Kalman filtering, etc., which can all be applied in this application.

[0177] Exemplarily, the target reconstruction data includes a plurality of degree-of-freedom data under the third degree of freedom. For example, if there are 21 third degrees of freedom, then the target reconstruction data has 21 degree-of-freedom data.

[0178] Please refer to FIG. 14 . In some embodiments, after step S103 , steps S1101 to S1102 may be further included:

[0179] Step S1101, obtaining a preset kinematic model, wherein the kinematic model has multiple nodes, and each node corresponds to one degree of freedom data;

[0180] Step S1102: input the target reconstruction data into the kinematic model, bind the data of each degree of freedom in the target reconstruction data to the corresponding node on the kinematic model, and map the motion image of the target object through the kinematic model after binding the data.

[0181] For example, a kinematic model is a model that converts input data into motion images. It can be a Unity model, but this is not a specific limitation. The kinematic model has multiple nodes, each corresponding to a degree of freedom. After inputting multiple degrees of freedom data, each node performs motion mapping based on these degrees of freedom data, thereby reproducing realistic motion.

[0182] Exemplarily, when the target object is a user, the target reconstruction data in the embodiment of the present application can characterize the angles of multiple joint positions of the human body and / or the contraction and relaxation of muscles. Taking the restoration of hand movements through the angles of hand joints as a specific example, the target reconstruction data maps the angles that multiple (such as 21) action nodes in the human hand kinematic model should rotate, corresponding to the various joints of the human hand. Therefore, by binding the target reconstruction data to multiple action sites on the human hand kinematic model, the actual movements of the human hand can be restored using the human hand kinematic model to obtain the corresponding action image.

[0183] It can be understood that the generated action image can be converted and displayed on the corresponding display device according to actual interaction needs. The display device can be a virtual reality (VR) device, a mixed reality (MR) device, an augmented reality (AR) device or a metaverse device. Therefore, the action image can be an image on a VR device, an MR device, an AR device or a metaverse device.

[0184] Please refer to FIG. 15 . In some embodiments, after step S1102 , steps S1201 to S1202 may be further included:

[0185] Step S1201, generating a virtual image of the target object based on the action image, and displaying the virtual image;

[0186] Step S1202 : In response to the interactive operation on the virtual image, a corresponding interactive display interface is displayed.

[0187] Illustratively, in the embodiment of the present application, after the action image of the display device is displayed, it is used to interact with the user, and a corresponding interactive display interface is displayed after the interaction.

[0188] For example, when a display device displays an action image, the displayed action image can be a virtual image, or a corresponding virtual image can be generated based on the action image after virtual processing. For example, if the action image is a 3D image of a hand, the action image can be directly used as a virtual image for virtual display on the display device; or, if the action image is a 3D image of a hand, the 3D image can be integrated into a virtual character displayed in 3D during display to generate a virtual image of the virtual character.

[0189] For example, after the virtual image is displayed, the user can perform interactive operations based on the virtual image. Specifically, when the user performs an interactive operation on the virtual image, the system will recognize the user's action, calculate the corresponding input instruction, process the input instruction, generate a corresponding screen based on the processing result, and display the corresponding interactive display interface on the display device.

[0190] Exemplarily, there are many ways to perform interactive operations, which are determined based on the way the virtual image is displayed. Taking VR devices as an example, for example, if the virtual image is displayed by a VR device, the user can make clicks, slides, flips, pushes, poses, and other actions in space with their hands. Certain acquisition devices on the VR device, such as cameras, sensors, etc., can recognize the actions made by the user, thereby recognizing the input instructions of the user's corresponding gestures, and completing the interactive operation after background processing; or, when the VR device displays the virtual image, the user can issue interactive actions through other devices, such as handheld input devices, mobile terminals (such as mobile phones) and consoles and other hardware devices for input. According to the input information of these devices, corresponding interactive operations can be generated. In the embodiments of the present application, there is no specific restriction on the implementation method of the interactive operation.

[0191] Referring to FIG. 16 , an embodiment of the present application further provides a motion data reconstruction system that can implement the above-mentioned motion data reconstruction method. The motion data reconstruction system includes:

[0192] A motion data acquisition module 1601 is configured to acquire initial motion data of a target object, wherein the initial motion data has a first degree of freedom, which is smaller than an actual degree of freedom of the target object;

[0193] A measurement matrix acquisition module 1602 is configured to acquire a preset target measurement matrix, wherein the target measurement matrix is ​​obtained by iteratively updating the residuals pre-calculated based on the reference motion data and the sample motion data, wherein the reference motion data has a second degree of freedom, which is determined based on the actual degree of freedom of the target object and is greater than the first degree of freedom.

[0194] The motion data reconstruction module 1603 is used to reconstruct the initial motion data based on the target measurement matrix to obtain target reconstructed data of the target object, wherein the target reconstructed data has a third degree of freedom, which corresponds to the second degree of freedom.

[0195] Exemplarily, the target object can be any object that can perform an action, for example, a user, a robot, or an animal. During the movement of a user or animal, the main body or limbs can perform corresponding movements, and the motion data generated by these movements can be used as initial motion data. During the movement of a robot, the bionic robotic arm can perform corresponding movements, and the motion data generated by these movements can be used as initial motion data. While the target object in the embodiments of the present application is described as a user, this does not constitute a limitation on the embodiments of the present application.

[0196] Exemplarily, the initial motion data is the motion data that needs to be processed. The initial motion data may be motion data detected by a sensor. The motion data may represent the motion situation. For example, it may be motion data such as the angle, position, height, etc. of the motion. No specific limitation is made here.

[0197] Exemplarily, the first degree of freedom is the number of degrees of freedom of the initial motion data, and the degree of freedom refers to the number of directions in which an object can freely change in a certain motion mode. For example, in 3D space, the degrees of freedom of an object usually include three directions, namely the degrees of freedom along the X-axis, Y-axis, and Z-axis. In robotics, the degrees of freedom of a robot determine the type and difficulty of tasks it can perform; in control engineering, the degrees of freedom can be used to describe the state space of the system and the design requirements of the controller. In short, the degree of freedom refers to the number of directions in which an object can freely change in a certain motion mode, and is an important concept for describing the posture and motion characteristics of an object in space. The first degree of freedom is the number of degrees of freedom contained in the initial motion data. In the embodiments of the present application, for the convenience of description, when comparing the degrees of freedom, the default comparison is the number of degrees of freedom.

[0198] For example, the initial motion data is the motion data collected by the sensors set up by the detection device. Since the layout space of the detection device on the user is limited, the degree of freedom of the collected initial motion data is lower than the actual degree of freedom of the user. Therefore, the initial motion data cannot fully express the actual motion of the user, and therefore requires subsequent reconstruction processing.

[0199] Exemplarily, the target measurement matrix is ​​a measurement matrix of an application process. The measurement matrix can be understood as a matrix consisting of a set of bases, vectors or functions that projects the original signal into a low-dimensional space. Under the action of the measurement matrix, the original signal will be compressed into a lower-dimensional representation. Conversely, the signal can also be restored to a higher-dimensional representation.

[0200] Exemplarily, the reference motion data is motion data with a higher degree of freedom, having a second degree of freedom. The definition of the second degree of freedom is similar to that of the first degree of freedom and is not further described here. The second degree of freedom of the reference motion is determined based on the actual degree of freedom of the target object. In one embodiment, the second degree of freedom is equal to the actual degree of freedom of the target object, or in other words, the second degree of freedom is close to the actual degree of freedom of the target object. For example, if the actual degree of freedom of the target object is 21, the second degree of freedom can be 21, or close to 21. Therefore, the second degree of freedom is greater than the first degree of freedom.

[0201] Exemplarily, the sample motion data is the motion data used as input data in the process of constructing the target measurement matrix. In one embodiment, the degree of freedom of the sample motion data is also lower than the second degree of freedom, or in other words, the degree of freedom of the sample motion data is equal to the first degree of freedom.

[0202] For example, during the pre-construction process of the target measurement matrix, the input sample motion data can be reconstructed, and the residuals can be calculated between the reconstruction situation and the reference motion data. Since the reference motion data represents the actual degrees of freedom of the target object, the obtained residuals can indicate the reconstruction level of the measurement matrix. Therefore, based on the residuals, the measurement matrix can be iterated multiple times and the measurement matrix can be continuously updated to obtain the required target measurement matrix. In this way, the target measurement matrix can accurately reconstruct the input data in practical applications.

[0203] Exemplarily, the target reconstructed data is motion data obtained after a reconstruction operation of the target measurement matrix. The target reconstructed data has a third degree of freedom, which is defined similarly to the first degree of freedom and will not be further elaborated here. The target reconstructed data can represent the actual motion data of the target object, or be infinitely close to the actual motion data of the target object. Therefore, its third degree of freedom can be the same as the second degree of freedom, or be infinitely close to the second degree of freedom, which is determined by the performance of the target measurement matrix.

[0204] For example, after the reconstruction of the target measurement matrix, the initial motion data can be converted from data with lower degrees of freedom, which is a compressed sensing processing method, also called a compressed sensing algorithm, to data with higher degrees of freedom. Since the target measurement matrix is ​​established based on the reference motion data based on the actual degrees of freedom of the target object, the degrees of freedom of the obtained target reconstructed data are also equal to or close to the actual degrees of freedom of the target object.

[0205] The embodiment of the present application processes the initial motion data through a preset target measurement matrix, wherein the first degree of freedom of the initial motion data is low and lower than the actual degree of freedom, and the target measurement matrix is ​​obtained after iterative updating of the residual calculated in advance based on the reference motion data and the sample motion data. The reference motion data has a second degree of freedom, and the second degree of freedom is determined based on the actual degree of freedom of the target object. Through continuous iterative updating, the sample motion data is reconstructed and close to the reference motion data. The established target measurement matrix can reconstruct the input low-degree-of-freedom motion data into high-degree-of-freedom data, and finally obtain target reconstructed data with a third degree of freedom, and the third degree of freedom corresponds to the second degree of freedom. Therefore, the initial motion data is reconstructed into high-degree-of-freedom data through compressed sensing, which can more accurately reflect the user's actual motion, thereby improving the accuracy of the motion data.

[0206] The specific implementation of the motion data reconstruction system is substantially the same as the specific embodiment of the motion data reconstruction method described above, and will not be repeated here. Under the premise of meeting the requirements of the embodiment of the present application, the motion data reconstruction system can also be provided with other functional modules to implement the motion data reconstruction method in the above embodiment.

[0207] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned motion data reconstruction method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, an in-vehicle computer, or the like.

[0208] Please refer to FIG17 , which illustrates a hardware structure of an electronic device according to another embodiment. The electronic device includes:

[0209] The processor 1701 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0210] The memory 1702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1702 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1702 and is called by the processor 1701 to execute the action data reconstruction method of the embodiments of this application.

[0211] Input / output interface 1703, used to implement information input and output;

[0212] Communication interface 1704, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0213] Bus 1705 , which transmits information between various components of the device (e.g., processor 1701 , memory 1702 , input / output interface 1703 , and communication interface 1704 );

[0214] The processor 1701 , the memory 1702 , the input / output interface 1703 and the communication interface 1704 are connected to each other in communication within the device via a bus 1705 .

[0215] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned action data reconstruction method is implemented.

[0216] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0217] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0218] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0219] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0220] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0221] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0222] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0223] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the above units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0224] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0225] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0226] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0227] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for reconstructing motion data, characterized in that: The method comprises: Acquire initial motion data of the target object, wherein the initial motion data has a first degree of freedom, and the first degree of freedom is smaller than an actual degree of freedom of the target object; Acquire a preset target measurement matrix, wherein the target measurement matrix is ​​obtained after iterative updating of residuals calculated in advance according to reference action data and sample action data, the reference action data has a second degree of freedom, the second degree of freedom is determined based on an actual degree of freedom of the target object, and the second degree of freedom is greater than the first degree of freedom; The initial motion data is reconstructed based on the target measurement matrix to obtain target reconstructed data of the target object, wherein the target reconstructed data has a third degree of freedom, and the third degree of freedom corresponds to the second degree of freedom.

2. The motion data reconstruction method according to claim 1, characterized in that: The obtaining of a preset target measurement matrix includes: Acquire the reference motion data and a plurality of the sample motion data, and establish an initial measurement matrix; In the current iteration cycle, any one of the sample action data is selected, and the sample action data is input into the initial measurement matrix for signal reconstruction to obtain sample reconstruction data; Obtaining the residual corresponding to the current sample action data according to the difference between the reference action data and the sample reconstructed data; The current initial measurement matrix is ​​adjusted according to the residual, and the residual corresponding to the next sample action data is continuously calculated in the next iteration cycle. The measurement matrix is ​​iteratively updated according to the next residual until the number of iterations reaches an iteration threshold, thereby obtaining the target measurement matrix.

3. The motion data reconstruction method according to claim 2, characterized in that: The steps of obtaining the sample reconstruction data in any iteration cycle are as follows, including: Obtaining the residual obtained in the previous iteration cycle and the measurement matrix in the current iteration cycle; Under preset relaxation parameters, combining the residual of the previous iteration cycle and the measurement matrix in the current iteration cycle, projecting the sample action data in the current iteration cycle to obtain projection data; Optimizing the projection data by using a preset iterative threshold function to obtain optimized new sample motion data; The optimized new sample action data is input into the measurement matrix in the current iteration cycle to perform signal reconstruction to obtain sample reconstruction data.

4. The motion data reconstruction method according to claim 3, characterized in that: The step of optimizing the projection data by using a preset iterative threshold function to obtain optimized new sample motion data includes: If the iterative threshold function is a hard threshold function, the projection data is compared with the hard threshold corresponding to the hard threshold function. The projection data is filtered to obtain the optimized new sample action data; If the iterative threshold function is a soft threshold function, the sparse solution of the projection data is calculated by the soft threshold function to obtain the optimized new sample motion data.

5. The motion data reconstruction method according to claim 1, characterized in that: The obtaining of the initial action data of the target object includes: Acquire initial motion data sent by a detection device, wherein the detection device is provided with a plurality of sensors, the detection device is used to be worn on a target object, and the sensors are used to acquire motion data around a joint corresponding to a target joint detection position on the target object, and use the data as the initial motion data; Alternatively, a plurality of target joint detection positions on the target object are determined, and motion data around corresponding joints are collected based on the plurality of target joint detection positions, and used as the initial motion data.

6. The motion data reconstruction method according to claim 5, characterized in that: If the target object is a hand, determining a plurality of target joint detection positions on the target object includes: Determine multiple joint detection locations on the target object; Determine from the plurality of joint detection positions a first joint detection position for detecting joint opening and closing between the distal phalanx and the proximal phalanx of the thumb, joint opening and closing between the proximal phalanx and the metacarpal of the thumb, and / or joint abduction and adduction between the metacarpal of the thumb and the trapezium, and also determine a second joint detection position for detecting joint opening and closing between the proximal phalanx and the metacarpal of other fingers except the thumb, a third joint detection position for detecting joint abduction and adduction between the proximal phalanx and the metacarpal of other fingers except the thumb, and a fourth joint detection position for detecting joint opening and closing between the proximal phalanx and the middle phalanx of other fingers except the thumb; A plurality of the first joint detection position, the second joint detection position, the third joint detection position, and the fourth joint detection position are used as target joint detection positions.

7. The motion data reconstruction method according to claim 1 or 5, characterized in that: The reference action data is obtained by the following steps, including: Acquire reference motion data sent by a detection device, wherein the detection device is provided with a plurality of sensors, the detection device is used to be worn on a target object, and the sensors are used to acquire motion data around a joint corresponding to a reference position of a target joint on the target object, and use the data as the reference motion data; Alternatively, a plurality of target joint reference positions on the target object are determined, and motion data around corresponding joints are collected based on the plurality of target joint reference positions and used as reference motion data; Wherein, the number of the target joint reference positions is greater than the number of the target joint detection positions.

8. The motion data reconstruction method according to claim 7, characterized in that: If the target object is a hand, multiple target joint reference positions on the target object are determined, including: determining a plurality of joint reference positions on a target object; Determine from the plurality of joint detection positions a first joint reference position for detecting the opening and closing between the distal phalanx and the proximal phalanx on the thumb, a second joint reference position for detecting the opening and closing between the proximal phalanx and the metacarpal, a third joint reference position for detecting the abduction and adduction between the proximal phalanx and the metacarpal, a fourth joint reference position for detecting the opening and closing between the metacarpal and the trapezium, and a fifth joint reference position for detecting the abduction and adduction between the metacarpal and the trapezium; Determine from the plurality of joint detection positions a sixth joint reference position for detecting the opening and closing between the distal phalanx and the middle phalanx, a seventh joint reference position for detecting the opening and closing between the middle phalanx and the proximal phalanx, an eighth joint reference position for detecting the opening and closing between the proximal phalanx and the metacarpal, and a ninth joint reference position for detecting the abduction and adduction between the proximal phalanx and the metacarpal on each of the other fingers; The first joint reference position, the second joint reference position, the third joint reference position, the fourth joint reference position, the fifth joint reference position, the sixth joint reference position, the seventh joint reference position, the eighth joint reference position and the ninth joint reference position are all used as target joint detection positions.

9. The motion data reconstruction method according to claim 1, characterized in that: The obtaining of the initial action data of the target object includes: Acquire original initial motion data of the target object, wherein the original initial motion data is a dimensionless value used to characterize the motion of the target object; The original initial motion data is calibrated to obtain the initial motion data, wherein the initial motion data is a dimensioned value used to characterize the motion of the target object.

10. The motion data reconstruction method according to claim 1, characterized in that: The reconstructing the initial motion data based on the target measurement matrix to obtain target reconstructed data of the target object includes: Reconstructing the initial motion data based on the target measurement matrix to obtain initial reconstructed data; The initial reconstructed data is filtered to obtain target reconstructed data of the target object.

11. The motion data reconstruction method according to claim 1, characterized in that: The target reconstruction data includes a plurality of degree-of-freedom data under the third degree-of-freedom; After obtaining the target reconstruction data of the target object, the method further includes: Acquire a preset kinematic model, wherein the kinematic model has a plurality of nodes, and each of the nodes corresponds to one degree of freedom data; The target reconstruction data is input into the kinematic model, and each degree of freedom data in the target reconstruction data is bound to the corresponding node on the kinematic model, and the motion image of the target object is mapped through the kinematic model after the data is bound.

12. The motion data reconstruction method according to claim 11, characterized in that: After mapping the action image of the target object through the kinematic model after binding the data, the method further includes: generating a virtual image of a target object based on the action image, and displaying the virtual image; In response to an interactive operation on the virtual image, a corresponding interactive display interface is displayed.

13. A motion data reconstruction system, characterized in that: The system comprises: A motion data acquisition module, used to acquire initial motion data of a target object, wherein the initial motion data has a first degree of freedom, and the first degree of freedom is smaller than an actual degree of freedom of the target object; A measurement matrix acquisition module is used to acquire a preset target measurement matrix, wherein the target measurement matrix is ​​obtained after iterative updating of residuals calculated in advance based on reference action data and sample action data, the reference action data has a second degree of freedom, the second degree of freedom is determined based on an actual degree of freedom of the target object, and the second degree of freedom is greater than the first degree of freedom; The motion data reconstruction module is used to reconstruct the initial motion data based on the target measurement matrix to obtain target reconstructed data of the target object, wherein the target reconstructed data has a third degree of freedom, and the third degree of freedom corresponds to the second degree of freedom.

14. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the motion data reconstruction method according to any one of claims 1 to 12 when executing the computer program.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the motion data reconstruction method according to any one of claims 1 to 12 is implemented.

Images