Method, device and readable storage medium for processing synchronous training data

Abstract

The present application provides a method, device and readable storage medium for processing synchronous training data, the method comprising: if the first training action information of the user collected by the manipulator is received, then the basic information of the user is obtained; the target training object is determined according to the basic information and the first training action information; the display device is controlled to display a first interface including an image of the target training object; the second training action information of the user collected by the manipulator is obtained; the target image information is generated according to the second training action information, and the target image information is the target image information used to update the dynamics of the target training object in the first interface; the display device is controlled to display the target image information. In the present application, the display content of the interface in the display device is updated by the training action information, so that the display content of the display device is more interactive, and the fun of the training process is improved.

Description

Technical Field

[0001] The present application relates to the field of robotics, and in particular to a method and device for processing synchronous training data and a readable storage medium. Background Art

[0002] To assist users in their training, engineers have designed various auxiliary mechanical devices to assist users in their limb training. For example, when training hands, users can wear a robot arm to train. However, existing robots perform training calculations according to fixed instructions, and the training process is simple. When users use the robot arm, they just repeat the same movements, resulting in low training efficiency. Summary of the invention

[0003] The present application provides a method, device and readable storage medium for processing synchronous training data, which updates the display content in a display device through training action information, making the display content more accurate and improving interactivity.

[0004] In a first aspect, an embodiment of the present application provides a method for processing synchronous training data, which is applied to a computing device in a hand function training system, wherein the hand function training system includes the computing device, a display device connected to the computing device, and a manipulator, and the method includes:

[0005] If the first training motion information of the user collected by the robot is received, the basic information of the user is obtained;

[0006] determining a target training object according to the basic information and the first training action information;

[0007] Controlling the display device to display a first interface, where the first interface is an interface generated according to the image of the target training object;

[0008] When the display device displays the first interface, obtaining second training action information of the user collected by the manipulator;

[0009] generating target image information according to the second training action information, wherein the target image information is target image information used to update the dynamic state of the target training object in the first interface;

[0010] Control the display device to display the target image information.

[0011] In a second aspect, an embodiment of the present application provides a synchronous training data processing device, which is applied to a computing device in a hand function training system, wherein the hand function training system includes the computing device, a display device and a manipulator connected to the computing device, and the device includes:

[0012] A first acquisition unit, configured to acquire basic information of the user upon receiving first training motion information of the user collected by the manipulator;

[0013] a determination unit, configured to determine a target training object according to the basic information and the first training action information;

[0014] A first control unit, configured to control the display device to display a first interface, wherein the first interface is an interface generated according to the image of the target training object;

[0015] A second acquisition unit, configured to acquire, when the display device displays the first interface, the second training motion information of the user collected by the manipulator;

[0016] a generating unit, configured to generate target image information according to the second training action information, wherein the target image information is target image information used to update the dynamic state of the target training object in the first interface;

[0017] The second control unit is used to control the display device to display the target image information.

[0018] In a third aspect, an embodiment of the present application provides a terminal device, comprising at least one processor, a communication interface and a memory, wherein the communication interface is used to send and / or receive data, the memory is used to store a computer program, and the at least one processor is used to call the computer program stored in the memory to implement any method as described in the first aspect of the present application.

[0019] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory is used to store computer program code, and the computer program code comprises computer instructions. When the processor executes the computer instructions, the electronic device executes instructions such as the steps in any method of the first aspect of the present application.

[0020] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes part or all of the steps described in the steps of any method in the first aspect of the embodiment of the present application.

[0021] In a sixth aspect, the present application provides a computer program, wherein the computer program is operable to cause a computer to execute some or all of the steps described in any method of the first aspect of the present application. The computer program may be a software installation package.

[0022] It can be seen that in the embodiment of the present application, if the first training action information of the user collected by the manipulator is received, the basic information of the user is obtained; the target training object is determined based on the basic information and the first training action information; the display device is controlled to display a first interface including an image of the target training object, so that the determined target training object is more personalized. The second training action information of the user collected by the manipulator is obtained; the target image information is generated based on the second training action information, and the target image information is the target image information used to update the dynamics of the target training object in the first interface; the display device is controlled to display the target image information. In the present application, the display device internal display content is updated by the training action information to improve interactivity, thereby improving the fun of the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A schematic diagram of the structure of a hand function training system provided in an embodiment of the present application;

[0025] Figure 2 A flowchart of a method for processing synchronous training data provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of the structure of a manipulator provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of an interface of a display device provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of a target training object provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of the movement of a target training object within a display device interface provided in an embodiment of the present application;

[0030] Figure 7 A schematic diagram of an image of a robot included in a display device interface provided in an embodiment of the present application;

[0031] Figure 8 A schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0032] Fig. 9A block diagram of the functional units of a synchronous training data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0035] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0036] See also Figure 1 , Figure 1 This is a schematic diagram of the structure of a hand function training system provided in an embodiment of the present application. Figure 1 As shown, the hand function training system includes a computing device, a manipulator and a display device. The computing device is connected to the manipulator and the display device respectively, so as to facilitate data exchange between the computing device and the manipulator, and between the computing device and the display device. It is understandable that the number of manipulators connected to the computing device can be set according to actual needs, for example, two or more manipulators can be connected, and there is no specific limitation here.

[0037] The computing device in this application can be a terminal device such as a smart phone, tablet computer, laptop computer, desktop computer, wearable device, head-mounted device, vehicle-mounted terminal, etc., which is installed with a control application (i.e., application client). The type of terminal device is not specifically limited here. It should be understood that when the application client runs on the computing device, it can be respectively connected to Figure 1The robot shown interacts with the display device.

[0038] Please combine the following Figure 2 The method in the embodiment of the present application is described in detail:

[0039] See also Figure 2 , Figure 2 A flowchart of a method for processing synchronous training data provided in an embodiment of the present application is shown as follows: Figure 2 As shown, a method for processing synchronous training data is applied to a computing device in a hand function training system, wherein the hand function training system includes the computing device, a display device connected to the computing device, and a manipulator, and the method includes:

[0040] S201: If first training motion information of a user collected by the robot is received, basic information of the user is obtained.

[0041] Please combine Figure 3 , Figure 3 A schematic diagram of the structure of a manipulator provided in an embodiment of the present application. Figure 3 As shown, the manipulator includes a manipulator palm module 32 and a finger joint drive module 31. The finger joint drive module 31 is connected to the manipulator palm module 32. A power supply and a Bluetooth module are provided in the manipulator palm module 32, and the power supply is used to store electrical energy and supply energy to the manipulator. The manipulator can exchange data with a computing device through the Bluetooth module.

[0042] Among them, each finger of the finger joint drive module 31 corresponds to at least one joint drive component, so that each finger of the manipulator can be driven independently. Each joint drive component includes a worm gear, a turbine gear and a drive member. The worm gear is connected to the drive member, and the worm gear and the turbine gear are meshed. Based on the training instructions, the turbine gear and the worm gear are driven by the drive member. Since the degree of the joint is adjusted by using the turbine gear and the worm gear, the manipulator can move more smoothly when performing training movements, reduce the noise when the manipulator performs movements, and have a higher control accuracy on the degree of the joint. For example Figure 3 As shown, the index finger of the manipulator includes a first joint drive component 311 and a second joint drive component 312. The first joint drive component 311 and the second joint drive component 312 are connected, and the first joint drive component 311 and the second joint drive component 312 respectively drive the various joints of the index finger to move, thereby improving the control accuracy of the joint degree.

[0043] The robot is also equipped with palm fixing straps and finger fixing straps. When the user uses the robot for training, the palm fixing straps and finger fixing straps of the robot are used to bind the hand to the robot. Then the robot makes training movements based on the training instructions, such as grasping, holding, pinching, etc. It drives the user's hand to make corresponding training movements, realizing the user's passive training. The robot can also collect the training movement information of the user's own training.

[0044] Specifically, the first training action information includes the joint angles of each joint of the manipulator when the user's hand performs the training action. The training action information of the action performed by the user's hand is collected by the manipulator to improve the accuracy of the collected information. When the computing device receives the first training action information of the user collected by the manipulator, the basic information of the user is obtained to provide data support for subsequent processing.

[0045] S202: Determine a target training object according to the basic information and the first training action information.

[0046] Wherein, the target training object is determined according to the basic information of the user and the first training action information, and the adaptability of the determined target training object is improved.

[0047] S203, controlling the display device to display a first interface.

[0048] Wherein, the first interface is an interface generated according to the image of the target training object. Specifically, after determining the target training object, the first interface is generated according to the image of the target training object. Figure 4 , Figure 4 This is a schematic diagram of an interface of a display device provided in an embodiment of the present application. Figure 4 As shown, the interface of the display device includes a first display area 41 and a second display area 42, wherein the first display area 41 includes fixed icons, such as a return icon on the left side of the first display area 41 representing return, and a display icon on the right side of the first display area 41 representing the year XXXX / XX / XX and the specific time XX:XX. After determining the target training object, the image of the target training object can be displayed in the second display area 42. For example, Figure 4 As shown, the target training object displayed in the second display area 42 is an image of the first vase 421. This allows the user to directly view the target training object. It is understandable that the second display area 42 may also be provided with prompt information, which is used to prompt the user to use the target training action to interact with the target training object. For example, if the target training action is a grasping action, the prompt information may be: Please grasp and lift the target training object. This improves the user experience.

[0049] S204: When the display device displays the first interface, obtain second training motion information of the user collected by the robot.

[0050] When the display device displays the first interface, the computing device obtains the second training motion information of the user collected by the robot arm to provide data support for the subsequent generation of target image information.

[0051] S205: Generate target image information according to the second training action information.

[0052] The target image information is used to update the dynamic target image information of the target training object in the first interface. The target image information is generated according to the second training action information, so that the generated target image information is more accurate and in line with the actual situation of the user, so that the content displayed by the display device is more synchronized.

[0053] S206, controlling the display device to display the target image information.

[0054] After the target image information is generated, the display device is controlled to display the target image information.

[0055] It can be seen that in this example, the target training object is determined based on the first training action information, so that the determined target training object is more personalized, and the user updates the dynamic target image information of the target training object in the display device based on the second training action information, so that the determined target image information is more in line with actual needs, and the synchronization of the display content of the display device is improved, thereby improving interactivity, making the training process more interesting, and thereby improving user training enthusiasm.

[0056] In a possible example, the basic information includes age data and body data of the user, and determining the target training object based on the basic information and the first training action information includes: obtaining the target training action in the first training action information; searching for a reference training object corresponding to the target training action from a preset database; and adjusting the reference training object according to the age data and the body data in the basic information to obtain the target training object.

[0057] In a specific example, the basic information includes the user's age data and body data. First, the joint angles of each joint of the manipulator in the first training action information are obtained, and the target training action is determined according to the joint angles of each joint of the manipulator. Then, the reference training object corresponding to the target training action is searched from the preset database. Since the users who use the manipulator are different, the reference training object is further adjusted according to the user's basic information to obtain the target training object, so that the target training object is more personalized. Specifically, the reference training object is adjusted according to the age data and body data in the basic information to obtain the target training object.

[0058] It can be seen that, in this example, the target training object is obtained based on the basic information of the user, so that the obtained target training object is more personalized.

[0059] In one possible example, a training action may correspond to a type of training object, and then a target training object is determined from the type of training objects based on historical training information. For example, if the training action is a gripping action, and the training objects corresponding to the training action are a water cup, a vase, and a toilet paper roll. Based on the user's historical training information, it is determined that the display device has displayed the training object of a water cup during the historical training period, then a random object is preferentially selected from the vase and the toilet paper roll as the target training object, thereby further increasing the fun of the training process.

[0060] In a possible example, adjusting the reference training object according to the age data and the body data in the basic information to obtain the target training object includes: determining first adjustment data corresponding to the age data for adjusting the size of the reference training object; determining second adjustment data corresponding to the body data for adjusting the mass of the reference training object; and adjusting the reference training object according to the first adjustment data and the second adjustment data to obtain the target training object.

[0061] In a specific example, the first adjustment data for adjusting the reference training object corresponding to the age data is obtained, and the first adjustment data is used to adjust the size of the reference training object. Since the younger the user is, the smaller the object that can be grasped is. For example, if the reference training object is determined to be a cup, and the initial diameter of the cup is 6 cm. The adjustment data for adjusting the diameter of the cup corresponding to the age of 8 to 10 years old is minus 2 cm. If the age value in the user's age data is determined to be 9, the corresponding first adjustment data is minus 2 cm, and the reference training object is adjusted according to the first adjustment data, so that the diameter of the target training object obtained after adjustment is 4 cm. It can be understood that different adjustment data can include multiple adjustment sub-data, and each adjustment sub-data is used to adjust different parts of the reference training object. For example, if the reference training object is a book, the adjustment data for adjusting the length, width and height of the book corresponding to the age of 8 to 10 years old can be set to minus 3 cm, minus 2 cm and minus 0.5 cm respectively. Different age ranges are set in advance. It can be understood that different adjustment data for adjusting different training objects corresponding to different age ranges can be set according to actual needs, and there is no limitation here.

[0062] Get the second adjustment data corresponding to the body data, wherein the second adjustment data is used to adjust the mass of the reference training object, and the mass value is proportional to the preset training duration. The healthier the user is, the longer the corresponding training duration can be. Therefore, the mass of the corresponding reference training object is determined according to the body data, so that the obtained target training object is more personalized, and the user can more intuitively determine the remaining training time during the training process. For example, if the determined reference training object is the second vase, the second vase has two flowers, the mass value corresponding to each flower is 1, and the training duration corresponding to each flower is 1 minute. If the determined second adjustment data is 2, please refer to Figure 5 , Figure 5 A schematic diagram of a target training object provided in an embodiment of the present application. The reference training object is adjusted according to the second adjustment data to obtain the target training object, such as Figure 5 As shown, the target training object is the third vase 51, and four flowers are inserted in the third vase 51. During user training, the number of flowers in the third vase 51 can be reduced by one for every one minute of training, further improving the fun of the user training process. It is understandable that the part of the training object used to characterize the mass can be determined according to the actual conditions of different training objects. For example, when the mass of a water cup increases, the water in the cup increases, so the part of the training object used to characterize the mass is not limited here.

[0063] The first training action information further includes the user's body spasm level, and the user's body data is determined based on the body spasm level and the professional's rating of the user's body. For example, if the body spasm is greater, the corresponding level value is lower, and the sum of the body spasm level value and the rating value is calculated, and the corresponding second adjustment data is determined according to the preset value range to which the sum belongs. Different preset value ranges are pre-set, and different preset value ranges correspond to different adjustment data.

[0064] After the first adjustment data and the second adjustment data are determined, the reference training object is adjusted based on the first adjustment data and the second adjustment data.

[0065] It can be seen that in this example, the target training object is determined based on the user's basic information, so that the determined target training object is more personalized, thereby improving the user's training enthusiasm.

[0066] In a possible example, generating target image information according to the second training action information includes: obtaining movement information of the manipulator in the second training action information; if the manipulator is determined to move according to the movement information, obtaining joint angle information of the manipulator after it starts to move from the second training action information; obtaining a set of joint angle range values ​​corresponding to the target training object, the joint angle range value set including a preset set of different joint angle range values ​​corresponding to different joints of the manipulator when the target training action is used to interact with the target training object; judging whether the manipulator has successfully interacted with the target training object through the training action corresponding to the joint angle information according to the joint angle information and the joint angle range value set; if it is judged that the manipulator has successfully interacted with the target training object through the training action corresponding to the joint angle information, determining the moving route of the target training object according to the movement information; generating motion information of the target training object according to the joint angle information; and generating the target image information according to the moving route and the motion information.

[0067] In a specific example, after obtaining the second training action information, the movement information of the manipulator in the second training action information is extracted. Specifically, the manipulator includes a gyroscope, and the rotation angle of the manipulator compared to the initial position is collected in real time based on the gyroscope. If the rotation angle is greater than the preset rotation angle, the movement of the manipulator is determined.

[0068] In a possible example, the manipulator may also be integrated with an accelerometer to measure the acceleration of the manipulator through the accelerometer and combine it with the rotation angle collected by the gyroscope to determine whether the manipulator moves, thereby improving the accuracy of the determination result.

[0069] If it is determined that the manipulator moves, the joint angle information after the manipulator starts to move is obtained from the second training action information. Since the joint angles of users of different ages and physical conditions vary greatly when performing training actions, a set of joint angle range values ​​corresponding to the target training object is obtained, and the accuracy of the judgment result is improved by judging whether the manipulator successfully interacts with the target training object through the training action corresponding to the joint angle information based on the joint angle information and the corresponding set of joint angle range values. If it is judged that the manipulator successfully interacts with the target training object through the training action corresponding to the joint angle information, the moving route of the target training object is determined based on the movement information. And, the motion information of the target training object is generated based on the joint angle information. Finally, the target image information is generated based on the moving route and the motion information.

[0070] For example, see Figure 6 , Figure 6 This is a schematic diagram of the movement of a target training object in a display device interface provided in an embodiment of the present application. Figure 6 As shown, if the target training object is Figure 6 After the fourth vase determines the moving route according to the moving information, the fourth vase moves from the starting point 61 of the moving route to the ending point 62 of the moving route. During the moving process, the fourth vase moves based on the motion information generated by the joint angle information. Specifically, the moving speed of the fourth vase can be determined according to the speed of the manipulator movement in the moving information, so as to further improve the synchronization of the training action information collected by the display device and the manipulator.

[0071] If it is determined that the manipulator fails to interact with the target training object through the training action corresponding to the joint angle information, training failure image information is generated according to the target training object and the preset image information, and the display device is controlled to display the training failure image information.

[0072] It can be seen that in this example, after determining that the manipulator has moved, the joint angle of the manipulator after the start of the movement is determined to determine whether the manipulator can interact with the target training object through the training action of the joint angle, thereby improving the accuracy of the determination result. And based on the movement information and joint angle information, the movement route and motion information of the target training object are determined, so that the target image information content is more correlated with the second training action information, thereby improving the interactivity.

[0073] In one possible example, judging whether the manipulator has successfully interacted with the target training object through the training action corresponding to the joint angle information based on the joint angle information and the joint angle range value set includes: extracting the maximum joint angle and the minimum joint angle corresponding to each joint in the joint angle information; searching the joint angle range value corresponding to each joint from the joint angle range value set; if the maximum joint angle and the minimum joint angle corresponding to each joint are within the joint angle range value, judging that the manipulator has successfully interacted with the target training object through the training action corresponding to the joint angle information.

[0074] In a specific example, the maximum joint angle and the minimum joint angle corresponding to each joint in the joint angle information are extracted, and the joint angle range value corresponding to each joint of the manipulator is obtained from the joint angle range value set. It is determined whether the maximum joint angle and the minimum joint angle corresponding to each joint are within the corresponding joint angle range value. For example, the maximum joint angle and the minimum joint angle corresponding to the proximal interphalangeal joint between the first phalanx and the second phalanx of the thumb in the joint angle information are extracted, the joint angle range value corresponding to the proximal interphalangeal joint of the thumb is determined, and it is determined whether the maximum joint angle and the minimum joint angle corresponding to the proximal interphalangeal joint are within the joint angle range value corresponding to the proximal interphalangeal joint. If it is determined that the maximum joint angle and the minimum joint angle of each joint in the joint angle information are within the corresponding joint angle range value, it is determined that the manipulator has successfully interacted with the target training object through the training action corresponding to the joint angle information.

[0075] It can be seen that in this example, based on the maximum joint angle and the minimum joint angle corresponding to each joint, it is determined whether the robot successfully interacts with the target training object through the training action corresponding to the joint angle information, thereby improving the determination efficiency and the accuracy of the determination results.

[0076] In a possible example, generating the motion information of the target training object according to the joint angle information includes: if the target training action is a preset type of training action, and it is determined according to the joint angle information that the joint angle of the manipulator changes from small to large, then obtaining the first joint change time period in the joint angle information and the maximum joint angle and the minimum joint angle corresponding to each joint; calculating the difference between the maximum joint angle and the minimum joint angle corresponding to each joint, and obtaining a difference set composed of the differences; searching for the maximum difference in the difference set; searching for the falling speed value of the target training object within a preset first unit time length from a preset database according to the maximum difference; and generating the motion information according to the first joint change time period, the first unit time length and the speed value.

[0077] In a specific example, if the target training action is a training action of a preset type, the training action of the preset type is a training action of a grasping type, and the joint angle of the manipulator is determined to change from small to large when the manipulator performs the target training action according to the joint angle information. That is, the joint angle corresponding to each joint at the preset interval time point in the joint angle information is extracted, and the joint angles of each joint extracted are sorted in chronological order, and then it is determined whether the sorted joint angles change from small to large. If it is determined that the joint angles corresponding to each joint of the manipulator change from small to large, the motion information of the falling target training object is generated to further improve the interactivity and fun in the training process. Specifically, the first joint change time period in the joint angle information and the maximum joint angle and minimum joint angle corresponding to each joint are obtained. The first joint change time period is the time period from the time point when the joint begins to change in the joint angle information to the time point when the joint change stops. Then the difference between the maximum joint angle and the minimum joint angle corresponding to each joint is calculated to obtain a difference set. For example, if the index finger includes a first joint and a second joint, the first difference and the second difference corresponding to the first joint and the second joint are calculated respectively, and the first difference and the second difference are stored in the difference set. Find the maximum difference in the difference set, and find the speed value of the target training object falling within the preset first unit time from the preset database according to the maximum difference, so that the determined speed value is more reasonable. It is understandable that the unit time can be set according to actual needs and is not limited here. Generate the motion information of the target training object falling according to the first joint change time period, the first unit time and the speed value.

[0078] It can be seen that in this example, when the target training action is determined to be a preset type of training action, when the manipulator joint angle is determined to change from small to large based on the joint angle information, the motion information of the target training object falling is further generated, thereby increasing the fun of the training process.

[0079] In a possible example, the generating of the motion information of the target training object according to the joint angle information includes: if it is determined according to the joint angle information that the joint angle of the manipulator is sometimes large and sometimes small, then obtaining a second joint change time period and a plurality of joint angle total values ​​in the joint angle information, the joint angle total value being the sum of the average joint angles of each joint of the manipulator within a single preset time period, each of the plurality of joint angle total values ​​corresponding to a different preset time period, the preset time period being a time period within the second joint change time period; calculating the standard deviation between the plurality of joint angle total values; searching a preset database for the jitter frequency of the target training object within a preset second unit time length according to the standard deviation; and generating the motion information according to the second joint change time period, the second unit time length and the jitter frequency.

[0080] In a specific example, if it is determined according to the joint angle information that the joint angle of the manipulator is sometimes large and sometimes small when performing the target training action, the second joint change time period and the total value of multiple joint angles in the joint angle information are obtained. The second joint change time period is the time period from the time point when the joint angle changes to the time point when the change stops. The specific steps for obtaining the total value of multiple joint angles are: determine the preset duration corresponding to the preset time period, divide the second joint change time period into multiple sub-time periods with the preset duration, and calculate the average joint angle of each joint of the manipulator in each sub-time period respectively. For example, the index finger includes a first joint and a second joint, and the joint angles of the first joint in the first sub-time period are 90 degrees and 80 degrees respectively, then the average joint angle of the first joint in the first sub-time period is calculated to be 85 degrees. If the joint angle of the second joint in the first sub-time period is always 80 degrees, the average joint angle of the second joint in the first sub-time period is 80 degrees. After determining the average joint angle corresponding to each joint of the manipulator in the first sub-time period, add the average joint angles corresponding to each joint to obtain the total value of the joint angle. Similarly, calculate the total value of the joint angle corresponding to each sub-time period to obtain multiple total values ​​of the joint angle. Then, the standard deviation between the total values ​​of the multiple joint angles is calculated, and the jitter frequency corresponding to the standard deviation is searched from a preset database, wherein the larger the standard deviation, the greater the change in the joint angle of the manipulator, and the higher the corresponding jitter frequency. The motion information of the manipulator jitter is generated according to the second joint change time period, the second unit time length and the jitter frequency.

[0081] It can be seen that in this example, the manipulator motion information is generated based on the joint angle information of the manipulator, so that the generated motion information reflects the shaking of the manipulator, improves the credibility of the generated target image information, and increases the interest in the training process.

[0082] In a possible example, after controlling the display device to display the target image information, the method further includes: obtaining a preset training time corresponding to the quality and an actual training time in the second training action information, the value of the quality being proportional to the preset training time; if the actual training time is less than the preset training time, and the training action information sent by the manipulator is not received within a preset waiting time after receiving the second training action information, generating training failure image information according to the target training object and preset image information; and controlling the display device to display the training failure image information.

[0083] In a specific example, after controlling the display device to display the target image information, the preset training duration corresponding to the quality and the actual training duration in the second training action information are obtained. The actual training duration is the time point from the time point when the manipulator moves to the time point when the joint angle corresponding to a single joint of the manipulator in the second training action information exceeds the corresponding joint angle range value, or, if the joint angle corresponding to each joint of the manipulator in the second training action information does not exceed the corresponding joint angle range value, the end time point of the second training action information is used as the end time point of the actual training duration. If the actual training duration is less than the preset training duration, it means that the actual training duration does not meet the training requirements. Therefore, it is further determined whether the training action information after the second training action information is received within the preset waiting time. If not received, the training failure image information is generated according to the target training object and the preset image information; the display device is controlled to display the training failure image information.

[0084] It can be seen that in this example, when it is determined that the actual training time does not reach the preset training time, training failure image information is generated to improve the interest of the training process.

[0085] In a possible example, the interface of the display device may also include an image of a robot. Figure 7 , Figure 7 A schematic diagram of an image of a robot included in a display device interface provided in an embodiment of the present application. Figure 7 As shown in FIG. 1 , if the target training action corresponding to the manipulator is determined to be a grasping action according to the first training action information, then Figure 7 The robot is shown in the grasping state, and if the target training object is determined to be the fifth vase 71 according to the first training action information. The robot moves according to the second training action information, and the robot successfully interacts with the target training object through the training action corresponding to the joint angle information, then the interaction information between the robot and the fifth vase 71 can be generated, and the interaction information is specifically that the robot successfully grasps the fifth vase 71. The target image information is generated according to the interaction information, the moving route and the motion information, so that the target image information is more intuitive, thereby improving the interactivity and fun.

[0086] See also Figure 8 , Figure 8 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. Figure 8As shown, the computing device includes a processor 801, a memory 803, a communication module 802, a program 804 and a Bluetooth module 805, and the processor 801 is connected to the memory 803, the communication module 802 and the Bluetooth module 805 through an internal communication bus. It is understood that the number of processors 801 can be set according to actual needs, for example, one, two or more, and the number of processors 801 is not specifically limited here. The number of programs 804 can be set according to actual needs, for example, one, two or more, and the number of programs 804 is not specifically limited here.

[0087] The program 804 is stored in the memory 803 and executed by the processor 801. The program 804 includes instructions for executing any step in the following method embodiments.

[0088] Among them, the processor 801 can be, for example, a central processing unit (CPU), a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, units and circuits described in conjunction with the disclosure of this application. The processor 801 can also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The communication unit can be a communication module 802, a transceiver, a transceiver circuit, etc., and the storage unit can be a memory 803.

[0089] The memory 803 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM).

[0090] Among them, the computing device can interact with the robot arm for data through the communication module 802, and can also interact with the robot arm for data through the Bluetooth module 805, and there is no specific limitation here.

[0091] It is understandable that the display device and the computing device may be provided separately, or the display device may be integrated into the computing device for an integrated provision, and no specific limitation is made here.

[0092] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the execution process on the method side. It is understandable that, in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0093] The embodiment of the present application can divide the electronic device into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0094] In the case of dividing each functional module according to each function, the following is combined Fig. 9 The processing device for synchronous training data in the embodiment of the present application is described in detail. Fig. 9 A functional unit composition block diagram of a synchronous training data processing device provided in an embodiment of the present application, a synchronous training data processing device, applied to a computing device in a hand function training system, the hand function training system comprising the computing device, a display device and a manipulator connected to the computing device, the device comprising:

[0095] A first acquisition unit 901 is configured to acquire basic information of the user upon receiving first training motion information of the user collected by the robot;

[0096] A determination unit 902, configured to determine a target training object according to the basic information and the first training action information;

[0097] A first control unit 903, used to control the display device to display a first interface, where the first interface is an interface generated according to the image of the target training object;

[0098] A second acquisition unit 904 is used to acquire the second training motion information of the user collected by the manipulator when the display device displays the first interface;

[0099] A generating unit 905, configured to generate target image information according to the second training action information, wherein the target image information is target image information used to update the dynamic state of the target training object in the first interface;

[0100] The second control unit 906 is used to control the display device to display the target image information.

[0101] In a possible example, the basic information includes the age data and body data of the user, and the determination unit 902 is further used to: obtain the target training action in the first training action information; and search for a reference training object corresponding to the target training action from a preset database; and adjust the reference training object according to the age data and the body data in the basic information to obtain the target training object.

[0102] In a possible example, the determination unit 902 is further used to: determine first adjustment data corresponding to the age data for adjusting the size of the reference training object; and determine second adjustment data corresponding to the body data for adjusting the mass of the reference training object; and adjust the reference training object according to the first adjustment data and the second adjustment data to obtain the target training object.

[0103] In a possible example, the generating unit 905 is further used to: obtain the movement information of the manipulator in the second training action information; and if the movement of the manipulator is determined based on the movement information, obtain the joint angle information of the manipulator after the manipulator starts to move from the second training action information; and obtain a set of joint angle range values ​​corresponding to the target training object, the joint angle range value set including a preset set of different joint angle range values ​​corresponding to different joints of the manipulator when the target training action is used to interact with the target training object; and determine whether the manipulator successfully interacts with the target training object through the training action corresponding to the joint angle information based on the joint angle information and the joint angle range value set; and if it is determined that the manipulator successfully interacts with the target training object through the training action corresponding to the joint angle information, determine the moving route of the target training object based on the movement information; and generate the motion information of the target training object based on the joint angle information; and generate the target image information based on the moving route and the motion information.

[0104] In a possible example, the generation unit 905 is also used to: extract the maximum joint angle and the minimum joint angle corresponding to each joint in the joint angle information; and search for the joint angle range value corresponding to each joint from the joint angle range value set; and if the maximum joint angle and the minimum joint angle corresponding to each joint are within the joint angle range value, it is determined that the manipulator has successfully interacted with the target training object through the training action corresponding to the joint angle information.

[0105] In a possible example, the generating unit 905 is further used for: if the target training action is a preset type of training action, and it is determined according to the joint angle information that the joint angle of the manipulator changes from small to large, then obtaining the first joint change time period in the joint angle information and the maximum joint angle and the minimum joint angle corresponding to each joint; and calculating the difference between the maximum joint angle and the minimum joint angle corresponding to each joint to obtain a difference set composed of the differences; and finding the maximum difference in the difference set; and according to the maximum difference, searching from a preset database for the speed value of the falling of the target training object within a preset first unit time length; and generating the motion information according to the first joint change time period, the first unit time length and the speed value.

[0106] In a possible example, the generating unit 905 is further used for: if it is determined according to the joint angle information that the joint angle of the manipulator is sometimes large and sometimes small, then obtaining a second joint change time period and a plurality of joint angle total values ​​in the joint angle information, the joint angle total value being the sum of the average joint angles of each joint of the manipulator within a single preset time period, each of the plurality of joint angle total values ​​corresponding to a different preset time period, the preset time period being a time period within the second joint change time period; and calculating the standard deviation between the plurality of joint angle total values; and searching a preset database for the jitter frequency of the target training object within a preset second unit time length according to the standard deviation; and generating the motion information according to the second joint change time period, the second unit time length and the jitter frequency.

[0107] In a possible example, the device also includes a third acquisition unit, which is used to obtain the preset training time corresponding to the quality and the actual training time in the second training action information, and the value of the quality is proportional to the preset training time; and if the actual training time is less than the preset training time, and the training action information sent by the manipulator is not received within the preset waiting time after receiving the second training action information, then generating training failure image information according to the target training object and preset image information; and controlling the display device to display the training failure image information.

[0108] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium may be a solid-state hard disk.

[0109] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps of any method described in the above method embodiments, and the above computer includes an electronic device.

[0110] An embodiment of the present application further provides a computer program product, which includes a computer program. The computer program can be operated to enable a computer to execute part or all of the steps of any method recorded in the above method embodiments.

[0111] The computer program product may be a software installation package, and the computer includes an electronic device.

[0112] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0113] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and systems can be implemented in other ways. For example, the device embodiments described above are merely schematic; for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0114] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0115] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0116] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0117] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions without departing from the spirit and scope of the present invention, and can make various changes and modifications, including the combination of the above-mentioned different functions and implementation steps, including software and hardware implementation methods, all of which are within the scope of protection of the present invention.

Claims

1. A method for processing synchronous training data, characterized in that: A computing device is applied to a hand function training system, the hand function training system comprises the computing device, a display device connected to the computing device and a manipulator, and the method comprises: If the first training motion information of the user collected by the robot is received, the basic information of the user is obtained; determining a target training object according to the basic information and the first training action information; Controlling the display device to display a first interface, where the first interface is an interface generated according to the image of the target training object; When the display device displays the first interface, obtaining second training action information of the user collected by the manipulator; generating target image information according to the second training action information, wherein the target image information is target image information used to update the dynamic state of the target training object in the first interface; Control the display device to display the target image information.

2. The method according to claim 1, characterized in that The basic information includes age data and physical data of the user, and determining the target training object according to the basic information and the first training action information includes: Obtaining a target training action in the first training action information; Searching for a reference training object corresponding to the target training action from a preset database; The reference training object is adjusted according to the age data and the body data in the basic information to obtain the target training object.

3. The method according to claim 2, characterized in that The step of adjusting the reference training object according to the age data and the body data in the basic information to obtain the target training object includes: determining first adjustment data corresponding to the age data for adjusting the size of the reference training object; determining second adjustment data corresponding to the body data for adjusting the mass of the reference training object; The reference training object is adjusted according to the first adjustment data and the second adjustment data to obtain the target training object.

4. The method according to claim 2 or 3, characterized in that: The generating target image information according to the second training action information includes: Acquire movement information of the manipulator in the second training action information; If it is determined according to the movement information that the manipulator moves, then obtaining the joint angle information after the manipulator starts to move from the second training motion information; Acquire a set of joint angle range values ​​corresponding to the target training object, wherein the set of joint angle range values ​​includes a set of different joint angle range values ​​corresponding to different joints of the manipulator when the target training action is used to interact with the target training object; According to the joint angle information and the joint angle range value set, determining whether the manipulator successfully interacts with the target training object through the training action corresponding to the joint angle information; If it is determined that the manipulator successfully interacts with the target training object through the training action corresponding to the joint angle information, determining the movement route of the target training object according to the movement information; generating motion information of the target training object according to the joint angle information; The target image information is generated according to the moving route and the motion information.

5. The method according to claim 4, characterized in that The step of judging, based on the joint angle information and the joint angle range value set, whether the manipulator successfully interacts with the target training object through the training action corresponding to the joint angle information comprises: Extracting the maximum joint angle and the minimum joint angle corresponding to each joint in the joint angle information; Searching for the joint angle range value corresponding to each joint from the joint angle range value set; If the maximum joint angle and the minimum joint angle corresponding to each joint are within the joint angle range, it is determined that the manipulator successfully interacts with the target training object through the training action corresponding to the joint angle information.

6. The method according to claim 4, characterized in that The step of generating the motion information of the target training object according to the joint angle information includes: If the target training action is a training action of a preset type, and it is determined according to the joint angle information that the joint angle of the manipulator changes from small to large, then the maximum joint angle and the minimum joint angle corresponding to each joint in the first joint change time period in the joint angle information are obtained; Calculate the difference between the maximum joint angle and the minimum joint angle corresponding to each joint to obtain a difference set consisting of the differences; Find the maximum difference in the difference set; According to the maximum difference, searching a preset database for a falling speed value of the target training object within a preset first unit time length; The motion information is generated according to the first joint change time period, the first unit duration and the speed value.

7. The method according to claim 4, characterized in that The step of generating the motion information of the target training object according to the joint angle information includes: If it is determined according to the joint angle information that the joint angle of the manipulator is sometimes large and sometimes small, then a second joint change time period and a plurality of joint angle total values ​​in the joint angle information are obtained, the joint angle total value being the sum of the average joint angles of each joint of the manipulator within a single preset time period, each joint angle total value in the plurality of joint angle total values ​​corresponding to a different preset time period, and the preset time period being a time period within the second joint change time period; Calculating a standard deviation between the plurality of joint angle total values; searching a preset database for the shaking frequency of the target training object within a preset second unit time length according to the standard deviation; The motion information is generated according to the second joint change time period, the second unit duration and the jitter frequency.

8. The method according to claim 3, characterized in that After controlling the display device to display the target image information, the method further includes: Obtaining a preset training duration corresponding to the quality and an actual training duration in the second training action information, wherein the value of the quality is proportional to the preset training duration; If the actual training duration is less than the preset training duration, and the training action information sent by the manipulator is not received within the preset waiting time after receiving the second training action information, training failure image information is generated according to the target training object and the preset image information; Control the display device to display the training failure image information.

9. A synchronous training data processing device, characterized in that: A computing device applied to a hand function training system, the hand function training system comprising the computing device, a display device and a manipulator connected to the computing device, the device comprising: A first acquisition unit, configured to acquire basic information of the user upon receiving first training motion information of the user collected by the manipulator; a determination unit, configured to determine a target training object according to the basic information and the first training action information; A first control unit, configured to control the display device to display a first interface, wherein the first interface is an interface generated according to the image of the target training object; A second acquisition unit, configured to acquire, when the display device displays the first interface, the second training motion information of the user collected by the manipulator; a generating unit, configured to generate target image information according to the second training action information, wherein the target image information is target image information used to update the dynamic state of the target training object in the first interface; The second control unit is used to control the display device to display the target image information.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 8.

Images