Tai Chi training interaction method based on knowledge graph and robot

Abstract

The embodiment of the invention discloses a Tai Chi training interaction method based on a knowledge graph and a robot, and relates to the technical field of knowledge graphs, and the method comprises the steps: collecting the image data of a target learner during Tai Chi training through the robot, obtaining the image features and action types of the target learner during Tai Chi training through the image data extraction, and obtaining the image features and action types of the target learner during Tai Chi training; generating corresponding question text data; traversing in a pre-created knowledge graph based on the question text data to obtain a first hypergraph and a second hypergraph, generating a first answer entity, and outputting a second answer entity having the highest similarity with the first answer entity in the pre-created knowledge graph; and displaying a Tai Chi action related to the second answer entity through the robot. According to the method and the device, the entities of complicated categories and the corresponding Tai Chi motion images can be processed, the problems of more Tai Chi motion contents, strong classification crossing and difficulty in reasoning are solved, the efficiency of searching the entity contents is improved, and the comprehensiveness and accuracy of intelligent teaching can be improved.

Description

Technical Field

[0001] The present application relates to the field of knowledge graph technology, and in particular to a Tai Chi training interaction method based on knowledge graph and robot. Background Art

[0002] As an intangible cultural heritage, Tai Chi is a form of exercise that is widely accepted and deeply loved by people. There are hundreds of known schools, and different schools of Tai Chi have different moves and standards, which can cause certain learning difficulties for beginners.

[0003] Therefore, learners need guidance from teachers or coaches during the learning process of Tai Chi. However, due to the large number of learners, it is usually difficult for teachers or coaches to provide one-on-one targeted guidance to each learner. Generally, they can only lead students to practice multiple times through demonstration. For teachers, multiple demonstrations and corrections of learners' mistakes will greatly increase their workload; for students, it is necessary for teachers to demonstrate and correct mistakes multiple times in order to correct mistakes in the movement. Moreover, due to the large number of Tai Chi movements, teachers cannot guide each student one by one on the details of each movement. In most cases, students need to learn by themselves according to the teaching content combined with teaching video resources. In this way, it is difficult for students to realize whether the movements are not standard, and it is difficult to confirm which movements do not meet the standards. This can easily lead to low overall learning efficiency. It is difficult for students to actively discover problems in learning, and it is impossible to improve learning results in a targeted manner.

[0004] Therefore, there is currently a lack of a solution that can provide students with auxiliary means in the process of learning Tai Chi, and it is difficult to actively discover problems that arise during students' practice. Summary of the invention

[0005] The embodiment of the present application provides a Tai Chi training interactive method based on a knowledge graph and a robot to solve the defects in the related technology. The technical solution is as follows: In a first aspect, an embodiment of the present application provides a Tai Chi training interaction method based on a knowledge graph and a robot, comprising: The robot collects image data of the target learner during Tai Chi training; Inputting the image data into a preset image-text mapping engine, extracting the image features of the target learner when performing Tai Chi training through the image data, determining the action type corresponding to the image feature, and generating question text data corresponding to the action type; Traversing the pre-created knowledge graph based on the question text data, generating a first hypergraph based on the first triples obtained through the traversal; traversing the pre-created knowledge graph based on the image features, generating a second hypergraph based on the second triples obtained through the traversal; Combining the first hypergraph and the second hypergraph to generate a first answer entity corresponding to the question text data, calculating the similarity between the first answer entity and each answer entity in the pre-created knowledge graph, and outputting a second answer entity with the highest similarity; Send a control instruction to the robot to enable the robot to demonstrate Tai Chi movements related to the second answer entity.

[0006] In an optional solution of the first aspect, after collecting the image data of the target learner performing Tai Chi training by the robot, the method further includes: Based on the acquisition time of each image frame in the image data, generating an image sequence consisting of image frames arranged in time sequence; Performing denoising and enhancement preprocessing on each image frame in the image sequence, and outputting the preprocessed image sequence; The step of inputting the image data into a preset image-text mapping engine is performed based on the preprocessed image sequence.

[0007] In an optional solution of the first aspect, the image data is input into a preset image-text mapping engine, image features of the target learner when performing Tai Chi training are extracted through the image data, and the action type corresponding to the image feature is determined, and question text data corresponding to the action type is generated, including: Extracting image features and posture labels corresponding to the image features from the image data through the preset image-text mapping engine, determining the action category of the target learner when performing Tai Chi training based on the posture labels, constructing a question text based on the image features, the posture labels and the action categories, and generating corresponding question text data; The question text is associated with the action category and the posture label.

[0008] In an optional solution of the first aspect, a text feature representation of the question text data is extracted based on a trained language model, and a corresponding text triple is obtained by extracting each entity and the association relationship between each two entities in the text feature representation, and determining the association relationship between the entity and the image feature in the text triple; The traversing the pre-created knowledge graph based on the question text data and generating a first hypergraph based on the first triple obtained by the traversal includes: Traversing the pre-created knowledge graph based on the text triples, traversing to obtain a first triple in the pre-created knowledge graph that contains any entity in the text triples, and generating the first hypergraph based on the first triple obtained through traversal; The traversing the pre-created knowledge graph based on the image features and the corresponding action types, and generating a second hypergraph based on the second triples obtained by the traversal, comprises: Based on the association relationship between the entities in the text triples and the image features, the pre-created knowledge graph is traversed to obtain a second entity associated with the action type and posture label corresponding to the image features in the text triples in the pre-created knowledge graph, and a second triplet containing the second entity is obtained, and the second hypergraph is generated based on the second triplet and an example image associated with the second triplet.

[0009] In an optional solution of the first aspect, the combining the first hypergraph and the second hypergraph to generate a first answer entity corresponding to the question text data includes: Taking the first hypergraph as a reference and combining it with the second hypergraph, generating an output result corresponding to the first hypergraph through a self-attention mechanism; Taking the second hypergraph as a reference and combining it with the first hypergraph, an output result corresponding to the second hypergraph is generated through a self-attention mechanism; The output result corresponding to the first hypergraph and the output result corresponding to the second hypergraph are aggregated to obtain the first answer entity.

[0010] In an optional solution of the first aspect, sending a control instruction to the robot so that the robot displays Tai Chi movements related to the second answer entity includes: Acquire an example image associated with the second answer entity and a posture label and an action type corresponding to the example image, acquire an action sequence of Tai Chi actions corresponding to the posture label and the action type, and send a control instruction to the robot to complete the action sequence; The robot is controlled to complete the action sequence through the control instruction, so that the robot displays Tai Chi movements related to the second answer entity.

[0011] In an optional solution of the first aspect, sending a control instruction to the robot so that the robot displays Tai Chi movements related to the second answer entity includes: Acquire an example image of a Tai Chi movement associated with the second answer entity, and send a control instruction to the robot to display the example image; The robot is controlled by the control instruction to display Tai Chi movements corresponding to the example image on a display device of the robot.

[0012] In a second aspect, the embodiment of the present application further provides a Tai Chi training interactive device based on a knowledge graph and a robot, comprising: An image acquisition module, used to collect image data of a target learner during Tai Chi training through a robot; A question text generation module, used for inputting the image data into a preset image-text mapping engine, extracting the image features of the target learner when performing Tai Chi training through the image data, determining the action type corresponding to the image feature, and generating question text data corresponding to the action type; A knowledge graph module, configured to traverse a pre-created knowledge graph based on the question text data, and generate a first hypergraph based on a first triple obtained through the traversal; and to traverse the pre-created knowledge graph based on the image features, and generate a second hypergraph based on a second triple obtained through the traversal; The knowledge graph module is further used to combine the first hypergraph and the second hypergraph to generate a first answer entity corresponding to the question text data, calculate the similarity between the first answer entity and each answer entity in the pre-created knowledge graph, and output the second answer entity with the highest similarity; The interactive module is used to send a control instruction to the robot so that the robot can display Tai Chi movements related to the second answer entity.

[0013] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method provided in the first aspect of the embodiment of the present application or any one of the implementation methods of the first aspect is implemented.

[0014] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided by the first aspect of the embodiment of the present application or any one of the implementations of the first aspect.

[0015] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least: This application utilizes a robot to collect image information of Tai Chi practitioners practicing, and performs feature extraction based on the image information to effectively determine image features and corresponding action types, and then generate associated question text data, thereby achieving a conversion process of accurately grasping the learner's purpose and intention through image information and automatically generating question text, which can effectively solve the problems of rigid sports learning scenes, high equipment requirements, and inconvenience in asking questions during exercise.

[0016] In addition, the present application traverses the pre-constructed knowledge graph based on the question text data and the corresponding image data of Tai Chi learners, and constructs a first hypergraph related to the question text and a second hypergraph related to the image features. Then, based on the first hypergraph and the second hypergraph, the intrinsic relationship between the knowledge graph and the specific problem can be deeply mined, so that entities of complex categories and Tai Chi movement images corresponding to the entities can be processed, thereby solving the problems of Tai Chi movement with rich content, strong classification overlap, and difficult reasoning, and improving the efficiency of finding relevant entity content; the solution provided by this application can improve the comprehensiveness and accuracy of intelligent teaching. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of a Tai Chi training interactive method based on a knowledge graph and a robot provided in an embodiment of the present application; Figure 2 It is a structural schematic diagram of a picture-text mapping engine of a Tai Chi training interactive method based on a knowledge graph and a robot provided in an embodiment of the present application; Figure 3 It is a structural schematic diagram of a Tai Chi training interactive device based on a knowledge graph and a robot provided in an embodiment of the present application; Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] The terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or modules is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other steps or modules that are inherent to these processes, methods, products or devices.

[0021] It should be noted that the terms "first\second" involved in the present application are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that "first\second" can be interchanged with a specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those described or illustrated herein.

[0022] The present application is described in detail below with reference to specific embodiments.

[0023] Next, combine Figure 1 , introduces a Tai Chi training interactive method based on knowledge graph and robot provided by the embodiment of the present application. For details, please refer to Figure 1 , Figure 1 The flowchart of a Tai Chi training interactive method based on a knowledge graph and a robot provided in an embodiment of the present application is shown. Figure 1 As shown, the method comprises the following steps: S101, collecting image data of a target learner performing Tai Chi training through a robot.

[0024] S102, inputting the image data into a preset image-text mapping engine, extracting the image features of the target learner during Tai Chi training through the image data, determining the action type corresponding to the image feature, and generating question text data corresponding to the action type.

[0025] S103, traversing the pre-created knowledge graph based on the question text data, and generating a first hypergraph based on the first triples obtained through the traversal; traversing the pre-created knowledge graph based on the image features, and generating a second hypergraph based on the second triples obtained through the traversal.

[0026] S104, combining the first hypergraph and the second hypergraph to generate a first answer entity corresponding to the question text data, calculating the similarity between the first answer entity and each answer entity in the pre-created knowledge graph, and outputting a second answer entity with the highest similarity.

[0027] S105, sending a control instruction to the robot to enable the robot to demonstrate Tai Chi movements related to the second answer entity.

[0028] In some embodiments, after S101, the method further includes: Based on the acquisition time of each image frame in the image data, an image sequence consisting of image frames arranged in time sequence can be generated.

[0029] Each image frame in the image sequence is preprocessed by denoising and enhancement, and the preprocessed image sequence is output.

[0030] The step of inputting the image data into a preset image-text mapping engine in S102 is further performed based on the preprocessed image sequence.

[0031] In some embodiments, the step of inputting the image data into a preset image-text mapping engine, extracting the image features of the target learner during Tai Chi training through the image data, determining the action type corresponding to the image features, and generating question text data corresponding to the action type includes: Extracting image features and posture labels corresponding to the image features from the image data through the preset image-text mapping engine, determining the action category of the target learner when performing Tai Chi training based on the posture labels, constructing a question text based on the image features, the posture labels and the action categories, and generating corresponding question text data; The question text is associated with the action category and the posture label.

[0032] Specifically, refer to Figure 2 The image-text mapping engine shown in the figure can first process the input image data through convolution, activation, pooling, etc. to obtain a feature map, and then pass through the convolution layer, reconstruction, softmax, reconstruction, and target area, and then pass through the image detector ( OD ) Extract image features and encode them to get image labels , image label It is passed to a nonlinear multilayer perceptron to model the regional weights in the image. , then use Gumbel-Softmax Distribute the sampled object In the decoding part, the processed pose label is obtained as a mask of the image label .

[0033] Further, the processed image labels Send to text encoder and calculate the category of image classification , that is, the action categories of the target learners during Tai Chi training, such as Tai Chi standing, hand techniques, footwork and body techniques, etc., and the obtained action categories are used One-hot After the representation, it is fed into the text encoder and connected with the image encoder to feed into the question decoder to obtain the question text Q , thereby generating the corresponding question text data.

[0034] It can be understood that posture labels may include action labels of parts such as the head, hands, legs, and body. Combined with the action type, the posture of the corresponding part in the action type can be obtained, and then it can be identified that when the learner is practicing the action type, the posture of the part corresponding to the posture label may not match the label, such as identifying the movement of the legs in the image of the learner's Hunyuan Posture.

[0035] Exemplarily, a question text can be generated based on the action type - Hunyuan Posture, and the posture label - legs, on how to show the correct leg movements in the image of Hunyuan Posture. This embodiment of the present application does not limit this.

[0036] In some embodiments, the preset image-text mapping engine can be trained based on the cross entropy loss function: ; in, is the cross loss entropy, is the loss function for predicting the starting position of the answer.

[0037] In some embodiments, in S103, the text feature representation of the question text data can be extracted based on the trained language model, and the association relationship between each entity and every two entities in the text feature representation is extracted to obtain the corresponding text triples, and the association relationship between the entities and image features in the text triples is determined.

[0038] Specifically, the question text data can be input into the pre-trained language model BERT, and the BERT model can be used to obtain the features of the question text, and then extract the entities in the question. and relationship , and has been based on the image detector ( OD )Get the posture label , based on entity and relationship And posture tags Perform entity connection, which is equivalent to entity-based and relationship Construct text triples and establish text triples and posture labels Since posture labels are also associated with corresponding action types, the deep relationship between each entity and posture labels and action types can be determined.

[0039] The method may be to traverse the pre-created knowledge graph based on the question text data, and generate a first hypergraph based on the first triplet obtained by the traversal, specifically including: The pre-created knowledge graph is traversed based on the text triples to obtain a first triple in the pre-created knowledge graph that contains any entity in the text triples, and the first hypergraph is generated based on the first triple obtained through the traversal.

[0040] Each word unit in the text triple corresponding to the question text data can be extracted, and the same node in the pre-created knowledge graph can be traversed based on each word unit as the starting node to obtain the triple corresponding to the same node. Multiple triples can be regarded as a set of relationships between entities, thereby obtaining multiple triples covering all such units in the question text data from the pre-created knowledge graph and outputting the first hypergraph.

[0041] The traversing the pre-created knowledge graph based on the image features and the corresponding action types, and generating a second hypergraph based on the second triples obtained by the traversal, comprises: Based on the association relationship between the entities in the text triples and the image features, the pre-created knowledge graph is traversed to obtain a second entity associated with the action type and posture label corresponding to the image features in the text triples in the pre-created knowledge graph, and a second triplet containing the second entity is obtained, and the second hypergraph is generated based on the second triplet and an example image associated with the second triplet.

[0042] In some embodiments, the pre-constructed knowledge graph in the above embodiments can be constructed based on information extraction of Tai Chi assessment standards, action essentials, books and journals, teaching materials, competition videos, demonstration images and other data. The basic movements of Tai Chi are composed of four major categories: standing exercises, hand techniques, footwork and body techniques. According to the different upper and lower relationships, each major category is divided into several subcategories; due to its changing evolution relationship, its movements can be divided into five major schools according to the schools, and each school has different moves; at the same time, combined with the characteristics of Tai Chi's yin and yang harmony, the knowledge graph can effectively represent its structural symmetry relationship.

[0043] Specifically, Figure 2 As shown, the pre-created knowledge graph can be traversed to obtain triplets with the same association relationship between entities and image features as those in the text triplets, and the triplets and entity images corresponding to the same posture labels and action types can be extracted. Figure 2 The posture labels shown in the figure are head, elbow, shoulder and leg. The action type of the image feature corresponding to the text triple is Hunyuanzhu. The entities containing Hunyuanzhu, head, elbow and leg can be extracted, and the images corresponding to these entities in the pre-created knowledge graph can be obtained, such as Figure 2 The example image and text description of the correct leg movement shown, that is, the example image mentioned above, are used to construct the second hypergraph.

[0044] Exemplarily, starting from the entity "double peaks piercing the ears" and using the relationship "hand movement" as the relationship, traverse the relationship line segment string. Through the construction of a hypergraph, there are arbitrarily many entities on each hyperedge, enabling better reasoning between knowledge.

[0045] Further, perform the steps of S104, specifically including: As Figure 2 shown, respectively obtain the first hypergraph embedding graph corresponding to the first hypergraph and the second hypergraph embedding graph corresponding to the second hypergraph, and apply the formula: ; where is the hyperedge in is the hyperedge embedding function, is the linear projection function. Then, pack the hyperedge representations in and in the same hypergraph into matrices .

[0046] Further, taking the first hypergraph embedding graph as a reference, combined with the second hypergraph embedding graph , generate the output result corresponding to the first hypergraph through the self-attention mechanism; Taking the second hypergraph embedding graph as a reference, combined with the first hypergraph embedding graph , generate the output result corresponding to the second hypergraph through the self-attention mechanism; Aggregate the output result corresponding to the first hypergraph and the output result corresponding to the second hypergraph to obtain the first answer entity.

[0047] Specifically including: Define the knowledge hyperedge and the problem hyperedge as query and key-value pairs respectively, set the query as , the key as , and the value as , where is the projection matrix. After attention calculation: ; where is the dimension of the query.

[0048] Through layer normalization , residual connection , and finally through the fully connected layer aggregate into a single vector representation, which are respectively the output results corresponding to the first hypergraph The output corresponding to the second hypergraph , after aggregation, the output result is ; in Representation layer normalization operation, .

[0049] Determine the first answer entity The specific steps include: ; in is the weight matrix, is the bias term.

[0050] Further, the similarity between the first answer entity and each answer entity in the pre-created knowledge graph is calculated, and the second answer entity with the highest similarity is output, that is, each answer entity in the pre-created knowledge graph is compared with the above first answer entity, and the second answer entity with the highest similarity with the first answer entity in the pre-created knowledge graph is output, such as Figure 2 As shown, the question is how to show the correct leg movement in the image of Hunyuan Posture, and the extracted answer is: the knee should be bent, and straighten in the bend. This embodiment of the application does not limit this.

[0051] In some embodiments, in S105, the correct action corresponding to the second answer entity may be displayed by a robot. For example, a control instruction may be sent to the robot to enable the robot to display Tai Chi actions related to the second answer entity, including: Acquire an example image associated with the second answer entity and a posture label and an action type corresponding to the example image, acquire an action sequence of Tai Chi actions corresponding to the posture label and the action type, and send a control instruction to the robot to complete the action sequence; The robot is controlled to complete the action sequence through the control instruction, so that the robot displays Tai Chi movements related to the second answer entity.

[0052] In some embodiments, an example image of a Tai Chi movement associated with the second answer entity may also be obtained, and a control instruction to display the example image may be sent to the robot; The robot is controlled by the control instruction to display Tai Chi movements corresponding to the example image on a display device of the robot.

[0053] It is understandable that the embodiments of the present application do not limit the manner in which the robot exhibits correct actions.

[0054] The following are device embodiments of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0055] See next Figure 3 , which is a schematic diagram of the structure of a Tai Chi training interaction device based on a knowledge graph and a robot provided by an exemplary embodiment of the present application. The device can be implemented as all or part of a terminal through software, hardware, or a combination of both, and can also be integrated on a server as an independent module. The Tai Chi training interaction device 30 based on a knowledge graph and a robot in the embodiment of the present application can be applied to a terminal or a cloud. The device 30 includes an image acquisition module 301, a question text generation module 302, a knowledge graph module 303, and an interaction module 304, wherein: The image acquisition module 301 is used to collect image data of the target learner when performing Tai Chi training through the robot; The question text generation module 302 is used to input the image data into a preset image-text mapping engine, extract the image features of the target learner when performing Tai Chi training through the image data, determine the action type corresponding to the image feature, and generate question text data corresponding to the action type; The knowledge graph module 303 is used to traverse the pre-created knowledge graph based on the question text data, and generate a first hypergraph based on the first triple obtained by the traversal; traverse the pre-created knowledge graph based on the image features, and generate a second hypergraph based on the second triple obtained by the traversal; The knowledge graph module 303 is further used to generate a first answer entity corresponding to the question text data by combining the first hypergraph and the second hypergraph, calculate the similarity between the first answer entity and each answer entity in the pre-created knowledge graph, and output the second answer entity with the highest similarity; The interaction module 304 is used to send a control instruction to the robot so that the robot can display Tai Chi movements related to the second answer entity.

[0056] It should be noted that the device 30 provided in the above embodiment only uses the division of the above functional modules as an example when executing the Tai Chi training interaction method based on the knowledge graph and the robot. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiment and the embodiment of the Tai Chi training interaction method based on the knowledge graph and the robot belong to the same concept. The implementation process is detailed in the method embodiment, which will not be repeated here.

[0057] An embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the program.

[0058] See also Figure 4 , which is a structural block diagram of an electronic device provided in an embodiment of the present application.

[0059] like Figure 4 As shown, the electronic device 400 includes: a processor 401 and a memory 402 .

[0060] In the embodiment of the present application, the processor 401 is the control center of the computer system, which can be a processor of a physical machine or a processor of a virtual machine. The processor 401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array).

[0061] The processor 401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also called a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state.

[0062] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments of the present application, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one instruction, which is used to be executed by the processor 401 to implement the method in the embodiment of the present application.

[0063] In some embodiments, the electronic device 400 further includes: a peripheral device interface 403 and at least one peripheral device 404. The processor 401, the memory 402 and the peripheral device interface 403 can be connected via a bus or a signal line. Each peripheral device 404 can be connected to the peripheral device interface 403 via a bus, a signal line or a circuit board. Specifically, the peripheral device 404 includes: a display screen, a camera and an audio circuit. The peripheral device interface 403 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 401 and the memory 402.

[0064] In some embodiments of the present application, the processor 401, the memory 402, and the peripheral device interface 403 are integrated on the same chip or circuit board; in some other embodiments of the present application, any one or two of the processor 401, the memory 402, and the peripheral device interface 403 can be implemented on a separate chip or circuit board. This embodiment of the present application does not specifically limit this.

[0065] The electronic device structure block diagram shown in the embodiment of the present application does not constitute a limitation on the electronic device 400. The electronic device 400 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0066] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the method of any of the above embodiments are implemented. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0067] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A Tai Chi training interactive method based on knowledge graph and robot, characterized in that: include: The robot collects image data of the target learner during Tai Chi training; Inputting the image data into a preset image-text mapping engine, extracting the image features of the target learner when performing Tai Chi training through the image data, determining the action type corresponding to the image feature, and generating question text data corresponding to the action type; Traversing the pre-created knowledge graph based on the question text data, and generating a first hypergraph based on the first triplet obtained by the traversal; Traversing the pre-created knowledge graph based on the image features, and generating a second hypergraph based on second triples obtained through the traversal; Combining the first hypergraph and the second hypergraph to generate a first answer entity corresponding to the question text data, calculating the similarity between the first answer entity and each answer entity in the pre-created knowledge graph, and outputting a second answer entity with the highest similarity; Send a control instruction to the robot to enable the robot to demonstrate Tai Chi movements related to the second answer entity.

2. The Tai Chi training interactive method based on knowledge graph and robot according to claim 1, characterized in that: After the robot collects the image data of the target learner during Tai Chi training, the method further includes: Based on the acquisition time of each image frame in the image data, generating an image sequence consisting of image frames arranged in time sequence; Performing denoising and enhancement preprocessing on each image frame in the image sequence, and outputting the preprocessed image sequence; The step of inputting the image data into a preset image-text mapping engine is performed based on the preprocessed image sequence.

3. The Tai Chi training interactive method based on knowledge graph and robot according to claim 1, characterized in that: The step of inputting the image data into a preset image-text mapping engine, extracting the image features of the target learner during Tai Chi training through the image data, determining the action type corresponding to the image feature, and generating question text data corresponding to the action type includes: Extracting image features and posture labels corresponding to the image features from the image data through the preset image-text mapping engine, determining the action category of the target learner when performing Tai Chi training based on the posture labels, constructing a question text based on the image features, the posture labels and the action categories, and generating corresponding question text data; The question text is associated with the action category and the posture label.

4. The Tai Chi training interactive method based on knowledge graph and robot according to claim 3, characterized in that: Extracting text feature representation of the question text data based on the trained language model, extracting the association relationship between each entity and each two entities in the text feature representation to obtain corresponding text triples, and determining the association relationship between the entities in the text triples and the image features; The traversing the pre-created knowledge graph based on the question text data and generating a first hypergraph based on the first triple obtained by the traversal includes: Traversing the pre-created knowledge graph based on the text triples, traversing to obtain a first triple in the pre-created knowledge graph that contains any entity in the text triples, and generating the first hypergraph based on the first triple obtained through traversal; The traversing the pre-created knowledge graph based on the image features and the corresponding action types, and generating a second hypergraph based on the second triples obtained by the traversal, comprises: Based on the association relationship between the entities in the text triples and the image features, the pre-created knowledge graph is traversed to obtain a second entity associated with the action type and posture label corresponding to the image features in the text triples in the pre-created knowledge graph, and a second triplet containing the second entity is obtained, and the second hypergraph is generated based on the second triplet and an example image associated with the second triplet.

5. A Tai Chi training interactive method based on knowledge graph and robot according to claim 1 or 4, characterized in that: The step of combining the first hypergraph and the second hypergraph to generate a first answer entity corresponding to the question text data includes: Taking the first hypergraph as a reference and combining it with the second hypergraph, generating an output result corresponding to the first hypergraph through a self-attention mechanism; Taking the second hypergraph as a reference and combining it with the first hypergraph, an output result corresponding to the second hypergraph is generated through a self-attention mechanism; The output result corresponding to the first hypergraph and the output result corresponding to the second hypergraph are aggregated to obtain the first answer entity.

6. The Tai Chi training interactive method based on knowledge graph and robot according to claim 1, characterized in that: The sending of a control instruction to the robot so that the robot displays Tai Chi movements related to the second answer entity includes: Acquire an example image associated with the second answer entity and a posture label and an action type corresponding to the example image, acquire an action sequence of Tai Chi actions corresponding to the posture label and the action type, and send a control instruction to the robot to complete the action sequence; The robot is controlled to complete the action sequence through the control instruction, so that the robot displays Tai Chi movements related to the second answer entity.

7. The Tai Chi training interactive method based on knowledge graph and robot according to claim 1, characterized in that: The sending of a control instruction to the robot so that the robot displays Tai Chi movements related to the second answer entity includes: Acquire an example image of a Tai Chi movement associated with the second answer entity, and send a control instruction to the robot to display the example image; The robot is controlled by the control instruction to display Tai Chi movements corresponding to the example image on a display device of the robot.

8. A Tai Chi training interactive device based on knowledge graph and robot, characterized in that: include: An image acquisition module, used to collect image data of a target learner performing Tai Chi training through a robot; A question text generation module, used for inputting the image data into a preset image-text mapping engine, extracting the image features of the target learner when performing Tai Chi training through the image data, determining the action type corresponding to the image feature, and generating question text data corresponding to the action type; A knowledge graph module, configured to traverse a pre-created knowledge graph based on the question text data, and generate a first hypergraph based on a first triple obtained through the traversal; and to traverse the pre-created knowledge graph based on the image features, and generate a second hypergraph based on a second triple obtained through the traversal; The knowledge graph module is further used to combine the first hypergraph and the second hypergraph to generate a first answer entity corresponding to the question text data, calculate the similarity between the first answer entity and each answer entity in the pre-created knowledge graph, and output the second answer entity with the highest similarity; The interactive module is used to send a control instruction to the robot so that the robot can display Tai Chi movements related to the second answer entity.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images