Taiji training interaction method based on knowledge graph and robot

Abstract

The embodiment of the application discloses a taichi training interaction method based on a knowledge graph and a robot, relates to the technical field of knowledge graphs, and comprises the following steps: collecting image data of a target learner during taichi training by using a robot, extracting image features and action types of the target learner during taichi training from the image data, and generating corresponding question text data; traversing the pre-created knowledge graph based on the question text data, obtaining a first hypergraph and a second hypergraph, generating a first answer entity, outputting a second answer entity with the highest similarity to the first answer entity in the pre-created knowledge graph, and displaying taichi actions related to the second answer entity by using the robot. The application can process complex entities and corresponding taichi movement images, solve the problems of a large amount of taichi movement content, strong classification cross and difficult reasoning, improve the efficiency of searching for entity content, and improve the comprehensiveness and accuracy of intelligent teaching.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge graph, and particularly relates to a Tai Chi training interaction method based on a knowledge graph and a robot. BACKGROUND

[0002] As intangible cultural heritage, Tai Chi is a kind of exercise mode that is widely accepted and loved by people. There are more than one hundred known schools, and the movements of different schools of Tai Chi are different, so it will bring some learning difficulties to beginners.

[0003] Therefore, in the learning process of Tai Chi, learners need to be guided by teachers or coaches, but due to the large number of learners, it is usually difficult for teachers or coaches to provide one-on-one targeted guidance to each learner, and they can only lead students to practice multiple times through demonstration. For teachers, multiple demonstrations and corrections of learners' mistakes will greatly increase the workload of teachers; for students, it is possible to correct the mistakes in the movement only by multiple demonstrations and corrections of teachers, and since there are many movements of Tai Chi, teachers cannot guide each student on the details of each movement one by one. In most cases, students need to learn by themselves according to the teaching content combined with teaching video resources, etc. In this way, students are difficult to realize whether the movements are not standard, and it is also difficult to confirm which movements do not meet the standard, which easily leads to low overall learning efficiency, and students are difficult to actively find problems in learning, so as to improve the learning effect.

[0004] Therefore, there is currently a lack of a scheme that can provide an auxiliary means for students in the learning process of Tai Chi, and it is difficult to actively find problems in the practice process of students. SUMMARY

[0005] The embodiment of the present application provides a Tai Chi training interaction method based on a knowledge graph and a robot, to solve the defects in the related art, and the technical solution is as follows:

[0006] In a first aspect, the embodiment of the present application provides a Tai Chi training interaction method based on a knowledge graph and a robot, comprising:

[0007] acquiring image data of a target learner performing Tai Chi training by a robot;

[0008] inputting the image data into a preset image-text mapping engine, extracting image features of the target learner performing Tai Chi training through the image data, determining the action type corresponding to the image features, and generating question sentence text data corresponding to the action type;

[0009] traversing in a pre-created knowledge graph based on the question text data, generating a first hypergraph based on a first triple obtained through the traversal; traversing in the pre-created knowledge graph based on the image feature, generating a second hypergraph based on a second triple obtained through the traversal;

[0010] combining the first hypergraph and the second hypergraph to generate a first answer entity corresponding to the question text data, calculating the similarity between the first answer entity and each answer entity in the pre-created knowledge graph, and outputting a second answer entity with the highest similarity;

[0011] sending a control instruction to the robot to make the robot demonstrate a Tai Chi action related to the second answer entity.

[0012] In an optional implementation of the first aspect, after the robot collects the image data of the target learner performing Tai Chi training, the method further includes:

[0013] generating an image sequence composed of image frames arranged in chronological order based on the time when each image frame in the image data is collected;

[0014] performing preprocessing, such as denoising and enhancement, on each image frame in the image sequence, and outputting a preprocessed image sequence;

[0015] performing the step of inputting the image data into the preset image-text mapping engine based on the preprocessed image sequence.

[0016] In an optional implementation of the first aspect, the step of inputting the image data into the preset image-text mapping engine includes extracting an image feature of the target learner performing Tai Chi training from the image data, determining an action type corresponding to the image feature, and generating question text data corresponding to the action type, including:

[0017] extracting an image feature and a pose label corresponding to the image feature from the image data through the preset image-text mapping engine, determining an action category of the target learner performing Tai Chi training based on the pose label, and constructing question text based on the image feature, the pose label, and the action category to generate corresponding question text data;

[0018] The question text is associated with the action category and the pose label.

[0019] In an optional implementation of the first aspect, the text feature representation of the question text data is extracted based on a trained language model, and each entity and the association relationship between each two entities in the text feature representation are extracted to obtain corresponding text triples, and the association relationship between the entities in the text triples and the image feature is determined.

[0020] The first supergraph is generated based on the first triple obtained by traversing the pre-created knowledge graph based on the question text data.

[0021] The first supergraph is generated based on the first triple obtained by traversing the pre-created knowledge graph based on the text triple.

[0022] The second supergraph is generated based on the second triple obtained by traversing the pre-created knowledge graph based on the image features and corresponding action types.

[0023] The second supergraph is generated based on the second triple obtained by traversing the pre-created knowledge graph based on the association between the entities in the text triple and the image features, and obtaining the second triple containing the second entity associated with the action type and the posture label corresponding to the image features in the text triple.

[0024] In an optional implementation of the first aspect, the first answer entity corresponding to the question text data is generated by combining the first supergraph and the second supergraph, including:

[0025] The output result corresponding to the first supergraph is generated by combining the second supergraph based on the first supergraph through a self-attention mechanism.

[0026] The output result corresponding to the second supergraph is generated by combining the first supergraph based on the second supergraph through a self-attention mechanism.

[0027] The first answer entity is obtained by aggregating the output result corresponding to the first supergraph and the output result corresponding to the second supergraph.

[0028] In an optional implementation of the first aspect, the control instruction is sent to the robot to make the robot demonstrate the Tai Chi action related to the second answer entity, including:

[0029] The example image associated with the second answer entity, the posture label and the action type corresponding to the example image, and the action sequence of the Tai Chi action corresponding to the posture label and the action type are obtained, and the control instruction for completing the action sequence is sent to the robot.

[0030] The robot is controlled to complete the action sequence through the control instruction, so that the robot demonstrates the Tai Chi action related to the second answer entity.

[0031] In an optional implementation of the first aspect, the sending of the control instruction to the robot to cause the robot to perform the Tai Chi action related to the second answer entity includes:

[0032] obtaining an example image of the Tai Chi action associated with the second answer entity, and sending a control instruction to the robot to cause the robot to perform the example image;

[0033] controlling the robot to perform the Tai Chi action corresponding to the example image on the display device of the robot through the control instruction.

[0034] In a second aspect, the embodiments of the present application further provide a Tai Chi training interactive device based on a knowledge graph and a robot, which includes:

[0035] an image acquisition module configured to acquire image data of a target learner performing Tai Chi training through a robot;

[0036] a question text generation module configured to input the image data into a preset image-text mapping engine, extract image features of the target learner performing Tai Chi training through the image data, determine an action type corresponding to the image features, and generate question text data corresponding to the action type;

[0037] a knowledge graph module configured to traverse a pre-created knowledge graph based on the question text data, generate a first hypergraph based on first triplets obtained through the traversal, traverse the pre-created knowledge graph based on the image features, generate a second hypergraph based on second triplets obtained through the traversal;

[0038] the knowledge graph module is further configured to generate a first answer entity corresponding to the question text data in combination with the first hypergraph and the second hypergraph, calculate a similarity between the first answer entity and each answer entity in the pre-created knowledge graph, and output a second answer entity with the highest similarity;

[0039] an interaction module configured to send a control instruction to the robot to cause the robot to perform a Tai Chi action related to the second answer entity.

[0040] In a third aspect, the embodiments of the present application further provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method provided in the first aspect or any of the implementation manners of the first aspect.

[0041] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the method provided in the first aspect of the embodiments of the present application or any of the implementation manners of the first aspect.

[0042] The technical solutions provided by some embodiments of the present application have at least the following beneficial effects:

[0043] By using the robot to collect image information of a Tai Chi practitioner during practice, and performing feature extraction based on the image information, the image features and corresponding action types can be effectively determined, and relevant question text data can be generated, so that the learning purpose and intention of the learner can be accurately grasped through the image information, and the conversion process of automatically generating question text is realized, which can effectively solve the problems of fixed sports learning scene, high device requirement, and inconvenience of inquiry during exercise.

[0044] In addition, based on the question text data and corresponding image data of the Tai Chi learner, the pre-constructed knowledge graph is traversed, and a first hypergraph related to the question text and a second hypergraph related to the image features are constructed, and then the internal relationship between the knowledge graph and the specific question can be deeply mined according to the first hypergraph and the second hypergraph, so that complex entities and Tai Chi exercise images corresponding to the entities can be processed, thereby solving the problems of multiple Tai Chi exercise contents, strong classification intersection, and difficult reasoning, and improving the efficiency of finding related entity content; the comprehensiveness and accuracy of intelligent teaching can be improved through the scheme provided by the present application. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the present application or related art, the following will briefly introduce the drawings needed to be used in the embodiments or related art descriptions. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0046] Figure 1 is a flowchart of a Tai Chi training interaction method based on a knowledge graph and a robot provided by an embodiment of the present application;

[0047] Figure 2 is a structural diagram of a graph-text mapping engine of a Tai Chi training interaction method based on a knowledge graph and a robot provided by an embodiment of the present application;

[0048] Figure 3 is a structural diagram of a Tai Chi training interaction device based on a knowledge graph and a robot provided by an embodiment of the present application;

[0049] Figure 4 is a structural diagram of an electronic device provided by an embodiment of the present application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or modules is not limited to the steps or modules listed, but may optionally include steps or modules not listed, or may optionally include other steps or modules inherent to such process, method, product, or apparatus.

[0052] It should be noted that the terms "first" and "second" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in an order other than those described or illustrated herein.

[0053] The present application will now be described in detail with reference to specific embodiments.

[0054] Next, combine Figure 1 This application introduces a Tai Chi training interaction method based on knowledge graphs and robots, provided by an embodiment of this application. For details, please refer to... Figure 1 , Figure 1 This illustration shows a flowchart of a Tai Chi training interaction method based on knowledge graphs and robots, provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0055] S101 uses a robot to collect image data of the target learner during Tai Chi training.

[0056] S102, the image data is input into a preset image-text mapping engine, the image features of the target learner during Tai Chi training are extracted from the image data, the action type corresponding to the image features is determined, and question text data corresponding to the action type is generated.

[0057] S103, based on the question text data, traverse the pre-created knowledge graph, and generate a first hypergraph based on the first triplet obtained from the traversal; based on the image features, traverse the pre-created knowledge graph, and generate a second hypergraph based on the second triplet obtained from the traversal.

[0058] S104, combine the first hypergraph and the second hypergraph to generate a first answer entity corresponding to the question text data, calculate the similarity between the first answer entity and each answer entity in the pre-created knowledge graph, and output the second answer entity with the highest similarity.

[0059] S105, send control commands to the robot to make the robot display the Tai Chi movements related to the second answer entity.

[0060] In some embodiments, after S101, the method further includes:

[0061] It can generate an image sequence composed of image frames arranged in chronological order based on the acquisition time of each image frame in the image data.

[0062] Each image frame in the image sequence is preprocessed with denoising and enhancement, and the preprocessed image sequence is output.

[0063] Further, based on the preprocessed image sequence, the step in S102 of inputting image data into a preset image mapping engine is performed.

[0064] In some embodiments, the step of inputting the image data into a preset image-text mapping engine, extracting image features of the target learner during Tai Chi training from the image data, determining the action type corresponding to the image features, and generating question text data corresponding to the action type includes:

[0065] The preset image-text mapping engine extracts image features and corresponding posture labels from the image data. Based on the posture labels, the movement category of the target learner during Tai Chi training is determined. Based on the image features, posture labels, and movement categories, question text is constructed, and corresponding question text data is generated.

[0066] The question text is associated with the action category and the gesture label.

[0067] Specifically, refer to Figure 2 The image mapping engine shown first processes the input image data through convolution, activation, pooling, etc. to obtain feature maps. Then, it passes through convolutional layers, reconstruction, softmax, target region mapping, and finally an image detector. OD Image features are extracted and encoded to obtain image labels. , image label is passed to a nonlinear multilayer perceptron, which is modeled to obtain the region weight in the image , then the image label Gumbel-Softmax processed is obtained by using distribution to obtain the sampling object processed pose label is obtained as a mask of the image label in the decoding part

[0068] Further, the processed image label is sent to the text encoder and the category of image classification is calculated , i.e. the action category of the target learner when performing Tai Chi training, such as Tai Chi stance, hand method, step method and body method, etc. The obtained action category is used One-hot to represent and fed back to the text encoder connected with the image encoder to feed into the question decoder to obtain the question text Q , so as to generate corresponding question text data.

[0069] It can be understood that the pose label can include action labels of head, hand, leg, body and the like. In combination with the action type, the pose of the corresponding part in the action type can be obtained, and further the situation that the pose of the part corresponding to the pose label may not conform to the label when the learner practices the action type can be identified, for example, the image of the learner mixing the meta-stance is identified.

[0070] Exemplarily, further, according to the action type-mixing meta-stance, the pose label-leg, the question text-how to show the correct action of the leg in the image of the mixing meta-stance can be generated, and the embodiments of the application are not limited thereto.

[0071] In some embodiments, the preset image-text mapping engine can be trained based on a cross-entropy loss function:

[0072] ;

[0073] wherein, is the cross-entropy loss, is the loss function of the predicted answer start position.

[0074] In some embodiments, in S103, the text feature representation of the question text data can be extracted based on the trained language model, and each entity and the association relationship between each two entities in the text feature representation are extracted to obtain corresponding text triplets, and the association relationship between the entities in the text triplets and the image features is determined.

[0075] Specifically, the question text data can be input into the pre-trained language model BERT, and the features of the question text are obtained using the BERT model, and then the entities in the question are extracted and relations , and the pose label OD has been obtained according to the image detector , the entity and the relation and the pose label are connected, that is, the text triple is constructed based on the entity and the relation , and the association between the text triple and the pose label is established, since the pose label is also associated with the action type, thereby determining the deep relationship between each entity and the pose label and the action type.

[0076] The first hypergraph can be generated based on the first triple obtained by traversing the pre-created knowledge graph based on the question text data, and the first hypergraph specifically includes:

[0077] The first hypergraph is generated based on the first triple obtained by traversing the pre-created knowledge graph based on the text triple.

[0078] Each word unit in the text triple corresponding to the question text data can be extracted, and the same node in the pre-created knowledge graph is traversed based on each word unit as a starting node, and the triple corresponding to the same node is obtained, and the plurality of triples can be regarded as a set of relations between entities and entities, so that a plurality of triples covering all units in the question text data are obtained from the pre-created knowledge graph, and the first hypergraph is output.

[0079] The second hypergraph is generated based on the second triple obtained by traversing the pre-created knowledge graph based on the image features and the corresponding action type, and the second hypergraph specifically includes:

[0080] The second entity associated with the action type and the pose label corresponding to the image feature in the text triple is obtained by traversing the pre-created knowledge graph based on the association between the entity in the text triple and the image feature, and the second triple containing the second entity is obtained, and the second hypergraph is generated based on the second triple and the example image associated with the second triple.

[0081] In some embodiments, the pre-built knowledge graph in the above embodiments can be obtained based on information extraction on Tai Chi examination standards, action essentials, books and journals, teaching materials, competition videos, demonstration images, etc. The basic movements of Tai Chi are composed of four categories: stances, hand techniques, footwork and body techniques. According to the different upper and lower relationships, each category is divided into several subcategories. Due to the relationship of evolution, the movements can be divided into five major schools according to the school. Each school has different techniques. At the same time, combined with the characteristics of the yin and yang of Tai Chi, therefore, the knowledge graph can effectively represent the symmetrical relationship of its structure.

[0082] Specifically, as shown in Figure 2 , the same triplets with the same association relationship between entities and image features in the text triplets can be obtained by traversing the pre-created knowledge graph, and triplets and entity images corresponding to the same posture label and action type are extracted. For example Figure 2 , the posture label-head, elbow and leg, and the action type of the image feature corresponding to the text triplet is mixed Yuanzhu, the entity containing mixed Yuanzhu, head, elbow and leg can be extracted, and the image corresponding to these entities in the pre-created knowledge graph is obtained, as shown in Figure 2 , the example image and text description of the correct action of the leg, i.e. the above example image, so as to construct the second hypergraph.

[0083] Exemplarily, starting with the entity "double peak ear" and the relationship "hand action" as the relationship, the traversal of the relationship line segment string is performed, and through the construction of the hypergraph, there are any number of entities on each hyperedge, which better performs the reasoning between knowledge.

[0084] Further, the step of performing S104 specifically includes:

[0085] As shown in Figure 2 , the first hypergraph embedding graph corresponding to the first hypergraph and the second hypergraph embedding graph corresponding to the second hypergraph are obtained respectively, and the formula

[0086] is applied.

[0087] Wherein, is the hyperedge in , is the hyperedge embedding function, is the linear projection function, then the hyperedges in the same hypergraph are represented and are packed into matrices and .

[0088] Further, taking the first hypergraph embedding graph as the benchmark, combined with the second hypergraph embedding graph generating an output result corresponding to the first hypergraph through a self-attention mechanism;

[0089] embedding the second hypergraph into a graph as a reference, combining the graph embedded by the first hypergraph generating an output result corresponding to the second hypergraph through a self-attention mechanism;

[0090] aggregating the output result corresponding to the first hypergraph and the output result corresponding to the second hypergraph to obtain the first answer entity.

[0091] Specifically, it comprises:

[0092] defining the knowledge hyperedge and the question hyperedge as a query and a key-value pair respectively, setting the query as , the key as , and the value as , wherein is a projection matrix. After attention calculation:

[0093] ;

[0094] wherein, is the dimension of the query.

[0095] through layer normalization , residual connection , and finally through a fully connected layer aggregated into a single vector representation, which is an output result corresponding to the first hypergraph and an output result corresponding to the second hypergraph , and the output result after aggregation is ;

[0096] wherein represents a layer normalization operation, .

[0097] The specific steps for determining the first answer entity comprise:

[0098] ;

[0099] wherein is a weight matrix, is a bias term.

[0100] Further, a similarity between the first answer entity and each answer entity in the pre-created knowledge graph is calculated, and a second answer entity with the highest similarity is output, that is, each answer entity in the pre-created knowledge graph is compared with the above first answer entity, and a second answer entity with the highest similarity to the first answer entity in the pre-created knowledge graph is output, as shown in Figure 2 The question is how to show the correct action of the leg in the image of the mixed pile, and the extracted answer is: the knee should be curved, and straight in the curve. The embodiments of the present application are not limited to this.

[0101] In some embodiments, the second answer entity corresponding correct action can be demonstrated by a robot in S105, for example, a control instruction can be sent to the robot to make the robot demonstrate the Tai Chi action related to the second answer entity, including:

[0102] An example image associated with the second answer entity and a posture label and an action type corresponding to the example image are obtained, and an action sequence of a Tai Chi action corresponding to the posture label and the action type is obtained, and a control instruction for completing the action sequence is sent to the robot;

[0103] The robot is controlled to complete the action sequence by the control instruction, so that the robot demonstrates the Tai Chi action related to the second answer entity.

[0104] In some embodiments, an example image of a Tai Chi action associated with the second answer entity can also be obtained, and a control instruction for demonstrating the example image is sent to the robot;

[0105] The robot is controlled to demonstrate the Tai Chi action corresponding to the example image on the display device of the robot by the control instruction.

[0106] It can be understood that the embodiments of the present application do not limit the way of the robot demonstrating the correct action.

[0107] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the present application.

[0108] Next, please refer to Figure 3A structural schematic diagram of a knowledge graph and robot-based taichi training interaction device is provided for an exemplary embodiment of the present application. The device can be realized as all or part of a terminal through software, hardware, or a combination of both, and can also be integrated as an independent module on a server. The knowledge graph and robot-based taichi training interaction device 30 in the embodiment of the present application can be applied to a terminal or cloud, and the device 30 includes an image acquisition module 301, a question sentence text generation module 302, a knowledge graph module 303, and an interaction module 304, wherein:

[0109] The image acquisition module 301 is configured to acquire image data of a target learner performing taichi training through a robot;

[0110] The question sentence text generation module 302 is configured to input the image data into a preset image-text mapping engine, extract image features of the target learner performing taichi training through the image data, determine an action type corresponding to the image features, and generate question sentence text data corresponding to the action type;

[0111] The knowledge graph module 303 is configured to traverse a pre-created knowledge graph based on the question sentence text data, generate a first hypergraph based on a first triple obtained through the traversal, traverse the pre-created knowledge graph based on the image features, generate a second hypergraph based on a second triple obtained through the traversal, and output a second answer entity with the highest similarity.

[0112] The knowledge graph module 303 is further configured to generate a first answer entity corresponding to the question sentence text data in combination with the first hypergraph and the second hypergraph, calculate the similarity between the first answer entity and each answer entity in the pre-created knowledge graph, and output a second answer entity with the highest similarity.

[0113] The interaction module 304 is configured to send a control instruction to the robot to enable the robot to demonstrate taichi actions related to the second answer entity.

[0114] It should be noted that the device 30 provided in the above embodiment, when executing the knowledge graph and robot-based taichi training interaction method, is only exemplarily divided into the above functional modules, and in actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above-described functions. In addition, the device provided in the above embodiment and the knowledge graph and robot-based taichi training interaction method embodiment belong to the same concept, and the implementation process is detailed in the method embodiment. Therefore, it is not repeated here.

[0115] The embodiment of the present application further provides an electronic device, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method of any one of the above embodiments when executing the program.

[0116] Please refer to Figure 4 , a structural block diagram of an electronic device provided by the embodiment of the present application.

[0117] As Figure 4 shown, the electronic device 400 includes a processor 401 and a memory 402.

[0118] In the embodiment of the present application, the processor 401 is the control center of the computer system, and can be the processor of a physical machine or the processor of a virtual machine. The processor 401 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 can be implemented in at least one of the following hardware forms: a DSP (Digital Signal Processing), a FPGA (Field-Programmable Gate Array) and a PLA (Programmable Logic Array).

[0119] The processor 401 can also include a main processor and a coprocessor. The main processor is a processor for processing data in a wake-up state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state.

[0120] The memory 402 can include one or more computer-readable storage media, which can be non-transitory. The memory 402 can also include a high-speed random access memory and a non-volatile memory such as one or more disk storage devices, flash storage devices. In some embodiments of the present application, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one instruction for being executed by the processor 401 to implement the method in the embodiment of the present application.

[0121] In some embodiments, the electronic device 400 further includes a peripheral device interface 403 and at least one peripheral device 404. The processor 401, the memory 402 and the peripheral device interface 403 can be connected through a bus or a signal line. Each peripheral device 404 can be connected to the peripheral device interface 403 through a bus, a signal line or a circuit board. Specifically, the peripheral device interface 403 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 401 and the memory 402.

[0122] In some embodiments of the present application, the processor 401, the memory 402 and the peripheral device interface 403 are integrated on the same chip or circuit board; in some other embodiments of the present application, any one or two of the processor 401, the memory 402 and the peripheral device interface 403 can be implemented on a separate chip or circuit board. The embodiments of the present application do not make specific limitations in this regard.

[0123] The structural block diagram of the electronic device shown in the embodiments of the present application does not constitute a limitation on the electronic device 400, and the electronic device 400 can include more or fewer components than shown in the diagram, or combine certain components, or adopt a different arrangement of components.

[0124] The embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any of the preceding embodiments. The computer readable storage medium can include, but is not limited to, any type of disk including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0125] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on such an understanding, the above technical solutions, in essence or in terms of related technology, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method described in each embodiment or some part of the embodiment.

[0126] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the same; although the present application has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A taichi training interaction method based on a knowledge graph and a robot, characterized in that, The method comprises the following steps: collecting image data of a target learner performing Tai Chi training by a robot; inputting the image data into a preset image-text mapping engine, extracting image features of the target learner performing Tai Chi training from the image data, determining an action type corresponding to the image features, and generating question text data corresponding to the action type; traversing a pre-created knowledge graph based on the question text data, generating a first hypergraph based on first triplets obtained through the traversal; traversing the pre-created knowledge graph based on the image features, generating a second hypergraph based on second triplets obtained through the traversal; combining the first hypergraph and the second hypergraph to generate a first answer entity corresponding to the question text data, calculating the similarity between the first answer entity and each answer entity in the pre-created knowledge graph, and outputting a second answer entity with the highest similarity; sending a control instruction to the robot to make the robot demonstrate Tai Chi actions related to the second answer entity. 2.The Tai Chi training interaction method based on a knowledge graph and a robot according to claim 1, wherein, After the step of collecting image data of a target learner performing Tai Chi training by a robot, the method further comprises the following steps: generating an image sequence composed of image frames arranged in chronological order based on the time when each image frame in the image data is collected; performing preprocessing such as denoising and enhancement on each image frame in the image sequence, and outputting a preprocessed image sequence; performing the step of inputting the image data into a preset image-text mapping engine based on the preprocessed image sequence. 3.The Tai Chi training interaction method based on a knowledge graph and a robot according to claim 1, wherein, The step of inputting the image data into a preset image-text mapping engine, extracting image features of the target learner performing Tai Chi training from the image data, determining an action type corresponding to the image features, and generating question text data corresponding to the action type comprises the following steps: extracting image features and posture labels corresponding to the image features from the image data by the preset image-text mapping engine, determining an action category of the target learner performing Tai Chi training based on the posture labels, constructing a question text based on the image features, the posture labels, and the action category, and generating corresponding question text data; wherein the question text is associated with the action category and the posture labels.

4. The Tai Chi training interaction method based on a knowledge graph and a robot according to claim 3, characterized in that, extracting text feature representations of the question text data based on a trained language model, extracting each entity and the association relationship between each two entities in the text feature representations to obtain corresponding text triplets, and determining the association relationship between the entities in the text triplets and the image features; The step of traversing a pre-created knowledge graph based on the question text data, generating a first hypergraph based on first triplets obtained through the traversal, comprises the following steps: traversing the pre-created knowledge graph based on the text triplets, obtaining first triplets containing any entity in the text triplets in the pre-created knowledge graph through the traversal, and generating the first hypergraph based on the first triplets obtained through the traversal; The step of traversing the pre-created knowledge graph based on the image features and corresponding action types, generating a second hypergraph based on second triplets obtained through the traversal, comprises the following steps: The pre-created knowledge graph is traversed based on the association relationship between the entities in the text triple and the image features, second entities associated with the action type and the posture label corresponding to the image feature in the text triple in the pre-created knowledge graph are obtained through traversal, and second triples containing the second entities are obtained. The second hypergraph is generated based on the second triples and example images associated with the second triples. 5.The knowledge graph and robot-based Tai Chi training interaction method of claim 1 or 4, wherein, The first answer entity corresponding to the question text data is generated by combining the first hypergraph and the second hypergraph, including: Taking the first hypergraph as a reference, the output result corresponding to the first hypergraph is generated by combining the second hypergraph through a self-attention mechanism; Taking the second hypergraph as a reference, the output result corresponding to the second hypergraph is generated by combining the first hypergraph through a self-attention mechanism; The first answer entity is obtained by aggregating the output result corresponding to the first hypergraph and the output result corresponding to the second hypergraph. 6.The Tai Chi training interaction method based on a knowledge graph and a robot according to claim 1, wherein, The control instruction is sent to the robot to make the robot demonstrate the Tai Chi action related to the second answer entity, including: An example image associated with the second answer entity, a posture label corresponding to the example image, and an action type corresponding to the posture label and the action type are obtained, and an action sequence of a Tai Chi action corresponding to the posture label and the action type is obtained. The control instruction for completing the action sequence is sent to the robot; The robot is controlled to complete the action sequence through the control instruction, so that the robot demonstrates the Tai Chi action related to the second answer entity.

7. The knowledge graph and robot-based tai chi training interaction method according to claim 1, characterized in that, The control instruction is sent to the robot to make the robot demonstrate the Tai Chi action related to the second answer entity, including: An example image of a Tai Chi action associated with the second answer entity is obtained, and a control instruction for displaying the example image is sent to the robot; The robot is controlled to display the Tai Chi action corresponding to the example image on the display device of the robot through the control instruction.

8. A taichi training interaction device based on a knowledge graph and a robot, characterized in that, It includes: An image acquisition module for acquiring image data of a target learner performing Tai Chi training through a robot; A question text generation module for inputting the image data into a pre-set image-text mapping engine, extracting image features of the target learner performing Tai Chi training through the image data, determining an action type corresponding to the image features, and generating question text data corresponding to the action type; A knowledge graph module for traversing a pre-created knowledge graph based on the question text data, generating a first hypergraph based on a first triple obtained through traversal, and traversing the pre-created knowledge graph based on the image features, generating a second hypergraph based on a second triple obtained through traversal; The knowledge graph module is also used to generate a first answer entity corresponding to the question text data by combining the first hypergraph and the second hypergraph, calculate the similarity between the first answer entity and each answer entity in the pre-created knowledge graph, and output a second answer entity with the highest similarity; An interaction module for sending a control instruction to the robot to make the robot demonstrate a Tai Chi action related to the second answer entity.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor, when executing the program, implements the steps of the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method according to any one of claims 1 to 7.

Images