Method and related device for creating personality skills

Abstract

The invention discloses a method and a related device for creating an own skill. The method comprises the following steps: receiving multi-mode demonstration data which is uploaded by a user and aims at a target skill; creating a first data expansion request message; the first data expansion request message is sent to the cloud agent, and the first data expansion request message is used for indicating the cloud agent to determine first training data; receiving a first data expansion response message from the cloud agent, displaying the first training data on a training data management interface, and recording the total data volume of the current training data; if the total data volume of the current training data is greater than or equal to a preset data volume and a training request of the user for the target skill is detected, creating a first skill training instruction; and sending the first skill training instruction to the robot with the body. The simulation learning efficiency of the robot with the body can be improved.

Description

Technical Field

[0001] The present application relates to the field of robot control, and in particular to a method for creating embodied skills and related devices. Background Art

[0002] Training robot skills through imitation learning has become a popular method for robot training. The traditional approach is to let human operators remotely operate robots through different control interfaces to complete various operation tasks, and use these demonstration data to train robots to perform tasks autonomously. Although this method has achieved good results on some simple tasks, when it comes to expanding to more complex scenarios and more diverse tasks, it may not be possible to learn sufficiently complex motion patterns from current demonstration data. Summary of the invention

[0003] The embodiments of the present application provide a method for creating embodied skills and related devices to improve the efficiency of imitation learning of embodied robots.

[0004] In a first aspect, an embodiment of the present application provides a method for creating an embodied skill, which is applied to a terminal device of a skill creation system, wherein the skill creation system further includes an embodied robot and a cloud agent, and the cloud agent is deployed on at least one server, and the method includes: Receive multimodal demonstration data for a target skill uploaded by a user, wherein the target skill is a skill determined by the user to be newly added; and create a first data expansion request message; and send the first data expansion request message to the cloud-based agent, wherein the first data expansion request message includes the multimodal demonstration data, and the first data expansion request message is used to instruct the cloud-based agent to determine first training data, wherein the first training data is first action data of the simulated training required for the embodied robot to achieve the target skill; receiving a first data expansion response message from the cloud agent, the first data expansion response message including the first training data, displaying the first training data on a training data management interface, and recording a total amount of current training data; If the total data volume of the current training data is greater than or equal to the preset data volume, and the user's training request for the target skill is detected, a first skill training instruction is created; and the first skill training instruction is sent to the embodied robot, so that the embodied robot trains the target skill according to the first training data to achieve the creation of the target skill.

[0005] Wherein, the method further comprises: If the total amount of the current training data is less than the preset amount of data, a first prompt window is displayed on the training data management interface, wherein the first prompt window includes a first prompt message, a first confirmation control, and a first denial control, wherein the first prompt message is used to indicate to the user that the amount of the current training data is insufficient and to determine whether to continue to expand the training data; Detecting a trigger operation on the first determination control, creating a second data expansion request message; and sending the second data expansion request message to the cloud-based agent, wherein the second data expansion request message includes the total data volume of the current training data, and the second data expansion request message is used to instruct the cloud-based agent to determine second training data, wherein the second training data is second action data of the simulated training required for the embodied robot to achieve the target skill; receiving a second data expansion response message from the cloud agent, the second data expansion response message including the second training data, displaying the second training data on the training data management interface, and updating the total data volume of the current training data for the first time according to the second training data; If the total data volume of the current training data obtained after the first update is greater than or equal to the preset data volume, and a training request of the user for the target skill is detected, a second skill training instruction is created; and the second skill training instruction is sent to the embodied robot, where the second skill training instruction is used to instruct the embodied robot to train the target skill according to the first training data and the second training data; receiving a training feedback message from the embodied robot, wherein the training feedback message is used to indicate the degree of completion of the target skill by the embodied robot; If the degree of completion is greater than or equal to a preset degree of completion, a control for the target skill is added to the embodied robot skill demonstration interface.

[0006] Wherein, the method further comprises: If the total data volume of the current training data obtained after the first update is less than the preset data volume, displaying the first prompt window on the training data management interface; Detecting a trigger operation on the first determination control, creating a third data expansion request message; and sending the third data expansion request message to the cloud-based agent, wherein the third data expansion request message is used to instruct the cloud-based agent to determine third training data, wherein the third training data is third action data required for the embodied robot to simulate training to achieve the target skill; receiving a third data expansion response message from the cloud agent, the third data expansion response message including the third training data, displaying the third training data on the training data management interface, and updating the total data volume of the current training data obtained after the first update according to the third training data; If the total data volume of the current training data obtained after the second update is greater than or equal to the preset data volume, and a training request of the user for the target skill is detected, a third skill training instruction is created; and the third skill training instruction is sent to the embodied robot, where the third skill training instruction is used to instruct the embodied robot to train the target skill according to the first training data, the second training data and the third training data; receiving a training feedback message from the embodied robot, wherein the training feedback message is used to indicate the degree of completion of the target skill by the embodied robot; If the degree of completion is greater than or equal to a preset degree of completion, a control for the target skill is added to the embodied robot skill demonstration interface.

[0007] The process of determining the first training data includes the following steps: Performing motion capture and audio capture on the multimodal demonstration data to obtain fourth motion data and first audio data; Decomposing the fourth action data into a single action to obtain a plurality of first sub-action data; and performing keyword search on each of the plurality of first sub-action data to obtain a plurality of second sub-action data; and parsing each of the first sub-action data and each of the plurality of second sub-action data to obtain sub-action training data; Determine a waveform of the first audio data; and, according to the waveform, search for M fifth action data corresponding to the first audio data; and, determine a matching degree between the fourth action data and each of the M fifth action data to obtain M matching degrees; and, according to the M matching degrees, screen the M second action data to obtain N second action data, where N is less than or equal to M; Determine the N fifth motion data as fourth training data; The first training data is determined according to the sub-action training data and the fourth training data.

[0008] Among them, the first data expansion response message also includes second prompt information, and the second prompt information is used to indicate that the demonstration angle of the sub-action data is sufficient, or to indicate that the demonstration angle of the sub-action data is insufficient. Among them, if the second prompt information is used to indicate that the demonstration angle of the sub-action data is insufficient, then the second prompt information includes information of the sub-action data with insufficient demonstration angle.

[0009] The process of determining the second training data includes the following steps: Determine the difference between the total data volume of the current training data and the preset data volume to obtain a target difference; Determining the target demonstration data according to the target difference, wherein the target demonstration data includes demonstration data of at least one mode in the multimodal demonstration data; Extracting feature data of the target demonstration data; Searching for corresponding action data according to the feature data to obtain P sixth action data; Performing motion capture on the multimodal demonstration data to obtain seventh motion data; Determine a matching degree between the seventh action data and each of the P sixth action data to obtain P matching degrees; Screening the P sixth action data according to the P matching degrees to obtain Q sixth action data, where Q is less than or equal to P; The Q sixth action data are determined as the second training data.

[0010] After the total data amount of the current training data obtained after the first update is updated according to the third training data, the method further includes: receiving a request from the user to perform an editing operation on target training data, displaying a second prompt window on the training data management interface, wherein the second prompt window includes third prompt information, a second confirmation control, and a second denial control, wherein the third prompt information is used to prompt the user whether to continue to perform the editing operation, and the target training data includes the first training data, and / or the second training data, and / or the third training data; A triggering operation on the second determination control is detected, the editing operation is performed, and the training data management interface is updated.

[0011] In a second aspect, an embodiment of the present application provides an embodied skill creation device, which is applied to a terminal device of a skill creation system, wherein the skill creation system further includes an embodied robot and a cloud agent, and the cloud agent is deployed on at least one server, including: A first processing unit is configured to receive multimodal demonstration data for a target skill uploaded by a user, wherein the target skill is a skill that needs to be newly added and determined by the user; and to create a first data expansion request message; and to send the first data expansion request message to the cloud-based agent, wherein the first data expansion request message includes the multimodal demonstration data, and the first data expansion request message is used to instruct the cloud-based agent to determine first training data, wherein the first training data is first action data of a simulated training required for the embodied robot to achieve the target skill; a display unit, configured to receive a first data expansion response message from the cloud agent, wherein the first data expansion response message includes the first training data, display the first training data on a training data management interface, and record a total amount of current training data; The second processing unit is used to create a first skill training instruction if the total data volume of the current training data is greater than or equal to the preset data volume and the user's training request for the target skill is detected; and send the first skill training instruction to the embodied robot, so that the embodied robot trains the target skill according to the first training data to achieve the creation of the target skill.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory storing execution instructions, wherein the memory stores one or more programs; when the processor executes the execution instructions stored in the memory, the processor executes the method described in the first aspect.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing an energy data management program, including execution instructions. When a processor of an electronic device executes the execution instructions, the processor executes the method described in the first aspect.

[0014] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect of the embodiment of the present application. The computer program product may be a software installation package.

[0015] It can be seen that in an embodiment of the present application, first, multimodal demonstration data for a target skill uploaded by a user is received, and the target skill is a skill that needs to be added determined by the user; and a first data expansion request message is created; and the first data expansion request message is sent to the cloud-based agent, the first data expansion request message includes the multimodal demonstration data, and the first data expansion request message is used to instruct the cloud-based agent to determine first training data, and the first training data is the first action data of the simulated training required for the embodied robot to achieve the target skill; then a first data expansion response message is received from the cloud-based agent, the first data expansion response message includes the first training data, the first training data is displayed on the training data management interface, and the total data volume of the current training data is recorded; if the total data volume of the current training data is greater than or equal to the preset data volume, and the user's training request for the target skill is detected, a first skill training instruction is created; and the first skill training instruction is sent to the embodied robot, so that the embodied robot trains the target skill according to the first training data to achieve the creation of the target skill.

[0016] After receiving an instruction from a user to create a new skill, compared to the embodied robot training the skill through a large amount of demonstration data collected by manpower, this solution is that the terminal device sends the demonstration data uploaded by the user to the cloud-based intelligent body, and the cloud-based intelligent body expands the demonstration data based on the demonstration data uploaded by the user, that is, a small amount of demonstration data is expanded to generate large-scale and diversified demonstration data, which reduces the cost of collecting demonstration data while ensuring the similarity between the expanded demonstration data and the demonstration data uploaded by the user. After the cloud-based intelligent body expands the diversified demonstration data, it sends it to the terminal device, so that the terminal device sends a skill training instruction to the robot, and then the robot trains the skill according to the diversified demonstration data, thereby realizing the creation of new skills and improving the learning efficiency of new skills, thereby improving the efficiency of imitation learning of the embodied robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is a system architecture diagram of a skill creation system provided by an embodiment of the present application; Figure 2 It is a flowchart of a method for creating embodied skills provided in an embodiment of the present application; Figure 3 is a schematic diagram of a training data management interface provided in an embodiment of the present application; Figure 4 is a schematic diagram of another training data management interface provided in an embodiment of the present application; Figure 5 is a schematic diagram of a skill demonstration interface provided in an embodiment of the present application; Figure 6 is a schematic diagram of a prompt window provided in an embodiment of the present application; Figure 7 is a schematic diagram of a training data editing interface provided in an embodiment of the present application; Figure 8 This is a block diagram of the functional units of a device for creating embodied skills provided in an embodiment of the present application; Fig. 9 It is a block diagram of the functional units of another device for creating embodied skills provided in an embodiment of the present application; Fig.10 It is a structural schematic diagram of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0021] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0022] When embodied robots learn new skills, current demonstration data cannot be extended to more complex scenarios and more diverse tasks.

[0023] In view of the above problems, the embodiments of the present application provide a method for creating embodied skills and related devices. The embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0024] See also Figure 1 , Figure 1 1 is a system architecture diagram of a skill creation system provided by an embodiment of the present application. Figure 1 As shown, the skill creation system 100 includes a terminal device 101, an embodied robot 102 and a cloud agent 103. The terminal device 101 can communicate with the embodied robot 102 and the cloud agent 103 respectively through a network or other connection methods. The terminal device 101 refers to a device used by a user. The user can remotely control the embodied robot through the application interface installed on the terminal device 101 to learn skills. Specifically, the user can click on the new skill control and upload demonstration data in the display screen interface of the terminal device. After receiving the demonstration data, the terminal device 101 sends it to the cloud agent 103. The terminal device 101 may include a tablet computer, a handheld computer, an in-vehicle electronic device, a server, a laptop computer, a mobile Internet electronic device (MID) or a wearable electronic device (such as a smart watch, a Bluetooth headset), a smart phone (such as an Android phone, an iOS phone, a Windows Phone phone, etc.), etc. The above are only examples, not exhaustive, and include but are not limited to the above electronic devices.

[0025] Among them, the cloud-based intelligent agent 103 can deploy multiple servers, each of which is interconnected. After receiving the user's demonstration data, the cloud-based intelligent agent 103 expands the demonstration data to obtain diversified demonstration data, and transmits the diversified demonstration data to the terminal device 101, so that the terminal device 101 sends training instructions to the embodied robot 102 based on the diversified demonstration data, so that the embodied robot 102 trains new skills according to the diversified demonstration data. After the training is completed, controls for the new skills are added to the skill interface of the embodied robot.

[0026] Among them, the embodied robot 102 embeds artificial intelligence into tangible entities such as robots, enabling them to have the ability to perceive, learn and dynamically participate in the surrounding environment. It can have a human-like appearance, including two feet, two arms and a head. It is a robot that imitates human functions and intelligence and can perform various tasks in human work and living environments.

[0027] See also Figure 2 , Figure 2 1 is a flow chart of a method for creating an embodied skill provided in an embodiment of the present application, the method is applied to a terminal device of a skill creation system, the skill creation system further includes an embodied robot and a cloud agent, the cloud agent is deployed on at least one server. The method includes the following steps.

[0028] S210, receiving multimodal demonstration data for a target skill uploaded by a user, wherein the target skill is a skill determined by the user to be added; and, creating a first data expansion request message; and, sending the first data expansion request message to the cloud-based intelligent agent.

[0029] Among them, the first data expansion request message includes the multimodal demonstration data, and the first data expansion request message is used to instruct the cloud-based intelligent agent to determine the first training data, and the first training data is the first action data of the simulated training required for the embodied robot to achieve the target skill.

[0030] Among them, receiving a user's request to add a new embodied robot skill, displaying a training data management interface on the display screen of the terminal device, see Figure 3 , Figure 3 is a schematic diagram of a training data management interface provided in an embodiment of the present application, such as Figure 3 As shown in the figure, the training data management interface includes multiple new data controls and training controls, and counts the current training data volume. The current training data volume is 0, and the new data control is represented by +. The user only needs to click the control to enter the data upload window for uploading data. The training control is used by the user to train the target skill when the current data volume is sufficient; the ellipsis is used to indicate that new data controls can continue to be added.

[0031] Among them, multimodal demonstration data can be data in various forms such as text, image, audio and video. When the user uploads data in the form of text or image, the cloud-based intelligent agent can convert the text or image into data such as video, and then expand the data.

[0032] For example, the newly added skill selected by the user may be dancing to the target music, and the uploaded multimodal demonstration data may be video data showing the dance according to the target music from multiple angles. Furthermore, by clicking the newly added data control, the data upload window is entered. The data upload window includes an upload control and a real-time recording control. If the user selects the upload control, the dance demonstration data of the user can be obtained. If the user selects the real-time recording control, the user demonstration video is recorded through the camera configured on the embodied robot. Specifically, the user plays music and shows dance movements based on the music, and the embodied robot captures the user's demonstration data based on the camera to obtain the initial demonstration data, i.e., the multimodal demonstration data.

[0033] In a possible embodiment, the process of determining the first training data includes the following steps: performing motion capture and audio capture on the multimodal demonstration data to obtain fourth motion data and first audio data; performing single motion decomposition on the fourth motion data to obtain multiple first sub-motion data; and performing keyword search on each first sub-action data in the multiple first sub-action data to obtain multiple second sub-action data; and parsing each first sub-action data and each second sub-action data in the multiple second sub-action data to obtain sub-action training data; determining the waveform of the first audio data; and searching for M fifth motion data corresponding to the first audio data based on the waveform; and determining the matching degree between the fourth motion data and each fifth motion data in the M fifth motion data to obtain M matching degrees; and screening the M second motion data based on the M matching degrees to obtain N second motion data, wherein N is less than or equal to M; determining the N fifth motion data as fourth training data; and determining the first training data based on the sub-action training data and the fourth training data.

[0034] Among them, motion capture technology can accurately record the user's movement trajectory and posture changes, making the demonstration process more vivid and intuitive. At the same time, audio capture technology can obtain various sound signals generated by the user during the demonstration, such as motion noise, voice output, etc., thereby providing richer demonstration information.

[0035] Among them, the user's action data needs to be segmented. Through the action recognition algorithm, the specific actions performed by the user in the demonstration can be identified and divided into a series of basic actions. Then the video data corresponding to the relevant basic actions are queried in the Internet database. Exemplarily, the relevant video data can be retrieved by entering the keywords or description information of the basic actions, and then the queried video data and the corresponding multiple basic actions are parsed. Exemplarily, video processing techniques such as inter-frame difference, optical flow estimation, etc. can be used to identify and track the actions in the video, extract the basic action information therein, and obtain sub-action training data.

[0036] The waveform of the first audio data can be obtained by audio processing technology. The waveform is the representation of the audio signal in the time domain, which reflects the change of the amplitude of the audio signal over time. By analyzing the waveform, the basic characteristics of the audio signal, such as frequency, amplitude, etc., are obtained. The corresponding action data is searched in the database according to the waveform characteristics of the first audio data, and the action data represents the action sequence matching the audio data. The similarity between the multiple action data obtained and the action data obtained by motion capture is calculated, and when the similarity is greater than the similarity threshold, it is determined as training data. The training data and the sub-action training data constitute the first training data.

[0037] Exemplarily, after capturing dance movements and music in a dance video uploaded by a user, the user's dance movements are divided into multiple basic movements, and then the video data corresponding to the relevant basic movements are queried in an Internet database, and then the queried video data is parsed into training data; and the music played by the user is determined, and a dance video corresponding to the same music is selected from the database, and the acquired dance video is matched with the dance video demonstrated by the user for similarity, and the dance video with a high degree of match is determined as a demonstration video to obtain training data.

[0038] It can be seen that in the embodiment of the present application, by analyzing and processing the multimodal data, similar data is searched in the database and screened according to the degree of similarity. While expanding the training data, the similarity between the expanded demonstration data and the demonstration data uploaded by the user is guaranteed, thereby improving the quality of the expanded data.

[0039] In a possible embodiment, the first data expansion response message also includes second prompt information, and the second prompt information is used to indicate that the demonstration angle of the sub-action data is sufficient, or the second prompt information is used to indicate that the demonstration angle of the sub-action data is insufficient, wherein if the second prompt information is used to indicate that the demonstration angle of the sub-action data is insufficient, then the second prompt information includes information about the sub-action data with insufficient demonstration angle.

[0040] Among them, after obtaining the sub-action training data, determine whether each sub-action in the sub-action training data includes data from multiple angles. If there are multiple video angles in the demonstration data corresponding to the current basic action, such as data from four angles of front, back, left and right, it is determined that the demonstration data of the basic action is sufficient. Otherwise, it is considered that the demonstration angle of the basic action needs to be supplemented, and a prompt message is output to prompt the user to demonstrate the basic action at the corresponding angle.

[0041] It can be seen that in this embodiment, by determining whether the video angle of each basic action is sufficient, the integrity of the training data is ensured, thereby improving the accuracy of the embodied robot in performing the target skill.

[0042] S220, receiving a first data expansion response message from the cloud-based intelligent agent, displaying the first training data on a training data management interface, and recording the total data volume of the current training data.

[0043] The first data expansion response message includes first training data.

[0044] For example, see Figure 3 and Figure 4 , Figure 4 is a schematic diagram of another training data management interface provided in an embodiment of the present application, such as Figure 4 As shown, the first newly added data control is replaced by the demonstration data. If you click on the control, you can view the demonstration video. The second newly added data control is replaced by the first extended data. If you click on the control, you can view the first extended data. The first training data includes the demonstration data and the first extended data. The total amount of demonstration data and the first extended data is 1789. The current amount of training data has changed from 0 to 1789.

[0045] S230: If the total data volume of the current training data is greater than or equal to the preset data volume, and a training request from the user for the target skill is detected, a first skill training instruction is created; and the first skill training instruction is sent to the embodied robot.

[0046] Among them, the first skill training instruction is used to instruct the embodied robot to perform training according to the first training data.

[0047] Among them, a training data volume threshold can be set for each newly added skill. Specifically, it can be determined according to the difficulty of each skill. The higher the difficulty of the skill, the larger the corresponding training data volume threshold. If the training data volume threshold corresponding to the skill that the current user needs to add is 1500, it is determined that the total data volume of the current training data is greater than or equal to the preset data volume, and the user can be prompted whether to start training. If the user triggers the training control, a first skill training instruction is created to instruct the embodied robot to train according to the first training data; after receiving the training instruction, the embodied robot trains according to the first training data and feeds back the training progress to the user terminal in real time. Specifically, the training progress can be reflected by the accuracy of the currently executed action, reaction time, etc.

[0048] Among them, if the terminal device receives a message that the training of the embodied robot for the target skill has been completed, a skill demonstration interface is displayed on the display screen, and a control of the target skill is added to the skill demonstration interface.

[0049] For example, see Figure 5 , Figure 5 is a schematic diagram of a skill demonstration interface provided in an embodiment of the present application, such as Figure 5 As shown, the original skill demonstration controls include shaking hands, bending, pouring water, squatting, and taking out garbage, etc. Now, a new skill section is added, and a control for dancing with music is added. Specifically, the user can click the control for dancing with music, play the target music, and the embodied robot 102 demonstrates the action in the demonstration scene 402 and the real scene at the same time.

[0050] In a possible embodiment, if the total data volume of the current training data is less than the preset data volume, a first prompt window is displayed on the training data management interface, the first prompt window including a first prompt message, a first confirmation control and a first denial control, the first prompt message being used to indicate to the user that the data volume of the current training data is insufficient, and to determine whether to continue to expand the training data; a trigger operation on the first confirmation control is detected, and a second data expansion request message is created; and the second data expansion request message is sent to the cloud-based intelligent agent, the second data expansion request message including the total data volume of the current training data, the second data expansion request message being used to instruct the cloud-based intelligent agent to determine the second training data, the second training data being the second action data required for the simulated training of the embodied robot to achieve the target skill; a second data expansion response message is received from the cloud-based intelligent agent , the second data expansion response message includes the second training data, the second training data is displayed on the training data management interface, and the total data volume of the current training data is updated for the first time according to the second training data; if the total data volume of the current training data obtained after the first update is greater than or equal to the preset data volume, and the user's training request for the target skill is detected, a second skill training instruction is created; and, the second skill training instruction is sent to the embodied robot, and the second skill training instruction is used to instruct the embodied robot to train the target skill according to the first training data and the second training data; a training feedback message is received from the embodied robot, and the training feedback message is used to indicate the completion degree of the embodied robot for the target skill; if the completion degree is greater than or equal to the preset completion degree, a control for the target skill is added to the skill demonstration interface of the embodied robot.

[0051] For example, if the training data volume threshold corresponding to the skill that the current user needs to add is 2000, then it is determined that the total data volume of the current training data is less than the preset data volume. Figure 6 , Figure 6 is a schematic diagram of a prompt window provided in an embodiment of the present application, such as Figure 6 As shown, a prompt window pops up, which includes prompt information, a confirmation control and a deny control. The prompt information is that the current training data is insufficient and whether to continue expanding. Whether to continue expanding is determined by clicking the confirmation control and the deny control.

[0052] In a possible embodiment, when it is detected that the user clicks the confirmation control, a request to continue expanding is sent to the cloud-based intelligent agent.

[0053] In a possible embodiment, the process of determining the second training data includes the following steps: determining the difference between the total data volume of the current training data and the preset data volume to obtain a target difference; determining the target demonstration data according to the target difference, wherein the target demonstration data includes demonstration data of at least one modality in the multimodal demonstration data; extracting feature data of the target demonstration data; searching for corresponding action data according to the feature data to obtain P sixth action data; performing motion capture on the multimodal demonstration data to obtain seventh action data; determining the matching degree between the seventh action data and each sixth action data in the P sixth action data to obtain P matching degrees; screening the P sixth action data according to the P matching degrees to obtain Q sixth action data, wherein Q is less than or equal to P; and determining the Q sixth action data as the second training data.

[0054] Among them, after the cloud-based intelligent agent receives the expansion request, the cloud-based intelligent agent determines the difference between the total data volume of the current training data carried in the request message and the preset data volume. In the case of a large difference, the key features of the demonstration data of each modality are extracted; in the case of a small difference, the key features of the demonstration data under a certain modality are extracted, such as extracting key features from audio data or extracting key features from action videos.

[0055] Specifically, for action videos, the key features can be mid-level feature extraction. Through the random forest framework, multiple underlying features are integrated to establish a mid-level feature representation with strong discrimination and description capabilities, so as to effectively represent actions and accurately identify the combination and segmentation of multiple actions. It can also be spatiotemporal context features, that is, the temporal context features are used as important information to describe actions, reflecting the context information between local spatiotemporal blocks, which helps to more accurately represent and identify actions.

[0056] Specifically, for audio data, key features may be time domain features, frequency domain features, time-frequency domain features, and the like.

[0057] Among them, time domain features may include short-time energy, which reflects the energy change of the audio signal in a short time interval, such as the zero-crossing rate, the number of times the signal crosses from positive to negative or from negative to positive per unit time, which effectively reflects the frequency characteristics of the audio signal; for example, the peak value, the maximum amplitude of the audio signal in a certain time period, can be used to describe the strength and dynamic range of the audio signal.

[0058] Among them, frequency domain features may include spectral flatness, which describes the smoothness of the audio signal spectrum; for example, the spectral centroid, the central frequency of the spectral energy distribution, which can reflect the main frequency components of the audio signal; for example, the Mel-frequency cepstral coefficient, which converts the spectrum into a set of coefficients by simulating the auditory perception mechanism of the human ear, and is applied to speech recognition and music classification.

[0059] Among them, the time-frequency domain features may include short-time Fourier transform. By windowing the audio signal and calculating the Fourier transform of each window, the time-frequency representation of the signal can be obtained; for example, waveform analysis can effectively capture the local characteristics of the signal by analyzing the short-term waveform characteristics of the audio signal, such as short-time energy, short-time zero crossings, etc.; for example, wavelet transform can obtain time and frequency information at the same time by performing wavelet decomposition of audio signals at different scales.

[0060] Among them, according to the determined key features, the action data corresponding to the same key features are searched in the database, and the similarity between the found action data and the action data obtained by motion capture is calculated. When the similarity is greater than a similarity threshold, it is determined as the second training data.

[0061] For example, according to the training data volume threshold 2000 and the current training data 1789, a difference of 211 is obtained, which is small, and a search can be performed for the features of a certain modality. For example, the beats per minute of the music played by the user can be determined, and a dance video corresponding to the music with the same beats per minute can be selected, and then the second training data can be determined based on the similarity between the currently acquired dance video and the dance video demonstrated by the user.

[0062] After the cloud-based intelligent agent obtains the second training data through expansion, it transmits it to the terminal device, so that the terminal device receives the training data and counts the amount of the training data.

[0063] Exemplarily, the third newly added data control in the training data management interface is replaced by the second extended data, which is the second training data. If the control is clicked, the second extended data is viewed, and the current training data volume changes from the previous 1789 to the current 3612. Determine whether the total data volume of the current training data is greater than or equal to the preset data volume, and prompt the user whether to start training. If the user triggers the training control, create a second skill training instruction to instruct the embodied robot to train the skill according to the first training data and the second training data; after receiving the training instruction, the embodied robot trains according to the first training data and the second training data, and gives real-time feedback on the training stage to the user terminal. If the terminal device receives a message that the embodied robot has completed the training of the target skill, the skill demonstration interface is displayed on the display screen, and the control of the target skill is added to the skill demonstration interface.

[0064] It can be seen that in the embodiment of the present application, key features are extracted from multiple demonstration data, and training data with high similarity is searched based on the key features, which ensures the similarity between the expanded demonstration data and the demonstration data uploaded by the user, reduces the cost of demonstration data collection, and improves the quality of the expanded data.

[0065] In a possible embodiment, if the total data volume of the current training data obtained after the first update is less than the preset data volume, the first prompt window is displayed on the training data management interface; a trigger operation on the first determination control is detected, and a third data expansion request message is created; and the third data expansion request message is sent to the cloud-based agent, the third data expansion request message is used to instruct the cloud-based agent to determine third training data, and the third training data is the third action data of the simulated training required for the embodied robot to achieve the target skill; a third data expansion response message is received from the cloud-based agent, the third data expansion response message includes the third training data, the third training data is displayed on the training data management interface, and the third training data is expanded according to the third data expansion response message. The third training data is updated by obtaining the total data volume of the current training data after the first update; if the total data volume of the current training data after the second update is greater than or equal to the preset data volume, and the user's training request for the target skill is detected, creating a third skill training instruction; and sending the third skill training instruction to the embodied robot, the third skill training instruction is used to instruct the embodied robot to train the target skill according to the first training data, the second training data and the third training data; receiving a training feedback message from the embodied robot, the training feedback message is used to indicate the completion degree of the embodied robot for the target skill; if the completion degree is greater than or equal to the preset completion degree, adding a control for the target skill in the skill demonstration interface of the embodied robot.

[0066] Among them, after obtaining the second training data, if the threshold of the amount of training data corresponding to the skill that the current user needs to add is 4000, it is determined that the total data volume of the current training data is less than the preset data volume, and a prompt window pops up to determine whether to continue to expand the training data. If it is detected that the user clicks the confirmation control in the prompt window, a request to continue expanding is sent to the cloud-based intelligent agent. After receiving the expansion request, the cloud-based intelligent agent searches the database for action data with high similarity based on the captured user's action data as the action data to be processed, and edits and synthesizes the action data to be processed, the first training data and the second training data to obtain the third training data.

[0067] Specifically, editing synthesis can be synthesized by matching body postures, similarity of movement amplitudes, unification of picture composition, coordination of color matching, coordination of movement and audio, etc.

[0068] Exemplarily, a dance video having a high similarity to the dance video initially demonstrated by the user may be selected as the dance video to be processed, and then the first training data, the second training data and the dance video to be processed are edited and synthesized to obtain the third training data.

[0069] Furthermore, the video frames in the dance video to be processed that are identical to the dance movements demonstrated by the user may be retained, and the video frames of other dance movements may be replaced with the video frames of the corresponding dance movements in the dance video demonstrated by the user or the first training data and the second training data.

[0070] After the cloud-based intelligent agent obtains the third training data through expansion, it transmits it to the terminal device, so that the terminal device receives the training data and counts the number of the training data.

[0071] Exemplarily, the fourth newly added data control in the training data management interface is replaced by the third extended data, which is the third training data. If the control is clicked, the third extended data is viewed, and the current training data volume changes from the previous 3612 to the current 5781. Determine that the total data volume of the current training data is greater than or equal to the preset data volume, and prompt the user whether to start training. If the user triggers the training control, create a third skill training instruction to instruct the embodied robot to train the skill according to the first training data, the second training data, and the third training data; after the embodied robot receives the training instruction, it trains according to the first training data, the second training data, and the third training data, and gives real-time feedback on the training stage to the user terminal. If the terminal device receives a message that the training of the embodied robot for the target skill has been completed, the skill demonstration interface is displayed on the display screen, and the control of the target skill is added to the skill demonstration interface.

[0072] It can be seen that in the embodiment of the present application, expansion is performed through the integration of multiple training data, which enriches the training data while ensuring the similarity between the demonstration data and the demonstration data uploaded by the user, reduces the cost of demonstration data collection, and improves the quality of the expanded data.

[0073] It can be seen that in the embodiment of the present application, after receiving the instruction from the user to create a new skill, compared with the embodied robot training the skill through a large amount of demonstration data collected by manpower, this solution is that the terminal device sends the demonstration data uploaded by the user to the cloud-based intelligent body, and the cloud-based intelligent body expands the demonstration data according to the demonstration data uploaded by the user, that is, a small amount of demonstration data is expanded to generate large-scale and diversified demonstration data, which reduces the cost of collecting demonstration data and ensures the similarity between the expanded demonstration data and the demonstration data uploaded by the user. After the cloud-based intelligent body expands the diversified demonstration data, it sends it to the terminal device, so that the terminal device sends a skill training instruction to the robot, and then the robot trains the skill according to the diversified demonstration data, thereby realizing the creation of new skills and improving the learning efficiency of new skills, thereby improving the efficiency of imitation learning of the embodied robot.

[0074] In a possible embodiment, after the total data volume of the current training data obtained after the first update according to the third training data is updated, the method further includes: receiving a request from the user to perform an editing operation on the target training data, displaying a second prompt window on the training data management interface, the second prompt window including third prompt information, a second confirmation control and a second denial control, the third prompt information being used to prompt the user whether to continue to perform the editing operation, the target training data including the first training data, and / or the second training data, and / or the third training data; detecting a trigger operation on the second confirmation control, performing the editing operation, and updating the training data management interface.

[0075] After obtaining multiple training data, the user can view or delete the training data. For example, the user can click on any training data to view and enter the training data editing interface. Figure 7 , Figure 7 is a schematic diagram of a training data editing interface provided in an embodiment of the present application, such as Figure 7 As shown, the training data editing interface includes multiple training data, such as training data 1, training data 2, training data 3, etc. Each training data is followed by a view control and a delete control. If the user clicks the view control to view training data 2 and feels that training data 2 has a poor match or a bad appearance, the user can click the delete control in the interface to delete it.

[0076] It can be seen that in the embodiments of the present application, the user experience can be enhanced, the storage space of the terminal device can be optimized, and the quality of the training data can be improved.

[0077] In a possible embodiment, the obtained training data can be classified and weighted, such as the user's initial demonstration data, basic action training data, and similar training data. When the embodied robot is training, the skill training can be carried out according to the weight of the training data, and more attention can be paid to important data types, thereby improving the overall efficiency of the training.

[0078] For the above embodiments, please refer to Figure 8 , Figure 8 is a block diagram of the functional units of a device for creating embodied skills provided in an embodiment of the present application, such as Figure 8 As shown, the embodied skill creation device 80 includes: a first processing unit 81, which is used to receive multimodal demonstration data for a target skill uploaded by a user, wherein the target skill is a skill that needs to be added as determined by the user; and, create a first data expansion request message; and, send the first data expansion request message to the cloud-based agent, wherein the first data expansion request message includes the multimodal demonstration data, and the first data expansion request message is used to instruct the cloud-based agent to determine first training data, wherein the first training data is first action data of the simulated training required for the embodied robot to achieve the target skill; a display unit 82, which is used to receive a first data expansion response message from the cloud-based agent, wherein the first data expansion response message includes the first training data, display the first training data on a training data management interface, and record the total data volume of the current training data; a second processing unit 83, which is used to create a first skill training instruction if the total data volume of the current training data is greater than or equal to a preset data volume and a training request for the target skill from the user is detected; and, send the first skill training instruction to the embodied robot, so that the embodied robot trains the target skill according to the first training data to achieve the creation of the target skill.

[0079] In one possible embodiment, in terms of creating skills, the embodied skill creation device 80 is specifically used to: if the total data volume of the current training data is less than the preset data volume, display a first prompt window on the training data management interface, the first prompt window including a first prompt message, a first confirmation control and a first denial control, the first prompt message being used to indicate to the user that the data volume of the current training data is insufficient, and determine whether to continue to expand the training data; detect a trigger operation on the first confirmation control, create a second data expansion request message; and send the second data expansion request message to the cloud-based agent, the second data expansion request message including the total data volume of the current training data, the second data expansion request message being used to instruct the cloud-based agent to determine the second training data, the second training data being the second action data of the simulated training required for the embodied robot to achieve the target skill; receive a second data expansion request message from the cloud-based agent. A second data expansion response message of the intelligent agent, the second data expansion response message includes the second training data, the second training data is displayed on the training data management interface, and the total data volume of the current training data is updated for the first time according to the second training data; if the total data volume of the current training data obtained after the first update is greater than or equal to the preset data volume, and the user's training request for the target skill is detected, a second skill training instruction is created; and, the second skill training instruction is sent to the embodied robot, the second skill training instruction is used to instruct the embodied robot to train the target skill according to the first training data and the second training data; a training feedback message is received from the embodied robot, the training feedback message is used to indicate the completion degree of the embodied robot for the target skill; if the completion degree is greater than or equal to the preset completion degree, a control for the target skill is added to the skill demonstration interface of the embodied robot.

[0080] Among them, the process of determining the first training data includes the following steps: performing motion capture and audio capture on the multimodal demonstration data to obtain fourth motion data and first audio data; performing single motion decomposition on the fourth motion data to obtain multiple first sub-motion data; and, performing keyword search on each first sub-action data in the multiple first sub-action data to obtain multiple second sub-action data; and, parsing each first sub-action data and each second sub-action data in the multiple second sub-action data to obtain sub-action training data; determining the waveform of the first audio data; and, searching for M fifth motion data corresponding to the first audio data based on the waveform; and, determining the matching degree between the fourth motion data and each fifth motion data in the M fifth motion data to obtain M matching degrees; and, screening the M fifth motion data based on the M matching degrees to obtain N fifth motion data, wherein N is less than or equal to M; determining the N fifth motion data as the fourth training data; and determining the first training data based on the sub-action training data and the fourth training data.

[0081] Among them, the first data expansion response message also includes second prompt information, and the second prompt information is used to indicate that the demonstration angle of the sub-action data is sufficient, or the second prompt information is used to indicate that the demonstration angle of the sub-action data is insufficient. Among them, if the second prompt information is used to indicate that the demonstration angle of the sub-action data is insufficient, then the second prompt information includes information about the sub-action data with insufficient demonstration angle.

[0082] In one possible embodiment, in terms of creating skills, the embodied skill creation device 80 is specifically used to: if the total data volume of the current training data obtained after the first update is less than the preset data volume, then display the first prompt window on the training data management interface; detect the trigger operation of the first determination control, create a third data expansion request message; and send the third data expansion request message to the cloud-based agent, the third data expansion request message is used to instruct the cloud-based agent to determine the third training data, the third training data is the third action data of the simulated training required for the embodied robot to achieve the target skill; receive the third data expansion response message from the cloud-based agent, the third data expansion response message includes the third training data, and display the third data expansion response message on the training data management interface. third training data, and updating the total data volume of the current training data obtained after the first update according to the third training data; if the total data volume of the current training data obtained after the second update is greater than or equal to the preset data volume, and the user's training request for the target skill is detected, creating a third skill training instruction; and, sending the third skill training instruction to the embodied robot, the third skill training instruction is used to instruct the embodied robot to train the target skill according to the first training data, the second training data and the third training data; receiving a training feedback message from the embodied robot, the training feedback message is used to indicate the completion degree of the embodied robot for the target skill; if the completion degree is greater than or equal to the preset completion degree, adding a control for the target skill in the skill demonstration interface of the embodied robot.

[0083] Among them, the process of determining the second training data includes the following steps: determining the difference between the total data volume of the current training data and the preset data volume to obtain a target difference; determining the target demonstration data according to the target difference, wherein the target demonstration data includes demonstration data of at least one modality in the multimodal demonstration data; extracting feature data of the target demonstration data; searching for corresponding action data according to the feature data to obtain P sixth action data; performing motion capture on the multimodal demonstration data to obtain seventh action data; determining the matching degree between the seventh action data and each sixth action data in the P sixth action data to obtain P matching degrees; screening the P sixth action data according to the P matching degrees to obtain Q sixth action data, wherein Q is less than or equal to P; determining the Q sixth action data as the second training data.

[0084] In a possible embodiment, after the total data volume of the current training data is obtained after the first update according to the third training data, the embodied skill creation device 80 is specifically used to: receive a request from the user to edit the target training data, display a second prompt window on the training data management interface, the second prompt window includes a third prompt information, a second confirmation control and a second denial control, the third prompt information is used to prompt the user whether to continue to perform the editing operation, the target training data includes the first training data, and / or the second training data, and / or the third training data; detect a trigger operation on the second confirmation control, execute the editing operation, and update the training data management interface.

[0085] It can be understood that since the method embodiment and the device embodiment are different presentation forms of the same technical concept, the content of the method embodiment part in this application should be synchronously adapted to the device embodiment part and will not be repeated here.

[0086] In the case of integrated units, see Fig. 9 , Fig. 9 is a block diagram of the functional units of another device for creating embodied skills provided in an embodiment of the present application, such as Fig. 9 As shown, the embodied skill creation device 80 includes: a processing module 802 and a communication module 801. The processing module 802 is used to control and manage the actions of the embodied skill creation device 80, for example, to execute the steps of the first processing unit 81, the display unit 82 and the second processing unit 83, and / or to execute other processes of the technology described herein. The communication module 801 is used for the interaction between the embodied skill creation device 80 and other devices. Fig. 9 As shown, the device 80 for creating embodied skills may further include a storage module 803, and the storage module 803 is used to store program codes and data of the device 80 for creating embodied skills.

[0087] Among them, the processing module 802 can be a processor or a controller, for example, a central processing unit (CPU), a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The communication module 801 can be a transceiver, an RF circuit or a communication interface, etc. The storage module 803 can be a memory.

[0088] All relevant contents of each scenario involved in the above method embodiment can be referred to the functional description of the corresponding functional module, which will not be repeated here. Figure 2 How to create embodied skills as shown.

[0089] See also Fig.10 , Fig.10 is a schematic diagram of the structure of an electronic device proposed in an embodiment of the present application, such as Fig.10 As shown, the electronic device 1000 includes a processor 1010, a memory 1020, a communication interface 1030 and one or more programs 1021. The one or more programs 1021 are stored in the memory and are configured to be executed by the processor. When the program is executed, it includes part or all of the steps of any one of the control methods of the embodied robot recorded in the above method embodiments. The processor, memory and communication interface are interconnected and complete communication with each other.

[0090] The memory may be a volatile memory such as a dynamic random access memory (DRAM), or a non-volatile memory such as a mechanical hard disk. The memory is used to store a set of executable program codes, and the processor is used to call the executable program codes stored in the memory, and may execute part or all of the steps of any embodied skill creation method described in the above embodiment of the embodied skill creation method.

[0091] It can be seen that the electronic device 1000 described in the embodiment of the present application, it can be seen that, in the embodiment of the present application, first receives the multimodal demonstration data for the target skill uploaded by the user, the target skill is the skill that the user determines needs to be added; and, creates a first data expansion request message; and, sends the first data expansion request message to the cloud-based agent, the first data expansion request message includes the multimodal demonstration data, the first data expansion request message is used to instruct the cloud-based agent to determine the first training data, the first training data is the first action data of the simulated training required for the embodied robot to achieve the target skill; then receives the first data expansion response message from the cloud-based agent, the first data expansion response message includes the first training data, displays the first training data on the training data management interface, and records the total data volume of the current training data; if the total data volume of the current training data is greater than or equal to the preset data volume, and the user's training request for the target skill is detected, creates a first skill training instruction; and, sends the first skill training instruction to the embodied robot, so that the embodied robot trains the target skill according to the first training data to achieve the creation of the target skill.

[0092] After receiving an instruction from a user to create a new skill, compared to the embodied robot training the skill through a large amount of demonstration data collected by manpower, this solution is that the terminal device sends the demonstration data uploaded by the user to the cloud-based intelligent body, and the cloud-based intelligent body expands the demonstration data based on the demonstration data uploaded by the user, that is, a small amount of demonstration data is expanded to generate large-scale and diversified demonstration data, which reduces the cost of collecting demonstration data while ensuring the similarity between the expanded demonstration data and the demonstration data uploaded by the user. After the cloud-based intelligent body expands the diversified demonstration data, it sends it to the terminal device, so that the terminal device sends a skill training instruction to the robot, and then the robot trains the skill according to the diversified demonstration data, thereby realizing the creation of new skills and improving the learning efficiency of new skills, thereby improving the efficiency of imitation learning of the embodied robot.

[0093] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps of any method described in the above method embodiments, and the above computer includes an electronic device.

[0094] The embodiment of the present application also provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, the computer program is operable to cause a computer to execute some or all of the steps of any method described in the method embodiment. The computer program product may be a software installation package, and the computer includes an electronic device.

[0095] It should be noted that, for the sake of simplicity, the aforementioned method implementations are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the implementations described in the specification are all optional implementations, and the actions and modules involved are not necessarily required by the present application.

[0096] In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0097] In the several embodiments provided in this application, it should be understood that the disclosed device can be implemented in other ways. For example, the device implementation described above is only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical or other forms.

[0098] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0099] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software program module.

[0100] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. According to this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program codes.

[0101] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable memory, which can include: a flash drive, a read-only memory, a random access memory, a magnetic disk or an optical disk, etc.

[0102] The above is a detailed introduction to the implementation methods of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above implementation methods is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for creating embodied skills, characterized in that: A terminal device applied to a skill creation system, wherein the skill creation system further comprises an embodied robot and a cloud-based intelligent agent, wherein the cloud-based intelligent agent is deployed on at least one server, and the method comprises: Receive multimodal demonstration data for a target skill uploaded by a user, wherein the target skill is a skill determined by the user to be newly added; and create a first data expansion request message; and send the first data expansion request message to the cloud-based agent, wherein the first data expansion request message includes the multimodal demonstration data, and the first data expansion request message is used to instruct the cloud-based agent to determine first training data, wherein the first training data is first action data of the simulated training required for the embodied robot to achieve the target skill; receiving a first data expansion response message from the cloud agent, the first data expansion response message including the first training data, displaying the first training data on a training data management interface, and recording a total amount of current training data; If the total data volume of the current training data is greater than or equal to the preset data volume, and the user's training request for the target skill is detected, a first skill training instruction is created; and the first skill training instruction is sent to the embodied robot, so that the embodied robot trains the target skill according to the first training data to achieve the creation of the target skill.

2. The method according to claim 1, characterized in that The method further comprises: If the total amount of the current training data is less than the preset amount of data, a first prompt window is displayed on the training data management interface, wherein the first prompt window includes a first prompt message, a first confirmation control, and a first denial control, wherein the first prompt message is used to indicate to the user that the amount of the current training data is insufficient and to determine whether to continue to expand the training data; Detecting a trigger operation on the first determination control, creating a second data expansion request message; and sending the second data expansion request message to the cloud-based agent, wherein the second data expansion request message includes the total data volume of the current training data, and the second data expansion request message is used to instruct the cloud-based agent to determine second training data, wherein the second training data is second action data of the simulated training required for the embodied robot to achieve the target skill; receiving a second data expansion response message from the cloud agent, the second data expansion response message including the second training data, displaying the second training data on the training data management interface, and updating the total data volume of the current training data for the first time according to the second training data; If the total data volume of the current training data obtained after the first update is greater than or equal to the preset data volume, and a training request of the user for the target skill is detected, a second skill training instruction is created; and the second skill training instruction is sent to the embodied robot, where the second skill training instruction is used to instruct the embodied robot to train the target skill according to the first training data and the second training data; receiving a training feedback message from the embodied robot, wherein the training feedback message is used to indicate the degree of completion of the target skill by the embodied robot; If the degree of completion is greater than or equal to a preset degree of completion, a control for the target skill is added to the embodied robot skill demonstration interface.

3. The method according to claim 2, characterized in that The method further comprises: If the total data volume of the current training data obtained after the first update is less than the preset data volume, displaying the first prompt window on the training data management interface; Detecting a trigger operation on the first determination control, creating a third data expansion request message; and sending the third data expansion request message to the cloud-based agent, wherein the third data expansion request message is used to instruct the cloud-based agent to determine third training data, wherein the third training data is third action data required for the embodied robot to simulate training to achieve the target skill; receiving a third data expansion response message from the cloud agent, the third data expansion response message including the third training data, displaying the third training data on the training data management interface, and updating the total data volume of the current training data obtained after the first update according to the third training data; If the total data volume of the current training data obtained after the second update is greater than or equal to the preset data volume, and a training request of the user for the target skill is detected, a third skill training instruction is created; and the third skill training instruction is sent to the embodied robot, where the third skill training instruction is used to instruct the embodied robot to train the target skill according to the first training data, the second training data and the third training data; receiving a training feedback message from the embodied robot, wherein the training feedback message is used to indicate the degree of completion of the target skill by the embodied robot; If the degree of completion is greater than or equal to a preset degree of completion, a control for the target skill is added to the embodied robot skill demonstration interface.

4. The method according to claim 1, characterized in that The process of determining the first training data comprises the following steps: Performing motion capture and audio capture on the multimodal demonstration data to obtain fourth motion data and first audio data; Decomposing the fourth action data into a single action to obtain a plurality of first sub-action data; and performing keyword search on each of the plurality of first sub-action data to obtain a plurality of second sub-action data; and parsing each of the first sub-action data and each of the plurality of second sub-action data to obtain sub-action training data; Determine a waveform of the first audio data; and, according to the waveform, search for M fifth motion data corresponding to the first audio data; and, determine a matching degree between the fourth motion data and each of the M fifth motion data, to obtain M matching degrees; and, screening the M fifth action data according to the M matching degrees to obtain N fifth action data, where N is less than or equal to M; Determine the N fifth motion data as fourth training data; The first training data is determined according to the sub-action training data and the fourth training data.

5. The method according to claim 4, characterized in that The first data expansion response message also includes second prompt information, and the second prompt information is used to indicate that the demonstration angle of the sub-action data is sufficient, or the second prompt information is used to indicate that the demonstration angle of the sub-action data is insufficient, wherein if the second prompt information is used to indicate that the demonstration angle of the sub-action data is insufficient, then the second prompt information includes information about the sub-action data with insufficient demonstration angle.

6. The method according to claim 2, characterized in that The process of determining the second training data comprises the following steps: Determine the difference between the total data volume of the current training data and the preset data volume to obtain a target difference; Determining target demonstration data according to the target difference, wherein the target demonstration data includes demonstration data of at least one mode in the multimodal demonstration data; Extracting feature data of the target demonstration data; Searching for corresponding action data according to the feature data to obtain P sixth action data; Performing motion capture on the multimodal demonstration data to obtain seventh motion data; Determine a matching degree between the seventh action data and each of the P sixth action data to obtain P matching degrees; Screening the P sixth action data according to the P matching degrees to obtain Q sixth action data, where Q is less than or equal to P; The Q sixth action data are determined as the second training data.

7. The method according to claim 3, characterized in that After the total data amount of the current training data obtained after the first update is updated according to the third training data, the method further includes: receiving a request from the user to perform an editing operation on target training data, displaying a second prompt window on the training data management interface, wherein the second prompt window includes third prompt information, a second confirmation control, and a second denial control, wherein the third prompt information is used to prompt the user whether to continue to perform the editing operation, and the target training data includes the first training data, and / or the second training data, and / or the third training data; A triggering operation on the second determination control is detected, the editing operation is performed, and the training data management interface is updated.

8. A device for creating embodied skills, characterized in that: A terminal device applied to a skill creation system, wherein the skill creation system further includes an embodied robot and a cloud agent, wherein the cloud agent is deployed on at least one server, and includes: A first processing unit is configured to receive multimodal demonstration data for a target skill uploaded by a user, wherein the target skill is a skill that needs to be newly added and determined by the user; and to create a first data expansion request message; and to send the first data expansion request message to the cloud-based agent, wherein the first data expansion request message includes the multimodal demonstration data, and the first data expansion request message is used to instruct the cloud-based agent to determine first training data, wherein the first training data is first action data of a simulated training required for the embodied robot to achieve the target skill; a display unit, configured to receive a first data expansion response message from the cloud agent, wherein the first data expansion response message includes the first training data, display the first training data on a training data management interface, and record a total amount of current training data; The second processing unit is used to create a first skill training instruction if the total data volume of the current training data is greater than or equal to the preset data volume and the user's training request for the target skill is detected; and send the first skill training instruction to the embodied robot, so that the embodied robot trains the target skill according to the first training data to achieve the creation of the target skill.

9. An electronic device, characterized in that: The method comprises a processor and a memory storing execution instructions, wherein the memory stores one or more programs; when the processor executes the execution instructions stored in the memory, the processor executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: An energy data management program is stored, including execution instructions. When a processor of an electronic device executes the execution instructions, the processor executes the method according to any one of claims 1-7.

Images