Embodied Service Method and Related Device for Teaching Videos

Abstract

The present application proposes a method and related device for embodied services of teaching videos. By implementing the embodiments of the present application, an action link configuration data set from a mobile terminal is received, a humanoid robot is controlled to perform operation teaching on a first target object, the humanoid robot is controlled to obtain a first object state image and a second object state image, the completion degrees of the robot task and the user task are determined according to the relationships between the first object state image, the second object state image and a single-frame image, and the subsequent single action task is based on the corresponding action link configuration data set according to the action task constraint conditions for operation teaching until it is confirmed that the teaching task has been completed. Compared with the existing strategy of directly generating a demonstration plan by a robot based on a human demonstration video, this solution improves the flexibility and intelligence of the embodied services of teaching videos of humanoid robots, is conducive to improving the personalization degree for different teaching situations, and enhances the user experience.

Description

Technical Field

[0001] The present invention relates to the technical field of robots, and in particular to a method and related device for embodied service of teaching videos. Background Art

[0002] Robotics has made remarkable progress in recent years and has gradually evolved from robotic arms to humanoid robots, which are used in various fields. Due to the anthropomorphic appearance, flexible movement, embodied intelligence, and friendly appearance of humanoid robots, humanoid robots have significant advantages in daily teaching and other aspects.

[0003] Embodied intelligence is a concept that integrates perception, decision-making, and action, enabling machines to have physical and motor abilities like humans, allowing robots to move autonomously according to environmental changes and flexibly adapt to different tasks and working environments. Due to the characteristics of embodied intelligence of humanoid robots, humanoid robots can perform embodied services based on the characteristics of embodied intelligence, and through real-time interaction between the intelligent system and the environment, provide users with a more personalized and efficient service experience.

[0004] Child education is the cultivation and training of children in aspects such as moral, intellectual, and physical education. It is an important link in improving the quality of the population. Currently, teaching young children often requires a lot of time and effort, and in many cases, caregivers may not have enough time to teach young children. Humanoid robots can make up for this problem and perform teaching tasks. Currently, humanoid robots need to first learn from human demonstrations and corresponding instructions, extract skill information from human demonstrations and learn, and then perform demonstration learning, which is not intelligent enough and cannot flexibly adjust the learning content. Summary of the Invention

[0005] The embodiments of the present application provide a method and related device for embodied service of teaching videos, which can flexibly perform teaching demonstrations for different input contents, and are beneficial to improving the flexibility and intelligence of the embodied service of humanoid robots.

[0006] In a first aspect, the embodiments of the present application provide a method for embodied service of teaching videos, which is applied to the controller of a humanoid robot. The humanoid robot is communicatively connected to a mobile terminal. The method includes:

[0007] Obtain a plurality of action link configuration data sets from the mobile terminal. The plurality of action link configuration data sets have a corresponding relationship with the plurality of action tasks of the teaching task. A single action link configuration data set of a single action task includes a single frame image, a single action parameter set, and a single action task constraint condition. The single action task constraint condition is a trigger condition for the single action task to transition to the next single action task or to confirm that the teaching task has been completed;

[0008] Controlling a humanoid robot to perform operation teaching on a first target object according to a single set of action parameters, where the first target object is the object to be operated by the humanoid robot;

[0009] Controlling the humanoid robot to obtain a first object state image and a second object state image. The first object state image is the image of the first target object when a single action task is completed, and the second object state image is the image of the second target object when a single action task is completed. The second target object is the object to be operated by the user;

[0010] Determining the completion degree of the robot task according to the relationship between the first object state image and a single frame image, and determining the completion degree of the user task according to the relationship between the second object state image and a single frame image;

[0011] According to the completion degree of the robot task, the completion degree of the user task, and the action task constraint conditions, perform operation teaching on the subsequent single action task based on the data set configured for the subsequent corresponding action link until it is confirmed that the teaching task has been completed.

[0012] In a possible embodiment, the humanoid robot further includes a head structure, and a picture acquisition device is arranged on the head structure. Controlling the humanoid robot to obtain the first object state image and the second object state image based on a target angle includes:

[0013] Obtaining the shooting angle of a single frame image according to the single frame image;

[0014] Obtaining a target angle according to the shooting angle and the position parameters of the head structure. The target angle is the rotation angle of the head structure of the humanoid robot, and the position parameters of the head structure are the relative positions of the head structure with respect to the first target object or the second target object;

[0015] Controlling the position of the head structure of the humanoid robot to be adjusted according to the target angle, so that the picture angle obtained by the picture acquisition device is consistent with the picture angle of the single frame image;

[0016] Obtaining the first object state image and the second object state image.

[0017] In a possible embodiment, determining the completion degree of the robot task according to the relationship between the first object state image and a single frame image includes:

[0018] Obtaining a first image feature according to the first object state image, and obtaining a second image feature according to the single frame image;

[0019] Performing error analysis according to the first image feature and the second image feature to obtain the completion degree of the robot task.

[0020] In a possible embodiment, determining the completion degree of the user task according to the relationship between the second object state image and a single frame image includes:

[0021] Obtain a third image feature based on the state images of two objects, and obtain a fourth image feature based on a single-frame image;

[0022] Perform error analysis based on the third image feature and the fourth image feature to obtain the user task completion degree.

[0023] In a possible embodiment, the action task constraint conditions include that the robot task completion degree meets the first completion degree threshold and / or the user task completion degree meets the second completion degree threshold. Based on the robot task completion degree, the user task completion degree, and the action task constraint conditions, the operation teaching of the subsequent single action task based on the action link configuration data set corresponding to the subsequent action includes:

[0024] Judge whether the robot task completion degree meets the first completion degree threshold, and judge whether the user task completion degree meets the second completion degree threshold. The first completion degree threshold and the second completion degree threshold are preset thresholds;

[0025] If the robot task completion degree meets the first completion degree threshold and the user task completion degree meets the second completion degree threshold, then transfer to the subsequent single action task, and perform operation teaching on the subsequent single action task based on the action link configuration data set corresponding to the subsequent action;

[0026] If the robot task completion degree does not meet the first completion degree threshold or the user task completion degree does not meet the first completion degree threshold, then re-perform operation teaching on the current single action task.

[0027] In a possible embodiment, the action link configuration data set further includes the task frame image of the action task. The task frame image is other frame images except the single-frame image. The task frame image is the influence information of the single action task in the non-completed state. The method further includes:

[0028] If the robot task completion degree meets the first completion degree threshold, store the action link configuration data set of the single action task corresponding to the first object state image; when receiving a query instruction issued by the user, control the image acquisition device of the humanoid robot to acquire a third object state image, and the third object state image is an image of the current operation target of the user acquired by the humanoid robot;

[0029] Match the corresponding task frame image or single-frame image based on the third object state image;

[0030] Based on the single action task corresponding to the task frame image or the single-frame image, retrieve the action link configuration data set corresponding to the single action task;

[0031] Based on the retrieved action link configuration data set and the action imitation system, control the humanoid robot to perform operation teaching on the first target object. The action imitation system is used to convert a single action parameter set into a machine task.

[0032] In a possible embodiment, the humanoid robot further includes an audio playback device, the action link configuration data set further includes multiple audios, a single audio corresponds to a single action task, and the method further includes:

[0033] When the humanoid robot is in the state of performing operation teaching on a single action task, control the audio playback device to play the single audio corresponding to the single action task;

[0034] If the operation teaching is in the paused teaching state, pause playing the single audio. The paused teaching state indicates that the current humanoid robot is not in the state of operation teaching.

[0035] In a possible embodiment, obtaining multiple action link configuration data sets from the mobile terminal includes:

[0036] Detect whether the user inputs a target teaching video. If the target teaching video is detected, perform frame image splitting on the target teaching video to obtain multiple teaching frame images;

[0037] Perform action task segmentation on the multiple teaching frame images to obtain multiple action tasks. A single action task includes at least one teaching frame image;

[0038] Generate an action link configuration data set according to the multiple action tasks.

[0039] In a second aspect, an embodiment of the present application provides a teaching video embodied service device, which is applied to the controller of a humanoid robot. The humanoid robot is communicatively connected to a mobile terminal. The device includes:

[0040] A data acquisition module, configured to acquire multiple action link configuration data sets from the mobile terminal. The multiple action link configuration data sets have a corresponding relationship with multiple action tasks of the teaching task. The timing constraint relationship represents the timing relationship of multiple action tasks in the target teaching video. The single action link configuration data set of a single action task includes a single frame image, a single action parameter set, and a single action task constraint condition. The single action task constraint condition is a trigger condition for the single action task to transition to the next action task or confirm that the teaching task has been completed;

[0041] A first operation teaching module, configured to control the humanoid robot to perform operation teaching on the first target object according to the single action parameter set. The first target object is the operation object of the humanoid robot;

[0042] A status image acquisition module, configured to control a humanoid robot to acquire a first object status image and a second object status image. The first object status image is an image of a first target object when a single action task is completed, and the second object status image is an image of a second target object when a single action task is completed. The second target object is an operation object of a user;

[0043] A completion degree determination module, configured to determine the completion degree of a robot task according to the relationship between the first object status image and a single frame image, and determine the completion degree of a user task according to the relationship between the second object status image and a single frame image;

[0044] A second operation teaching module, configured to perform operation teaching on a subsequent single action task based on a data set configured for a subsequent corresponding action link according to the completion degree of the robot task, the completion degree of the user task, and action task constraint conditions until it is confirmed that the teaching task has been completed.

[0045] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which a teaching video embodiment service program is stored, including execution instructions. When a processor executes the execution instructions, the processor executes some or all of the steps described in the first aspect.

[0046] In a fourth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory storing execution instructions. The above one or more programs are stored in the above memory. When the processor executes the execution instructions stored in the memory, the processor executes some or all of the steps described in the first aspect.

[0047] In a fifth aspect, an embodiment of the present application provides a computer program product. The computer program product includes a non-transitory computer-readable storage medium storing a computer program. The computer program is operable to cause a computer to execute some or all of the steps described in the first aspect of the embodiments of the present application. The computer program product may be a software installation package.

[0048] By implementing the embodiments of the present application, multiple action link configuration data sets from a mobile terminal are obtained. The multiple action link configuration data sets have a corresponding relationship with multiple action tasks of a teaching task. A single action link configuration data set of a single action task includes a single frame image, a single action parameter set, and a single action task constraint condition. The single action task constraint condition is a trigger condition for the single action task to transition to the next action task or confirm that the teaching task has been completed. According to the single action parameter set, a humanoid robot is controlled to perform operation teaching on a first target object, where the first target object is the operation object of the humanoid robot. The humanoid robot is controlled to obtain a first object state image and a second object state image. The first object state image is the image of the first target object when a single action task is completed, and the second object state image is the image of a second target object when a single action task is completed. The second target object is the operation object of the user. The robot task completion degree is determined according to the relationship between the first object state image and the single frame image, and the user task completion degree is determined according to the second object state image and the single frame image. According to the robot task completion degree, the user task completion degree, and the action task constraint condition, the subsequent single action task is operated and taught based on the corresponding action link configuration data set until it is confirmed that the teaching task has been completed. Compared with the existing strategy of directly generating a demonstration plan by a robot based on a human demonstration video, this solution splits the target teaching video to be demonstrated and adds a transition judgment condition between each action task, demonstrates, pauses, or repeats the demonstration based on the actual situation of the user, improves the flexibility and intelligence of the embodied service of the humanoid robot teaching video, is conducive to improving the personalization degree for different teaching situations, and enhances the user experience. Brief Description of the Drawings

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the background art, the drawings required to be used in the embodiments of the present invention or the background art will be described below.

[0050] Figure 1 It is a schematic diagram of the architecture of a teaching video embodied service system provided by an embodiment of the present application;

[0051] Figure 2 It is a block diagram of the functional modules of a humanoid robot provided by an embodiment of the present application;

[0052] Figure 3 It is a schematic flowchart of a teaching video embodied service method provided by an embodiment of the present application;

[0053] Figure 4 It is a schematic diagram of the user interaction terminal interface of a teaching video embodied service provided by an embodiment of the present application;

[0054] Figure 5It is a schematic diagram of the scenario of an embodied service for teaching videos provided by an embodiment of the present application;

[0055] Figure 6 It is a schematic flowchart of a method for an embodied service of teaching videos provided by an embodiment of the present application;

[0056] Figure 7 It is a schematic structural diagram of a device for an embodied service of teaching videos proposed by an embodiment of the present application;

[0057] Figure 8 It is a schematic structural diagram of a humanoid robot proposed by an embodiment of the present application. Detailed implementation manners

[0058] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present application.

[0059] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or electronic device that includes a series of steps or units is not limited to the listed steps or units, but in an alternative example, it also includes steps or units not listed, or in an alternative example, it also includes other steps or units inherent to these processes, methods, products, or electronic devices.

[0060] Referring to "embodiment" herein means that a specific feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein may be combined with other embodiments.

[0061] The method for an embodied service of teaching videos and related devices provided by the embodiments of the present application can be applied to a teaching video embodied service system as shown in Figure 1 Please refer to Figure 1 , Figure 1It is a schematic architecture diagram of an embodied service system for teaching videos provided by an embodiment of the present application. The teaching video embodied service system 100 includes a terminal 110 and a controller 120 of a humanoid robot. The terminal 110 can transmit data to the controller 120 of the humanoid robot through a network. The terminal 110 refers to the device used by the user, corresponding to the user interaction device in this solution, such as a smart phone, a computer, etc.

[0062] In this solution, the terminal 110 has a display screen. The terminal 110 installs application programs that can be used by the user. The application programs are provided with user interfaces, enabling the user to conveniently input operations such as target teaching videos, and perform other human-computer interaction operations. The terminal 110 can also perform video analysis and processing on the acquired target teaching video to obtain an action link configuration data set for controlling the actions of the humanoid robot. The controller 120 of the humanoid robot is used to perform control operations on the humanoid robot according to the data transmitted by the terminal 110.

[0063] Please refer to Figure 2 , Figure 2 It is a block diagram of the functional modules of a humanoid robot provided by an embodiment of the present application. As Figure 2 shown, the functional modules of the humanoid robot may include: a control module, a camera module, a data acquisition module, a data transceiver module, etc., which are not limited herein.

[0064] Among them, the control module, as the "brain" of the humanoid robot, is responsible for overall planning and coordinating the work of other modules, and can correspond to the controller of the humanoid robot in this solution. It processes data from each module, makes decisions according to preset programs and algorithms, and commands the actions, behaviors, and task executions of the humanoid robot. For example, it determines the actions and postures of the humanoid robot, controls the shooting angle of the camera module, and processes the data acquired by the data acquisition module.

[0065] Among them, the camera module is equivalent to the "eyes" of the humanoid robot, and is used to acquire image and video information of the surrounding environment. These visual information can help the humanoid robot identify objects, people, etc., and provide a basis for robot target tracking and interaction with objects. For example, during the embodied service process based on teaching videos, it acquires the target object to be operated by the user or the target object to be operated by the humanoid robot itself, or acquires the user's actions. The camera module can be set on the head structure of the robot, or can be set on other parts. This solution only describes the situation of being set on the head structure, but it does not mean that it can only be set on the head and is not limited.

[0066] Among them, the data acquisition module is like the "sensory organ" of the humanoid robot. For example, it can be a sound collector and is responsible for collecting external voice data, etc.

[0067] Among them, the signal transceiver module is the interface for the humanoid robot to communicate with an external control source or other devices. It can receive instructions, data, and information from the terminal, enabling the humanoid robot to perform tasks according to the control requirements of the instructions, data, and information from the terminal.

[0068] Through the mutual cooperation among these modules, the humanoid robot can perceive the environment, receive instructions, make decisions, and execute corresponding actions to complete various embodied service tasks.

[0069] Based on this, the present application provides a method and related device for embodied service of teaching videos for humanoid robots. The present application will be described in detail below with reference to the accompanying drawings.

[0070] Please refer to Figure 3 , Figure 3 which is a schematic flowchart of a method for embodied service of teaching videos provided by an embodiment of the present application. This method is applied to the controller of a humanoid robot, and the humanoid robot is communicatively connected to a mobile terminal. As Figure 3 shown, the method includes the following steps:

[0071] S310, obtain multiple action link configuration data sets from the mobile terminal. The multiple action link configuration data sets have a corresponding relationship with multiple action tasks of the teaching task. The single action link configuration data set of a single action task includes a single frame image, a single action parameter set, and a single action task constraint condition.

[0072] Among them, the single frame image is the video information of a single action task in the completed state, the single action parameter set is the action execution data mapped to the humanoid robot, and the single action task constraint condition is the trigger condition for the single action task to transition to the next action task or confirm that the teaching task has been completed.

[0073] Specifically, obtain multiple action link configuration data sets with time sequence constraint relationships from the mobile terminal for the target teaching video. The multiple action link configuration data sets correspond one by one to multiple action tasks required to be completed by the teaching task of the target teaching video. The time sequence constraint relationship represents the time sequence relationship of multiple action tasks in the target teaching video. The single action link configuration data set of a single action task includes the single frame image, the single action parameter set, and the single action task constraint condition of this action task.

[0074] Before receiving multiple action link configuration data sets of the target teaching video from the mobile terminal, which have sequential constraint relationships, it is also necessary to wait for the user's confirmation information. The mobile terminal asks the user whether to start the teaching through an inquiry message. If a confirmation instruction from the user is received, the mobile terminal sends a confirmation message to the humanoid robot, indicating that the humanoid robot can start receiving multiple action link configuration data sets of the target teaching video from the mobile terminal.

[0075] Among them, the picture content of the target teaching video mentioned in this article can be a video containing the human body structure and the objects manipulated by humans, or a video only including the human body structure, or a video only including the manipulated objects. This video can have background sounds for explanation, can not have background sounds for explanation, can also have meaningless background sounds, etc., or can not have background sounds, which is not limited here. The sequential constraint relationship can be that there is a front-to-back time sequence relationship between multiple action tasks, and there is a one-to-one correspondence between a single action task and a single action link configuration data set. Therefore, there is a sequential constraint relationship between the action link configuration data sets. An action link configuration data set includes, but is not limited to, a single frame image corresponding to the action task, a single action parameter set, and a single action task constraint condition. The single frame image is the frame image included in this action task at the end of this action task, which can be understood as the frame image of the end pose of the action task; the single action parameter set can be the control parameters for controlling the humanoid robot to execute this action task, or a control instruction. The control instruction can be an instruction generated by the humanoid robot according to the control parameters, or the action parameter set sent by the mobile terminal is the instruction for controlling the humanoid robot; the single action task constraint condition can be the trigger condition for the single action task to transfer to the next action task, or the trigger condition for confirming that the teaching task has been completed.

[0076] Exemplarily, the single action task constraint condition can be to judge the completion degrees of the operations of the humanoid robot and the user after the execution of the single action task. If the completion degree of the operation of the humanoid robot or the user does not meet the completion requirements, it can be considered that the transfer to the next consecutive action task for teaching has not been triggered; if the completion degrees of the operations of the humanoid robot and the user meet the completion requirements, it can be considered that the transfer to the next consecutive action task for teaching has been triggered. The above-mentioned completion requirements can be a value for parameter judgment, or others, which is not limited here. The single action task constraint condition can also be to judge whether the last action task in the time sequence has been executed and meets the completion requirements. If the last action task has been executed and meets the completion requirements, it can be confirmed that the teaching task has been completed.

[0077] Specifically, the method for determining the action link configuration data set includes the following steps:

[0078] 11. The mobile terminal detects whether the user inputs a target teaching video. If the target teaching video is detected, the target teaching video is split into frame images to obtain a plurality of teaching frame images.

[0079] 12. The mobile terminal performs action task segmentation on the basis of the plurality of teaching frame images to obtain a plurality of action tasks. A single action task includes at least one teaching frame image.

[0080] 13. The mobile terminal generates an action parameter set based on the plurality of action tasks according to an action parameter mapping model. The action imitation system is used to convert the task actions of the video into action parameters.

[0081] Among them, the above-mentioned target teaching video can be a video obtained by inputting a network link, or a locally downloaded video. The user can input the target teaching video into the mobile terminal by entering a video link or calling a local video. The above-mentioned teaching frame image is the smallest unit in the target teaching video and is a static image.

[0082] Among them, the mobile terminal detects whether the user inputs a target teaching video. If the mobile terminal detects that the user inputs a target teaching video, the target teaching video is frame-segmented to obtain a plurality of teaching frame images, and then the plurality of teaching frame images are task-segmented and summarized to obtain a plurality of action tasks. An action task can include one or more teaching frame images, which is not limited here. Then, the mobile terminal performs action parameter mapping according to the plurality of teaching frame images to obtain a plurality of environment configuration data sets. Each environment configuration data set corresponds to an action task. An action data set includes a plurality of action parameters, and a single action parameter corresponds to a teaching frame image.

[0083] Among them, the action parameter set has time stamps for alignment, which is used to ensure that the humanoid robot can be time-aligned when teaching according to the action parameter set.

[0084] For example, please refer to Figure 4 , Figure 4 which is a schematic diagram of the user interaction terminal interface of a teaching video embodiment service provided by this application. As Figure 4As shown, the user interaction terminal is installed with an application program for providing embodied services for teaching videos. The functional interface of this application program is set with an enhanced playback function for teaching videos. After the user clicks, the page jumps and prompts the user to enter the video link or local video to be played. After the user inserts the video source in the box and clicks "OK", the page jumps to the demand prompt information page. For example, if it is a video link, the mobile terminal APP obtains the video from the website, recognizes the task props of the teaching video based on several frame images of the current video, and creates and outputs a pre-play user preparation demand prompt information according to the recognition result. For example, for an origami airplane teaching video, a pop-up box on the current page prompts "Two A4 papers need to be prepared", and prompts to specify the desktop in the current environmental space for placing the A4 papers and subsequent operation teaching as "Place on the desktop". When the user is ready according to the demand prompt information, they can click "OK" to inform the user interaction terminal that they are ready with the A4 papers and have placed them on the desktop.

[0085] In a possible embodiment, before or after receiving multiple action link configuration data sets of a target teaching video from a mobile terminal, the method further includes: receiving initial pose data from the mobile terminal, where the initial pose data is used to restrict the preparation pose of the humanoid robot before starting teaching, including kneeling, sitting, standing, or other complex poses, which are not specifically limited here.

[0086] Among them, the determination of the initial pose data includes the following steps: after the mobile terminal detects the target teaching video, the mobile terminal obtains the body data and environmental data of the humanoid robot. The body data represents the height of the humanoid robot and the lengths of its limbs, and the environmental data is the height at which the first target object and the second target object are placed. The initial pose data of the robot is obtained based on the body data and environmental data, and the initial pose data is sent to the humanoid robot after establishing a connection between the mobile terminal and the humanoid robot.

[0087] S320, controlling the humanoid robot to conduct operation teaching on the first target object according to a single action parameter set, where the first target object is the operation object of the humanoid robot.

[0088] Among them, before conducting operation teaching, the user needs to prepare the items to be operated, that is, the first target object and the second target object. The first target object is the object on which the robot conducts operation teaching, and the second target object is the object for the user to operate. The placement positions of the first target object and the second target object need to be prompted by the mobile terminal based on the target teaching video.

[0089] Specifically, after the mobile terminal splits the target teaching video into multiple teaching frame images, it performs image analysis on the multiple teaching frame images to generate prompt information. The prompt information is used to indicate the first target object and the second target object that the user needs to prepare, as well as the placement positions of the first target object and the second target object, and the preparatory actions. The prompt information can be audio information, text information, etc. The prompt method can be audio broadcast, pop-up window, etc., which is not limited here.

[0090] Among them, the first target object and the second target object are not limited to a single whole object and can be a combination including multiple sub-objects, which is not limited here.

[0091] Specifically, according to the single action parameter set corresponding to a single action task and the action imitation system, the humanoid robot is controlled to conduct operation teaching on the first target object. The first target object is the operation object of the humanoid robot, and the action imitation system is used to convert the single action parameter set into a machine task. Among them, the action imitation system is set in the humanoid robot. The action imitation system includes three main components: an action imitator, an action space compressor, and a high-order policy for downstream tasks. The action imitator learns to imitate the action sequences in a large motion dataset by using a recursive reinforcement learning algorithm. The action space compressor then uses the information bottleneck technique to compress the action representation learned by the action imitator into a low-dimensional vector space. Finally, the high-order policy is trained to perform downstream tasks in the low-dimensional vector space.

[0092] Specifically, controlling the humanoid robot to conduct operation teaching on the first target object according to the single action parameter set corresponding to a single action task and the action imitation system may include: inputting the single action parameter set with a time sequence constraint relationship into the action imitator. The action imitator uses the recursive reinforcement learning algorithm to learn the action parameter set to obtain an action policy. Then, the action compressor compresses the action policy into a low-dimensional space to obtain a low-dimensional vector, which is used as the input data of the high-order policy. The high-order policy clearly executes the task based on the low-dimensional vector to obtain the action of the operation teaching performed by the humanoid robot on the first target object.

[0093] S330, control the humanoid robot to obtain the first object state image and the second object state image. The first object state image is the image of the first target object when a single action task is completed, and the second object state image is the image of the second target object when a single action task is completed. The second target object is the operation object of the user.

[0094] It should be noted that after the teaching of each action task is completed, it is necessary to evaluate the operation results of the humanoid robot and the operation results of the user. In practical applications, the execution time of the action task should be relatively short, that is, the process of this evaluation state should be regarded as real-time.

[0095] Among them, the state of the first object is the state of the first target object when the humanoid robot finishes the current action task, which can be, for example, a pattern, a posture, a form, a shape, etc., and is not limited here. The state of the second object is the state of the second object when the user follows the teaching of the humanoid robot to operate the second target object and finishes the action task, which can be the state type corresponding to the state of the first target object, and can be, for example, a pattern, a posture, a form, a shape, etc., and is not limited here.

[0096] Among them, the humanoid robot can obtain the state image based on the camera module. The target angle is the image acquisition angle determined according to the last frame image of the action task, that is, a single frame image, and is used to be closer to the shooting angle in the video. The controller controls the humanoid robot to rotate the component where the camera module is located based on the target angle, which can be the head and related joints of the humanoid robot in practical applications, so that the first object state image and the second object state image obtained by the humanoid robot are close to the single frame image.

[0097] In a possible embodiment, the humanoid robot further includes a head structure, and a picture acquisition device is arranged on the head structure. Controlling the humanoid robot to obtain the first object state image and the second object state image based on the target angle includes:

[0098] 31. Obtain the shooting angle of the single frame image according to the single frame image;

[0099] 32. Obtain the target angle according to the shooting angle and the position parameters of the head structure. The target angle is the rotation angle of the head structure of the humanoid robot, and the position parameters of the head structure are the relative positions of the head structure with respect to the first target object or the second target object;

[0100] 33. Control the head structure of the humanoid robot to adjust its position according to the target angle, so that the picture angle obtained by the picture acquisition device is consistent with the picture angle of the single frame image;

[0101] 34. Obtain the first object state image and the second object state image.

[0102] Among them, obtaining the shooting angle according to the single frame image can be achieved by analyzing the exchangeable image file format data in the frame image, using computer vision technology or geometric analysis with a reference object, etc., or other methods, which are not limited here. Obtain the angle that the head structure of the humanoid robot needs to be adjusted according to the shooting angle of the single frame image and the relative position of the current robot head structure with respect to the first target object or the second target object, that is, the target angle. Adjust the position of the head structure according to the target angle, so that the past picture angle of the head structure can be maximally consistent with the picture angle of the single frame image, thereby achieving the best comparison effect.

[0103] It can be seen that in this embodiment, the target angle can be obtained by comparing with the angle of the frame image, and the shooting angle of the robot's head can be adjusted according to the target angle, so as to more accurately obtain the operation states of the user and the humanoid robot itself on the target object, improving the accuracy of the embodied service of the humanoid robot.

[0104] S340. Determine the robot task completion degree according to the relationship between the first object state image and a single frame image, and determine the user task completion degree according to the second object state image and a single frame image.

[0105] Among them, the robot task completion degree represents the completion degree of the humanoid robot for a single action task, and the user task completion degree represents the completion degree of the user for a single action task.

[0106] Among them, after each action task ends the operation, it is necessary to track the completion degree of each action task to ensure the operation accuracy of the humanoid robot for the action task and the teaching following degree of the user based on the humanoid robot. After each action task ends, the task completion degree is determined according to the obtained first object state image and second object state image.

[0107] Specifically, the robot task completion degree of the humanoid robot is analyzed according to the first object state image and a single frame image, and the user task completion degree is analyzed according to the second object state image and a single frame image. Among them, the analysis of the completion degree can be the analysis of the matching degree between the first object state image or the second object state image and a single frame image, and can be judged by calculating the similarity degree between the images. Exemplarily, it can be based on image processing software, digital image correlation method, feature point matching method, histogram comparison method, structural similarity index method, perceptual hashing algorithm, machine learning method, transform domain comparison method, etc. The specific method used can be selected according to the actual situation and is not limited herein.

[0108] In a possible embodiment, determining the robot task completion degree according to the relationship between the first object state image and a single frame image includes: obtaining a first image feature according to the first object state image, and obtaining a second image feature according to a single frame image; performing error analysis according to the first image feature and the second image feature to obtain the robot task completion degree.

[0109] Among them, the feature point matching method of the image can be selected for the analysis of the task completion degree.

[0110] Specifically, first, image preprocessing is performed on the first object state image. The preprocessing of the first object state image includes operations such as denoising, grayscale conversion, and binarization to improve the accuracy and efficiency of subsequent feature extraction. Then, feature extraction is carried out. Image processing algorithms (such as edge detection, corner detection, texture analysis, etc.) are used to extract key features from the preprocessed image. These features may include the shape, size, position, color distribution, etc. of the object. Then, the image is represented by features, and the extracted features are converted into a form suitable for computer processing, such as vectors, matrices, or tensors. These features will be used for subsequent error analysis and task completion degree evaluation. The processing method for a single-frame image can be the same as that of the first object state image to obtain an image represented by features. Then, feature matching is performed on the first object state image and the single-frame image: the features of the first object state image are matched with the features in the frame image sequence to find similarities or differences. This can be achieved by calculating the distance or similarity between the features. Then, error calculation is performed on the result of feature matching. Based on the matching result, the error between the actual task execution and the preset task is calculated, that is, the robot task completion degree, which may include position error, shape error, trajectory error, etc.

[0111] Among them, after obtaining the error analysis, the specific position of the error can be further analyzed to obtain the specific position where the first target state of the first target object and the single-frame image are different, and then the position where the humanoid robot's action has an error can be obtained. If the action task with unsatisfactory completion degree needs to be re-operated subsequently, additional attention can be paid to the position with errors, thereby increasing the accuracy.

[0112] It can be seen that in this embodiment, by performing feature extraction on the state image of the first target object after the end of an action task of the humanoid robot and performing error analysis with the features of the single-frame image, the completion degree of the humanoid robot in the action task is obtained, which is beneficial to taking measures in a timely manner when it is found that there are errors in the operation, improving the intelligence and accuracy of teaching, and thus improving the user experience.

[0113] In a possible embodiment, determining the user task completion degree according to the second object state image and the single-frame image includes: obtaining the third image feature according to the second object state image, and obtaining the fourth image feature according to the single-frame image; performing error analysis according to the third image feature and the fourth image feature to obtain the user task completion degree.

[0114] Among them, the feature point matching method of the image can be selected for the analysis of the task completion degree.

[0115] Specifically, first, perform image preprocessing on the second object state image. The preprocessing of the second object state image includes operations such as denoising, grayscale conversion, and binarization to improve the accuracy and efficiency of subsequent feature extraction. Then, perform feature extraction. Use image processing algorithms (such as edge detection, corner detection, texture analysis, etc.) to extract key features from the preprocessed image. These features may include the shape, size, position, color distribution, etc. of the object. Then, represent the image with features to obtain the third image feature, and convert the extracted features into a form suitable for computer processing, such as vectors, matrices, or tensors. These features will be used for subsequent error analysis and task completion degree evaluation. The processing method for a single-frame image can be the same as that of the second object state image to obtain the fourth image feature represented by features. Then, perform feature matching on the second object state image and the single-frame image: match the features of the second object state image with the features in the frame image sequence to find similarities or differences. This can be achieved by calculating the distance or similarity between the features. Then, calculate the error of the feature matching result. Based on the matching result, calculate the error between the actual task execution and the preset task, that is, the user task completion degree. Examples can include position error, shape error, trajectory error, etc.

[0116] Among them, after obtaining the error analysis, it is also possible to further analyze the specific location of the error, obtain the difference between the second target state of the second target object and the single-frame image, and then obtain the location where the user action has an error. If subsequent re-operation is required for action tasks with unsatisfactory completion degrees, additional attention can be paid to the locations with errors, thereby increasing the accuracy.

[0117] It can be seen that in this embodiment, by performing feature extraction on the state image of the second target object after a user's action task ends and performing error analysis with the features of a single-frame image, the completion degree of the user's action task is obtained, which is beneficial to taking measures in a timely manner when it is found that there are errors in the operation, improving the intelligence and accuracy of teaching, and thus improving the user experience.

[0118] S350, according to the robot task completion degree, the user task completion degree, and the action task constraint conditions, perform operation teaching on the subsequent single action task based on the data set configured for the corresponding action link until it is confirmed that the teaching task has been completed.

[0119] Among them, the action task constraint conditions can be used to determine whether the state after the execution of the current action task can be used for teaching the subsequent action task. Specifically, it can be analyzed by comparing the robot task completion degree and the user task completion degree with the action task constraint conditions respectively. The action task constraint conditions can be thresholds or other forms of conditions, which are not limited here and are used to judge whether the robot task completion degree and the user task completion degree meet the conditions.

[0120] Among them, if it is determined according to the action task constraint conditions that the operation teaching of a single subsequent action task can be performed, but the current action task is already the last action task, it indicates that the teaching task has been completed. The teaching task can represent all the action tasks transmitted from the user interaction terminal.

[0121] In a possible embodiment, the action task constraint conditions include that the robot task completion degree meets the first completion degree threshold and / or the user task completion degree meets the second completion degree threshold. According to the robot task completion degree, the user task completion degree, and the action task constraint conditions, the operation teaching of a single subsequent action task based on the action link configuration data set corresponding to the subsequent action includes: judging whether the robot task completion degree meets the first completion degree threshold, and judging whether the user task completion degree meets the second completion degree threshold. The first completion degree threshold and the second completion degree threshold are preset thresholds; if the robot task completion degree meets the first completion degree threshold and the user task completion degree meets the second completion degree threshold, then transfer to the single subsequent action task, and perform operation teaching on the single subsequent action task based on the action link configuration data set corresponding to the subsequent action; if the robot task completion degree does not meet the first completion degree threshold or the user task completion degree does not meet the first completion degree threshold, then re-perform operation teaching on the current single action task.

[0122] Among them, the action task constraint conditions can be that the robot task completion degree meets the first completion degree threshold, or the user task completion degree meets the second completion degree threshold, or the robot task completion degree meets the first completion degree threshold and the task completion degree meets the second completion degree threshold. The setting of the first completion degree threshold and the second completion degree threshold can be reasonable values obtained in the previous research and preset in the humanoid robot.

[0123] Among them, please refer to Figure 6 , Figure 6 is a schematic flowchart of a teaching video embodiment service method provided by an embodiment of the present application. As Figure 6As shown, when the action task constraint condition is that the robot task completion degree meets the first completion degree threshold and the task completion degree meets the second completion degree threshold, after the operation of a single action task ends and the robot task completion degree and the user task completion degree are obtained, it is judged whether the robot task completion degree meets the first completion degree threshold and whether the user task completion degree meets the second completion degree threshold. If it is judged that the robot task completion degree meets the first completion degree and the user task completion degree meets the second completion degree threshold, it means that the current single action task has completed the teaching and can be transferred to the subsequent action task for operation teaching or end the teaching task. If the robot task completion degree does not meet the first completion degree threshold or the user task completion degree does not meet the second completion degree threshold, it means that the current action task needs to be re-taught.

[0124] Among them, the process of re-teaching can be to obtain an inverse action parameter set according to the single action parameter set corresponding to the single action task. The inverse action parameter set is used to restore the first target object from the current first object state to the initial state of the current action task. The first object state is the object state corresponding to the first object state image; perform an inverse action operation according to the inverse action parameter set so that the first target object is in the third object state, and the third object state is the initial state of the action task. At the same time, the user follows the humanoid robot to perform an inverse action operation so that the second target object is restored to the fourth object state, and the fourth state is the initial state of the action task. Then repeat the teaching of the action task and repeat the above steps until the current single action task meets the constraint conditions and is transferred to the subsequent action task or the teaching task is completed.

[0125] It can be seen that in this embodiment, by judging whether the robot task completion degree and the user task completion degree meet the conditions for transferring to the subsequent action task, if the transfer conditions are not met, the operation teaching is repeated, fully considering the user's progress, which can make the teaching of the humanoid robot accurate and rigorous, improve the accuracy and serviceability of the humanoid robot's embodied service, and enhance the user experience.

[0126] In a possible embodiment, the action link configuration data set further includes the task frame images of the action task. The task frame images are other frame images except for the single frame image, and the task frame images are the influence information of a single action task in a non-completed state. The method further includes: if the robot task completion degree of the humanoid robot meets the first completion degree threshold, storing the action link configuration data set of the single action task corresponding to the first object state image; when receiving a query instruction issued by the user, controlling the screen acquisition device of the humanoid robot to acquire a third object state image, where the third object state image is an image of the current operation target of the user acquired by the humanoid robot; matching the corresponding task frame image or single frame image based on the third object state image; based on the single action task corresponding to the task frame image or single frame image, retrieving the action link configuration data set corresponding to the single action task; and controlling the humanoid robot to perform operation teaching on the first target object based on the retrieved action link configuration data set and the action imitation system.

[0127] Among them, after it is determined that the robot task completion degree of the humanoid robot meets the first completion degree threshold, it means that the operation of the humanoid robot on the action task is accurate, and then the single action task and the corresponding action link configuration data set can be stored. The storage location can be a storage location that can be retrieved at any time, and specific limitations are not made. When receiving a query instruction issued by the user, controlling the humanoid robot to acquire a third object state image of the user's second target object, where the third object state image is an image of the current state of the second target object and is used for matching the action task. Then, according to the obtained third object state image, match it with the task frame images and single frame images of the action link configuration data set of the historical action tasks stored.

[0128] Among them, the action link configuration data set may further include all action frame images corresponding to the action task except for the single frame image, that is, the task frame images. When storing the action task, the action link configuration data set of the action task is stored. When matching according to the obtained third object state image with the historical action tasks stored, it can be based on all task frame images of the action task for image matching. The specific image matching process can be a feature extraction matching method or others, and no limitation is made here. Perform repeated teaching tasks according to the matched action task and the action imitation system. The usage method of the action imitation system has been described above and will not be elaborated here.

[0129] Among them, the query instruction of the user can be a voice instruction or an instruction input on the user interaction terminal. The recognition of the voice instruction can be a pre-set and trained voice recognition model, and specific limitations are not made here.

[0130] It can be seen that in this embodiment, the response to user requirements is increased. If a user's query instruction is received, the user's query instruction can be responded to, and repeated teaching can be carried out according to the actual situation of the user. Furthermore, the teaching of the humanoid robot is made more accurate and the user experience is better.

[0131] In a possible embodiment, the humanoid robot further includes an audio playback device. The action link configuration data set further includes multiple audio files. Each individual audio file corresponds to a single action task. The method further includes: when the humanoid robot is in a state of teaching a single action task, controlling the audio playback device to play the single audio file corresponding to the single action task; if the teaching of the action task is in a paused state, pausing the playback of the single audio file. The paused state indicates that the current humanoid robot is not in a state of teaching an action task.

[0132] Among them, the multiple audio files are the explanatory audio files corresponding to a single action task and other background sounds. The audio is the sound data extracted by the mobile terminal from the target teaching video, such as background music, etc. It can also only include background music, or it can refer to including explanatory audio, which can be the audio information extracted and split from the target explanatory video.

[0133] Among them, when it is detected that the humanoid robot is teaching an action task, the matching audio is played. If it is detected that the humanoid robot is in a paused state, the playback of the audio is paused. The paused state can be that the humanoid robot is not in a state of teaching an action task. Specifically, it can be a process of judging the completion degree. The playback of the audio only follows the execution of the action task. If a certain action task is repeatedly executed, the audio of that action task is played. If it is switched to another action task, for example, the user issues a query instruction, the corresponding audio is played according to which action task is re-taught after the jump. The audio is not played from beginning to end completely according to the audio rhythm of the original target teaching video.

[0134] For example, action task A corresponds to audio A, action task B corresponds to audio B, and action task C corresponds to audio C. If action task A is currently being executed, the corresponding audio A is played. After the execution is completed, a completion degree test is performed on action task A, and at this time, no audio is played. When it is detected that teaching of a subsequent action task can be carried out, that is, teaching of action task B is carried out, the corresponding audio B is played. If it is detected that the completion degree does not meet the action task constraint conditions, then it may be necessary to re-execute action task A at this time, and the corresponding audio A is played. After action task C is executed and a user's query instruction is received, and it is detected that the action task corresponding to the user's second target object is action B, then teaching of action task B is carried out and audio B is played.

[0135] It can be seen that in this embodiment, the played audio is split according to the action task and played in a matching manner, enabling the audio to match the teaching of the action task, resulting in a better teaching effect, improving the accuracy of the embodied service, enhancing the user's learning effect, and thus increasing the user's satisfaction.

[0136] For example, please refer to Figure 5 , Figure 5 which is a schematic diagram of a scenario of an embodied service for teaching videos provided by an embodiment of the present application. Suppose a humanoid robot needs to teach a child origami. Based on the robot's body data, the initial adaptive pose of the robot at the teaching stage is determined. For example, since the child's desk is very low, optional initial postures for the robot are given, such as kneeling or squatting. And, based on the target teaching video that the user has imported from the user interaction terminal, a dataset for action link configuration is obtained. Also, the user interaction terminal prompts the user to establish a connection between the user interaction terminal and the humanoid robot, which can be by inserting into the robot body interface, connecting through a data cable, or establishing a wireless communication connection. After identifying the communication status of the interface, the user is informed by voice that the function has been initialized and asked whether to start. When the robot detects a user confirmation operation, the voice confirmation operation can be a voice or an input instruction. The robot body performs an operation teaching of the corresponding action task on the first target object (paper), and the user performs an action on the second target object (paper) following the teaching of the humanoid robot. During the process, the robot detects the completion degrees of the robot tasks and the user tasks for each action task to ensure that the completion degrees meet the conditions. And, corresponding audio is played during the action task teaching. When the user has doubts about the currently taught step and issues a question instruction, the robot can scan the current origami shape of the user based on the question instruction. Then, the robot determines the current action task where the user is located based on the previously completed action tasks and the action link configuration dataset, and then the robot returns to this action task again to demonstrate origami.

[0137] It can be seen that in this embodiment, by implementing the embodiments of the present application, multiple action link configuration data sets from a mobile terminal are obtained. The multiple action link configuration data sets have a corresponding relationship with multiple action tasks of a teaching task. The single action link configuration data set of a single action task includes a single frame image, a single action parameter set, and a single action task constraint condition. The single action task constraint condition is a trigger condition for the single action task to transition to the next action task or to confirm that the teaching task has been completed. According to the single action parameter set, the humanoid robot is controlled to perform operation teaching on a first target object, where the first target object is the operation object of the humanoid robot. The humanoid robot is controlled to obtain a first object state image and a second object state image. The first object state image is the image of the first target object when the single action task is completed, and the second object state image is the image of a second target object when the single action task is completed. The second target object is the operation object of the user. The robot task completion degree is determined according to the relationship between the first object state image and the single frame image, and the user task completion degree is determined according to the second object state image and the single frame image. According to the robot task completion degree, the user task completion degree, and the action task constraint condition, the subsequent single action task is operated and taught based on the corresponding action link configuration data set until it is confirmed that the teaching task has been completed. Compared with the existing strategy of directly generating a demonstration plan by the robot based on a human demonstration video, this solution splits the target teaching video to be demonstrated and adds a transition judgment condition between each action task, demonstrates, pauses, or repeats the demonstration based on the actual situation of the user, improves the flexibility and intelligence of the embodied service of the humanoid robot teaching video, is beneficial to improving the personalization degree for different teaching situations, and enhances the user experience.

[0138] Please refer to Figure 7 , Figure 7FIG. 0 is a schematic structural diagram of an embodied service device for teaching videos proposed in an embodiment of the present application. The device is applied to a controller of a humanoid robot, and the humanoid robot is communicatively connected to a mobile terminal. The embodied service device 700 for teaching videos includes: a data acquisition module 710, a first operation teaching module 720, a state image acquisition module 730, a completion degree determination module 740, and a second operation teaching module 750. Among them: The data acquisition module 710 is configured to acquire a plurality of action link configuration data sets from the mobile terminal. The plurality of action link configuration data sets have a corresponding relationship with a plurality of action tasks of the teaching task. A single action link configuration data set of a single action task includes a single frame image, a single action parameter set, and a single action task constraint condition. The single action task constraint condition is a trigger condition for a single action task to transition to the next single action task or to confirm that the teaching task has been completed; The first operation teaching module 720 is configured to control the humanoid robot to perform operation teaching on a first target object according to the single action parameter set. The first target object is an operation object of the humanoid robot; The state image acquisition module 730 is configured to control the humanoid robot to acquire a first object state image and a second object state image. The first object state image is an image of the first target object when a single action task is completed, and the second object state image is an image of the second target object when a single action task is completed. The second target object is an operation object of the user; The completion degree determination module 740 is configured to determine the robot task completion degree according to the relationship between the first object state image and the single frame image, and to determine the user task completion degree according to the relationship between the second object state image and the single frame image; The second operation teaching module 750 is configured to perform operation teaching on the subsequent single action task based on the corresponding action link configuration data set according to the robot task completion degree, the user task completion degree, and the action task constraint condition until it is confirmed that the teaching task has been completed.

[0139] In a possible implementation manner, the humanoid robot further includes a head structure, and a picture acquisition device is arranged on the head structure. In terms of controlling the humanoid robot to acquire the first object state image and the second object state image, the state image acquisition module 730 is specifically configured to:

[0140] Acquire the shooting angle of the single frame image according to the single frame image;

[0141] Obtain a target angle according to the shooting angle and the position parameter of the head structure. The target angle is the rotation angle of the head structure of the humanoid robot, and the position parameter of the head structure is the relative position between the head structure and the first target object or the second target object;

[0142] Control the head structure of the humanoid robot to adjust its position according to the target angle so that the picture angle acquired by the picture acquisition device is the same as the picture angle of the single frame image;

[0143] Obtain the first object state image and the second object state image.

[0144] In a possible implementation, the completion determination module 740 is specifically configured to determine the robot task completion degree based on the relationship between the first object state image and the single-frame image as follows:

[0145] Obtain the first image feature from the first object state image and obtain the second image feature from the single-frame image;

[0146] Perform error analysis based on the first image feature and the second image feature to obtain the robot task completion degree;

[0147] Determining the user task completion degree based on the second object state image and the single-frame image includes:

[0148] Obtain the third image feature from the second object state image and obtain the fourth image feature from the single-frame image;

[0149] Perform error analysis based on the third image feature and the fourth image feature to obtain the user task completion degree.

[0150] In a possible implementation, the action task constraint conditions include that the robot task completion degree meets the first completion threshold and / or the user task completion degree meets the second completion threshold. The second operation teaching module 750 is specifically configured to perform operation teaching on the subsequent single action task based on the action link configuration data set corresponding to the subsequent according to the robot task completion degree, the user task completion degree, and the action task constraint conditions as follows:

[0151] Judge whether the robot task completion degree meets the first completion threshold and judge whether the user task completion degree meets the second completion threshold. The first completion threshold and the second completion threshold are preset thresholds;

[0152] If the robot task completion degree meets the first completion threshold and the user task completion degree meets the second completion threshold, then transfer to the subsequent single action task and perform operation teaching on the subsequent single action task based on the action link configuration data set corresponding to the subsequent;

[0153] If the robot task completion degree does not meet the first completion threshold or the user task completion degree does not meet the first completion threshold, then re-perform operation teaching on the current single action task.

[0154] In a possible implementation, the action link configuration data set further includes the task frame images of the action task. The task frame images are other frame images except for the single frame image. The task frame images are the influence information of a single action task in an unfinished state. The completion degree determination module 740 is specifically further used for:

[0155] If the robot task completion degree meets the first completion degree threshold, store the action link configuration data set of the single action task corresponding to the first object state image;

[0156] When receiving a query instruction issued by the user, control the screen acquisition device of the humanoid robot to acquire a third object state image, where the third object state image is an image of the current operation target of the user acquired by the humanoid robot;

[0157] Match the corresponding task frame image or single frame image based on the third object state image;

[0158] Based on the single action task corresponding to the task frame image or single frame image, retrieve the action link configuration data set corresponding to the single action task;

[0159] Based on the retrieved action link configuration data set and the action imitation system, control the humanoid robot to perform operation teaching on the first target object. The action imitation system is used to convert the single action parameter set into a machine task.

[0160] In a possible embodiment, the humanoid robot further includes an audio playback device. The action link configuration data set further includes multiple audios. Each single audio corresponds to a single action task. The first operation teaching module 720 is specifically further used for:

[0161] When the humanoid robot is in the state of performing operation teaching on a single action task, control the audio playback device to play the single audio corresponding to the single action task;

[0162] If the operation teaching is in a paused teaching state, pause playing the single audio. The paused teaching state indicates that the current humanoid robot is not in the state of operation teaching.

[0163] It should be noted that for the specific functional implementation manner of the teaching video embodied service device 700, refer to the above Figure 3Description of the teaching video embodied service method shown, for example, the data acquisition module 710 is used to implement the relevant content of executing S310. Each unit or module in the teaching video embodied service device 700 can be separately or entirely combined into one or several other units or modules to form, or some of the units or modules can be further split into multiple smaller units or modules in terms of function to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above units or modules are divided based on logical functions. In practical applications, the function of one unit (or module) is realized by multiple units (or modules), or the functions of multiple units (or modules) are realized by one unit (or module).

[0164] It can be seen that the teaching video embodied service device described in the embodiments of the present application obtains multiple action link configuration data sets from a mobile terminal. The multiple action link configuration data sets have a corresponding relationship with multiple action tasks of a teaching task. A single action link configuration data set of a single action task includes a single frame image, a single action parameter set, and a single action task constraint condition. The single action task constraint condition is a trigger condition for a single action task to transition to the next single action task or confirm that the teaching task has been completed; control a humanoid robot to perform operation teaching on a first target object according to the single action parameter set, where the first target object is the operation object of the humanoid robot; control the humanoid robot to obtain a first object state image and a second object state image. The first object state image is the image of the first target object when a single action task is completed, and the second object state image is the image of a second target object when a single action task is completed. The second target object is the operation object of the user; determine the robot task completion degree according to the relationship between the first object state image and the single frame image, and determine the user task completion degree according to the second object state image and the single frame image; according to the robot task completion degree, the user task completion degree, and the action task constraint condition, perform operation teaching on the subsequent single action task based on the corresponding action link configuration data set until it is confirmed that the teaching task has been completed. Compared with the existing strategy of a robot directly generating a demonstration plan based on a human demonstration video, this solution splits the target teaching video to be demonstrated and adds a transfer judgment condition between each action task, and demonstrates, pauses, or repeats the demonstration based on the actual situation of the user, improving the flexibility and intelligence of the humanoid robot teaching video embodied service, being beneficial to improving the personalization degree for different teaching situations, and enhancing the user experience.

[0165] Please refer to Figure 8 , Figure 8 is a schematic structural diagram of a humanoid robot proposed in the embodiments of the present application, as Figure 8As shown, the humanoid robot 800 includes a processor 810, a memory 820, a communication interface 830, and one or more programs 821. The one or more programs 821 are stored in the memory 820 and are configured to be executed by the processor 810. The processor 810, the memory 820, and the communication interface 830 are interconnected and complete communication with each other.

[0166] Among them, the memory 820 can be a volatile memory such as a dynamic random access memory DRAM, or a non-volatile memory such as a mechanical hard disk. The memory 820 is used to store a set of executable program codes, and the processor 810 is used to call one or more programs 821 stored in the memory 820 and can execute some or all of the steps of any teaching video embodied service method described in the foregoing embodiments.

[0167] The embodiment of the present application also provides a computer storage medium. Among them, the computer storage medium stores a computer program for electronic data exchange, and the computer program enables the computer to execute some or all of the steps of any method described in the foregoing method embodiments. The computer includes an electronic device.

[0168] The embodiment of the present application also provides a computer program product. The computer program product includes a non-transitory computer-readable storage medium storing a computer program. The computer program is operable to enable the computer to execute some or all of the steps of any method described in the foregoing method embodiments. The computer program product can be a software installation package, and the computer includes an electronic device.

[0169] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0170] In the foregoing embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0171] In several embodiments provided by the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the above division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling, direct coupling, or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical or other form.

[0172] The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] In addition, in each embodiment of the present application, the functional units can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0174] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer electronic device (which can be a personal computer, an electronic device, or a network electronic device, etc.) to execute all or part of the steps of the above methods in each embodiment of the present application. The aforementioned memory includes: USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), mobile hard disks, magnetic disks, or optical disks and other media that can store program codes.

[0175] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program. This program can be stored in a computer-readable memory. The memory can include: flash drives, read-only memories (English: Read-Only Memory, abbreviated: ROM), random access memories (English: Random Access Memory, abbreviated: RAM), magnetic disks, or optical disks, etc.

[0176] The above has introduced the embodiments of the present application in detail. Specific examples are used herein to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A method for embodied services of teaching videos, characterized in that, A controller for a humanoid robot, the humanoid robot being communicatively connected to a mobile terminal, the method comprising: Obtaining a plurality of action link configuration data sets from the mobile terminal, the plurality of action link configuration data sets having a corresponding relationship with a plurality of action tasks of a teaching task, a single action link configuration data set of a single action task including a single frame image, a single action parameter set, and a single action task constraint condition, the single action task constraint condition being a trigger condition for the single action task to transition to the next single action task or to confirm that the teaching task has been completed; Controlling the humanoid robot to perform operation teaching on a first target object according to the single action parameter set, the first target object being an operation object of the humanoid robot; Controlling the humanoid robot to obtain a first object state image and a second object state image, the first object state image being an image of the first target object when the single action task is completed, the second object state image being an image of a second target object when the single action task is completed, the second target object being an operation object of the user; Determining the robot task completion degree according to the relationship between the first object state image and the single frame image, and determining the user task completion degree according to the relationship between the second object state image and the single frame image; According to the robot task completion degree, the user task completion degree, and the action task constraint condition, performing the operation teaching on the subsequent single action task based on the action link configuration data set corresponding to the subsequent single action task until it is confirmed that the teaching task has been completed.

2. The method according to claim 1, wherein The humanoid robot further includes a head structure, and a picture acquisition device is provided on the head structure. The controlling the humanoid robot to obtain a first object state image and a second object state image includes: Obtaining the shooting angle of the single frame image according to the single frame image; Obtaining a target angle according to the shooting angle and the position parameter of the head structure, the target angle being the rotation angle of the head structure of the humanoid robot, and the position parameter of the head structure being the relative position of the head structure to the first target object or the second target object; Controlling the position of the head structure of the humanoid robot to be adjusted according to the target angle so that the picture angle obtained by the picture acquisition device is consistent with the picture angle of the single frame image; Obtaining the first object state image and the second object state image.

3. The method according to claim 2, wherein Determining the robot task completion degree according to the relationship between the first object state image and the single frame image includes: Obtaining a first image feature according to the first object state image, and obtaining a second image feature according to the single frame image; Performing error analysis according to the first image feature and the second image feature to obtain the robot task completion degree; The determining the user task completion degree according to the relationship between the second object state image and the single frame image includes: Obtaining a third image feature according to the two-object state image, and obtaining a fourth image feature according to the single frame image; Error analysis is performed based on the third image feature and the fourth image feature to obtain the user task completion degree.

4. The method according to claim 1 or 3, characterized in that, The action task constraint conditions include that the robot task completion degree meets a first completion degree threshold and / or the user task completion degree meets a second completion degree threshold. Based on the robot task completion degree, the user task completion degree, and the action task constraint conditions, the operation teaching for the subsequent single action task based on the data set configured for the corresponding action link includes: Determine whether the robot task completion degree meets the first completion degree threshold, and determine whether the user task completion degree meets the second completion degree threshold. The first completion degree threshold and the second completion degree threshold are preset thresholds; If the robot task completion degree meets the first completion degree threshold and the user task completion degree meets the second completion degree threshold, then transfer to the subsequent single action task, and perform the operation teaching on the subsequent single action task based on the data set configured for the corresponding action link; If the robot task completion degree does not meet the first completion degree threshold or the user task completion degree does not meet the first completion degree threshold, then re-perform the operation teaching on the current single action task.

5. The method according to claim 4, wherein The data set configured for the action link further includes the task frame image of the action task. The task frame image is other frame images except for the single frame image. The task frame image is the influence information of the single action task in the non-completed state. The method further includes: If the robot task completion degree meets the first completion degree threshold, store the data set configured for the action link of the single action task corresponding to the first object state image; When a query instruction issued by the user is received, control the image acquisition device of the humanoid robot to acquire a third object state image, where the third object state image is an image of the current operation target of the user acquired by the humanoid robot; Match the corresponding task frame image or single frame image based on the third object state image; Based on the single action task corresponding to the task frame image or the single frame image, retrieve the data set configured for the action link corresponding to the single action task; Based on the retrieved data set configured for the action link and the action imitation system, control the humanoid robot to perform the operation teaching on the first target object. The action imitation system is used to convert the single action parameter set into a machine task.

6. The method according to claim 1, wherein The humanoid robot further includes an audio playback device. The data set configured for the action link further includes multiple audio files. A single audio file corresponds to the single action task. The method further includes: When the humanoid robot is in the state of performing the operation teaching on the single action task, control the audio playback device to play the single audio file corresponding to the single action task; If the operation teaching is in the paused teaching state, pause playing the single audio file. The paused teaching state indicates that the current humanoid robot is not in the state of performing the operation teaching.

7. The method according to claim 1, characterized in that The obtaining of a plurality of action link configuration data sets from the mobile terminal includes: Detecting whether a user inputs a target teaching video. If the target teaching video is detected, splitting the target teaching video into frame images to obtain a plurality of teaching frame images; Performing action task segmentation according to the plurality of teaching frame images to obtain the plurality of action tasks, where a single action task includes at least one of the teaching frame images; Generating an action link configuration data set according to the plurality of action tasks.

8. An embodied service device for teaching videos, characterized in that, Applied to a controller of a humanoid robot, the humanoid robot being communicatively connected to a mobile terminal, the device includes: A data acquisition module, configured to acquire a plurality of action link configuration data sets from the mobile terminal, the plurality of action link configuration data sets having a corresponding relationship with a plurality of action tasks of a teaching task. A single action link configuration data set of a single action task includes a single frame image, a single set of action parameters, and a single action task constraint condition, and the single action task constraint condition is a trigger condition for the single action task to transition to the next action task or to confirm that the teaching task has been completed; A first operation teaching module, configured to control the humanoid robot to perform operation teaching on a first target object according to the single set of action parameters, where the first target object is an operation object of the humanoid robot; A state image acquisition module, configured to control the humanoid robot to acquire a first object state image and a second object state image. The first object state image is an image of the first target object when the single action task is completed, and the second object state image is an image of a second target object when the single action task is completed. The second target object is an operation object of the user; A completion degree determination module, configured to determine the robot task completion degree according to the relationship between the first object state image and the single frame image, and to determine the user task completion degree according to the second object state image and the single frame image; A second operation teaching module, configured to perform the operation teaching on the subsequent single action task based on the corresponding action link configuration data set according to the robot task completion degree, the user task completion degree, and the action task constraint condition until it is confirmed that the teaching task has been completed.

9. A computer-readable storage medium, characterized in that, A storage medium stores a teaching video embodied service program, including execution instructions. When a processor executes the execution instructions, the processor executes the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes a processor and a memory storing execution instructions, the memory storing one or more programs; when the processor executes the execution instructions stored in the memory, the processor executes the method according to any one of claims 1 to 7.

Images