Sign language teaching method based on humanoid robot and related products

Abstract

The embodiment of the present invention discloses a sign language teaching method based on a humanoid robot and related products, the method comprising: obtaining a sign language teaching instruction; determining first relative position information; adjusting the initial position, initial posture and initial orientation of the humanoid robot based on the first relative position information, so that the humanoid robot is in a first position, a first posture and a first orientation to interact with a user; when the target scene mode is a teaching mode, determining a first sign language action corresponding to a target learning content; adjusting the first position, first posture and first orientation of the humanoid robot based on the first sign language action, so that the humanoid robot is in a second position, a second posture and a second orientation to interact with the user; demonstrating the first sign language action and obtaining the second sign language action of the user; determining the similarity between the first sign language action and the second sign language action to obtain a first similarity; determining the sign language accuracy of the second sign language action based on the first similarity, and feeding back the sign language accuracy to the user.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a sign language teaching method based on a humanoid robot and related products. Background Art

[0002] As a unique language system, sign language has a rich vocabulary and grammatical structure and can accurately convey a variety of complex information. The hearing-impaired mainly rely on sign language to communicate in their daily lives. Sign language is an important tool for them to express themselves and understand others. As society pays more and more attention to the disabled, it has become crucial to promote effective communication between the hearing-impaired and the hearing people. All sectors of society are also paying more and more attention to the popularization of sign language. Many volunteers, service industry personnel, etc. have the need to learn sign language to better provide services for the hearing-impaired and promote social integration and inclusion. In addition, in addition to being used for effective communication between the hearing-impaired and the hearing people, sign language can also be used for communication in noisy environments such as factories or public places.

[0003] Traditional sign language teaching mainly relies on on-site demonstrations and explanations by sign language teachers, or teaching through videos, pictures and other materials. These teaching methods have certain limitations, such as limited teacher resources, the difficulty of teaching progress to meet the needs of different sign language learners, the single look and feel of videos and pictures, etc., making it difficult to teach intuitively, resulting in the learning efficiency of sign language learners not being guaranteed. Current general-purpose robots do not have the ability to teach sign language.

[0004] Therefore, there is an urgent need for a sign language teaching method based on a humanoid robot to meet the sign language learning needs of sign language learners and improve their sign language learning efficiency. Summary of the invention

[0005] In order to solve the above problems, the embodiments of the present invention provide a sign language teaching method and related products based on a humanoid robot, in which sign language learners are taught sign language by a humanoid robot to meet the sign language learning needs of sign language learners and improve their sign language learning efficiency.

[0006] In a first aspect, an embodiment of the present invention provides a sign language teaching method based on a humanoid robot, wherein the sign language teaching method based on a humanoid robot is applied to a humanoid robot, wherein the humanoid robot includes a visual module and a sign language demonstration component; the sign language teaching method based on a humanoid robot includes:

[0007] Obtain the user's sign language teaching instructions;

[0008] In response to the sign language teaching instruction, determining first relative position information between the user and the humanoid robot through the visual module;

[0009] Adjusting the initial position, initial posture and initial orientation of the humanoid robot based on the first relative position information to obtain a first position, a first posture and a first orientation, so that the humanoid robot is in the first position, the first posture and the first orientation to interact with the user;

[0010] Get the target scene mode;

[0011] When the target scenario mode is a teaching mode, obtaining target learning content;

[0012] Determining a first sign language action corresponding to the target learning content;

[0013] Adjusting the first position, the first posture, and the first orientation of the humanoid robot based on the first sign language action to obtain a second position, a second posture, and a second orientation, so that the humanoid robot is in the second position, the second posture, and the second orientation to interact with the user;

[0014] Controlling the humanoid robot to demonstrate the first sign language action through the sign language demonstration component to guide the user to follow the humanoid robot to learn the first sign language action, and obtaining the user's second sign language action through the visual module;

[0015] Determine the similarity between the first sign language action and the second sign language action to obtain a first similarity;

[0016] The sign language accuracy of the second sign language action is determined based on the first similarity, and the sign language accuracy is fed back to the user.

[0017] In a second aspect, an embodiment of the present invention provides a sign language teaching device based on a humanoid robot, wherein the sign language teaching device based on a humanoid robot is applied to the humanoid robot, wherein the humanoid robot includes a visual module and a sign language demonstration component; the sign language teaching device based on the humanoid robot includes an acquisition unit and a processing unit;

[0018] The acquisition unit is used to acquire the user's sign language teaching instructions;

[0019] The processing unit is used to determine the first relative position information between the user and the humanoid robot through the visual module in response to the sign language teaching instruction;

[0020] Adjusting the initial position, initial posture and initial orientation of the humanoid robot based on the first relative position information to obtain a first position, a first posture and a first orientation, so that the humanoid robot is in the first position, the first posture and the first orientation to interact with the user;

[0021] Get the target scene mode;

[0022] When the target scenario mode is a teaching mode, obtaining target learning content;

[0023] Determining a first sign language action corresponding to the target learning content;

[0024] Adjusting the first position, the first posture, and the first orientation of the humanoid robot based on the first sign language action to obtain a second position, a second posture, and a second orientation, so that the humanoid robot is in the second position, the second posture, and the second orientation to interact with the user;

[0025] Controlling the humanoid robot to demonstrate the first sign language action through the sign language demonstration component to guide the user to follow the humanoid robot to learn the first sign language action, and obtaining the user's second sign language action through the visual module;

[0026] Determine the similarity between the first sign language action and the second sign language action to obtain a first similarity;

[0027] The sign language accuracy of the second sign language action is determined based on the first similarity, and the sign language accuracy is fed back to the user.

[0028] In a third aspect, an embodiment of the present invention provides a humanoid robot, comprising: a processor and a memory, wherein the processor is connected to the memory, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the humanoid robot performs the method described in the first aspect.

[0029] Implementing the embodiments of the present application has the following beneficial effects:

[0030] In the implementation mode of the present application, the user's sign language teaching instruction is first obtained, and in response to the sign language teaching instruction, the first relative position information between the user and the humanoid robot is determined through the visual module, and the initial position, initial posture and initial orientation of the humanoid robot are adjusted based on the first relative position information to obtain the first position, first posture and first orientation, so that the humanoid robot is in the first position, first posture and first orientation to interact with the user, and then, the target scene mode is obtained, and when the target scene mode is the teaching mode, the target learning content is obtained, and the first sign language action corresponding to the target learning content is determined, and the first position, first posture and first orientation of the humanoid robot are adjusted based on the first sign language action to obtain the second position, second posture and second orientation, so that the humanoid robot is in the second position, second posture and second orientation to interact with the user, and then, the humanoid robot is controlled to demonstrate the first sign language action through the sign language demonstration component to guide the user to follow the humanoid robot to learn the first sign language action, and the second sign language action of the user is obtained through the visual module, and the similarity between the first sign language action and the second sign language action is determined to obtain the first similarity, and the sign language accuracy of the second sign language action is determined based on the first similarity, and the sign language accuracy is fed back to the user. Therefore, by adjusting the position, posture and orientation of the humanoid robot, acquiring the target learning content in the teaching mode and determining the first sign language action corresponding to the target learning content, and then demonstrating the first sign language action through the sign language demonstration component, and feeding back the user's sign language accuracy to the user, personalized sign language teaching can be achieved according to different users, thereby improving the user's sign language learning effect and sign language learning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the background technology, the drawings required for use in the embodiments of the present invention or the background technology will be described below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 is a schematic structural diagram of a humanoid robot provided in an embodiment of the present application;

[0033] Figure 2 is a flow chart of a sign language teaching method based on a humanoid robot provided in an embodiment of the present application;

[0034] Figure 3 is a schematic diagram of the positional relationship between a humanoid robot and a user during sign language teaching provided by an embodiment of the present application;

[0035] Figure 4 It is a structural schematic diagram of a sign language teaching device based on a humanoid robot provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0037] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other steps or modules inherent to these processes, methods, products or devices.

[0038] Reference to "embodiments" herein means that a particular feature, result, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0039] The humanoid robot-based sign language teaching method provided in the embodiment of the present application is applied to the humanoid robot, see Figure 1 , Figure 1 is a schematic diagram of the structure of a humanoid robot provided in an embodiment of the present application, such as Figure 1As shown, the humanoid robot includes a visual module 11 and a sign language demonstration component 12. The visual module 11 can be a camera device with a data acquisition function, and the sign language demonstration component 12 is used to demonstrate the first sign language action. The visual module 11 can capture the surrounding environment and the user's behavior, expression, posture and other information in real time and carefully. Whether in a bright day or in a relatively dim indoor environment, the visual module 11 can obtain clear and accurate data images through its sensitive photosensitive element and intelligent image adjustment function, providing a reliable basis for subsequent analysis and processing. The sign language demonstration component 12 has a highly flexible mechanical structure and can accurately simulate various complex sign language actions. From simple single-hand gestures to complex actions coordinated by both hands, the sign language demonstration component 12 can demonstrate smoothly and naturally through its internal precise driving system and optimized control algorithm to ensure that the details of each sign language action can be accurately displayed, whether it is the bending angle of the fingers or the direction of the palm, or the movement speed and trajectory of the hand, etc., can achieve a high degree of restoration, so that the user can clearly and intuitively learn and understand the meaning expressed by the first sign language action.

[0040] See also Figure 2 , Figure 2 is a flow chart of a sign language teaching method based on a humanoid robot provided in an embodiment of the present application. Figure 2 As shown, the method includes but is not limited to the following steps:

[0041] Step S101: Obtaining a user's sign language teaching instruction;

[0042] Step S102: In response to the sign language teaching instruction, determining first relative position information between the user and the humanoid robot through a visual module;

[0043] Step S103: adjusting the initial position, initial posture and initial orientation of the humanoid robot based on the first relative position information to obtain a first position, a first posture and a first orientation, so that the humanoid robot is in the first position, the first posture and the first orientation to interact with the user;

[0044] Step S104: obtaining a target scene mode;

[0045] Step S105: when the target scene mode is the teaching mode, obtaining the target learning content;

[0046] Step S106: determining a first sign language action corresponding to the target learning content;

[0047] Step S107: adjusting the first position, the first posture and the first orientation of the humanoid robot based on the first sign language action to obtain the second position, the second posture and the second orientation, so that the humanoid robot is in the second position, the second posture and the second orientation to interact with the user;

[0048] Step S108: controlling the humanoid robot to demonstrate the first sign language action through the sign language demonstration component to guide the user to follow the humanoid robot to learn the first sign language action, and obtaining the user's second sign language action through the visual module;

[0049] Step S109: determining the similarity between the first sign language action and the second sign language action to obtain a first similarity;

[0050] Step S110: determining the sign language accuracy of the second sign language action based on the first similarity, and feeding back the sign language accuracy to the user.

[0051] In a possible embodiment, the user may be a hearing-impaired person or a hearing-impaired person, and the sign language teaching instruction may be a preset action instruction, which is used to indicate that the user wishes to start the sign language teaching function of the humanoid robot. After the action instruction is obtained by the visual module of the humanoid robot, the action instruction can be used as a sign language teaching instruction and respond to the sign language teaching instruction. The sign language teaching instruction may also be a preset voice instruction or a page interaction instruction. For example, the humanoid robot may be provided with a human-machine interaction device or connected to a user device. The human-machine interaction device may be a voice receiving device or a touch display screen. The user's sign language teaching instruction may be obtained by obtaining the user's voice through the voice receiving device, or by obtaining the user's click operation on a preset virtual touch button through the touch display screen. After the voice instruction or page interaction instruction issued by the user is received by the human-machine interaction device, the sign language teaching instruction may be responded to. By providing a variety of instruction receiving methods such as a touch display screen and a voice receiving device, the usage habits of different users can be met, and the convenience and friendliness of the user's interaction with the humanoid robot are enhanced.

[0052] In a possible embodiment, the image information of the user is obtained through a visual module, the relative position coordinates of the user and the humanoid robot are determined through an image recognition algorithm, and the relative position coordinates are used as the first relative position information. A two-dimensional relative coordinate system or a three-dimensional relative coordinate system is established with the humanoid robot as the origin, the position information of the user is determined through the image information of the user, and the position information of the user is converted into relative position coordinates. For example, the visual module detects that the user is 2 meters in front of the humanoid robot and 30 degrees to the left, then the coordinates of the user in the relative coordinate system may be: x=1.73 meters, y=1 meter and z=0 meters. Determining the first relative position information through the visual module provides accurate data support for the subsequent movement, posture adjustment and interaction of the humanoid robot, which can make the interaction between the humanoid robot and the user more natural and efficient, and avoid interaction obstacles caused by inaccurate position.

[0053] In a possible embodiment, the humanoid robot includes a lower limb component, which is used to control the movement of the humanoid robot. The lower limb component may include wheels or tracks. The humanoid robot is controlled to adjust and move from an initial position to a first position according to the first relative position information. In the process of moving from the initial position to the first position, the moving speed may be adjusted according to the distance and the complexity of the environment. The moving speed may be faster in an open environment without obstacles. The initial posture may be adjusted by controlling the waist, neck, shoulder, leg and other joint components of the humanoid robot to rotate and bend according to the first relative position information. The humanoid robot is placed in the first position, the first posture and the first orientation to interact with the user so that the humanoid robot can maintain a suitable distance from the user. Adjusting the moving speed of the humanoid robot in different environments can ensure that the humanoid robot reaches the first position safely and quickly, thereby improving the interaction efficiency. By adjusting the initial posture and initial orientation of the humanoid robot, the humanoid robot can interact and communicate with the user in a comfortable and natural way. For example, adjusting the neck and waist joints can enhance the affinity of the interaction, make the user feel more intimate, and improve the enthusiasm for learning.

[0054] In a possible embodiment, the target scene mode may include one of the following: teaching mode, question-and-answer mode, practice mode, and test mode, etc. The target scene mode is determined by obtaining the user's actions, voice, or click data on the touch screen. When the target scene mode is the teaching mode, the target learning content can be determined according to the user's sign language learning progress, or the user can choose the content to be learned by himself. In this way, the personalized choice of users at different learning stages and learning needs is met. For example, in the early stage of learning, users may need a teaching mode for systematic learning. After mastering a certain sign language foundation, they can switch to a practice mode or a test mode for consolidation and testing. This flexible mode selection can improve the learning effect and meet the learning styles and progress of different users.

[0055] In a possible embodiment, the first position, first posture and first orientation of the humanoid robot are adjusted based on the first sign language action to obtain the second position, second posture and second orientation, so that the humanoid robot is in the second position, second posture and second orientation to interact with the user. By adjusting the first orientation and the first posture, it can be ensured that the user has the best visual effect when watching the humanoid robot demonstrate sign language, making the demonstration of sign language actions clearer, which helps users learn sign language more accurately. By adjusting the first position, the user will not be able to see the sign language action of the humanoid robot clearly due to the long distance, nor will the user feel oppressed due to the close distance.

[0056] In a possible embodiment, the sign language demonstration component of the humanoid robot may include at least one of the following: a shoulder component, a torso component, a hand component and a head component, wherein the hand component may be a mechanical arm, and the head component includes a facial component, which is used to demonstrate the facial expression and lip movement in the first sign language movement. The first sign language movement can be demonstrated by the sign language demonstration component of the humanoid robot. During the demonstration process, the demonstration speed can be adjusted according to the learning progress and learning level of the user. For beginners, the movement can be slowed down, and the pause time between each movement can be appropriately extended. The humanoid robot can output text prompt information or voice prompt information through a human-computer interaction device to guide the user to follow the first sign language movement learning. By capturing the user's movement through a visual module, the user's hand and body movements can be obtained from different angles to obtain the user's second sign language movement. By performing the first sign language movement through the sign language demonstration component, while demonstrating the hand movement, the body posture movement and facial expression movement corresponding to the hand movement can be demonstrated, so that the meaning of the sign language can be vividly and accurately displayed, so that the user can better understand the details of the sign language, and the rich body language and expression can assist the user in understanding the emotions and context expressed by the sign language vocabulary. Acquiring the user's actions from multiple angles through the visual module can avoid misjudgment of actions caused by a single perspective and provide more reliable data for subsequent similarity calculation and feedback.

[0057] In a possible embodiment, the first sign language action and the second sign language action are decomposed, key feature points such as the position of the hand joints and the bending angle of the fingers are extracted, and the degree of difference of the key feature points is compared. According to the comparison result of the feature points, the similarity between the first sign language action and the second sign language action is calculated using a similarity calculation formula to obtain a first similarity, and the sign language accuracy calculated by the first similarity is fed back to the user in the form of voice or text. By decomposing the action and comparing the key feature points, the difference between the user's sign language action and the standard action can be analyzed, and the user's learning effect can be objectively and quantitatively evaluated, so that the user can intuitively understand how close he is to the standard action, which is convenient for measuring the learning progress and level.

[0058] Optionally, in step S103, adjusting the initial position, initial posture and initial orientation of the humanoid robot based on the first relative position information to obtain the first position, first posture and first orientation may include the following steps:

[0059] Step S201: determining the current position of the user based on the first relative position information;

[0060] Step S202: obtaining the current position and current posture of the user, and first obstacle information between the initial position and the current position through a visual module, where the first obstacle information is used to reflect the spatial distribution of the obstacles;

[0061] Step S203: determining a first position and a first orientation based on the first obstacle information, a preset distance relationship, a preset orientation relationship, a current position and a current orientation;

[0062] Step S204: determining a first posture based on the relationship between the current posture and a preset posture;

[0063] Step S205: if the initial position is inconsistent with the first position, determining a moving path based on the initial position, the first position and the first obstacle information;

[0064] Step S206: if the initial posture is inconsistent with the first posture, determining a vertical offset based on the initial posture and the first posture;

[0065] Step S207: if the initial orientation is inconsistent with the first orientation, determining a rotation direction and a rotation angle based on the initial orientation and the first orientation;

[0066] Step S208: adjusting the initial position, initial posture and initial orientation based on the moving path, the vertical offset, the rotation direction and the rotation angle to obtain a first position, a first posture and a first orientation.

[0067] In a possible embodiment, the preset orientation relationship may be that the first orientation of the humanoid robot is opposite to the current orientation of the user, that is, the humanoid robot is facing the user, the preset distance relationship may be determined according to the arm length of the user, and the preset posture relationship may be that the center reference point of the humanoid robot and the center reference point of the user are at the same horizontal height. The reference orientation is determined based on the preset orientation relationship and the current orientation, and the reference position is determined based on the preset distance relationship, the reference orientation, and the current position. When it is determined through the first obstacle information that there is no obstacle in the spatial area corresponding to the reference position, the reference position is used as the first position, and the reference orientation is used as the first orientation. When it is determined through the first obstacle information that there is an obstacle in the spatial area corresponding to the reference position, the position of the humanoid robot is re-determined based on the first obstacle information to obtain the first position, and the first orientation is determined based on the first position and the current position of the user. If the initial position is inconsistent with the first position, the moving path is determined based on the initial position, the first position, and the first obstacle information, and the humanoid robot is controlled to move from the initial position to the first position based on the moving path. If the initial posture is inconsistent with the first posture, a vertical offset is determined based on the initial posture and the first posture, and the humanoid robot is controlled to adjust the initial posture based on the vertical offset to obtain the first posture. If the initial orientation is inconsistent with the first orientation, a rotation direction and a rotation angle are determined based on the initial orientation and the first orientation, and the humanoid robot is controlled to select based on the rotation direction and the rotation angle to be in the first orientation.

[0068] For example, see Figure 3 , Figure 3 is a schematic diagram of the positional relationship between a humanoid robot and a user during sign language teaching provided by an embodiment of the present application, such as Figure 3 As shown, when teaching sign language, if the user takes a sitting position to learn sign language, in the absence of obstacles, the humanoid robot can adjust to a reference position, reference posture and reference orientation according to a preset orientation relationship, a preset distance relationship and a preset posture relationship, so as to better interact with the user.

[0069] In the embodiment of the present application, the humanoid robot is controlled to make personalized interaction adjustments according to the actual situation of the user. For example, when the user turns sideways, the position and posture of the humanoid robot can be adjusted to ensure that the user can comfortably see the sign language demonstration action of the humanoid robot. The first position and the first orientation are determined according to the preset distance relationship and orientation relationship, which can optimize the interaction distance and viewing angle between the humanoid robot and the user. The appropriate distance allows the user to clearly see the actions and expressions of the humanoid robot. The frontal orientation helps to enhance the directness and affinity of the interaction. The first posture is determined according to the preset posture relationship, which can make the humanoid robot and the user match each other in posture and enhance the interactive experience. When the humanoid robot and the user maintain a level and frontal posture, the user will feel more natural and comfortable, as if communicating with a real person, which improves the user's acceptance and satisfaction with the humanoid robot.

[0070] Optionally, in step S107, adjusting the first position, the first posture, and the first orientation of the humanoid robot based on the first sign language action to obtain the second position, the second posture, and the second orientation may include the following steps:

[0071] Step S301: obtaining a motion trajectory of a first sign language motion;

[0072] Step S302: determining the key points and demonstration range of the first sign language action based on the action trajectory;

[0073] Step S303: Determine the demonstration orientation and the demonstration posture based on the action points, the current orientation and the current posture;

[0074] Step S304: determining a demonstration distance based on the demonstration amplitude and the current position;

[0075] Step S305: determining a demonstration space based on the motion trajectory;

[0076] Step S306: obtaining the second obstacle information in the demonstration space through the visual module;

[0077] Step S307: Determine the demonstration position based on the demonstration distance, the motion trajectory and the second obstacle information;

[0078] Step S308: Based on the demonstration position, the demonstration posture and the demonstration orientation, the first position, the first posture and the first orientation are adjusted respectively to obtain the second position, the second posture and the second orientation.

[0079] In a possible embodiment, the motion trajectory of the first sign language action is the movement path of the sign language demonstration component over time in three-dimensional space, which can be represented by the coordinate point information on the movement path. The action key points of the first sign language action can be the shape change or the change in movement direction of the sign language demonstration component in the process of demonstrating the first sign language action. The demonstration amplitude is the maximum movement range of the sign language demonstration component in the process of demonstrating the first sign language action. If the demonstration amplitude is small, the humanoid robot is controlled to be close to the user so that the user can better observe the first sign language action with subtle changes. If the demonstration amplitude is large, the humanoid robot is controlled to be away from the user so that the user can observe the complete first sign language action. According to the current position and current posture of the user, and the key points of the first sign language action, the demonstration position and demonstration posture that are convenient for the user to observe are determined. For example, when the first sign language action is demonstrated through the sign language demonstration component, the key points of the first sign language action are all action changes at the same horizontal height. When the humanoid robot is in the first position, the first posture and the first position to demonstrate the sign language through the hand demonstration component, part of the sign language action may be blocked, and the user cannot observe the key points of the action. At this time, the humanoid robot can be adjusted to move from the front of the user to the side of the user, and the user can observe the horizontal action changes from the side of the humanoid robot through the rotation operation. In addition, after determining the demonstration space required for the demonstration process based on the demonstration trajectory, the second obstacle information in the demonstration space is obtained through the visual module. If there is no obstacle in the demonstration space, the demonstration position is determined according to the demonstration distance. If there is an obstacle in the demonstration space, it means that the humanoid robot may encounter an obstacle when demonstrating the first sign language action through the sign language demonstration component. At this time, it is necessary to determine the demonstration position in combination with the second obstacle information so that the humanoid robot will not encounter the user or other objects that hinder the demonstration during the demonstration. After determining the demonstration position, demonstration posture and demonstration orientation, the demonstration position, demonstration posture and demonstration orientation are used as the second position, second posture and second orientation respectively, and the humanoid robot is controlled to adjust from the first position, first posture and first orientation to the second position, second posture and second orientation. If an obstacle appears in the demonstration space through the visual module during the demonstration, the demonstration of the first sign language action can be paused first, and the demonstration can be continued when it is recognized that there is no obstacle in the demonstration space.

[0080] In the embodiment of the present application, determining the action points of the first sign language action helps to highlight the core part of the first sign language action. For learners, it is easier to grasp the changes in the key hand shape, position and moving direction. Adjusting the demonstration orientation according to the user's current position and action points can ensure that the user is always at the best viewing angle. Calculating the demonstration distance based on the current position and demonstration amplitude can ensure that the user observes the sign language demonstration at a suitable distance, and neither the range of the entire action can be seen clearly because the distance is too close, nor the details of the action can be ignored because the distance is too far, which helps to improve the user's understanding and learning effect of the sign language action. By determining the demonstration space, the spatial range required for the demonstration of the hand demonstration component can be predicted in advance, so as to select a suitable position for demonstration in a complex environment, avoiding limited movement due to insufficient space or collision with obstacles in the surrounding environment. Obtaining obstacle information in the demonstration space through the visual module enables the humanoid robot to understand the obstacle situation in the environment in real time. For example, in places such as classrooms or conference rooms, people may move or objects may move at any time. Obtaining the second obstacle information in time can avoid collision with obstacles during the demonstration process, ensuring the continuity and stability of the demonstration.

[0081] Optionally, in step S108, controlling the humanoid robot to demonstrate the first sign language action through the sign language demonstration component may include the following steps:

[0082] Step S401: Obtaining the user's sign language learning progress;

[0083] Step S402: determining the user's sign language learning level based on the sign language learning progress;

[0084] Step S403: determining a first complexity level of the first sign language action based on the sign language learning level;

[0085] Step S404: if the first complexity level is higher than a preset complexity level threshold, decomposing the first sign language action to obtain at least one sub-action;

[0086] Step S405: determining the complexity of each sub-action in at least one sub-action based on the sign language learning level, and obtaining at least one second complexity level;

[0087] Step S406: determining a demonstration speed of at least one sub-action based on the second complexity level to obtain at least one first demonstration speed;

[0088] Step S407: Based on the demonstration order of at least one sub-action and at least one first demonstration speed, control the humanoid robot to demonstrate at least one sub-action through the sign language demonstration component to complete the demonstration of the first sign language action.

[0089] In a possible embodiment, a user's sign language learning record is obtained, and the number of sign language vocabulary learned by the user, the length of learning time, the number of times each sign language action has been practiced, and learning progress information such as the accuracy and proficiency of the user during testing and practice are determined based on the sign language learning record. The user's sign language learning progress is determined based on the learning progress information. Different learning progresses correspond to different sign language learning levels. The sign language learning level is used to reflect the user's sign language learning ability. For users with different sign language learning levels, when learning the same first sign language action, the complexity perceived by the users is also different. A first complexity of the first sign language action is determined based on the sign language learning level. The preset complexity threshold can be determined based on the user's sign language learning level. If the user's sign language learning level is high, the complexity threshold can also be set higher. If the first complexity level is higher than the preset complexity level threshold, it means that the first sign language action is more difficult to learn for the user. At this time, if the first sign language action is demonstrated according to the initial demonstration speed, it may cause the user to have difficulty keeping up with the initial demonstration speed when following the humanoid robot to learn sign language, which will reduce the user's sign language learning experience. Therefore, if the first complexity level is higher than the preset complexity level threshold, the first sign language action is decomposed to obtain at least one sub-action, and the complexity of each sub-action in at least one sub-action is determined based on the sign language learning level to obtain at least one second complexity level, and the demonstration speed of at least one sub-action is determined based on the second complexity to obtain at least one first demonstration speed, and based on the demonstration order of at least one sub-action and at least one first demonstration speed, the humanoid robot is controlled to demonstrate at least one sub-action through the sign language demonstration component to complete the demonstration of the first sign language action. Among them, when determining the first complexity of the first sign language action, the learning difficulty of the first sign language action is first determined based on factors such as the number of hand movements involved in the first sign language action, the degree of change of the sign language movement, and the degree of body coordination. For example, if the first sign language action requires both hands to make different shape changes at the same time, and is accompanied by body movement, then it can be judged that the learning difficulty of the first sign language action is relatively high. Then, the first complexity of the first sign language action for the user is determined based on the learning difficulty of the first sign language action and the user's sign language learning level. When decomposing the first sign language action, it can be decomposed according to the interpretation of the first sign language action, and it can also be decomposed according to the key change nodes of the first sign language action, so that each sub-action is relatively independent and has clear action characteristics, so as to facilitate user understanding and learning.

[0090] In the embodiment of the present application, different demonstration speeds are determined for users with different learning levels, so that the needs of users can be better met and learning efficiency can be improved. When the first sign language action is too complex, the complex first sign language action is decomposed into sub-actions, and an overall complex task is split into multiple relatively simple small tasks, which reduces the learning difficulty of the user, so that the user can understand the details and key points of each action more clearly, and avoid frustration caused by facing too complex actions at one time. In addition, the demonstration speed is determined according to the complexity of the sub-action and the user's learning level. For sub-actions with high complexity, slowing down the speed can allow users to have enough time to observe and understand. For simple sub-actions, speeding up the speed can save time without affecting the learning effect, improve overall learning efficiency, and optimize the user's learning experience.

[0091] Optionally, in step S108, obtaining the user's second sign language action through the visual module may include the following steps:

[0092] Step S501: Obtaining the demonstration start time and demonstration end time of the first sign language action;

[0093] Step S502: Determine the delay detection time based on the sign language learning level;

[0094] Step S503: determining a target detection time period based on the demonstration start time, the demonstration end time and the delayed detection time;

[0095] Step S504: adjusting the initial detection angle of the visual module based on the action points to obtain a target detection angle;

[0096] Step S505: Based on the target detection angle, the user's sign language video is collected within the target detection time period through the visual module;

[0097] Step S506: determining a starting action feature and an ending action feature based on the first sign language action;

[0098] Step S507: Determine a second sign language action based on the sign language video, the start action feature, and the end action feature.

[0099] In a possible embodiment, the reaction time and completion time of users with different sign language learning levels when following the humanoid robot to complete the first sign language action are different. For users with higher sign language learning levels, the delayed detection time can be correspondingly shorter, the starting time of the target detection time period can be the demonstration start time, and the end time of the target detection time period can be the delayed detection time after the demonstration end time. For example, the delayed detection time is 30 seconds, the demonstration start time is 9:30, and the demonstration end time is 9:31:30 seconds, then the target detection time period is 9:30 to 9:32. The initial detection angle of the visual module is adjusted based on the main movement direction and action angle of the hand in the action points to obtain the target detection angle. If the action point of the first sign language action is that the hand moves from the left side of the body to the right side and the palm is displayed forward, then the angle of the visual module is adjusted to the angle that is facing the movement direction of the user's hand and can detect the direction of the user's palm. At the target detection angle, the visual module performs data acquisition at a preset frame rate within the target detection time period, converts the light signal into a digital image signal, and stores the image frame according to the acquisition time to form the user's sign language video. The starting action feature may be key information such as the hand shape and position at the beginning of the first sign language action, which may be determined by the initial shape and position of the sign language demonstration component. The ending action feature may be key information such as the hand shape and position at the completion of the first sign language action. The starting action feature and the ending action feature may also be determined by the preset starting action and ending action, respectively. For example, the starting action feature is that both hands hang naturally, and the ending action feature is that both hands cross in front of the chest. The sign language action between the starting action feature and the ending action feature in the sign language video is taken as the user's second sign language action.

[0100] In the embodiment of the present application, the delay detection time is determined according to different sign language learning levels, which can better adapt to the reaction speed and learning ability of different users. For users with lower sign language learning levels, more reaction time is set to avoid missing detection due to nervousness or unskilledness. For users with higher sign language learning levels, detection feedback can be obtained in a more compact time, thereby improving detection efficiency. By determining the target detection time period, the detection of the visual module is concentrated in the time period when the user is most likely to react, avoiding detection within an unnecessary time range, reducing the amount of data processing, and improving the accuracy and efficiency of detection. By adjusting the detection angle of the visual module based on the key points of the action, it can be ensured that the visual module can capture the user's sign language action at the best viewing angle, so that the key part of the sign language action can be clearly seen, avoiding partial action being blocked or deformed due to poor angles, and improving the quality of the collected data. By determining the starting action feature and the ending action feature of the first sign language action, and matching them in the collected sign language video, the second sign language action performed by the user can be accurately determined. This determination method based on feature matching avoids the error of subjective judgment and improves the accuracy of sign language action recognition.

[0101] Optionally, the method may further include the following steps:

[0102] Step S601: when demonstrating the first sign language action, obtaining the user's expression and the user's sight angle through the visual module;

[0103] Step S602: Determine a first expression tag corresponding to the user's expression;

[0104] Step S603: when the first expression tag is consistent with the preset expression tag, obtaining the target sub-action, adjusting the first demonstration speed corresponding to the target sub-action to obtain the second demonstration speed, and demonstrating the target sub-action based on the second demonstration speed;

[0105] Step S604: determining the user's concentration level based on the user's sight angle;

[0106] Step S605: When the concentration level is lower than a preset concentration level threshold, the sign language demonstration component is controlled to pause the demonstration.

[0107] In a possible embodiment, the first expression label may be confused, frustrated, happy, positive, etc., and the preset expression label may be an expression label that does not match the first sign language action. For example, "confused" is used as the preset expression label. When the user's expression shows an expression such as frowning to indicate confusion, the first expression label is judged to be confused. At this time, the first expression label is consistent with the preset expression label, indicating that the demonstration speed of the first sign language action needs to be adjusted. The target sub-action may be a sub-action that causes the user to be confused. The target sub-action may be determined according to the complexity and novelty of the sub-action. The first demonstration speed corresponding to the target sub-action is adjusted. If the first demonstration speed is fast, when the user shows a confused expression, the first demonstration speed is slowed down to the second demonstration speed, and the target sub-action is re-demonstrated at the adjusted second demonstration speed to make the action clearer and slower, so that the user can better observe and understand. The user's concentration is determined according to the user's sight angle. For example, when the angle between the user's sight angle and the demonstration area of ​​the first sign language action is greater than 30 degrees and the duration exceeds 5 seconds, or when the user's sight angle frequently deviates from the demonstration area, it is determined that the user's concentration is low. When the concentration level is lower than a preset concentration level threshold, the sign language demonstration component is controlled to pause the demonstration. After the demonstration is paused, the user can be reminded to concentrate through voice prompts or flashing indicator lights.

[0108] In the embodiment of the present application, by collecting the user's expression and sight angle in real time, the user's immediate reaction in the process of learning sign language can be timely understood. By using image capture and feature extraction technology, the user's facial features can be accurately analyzed. The expression and sight angle information helps to more accurately judge the user's learning situation. The demonstration speed is adjusted in time according to the user's expression, and personalized teaching is achieved. When the user has expressions such as confusion, slowing down the demonstration speed can give the user more time to understand and digest complex sign language movements, meet the learning needs of different users, and adjust the speed according to the user's confusion points, so that the user can better keep up with the teaching rhythm and reduce the frustration caused by not keeping up, thereby improving the effect and enthusiasm of learning sign language. Controlling the pause and continuation of the demonstration according to the user's concentration level increases the interaction with the user. When the user is not focused, the demonstration is paused in time and the user is reminded, so that the user can feel the humanoid robot's attention to its learning status, improve the user's participation, avoid continuing the demonstration when the user is not focused, save teaching time, and ensure that the user receives sign language information in a focused state, thereby improving learning efficiency.

[0109] Optionally, the method may further include the following steps:

[0110] Step S701: if the sign language accuracy is less than or equal to the first preset accuracy threshold, determine a third demonstration speed of the first sign language action based on the sign language accuracy, and control the humanoid robot to demonstrate the first sign language action through the sign language demonstration component again;

[0111] Step S702: if the sign language accuracy is greater than a first preset accuracy threshold and less than a second preset accuracy threshold, determine the accuracy of at least one sub-action based on the sign language accuracy to obtain the accuracy of at least one sub-action, and the first preset accuracy threshold is less than the second preset accuracy threshold;

[0112] Step S703: determining at least one candidate sub-action based on the accuracy of at least one sub-action;

[0113] Step S704: determining a demonstration speed of at least one candidate sub-action based on the accuracy of at least one sub-action, and obtaining at least one fourth demonstration speed;

[0114] Step S705: Based on at least one fourth demonstration speed, the humanoid robot is controlled to demonstrate at least one candidate sub-action through a sign language demonstration component.

[0115] In a possible embodiment, when the user's sign language accuracy is less than or equal to the first preset accuracy threshold, it means that the user's overall mastery of the first sign language action is poor. At this time, it is necessary to adjust the demonstration speed of the first sign language action according to the sign language accuracy. A first mapping relationship between the sign language accuracy and the third demonstration speed can be established, and then the third demonstration speed is determined according to the first mapping relationship. When the user's sign language accuracy is greater than the first preset accuracy threshold and less than the second preset accuracy threshold, it means that the user has a certain understanding of the first sign language action, but there are still some problems. At this time, the accuracy of each sub-action is determined, and a second mapping relationship between the sub-action accuracy and the fourth demonstration speed is established, and then the demonstration speed of the candidate sub-action is determined according to the second mapping relationship.

[0116] In the embodiment of the present application, the accuracy rate of sign language and the accuracy rate of sub-movements are analyzed to determine the sign language movements that need to be demonstrated again and the corresponding demonstration speeds, thereby focusing on the specific problem points of the user, avoiding wasting time on the parts that the user has already mastered well, and improving teaching efficiency.

[0117] Optionally, the method may further include the following steps:

[0118] Step S801: when the target scenario mode is the question-and-answer mode, obtaining the user's question content;

[0119] Step S802: Determine the reply content corresponding to the question content;

[0120] Step S803: determining a third sign language action corresponding to the reply content;

[0121] Step S804: adjusting the first position, the first posture, and the first orientation of the humanoid robot based on the third sign language action to obtain a third position, a third posture, and a third orientation, so that the humanoid robot is in the third position, the third posture, and the third orientation to interact with the user;

[0122] Step S805: Control the humanoid robot to perform a third sign language action through the sign language demonstration component to express the reply content.

[0123] In a possible embodiment, the type of the question content may be at least one of a text type, a voice type, a sign language type, and a lip reading type. When determining the third sign language action corresponding to the reply content, the target text of the reply content that requires a sign language demonstration is determined, and a sign language learning type parameter set by the user is obtained. The sign language learning type parameter is used to determine the type of sign language the user expects to learn. The sign language type includes one of a grammatical sign language, a universal sign language, and a natural sign language. The target text is decomposed to obtain at least one first decomposed word, and the word order of each first decomposed word in the at least one first decomposed word is determined to obtain at least one first word order. The at least one first word order is used to represent the order of each first decomposed word in the at least one first decomposed word. The grammatical feature corresponding to the sign language learning type parameter is obtained, and the at least one first decomposed word and the at least one first word order are adjusted based on the grammatical feature to obtain at least one second decomposed word and at least one second word order. , based on at least one second word order, determine a first comparison order of at least one second decomposed word, based on the first comparison order, compare at least one second decomposed word in a preset sign language teaching content database in turn, obtain the gesture action and body posture action corresponding to each second decomposed word in at least one second decomposed word, obtain at least one gesture action and at least one body posture action, determine the facial expression action feature value and the lip movement feature value based on the sign language learning type parameter, the facial expression action feature value and the lip movement feature value are respectively used to determine the degree of change of the facial expression action and the lip movement, determine the facial expression action and the lip movement corresponding to each gesture action in at least one gesture action based on the facial expression action feature value and the lip movement feature value, obtain at least one facial expression action and at least one lip movement, respectively integrate at least one gesture action, at least one body posture action, at least one facial expression action and at least one lip movement, and obtain a third sign language action.

[0124] In the embodiment of the present application, the third sign language action is determined according to the reply content, which ensures the correctness of the sign language expression, enables the user to receive action information that conforms to the sign language specification, avoids the difficulty of understanding caused by inaccurate sign language actions, and improves the accuracy of information transmission. And by adjusting the position, posture and orientation of the humanoid robot, the interaction between the humanoid robot and the user is made more natural and comfortable.

[0125] In summary, in the implementation mode of the present application, the user's sign language teaching instruction is first obtained, and in response to the sign language teaching instruction, the first relative position information between the user and the humanoid robot is determined through the visual module, and the initial position, initial posture and initial orientation of the humanoid robot are adjusted based on the first relative position information to obtain the first position, the first posture and the first orientation, so that the humanoid robot is in the first position, the first posture and the first orientation to interact with the user, and then, the target scene mode is obtained, and when the target scene mode is the teaching mode, the target learning content is obtained, and the first sign language action corresponding to the target learning content is determined, and based on the first The sign language action adjusts the first position, the first posture and the first orientation of the humanoid robot to obtain the second position, the second posture and the second orientation, so that the humanoid robot is in the second position, the second posture and the second orientation to interact with the user. Then, the humanoid robot is controlled to demonstrate the first sign language action through the sign language demonstration component to guide the user to follow the humanoid robot to learn the first sign language action, and the user's second sign language action is obtained through the visual module, and the similarity between the first sign language action and the second sign language action is determined to obtain the first similarity. The sign language accuracy of the second sign language action is determined based on the first similarity, and the sign language accuracy is fed back to the user. Thus, by adjusting the position, posture and orientation of the humanoid robot, obtaining the target learning content in the teaching mode and determining the first sign language action corresponding to the target learning content, and then demonstrating the first sign language action through the sign language demonstration component, and feeding back the user's sign language accuracy to the user, personalized sign language teaching can be achieved according to different users, and the user's sign language learning effect and sign language learning efficiency are improved.

[0126] The method of the embodiment of the present invention is described in detail above, and the device of the embodiment of the present invention is provided below.

[0127] See also Figure 4 , Figure 4 1 is a schematic diagram of a humanoid robot-based sign language teaching device according to an embodiment of the present application. The humanoid robot-based sign language teaching device 900 is applied to the humanoid robot, the humanoid robot includes a visual module and a sign language demonstration component, and the humanoid robot-based sign language teaching device 900 includes an acquisition unit 901 and a processing unit 902;

[0128] An acquisition unit 901 is used to acquire a user's sign language teaching instruction;

[0129] The processing unit 902 is used to determine the first relative position information between the user and the humanoid robot through the visual module in response to the sign language teaching instruction;

[0130] Adjusting the initial position, initial posture and initial orientation of the humanoid robot based on the first relative position information to obtain a first position, a first posture and a first orientation, so that the humanoid robot is in the first position, the first posture and the first orientation to interact with the user;

[0131] Get the target scene mode;

[0132] When the target scenario mode is the teaching mode, the target learning content is obtained;

[0133] Determine the first sign language movement that corresponds to the target learning content;

[0134] Adjusting a first position, a first posture, and a first orientation of the humanoid robot based on the first sign language action to obtain a second position, a second posture, and a second orientation, so that the humanoid robot is in the second position, the second posture, and the second orientation to interact with the user;

[0135] Controlling the humanoid robot to demonstrate a first sign language action through a sign language demonstration component to guide the user to follow the humanoid robot to learn the first sign language action, and obtaining the user's second sign language action through a visual module;

[0136] Determine the similarity between the first sign language action and the second sign language action to obtain a first similarity;

[0137] The sign language accuracy of the second sign language action is determined based on the first similarity, and the sign language accuracy is fed back to the user.

[0138] In a possible embodiment, in adjusting the initial position, initial posture, and initial orientation of the humanoid robot based on the first relative position information to obtain the first position, the first posture, and the first orientation, the processing unit 902 is specifically configured to:

[0139] determining a current location of the user based on the first relative location information;

[0140] Acquire the current position and current posture of the user, and first obstacle information between the initial position and the current position through the visual module, where the first obstacle information is used to reflect the spatial distribution of the obstacles;

[0141] Determining a first position and a first orientation based on the first obstacle information, a preset distance relationship, a preset orientation relationship, a current position and a current orientation;

[0142] Determining a first posture based on a relationship between a current posture and a preset posture;

[0143] If the initial position is inconsistent with the first position, determining a moving path based on the initial position, the first position and the first obstacle information;

[0144] If the initial posture is inconsistent with the first posture, determining a vertical offset based on the initial posture and the first posture;

[0145] If the initial orientation is inconsistent with the first orientation, determining a rotation direction and a rotation angle based on the initial orientation and the first orientation;

[0146] The initial position, initial posture and initial orientation are adjusted based on the moving path, the vertical offset, the rotation direction and the rotation angle to obtain a first position, a first posture and a first orientation.

[0147] In a possible embodiment, in adjusting the first position, the first posture, and the first orientation of the humanoid robot based on the first sign language action to obtain the second position, the second posture, and the second orientation, the processing unit 902 is specifically configured to:

[0148] Obtaining the motion trajectory of the first sign language motion;

[0149] Determine the key points and demonstration range of the first sign language action based on the action trajectory;

[0150] Determine the demonstration orientation and the demonstration posture based on the action points, the current orientation and the current posture;

[0151] Determining a demonstration distance based on the demonstration amplitude and the current position;

[0152] Determine the demonstration space based on the motion trajectory;

[0153] Acquire the second obstacle information in the demonstration space through the visual module;

[0154] Determine the demonstration position based on the demonstration distance, the motion trajectory and the second obstacle information;

[0155] Based on the demonstration position, the demonstration posture and the demonstration orientation, the first position, the first posture and the first orientation are adjusted respectively to obtain the second position, the second posture and the second orientation.

[0156] In a possible embodiment, in controlling the humanoid robot to demonstrate the first sign language action through the sign language demonstration component, the processing unit 902 is specifically configured to:

[0157] Get the user's sign language learning progress;

[0158] Determining the user's sign language learning level based on the sign language learning progress;

[0159] determining a first complexity level of a first sign language movement based on the sign language learning level;

[0160] If the first complexity is higher than a preset complexity threshold, decomposing the first sign language action to obtain at least one sub-action;

[0161] Determine the complexity of each sub-action in at least one sub-action based on the sign language learning level to obtain at least one second complexity level;

[0162] Determine a demonstration speed of at least one sub-action based on the second complexity level to obtain at least one first demonstration speed;

[0163] Based on the demonstration order of at least one sub-action and at least one first demonstration speed, the humanoid robot is controlled to demonstrate at least one sub-action through the sign language demonstration component to complete the demonstration of the first sign language action.

[0164] In a possible embodiment, in acquiring the second sign language action of the user through the visual module, the processing unit 902 is specifically configured to:

[0165] Get the demonstration start time and demonstration end time of the first sign language action;

[0166] Delayed testing time is determined based on the level of sign language learning;

[0167] Determine a target detection time period based on a demonstration start time, a demonstration end time, and a delayed detection time;

[0168] Adjust the initial detection angle of the visual module based on the action points to obtain the target detection angle;

[0169] Based on the target detection angle, the user's sign language video is collected during the target detection period through the visual module;

[0170] Determine a starting action feature and an ending action feature based on the first sign language action;

[0171] A second sign language action is determined based on the sign language video, the start action feature, and the end action feature.

[0172] In a possible embodiment, the processing unit 902 is further configured to:

[0173] When demonstrating the first sign language action, the user's expression and the user's sight angle are obtained through the visual module;

[0174] Determine a first expression label corresponding to the user's expression;

[0175] When the first expression tag is consistent with the preset expression tag, obtaining the target sub-action, adjusting the first demonstration speed corresponding to the target sub-action to obtain the second demonstration speed, and demonstrating the target sub-action based on the second demonstration speed;

[0176] Determine the user's focus based on their line of sight;

[0177] When the concentration level is lower than a preset concentration level threshold, the sign language demonstration component is controlled to pause the demonstration.

[0178] In a possible embodiment, the processing unit 902 is further configured to:

[0179] If the sign language accuracy is less than or equal to the first preset accuracy threshold, determine a third demonstration speed of the first sign language action based on the sign language accuracy, and control the humanoid robot to demonstrate the first sign language action through the sign language demonstration component again;

[0180] If the sign language accuracy is greater than a first preset accuracy threshold and less than a second preset accuracy threshold, determine the accuracy of at least one sub-action based on the sign language accuracy to obtain the accuracy of at least one sub-action, and the first preset accuracy threshold is less than the second preset accuracy threshold;

[0181] determining at least one candidate sub-action based on the accuracy rate of at least one sub-action;

[0182] Determine a demonstration speed of at least one candidate sub-action based on the accuracy of at least one sub-action, and obtain at least one fourth demonstration speed;

[0183] Based on at least one fourth demonstration speed, the humanoid robot is controlled to demonstrate at least one candidate sub-action through a sign language demonstration component.

[0184] In a possible embodiment, the processing unit 902 is further configured to:

[0185] When the target scenario mode is the question-and-answer mode, obtain the user's question content;

[0186] Determine the response content corresponding to the question content;

[0187] determining a third sign language action corresponding to the response content;

[0188] adjusting the first position, the first posture, and the first orientation of the humanoid robot based on the third sign language action to obtain a third position, a third posture, and a third orientation, so that the humanoid robot is in the third position, the third posture, and the third orientation to interact with the user;

[0189] The humanoid robot is controlled to perform a third sign language action through the sign language demonstration component to express the reply content.

[0190] The humanoid robot provided in the embodiment of the present application also includes a processor and a memory. The processor is connected to the memory. The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory.

[0191] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement part or all of the steps of any one of the methods for implementing sign language teaching based on a humanoid robot as recorded in the above method embodiments.

[0192] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps of any one of the humanoid robot-based sign language teaching methods recorded in the above method embodiments.

[0193] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0194] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0195] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the modules, which is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or module can be electrical or other forms.

[0196] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0197] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software program modules.

[0198] If the integrated module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0199] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A sign language teaching method based on a humanoid robot, characterized in that: Applied to a humanoid robot, the humanoid robot includes a visual module and a sign language demonstration component; the method includes: Obtain the user's sign language teaching instructions; In response to the sign language teaching instruction, determining first relative position information between the user and the humanoid robot through the visual module; Adjusting the initial position, initial posture and initial orientation of the humanoid robot based on the first relative position information to obtain a first position, a first posture and a first orientation, so that the humanoid robot is in the first position, the first posture and the first orientation to interact with the user; Get the target scene mode; When the target scenario mode is a teaching mode, obtaining target learning content; Determining a first sign language action corresponding to the target learning content; Adjusting the first position, the first posture, and the first orientation of the humanoid robot based on the first sign language action to obtain a second position, a second posture, and a second orientation, so that the humanoid robot is in the second position, the second posture, and the second orientation to interact with the user; Controlling the humanoid robot to demonstrate the first sign language action through the sign language demonstration component to guide the user to follow the humanoid robot to learn the first sign language action, and obtaining the user's second sign language action through the visual module; Determine the similarity between the first sign language action and the second sign language action to obtain a first similarity; The sign language accuracy of the second sign language action is determined based on the first similarity, and the sign language accuracy is fed back to the user.

2. The method according to claim 1, characterized in that The step of adjusting the initial position, initial posture and initial orientation of the humanoid robot based on the first relative position information to obtain a first position, a first posture and a first orientation includes: determining a current location of the user based on the first relative location information; Acquire the current position and current posture of the user, and first obstacle information between the initial position and the current position through the visual module, where the first obstacle information is used to reflect the spatial distribution of obstacles; Determine the first position and the first orientation based on the first obstacle information, a preset distance relationship, a preset orientation relationship, the current position and the current orientation; Determining the first posture based on the current posture and a preset posture relationship; If the initial position is inconsistent with the first position, determining a moving path based on the initial position, the first position and the first obstacle information; If the initial posture is inconsistent with the first posture, determining a vertical offset based on the initial posture and the first posture; If the initial orientation is inconsistent with the first orientation, determining a rotation direction and a rotation angle based on the initial orientation and the first orientation; The initial position, the initial posture, and the initial orientation are adjusted based on the moving path, the vertical offset, the rotation direction, and the rotation angle to obtain the first position, the first posture, and the first orientation.

3. The method according to claim 2, characterized in that The adjusting the first position, the first posture and the first orientation of the humanoid robot based on the first sign language action to obtain a second position, a second posture and a second orientation includes: Acquire a motion trajectory of the first sign language motion; Determine the action key points and demonstration range of the first sign language action based on the action trajectory; Determine a demonstration position and a demonstration posture based on the action points, the current position and the current posture; determining a demonstration distance based on the demonstration amplitude and the current position; determining a demonstration space based on the motion trajectory; Acquiring second obstacle information in the demonstration space through the visual module; Determining a demonstration position based on the demonstration distance, the motion trajectory and the second obstacle information; The first position, the first posture and the first orientation are adjusted respectively based on the demonstration position, the demonstration posture and the demonstration orientation to obtain the second position, the second posture and the second orientation.

4. The method according to claim 3, characterized in that The controlling the humanoid robot to demonstrate the first sign language action through the sign language demonstration component comprises: Obtaining the user's sign language learning progress; determining the user's sign language learning level based on the sign language learning progress; determining a first complexity level of the first sign language action based on the sign language learning level; If the first complexity level is higher than a preset complexity level threshold, decomposing the first sign language action to obtain at least one sub-action; Determine the complexity of each sub-action in the at least one sub-action based on the sign language learning level to obtain at least one second complexity level; Determine a demonstration speed of the at least one sub-action based on the second complexity level to obtain at least one first demonstration speed; Based on the demonstration order of the at least one sub-action and the at least one first demonstration speed, the humanoid robot is controlled to demonstrate the at least one sub-action through the sign language demonstration component to complete the demonstration of the first sign language action.

5. The method according to claim 4, characterized in that The acquiring the second sign language action of the user through the visual module includes: Obtaining a demonstration start time and a demonstration end time of the first sign language action; determining a delayed detection time based on the sign language learning level; Determining a target detection time period based on the demonstration start time, the demonstration end time, and the delayed detection time; Adjusting the initial detection angle of the visual module based on the action points to obtain a target detection angle; Based on the target detection angle, collecting the user's sign language video within the target detection time period through the visual module; Determine a starting action feature and an ending action feature based on the first sign language action; The second sign language action is determined based on the sign language video, the starting action feature and the ending action feature.

6. The method according to claim 4 or 5, characterized in that The method further comprises: When demonstrating the first sign language action, obtaining the user's expression and the user's sight angle through the visual module; Determine a first expression label corresponding to the user expression; When the first expression tag is consistent with the preset expression tag, obtaining a target sub-action, adjusting a first demonstration speed corresponding to the target sub-action to obtain a second demonstration speed, and demonstrating the target sub-action based on the second demonstration speed; Determining the user's concentration level based on the user's sight angle; When the concentration level is lower than a preset concentration level threshold, the sign language demonstration component is controlled to pause the demonstration.

7. The method according to claim 6, characterized in that The method further comprises: If the sign language accuracy rate is less than or equal to a first preset accuracy rate threshold, determining a third demonstration speed of the first sign language action based on the sign language accuracy rate, and controlling the humanoid robot to demonstrate the first sign language action through the sign language demonstration component again; If the sign language accuracy is greater than the first preset accuracy threshold and less than the second preset accuracy threshold, determining the accuracy of the at least one sub-action based on the sign language accuracy to obtain the at least one sub-action accuracy, and the first preset accuracy threshold is less than the second preset accuracy threshold; Determine at least one candidate sub-action based on the at least one sub-action accuracy; Determine the presentation speed of the at least one candidate sub-action based on the accuracy of the at least one sub-action, and obtain at least one fourth presentation speed; Based on the at least one fourth demonstration speed, the humanoid robot is controlled to demonstrate the at least one candidate sub-action through the sign language demonstration component.

8. The method according to claim 7, characterized in that The method further comprises: When the target scenario mode is a question-and-answer mode, obtaining the question content of the user; Determine the reply content corresponding to the question content; Determining a third sign language action corresponding to the response content; adjusting the first position, the first posture, and the first orientation of the humanoid robot based on the third sign language action to obtain a third position, a third posture, and a third orientation, so that the humanoid robot is in the third position, the third posture, and the third orientation to interact with the user; The humanoid robot is controlled to perform the third sign language action through the sign language demonstration component to express the reply content.

9. A sign language teaching device based on a humanoid robot, characterized in that: Applied to a humanoid robot, the humanoid robot comprises a visual module and a sign language demonstration component; the sign language teaching device based on the humanoid robot comprises an acquisition unit and a processing unit; The acquisition unit is used to acquire the user's sign language teaching instructions; The processing unit is used to determine the first relative position information between the user and the humanoid robot through the visual module in response to the sign language teaching instruction; Adjusting the initial position, initial posture and initial orientation of the humanoid robot based on the first relative position information to obtain a first position, a first posture and a first orientation, so that the humanoid robot is in the first position, the first posture and the first orientation to interact with the user; Get the target scene mode; When the target scenario mode is a teaching mode, obtaining target learning content; Determining a first sign language action corresponding to the target learning content; Adjusting the first position, the first posture, and the first orientation of the humanoid robot based on the first sign language action to obtain a second position, a second posture, and a second orientation, so that the humanoid robot is in the second position, the second posture, and the second orientation to interact with the user; Controlling the humanoid robot to demonstrate the first sign language action through the sign language demonstration component to guide the user to follow the humanoid robot to learn the first sign language action, and obtaining the user's second sign language action through the visual module; Determine the similarity between the first sign language action and the second sign language action to obtain a first similarity; The sign language accuracy of the second sign language action is determined based on the first similarity, and the sign language accuracy is fed back to the user.

10. A humanoid robot, characterized in that: include: A processor and a memory, the processor is connected to the memory, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the humanoid robot executes the method according to any one of claims 1 to 8.

Images