A robot, a robot interaction method, and a storage medium

Abstract

The present application provides a robot, a robot interaction method and a storage medium, including: obtaining user activity information and environmental information sensed by a sensing module; determining task content according to the user activity information and the environmental information; determining a projection assistance strategy according to the task content and the environmental information; and performing an assistance operation according to the projection assistance strategy through a projection module. The present application can identify the situation of the user and the situation of the environment and can perform real-time reasoning, decision-making and dynamic response according to the situations of the user and the environment to obtain the needs of the user and then determine the task content; determine a projection assistance strategy according to the task content and the environmental information to provide the required content for the user in a suitable manner, realize effective interaction with the real world, respond in real time to events occurring in the outside world, improve scene adaptability, and provide a more efficient and flexible projection interaction experience.

Description

Technical Field

[0001] This application belongs to the technical field of data processing, and particularly relates to a robot, a robot interaction method, and a storage medium. Background Art

[0002] Currently, the main function of projection devices is to display content, specifically, to display the content required by users based on pre-programmed fixed logical content. Projection devices generally only recognize simple instructions from users.

[0003] This makes it impossible for projection devices to effectively interact with the real world and respond in real time to events occurring in the outside world. Summary of the Invention

[0004] Embodiments of this application provide a robot, a robot interaction method, and a storage medium to solve the problem of being unable to effectively interact with the real world.

[0005] In a first aspect, embodiments of this application provide a robot interaction method, including:

[0006] Obtaining user activity information and environmental information sensed by a sensing module;

[0007] Determining task content based on the user activity information and the environmental information;

[0008] Determining a projection assistance strategy based on the task content and the environmental information;

[0009] Performing a projection operation through a projection module according to the projection assistance strategy.

[0010] In a second aspect, embodiments of this application provide a robot, including: a sensing module, an information processing module, and a projection module;

[0011] The sensing module is configured to sense user activity information and environmental information;

[0012] The information processing module is configured to determine task content based on the user activity information and the environmental information;

[0013] It is further configured to determine a projection assistance strategy based on the task content and the environmental information;

[0014] The projection module is configured to perform a projection operation according to the projection assistance strategy.

[0015] In a third aspect, embodiments of this application provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any item of the first aspect above is implemented.

[0016] Fourthly, an embodiment of the present application provides a computer program product. When the computer program product runs on a robot, the robot is enabled to execute the method described in any one of the first aspects above.

[0017] The beneficial effects of the embodiments of the present application compared with the prior art are as follows:

[0018] In the embodiments of the present application, by obtaining the user activity information and environmental information sensed by the sensing module; determining the task content according to the user activity information and environmental information; determining the projection assistance strategy according to the task content and environmental information; and executing the assistance operation according to the projection assistance strategy through the projection module, it is possible to identify the user's situation and the environmental situation and perform real-time reasoning, decision-making, and dynamic response according to the user and environmental situations, so as to obtain the user's needs and then determine the task content; determine the projection assistance strategy according to the task content and environmental information, and provide the required content for the user in a suitable manner, realize effective interaction with the real world, respond to events occurring in the outside world in real time, improve the scene adaptability, and provide a more efficient and flexible projection interaction experience.

[0019] It can be understood that the beneficial effects of the above second to fifth aspects can be referred to the relevant descriptions in the first aspect above, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0021] Figure 1 is the first flowchart of the robot interaction method provided by an embodiment of the present application;

[0022] Figure 2 is the second flowchart of the robot interaction method provided by an embodiment of the present application;

[0023] Figure 3 is the first example diagram of the application scenario provided by an embodiment of the present application;

[0024] Figure 4 is the second example diagram of the application scenario provided by an embodiment of the present application;

[0025] Figure 5 is the third example diagram of the application scenario provided by an embodiment of the present application;

[0026] Figure 6 is the fourth example diagram of the application scenario provided by an embodiment of the present application;

[0027] Figure 7 It is the fifth example diagram of the application scenario provided by an embodiment of the present application;

[0028] Figure 8 It is the sixth example diagram of the application scenario provided by an embodiment of the present application;

[0029] Figure 9 It is the seventh example diagram of the application scenario provided by an embodiment of the present application;

[0030] Figure 10 It is the eighth example diagram of the application scenario provided by an embodiment of the present application;

[0031] Figure 11 It is the ninth example diagram of the application scenario provided by an embodiment of the present application;

[0032] Figure 12 It is the tenth example diagram of the application scenario provided by an embodiment of the present application;

[0033] Figure 13 It is the eleventh example diagram of the application scenario provided by an embodiment of the present application;

[0034] Figure 14 It is the twelfth example diagram of the application scenario provided by an embodiment of the present application;

[0035] Figure 15 It is the thirteenth example diagram of the application scenario provided by an embodiment of the present application;

[0036] Figure 16 It is the fourteenth example diagram of the application scenario provided by an embodiment of the present application;

[0037] Figure 17 It is the fifteenth example diagram of the application scenario provided by an embodiment of the present application;

[0038] Figure 18 It is the sixteenth example diagram of the application scenario provided by an embodiment of the present application;

[0039] Figure 19 It is the seventeenth example diagram of the application scenario provided by an embodiment of the present application;

[0040] Figure 20 It is the eighteenth example diagram of the application scenario provided by an embodiment of the present application;

[0041] Figure 21 It is the nineteenth example diagram of the application scenario provided by an embodiment of the present application;

[0042] Figure 22 It is the twentieth example diagram of the application scenario provided by an embodiment of the present application;

[0043] Figure 23 It is the twenty - first example diagram of the application scenario provided by an embodiment of the present application;

[0044] Figure 24 It is the twenty - second example diagram of the application scenario provided by an embodiment of the present application;

[0045] Figure 25 It is the twenty - third example diagram of the application scenario provided by an embodiment of the present application;

[0046] Figure 26 It is the twenty - fourth example diagram of the application scenario provided by an embodiment of the present application;

[0047] Figure 27 It is the twenty - fifth example diagram of the application scenario provided by an embodiment of the present application;

[0048] Figure 28 It is the twenty - sixth example diagram of the application scenario provided by an embodiment of the present application;

[0049] Figure 29 It is the twenty - seventh example diagram of the application scenario provided by an embodiment of the present application;

[0050] Figure 30 It is the twenty - eighth example diagram of the application scenario provided by an embodiment of the present application;

[0051] Figure 31 It is the twenty - ninth example diagram of the application scenario provided by an embodiment of the present application;

[0052] Figure 32 It is the thirtieth example diagram of the application scenario provided by an embodiment of the present application;

[0053] Figure 33 It is the thirty - first example diagram of the application scenario provided by an embodiment of the present application;

[0054] Figure 34 It is the thirty - second example diagram of the application scenario provided by an embodiment of the present application;

[0055] Figure 35 It is the thirty - third example diagram of the application scenario provided by an embodiment of the present application;

[0056] Figure 36 It is the thirty - fourth example diagram of the application scenario provided by an embodiment of the present application;

[0057] Figure 37 It is the thirty - fifth example diagram of the application scenario provided by an embodiment of the present application;

[0058] Figure 38It is the thirty-sixth example diagram of the application scenario provided by an embodiment of the present application;

[0059] Figure 39 It is the thirty-seventh example diagram of the application scenario provided by an embodiment of the present application;

[0060] Figure 40 It is the thirty-eighth example diagram of the application scenario provided by an embodiment of the present application;

[0061] Figure 41 It is the thirty-ninth example diagram of the application scenario provided by an embodiment of the present application;

[0062] Figure 42 It is the fortieth example diagram of the application scenario provided by an embodiment of the present application;

[0063] Figure 43 It is the forty-first example diagram of the application scenario provided by an embodiment of the present application;

[0064] Figure 44 It is the forty-second example diagram of the application scenario provided by an embodiment of the present application;

[0065] Figure 45 It is the forty-third example diagram of the application scenario provided by an embodiment of the present application;

[0066] Figure 46 It is the forty-fourth example diagram of the application scenario provided by an embodiment of the present application;

[0067] Figure 47 It is the forty-fifth example diagram of the application scenario provided by an embodiment of the present application;

[0068] Figure 48 It is the forty-sixth example diagram of the application scenario provided by an embodiment of the present application;

[0069] Figure 49 It is the forty-seventh example diagram of the application scenario provided by an embodiment of the present application;

[0070] Figure 50 It is the forty-eighth example diagram of the application scenario provided by an embodiment of the present application;

[0071] Figure 51 It is the forty-ninth example diagram of the application scenario provided by an embodiment of the present application;

[0072] Figure 52 It is the fiftieth example diagram of the application scenario provided by an embodiment of the present application;

[0073] Figure 53 It is the fifty-first example diagram of the application scenario provided by an embodiment of the present application;

[0074] Figure 54 It is the first schematic structural diagram of the robot provided by an embodiment of the present application;

[0075] Figure 55 It is the second schematic structural diagram of the robot provided by an embodiment of the present application;

[0076] Figure 56 It is the third schematic structural diagram of the robot provided by an embodiment of the present application;

[0077] Figure 57 It is the fourth schematic structural diagram of the robot provided by an embodiment of the present application. Detailed implementation manners

[0078] In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0079] It should be understood that when used in the specification and the appended claims of the present application, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0080] It should also be understood that the term "and / or" as used in the specification and the appended claims of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0081] As used in the specification and the appended claims of the present application, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" according to the context. Similarly, the phrase "if determined" or "if detecting [the described condition or event]" can be interpreted as meaning "once determined", "in response to determining", "once detecting [the described condition or event]", or "in response to detecting [the described condition or event]" according to the context.

[0082] In addition, in the description of the specification and the appended claims of the present application, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0083] References to "one embodiment" or "some embodiments" or the like described in the specification of the present application mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the present application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprising", "including", "having" and their variants mean "including but not limited to", unless otherwise specifically emphasized.

[0084] In one embodiment, refer to Figure 1 , the method is applied to an information processing module of a robot, and the robot includes a sensing module, an information processing module, and a projection module.

[0085] Specifically, it includes:

[0086] S11: Obtain user activity information and environmental information sensed by the sensing module.

[0087] In an application, the user activity information includes user language information and user physical state information. Among them, the user language information includes user conversation information (for example: conversation between the user and the robot, conversation between the user and others), user voice information (conversation of the user with himself), user tone information, and the user physical state information includes the position of gaze, gestures, body movements, expressions, etc. The environmental information includes temperature, projection screen, time, location, light, sound, the position of the user and the physical and optical characteristic information, state information, position, direction of each object, as well as environmental change information, etc. The state information of an object includes static change information and dynamic change information. The dynamic change information can be determined according to the change situation of the object before and after.

[0088] Exemplarily, the state information of an object includes the cooking degree of food, the cleanliness of clothes, the position movement of a remote control, items stored inside an associated device (for example, there is milk stored inside a refrigerator in the environment), static or dynamic text symbol graphics on the surface of an object, the surface material of an object, etc.

[0089] It can be understood that the sensing of user activity information is carried out after obtaining the consent of the user. The execution time and sequence of the steps of sensing user activity information and sensing environmental information are not restricted.

[0090] It should be noted that the obtained user activity information and environmental information are real-time information at the current moment.

[0091] S12: Determine the task content according to the user activity information and the environmental information.

[0092] In a possible implementation, the task content includes task planning information, answer information, information related to the first goal, information related to the second goal, information on relevant visual elements, and work status information. That is, when at least one of the task planning information, answer information, information related to the first goal, information related to the second goal, and information on relevant visual elements is determined according to the user's needs, the task content correspondingly includes this information.

[0093] Step S12 includes:

[0094] S121: Determine the user's intention based on the user activity information and environmental information.

[0095] In a possible implementation, step S121 includes:

[0096] S21: Determine the user behavior information and user mental state information based on the user activity information.

[0097] S22: Determine the interaction between the user and the environment based on the user activity information and environmental information.

[0098] In an application, by combining the user's physical state information and environmental information, determine the interaction between the user and the environment and / or the interaction behavior between the user and objects or people in the environment, and obtain the interaction between the user and the environment. Among them, the interaction situation characterizes the impact of the current environment on the user and the impact of the environmental changes before and after on the user.

[0099] Exemplarily, the interaction situation includes the user gazing at an object, the user picking up an object, the user operating an object, etc. The user behavior information includes the user moving, the user traveling, etc. The user mental state information includes the user's facial expression, micro-movements, and the clues of mental activities contained in the user's language, etc.

[0100] S23: Determine the user's intention based on at least one of the interaction situation, user behavior information, user mental state information, and user language information.

[0101] In an application, when at least one of the interaction situation, user behavior information, user mental state information, and user language information is obtained, perform reasoning and analysis on this information to determine the user's intention.

[0102] In a possible implementation, the information can be reasoned and analyzed based on an analysis algorithm including a pre-trained artificial intelligence model to understand the user's intention and learn the user's preference requirements, and thus obtain the user's intention.

[0103] S122: Determine the user's needs based on the user's intention, scene information, and context information.

[0104] In an application, based on the user's intention, combined with scenario information and context information, the user's needs are determined.

[0105] In a possible implementation, an analysis algorithm including a pre-trained artificial intelligence model can be used to analyze the user's intention, scenario information, and context information to determine the user's needs.

[0106] Among them, the scenario information is determined according to the environmental information. The context information is determined according to the scenario information, user activity information, and environmental information. The context information includes various information related to the scenario, environment, and user, and may include various scenario information, environmental information of various scenarios, historical behavior information of users in various scenarios (such as: users' daily habits, preference records, recent activities, etc.), immediate perception behaviors (such as: users' actions, expressions, tones, etc.), interaction information with the environment or others (such as: conversations between users and others, the number of people around the user), etc.

[0107] S123: Determine at least one of the task planning information, answer information, information related to the first target, information related to the second target, and information related to relevant visual elements according to the user's needs.

[0108] Among them, the first target is a static object within the user's attention range. A static object refers to an object that does not move during the interaction between the robot and the outside world. The second target is an object in a moving state within the user's attention range. The moving state refers to a state in which continuous or short-term movement occurs during the interaction between the robot and the outside world. The second target can also be determined through user activity information.

[0109] Exemplarily, the second target can be an object held by the user, an object held by the robot, an object worn by the user, an object worn by the robot, or an object held or worn by other people in the environment, or other objects in a moving state in the environment.

[0110] In an application, task planning information is generated according to the user's needs. Since the robot will interact with the outside world in real time after entering the working state and can interact with the user when it detects that the user's needs include needs related to task content, the task planning information can be either initially generated task planning or adjusted task planning.

[0111] Determine information related to the first target and / or the second target according to the user's needs. The related information includes information to be supplemented and information to be enhanced. For example: information that the user may potentially be interested in or need, relevant explanations, extended information, etc. The information related to the first target and / or the second target can improve the user's understanding of the task scenario and meet the user's needs for the attention target.

[0112] Determine answer information related to the needs according to the user's needs to meet the user's needs.

[0113] Determine the visual elements related to the task according to the user's requirements, and infer the information of the relevant visual elements. The information of the relevant visual elements includes which visual elements need to appear in the current projection screen, and how to perform transformation operations such as arrangement, combination, scaling, and deformation on these relevant visual elements for the current task, as well as information such as which visual elements need to be generated or pre-loaded in advance. Determine how to perform operations such as arrangement, combination, transformation, scaling, and deformation on these relevant visual elements according to the information of the relevant visual elements, and finally form the visualized content in the projection screen.

[0114] Among them, visual elements refer to graphics and images generated by the program, or pre-stored locally, or in networked devices. Relevant visual elements include icons, text, videos, animations, color coding, spatial positioning diagrams, charts, simulated physical objects, borders, dividing lines, background images, texture images, 3D models, light and shadow maps, 3D scenes, etc. Relevant visual elements are the basic visualized materials used to constitute the visualized content (including the first visualized content and / or the second visualized content).

[0115] For example, icons include small symbols such as arrows, etc., used to represent specific functions, objects or states. Text includes text prompts, labels, etc. Color coding includes black, red, etc., used to distinguish categories and display states. Simulated physical objects include projected simulated color palettes, digital simulations of physical entities, projected virtual clothing, etc.

[0116] S124: Generate working state information according to the working state of the robot.

[0117] In the application, generate working state information according to the working state of the robot. The working state information includes the execution progress of the task, resource usage, etc.

[0118] It can be understood that by generating the working state information, the user can understand the situation of the robot performing tasks through the working state information, which can enhance the interpretability in the cooperation between the robot and the user and the environment, facilitate the user to understand and intervene in a timely manner, maintain the awareness of the robot's decision-making, realize closer human-machine interaction and cooperation, and enhance the adaptability and intelligence of the interaction. The subsequent display of the working state information is not limited in form and can be displayed in a multi-modal manner.

[0119] For example, the working state information includes that the sorting of clothes has been completed, the current task is to classify and put the clothes into the washing machine, and the next task is to add bleach and adjust the washing machine to the bleach washing mode.

[0120] S13: Determine the projection assistance strategy according to the task content and environmental information.

[0121] In a possible implementation, the projection assistance strategy includes a projection position, a projection angle, a first visualization content, a second visualization content, and a target visual element.

[0122] Step S13 includes:

[0123] S131: Based on a preset rule, determine a projection method according to the task content and environmental information.

[0124] Among them, the projection method includes a display method, a projection content placement method, and projectable surface information.

[0125] Among them, the preset rules include the basic cognitive rules of people, the spatial and logical relationships of objects in the environment, the common sense rules of the physical world, the cultural and social common sense rules, and physical properties, etc.

[0126] The basic cognitive rules of people include a clear hierarchical structure or an intuitive graphical display method related to the cognitive characteristics of users, etc., so that complex and simple information can be presented in an easy-to-understand form. The spatial and logical relationships of objects in the environment include that important information is preferentially projected within the user's line of sight, important information is preferentially projected near the target, and how the spatial relationships of objects in the environment can be used for the superposition and presentation of projection information, etc., so that users can quickly obtain the core content. The common sense rules of the physical world include the basic usage methods of each object or the influence of the gravity direction on the position and angle of the projection content, etc. The cultural and social common sense rules include idiomatic expressions, customary colors, common symbols, cultural allusions, cultural customs, and cultural taboos in different cultural scenarios. Physical properties include the physical and optical characteristics of various surfaces (for example: color, texture, shape, contour, geometric structure, size, reflectivity, refractive index, etc.).

[0127] Using the preset rules, determine a display method and a projection content placement method suitable for users according to the state information and positions of objects in the task content and environmental information. And using the preset rules, analyze the physical and optical characteristics information, positions, and directions of each surface in the environmental information, and determine projectable surface information suitable for users. This projection surface can completely and clearly display the projection content.

[0128] Among them, suitable projection surfaces generally include walls, floors, ceilings, curtain walls formed by water mist, human or animal surfaces, object surfaces and / or inner cavities, or specific areas associated with objects.

[0129] It can be understood that selecting a suitable projection method ensures the efficient and accurate transmission of the projection content, enabling users to quickly obtain the projected information.

[0130] S132: Based on the projection method, determine the projection position and projection angle according to at least one of the position of the robot, the position of the user, the position of the first target, and the position of the second target.

[0131] Among them, the environmental information includes at least one of the position of the user, the position of the first target, and the position of the second target, and the position of the robot is determined by positioning.

[0132] It can be understood that when the environmental information includes the position of the user and the position of the first target, determine the projection position and projection angle according to the position of the robot, the position of the user, and the position of the first target. When the environmental information includes the position of the user and the position of the second target, determine the projection position and projection angle according to the position of the robot, the position of the user, and the position of the second target. When the environmental information includes the position of the user, the position of the first target, and the position of the second target, determine the projection position and projection angle according to the position of the robot, the position of the user, the position of the first target, and the position of the second target.

[0133] In the application, based on the projection method, during the movement of the robot, adjust the projection position and projection angle according to the position of the robot to maintain the stability of the projection screen and the clear and continuous presentation of the projection content during the movement.

[0134] Based on the projection method, when the user moves or switches positions in the scene, adjust the projection angle according to the position of the user so that the projection screen is in the direction suitable for the user to watch, and the projection content is within the best line of sight of the user.

[0135] Based on the projection method, when the second target moves or switches positions in the scene, adjust the projection position according to the position of the second target so that the projection content adapts to the movement trajectory of the second target and remains clearly visible.

[0136] It can be understood that the execution time and order of the above steps are not restricted. The obtained projection position and projection angle can provide a stable, clear, and easy-to-observe projection screen for the user.

[0137] S133: Determine the first visualization content according to the task content and context information.

[0138] In the application, analyze the task content and context information to determine the first visualization content. The first visualization content is used to display the task content, the background information related to the task content, the guiding information related to the task content, and the feedback information, so that the user can quickly and comprehensively obtain the required information through the first visualization content.

[0139] It is understandable that the first visualization content can completely and directly convey the task content and other information related to the task content, without relying on the first target and the second target. In addition, the first visualization content can be projected at a suitable position, without being restricted by the positions of the first target and the second target.

[0140] In a possible implementation, the task content can be analyzed and inferred based on an algorithm containing a pre-trained artificial intelligence model, and the first visualization content can be determined in combination with the context information.

[0141] For example, the first visualization content projected by the robot includes pictures and information of visually recommended products, promotional information of the store, etc. in the shopping guide scenario, background information of exhibits in the museum tour scenario, vital sign information and first aid operation guidelines in the first aid scenario, and video relaxation guidance in the auxiliary relaxation scenario.

[0142] S134: Determine the second visualization content according to at least one of the task content, context information, relevant information of the first target, and relevant information of the second target.

[0143] Among them, the second visualization content is used to project and enhance the first target and / or the second target. Specifically, based on the second visualization content, projection images are superimposed on the surface and adjacent areas of the first target and / or the second target, and users can obtain the required information through the combination of the second visualization content and the projected target.

[0144] It is understandable that the second visualization content is associated with the first target, the second target itself, and their positions, that is, associated with the projected object and its position. The second visualization content needs to be combined with the surface and adjacent areas of the first target and / or the second target to jointly present information, and depends on the first target and / or the second target.

[0145] In the application, at least one of the task content, context information, information to be supplemented, and information to be enhanced of the first target and the second target is analyzed to determine the second visualization content.

[0146] For example, in the tour scenario, the second visualization content projected by the robot includes visual highlighting of exhibits or posters, and display of missing parts of sculptures. In the teaching scenario, the second visualization content projected by the robot includes visual explanation information of teaching aids. In the shopping guide scenario, the second visualization content projected by the robot includes virtual clothes projected on the user.

[0147] S135: Determine the target visual element according to the first visualization content, the second visualization content, information of relevant visual elements, user activity information, and robot interaction status information.

[0148] Among them, the robot interaction status information is the information about the interaction between the robot and the outside world.

[0149] It can be understood that the working status information is coarse-grained information used to characterize the task execution situation of the robot. The robot interaction status information is fine-grained information used to characterize the specific situation of the robot's operation.

[0150] In the application, based on the information of relevant visual elements, analyze the first visualization content, the second visualization content, and the relevant visual element information, and select the target visual element to enable the user to better obtain the required information. Indicate or emphasize some content in the first visualization content and / or the second visualization content through the target visual element.

[0151] The target visual element is a visual element that guides the user's visual focus through dynamic changes in the projection screen and provides timely indication or emphasis on other visualization content. During the interaction between the robot and the user, the target visual element changes dynamically along with the robot's voice and body movement, as well as the interaction behavior between the user and the robot, guiding the user to focus on the information most relevant to the current interaction in the projection screen, making the communication and cooperation between the robot and the user smooth and natural.

[0152] Exemplarily, the target visual element may include a pointer, highlighting, color change, shape change, indicative dynamic effect, dynamic step prompt, interaction prompt, and other visual elements that can be used for indication or emphasis.

[0153] It can be understood that after the robot enters the working state, it will interact with the outside world in real time, loop through steps S11 to S12, and can adjust the task content in real time according to the content of the interaction with the user and the content of the interaction with the environment, so as to clarify the task planning information, that is, the specific goals and relevant information of the task, ensuring that the entire process can comprehensively and accurately understand the user's intention and needs, and providing accurate and comprehensive assistance to the user throughout the process. After the assistance for the current time ends, the robot will wait for the next user demand to trigger and enter the working state again.

[0154] S14: Execute the projection operation according to the projection assistance strategy through the projection module.

[0155] In a possible implementation manner, step S14 includes:

[0156] S141: Project the first visualization content and the target visual element at the projection position and at the projection angle through the projection module.

[0157] S142: Project the second visualization content and the target visual element at the position of the first target and / or the second target through the projection module.

[0158] In an application, the robot moves so that the projection screen is in the projection position and projects the first visualization content and the target visual element at a projection angle. And at the position of the first target (for example: the first target itself or near the first target, etc.) and / or the position of the second target (for example: the second target itself or near the second target, etc.), the second visualization content is projected.

[0159] It can be understood that when it comes to the relevant information of the first target, the second visualization content is projected at the position of the first target. When it comes to the relevant information of the second target, the second visualization content is projected at the position of the second target. When it comes to the relevant information of the first target and the second target, the second visualization content is projected at the positions of the first target and the second target.

[0160] For example, in a guided tour scenario, the poster of the exhibit is highlighted, and the missing part is projected and complemented at the missing position of the sculpture. In a teaching scenario, projection is superimposed on teaching aids. In a shopping guide scenario, virtual clothing is projected on the user.

[0161] In this embodiment, by obtaining the user activity information and environmental information sensed by the perception module; determining the task content according to the user activity information and environmental information; determining the projection assistance strategy according to the task content and environmental information; and executing the assistance operation according to the projection assistance strategy through the projection module, it can identify the user's situation and the environmental situation and can perform real-time reasoning, decision-making and dynamic response according to the user and environmental situations to obtain the user's needs and then determine the task content; determine the projection assistance strategy according to the task content and environmental information, and provide the required content for the user in a suitable way to achieve effective interaction with the real world, respond to events occurring in the outside world in real time, improve the scene adaptability and response ability, and provide a more efficient and flexible projection interaction experience.

[0162] It can be understood that by the robot based on the projection method, determining the appropriate projection position and projection angle according to each position, it can project the content completely in the appropriate area, so that the projection is not limited to a fixed projection range and can adapt to the needs of a large range or dynamic moving scenarios.

[0163] And by projecting the first visualization content, the target visual element and the second visualization content, a graphical interface that overlays and mixes with the objects and spaces in the real world is realized to interact with the user to clarify the user's intention and task, and solve the problem of inability to effectively interact with the real world.

[0164] In one embodiment, the robot further includes a positioning and navigation module, a communication module and a motion module, and the perception module includes a sensor module, a vision module and an I / O interaction module.

[0165] Please refer to Figure 2 , the method further includes:

[0166] S15: Determine the voice assistance strategy and action assistance strategy according to the task content.

[0167] In the application, analyze the task content to determine the voice assistance strategy and action assistance strategy.

[0168] Among them, the action assistance strategy is the physical assistance operation to assist the user in various scenarios. The physical assistance operation is to directly intervene in objects and spaces in the real world through physical or spatial operations, such as applying force, moving, manipulating, transporting, etc., and can provide support to the user in the real world, enhance, supplement or replace the user's own operations to meet the user's needs.

[0169] The voice assistance strategy is to provide real-time voice feedback and guidance to the user in various scenarios, specifically including task voice guidance, real-time voice prompts, voice error correction and optimization, etc., which can enhance the user experience and operation efficiency.

[0170] S16: Execute voice operations according to the voice assistance strategy through the I / O interaction module.

[0171] S17: Execute actions according to the action assistance strategy through the motion module.

[0172] Among them, the robot interaction state information includes the execution state information of the voice assistance strategy and the action assistance strategy.

[0173] For example, the voice assistance strategy is to play the voice sequence abcdefg, and the execution state information of the voice assistance strategy is that the robot is currently playing up to d. The action assistance strategy is that the robot executes the action sequence xyz, and the execution state information of the action assistance strategy is that the robot is currently executing y.

[0174] In a possible implementation manner, step S17 includes:

[0175] Control the robot to move and / or the operation components to execute actions according to the action assistance strategy through the motion module.

[0176] Among them, the operation components include the robotic arm and manipulator installed on the robot, etc.

[0177] In the application, control the robot to move to the destination and / or the operation components to execute actions related to the task content according to the action assistance strategy, improve the efficiency, accuracy and safety of task completion, and provide reliable action assistance support for the user in various scenarios to meet the user's needs.

[0178] For example, in a housework scenario, the control operation component assists the user in completing housework tasks. In a fitness scenario, the control operation component provides motion guidance and assisted exercises for the user, uses gestures with introductory, directive, and guiding functions during interaction, and massages the user, etc.

[0179] Play voice according to the voice assistance strategy.

[0180] In the application, control the voice output module to play voice according to the voice assistance strategy, provide a more intuitive and smooth interaction, and enhance the user experience and operation efficiency.

[0181] For example, in scenarios such as cooking, painting, and doing experiments, play voice to provide step-by-step guidance and assist the user in completing operations. In an exhibition scenario, broadcast exhibit information or recommended information. In teaching and performance scenarios, play voice to correct misoperations, optimize the interaction experience, or synchronize the rhythm.

[0182] To better understand the method described in the above embodiments, interaction examples in different scenarios are provided:

[0183] Please refer to Figure 3 , in a smart home scenario, the robot senses that the user picks up clothes (the second target), as well as the material, stain type, and degree of dirt of the clothes in the environment. According to the projection assistance strategy, highlight the stain area (the second visualization content) on the clothes picked up by the user. Then, based on the user's picking-up action and the material, stain type, and degree of dirt of the clothes, clarify the user's laundry needs and determine the task content: a preliminary washing plan, where the preliminary washing plan is to wash the clothes separately according to color. Project the washing plan (the first visualization content) on the wall according to the projection assistance strategy, and provide voice feedback according to the voice assistance strategy: ask whether to wash separately by color, whether to select clothes suitable for dry cleaning or hand washing, etc. Through the user's voice, detect the user's needs for the task content: want to wash underwear and outerwear separately, and then determine the task content: an adjusted washing plan to be able to update the visualization content in a timely manner. The adjusted washing plan includes the washing sequence and washing duration of the clothes. Project the first visualization content on the wall according to the new projection assistance strategy, so that the user can directly understand the specific work plan and execution process of the robot, make decisions or adjust the plan, enhance the user's control and participation in the task, and make the human-machine collaboration more natural and efficient. And pick up the user's clothes and put them into the washing machine according to the action assistance strategy. During the task execution process, adjust the task plan in real time. The adjusted task plan includes the current task status, unexecuted steps, and executed steps. Project the first visualization content on the wall or other areas convenient for the user to view according to the new projection assistance strategy. And interact with the user so that the user can know the task progress at any time and give timely feedback until the washing task is completed and the assistance ends.

[0184] Please refer toFigure 4 In the smart home scenario, the robot senses that the user's finger points towards the open wardrobe (the first target), as well as the types, sizes, materials, etc. of the clothes in the wardrobe in the environment. Then, based on the user's action of pointing to the wardrobe and the types, sizes, materials of the clothes, it determines the user's need to organize the wardrobe and defines the task content: a preliminary organization plan (organize by type). It projects the first visualized content onto the surface of the wardrobe according to the projection assistance strategy and provides voice feedback according to the language assistance strategy: asks whether certain clothes are needed the next day or the day after. By detecting the user's language, it determines the user's need for the task content: the coat needs to be hung up for the user to wear tomorrow, and the others are arranged according to the plan. Then it determines the task content: the adjusted organization plan (hang up the coat). It projects the first visualized content onto the surface of the wardrobe according to the new projection assistance strategy and stacks and organizes the clothes according to the action assistance strategy. During the task execution, it adjusts the task plan in real time (the current task status, unexecuted steps, and executed steps). And it interacts with the user until the organization task is completed and the assistance ends.

[0185] Please refer to Figures 5 to 6 In the smart home scenario, the robot senses the user's instruction to look after the child and determines the task content: a preliminary looking-after plan (help the child and perform the looking-after operations issued by the user). After the child comes home from school, it senses the child, the schoolbag (the first target), the shoes (the first target) in the environment, and no other people. Based on the child, the schoolbag, the shoes, and no other people in the environment, it determines the task content: the adjusted looking-after plan (carry the schoolbag, place the shoes, and project the mother's portrait, matter guidance: guide the user to get the bottled mung bean soup from the refrigerator). It projects the first visualized content onto the wall near the shoe cabinet according to the projection assistance strategy and plays the voice: Welcome home. Let me help you put the shoes in the shoe cabinet according to the language assistance strategy. It places the shoes in the shoe cabinet according to the action assistance strategy. After the child enters the kitchen according to the guidance, the robot follows and senses the child and the refrigerator in the environment, and determines the task content: the adjusted looking-after plan (get the mung bean soup in the refrigerator, project the mother's portrait, matter guidance: the robot helps you heat the food in the refrigerator and reminds you to feed the pet). It projects the second visualized content about the information in the refrigerator onto the surface of the refrigerator in an overlay manner according to the projection assistance strategy and opens the refrigerator, identifies the items, and takes out the mung bean soup according to the action assistance strategy. After the child goes to the balcony according to the guidance, the robot follows and determines the task content: the adjusted looking-after plan (guide the child to feed the pet). It projects the video of feeding the pet onto the wall and the ground near the pet feeding area according to the projection assistance strategy and uses body language to guide the child to feed the pet according to the action assistance strategy. The looking-after plan enhances the long-distance communication and interaction among family members. When the looking-after plan is completed, the assistance ends.

[0186] In the smart home scenario, the robot senses that the user is moving and the objects in the environment such as tables, cups, kettles, time, temperature, etc. Based on the user's movement and the tables, cups, and kettles in the environment, and the context information: the user's historical behavior information (drinking records within the sensed time), the robot determines the user's drinking needs and the task content: a preliminary drinking plan (recommending the best drinking time and the appropriate amount of water for this time). According to the projection assistance strategy, the second visualized content is projected and superimposed on the surface of the water cup and the area near the water cup to achieve interaction with real objects.

[0187] In the smart home scenario, the robot senses that the user picks up a book (the second target) and the temperature and light in the environment. Based on the user's action of picking up the book and the temperature and light in the environment, the robot determines the user's need to read the book and the task content: a preliminary reading assistance plan (creating a reading environment) and the historical electronic annotations of the book, where the historical electronic annotations are the answering information and the relevant information of the second target. According to the projection assistance strategy, the electronic annotations of the book (the second visualized content) are projected onto the pages so that the user can continue the previous reading progress and review the key content, and according to the action assistance strategy, a soft reading light is projected to avoid interference from the room light. When the user finishes reading, the robot senses that the user closes the book and the temperature and light in the environment. Based on the user's action of closing the book and the temperature and light in the environment, the robot determines the user's need to end reading and the task content: an adjusted reading assistance plan (putting the book back in place and showing the location where the book is to be put back). According to the projection assistance strategy, the location where the book is to be put back (the second visualized content) is projected in the bookshelf, and according to the action assistance strategy, the book is placed in that location, thus completing the assistance and providing the user with efficient, personalized, and embodied life support, improving the level of human-robot collaboration.

[0188] Please refer to Figure 7, in a hotel scenario, the robot perceives people and elements in the environment such as the hotel entrance and the front desk. Based on the people and these elements in the environment, it determines that the user has hotel-related needs and defines the task content: the initial assistance plan (welcoming guests and asking about their needs). It projects personalized welcome messages (the first visualized content) onto suitable locations for the user to view, such as the ground, walls, or the flat surface of a navigation desk, in accordance with the projection assistance strategy, and actively greets the guests according to the movement assistance strategy. It plays the voice message: "Dear sir, hello. May I help you with anything?" according to the voice assistance strategy. By detecting the user's voice, it determines the user's need for the task content: "How can I get to Room 501?" and defines the task content: the adjusted assistance plan (helping the guest get to Room 501). It projects the floor plan, navigation path, arrow navigation guidance, and specific travel distance and time (the first visualized content) onto the ground in accordance with the projection assistance strategy, and plays the message: "Dear sir, hello. Please turn right at the intersection ahead to reach your destination." according to the voice assistance strategy, enabling the user to obtain the itinerary route plan in an intuitive and concise manner. The guest proceeds towards the destination according to the guidance, and the robot follows, defining the task content: the navigation plan (projecting navigation guidance along the way as the guest moves). It dynamically projects direction arrows, floor plans, and facility introduction information onto the ground or walls along the way according to the projection assistance strategy, providing the user with clear route and environment guidance and giving the guest an intuitive understanding of the itinerary.

[0189] Please refer to Figure 8 , in a hotel scenario, the robot perceives the elevator in the people and the environment. When the user is waiting for the elevator, the robot determines the user's need to go to the room or a specific hotel facility based on the elevator in the people and the environment, and defines the task content: the initial guidance plan (providing services to the guest). It interacts with the user according to the voice assistance strategy to obtain the user's destination, defines the task content: the adjusted guidance plan (assisting the guest to the destination), and projects the floor plan of the user's destination floor, facility location, and path planning (the first visualized content) onto the wall near the elevator according to the projection assistance strategy, enhancing the user's spatial perception of the target area and optimizing the guidance path according to the environmental characteristics of the environmental information. During the task execution, it continuously perceives the user's activity information and environmental information and dynamically adjusts each assistance strategy to guide the guest to the destination, ensuring the real-time and accuracy of the guidance.

[0190] In a hotel scenario, the robot perceives the instruction from the user to inquire about recommended nearby restaurants. Based on the user's instruction, it determines the user's need for recommended restaurants and defines the task content: recommending nearby restaurants. It projects the map information, menu information, and reservation information of the recommended restaurants onto suitable locations for the user to view according to the projection assistance strategy, providing diverse options. It assists the user in completing the reservation operation according to the movement assistance strategy.

[0191] Similarly, in the scenario of going to a meeting room, the robot projects the floor plan of the meeting room, the navigation path, and meeting-related information, including meeting topics, participant lists, time schedules, and other details, according to the projection strategy to help users efficiently complete meeting preparations.

[0192] In a shopping mall scenario, the robot perceives the user's instruction to inquire about the mall situation. It determines the user's need to know about the mall situation based on the user's instruction and defines the task content: introducing the mall situation. According to the projection assistance strategy, it projects the brief introductions of the stores along the way, promotion activity information, details of special services, and recommends relevant stores and products based on user preferences, providing accurate and personalized shopping navigation and information services for users.

[0193] Please refer to Figures 9 to 10 , similarly, in a hospital scenario, the robot determines the user's navigation needs through the user's voice (such as "How can I get to the surgical department of the hospital?" or "How can I get to the children's area?") and provides navigation services for the user. It projects the first visualized content at a place suitable for the user to view according to the projection assistance strategy, such as on the walls or ceilings of the hospital (the first target). Additionally, it superimposes the second visualized content on the floor signs hanging from the ceiling, clinic signs, or the map held by the user (the second target) according to the projection assistance strategy. At the same time, it highlights and dynamically displays the first target or the second target to help patients quickly obtain important location information. It plays according to the voice assistance strategy: "Hello, the surgical department of the hospital is in this direction / at the lower right corner of the hospital guidebook. You can follow this route." During the task execution, it dynamically adjusts each assistance strategy in real-time by perceiving the user's activity information and environmental information, and dynamically adjusts the navigation path to guide the patient to the target clinic or area.

[0194] Similarly, in a medical scenario, the robot at the medical guidance desk helps patients with triage through various assistance strategies, communicates with other devices in the area, and guides other devices to receive patients to provide accurate navigation and guidance, offering efficient spatial location guidance for users.

[0195] Similarly, in a guided tour scenario, the task content of the robot is to provide personalized tour routes for each visitor according to the visitor's interest points, preferences, the passenger flow in the environment, and the time schedule. According to the projection assistance strategy, it dynamically superimposes the second visualized content such as recommended scenic spots, thumbnail images of exhibits, and explanatory content on the map or travel brochure held by the visitor (the second target), or projects it onto the map display board near the visitor to provide intuitive guiding information. During the task execution, it dynamically adjusts each assistance strategy in real-time by perceiving the user's activity information and environmental information, and dynamically adjusts the navigation path to enable users to obtain the best tour experience.

[0196] Through the above navigation interactions, users can easily reach their destinations, and can also learn valuable information related to the path, improving the practicality and convenience of navigation.

[0197] Please refer to Figures 11 to 12 , in the shopping guide scenario, the robot perceives various shelf and product information of people and the environment (the first goal). Based on the various product information of people and the environment, it determines that the user has a need to purchase items, and determines the task content: assist with the purchasing plan (follow the user and provide assistance), avoiding the pressure and unnatural feeling brought by traditional personnel shopping guides. During the user's purchasing process, it perceives that the user has been staring at milk for a long time and / or asks for information related to milk, or a recommended voice (May I ask when this milk is suitable for drinking), and then determines the user's need for milk information, and determines the task content: highlight the description information of milk (the information to answer and the relevant information of the first goal). The description information includes highlighted features, competitor analysis, price trends, and current promotion efforts, etc. According to the projection assistance strategy, project the recommended plan of the product (the second visualization content) on the surface of the auxiliary object held by the robotic arm (the second goal) and superimpose the highlighted expiration date of milk on the product packaging surface (the second visualization content), or it can also be projected on the ground, and according to the action assistance strategy, point to the description area on the milk surface, and according to the voice assistance strategy, play: According to the current date label, the milk of this brand is the freshest and is on sale today. You can see the detailed information and nutritional composition description here, and this low-sugar organic milk is suitable for drinking in the morning. Let me provide you with a specific matching plan or more product information. The robot interacts efficiently with the user, provides comprehensive product recommendations and information display services, helps the user quickly understand the historical price changes and current preferential policies of the product, assists the user in making a wise purchase decision, and at the same time provides promotional information of derivative products related to the product, bringing a personalized shopping experience to the user.

[0198] When the robot perceives that the user asks about the location of the cashier, expresses the voice of purchasing goods or other information, it determines the user's need to purchase goods, and determines the task content: provide checkout service. According to the projection assistance strategy, project the relevant information of the product (the first visualization content) on a suitable plane, and according to the voice assistance strategy, play: Hello, please confirm whether the product information is the product you selected. I will put it in the shopping bag for you for confirmation. According to the action assistance strategy, adjust its own posture, pick up the product and identify the barcode to complete the checkout and packing of the product for the user, which can save the waiting time of traditional queuing for checkout and achieve a seamless connection of the purchasing service.

[0199] In the shopping guide scenario, the robot perceives the types of people and food in the environment. Based on the types of people, food in the environment, and context information: the user's shopping list and daily diet records, it determines the user's dish-making requirements and the task content: preliminary dietary planning (dietary combination recommendations and asking the user for adjustments). When it senses that the user picks up a certain food (the second target), it can also determine the user's need to know about this food based on the food selected by the user, and determine the task content: information about this food (answer information and relevant information about the first target). The relevant information includes energy value, calorie content, ingredient list, nutritional components, and impacts on health, etc. According to the projection assistance strategy, it projects the relevant information of this food on the surface of or near the food ingredient, enabling the user to interact with the robot through a multi-modal interaction method, understand the nutritional value of each ingredient, and obtain personalized dietary suggestions according to their own needs. Through user voice detection of the user's need for the task content: the user's voice asking about ingredient combinations. Based on the user's voice asking about ingredient combinations and context information: the user's intake of a certain nutrient has reached an excessive level, it determines the user's need for the task content involved and determines the task content: adjusted dietary plan (suitable ingredient combinations, nutritional matching plans, and the body change model after re-intaking a certain nutrient). According to the projection assistance strategy, it projects the adjusted dietary plan (the first visualized content) at a position suitable for the user to view, enhancing the user's intuitive understanding of healthy eating. After the user finishes shopping, based on the ingredients purchased by the user, it determines the user's need to know about the nutritional situation of the ingredients and determines the task content: overall nutrition plan (the daily dietary energy, nutritional distribution, and suggestions for adjusting the dietary structure for the next meal). According to the projection assistance strategy, it projects the first visualized content at a position suitable for the user to view, helping the user optimize their diet plan and achieve the purpose of scientific diet and health management.

[0200] Please refer to Figure 13 , when the user stops in front of a large item (the first target) (for example: a washing machine), the robot perceives the person's position, the voice asking for recommendations (I want to buy a washing machine, please help me recommend), and the washing machine in the environment. Based on the person's stopping action and the washing machine in the environment, it determines that the user has a need to know about the washing machine and determines the task content: relevant information about the washing machine (the first target). The relevant information includes the internal structure cross-sectional view of the washing machine, the working process of key components, etc. According to the projection assistance strategy, it projects and overlays the internal structure cross-sectional view, the working process of key components (the second visualized content), etc. on the surface of the washing machine, and according to the action assistance strategy, it locally magnifies and projects the internal structure to help the user understand the key functional modules or the core advantages of product improvements in more detail, and according to the voice assistance strategy, it plays: This washing machine has an efficient motor that drives the inner drum to run smoothly, and is equipped with intelligent sensors to automatically adjust the water volume and washing time.

[0201] Please refer to Figure 14, in the shopping guide scenario, the robot perceives the user's inquiry for recommendations (wanting to buy a top, can you recommend some to me), the voices of other communications, and the colors and styles of various types of clothes, the clothes, pants, and shoes (the second target) worn by the user in the environment, determines the user's purchase needs, and determines the task content: recommendation solutions (outfit recommendations, current promotional information, etc.). According to the projection assistance strategy, project the outfit recommendations expressed by the clothing graphics and the promotional information represented by the combined patterns (the first visualized content) on the wall near the clothing, and play according to the voice assistance strategy: Here is the top I recommend for you and some outfit recommendations. Enjoy a discount when purchasing in a combination.

[0202] Please refer to Figure 15 , during the user's try-on process, the robot perceives the user's voice for projecting the clothing (I want to try on the effect of this top, can you help me). Determine the user's clothing try-on needs based on the user's voice, and determine the task content: preliminary clothing display plan (the top of the style selected by the customer). According to the projection assistance strategy, project the clothing effect of the style, material, and color required by the user (the second visualized content) on top of the user's white top. And play according to the language assistance strategy: Okay, projecting the upper body effect of this piece of clothing for you. During the projection process, detect the user's needs for the task content through the user's voice: I think this style is quite good. Can you change the color for me? Determine the task content: adjusted clothing display plan (tops of other colors of the same style). According to the projection assistance strategy, project the clothing effect of the style, material, and color required by the user on the user's white top, and play according to the voice assistance strategy: Okay, projecting tops of other colors of the same style for you. By projecting on the white clothing worn by the user, the clothing effect of the style, material, and color required by the user can be simulated, enabling the user to view the effect without the hassle of changing multiple sets of clothing. And during the projection process, move the mirror according to the action assistance strategy and automatically adjust the mirror angle to meet the user's best viewing needs. For example, when the user wants to view the back effect of the clothing, the robot moves the mirror behind the user to provide a full range of clothing try-on effects.

[0203] Please refer to Figure 16 , in the shopping guide scenario, the robot perceives the user's voice (understanding the furniture material) and the information of the furniture (the first target) in the environment, determines the user's need to understand the furniture material, and determines the task content: preliminary furniture material and color display plan. According to the projection assistance strategy, project different materials of the furniture (the second visualized content) (such as wood, metal, fabric, etc.) on the surface of the furniture for the user to visually compare. During the task execution, the robot interacts with the user to change the material, color, and simulate the effects under different lighting conditions to facilitate the user's accurate decision-making.

[0204] Please refer to Figure 17, in the shopping guide scenario, the robot perceives the user's voice (want to modify the vehicle) and the information of the vehicle in the environment, determines the user's need to modify the vehicle, and determines the task content: the preliminary vehicle painting plan. According to the projection assistance strategy, different painting effects (the second visualization content) are projected on the surface of the vehicle. During the task execution, the robot interacts with the user to change the painting effect and dynamically adjusts the projection direction according to the user's position to achieve a fast presentation of the display effects from multiple angles and under various appearance designs. The robot perceives the user's voice (just this kind of painting) and determines the task content: generate an order plan. Generate an order according to the action assistance operation and connect to the settlement system of the store to efficiently complete the shopping process.

[0205] Please refer to Figure 18 , in the shopping guide scenario, the robot perceives the user's long-term gaze at a certain displayed electronic product (the first target) (such as: mobile phone, tablet or computer) and the information of the electronic product in the environment, determines the user's need to understand the electronic product, and determines the task content: the preliminary supporting recommendation information (answer information) and promotional information (related information of the first target) of the electronic product. According to the projection assistance strategy, the first visualization content: supporting recommendation information and promotional information (combined preferential promotion information) is projected at a suitable position near the product, and is played according to the voice assistance strategy: These are some popular software applications for mobile phones recommended to you. You can check the prices of different mobile phone models here in the projection to help the user comprehensively understand the purchase options and enhance the consumption experience.

[0206] Please refer to Figure 19 , in the exhibition scenario, the robot perceives the user's gaze at the painting and the information of the painting in the environment, determines the user's need to understand the painting, and determines the task content: the preliminary exhibition item introduction plan, the answer information of the painting, and the related information (related information of the first target). According to the projection assistance strategy, the exhibition item information (the first visualization content) (raw materials, technological characteristics, internal structure, creation process, creation background, knowledge principle, texture material, origin of the past production of the exhibition item, how to use) is projected in the area near the exhibition item and at a position suitable for visitors to watch. Through the distributed animation demonstration, the complete creation process from sketch drawing to detail polishing is presented to help visitors intuitively understand the subtleties of artistic creation, display the historical timeline of the creation period, highlight the association between the exhibition item and the art movement and important historical events, and reproduce the historical background and story details through short plot animations. In addition, the second visualization content is projected and superimposed on the art reproduction or miniature model. The classical painting is transformed into a modern abstract style through the style transfer technology, and the dynamic change of the art style is demonstrated in real time. And through the simulation of art cooperation, the multi-style fusion effect of the painting is demonstrated to explore the diversity and creative possibilities of artistic expression, and comprehensively enhance the depth and interest of visitors' exhibition viewing. And according to the action assistance strategy, the explanation effect is enhanced through gestures. The gestures point to the key elements in the painting or simulate the hand habits of the artist when painting to improve the visitors' attention and understanding of the details of the exhibition item.

[0207] During the demonstration, the robot perceives the user's speech (there is too much background information to read, please help me find the key information), determines that the user understands the key information needs, and determines the task content: the key content of the background information (information related to the first objective). According to the projection assistance strategy, a projection is superimposed on the background information display board to highlight the key content (the second visualization content), and the following is played according to the voice assistance strategy: Okay, the key content of the creation background information is being highlighted.

[0208] In addition, the robot perceives the user's speech (I want to see what the simple drawing of this painting looks like) and the information of the painting in the environment, determines the user's need to view the simple drawing, and determines the task content: the simple drawing display plan. According to the projection assistance strategy, the simple drawing (the second visualization content) is projected onto the robot's handheld object (the second objective), and the following is played according to the voice assistance strategy: The simple drawing of this painting is being generated for you. Please look at the projection display in the picture frame.

[0209] Please refer to Figure 20 , in the exhibition scenario, the robot perceives the user's speech (I want to see what the complete sculpture looks like) and the information of the sculpture in the environment, determines the user's need to understand the shape of the sculpture, and determines the task content: the preliminary completion plan. According to the projection assistance strategy, the missing part of the sculpture is projected to complete the restoration or the original color, material, texture and decoration are superimposed in real time to restore the original appearance of the sculpture before fading (the second visualization content). During the restoration process, the projection effect is dynamically enhanced so that the user can feel the original charm and historical value of the exhibit in an immersive experience. And the following is played according to the voice assistance strategy: Okay, the appearance of the complete sculpture is being projected and completed for you. In the completion plans of other exhibits, the interactivity and vividness of the explanation can also be enhanced on the surface of the exhibit (the second objective) held by the robot.

[0210] In the exhibition scenario, the robot projects the complete forms of ancient buildings and large cultural relics in positions suitable for visitors to view according to the projection assistance strategy, and dynamically demonstrates the construction process, construction methods, tools and technical details to help visitors deeply understand how the ancients overcame technical difficulties to build magnificent buildings, and the dynamic scene reconstruction restores historical moments and storylines, enabling visitors to be in the historical environment at that time and feel the cultural significance and historical value of the exhibits from multiple dimensions.

[0211] Please refer to Figure 21, in the rehabilitation scenario, the robot senses the user lying in bed, information about the bed in the environment, and context information: the user's activity status, physiological data (e.g., heart rate, respiratory rate, muscle tension, etc.), determines the user's need for rest and relaxation, and determines the task content: a preliminary flat massage plan. After the user confirms, the robot performs massage on areas such as the user's chest according to the motion assistance strategy. Plays according to the voice assistance strategy: Noticed that you seem very tired today. Let me play some soothing images and sounds of the ocean for you to relax. And projects an image of the ocean on the ceiling according to the projection strategy (the first visualization content). And dynamically adjusts the massage intensity in real time according to the specific situation during the massage process to provide precise personalized massage services for the user. The robot senses that the user is sitting on the bed and information about the bed in the environment, determines the task content: an adjusted back massage plan. After adjusting to the optimal massage position according to the motion assistance strategy, changes the massage area and performs massage on areas such as the user's back, and adjusts the massage mode in real time. Projects the ocean image on the wall in front of the user's eyes according to the projection strategy so that the user can always watch the projection image in the most comfortable way, and plays the massage situation and precautions according to the voice assistance strategy (This part of the muscle is a bit tense. I'll massage it more for you. Remember not to overuse your hands next time. I'll record it and remind you).

[0212] Please refer to Figure 22 , during the massage process, the robot senses the user's voice (Please tell me what my specific daily schedule is tomorrow), determines the user's need to know tomorrow's schedule, and determines the task content: a preliminary tomorrow's schedule plan. Projects the daily schedule (the first visualization content) onto the wall in front of the user's eyes according to the projection assistance strategy. The robot senses a meeting notice and determines the task content: an adjusted daily plan. Projects the daily schedule onto the wall in front of the user's eyes and plays according to the voice assistance strategy: Just received a notice that tomorrow's meeting event has been postponed. Here is the updated schedule.

[0213] Similarly, in the rehabilitation scenario for fracture rehabilitation, the robot uses the robotic arm as a rehabilitation aid according to the motion assistance strategy, adjusts the robotic arm to a form suitable for the patient's weight-bearing training through the teaching mode, and adjusts the required torque of the robotic arm in real time according to the patient's rehabilitation situation. And provides muscle relaxation massage services for the patient after the training to promote muscle recovery and relaxation. And highlights and projects the correct muscle activation situation (the second visualization content) on the corresponding area of the patient's body according to the projection assistance strategy. For example, for a patient with a leg fracture, the robotic arm can be positioned in front of the patient's leg to provide appropriate weight-bearing support to help the patient with rehabilitation training. Highlights and projects the correct muscle activation situation on the patient's leg dynamically to guide the patient to complete standard rehabilitation movements.

[0214] Please refer to Figure 23, in the rehabilitation scenario, the tasks of the robot are as follows: physical therapy plan. According to the projection assistance strategy, the human body meridians, key acupoints, and treatment areas (the second visualized content) are projected and superimposed on the patient's body. According to the motion assistance strategy, the acupoints are accurately located and basic physical therapy operations are performed (for example: massage, acupuncture, cupping for acupoints or position calibration, assisting in the execution of massage movements) to reduce the workload of doctors, and according to the voice assistance strategy, the voice is played in cooperation with the doctor's voice (if you feel any discomfort, please tell me at any time, and the robot and I will make adjustments immediately): This is the Ganshu acupoint, which can promote liver metabolism. We still need to press for another minute.

[0215] During the task execution, the robot adjusts the projection content in real time according to the projection assistance strategy, helps the doctor find the correct acupoints in real time through projection, distinguishes the completed and uncompleted massage areas by color and provides detailed guidance for the next operation, etc., so that the doctor can grasp the physical therapy status in real time, improve the treatment effect, and project the working status at the appropriate position to facilitate the user to understand the physical therapy progress and subsequent plans in real time. Also, the robot adjusts the projection pose in real time according to the projection assistance strategy so that the projection content and the massage area follow in real time when the patient turns the body, ensuring the continuity and accuracy of the collaborative operation with the doctor. And interact with the user in real time through a variety of interaction methods, collect the user's feedback in real time (for example: pain perception, massage strength perception) and dynamically adjust the physical therapy plan according to the feedback to achieve personalized optimization.

[0216] Please refer to Figure 24 , in the hospital scenario, the robot senses the information of the patient sitting on the chair and the tables and chairs in the environment, determines the patient's medical needs, and determines the task content: guiding the patient's medical treatment plan and providing medical treatment answers. According to the projection assistance strategy, the path is projected at a suitable position to guide the patient to the designated physical examination room and project the relevant information of the physical examination and medical treatment, so that the patient can intuitively know the required operation steps, and project the patient's body data (questions, blood pressure, heart rate) (the first visualized content) at a position convenient for the patient to observe, enhancing the patient's understanding of their own health status. According to the motion assistance strategy, assist the patient to complete pre-diagnosis operations such as taking body temperature and measuring blood pressure, and play according to the voice strategy: Please let me measure your body temperature and blood pressure for you. Your physical examination information is shown in the projection.

[0217] After the pre-diagnosis task is completed, all the data is uploaded to the hospital database according to the motion assistance strategy to provide data for the doctor's subsequent medical treatment, enabling the doctor to directly access the patient data on the platform during the medical treatment without repeated measurement, improving the diagnosis efficiency and reducing the patient's ineffective waiting time.

[0218] Please refer to Figure 25In the hospital scenario, the robot perceives the user's voice (there are several people in front, and they have been waiting for a long time), determines that the user is impatient and needs to understand the waiting information needs, and determines the task content: relieve the user's emotions and answer relevant information about the medical treatment. According to the projection assistance strategy, the medical treatment information (first visual content) is projected at the appropriate location, such as: the number of people waiting in front, the doctor's qualifications and the estimated waiting time, so that the patient can obtain practical information and relieve tension, and plays according to the voice assistance strategy: The doctor information and waiting information you are seeing can be viewed on the projection, please wait patiently. And according to the projection assistance strategy, interesting GIF animations, emoticons and humorous content (second visual content) are projected on the surface of the robot to interact with the patient, and according to the voice assistance strategy, it plays: Sorry, I know you are very anxious, can I tell you some happy stories, provide emotional support and information consultation services for patients, and create a soothing and pleasant waiting environment.

[0219] While the patient is waiting, the robot senses that the patient is sitting down and determines the task content: preliminary consultation waiting plan. According to the projection assistance strategy, vivid and interesting health science popularization content (for example: prevention methods of common diseases, healthy lifestyle suggestions, etc.) is projected at the appropriate location, and easy-to-understand health education is presented in a graphic and textual manner. The robot senses that the patient stands up and walks around, and determines the task content: adjusted consultation waiting plan. According to the action assistance strategy, the robot follows the patient's steps, and according to the projection assistance strategy, the dynamic content (secondary visual content) is projected to a location suitable for the patient to watch. Projected content, such as: fun animations of cartoon characters imitating the patient's actions, warm and encouraging text.

[0220] In the hospital scenario, the robot perceives the user's voice (asking questions), determines the user's need to solve the problem, and determines the task content: the answer to the problem. According to the projection assistance strategy, the robot projects the content in a location suitable for the patient to watch.

[0221] In the hospital scenario, the robot perceives the information of the user's medication window in the mobile environment, determines the user's medication needs, and determines the task content: preliminary medication planning and medication answer information. According to the action assistance strategy, the patient is guided to the designated medication window, and the complete prescription information is seamlessly transmitted to the medication robot. After being transmitted to the medication robot, the name, formula ratio, function, efficacy, dosage, service frequency, time period and other information of the medicine and the medication process are highlighted or displayed dynamically in color according to the projection assistance strategy. During the task execution, according to the projection assistance strategy, after receiving the operation data transmitted by the medication dispensing robot, the medication dispensing process is projected at an appropriate location to intuitively display the details of each step. And after interacting with the patient, the specified screen is enlarged according to the action assistance strategy so that the user can clearly see the characteristics and details of the medicinal materials.

[0222] In psychotherapy, the robot perceives the user's painful expression and the information about items in the environment, determines the user's comfort needs, and determines the task content: preliminary comfort planning, provides emotional support and psychological comfort to users, and builds an interactive experience with healing effects. According to the projection assistance strategy, the robot projects the scene images generated based on the photos, videos or audio provided by the user. For example, when the patient is sad because of the death of a pet, the robot projects the image of the pet accompanying the loved ones as an angel, or displays the quiet natural landscape to create a comfortable and cozy atmosphere to relieve the patient's emotions. Or when the patient is sad because of the death of a loved one, the robot projects the interactive images of the deceased loved one's belongings before death to build emotional connections and comfort for the patient. And provide physical emotional support according to the action assistance strategy, such as patting the shoulder or shaking hands with the patient to enhance the healing effect.

[0223] During the task execution, the robot senses the user's choice of real objects as emotional personification carriers and determines the task content: adjust the comfort plan. Superimpose the virtual character on the surface of the real object according to the projection assistance strategy. Conduct emotional dialogues with patients according to the voice assistance strategy, allowing patients to express their inner conflicts and thoughts, and cooperate with therapists to provide guidance.

[0224] Please refer to Figure 26 ,In the game scene, the robot perceives the user's activity information and the information of the game items (first target) in the environment, determines the user's need to play the game, and determines the task content: preliminary assistance planning, as a referee or game assistant, to improve the player's gaming experience and create a more immersive interactive environment. According to the projection assistance strategy, dynamic visual effects (secondary visualization content) are projected on the surface of the items used by the player, and the corresponding game effects and animations are projected at appropriate locations according to the specific operations of the identified players, providing players with a unique interactive experience. For example: in the card game, the player is identified to play the attack card, and the corresponding attack animation is projected in real time to intuitively reproduce the character's actions and effects. In the Landlord game, the player is identified to play the bomb card, and the corresponding explosion effect is projected in real time to enhance the game atmosphere. Play according to the voice assistance strategy: Now it is the turn of player No. 1, please ask player No. 1 to operate the game.

[0225] During the task execution, dynamic visual effects or prompt information are projected in real time according to the game progress, supporting players' instant query of game rules or strategies in real time.

[0226] Please refer to Figure 27, in an immersive game scenario, the robot perceives information about people and game props in the environment, determines the user's game start requirement, and determines the task content: preliminary game development plan. According to the projection assistance strategy, the corresponding game content (the second visualized content) is projected onto the surface of the prop or the area near the prop, transforming static props into highly interactive game elements. For example, the robot follows the player and integrates the walls, floors, ceilings, and props in the space into the game scenario through projection, creating a highly interactive virtual-real combined game experience (projecting the image of a magic manual on the surface of the book prop in the player's hand, dynamically displaying the runes and clues in it, or projecting special effects on the surface of the broom in the player's hand, transforming it into a magic prop to enhance the player's immersion). And according to the projection assistance strategy, the assistance screen is projected at a suitable position, such as: route guidance, hidden clues, or projection of decryption progress. And according to the action assistance strategy, game props are delivered, physical effects are simulated, and the player's role is played to interact with other players, enhancing the interactivity and fun of the game.

[0227] Please refer to Figure 28 , in a ball game scenario, the robot perceives the user's voice (I want to hit the No. 1 ball, please assist me) and information about the billiard balls in the environment, determines the user's requirement to hit the No. 1 ball, and determines the task content: preliminary assistance plan, providing route recommendations, action correction, and visual enhancement effects to improve the sports experience. According to the projection assistance strategy, the corresponding hitting point (the second visualized content) is dynamically displayed on the surface of the white ball from the hitting perspective behind the player, so that the hit white ball can hit the target ball (the first target). The hitting route (the second visualized content) generated based on the player's real-time aiming selection and cue angle is projected and superimposed on the table, helping the user accurately grasp the hitting direction and strength, and the movement trajectory of the ball after hitting is projected on the table, enabling the player to intuitively know the movement and collision effects of the ball at different angles and strengths, and helping the user optimize strategies and improve technical levels. According to the action assistance strategy, the hitting action posture is provided, and according to the voice assistance strategy, it is played: Okay, you can adjust your action according to my action posture, and I will project the suitable hitting position and trajectory prediction.

[0228] The robot perceives that the user is ready to hit the ball and the corresponding standing posture, center of gravity shift, and cue holding method, and determines the task content: adjusted assistance plan, providing specific guidance. According to the projection assistance strategy, the cue holding position indication (the second visualized content) is projected on the surface of the cue (the second target), and the teaching video (the first visualized content) is projected at a suitable position (such as the wall or the table), and according to the action assistance strategy, physical assistance is provided to help the user adjust the posture and master the correct hitting technique. The robot perceives that the user hits the ball, and determines the task content: adjusted assistance plan, providing visual effects to provide the user with a dynamic feedback experience. According to the projection assistance strategy, visual effects (the second visualized content) are superimposed on the movement trajectory of the ball, such as: flames or meteor trails, etc.

[0229] Please refer to Figure 29 , in the learning scenario, the robot perceives the information of people and learning items in the environment, determines the interaction needs with students, and determines the task content: preliminary interaction planning, dynamically responds to environmental changes, position changes, and students' needs. According to the action assistance strategy, it maintains an appropriate distance from the teacher and moves to the position of the student where the teacher cannot pay attention in time. And according to the projection assistance strategy, it provides diverse interactive support on the surface, nearby area or desktop of devices such as students' books, mobile phones, computer screens, etc. (the first target).

[0230] Please refer to Figure 30 , in the learning scenario, the robot perceives the information of the user answering questions and the books in the environment, determines the user's learning needs, and determines the task content: preliminary teaching plan, question answering information, so that the user can comprehensively understand the questions. According to the projection assistance strategy, it highlights the key knowledge points (the second visualization content) on the surface of the book, projects the problem-solving ideas and key steps at appropriate positions, and projects correct or wrong marks (the second visualization content) near the user's answers. During the task execution, it interacts with the user to provide teacher connection answers and project teacher feedback information.

[0231] Please refer to Figures 31 to 32 , similarly, in the learning scenario, during the projection learning process, when users all simultaneously gaze at a certain content, the robot dynamically adjusts the size and position of the projection area in real time according to the positions and gaze directions of each user, so as to cover the line-of-sight ranges of all users. When users have different gaze directions, it provides corresponding projection pictures for users in different positions, provides an adapted projection method for multiple users, and provides clear projection content for each user, facilitating users to view.

[0232] Please refer to Figures 33 to 34 , in the teaching scenario (such as physics, biology, chemistry, etc.), the robot perceives the information of teaching aids (the second target) in people and the environment, determines the user's needs for using teaching aids, and determines the task content: preliminary enhanced teaching aid plan, realizing dynamic and intuitive teaching demonstrations. According to the projection assistance strategy, it superimposes dynamic projection content (the second visualization content) on the surface or nearby area of the teaching aid, including the operating mechanism of the system, the highlighted important parts, and the correlation between different components. For example: project the correct circuit connection diagram on the desktop and gradually show the operation steps and connection details to help students intuitively understand the experimental process. Project the scale markings and dropping steps on the surface of the measuring cup to illustrate the dosage of the reagent, ensuring the safety of the experiment operation and the accuracy of the data. And according to the voice assistance strategy, play: Classmates, please look at the highlight of my projection. Next, I will explain this part.

[0233] Please refer to Figure 35, in a teaching scenario, the robot senses the information of the user holding experimental equipment (the second target) and experimental objects in the environment (the first target), determines the user's experimental needs, and determines the task content: initially assist in planning, actively intervene to provide intelligent and multi-modal auxiliary support, and answer the user's questions. Adjust the pose in real time according to the projection assistance strategy and project the detailed experimental guidance plan generated according to the current experimental progress on the desktop, experimental equipment, etc., for the user to view. For example: in the teaching of the circulatory system in a biology class, the robot projects the dynamic operation process of the circulatory system (the second visualization content) on the surface of a human model (the first target) or the student's body (the second target), enhancing the students' understanding of complex concepts through visualization. And play according to the voice assistance strategy: Classmates, please look at the highlight of my projection. Next, I will explain this part. During the task execution, adjust the position of the projection in real time according to the needs of students or teachers according to the projection assistance strategy, and display the circulatory functions of different parts of the human body to provide intuitive learning support for the user.

[0234] In a handicraft scenario, the robot senses the information of the user making a work (the second target), the user's voice, the user's gestures, and the objects in the environment, determines the user's need to make a work, and determines the task content: initially assist in planning, provide personalized help, and provide intelligent tutoring for the user. Provide physical assistance according to the action assistance strategy, cooperate with the user to complete the production task, and project the targeted guidance plan (detailed operation steps) generated according to the work and production steps on the desktop according to the projection assistance strategy, and project the production guidance screen (the second visualization content) on the surface of the work. For example: gradually project the folding steps and the corresponding key folding lines and cutting lines (the second visualization content) on the surface of the origami to help the user complete the operation accurately. Display detailed guidance on the surface of complex works such as carving or models, showing the correct carving shape or the order of component organization.

[0235] In an outdoor scenario (such as: botanical garden, science and technology museum, field investigation and scientific research environment, etc.), the robot senses the shape, position, and type of plants (the first target) of the user during movement and in the environment, determines the user's need to understand plants, and determines the task content: initially introduce plant planning and detailed information about plants (answer information and relevant information about the first target), and provide an interactive learning experience for the user. Project the detailed information of the plant (scientific name, ecological characteristics, distribution area, etc.) in the nearby area of the plant according to the projection assistance strategy, dynamically animate the life cycle of the plant (the complete process from seed germination to flowering and fruiting to withering) in the nearby area of the plant to help the user comprehensively understand the growth mechanism of the plant, and project the dynamic operation process of the root system in the nearby area of the plant to demonstrate the way of absorbing water and nutrients, and display physiological characteristics such as leaf veins and the internal structure of the stem on the surface of the plant, enabling the user to intuitively feel the internal operation mechanism of the plant.

[0236] Please refer toFigure 36 In an outdoor scenario, the robot senses information about people and wild plants (the first target) in the environment, determines the user's outdoor operation requirements, and determines the task content: preliminary outdoor exploration planning and information on the answers to wild plants. According to the projection assistance strategy, project the toxicity attributes (edible or inedible plant species) (the second visualization content) on the surface of the plants to help the user make a correct judgment. According to the action assistance strategy, assist the user in completing tasks such as picking, material collection, and carrying, improving the safety and efficiency of outdoor operations.

[0237] Please refer to Figure 37 In a fitness scenario, the robot senses the user's voice (I want to do bicep curl training. Can you tell me the training movements?) and information about fitness items (the first target) in the environment, determines the user's bicep training needs, and determines the task content: preliminary training planning and providing full assistance during the user's training process. According to the projection assistance strategy, project the action demonstration and teaching content and the user's training status (the first visualization content) at a position suitable for the user to watch, so that the projected content is clearly visible and matches the user's perspective, providing an intuitive and easy-to-understand display of the action essentials for the user. For example, when the user is sitting for exercise, the projection position is the wall facing the user, or when the user is lying down for training, the projection position is the ceiling, and project a dynamic schematic diagram of muscle exertion when performing the action on the corresponding area of the user's body (the second visualization content) so that the user can intuitively view the exercise effect. For example, when the user is doing dumbbell curls, project the outline of the biceps brachii on the user's arm and dynamically display the activation state of the muscle using color changes (the second visualization content). Red indicates that the muscle group is fully involved in the training, helping the user master the correct exertion method and adjust in real time. According to the action assistance strategy, display the standard demonstration action, and according to the voice assistance strategy, play: No problem. Please follow my movements for training. Pay attention to the exertion of this muscle. You can also view the training video projected by me.

[0238] Please refer to Figure 38, when the robot senses the information of the user's posture and fitness objects in the environment, and identifies that the user's posture is inaccurate, it determines the user's correct training needs and the task content: adjusted training plan (adjust the user's posture), helps the user correct the posture, determines the planning and safety of the movements, further optimizes the user's fitness experience, and intuitively guides the user to complete the training. It guides the user's limbs to the correct position according to the movement assistance strategy and projects the correct fitness movements according to the projection assistance strategy. The robot senses the user's voice ("I can't lift it anymore, come and help me"), determines the user's need to help lift the dumbbell, and determines the task content: adjusted training plan (assist the user to use the dumbbell (second goal)). It grabs the dumbbell according to the movement assistance strategy and provides appropriate assistance to help the user safely break through the training bottleneck. For example, it assists the user to complete the last few movements in dumbbell presses or barbell training, and plays according to the voice assistance strategy: "Okay, assisting you to put the equipment back." And during the task execution, it dynamically adjusts the assistance intensity according to the user's physical condition and training progress, ensures the training effect and avoids the risks caused by excessive load, and creates a safer, more efficient and personalized fitness support platform for the user.

[0239] Please refer to Figure 39 , in the action teaching scenarios (such as: dance, gymnastics, yoga, ballroom dance, skating, etc.), the task content of the robot is: to teach the user to learn dance movements, provide action design and precise guidance, and provide systematic and personalized technical support for action training. It shows the postures of the dance movement joints according to the movement assistance strategy, helps the user intuitively feel and learn the dance movements, and precisely imitate the dance movements (such as: arm postures or body balance postures). It projects the imitated dance movements and the sensed own movement gestures to the user according to the projection assistance strategy (second visualization content), especially the enlarged display of the hand detail movements, which is convenient for the user to better observe and learn the imitation. During the task execution, the robot analyzes the user's dance movements in real time, and dynamically displays the previously learned action steps and subsequent actions, step-by-step action demonstrations (first visualization content) according to the projection assistance strategy, ensures that the user understands the action logic, and slows down the projection rhythm at the key actions to emphasize the posture requirements and force details, and helps the user master the skills of complex actions.

[0240] Please refer to Figure 40, in the outdoor physical education teaching scenario, the tasks of the robot are as follows: to assist users with outdoor activities, provide intelligent support and personalized services to users, so that users can obtain intuitive and vivid guidance during the activity, and ensure that users receive appropriate teaching content for their progress to achieve the best learning effect. According to the projection assistance strategy, warm-up exercises such as stretching, bending, and jumping are projected on the ground projection game area (the second visualized content) (for example: hopscotch grids or the marked area for picking rubber bands), as well as the guiding images of the teacher's movements, so that users can directly follow the projected content to complete the action practice. According to the action assistance strategy, guide users to perform systematic warm-up exercises, and command and demonstrate users' actions through gestures to enhance the interaction with users and improve teaching efficiency.

[0241] Please refer to Figure 41 , in the cooking scenario, the robot perceives the user's voice (I want to fry an egg, help me beat the egg, and then tell me how to make a poached egg), the information of kitchen objects in the environment, and the context information: the user's taste preferences and the user's physical condition, determines the user's need to fry an egg, and determines the task content: preliminary cooking plan, assist the user to complete the operations of different links in cooking, and improve the efficiency and experience of the cooking process. According to the projection assistance strategy, project the poached egg production process on the wall in front of the user, display the raw materials required for the dish, the specific cooking method, and the finished product effect diagram (the first visualized content), and provide projection information and prompts for the user. According to the action assistance strategy, prepare the ingredients, and play according to the voice assistance strategy: Okay, I'm starting to prepare the ingredients now. The poached egg production process is projected in front of you. During the task execution, the robot adjusts the tutorial content and rhythm in real time according to the user's cooking progress, and asks the user whether to add condiments or sauces, and records the user's choices, so as to provide recommendations for the user when making the dish again and actively ask for the user's choices.

[0242] Please refer to Figure 42 , in the cooking scenario, the robot perceives the user's voice (How to slice this potato (the second target)) and the information of kitchen objects in the environment, determines the user's need to slice the potato, and determines the task content: potato slicing plan. According to the projection assistance strategy, project a demonstration video of the potato slicing steps on the desktop (the first visualized content), and project and superimpose cutting auxiliary lines on the surface of the potato (the second visualized content), and demonstrate the cutting skills according to the action assistance strategy to ensure that the user accurately completes the ingredient processing operation. Play according to the voice assistance strategy: You can first cut it in half, and then cut it into slices along the projected line. During the task execution, project the information of key steps, heat control, and condiment addition amount in real time according to the projection assistance strategy to help the user master the best cooking parameters, optimize the production of the dish, and execute the transfer of ingredients or condiments according to the action assistance strategy to assist the user in performing basic physical operations such as putting, lifting, and covering, and optimize the cooking process.

[0243] Similarly, in a cooking scenario, the robot can also assist the user in washing the ingredients before cutting them according to the action assistance strategy, or transfer the ingredients from the washing area to the cutting area, and project the best processing plan (the first visualization content) generated based on the ingredient type, environmental information, and common sense at a place suitable for the user to view according to the projection assistance strategy. Project the cutting method preview (the second visualization content) on the surface of the ingredients according to the projection assistance strategy. And interact with the user to confirm the cutting method (e.g., dicing, shredding, etc.) according to the voice assistance strategy. The robot continuously interacts with the user to enable the user to complete each step according to the tutorial and optimize the task plan in real time.

[0244] Please refer to Figure 43 , Similarly, in a restaurant scenario, the robot senses the information of the user sitting down and the items in the environment (the first target) (e.g., the information of knives, forks, and plates), determines the user's ordering needs, and confirms the task content: preliminary ordering plan and dish answering information, and provides the functions of ordering, information display, and delivery for the user. Project the menu information (the first visualization content) on the table according to the projection assistance strategy, and place the tableware or pour tea for the user according to the action assistance strategy. The robot senses the user's voice (What kind of fish is used for this steamed fish?), confirms that the user understands the fish variety category needs, and determines the task content: the answering information of the steamed fish. Project the detailed content of the dish on the table according to the projection assistance strategy, including: the source of the ingredients, freshness, nutritional value, and allergen reminder, to help the user intuitively understand the characteristics of the dish and make a choice, and play according to the voice assistance strategy: This steamed fish is made with freshly killed grass carp and is our signature dish. The robot senses the user's voice (I want the steamed fish), determines the user's order placement needs, and determines the task content: order placement plan. Transmit the order information to the kitchen according to the action assistance strategy to instruct the kitchen to prepare the dish, and project the waiting time for serving and the dynamic countdown action on the table or the wall according to the projection assistance strategy to strengthen the user's sense of control over the dining process.

[0245] Please refer to Figures 44 to 45 , In a remote space assistance scenario (e.g., remote lectures, teaching, etc.), the robot's task content: remote assistance planning. As the agent of the remote speaker, present the speech situation in an intuitive form in the target space to achieve efficient cross-space information transmission and interactive communication. Project the speaker's content, upper body image, actions, and speech subtitles (the first visualization content) at a position suitable for the participants to view according to the projection assistance strategy. The projected content can be projected in one place or in different places. Ensure clear transmission of remote information, and simulate the speaker's body language and gestures in real time according to the action assistance strategy to provide a better presentation experience for the user. Play according to the voice assistance strategy: Welcome everyone to my remote speech. I can see all my friends here. Thank you all for coming.

[0246] For example, when a remote speaker emphasizes key content, highlight and project the emphasized content of the speaker, and simulate the speaker's explanatory gestures to strengthen the display of key information, attract the attention of participants, and improve the efficiency and effect of information transmission.

[0247] During the task execution, in accordance with the action assistance strategy, adjust the booth position and angle in the on-site space in real time according to the position and actions of the speaker in their environment, so that the projected content and action display are highly coordinated, ensuring that viewers can always clearly see the speech content and the dynamic demonstration of the speaker, creating a sense of authenticity as if the remote speaker is on-site. At the same time, in accordance with the projection assistance strategy, project the real-time picture of the viewer (the second target) onto the screen that the speaker is looking at, enabling the speaker to learn about the reactions and actions of the visitors in real time, generating an immersive speech experience, and strengthening the on-site atmosphere of the remote speech.

[0248] In addition, transmit the positioning information of the viewer and the close-up pictures of the audience who ask questions to the remote speaker's screen for the remote speaker to select. Interact with the selected position of the visitor by the speaker in accordance with the voice assistance strategy and the action assistance strategy, simulate the voice content of the remote speaker (now invite a friend present to answer my question) and the invitation gesture, guide the visitors to ask questions, and optimize the interactivity and immersion of remote collaboration.

[0249] Similarly, in a live broadcast scenario, the robot follows the user in real time in accordance with the action assistance strategy, and projects the key features, function descriptions or script content of the product dynamically at a position suitable for the user to watch in accordance with the projection assistance strategy, helping the user to complete the explanation smoothly and confidently, switching the live broadcast screen in real time according to the user's situation, and projecting the bullet screens, feedback and question content of the audience, facilitating the user to interact immediately and answer the questions of the audience. It can also adjust its own pose in accordance with the projection assistance strategy and the action assistance strategy to clearly project the detailed parts of the product in close-up (such as interfaces or internal structures), ensuring that the audience can obtain an intuitive and detailed viewing experience, and providing comprehensive intelligent assistance for live broadcast activities in factories, homes or retail.

[0250] Please refer to Figure 46, in a scenario with complex equipment, the robot interacts with the user, senses the user's voice, actions, and information about objects in the environment (e.g., the interface of the equipment (second target), the location and function of physical buttons (first target), etc.), determines the user's need to operate the equipment (e.g., ATM machines, self-service registration machines, report pick-up machines, etc.), determines the task content: the operation device guidance plan and relevant information of the equipment, provides the user with an interaction medium with the equipment to provide intuitive and intelligent operation guidance. Highlight prompts (second visualization content) in key areas of the equipment according to the projection assistance strategy to guide the user to complete the operation process, such as the card insertion entrance, option buttons, password input area, etc., to ensure that the user can intuitively understand how to execute each step on the operation interface of the equipment. And play according to the voice assistance strategy: Dear customer, please enter your password in the highlighted area. During the task execution, adjust the projection content and guidance rhythm in real time according to the user's feedback according to the projection assistance strategy, such as providing a detailed projection display explanation for the user's confusion, and assisting the user to complete specific operations after obtaining the user's confirmation according to the action assistance strategy.

[0251] Please refer to Figure 47 , in the scenario of vocational training, the robot senses the user's voice (The operation buttons of this machine tool seem a bit complicated. Can you tell me how to complete the work of cutting templates?) and information about the machine tool (first target) in the environment, determines the user's need for help, determines the task content: explaining the equipment plan and the explanation information of the machine tool (answer information and information of the first target), provides the user with efficient operation guidance, and learns the operation process. Highlight relevant physical buttons, operation prompt content, and additional step number labels (second visualization content) on the equipment interface and buttons according to the projection assistance strategy, and gradually guide the user to press the correct buttons in an intuitive way, providing the user with clear and definite operation guidance to help the user complete the entire required operation process. And play according to the voice assistance strategy: Of course no problem. You can operate the buttons and each step and function in the marked numerical order in turn, and assist the user to complete some physical operations that are difficult for beginners to master according to the action assistance strategy, such as adjusting the workpiece position or activating the safety switch, effectively improving the safety and accuracy of equipment operation. During the task execution, interact with the user in real time, and dynamically adjust the guidance rhythm according to the user's feedback and progress, such as slowing down the switching speed of the highlighted buttons or repeating the prompts for key steps.

[0252] Please refer to Figure 48Similarly, in the instrument teaching scenario, the robot's task content is to guide the learning of instrument planning and provide all-round intelligent assistance and guidance for instrument practitioners. According to the projection assistance strategy, the teaching video (first visual content) of the music score and the standard playing posture of the instrument is projected within the user's visual range to help users learn and imitate intuitively, and the pitch calibration image is projected to intuitively understand the deviation and correction method of the standard pitch, and assist users to accurately adjust their playing skills. And according to the action assistance strategy, the user's playing posture is adjusted in real time to enable the user to maintain a standardized playing posture during the practice process (for example: when the user practices a string instrument, the angle of the user's arm or bow is assisted to correct the action of holding the bow or pulling the string), and the gesture command is dynamically followed by the performance melody to provide rhythmic guidance for the user, enhance the fluency and expressiveness of the performance, and optimize the teaching interaction and experience. During the task execution, the projection content and angle are dynamically adjusted according to the position of the practitioner and the viewing angle requirements of the instrument according to the projection assistance strategy and the action assistance strategy.

[0253] Please refer to Figure 49 In public service collaboration scenarios (for example, subway stations, airports, railway stations and other crowded places), the robot's tasks include: providing public service planning, assisting public service professionals or completing part of the work. According to the projection assistance strategy, the robot dynamically projects path-guiding arrows and other visual content (first visual content) generated based on the identified congested areas and personnel flow directions on suitable surfaces such as walls, floors or load-bearing columns (first target), providing intuitive walking guidance for the crowd, and guiding the crowd's walking direction according to the action assistance strategy, guiding the crowd to move in a specific direction, effectively alleviating the congestion in key areas, and realizing real-time diversion and guidance of the flow of people. According to the voice assistance strategy: There is a large flow of people here, please move forward, providing crowd management and guidance. During the execution of the task, according to the projection assistance strategy, the guidance situation is dynamically adjusted according to the environmental conditions and the flow of people in real time, for example: increasing or reducing the display frequency of the path guidance, adjusting the projection position and the coverage of the content, to ensure the clarity and effectiveness of the guidance information.

[0254] Please refer to Figure 50In the first aid scenario, the robot's task content is: first aid guidance planning, providing first aid guidance to the surrounding people or medical staff, and ensuring that first aid information is easy to understand and execute at critical moments. When using the device, project the device's operating steps (first visual content) at a location suitable for users to watch according to the projection assistance strategy, including how to turn on the device, connect the electrode patch and the correct placement. And pass first aid medicines, equipment or tools (second goal) to assisting medical staff according to the action collaboration strategy to assist medical staff to quickly complete first aid preparations. When medical staff want to perform operations, project key parameters such as vital signs information on the wall or near the patient according to the projection assistance strategy (first visual content), project instructions or feedback information such as the strength, position and rhythm of the superimposed compression on the patient's body (second visual content), and assist in performing the operation according to the action assistance strategy, and play according to the voice assistance strategy: the patient's heart rate has recovered to 60 times per minute, and it is recommended to continue cardiopulmonary resuscitation compression, so that medical staff can intuitively grasp the operation status and effect. During the execution of the task, dynamically adjust the action and projection content according to the progress of first aid in real time, for example: real-time update of compression rhythm or pressure feedback.

[0255] Please refer to Figure 51 In high-risk scenarios (e.g., disaster relief or fire response), the robot's tasks include: guiding planning, cooperating with professional rescuers, and performing dangerous operations on behalf of personnel. Dynamically display fire maps or escape route maps (first visual content) according to the projection assistance strategy, provide visual guidance of safe paths for rescuers or trapped persons, and dynamically display changes in the on-site environment by projecting superimposed information (second visual content) on environmental objects, such as the direction of fire spread, unstable structures, etc. And carry collapsed obstacles, clear escape routes, or transport rescue supplies according to the action assistance strategy. Play instructions according to the voice assistance strategy, for example: Danger, please leave quickly.

[0256] In scenarios that require fine manipulation (e.g. surgery, medical aesthetics, etc.), the robot's mission is to assist planning and improve operational efficiency and accuracy. According to the projection assistance strategy, the operation location or medical imaging information of the body (second visual content) is accurately displayed on the surface of the patient's body (second target), providing real-time visual guidance for medical staff to assist in formulating plans or performing complex operations, such as incision locations, X-rays, etc. And according to the action assistance strategy, tools and equipment are delivered to medical staff and high-precision operations are performed (e.g. suturing or precision organ operations in minimally invasive surgery), providing surgical efficiency, safety and accuracy.

[0257] Similarly, in the tattoo scenario, the tasks of the robot are as follows: assisting in planning to improve operation efficiency and precision. Projecting a preview effect diagram of the tattoo (the second visualization content) on the skin area according to the projection assistance strategy to help users quickly customize and confirm the pattern design. And completing the outlining and filling of the pattern or displaying the pen movement path and target area according to the action assistance strategy to assist the tattoo artist. During the task execution, project the complete effect of the future pattern (the second visualization content) in real time according to the projection assistance strategy for the tattoo artist to adjust the design.

[0258] Similarly, in the medical aesthetics and beauty makeup scenarios, the tasks of the robot are as follows: assisting in planning. Dynamically display the makeup trajectory on the user's face according to the projection assistance strategy, and implement the makeup steps according to the action assistance strategy to provide users with personalized beauty makeup solutions.

[0259] Please refer to Figure 52 , in the scenario of human-machine co-created artistic expression, the robot interacts with the user (for example: you can continue to talk to me to generate pictures or paint based on the projection) to perceive the user's creative content (for example: the sun rises from the mountains in the east and shines its brilliance on the earth.) and the information of objects in the environment, determine the user's painting needs, and determine the task content: assisting in creative planning and the information of artistic elements to provide precise guidance for the user's next creation. Dynamically project visual patterns (such as the sun, clouds, mountains, etc.) on the canvas according to the projection assistance strategy. And transfer tools (the second target) and clean tools according to the action assistance strategy to provide convenience for the user's creation and provide seamless operation support for the user. For example: assisting the user in getting the paint palette, color mixing tools or cleaning the paintbrush. During the task execution, adjust the projection content in real time according to the user's feedback, such as modifying the color, shape or position of the elements to meet the user's creative needs.

[0260] Please refer to Figure 53 , similarly, in the stage performance scenario, the tasks of the robot are as follows: enhancing performance planning, enhancing the stage effect, and enriching the layering and forms of stage performances. Project virtual images and flowing special effects (the second visualization content) (such as particle effects like air currents, waves, water ripples, etc.) generated according to the movements, rotations, and jumps of the dancers (the second target) in different dance segments according to the projection assistance strategy, project virtual costumes (the second visualization content) on the surface of the dancers' bodies, project the stage background on the stage, and project the light and shadow streamlines generated according to the dancers' waving of their arms. During the task execution, dynamically adjust the costumes and stage background in real time according to the different dance plots according to the projection assistance strategy.

[0261] Moreover, according to the action assistance strategy, it can also cooperate with the projection effect to participate in the stage performance, synchronize actions with the dancers' gazes, and jointly complete an innovative performance integrating technology and art, creating a unique artistic effect of the integration of virtual and reality, and the interaction between humans and machines.

[0262] In the non - heritage cultural performance scenarios (such as puppet shows, shadow plays, etc.), the tasks of the robot are as follows: assisting in performance planning, providing new possibilities for the inheritance and innovation of art. Generating backgrounds, light sources, and dynamic images on the curtain according to the projection assistance strategy, and controlling the puppets or shadow puppets according to the action assistance strategy. During the task execution, the background on the curtain is switched in real - time according to the plot of the story according to the projection assistance strategy, closely combined with the action performance of the puppets or shadow puppets, enabling the audience to appreciate the fine operations of the characters and experience the scene transition effect synchronized with the plot of the story, giving new vitality to traditional non - heritage culture.

[0263] It should be understood that the magnitudes of the sequence numbers of the steps in the above - mentioned embodiments do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of this application. And the perception of the data in the above - mentioned embodiments is compliant, and its use or implementation does not involve harming the public interest.

[0264] Corresponding to the method described in the above - mentioned embodiments, for the sake of convenience of description, only the parts related to the embodiments of this application are shown.

[0265] In one embodiment, please refer to Figure 54 , the robot includes: a perception module, an information processing module, and a projection module;

[0266] The perception module is used to perceive user activity information and environmental information.

[0267] In a possible implementation manner, the perception module provides the perception function of multiple - modality data, providing basic support for the external perception, reasoning, and decision - making of the robot. The perception module includes a sensor module, a vision module, and an I / O interaction module.

[0268] The sensor module is used to perceive user activity information and environmental information.

[0269] The vision module is used to perceive environmental information.

[0270] The I / O interaction module is used to perceive user activity information.

[0271] In a possible implementation manner, the sensor module includes different types of sensors such as tactile sensors, depth sensors, distance sensors, light sensors, sound sensors, and vision sensors to perceive data in the real world.

[0272] In a possible implementation manner, the data perceived by each sensor in the sensor module is judged and compared with the data in the dataset of the information processing module to activate single or multiple modules in the robot related to the task, which can reduce the computing power and performance requirements of the robot.

[0273] Exemplarily, the voice of the user is captured by a sound sensor, and gestures or behavioral actions of the user are recognized by sensors such as a tactile sensor and a visual sensor. After judgment and comparison, the I / O interaction module is activated. According to the specific interaction scenario, the user can be a single or multiple persons.

[0274] After multi-modal data is sensed by sensors that can sense environmental information such as a depth sensor, a distance sensor, and a light sensor, after judgment and comparison, the visual module is activated to accurately locate objects and positions in the environment, identify dynamic changes, process and analyze the environmental information, and infer the impact of the environmental changes before and after on the user.

[0275] In a possible implementation manner, the visual module integrates various visual perception devices, including RGB-D cameras, laser scanners, object detection cameras, thermal imaging cameras, multi-spectral cameras, HDR cameras, etc., to sense the visual information of the real world in real time, and can identify various environments according to different types of image data, including the object features, spatial layout, users or non-user personnel in the environment and their activities and environmental changes, and provide reliable visual information support.

[0276] In a possible implementation manner, the I / O interaction module includes a voice interaction module, an embodied interaction module, a physical interaction module, etc., to improve multiple input and output methods and be able to interact with the user for information in different scenarios.

[0277] The voice interaction module includes a sound localization module, a speaker, a microphone, etc., for speech recognition and natural language processing, and can capture the voice of the user through accurate sound localization.

[0278] The embodied interaction module includes a face recognition module, a posture recognition module, a gesture recognition module, a behavioral action recognition module, etc., and can identify the non-verbal information of the user in real time to sense the user activity information.

[0279] The physical interaction module includes physical buttons, physical switches, displays, vibration feedback modules, etc., and can provide multiple types of interaction methods to achieve diverse interactions with the user and ensure that the multi-level interaction needs of the user are met in different scenarios.

[0280] The information processing module is used to determine the task content according to the user activity information and the environmental information;

[0281] It is also used to determine the projection assistance strategy according to the task content and the environmental information.

[0282] In an application, after receiving information from a sensing module, an information processing module performs inference and decision-making to obtain task content, determines a projection assistance strategy based on the task content and environmental information, and then transmits the information to a projection module through information transmission so that the projection module can perform an assistance operation.

[0283] It can be understood that the information processing module includes an information processing function and an information transmission function to ensure effective coordination and interaction among modules, and improve the overall performance and response ability of the robot.

[0284] A projection module is used to perform a projection operation according to the projection assistance strategy.

[0285] In a possible implementation, the projection module includes an optical projection module, an optical correction module, a dynamic stabilization module, a projection angle adjustment module, and a graphics processing module. It can perform multi-directional and multi-scene planar projections, interact with the real world in various scenarios, enhance and superimpose objects and spaces in the scene, and improve the user's perception and interaction ability with the environment. The optical projection module integrates various types of projection devices, such as laser projectors, LED projectors, digital light processing projectors, etc., and can adapt to the display requirements of different scenarios to determine high-brightness, wide-color gamut, and accurate imaging of the projection screen.

[0286] The optical projection module provides a projection function. The optical correction module provides functions of automatic focusing and trapezoidal correction of the projection surface to ensure that users can obtain clear and accurate projection displays at any projection angle or distance. The dynamic stabilization module and the projection angle adjustment module provide flexible projections in multiple directions and at multiple angles, and can effectively maintain the stability of the screen during the movement of the robot or the change of the projection position, improving the adaptability in complex environments. The graphics processing module provides accelerated image rendering and processing to ensure high-definition and smooth projection effects in dynamic scenarios, and improve the real-time performance and image quality stability of the projection content.

[0287] Please refer to Figure 55 , the installation position and method of the projection module can be selected according to the scene requirements. The installation position and method include the top of the robot's head, the forearm, the shoulder, the chest, and the end of an additional robotic arm, etc.

[0288] The number of projection modules installed on the robot can be selected according to the scene requirements. Each projection module can be installed at different positions on the robot to achieve a wider range and multi-view projection function. And each projection module can use different types of projection systems and technologies.

[0289] The projection module is specifically used to project the first visualization content and target visual elements at a projection angle at the projection position through the projection module; project the second visualization content and target visual elements at the position of the first target and / or the second target through the projection module.

[0290] In an application, when the projection position can only be reached by moving the robot body and / or the projection module, after moving the robot body and / or the projection module, project the content at the projection position and project the content at the position of the first target and / or the second target.

[0291] Specifically, the optical correction module and the dynamic stabilization module are used to respond to the projection position and projection angle in real time during the change of the robot position and the position of the second target, ensuring the stability of the projection screen during the movement, so that the optical projection module projects the content clearly and continuously during the change of the robot position. The projection angle adjustment module is used to respond to the projection angle in real time during the change of the user position, so that the projection screen of the optical projection module is always in the direction suitable for the user to watch and within the best line of sight range.

[0292] In an application, the robot includes a power module, and the power module includes a power supply, a wired / wireless charging management module, a USB interface, a heat dissipation module, etc., providing stable and safe power supply for each module, and realizing efficient power management and heat dissipation control.

[0293] Please refer to Figure 56 , it can be understood that the external form of the robot includes various forms, realizing modular disassembly and assembly combination, such as the installation positions and configuration methods of actuators such as mobile modules, robotic arms, and dexterous hands, which can be specifically selected according to the actual scenario requirements. By way of example, the robot can be in the form of a robotic arm gripper based on a traditional mobile chassis. A single robotic arm gripper form of a wheeled balanced mobile chassis. A humanoid robot form of a traditional mobile chassis. A humanoid robot form of a wheel-legged balanced mobile chassis. And a bipedal humanoid robot form.

[0294] The robotic arm can be designed to have two or more degrees of freedom and be connected by joint connectors. The form of the robotic arm includes various forms, which can be specifically selected according to the actual scenario requirements, including traditional robotic arms, movable pan-tilt heads, octopus-arm robotic arms, snake-shaped robotic arms, etc. The joint connection methods can include mechanical connections, electrical connections, etc., so that the robotic arm can achieve translational motion, rotational motion and any combination thereof in any direction.

[0295] In one embodiment, please refer to Figure 57 , the robot further includes a positioning and navigation module, a communication module and a motion module.

[0296] The positioning and navigation module is used to locate the position of the robot.

[0297] In a possible implementation, the positioning and navigation module includes a positioning module, a SLAM module, etc., providing high-precision spatial positioning and navigation support in different scenarios, ensuring accurate and stable determination of the robot's position and orientation in the environment in different scenarios, including complex dynamic scenarios.

[0298] The positioning module integrates multiple positioning hardware, including a UWB positioning module, a GPS positioning module, an IMU inertial measurement module, a magnetometer, a radar, etc. The UWB positioning module provides high-precision positioning in a small indoor area. The GPS positioning module provides wide-area positioning for outdoor scenarios. The IMU inertial measurement module includes a gyroscope and an accelerometer, improving the attitude perception ability. The magnetometer can assist in positioning, ensuring accurate and stable determination of the position and orientation in different scenarios.

[0299] The SLAM module provides real-time map construction and precise positioning functions, capable of identifying and avoiding obstacles, optimizing path planning, enabling the robot to autonomously and efficiently complete navigation tasks in different scenarios.

[0300] The communication module is used to communicate with external devices.

[0301] In a possible implementation, the communication module integrates a wired communication module and a wireless communication module, achieving efficient data transmission, flexible interaction, and providing fast and stable connections and information transfer in scenarios with large data transmission requirements or high security, supporting the data transmission and control requirements between the robot and different devices and users, improving the operation efficiency and interaction fluency in different scenarios, including complex dynamic scenarios. The wireless communication module includes a mobile network module, a Bluetooth module, a radio frequency identification module, a wireless local area network, a global navigation satellite system, a near-field communication module, an infrared communication module, a frequency modulation communication module, etc. The wired communication module includes a high-bandwidth Ethernet module.

[0302] In applications, when the robot has a need to communicate with external devices, the information processing module communicates with external devices through the communication module to assist the information processing module in processing information.

[0303] The information processing module is also used to determine voice assistance strategies and action assistance strategies according to the task content;

[0304] The motion module is used to execute actions according to the action assistance strategy.

[0305] In a possible implementation, the motion module includes a motor module, a motion buffer module, a collision prevention safety module, a mobile chassis, and an operation component, etc., capable of providing flexible and diverse interaction methods for users in the case of continuous changes in the user, the environment, and the robot's position, improving the environmental adaptability and operation flexibility of projection interaction.

[0306] Among them, the operation components include robotic arms, robotic hands, etc. The motor module includes motion and joint encoders, servo motors, motor controllers, and drive units, which are used to control the coordinated movement of the robotic arm, robotic hand, and mobile chassis, ensuring precise and efficient operation of the robot in different environments, including complex environments, and performing assistance operations according to the action assistance strategy. The motion buffer module includes hydraulic buffers, spring dampers, etc., which are used to effectively absorb and relieve impact forces during the operation of the robot, reduce vibrations caused by rapid movement or sudden stops, and improve operation stability and operation accuracy. The anti-collision safety module is used to sense the positions of objects in the environment in real time, adjust the motion path in a timely manner, avoid collision accidents, and ensure the safety and reliability of the robot.

[0307] Specifically, the motor module controls the robotic arm, robotic hand, and mobile chassis to perform assistance operations.

[0308] The I / O interaction module is also used to perform voice operations according to the voice assistance strategy.

[0309] In a possible implementation manner, the voice interaction module performs assistance operations according to the voice assistance strategy, and can combine functions such as voice parsing and real-time voice dialogue to provide smooth voice feedback to the user, ensuring natural language interaction with the user.

[0310] It should be noted that for the information interaction, execution process, etc. between the above-mentioned devices / units, since they are based on the same concept as the method embodiments of the present application, their specific functions and the technical effects brought can be specifically referred to in the method embodiment part, and will not be elaborated here.

[0311] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the above-mentioned division of each functional unit and module is used for illustration. In practical applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0312] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0313] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, it enables the electronic device to execute the steps in the above-mentioned method embodiments.

[0314] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-mentioned method embodiments of the present application, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps in the above-mentioned method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the photographing device / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk, or an optical disc, etc. In some cases, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0315] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0316] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but this implementation should not be considered to exceed the scope of the present application.

[0317] In the embodiments provided in the present application, it should be understood that the disclosed device / network device and method can be implemented in other ways. For example, the device / network device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.

[0318] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0319] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A robot interaction method, characterized in that: include: Obtain user activity information and environmental information perceived by the perception module; Determining task content according to the user activity information and the environment information; Determining a projection assistance strategy according to the task content and the environmental information; Performing a projection operation according to the projection assistance strategy through a projection module; Wherein, determining the projection assistance strategy according to the task content and the environmental information includes: Based on a preset rule, according to the task content and the environmental information, determining a projection mode, wherein the projection mode includes a display mode, a projection content placement mode and projectable surface information; Based on the projection mode, a projection position and a projection angle are determined according to the position of the robot, the position of the user, and at least one of the position of the first target and the position of the second target, wherein the environmental information includes the position of the user and at least one of the position of the first target and the position of the second target, and the position of the robot is determined by positioning; Determine first visualization content according to the task content and context information; Determine a second visualization content according to at least one of the task content, the context information, the relevant information of the first target, and the relevant information of the second target; Determine a target visual element according to the first visualization content, the second visualization content, information of related visual elements, the user activity information and robot interaction state information, wherein the robot interaction state information is information about the interaction between the robot and the outside world; Among them, the projection assistance strategy includes the projection position, the projection angle, the first visualization content, the second visualization content and the target visual element; the first visualization content is used to display the task content, background information related to the task content and guidance information and feedback information related to the task content, and the second visualization content is used to supplement and enhance the projection of the first target and / or the second target.

2. The method according to claim 1, characterized in that The determining of the task content according to the user activity information and the environment information includes: determining a user intention based on the user activity information and the environment information; Determining user needs according to the user intention, scenario information and context information, wherein the scenario information is determined according to the environment information, and the context information is determined according to the scenario information, the user activity information and the environment information; Determine at least one of task planning information, answer information, relevant information of a first target, relevant information of a second target, and information of relevant visual elements according to the user's needs, wherein the first target is a static object within the user's attention range, the second target is an object in motion within the user's attention range, and the relevant information includes information to be supplemented and information to be enhanced; Generate working status information according to the working status of the robot; The task content includes the task planning information, the answer information, relevant information of the first goal, relevant information of the second goal, information of the relevant visual elements and the working status information.

3. The method according to claim 1, characterized in that The projection module performs a projection operation according to the projection assistance strategy, including: projecting the first visualization content and the target visual element at the projection angle at the projection position by a projection module; The second visualization content and the target visual element are projected at the position of the first target and / or the second target by a projection module.

4. The method according to claim 1, characterized in that: Also includes: Determine a voice assistance strategy and a motion assistance strategy according to the task content; Execute voice operation according to the voice assistance strategy through the I / O interaction module; The motion module executes the motion according to the motion assistance strategy.

5. The method according to claim 4, characterized in that Executing actions according to the action assistance strategy through the motion module, including: The motion module controls the robot to move and / or operate the components to perform actions according to the action assistance strategy.

6. The method according to claim 2, characterized in that The determining the user intention according to the user activity information and the environment information includes: Determining user behavior information and user psychological state information based on the user activity information; Determining the interaction between the user and the environment according to the user activity information and the environment information; The user intention is determined according to at least one of the interaction situation, the user behavior information, the user psychological state information, and the user language information, and the user activity information includes the user language information.

7. A robot, characterized in that: include: Perception module, information processing module and projection module; The perception module is used to perceive user activity information and environmental information; The information processing module is used to determine the task content according to the user activity information and the environment information; Also used to determine a projection assistance strategy according to the task content and the environmental information; The projection module is used to perform the projection operation according to the projection assistance strategy; The information processing module is specifically used to determine the projection mode based on a preset rule, according to the task content and the environmental information, and the projection mode includes a display mode, a projection content placement mode and projectable surface information; Specifically used to determine the projection position and the projection angle based on the projection mode and according to the position of the robot, the position of the user and at least one of the position of the first target and the position of the second target, wherein the environmental information includes the position of the user and at least one of the position of the first target and the position of the second target, and the position of the robot is determined by positioning; Specifically used to determine the first visualization content according to the task content and the context information; Specifically used to determine the second visualization content according to at least one of the task content, the context information, the relevant information of the first target, and the relevant information of the second target; Specifically used to determine a target visual element according to the first visualization content, the second visualization content, information of related visual elements, the user activity information and robot interaction state information, wherein the robot interaction state information is information about the interaction between the robot and the outside world; Among them, the projection assistance strategy includes the projection position, the projection angle, the first visualization content, the second visualization content and the target visual element; the first visualization content is used to display the task content, background information related to the task content and guidance information and feedback information related to the task content, and the second visualization content is used to supplement and enhance the projection of the first target and / or the second target.

8. The robot according to claim 7, characterized in that: It also includes a positioning and navigation module, a communication module and a motion module, and the perception module includes a sensor module, a vision module and an I / O interaction module; The positioning and navigation module is used to locate the position of the robot; The communication module is used to communicate with external devices; The sensor module is used to sense the user activity information and the environmental information; The visual module is used to perceive the environmental information; The information processing module is further used to determine a voice assistance strategy and a motion assistance strategy according to the task content; The motion module is used to perform the action according to the action assistance strategy; The I / O interaction module is used to sense the user activity information; It is also used to perform voice operations according to the voice assistance strategy.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images