Robot control method, storage medium, and program product

Abstract

The invention discloses a robot control method, a storage medium and a program product, and relates to the technical field of automation control, the robot control method is applied to a robot, the robot is provided with an image sensor, and the method comprises the following steps: receiving a user instruction, a first image of the external environment where the robot is located is collected based on the image sensor; analyzing the first image to obtain user posture data, and determining an operation object from the external environment based on the user posture data; analyzing the user instruction to obtain an instruction instruction operation, and generating a robot action sequence based on the instruction instruction operation; and controlling the robot to execute the robot action sequence on the operation object. The problem that the robot mistakenly executes the task or cannot execute the task due to the fuzzy instruction is avoided, so that the task execution efficiency of the robot facing the fuzzy instruction is improved.

Description

Technical Field

[0001] The present application relates to the field of automation control technology, and in particular to a robot control method, a storage medium, and a program product. Background Art

[0002] In today's era of rapid technological advancement, robots have been widely used in many fields, including industrial production, logistics and warehousing, and intelligent services. For example, voice-controlled robotic arm systems and cleaning robots allow operators to conveniently control robots to perform tasks through voice commands, improving work efficiency and reducing labor costs. However, in complex real-world work environments, operators' commands are not always strictly and precisely structured. Instead, they may contain ambiguous expressions that require context to accurately understand, such as "tidy up here." When robots receive such commands, they often struggle to accurately infer the operator's true intentions, and thus fail to correctly perform the task. This impacts the robot's efficiency and makes it difficult for robots to meet diverse and intelligent work demands.

[0003] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a robot control method, storage medium and program product, aiming to solve the technical problem of low task execution efficiency when the robot receives ambiguous instructions.

[0005] To achieve the above objectives, the present application proposes a robot control method, which is applied to a robot equipped with an image sensor. The robot control method includes:

[0006] receiving a user instruction and collecting a first image of the external environment of the robot based on the image sensor;

[0007] parsing the first image to obtain user posture data, and determining an operation object from the external environment based on the user posture data;

[0008] Parsing the user instruction to obtain an instruction-indicated operation, and generating a robot action sequence based on the instruction-indicated operation;

[0009] The robot is controlled to execute the robot action sequence on the operation object.

[0010] In one embodiment, the step of parsing the first image to obtain user posture data includes:

[0011] Inputting the first image into a pre-trained posture extraction network to obtain joint points corresponding to each joint of the user and a head rotation angle, wherein the head rotation angle includes a head pitch angle, a head yaw angle, and a head roll angle;

[0012] The joint points and the head rotation angle are determined as user posture data.

[0013] In one embodiment, the step of determining the operation object from the external environment based on the user gesture data includes:

[0014] Determining the user's facial orientation based on the head rotation angle, determining a first projection ray with the user's head center point as a starting point and the facial orientation as a first projection direction, constructing a first cone space model with the first projection ray as a central axis, and projecting the first cone space model onto the first image to obtain a sightline focus area;

[0015] Determining a second projection ray from a limb end joint point among the joint points as a starting point and a pointing direction of the limb where the limb end joint point is located as a second projection direction, constructing a second cone space model with the second projection ray as a central axis, and projecting the second cone space model onto the first image to obtain a limb indication area;

[0016] An area in the external environment corresponding to the sight focus area and the body indication area is determined as an area to be operated, and an object in the area to be operated is determined as an operation target.

[0017] In one embodiment, the step of generating a robot action sequence based on the instruction instruction operation includes:

[0018] If the instruction indicates an explicit pointing action, determining the current posture of the operation object based on the first image, obtaining a preset reference posture corresponding to the operation object, and performing trajectory planning to generate a robot motion sequence using the current posture as an initial value, the reference posture as a target value, and the instruction indicates an action constraint condition;

[0019] If the instruction indicates that the operation is not the clear pointing action, the user's historical behavior associated with the operation object is determined from the user's historical behavior data, and the trajectory planning is performed to generate the robot action sequence with the current posture as the initial value, the reference posture as the target value, and the user's historical behavior as the action constraint condition.

[0020] In one embodiment, after the step of receiving the user instruction, the method further includes:

[0021] Detecting whether the user instruction is a clear-pointing instruction; wherein the clear-pointing instruction is an instruction including a specific object and a specific action;

[0022] If the user instruction is not the explicit pointing instruction, executing the step of capturing the first image of the external environment of the robot based on the image sensor and subsequent steps;

[0023] If the user instruction is the explicit pointing instruction, the user instruction is parsed to obtain the instruction indication object and the instruction indication operation, a target annotation matching the instruction indication object is determined from the various semantic annotations of the preset semantic map, the entity object corresponding to the target annotation is determined as the operation object, a robot action sequence is generated based on the instruction indication operation, and the robot is controlled to execute the robot action sequence on the operation object.

[0024] In one embodiment, the step of detecting whether the user instruction is an explicit directed instruction includes:

[0025] Performing semantic analysis on the user instruction to extract instruction actions and instruction objects in the user instruction;

[0026] If there is a unique matching annotation in each of the semantic annotations and the action represented by the instruction action is a clear pointing action, then the user instruction is determined to be a clear pointing instruction, wherein the matching annotation is the annotation in each of the semantic annotations that matches the instruction object;

[0027] If there are multiple matching annotations or no matching annotation in each of the semantic annotations, and / or the action represented by the instruction action is not the explicit pointing action, it is determined that the user instruction is not the explicit pointing instruction.

[0028] In one embodiment, after the step of controlling the robot to perform the robot action sequence on the operation object, the method further includes:

[0029] Capturing a second image of the external environment based on the image sensor, and analyzing the second image to obtain an actual posture of the operation object;

[0030] Mapping the actual posture to a posture at a reference acquisition angle to obtain a mapped posture, wherein the reference acquisition angle is an acquisition angle when the image sensor acquires a preset reference image;

[0031] Determine a posture deviation between the mapped posture and a preset reference posture corresponding to the operation object, generate a correction action sequence according to the posture deviation, and control the robot to execute the correction action sequence on the operation object.

[0032] In one embodiment, before the step of mapping the actual posture to a posture at a reference acquisition angle to obtain a mapped posture, the method further includes:

[0033] Acquiring an image of the external environment by the image sensor according to a preset period to obtain a plurality of periodic images, and determining an offset image from the plurality of periodic images, wherein a position of the operating object in the offset image is different from a position of the operating object in a preset initial frame image;

[0034] If the proportion of the offset image in the multiple frames of the periodic image is greater than or equal to a preset proportion and the number of the offset images is greater than or equal to a preset number, parsing the offset image of each frame to obtain the offset posture of the operation object, fitting each of the offset postures to obtain a fitting posture, determining the posture with the highest similarity to the fitting posture among the offset postures as the target posture, and determining the offset image corresponding to the target posture as the reference image;

[0035] If the image proportion is smaller than the preset proportion and / or the number of the offset images is smaller than the preset number, the initial frame image is determined as the reference image.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the robot control method described above are implemented.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the robot control method as described above.

[0038] A robot control method proposed in the present application is applied to a robot equipped with an image sensor. The robot control method receives user instructions and, based on the image sensor, collects a first image of the robot's external environment; parses the first image to obtain user posture data, and determines an operation object from the external environment based on the user posture data; parses the user instructions to obtain instruction-indicated operations, and generates a robot action sequence based on the instruction-indicated operations; and controls the robot to execute the robot action sequence on the operation object.

[0039] This application uses environmental images and user posture data collected by image sensors to clarify the operation object, and then determines the specific robot action sequence based on the instruction instructions, avoiding the problem of the robot performing tasks incorrectly or being unable to perform tasks due to ambiguous instructions, thereby improving the robot's task execution efficiency when facing ambiguous instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0042] Figure 1 A schematic diagram of a flow chart of the first embodiment of the robot control method of the present application;

[0043] Figure 2 A schematic diagram of a flow chart of the second embodiment of the robot control method of this application;

[0044] Figure 3 A flowchart of the third embodiment of the robot control method of this application is provided;

[0045] Figure 4 A schematic diagram of the application flow of the robot control method provided in one embodiment of the present application;

[0046] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the robot control method in the embodiment of the present application.

[0047] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0049] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0050] The main solution of the embodiment of the present application is: receiving user instructions and collecting a first image of the external environment in which the robot is located based on an image sensor; parsing the first image to obtain user posture data, and determining an operation object from the external environment based on the user posture data; parsing the user instructions to obtain the instruction-indicated operation, and generating a robot action sequence based on the instruction-indicated operation; controlling the robot to execute the robot action sequence on the operation object.

[0051] In this embodiment, for ease of description, the following description is made with a robot as the execution subject.

[0052] In complex actual working environments, the instructions issued by operators are not always strict and precise fixed sentences, but may contain ambiguous expressions that require the context of the scene to be accurately understood. When robots receive such instructions, it is usually difficult to accurately infer the operator's true intentions, and thus cannot perform the task correctly, affecting the robot's task execution efficiency and making it difficult for the robot to meet diverse and intelligent work needs.

[0053] The present application provides a solution that uses the first image captured by the image sensor and the user posture data to clarify the operation object, and then determines the specific robot action sequence based on the instruction operation, avoiding the problem of the robot performing the task incorrectly or being unable to perform the task due to ambiguous instructions, thereby improving the robot's task execution efficiency when facing ambiguous instructions.

[0054] It should be noted that the execution entity of this embodiment can be a computing service device with data processing, network communication, and program execution capabilities, which is connected to the robot, such as a tablet computer, personal computer, mobile phone, etc., or a robot. The following uses a robot as an example to illustrate this embodiment and the following embodiments.

[0055] Based on this, the embodiment of the present application provides a robot control method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the robot control method of the present application.

[0056] In this embodiment, the robot is equipped with an image sensor, and the robot control method includes steps S10 to S40:

[0057] Step S10: receiving a user instruction and capturing a first image of the external environment of the robot based on an image sensor.

[0058] User commands are information carriers that convey users' intentions to the robot and require it to perform specific tasks. User commands can take many forms, which are not limited here, such as voice commands, touch commands, and commands sent from external devices via wireless communication methods such as Bluetooth.

[0059] The robot is equipped with an image sensor. The placement and number of these image sensors are not restricted. Upon receiving a user command, the robot activates the image sensor, which captures the robot's surrounding environment based on pre-set parameters such as resolution and frame rate to produce a first image. This first image can be a point cloud image, a depth map, a planar image, or other type, depending on the type of image sensor and is not otherwise restricted.

[0060] In this embodiment, by using the image sensor to capture the first image, basic data support is provided for the subsequent robot to understand the environment, identify the operation object and perform the task, thereby enhancing the robot's ability to perceive the environment and the accuracy of task execution.

[0061] Step S20 , analyzing the first image to obtain user posture data, and determining an operation object from the external environment based on the user posture data.

[0062] The user posture data is used to characterize the user's body posture and is a collection of information about the user's body posture and movements. Specifically, it may include data characterizing the user's limb movements and body orientation. In this embodiment, the method of parsing the first image to obtain the user posture data is not limited. In one feasible implementation, the user posture data may be extracted through a neural network model. For example, the collected first image is input into a user posture recognition model trained with a large number of human posture samples, and the model outputs the user posture data. In another feasible implementation, the user posture data may also be extracted based on target detection and image analysis algorithms, for example.

[0063] There is no limitation on the method of determining the operation object from the external environment based on user posture data, and it can be implemented by image analysis or target detection. For example, in a feasible implementation method, the user posture data may include head posture data and user limb data. The specific implementation process of determining the operation object based on the user posture data may be: converting the head posture data into a gaze direction vector, combining the eye tracking algorithm to calculate the line of sight focus area, and marking the objects with built-in confidence values in the focus area exceeding a first threshold as candidate objects; identifying the chain structure of hand key points based on user limb data, and when it is detected that the angle between the index finger extension direction vector and the palm normal vector is less than a second threshold, establishing a pointing ray along the forearm axis; performing spatial orthogonal calculation based on the environmental map and the pointing ray, determining the nearest neighbor object among the candidate objects as the operation object.

[0064] It can be understood that by analyzing the first image to obtain user posture data, the robot can understand the user's intention and accurately determine the operation object, laying the foundation for the subsequent accurate execution of the user's desired operations, thereby improving the pertinence and effectiveness of the robot's operations.

[0065] Step S30: parsing the user instruction to obtain the instruction-indicated operation, and generating a robot action sequence based on the instruction-indicated operation.

[0066] A command-indicated operation is information about the action the user expects the robot to perform, derived from user commands using natural language processing technology. It should be noted that user commands can be either explicit or ambiguous. An explicit command includes a specific executable object and a specific action. For example, in the user command "Close the window," "close" is a specific executable action, and the window is a specific executable object. "Close the window" is an explicit command. In contrast, in "Tidy up here," "tidy up" is an ambiguous operation without a clear action, and "here" also refers to an unclear object. "Tidy up here" is an explicit command. It is understood that when a user command is explicit, the command-indicated operation is a specific executable action.

[0067] Specifically, to parse the user instruction to obtain the instruction-indicated operation, semantic analysis, grammatical parsing, and other techniques from the field of natural language processing can be used to process the instruction text entered by the user and extract key operation information, such as using a semantic analysis algorithm based on a word vector model, which is not limited here. For example, in one feasible implementation, the text-based user instruction can be segmented, and a semantic analysis algorithm can be used to determine the verb and instruction object, and the verb and instruction object can be determined as the instruction-indicated operation.

[0068] Generating a robot action sequence based on instruction-indicated operations can be done by using the instruction-indicated operations as action constraints to determine the robot action sequence, or by combining the instruction-indicated operations and the user's historical behavior data to determine the action constraints to generate the action sequence. There is no limitation here.

[0069] In step S40 , the robot is controlled to execute a robot action sequence on the operation object.

[0070] The robot's central control unit generates and sends corresponding control signals to the robot's actuators based on the robot's action sequence and the operation object, so that the robot can execute the robot action sequence on the operation object.

[0071] Furthermore, in a feasible implementation, during the execution of an operation, a sudden failure of the actuator may occur, resulting in operation failure. A backup actuator can be deployed on the robot. When the main actuator fails, the backup actuator automatically switches and takes over the operation to ensure the smooth completion of the task.

[0072] In one feasible implementation, step S20, the step of parsing the first image to obtain user posture data, includes:

[0073] Step S201: Input the first image into a pre-trained posture extraction network to obtain joint points corresponding to each joint of the user and the head rotation angle, wherein the head rotation angle includes the head pitch angle, the head yaw angle and the head roll angle.

[0074] The pre-trained posture extraction network is a neural network model that has been trained with a large amount of data and has the ability to extract human posture-related information from images. For example, a network can be constructed with HRNet-W48 as the backbone network, followed by a joint point heat map prediction branch, a three-dimensional coordinate regression branch, and a head posture estimation branch.

[0075] The acquired first image is input into the pre-trained posture extraction network to obtain the joint points and head rotation angles corresponding to each joint of the user. For example, in a feasible implementation, the posture extraction network is a network constructed with HRNet-W48 as the backbone network, followed by a joint point heat map prediction branch, a three-dimensional coordinate regression branch, and a head posture estimation branch. In this implementation, the backbone network HRNet-W48 in the network extracts features from the image, and different branches perform corresponding tasks respectively. The joint point heat map prediction branch outputs a Gaussian heat map of the joint point, and the position of the joint point can be determined by analyzing the heat map; the three-dimensional coordinate regression branch outputs a multidimensional vector through a fully connected layer, and converts the two-dimensional joint point coordinates into three-dimensional space coordinates in combination with depth information; the head posture estimation branch uses a 6D rotation representation to output a head rotation matrix, and then decomposes the rotation matrix into head pitch angle, head yaw angle, and head roll angle. Among them, joint points are the position identification points corresponding to the joints of the human body in the image. Taking the COCO key point definition standard (Microsoft Common Objects in ContextKeypoint Definition Standard) as an example, they include key points such as the left and right wrists, left and right ankles, and the tips of the left and right middle fingers. The head rotation angle is used to describe the orientation angle of the human head in three-dimensional space. Among them, the head pitch angle indicates the degree of tilt of the head up and down, the head yaw angle indicates the left and right rotation angle of the head, and the head roll angle indicates the rotation angle of the head around its own central axis. Implementation method flow:

[0076] Step S202: Determine the joint points and the head rotation angle as user posture data.

[0077] The extracted joint posture data and head rotation angle are determined as user posture data, providing the data foundation for subsequent determination of the manipulated object based on this user posture data. Compared to single joint posture data or head posture information, user posture data can more accurately reflect the user's overall movements and focus of attention, improving the robot's accuracy in determining the manipulated object.

[0078] In one feasible implementation, step S20, the step of determining the operation object from the external environment based on the user gesture data, includes:

[0079] Step S203: Determine the user's facial orientation based on the head rotation angle, determine the first projection ray with the user's head center point as the starting point and the facial orientation as the first projection direction, construct a first cone space model with the first projection ray as the central axis, and project the first cone space model onto the first image to obtain the line of sight focus area.

[0080] The user's facial orientation is calculated based on the head rotation angle data. Facial orientation is the direction the front of the head points. For example, assuming the head yaw angle is β, the pitch angle is α, and the roll angle is γ, the direction vector of the facial orientation in the spatial coordinate system can be calculated using the corresponding rotation matrix. The user's head center point can be determined by performing geometric calculations on key head points such as the tip of the nose, ears, and back of the head. For example, by obtaining the coordinates of the nose tip (x1, y1, z1), the coordinates of the left ear tip (x2, y2, z2), the coordinates of the right ear tip (x3, y3, z3), and the coordinates of the center of the back of the head (x4, y4, z4), a specific geometric algorithm (such as the center of mass algorithm) is used to calculate the coordinates of the head center point (x0, y0, z0).

[0081] The first projection ray is determined with the head center as the starting point and the facial orientation as the direction. The first cone space model is constructed based on the preset opening angle and effective length. For example, in a home scenario, the opening angle is set to 30°±5° and the effective length is set to 1.8m. This can be configured according to actual needs and is not limited here.

[0082] The constructed first cone space model is projected onto the first image through perspective projection and other algorithms to obtain the visual focus area. The projection process can use a pinhole camera model to convert the points in the cone space model into image pixel coordinates according to the projection formula, and then determine the projection area on the first image.

[0083] Step S204: determine a second projection ray starting from the end joint point of each joint point and taking the pointing direction of the limb where the end joint point of the limb is located as the second projection direction, construct a second cone space model with the second projection ray as the central axis, and project the second cone space model onto the first image to obtain a limb indication area.

[0084] The end joints of the limbs refer to the joints at the farthest ends of the human limbs. Taking the COCO key point definition standard as an example, the end joints of the upper limbs are such as the left and right hands, and the end joints of the fingers are such as the tips of the middle fingers of the left and right hands; the end joints of the lower limbs are such as the left and right ankles, and the end joints of the feet are such as the left and right toes.

[0085] First, calculate the pointing direction of the limb. Different calculation methods can be used for different limbs. For example, for the upper limbs, it is determined according to the vector rules from the elbow joint to the wrist and from the wrist to the middle fingertip; for the lower limbs, the weighted average direction of the historical movement trajectory of the foot is used for calculation. There are no specific restrictions here.

[0086] A second projection ray is determined, starting from the distal joint of the limb and oriented in the limb's direction as the second projection direction. A second cone-space model is constructed based on a preset opening angle and effective length. The second cone-space model is projected onto the first image using an algorithm such as perspective projection to obtain the limb's indicated area. The projection process can utilize methods such as a pinhole camera model to convert points in the cone-space model into image pixel coordinates to determine the projected area, though this is not a limitation.

[0087] Step S205 : determining an area in the external environment corresponding to the sight focus area and the body indication area as an area to be operated, and determining an object in the area to be operated as an operation target.

[0088] The area in the external environment corresponding to the visual focus area and the limb indication area is determined as the area to be operated. In one feasible implementation manner, all areas in the external environment corresponding to the visual focus area and the limb indication area can be determined as the area to be operated; in another feasible implementation manner, the overlapping area of all areas in the external environment corresponding to the visual focus area and the limb indication area can be determined as the area to be operated.

[0089] The system uses a preset object recognition algorithm to identify objects within the area to be operated and determines them as the target of the operation. If there are multiple objects, the system further screens and determines the target of the operation through multimodal verification, for example, by combining voice command information with the object's functional attribute labels to determine the final target of the operation.

[0090] In one feasible implementation, step S30, the step of generating a robot action sequence based on the instruction instruction operation, includes:

[0091] Step S301: If the instruction indicates that the operation is a clear pointing action, the current posture of the operation object is determined based on the first image, and the preset reference posture corresponding to the operation object is obtained. The trajectory planning is performed with the current posture as the initial value, the reference posture as the target value, and the instruction indication operation as the action constraint condition to generate the robot action sequence.

[0092] The current pose of the operation object is the position and posture information of the operation object at the current moment. The preset reference pose is a pre-set standard placement pose of the operation object, which can be obtained through a preset template or online.

[0093] Determine whether the instruction-indicated operation is a clear-pointing action. The specific judgment method is not limited here. In one feasible implementation, it can be determined by detecting whether the instruction-indicated operation is an action in the robot's preset action library. If the instruction-indicated operation is an action in the preset action library, it is determined to be a clear-pointing action. In another feasible implementation, it can be determined by detecting whether the instruction-indicated operation is an action with a clear meaning. If so, it is determined to be a clear-pointing action. That is, a clear-pointing action is an action that can be executed by a robot with a clear meaning, such as cleaning, placing, and taking, while ambiguous and ambiguous operations such as tidying up and fiddling are not clear-pointing actions.

[0094] If the instruction indicates an explicit pointing action, the first image is used to determine the current posture of the operation object through target detection and posture estimation algorithm, and the preset reference posture of the operation object is obtained. The current posture is used as the initial value, the reference posture is used as the target value, and the instruction indicates the operation as the action constraint condition for trajectory planning. According to the planned trajectory, a robot action sequence is generated, and the trajectory is converted into a motion instruction sequence for each joint of the robot. The specific trajectory planning algorithm is not limited here.

[0095] It can be understood that when the instruction is a clear pointing action, the first image data can be used to accurately obtain the current posture of the operation object, and the trajectory planning can be performed in combination with the preset reference posture, so that the robot can perform actions according to clear goals and constraints, thereby improving the accuracy and efficiency of the operation.

[0096] Step S302: If the instruction indicates that the operation is not a clear pointing action, the user's historical behavior associated with the operation object is determined from the user's historical behavior data, and the current posture is used as the initial value, the reference posture is used as the target value, and the user's historical behavior is used as the action constraint condition to perform trajectory planning to generate a robot action sequence.

[0097] User historical behavior data records past user actions on an object. For example, a user may have repeatedly performed the "pick up a cup" operation. The time of these actions, the cup's identifier, the robot's motion trajectory, and the type of operation constitute this historical behavior data. User historical behavior is the user's action patterns associated with the current action object, extracted from this historical behavior data. For example, if historical data analysis reveals that a user always places a cup in a specific location after picking it up, this is considered a user historical behavior.

[0098] When it is determined that the instruction indicates an action that is not a clear pointing action, the historical behavior associated with the current operation object is filtered out from the stored user historical behavior data. The trajectory is planned with the current posture as the initial value, the reference posture as the target value, and the user's historical behavior as the action constraint condition. The robot action sequence is generated according to the planned trajectory.

[0099] It is understandable that when the instructions are not clearly directed to actions, by analyzing the user's historical behavior data, it is possible to discover the user's operating habits and patterns, and use them as action constraints for trajectory planning, so that the robot's actions are more in line with the user's expectations, improving the user experience, and using historical behavior data for planning, reducing blindness in the case of unclear instructions, improving the rationality and accuracy of the robot's actions, and enhancing the robot's intelligence and adaptability.

[0100] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 2 After the step S10 of receiving the user instruction, the method further includes:

[0101] Step S50 , detecting whether the user instruction is a clear-pointing instruction; wherein a clear-pointing instruction is an instruction containing a specific object and a specific action.

[0102] After receiving the user's instructions, the robot is tested to see if they are clearly directed instructions. If they are clearly directed instructions, the robot will enter the direct parsing and execution process; if not, the robot will be guided to understand the user's intentions through environmental perception.

[0103] The specific process of detecting whether the user instruction is an explicit pointing instruction is not limited here. For example, in one feasible implementation, it can be to detect whether the object and action indicated by the user instruction have a unique and clear meaning. If so, it is determined to be an explicit pointing instruction. In another feasible implementation, the detection can be based on the semantic map and the user instruction, by detecting whether the object in the user instruction is an object with a unique semantic annotation in the semantic map and detecting whether the operation in the user instruction has a unique meaning. If so, it is determined to be an explicit pointing instruction.

[0104] Step S60: If the user instruction is not an explicit pointing instruction, the step of collecting a first image of the external environment of the robot based on the image sensor and subsequent steps are executed.

[0105] When the user command is judged to be a non-explicitly directed command, the robot will understand the user's intention through environmental perception, and perform subsequent steps such as user posture analysis and operation object determination to determine the execution of the task. The details will not be repeated here.

[0106] Step S70: If the user instruction is a clear pointing instruction, the user instruction is parsed to obtain the instruction indication object and the instruction indication operation, and the target annotation that matches the instruction indication object is determined from the various semantic annotations of the preset semantic map. The entity object corresponding to the target annotation is determined as the operation object, and a robot action sequence is generated based on the instruction indication operation, and the robot is controlled to execute the robot action sequence on the operation object.

[0107] A preset semantic map is a map that associates physical spaces and objects in an environment with semantic information. It contains various physical objects in the environment and their semantic annotations. For example, a preset semantic map of a home environment would include the location of each room, the furniture within the room (such as sofas, beds, tables, etc.), and the spatial relationships between them. The preset semantic map can be constructed by a robot using image sensors or pre-configured in the robot, without limitation. It should be noted that the algorithm for constructing the semantic map can refer to known construction methods. Semantic annotation is a semantic description of each physical object in the preset semantic map, used to identify information such as the object's type, name, and location.

[0108] After the user instruction is determined to be a clear-pointing instruction, it is executed. The clear-pointing instruction is parsed using natural language processing technology to extract the instruction object and the instruction operation. Using string matching, similarity calculation and other algorithms, the instruction object is compared with the various semantic annotations in the semantic map, and the target annotation that matches the instruction object is determined from the various semantic annotations in the preset semantic map. The entity object corresponding to the target annotation is determined as the operation object, and the instruction operation is converted into a robot action sequence that can be executed by the robot. The robot is controlled to move to the corresponding position to perform the instruction operation on the operation object.

[0109] In one feasible implementation, step S50, the step of detecting whether the user instruction is an explicit directed instruction, includes:

[0110] Step S501 : performing semantic analysis on the user instruction to extract the instruction action and instruction object in the user instruction.

[0111] Obtain the instructions input by the user, perform semantic analysis on the user instructions to extract the sentence structure, namely the subject, predicate (instruction action) and object (instruction object), where the predicate is the instruction action and the object is the instruction object. For example, for the sentence "pick up the ball on the ground", it can be clearly understood through dependency syntactic analysis that "pick up" is the instruction action and "ball" is the instruction object. There is no restriction on the method of semantic analysis, for example, it can be analyzed through part-of-speech tagging, syntactic analysis, semantic role tagging, etc., which are not restricted here. It should be noted that the user instructions may be non-text instructions, so the user instructions are converted into text form before semantic analysis.

[0112] Semantic analysis can decompose the natural language instructions input by the user into instruction actions and instruction objects that are easy to understand and process, providing basic data for subsequent judgment of instruction types. By accurately extracting instruction actions and instruction objects, the robot can initially understand the user's operating intentions, improve the system's ability to parse user instructions, and lay the foundation for the correct execution of instructions.

[0113] Step S502 : If there is a unique matching annotation in each semantic annotation and the action represented by the instruction action is a clear pointing action, then the user instruction is determined to be a clear pointing instruction, wherein the matching annotation is the annotation in each semantic annotation that matches the instruction object.

[0114] After extracting the command action and command object, query the semantic annotation library of the preset semantic map. The semantic annotation library contains semantic annotation information of various common objects and actions. Check whether there is a unique matching annotation that matches the command object among all semantic annotations. The matching annotation is the annotation that semantically matches the command object among all semantic annotations. Specifically, matching can be achieved through similarity comparison and semantic comparison, which are not restricted here.

[0115] If the instruction object has only one unique matching annotation and the instruction action is an explicit targeting action, then the user instruction is determined to be an explicit targeting instruction. It can be understood that by checking the matching of semantic annotations and judging the instruction action, explicit targeting instructions can be accurately identified, improving the accuracy and reliability of instruction type judgment.

[0116] Step S503 : If there are multiple matching annotations or no matching annotation in each semantic annotation, and / or the action represented by the instruction action is not an explicit pointing action, it is determined that the user instruction is not an explicit pointing instruction.

[0117] If multiple semantic annotations match the instruction object, or if no matching annotation exists at all, indicating that the operation object exists in multiple contexts, the instruction object is ambiguous and the user instruction is not a clearly directed instruction. If the instruction action is not a clearly directed action, the user instruction is also not a clearly directed instruction. That is, if multiple matching annotations exist or no matching annotations exist in each semantic annotation, and the action represented by the instruction action is not a clearly directed action, if either of these two conditions is met, the user instruction is not a clearly directed instruction.

[0118] It can be understood that by identifying non-explicitly directed instructions, they can be avoided from being misjudged as explicitly directed instructions, thereby improving the comprehensiveness and accuracy of instruction type judgment.

[0119] In this implementation, by detecting whether a user instruction is explicit, the robot provides a basis for subsequent system operations. For explicit instructions, the robot can employ targeted and efficient processing, such as quickly identifying the action target and planning the action sequence. For ambiguous instructions, the robot can further clarify the user's intent before proceeding with the task. This detection process thus enhances the system's ability to understand and process natural language instructions, enabling the robot to better adapt to complex and diverse user instructions, and improving its practicality and intelligence in human-computer interaction scenarios.

[0120] Based on the first and / or second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the first and / or second embodiments can be referred to above and will not be described in detail. Figure 3 After the step S40 of controlling the robot to execute a robot action sequence on the operation object, the method further includes:

[0121] Step A10: Capture a second image of the external environment based on the image sensor, and analyze the second image to obtain the actual posture of the operation object.

[0122] The second image is an image of the external environment captured by the image sensor after the robot performs an action sequence on the operation object. The actual position and posture of the operation object is the actual position and posture of the operation object in real space after the robot completes the action sequence.

[0123] When the robot completes the action sequence on the operation object, it triggers the image sensor to capture a second image of the external environment, and uses image processing and computer vision algorithms to analyze the second image. The specific analysis process is not limited here. For example, in one feasible implementation, the operation object can be first located based on a deep learning target detection model, and then the posture information can be obtained through a posture estimation network.

[0124] It can be understood that by collecting the second image and analyzing it to obtain the actual posture of the operating object, the actual state of the operating object after performing the action can be obtained in real time, providing a key data basis for subsequent judgment of the action execution effect and correction.

[0125] Step A20 , mapping the actual posture to a posture at a reference acquisition angle to obtain a mapped posture, wherein the reference acquisition angle is an acquisition angle when the image sensor acquires a preset reference image.

[0126] The baseline acquisition angle is the angle used by the image sensor when capturing a preset baseline image. For example, during system initialization, the camera captures an image of the standard pose of the object at a fixed height and angle. This acquisition angle is called the baseline acquisition angle.

[0127] Determine the relevant parameters of the reference acquisition angle, and determine the actual acquisition angle of the image sensor during the second image acquisition. This is determined specifically by the internal and external parameters of the image sensor. Detailed description is omitted here, and reference may be made to existing technologies. Based on the relationship between the reference acquisition angle and the actual acquisition angle, coordinate transformation and posture conversion algorithms are used to map the actual posture to the posture at the reference acquisition angle, i.e., the mapped posture. For example, in one feasible implementation, the coordinates and posture information of the actual posture can be converted to the coordinate system corresponding to the reference acquisition angle by performing rotation and translation matrices to obtain the mapped posture.

[0128] By mapping the actual posture to the posture at the reference acquisition angle, the posture information obtained at different acquisition angles is made comparable and consistent, providing a basis for subsequent comparison with the preset reference posture and improving the accuracy of posture deviation calculation.

[0129] Step A30: determining a posture deviation between the mapped posture and a preset reference posture corresponding to the operation object, generating a correction action sequence according to the posture deviation, and controlling the robot to execute the correction action sequence on the operation object.

[0130] The difference between the mapped pose and the preset reference pose is calculated to obtain the pose deviation. The pose deviation can be divided into position deviation (that is, the coordinate difference in the X, Y, and Z directions under the world coordinate axis) and attitude deviation (that is, the rotation angle difference around the X, Y, and Z axes under the world coordinate axis).

[0131] According to the size and direction of the posture deviation, a preset motion planning algorithm is used to generate a correction action sequence. The correction action sequence is used to adjust the posture of the operating object to make it close to the robot action sequence of the reference posture. The specific motion planning algorithm is not limited here.

[0132] The robot is controlled to perform actions on the object according to the generated corrected action sequence to reduce the pose deviation and bring the pose of the object closer to the reference pose. It can be understood that by calculating the pose deviation and generating the corrected action sequence, the pose of the object after the robot performs the action can be adjusted and corrected, improving the accuracy and precision of the operation.

[0133] In this embodiment, by proposing a closed-loop feedback control mechanism, errors generated during the execution of robot actions can be corrected in real time, thereby improving the accuracy and precision of robot operations, improving the adaptability and robustness of the robot, improving the intelligence level and degree of automation of the robot, reducing manual intervention, and improving work efficiency and user experience.

[0134] In one feasible implementation, before step A10 of mapping the actual posture to the posture at the reference acquisition angle to obtain the mapped posture, the method further includes:

[0135] In step A01, an image sensor is used to capture images of the external environment according to a preset period to obtain a multi-frame periodic image, and an offset image is determined from the multi-frame periodic image, wherein the position of the operating object in the offset image is different from the position of the operating object in the preset initial frame image.

[0136] The preset period is a pre-set image acquisition time interval, which can be set according to actual needs. The image sensor continuously acquires images of the external environment according to the preset period, and the obtained image is the periodic image.

[0137] The initial frame image is a pre-set image used as a reference, and is usually an image captured before the operation begins. In this embodiment, a posture analysis is performed on each frame period image, and the posture of the operating object in each frame period image is compared with the posture in the initial frame image. If the posture of the operating object in a certain frame period image is different from the posture in the initial frame image, then the frame image is determined to be an offset image. There is no limitation on the method of posture extraction, and specifically, methods such as feature matching, target detection, and posture estimation can be used to obtain the posture information of the operating object. For example, in a feasible embodiment, a scale-invariant feature transformation algorithm can be used to extract feature points in the image. Feature matching is then used to determine whether the posture of the operating object has changed.

[0138] It can be understood that by collecting multi-frame periodic images according to a preset period and determining the offset image therefrom, the posture changes of the operating object can be monitored in real time, providing basic data for subsequent judgment of the state of the operating object and selection of a suitable reference image.

[0139] Step A02: If the proportion of the offset image in the multi-frame periodic image is greater than or equal to a preset proportion and the number of the offset images is greater than or equal to a preset number, the offset images of each frame are parsed to obtain the offset posture of the operation object, and each offset posture is fitted to obtain a fitted posture. The posture with the highest similarity to the fitted posture among each offset posture is determined as the target posture, and the offset image corresponding to the target posture is determined as the reference image.

[0140] The image percentage is the ratio of offset images to the total number of frames in a multi-frame image cycle. For example, if 30 frames out of a 100-frame image cycle contain offset images, the image percentage is 30%. The preset percentage is a pre-set threshold for determining whether the offset image percentage is large enough. The preset number is a pre-set threshold for determining whether the number of offset images is sufficient.

[0141] The image ratio of the offset image in the multi-frame periodic image is calculated, and the number of offset images is counted. If the image ratio of the offset image in the multi-frame periodic image is greater than or equal to a preset ratio and the number of offset images is greater than or equal to a preset number, it indicates that the posture of the object being operated in the user's daily usage habits is different from the posture in the initial frame image. The posture of the object being operated in the user's daily usage habits can be used as a basis for comparison for subsequent robot operations. Specifically, each frame of the offset image is parsed, and a target detection and posture estimation algorithm is used to obtain the posture of the object being operated, i.e., the offset posture; a fitting algorithm is used to fit each offset posture, and the resulting posture is also known as the fitted posture; the similarity between each offset posture and the fitted posture is calculated. Specifically, a distance measurement method such as Euclidean distance, cosine similarity, etc. can be used, but this is not limited here. The offset posture with the highest similarity is determined as the target posture, and the offset image corresponding to the target posture is determined as the reference image.

[0142] Step A03: If the image proportion is less than a preset proportion and / or the number of offset images is less than a preset number, the initial frame image is determined as the reference image.

[0143] If the image ratio is less than the preset ratio and / or the number of offset images is less than the preset number, the initial frame image will be determined as the reference image, ensuring that the robot can still have a stable reference image when the posture of the operated object does not change significantly, avoiding the problem of inaccurate reference image selection due to a small number of abnormal offset images, and improving the stability and reliability of the system.

[0144] In this embodiment, by analyzing and processing multi-frame periodic images, a suitable reference image is dynamically selected according to conditions such as the proportion and number of offset images, so as to adapt to different changes in the posture of the operation object, improve the accuracy and representativeness of the reference image, and provide a reliable reference for subsequent operation robot motion planning, posture deviation calculation, etc., thereby improving the accuracy and precision of robot operation and enhancing the stability and adaptability of the system.

[0145] For example, in order to help understand the implementation process of the robot control method obtained by combining this embodiment with the above embodiments, please refer to Figure 4 , Figure 4 A simplified flow chart of a robot control method is provided. In this embodiment, the robot is equipped with an image sensor and a laser radar. Specifically:

[0146] Step 1. Perform environmental recognition based on lidar and image sensors to build an environmental semantic map (also known as a preset semantic map).

[0147] Step 2. Receive the user instruction and determine whether the user instruction is a clear-pointing instruction; if it is not a clear-pointing instruction, go to step 3; if it is a clear-pointing instruction, go to step 4. If the user instruction is a voice instruction, the robot will recognize it and convert it into text, and detect it by comparing the text results. For example, when building a semantic map, the semantic roles of the environment will be marked. If there is a precisely directed voice input at this time, such as "Help me get a broom from the kitchen", the keywords "kitchen" and "broom" can be identified and extracted, and it can be determined to be a clear-pointing instruction; if the user instruction is "Clean up here", the operation and object indicated in the instruction are vague, and it can be determined that it is not a clear-pointing instruction.

[0148] Step 3. Understanding the intention of the instruction and planning the robot action (i.e., collecting a first image of the robot's external environment based on the image sensor; parsing the first image to obtain user posture data, and determining the operation object from the external environment based on the user posture data; parsing the user instruction to obtain the instruction-indicated operation, and generating a robot action sequence based on the instruction-indicated operation; controlling the robot to execute the robot action sequence on the operation object). The underlying intention is parsed through the content of the instruction, and the object indicated by the user (i.e., the operating object) is detected through the image captured by the image sensor. The robot action is then determined through the user behavior history. The robot action sequence is then planned based on the determined robot action. When planning the action sequence, the action is planned by comparing it with the reference posture of the operating object, and the action is adjusted in real time through visual feedback after the task is performed (i.e., a second image of the external environment is captured based on the image sensor, and the second image is parsed to obtain the actual posture of the operating object; the actual posture is mapped to the posture under the reference acquisition angle to obtain the mapped posture, wherein the reference acquisition angle is the acquisition angle when the image sensor captures the preset reference image; the posture deviation between the mapped posture and the preset reference posture corresponding to the operating object is determined, a correction action sequence is generated based on the posture deviation, and the robot is controlled to execute the correction action sequence on the operating object). Specifically, "tidy up here" is a vague intention, which is analyzed: "tidy up" = tidying action, the finger points to the dining table, "here" = the dining area, and then a comparative test is performed, and it is found that the stool is not placed inside the table. Referring to the user's habit records, it is found that the stool is placed inside the table in a certain posture many times, so the conclusion is: put the stool inside the table.

[0149] In this embodiment, when the semantic map is first constructed, the environment is recorded as a comparison frame (i.e., a reference image). A learning function is added here to record and update this comparison frame. For example, in the initial frame (i.e., the initial image), the stool is placed inside the dining table, but later the customer wants to place the stool outside the dining table and performs the operation multiple times. After recording this action, the comparison frame will be updated (i.e., based on the image sensor, images of the external environment are collected according to a preset period to obtain multi-frame periodic images, and an offset image is determined from the multi-frame periodic images, wherein the posture of the operation object in the offset image is different from the posture of the operation object in the preset initial frame image; if the image proportion of the offset image in the multi-frame periodic image is greater than or equal to a preset proportion and the number of offset images is greater than or equal to a preset number, then the offset images of each frame are parsed to obtain the offset posture of the operation object, and the offset postures are fitted to obtain a fitted posture, and the posture with the highest similarity to the fitted posture among the offset postures is determined as the target posture, and the offset image corresponding to the target posture is determined as the reference image; if the image proportion is less than the preset proportion and / or the number of offset images is less than the preset number, then the initial frame image is determined as the reference image).

[0150] Step 4. Parse user instructions and perform action planning (that is, parse user instructions to obtain the instruction object and instruction operation, determine the target annotation that matches the instruction object from the various semantic annotations of the preset semantic map, determine the entity object corresponding to the target annotation as the operation object, generate a robot action sequence based on the instruction operation, and control the robot to execute the robot action sequence on the operation object).

[0151] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the robot control method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0152] This application also provides a robot control device, which is applied to a robot equipped with an image sensor. The robot control device provided by this application, employing the robot control method of the aforementioned embodiment, can resolve the technical issue of low task execution efficiency when the robot receives ambiguous instructions. Compared to the prior art, the beneficial effects of the robot control device provided by this application are the same as those of the robot control method provided by the aforementioned embodiment, and the other technical features of the robot control device are the same as those disclosed in the aforementioned embodiment, and are not further described here.

[0153] The present application provides a robot control device. The robot control device can be an electronic device communicatively connected to a robot, or it can be a robot, without limitation. The robot control device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the robot control method described in the first embodiment.

[0154] Reference below Figure 5 , which shows a schematic diagram of the structure of a robot control device suitable for implementing the embodiments of the present application. The robot control device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The robot control device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0155] like Figure 5 As shown, the robot control device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to programs stored in a read-only memory 1002 or programs loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the robot to control its motion sequences. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: an input device 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the robot control device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a robot control device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented or have alternatively.

[0156] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0157] The robot control device provided in this application, employing the robot control method of the aforementioned embodiment, can resolve the technical issue of low task execution efficiency when a robot receives ambiguous instructions. Compared to the prior art, the beneficial effects of the robot control device provided in this application are the same as those of the robot control method provided in the aforementioned embodiment. Other technical features of this robot control device are the same as those disclosed in the aforementioned embodiment and are not further elaborated here.

[0158] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0159] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0160] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, and the computer-readable program instructions are used to execute the robot control method in the above-mentioned embodiment.

[0161] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0162] The computer-readable storage medium may be included in the robot control device, or may exist independently without being assembled into the robot control device.

[0163] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the robot control device, the robot control device implements the various embodiments of the robot control method.

[0164] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0165] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0166] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0167] The computer-readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned robot control method. This computer-readable storage medium can address the technical issue of low task execution efficiency when a robot receives ambiguous instructions. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the robot control method provided in the aforementioned embodiments, and are not further elaborated here.

[0168] The present application also provides a computer program product, comprising a computer program, which implements the steps of the robot control method as described above when executed by a processor.

[0169] The computer program product provided in this application can solve the technical problem of low task execution efficiency when a robot receives ambiguous instructions. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the robot control method provided in the above embodiment, and will not be elaborated here.

[0170] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A robot control method, characterized in that: The robot control method is applied to a robot, wherein the robot is equipped with an image sensor, and the robot control method includes: receiving a user instruction and collecting a first image of the external environment of the robot based on the image sensor; parsing the first image to obtain user posture data, and determining an operation object from the external environment based on the user posture data; Parsing the user instruction to obtain an instruction-indicated operation, and generating a robot action sequence based on the instruction-indicated operation; The robot is controlled to execute the robot action sequence on the operation object.

2. The robot control method according to claim 1, wherein: The step of parsing the first image to obtain user posture data includes: Inputting the first image into a pre-trained posture extraction network to obtain joint points corresponding to each joint of the user and a head rotation angle, wherein the head rotation angle includes a head pitch angle, a head yaw angle, and a head roll angle; The joint points and the head rotation angle are determined as user posture data.

3. The robot control method according to claim 2, wherein: The step of determining an operation object from the external environment based on the user gesture data comprises: Determining the user's facial orientation based on the head rotation angle, determining a first projection ray with the user's head center point as a starting point and the facial orientation as a first projection direction, constructing a first cone space model with the first projection ray as a central axis, and projecting the first cone space model onto the first image to obtain a sightline focus area; Determining a second projection ray from a limb end joint point among the joint points as a starting point and a pointing direction of the limb where the limb end joint point is located as a second projection direction, constructing a second cone space model with the second projection ray as a central axis, and projecting the second cone space model onto the first image to obtain a limb indication area; An area in the external environment corresponding to the sight focus area and the body indication area is determined as an area to be operated, and an object in the area to be operated is determined as an operation target.

4. The robot control method according to claim 1, wherein: The step of generating a robot action sequence based on the instruction instruction operation includes: If the instruction indicates an explicit pointing action, determining the current posture of the operation object based on the first image, obtaining a preset reference posture corresponding to the operation object, and performing trajectory planning to generate a robot motion sequence using the current posture as an initial value, the reference posture as a target value, and the instruction indicates an action constraint condition; If the instruction indicates that the operation is not the clear pointing action, the user's historical behavior associated with the operation object is determined from the user's historical behavior data, and the trajectory planning is performed to generate the robot action sequence with the current posture as the initial value, the reference posture as the target value, and the user's historical behavior as the action constraint condition.

5. The robot control method according to claim 1, wherein: After the step of receiving the user instruction, the method further includes: Detecting whether the user instruction is a clear-pointing instruction; wherein the clear-pointing instruction is an instruction including a specific object and a specific action; If the user instruction is not the explicit pointing instruction, executing the step of capturing the first image of the external environment of the robot based on the image sensor and subsequent steps; If the user instruction is the explicit pointing instruction, the user instruction is parsed to obtain the instruction indication object and the instruction indication operation, a target annotation matching the instruction indication object is determined from the various semantic annotations of the preset semantic map, the entity object corresponding to the target annotation is determined as the operation object, a robot action sequence is generated based on the instruction indication operation, and the robot is controlled to execute the robot action sequence on the operation object.

6. The robot control method according to claim 5, characterized in that: The step of detecting whether the user instruction is a clear pointing instruction includes: Performing semantic analysis on the user instruction to extract instruction actions and instruction objects in the user instruction; If there is a unique matching annotation in each of the semantic annotations and the action represented by the instruction action is a clear pointing action, then the user instruction is determined to be a clear pointing instruction, wherein the matching annotation is the annotation in each of the semantic annotations that matches the instruction object; If there are multiple matching annotations or no matching annotation in each of the semantic annotations, and / or the action represented by the instruction action is not the explicit pointing action, it is determined that the user instruction is not the explicit pointing instruction.

7. The robot control method according to any one of claims 1 to 6, characterized in that: After the step of controlling the robot to perform the robot action sequence on the operation object, the method further includes: Capturing a second image of the external environment based on the image sensor, and analyzing the second image to obtain an actual posture of the operation object; Mapping the actual posture to a posture at a reference acquisition angle to obtain a mapped posture, wherein the reference acquisition angle is an acquisition angle when the image sensor acquires a preset reference image; Determine a posture deviation between the mapped posture and a preset reference posture corresponding to the operation object, generate a correction action sequence according to the posture deviation, and control the robot to execute the correction action sequence on the operation object.

8. The robot control method according to claim 7, wherein: Before the step of mapping the actual posture to the posture under the reference acquisition angle to obtain the mapped posture, the method further includes: Acquiring images of the external environment by the image sensor according to a preset period to obtain a plurality of periodic images, and determining an offset image from the plurality of periodic images, wherein a position of the operating object in the offset image is different from a position of the operating object in a preset initial frame image; If the proportion of the offset image in the multiple frames of the periodic image is greater than or equal to a preset proportion and the number of the offset images is greater than or equal to a preset number, parsing the offset image of each frame to obtain the offset posture of the operation object, fitting each of the offset postures to obtain a fitting posture, determining the posture with the highest similarity to the fitting posture among the offset postures as the target posture, and determining the offset image corresponding to the target posture as the reference image; If the image proportion is smaller than the preset proportion and / or the number of the offset images is smaller than the preset number, the initial frame image is determined as the reference image.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the robot control method according to any one of claims 1 to 8 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the robot control method according to any one of claims 1 to 8 are implemented.

Images