A mobile robot intelligent control method

By setting up vision sensors on the joints of industrial robots and using speech recognition technology, the environmental semantic map is built, which solves the problem of reduced work accuracy caused by robot movement blind spots and occlusion, improves the safety and stability of movement, and ensures the correctness of operations and integrity.

CN117798925BActive Publication Date: 2024-12-03TAER (SHENZHEN) INTELLIGENT ROBOT TECH CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202410085083.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-20
Publication Date
2024-12-03
Estimated Expiration
2044-01-20

AI Technical Summary

Technical Problem

In the prior art, industrial robots may generate blind spots and shades during movement, resulting in reduced working accuracy and the robotic arm cannot operate effectively in a controllable position.

Method used

By setting up vision sensors on the robot joints, obtaining environmental data and building an environmental semantic map, combining speech recognition technology, determining the operable area and movement path of the robot can be ensured to execute operation instructions safely and effectively.

Benefits of technology

It improves the safety and stability of robot movement, ensures the correctness and integrity of operations, and avoids the problem of reduced working accuracy caused by blind spots and shading.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117798925B_ABST
    Figure CN117798925B_ABST
Patent Text Reader

Abstract

The present application discloses a mobile robot intelligent control method, which relates to the field of robot technology, and comprises: step S1, based on the visual sensor arranged on the robot joint, obtaining the environmental data of the operable area in the visual sensor; step S2, based on the environmental data, determining that the current operable area is in an executable state, and recognizing that the robot can move and can execute the currently running operation, then the operable area is determined to be in an executable state, otherwise the operable area is determined to be in a non-executable state; step S3, recognizing the currently sent voice command, determining the operable area based on the voice command, and determining the robot's operation command based on the recognition result of the target in the operable area; the robot's adaptability in complex environments can be improved, and the robot's work efficiency can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of robot technology, and in particular to an intelligent control method for a mobile robot. Background Art

[0002] With the increasing demand for upgrading of labor-intensive industries, industrial robots are increasingly being used in manufacturing and other fields. Industrial robots can maintain stable and reliable operation and can efficiently complete manufacturing tasks according to pre-set programs. When industrial robots are operating, they usually need to obtain the surrounding environment in order to control the robot according to the surrounding environment.

[0003] The method of obtaining the surrounding environment in the prior art is usually to install a camera in the surrounding environment. However, the robot may move while working, and the surrounding environment may have blind spots due to the movement of the robot. At the same time, the robot may be obscured due to the moving route when working, which reduces the original working accuracy and makes the robotic arm unable to perform effective operations in the controllable position. Summary of the invention

[0004] The embodiments of the present application provide an intelligent control method for a mobile robot, thereby solving the technical problem in the prior art that the robot movement is difficult due to blind spots in the robot movement, and improving the safety and stability of the robot movement.

[0005] The present application provides a mobile robot intelligent control method, including:

[0006] Step S1, based on a visual sensor disposed on a robot joint, obtaining environmental data of an operable area in the visual sensor;

[0007] Step S2, based on the environmental data, determining that the current operable area is in an executable state, recognizing that the robot can move and can perform the currently running operation, then determining that the operable area is in an executable state, otherwise determining that the operable area is in a non-executable state;

[0008] Step S3, identifying the voice command currently sent, determining the operable area based on the voice command, and determining the operation command of the robot based on the recognition result of the target in the operable area.

[0009] Step S3 also includes the following implementation contents:

[0010] Step S31, after recognizing the corresponding voice command, constructing an environmental semantic map of the operable area based on the data collected by the robot;

[0011] Step S32, determining a movement path of the robot corresponding to the target object based on the environment semantic map;

[0012] Step S33, determining the operation instructions of the robot according to the reception status of the voice instructions.

[0013] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0014] By setting visual sensors on the joints of the robot to obtain environmental data, the robot can identify the operable area around it and build an environmental semantic map based on the collected data, so that the robot can understand the meaning of its surroundings. According to the environmental data, the location of the target object and the robot's posture information, the robot can determine its movement path to ensure that it reaches the target location safely and efficiently.

[0015] Through voice recognition technology, the robot can understand the user's voice commands and perform corresponding operations based on these commands. When receiving continuous voice commands, the robot can dynamically determine its operation instructions based on the time interval between commands and the consistency of operations. When it is found that the actual operation does not match the target operation, the robot can generate prompt information and adjust the operation instructions it executes to ensure the correctness and safety of the operation.

[0016] The robot can identify target objects during movement and adjust the corresponding position after movement according to the corresponding position of the robot, thereby verifying the correctness and completeness of the operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a flow chart of an intelligent control method for a mobile robot;

[0018] Figure 2 Schematic diagram of the process of step S3 of a mobile robot intelligent control method. DETAILED DESCRIPTION

[0019] To facilitate the understanding of the present invention, the present application will be described more comprehensively below with reference to the relevant drawings; the drawings show preferred embodiments of the present invention, but the present invention can be implemented in many different forms and is not limited to the embodiments described herein; on the contrary, the purpose of providing these embodiments is to enable a more thorough and comprehensive understanding of the disclosed content of the present invention.

[0020] It should be noted that the terms “vertical”, “horizontal”, “up”, “down”, “left”, “right” and similar expressions used in this document are only for illustrative purposes and do not represent the only implementation method.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which the present invention belongs; the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the term "and / or" used herein includes any and all combinations of one or more related listed items. Example

[0022] like Figure 1 As shown, the present application provides a mobile robot intelligent control method comprising:

[0023] Step S1, based on the visual sensor set on the robot joint, obtain the environmental data of the operable area in the visual sensor; the environmental data includes sensor data and image data, the sensor data includes physical quantities such as temperature, humidity, pressure, light intensity, magnetic field intensity, etc., these data can help the robot understand the physical characteristics of the surrounding environment; the image data includes pictures or videos of the surrounding environment collected by the camera, these data can help the robot recognize objects, judge distances and obstacles and other information; the operable area is the area where the current robot can perform operations after relative voice analysis.

[0024] Step S2, based on the environmental data, determine that the current operable area is in an executable state, identify that the robot can move and can perform the currently running operation, then determine that the operable area is in an executable state, otherwise determine that the operable area is in a non-executable state.

[0025] If the operable area is in an executable state, the device moves according to the current image recognition and executes the operation instruction when it reaches the operable area.

[0026] If the operable area is in a non-executable state, the moving path of the robot is adjusted according to the area information recognized by the image.

[0027] Step S3, identifying the voice command currently sent, determining the operable area based on the voice command, and determining the operation command of the robot based on the recognition result of the target in the operable area.

[0028] When the robot receives a voice command, it identifies the first target in the operation content according to the operation content corresponding to the voice command. The first target is the priority target to be executed after language analysis. For example, the robot currently needs to perform operations in two places. At this time, according to the results of the current language analysis, the area with a larger voice analysis weight is selected as the first target for the robot to execute, and based on the distance between the robot and the target, the target that is currently easiest to execute is selected as the first target.

[0029] Preferably, identifying the operable area also includes determining a first position corresponding to the movement of the robot based on the displacement of the robot; the first position indicates the position to which the robotic arm moves after the robot moves; based on an operation instruction executed by the robot, determining a second position corresponding to the first position, the second position being used to indicate the position to which the robotic arm moves after the robot performs the operation; and comparing the relationship between the first position and the second position to determine the distance moved before and after the movement of the robotic arm.

[0030] Preferably, the current robot recognition content also includes identifying the target object, determining the target position of the robot's movement based on the target object, the information corresponding to the target object includes the cargo code, cargo shape, cargo size, cargo type, cargo weight, and adjusting the robot's working mode based on the target object.

[0031] The present invention sets corresponding information to determine that after the robot identifies the operable area and the corresponding target object, it can select a working mode according to voice recognition, thereby improving the effect of the robot's operation, and adjust the work of the robotic arm according to the identified position during operation, thereby improving the ability of the robotic arm to locate and process goods. Example

[0032] After the robot identifies the working area and the corresponding items, due to the different sizes, types, and weights of different items, some goods will affect the robot's visual image acquisition when being transported, resulting in the robot being unable to observe the surrounding environment in time when moving, making it impossible for the robot to avoid obstacles in time.

[0033] Preferably, step S3 also includes the following implementation content:

[0034] Step S31, after recognizing the corresponding voice command, construct an environmental semantic map of the operable area according to the data collected by the robot; the voice command is used to determine the content of the work and the moving area of ​​the robot, and based on the semantic recognition of the voice command, the operation command executed by the robot after the semantic recognition is determined; the environmental semantic map is an advanced map representation form, which not only contains the geometric information of the geographic space position, but also contains the semantic information of various objects, roads, traffic rules, etc. in the environment. By converting the real-world scenes into a form that can be recognized and understood by the computer, it can assist the robot to recognize the surrounding environment when moving and adjust its work according to the environment.

[0035] In the absence of an environmental semantic map, image features of the environmental data and robot posture information are determined based on the acquired environmental data, image features are weighted fused based on the robot's posture information, a local image model is determined, and an environmental semantic map is determined based on the local image model.

[0036] The posture information of the robot can include the following aspects: Position information: describes the position of the robot's end effector or tool in space, usually expressed in three-dimensional coordinate values ​​(X, Y, Z); Direction information: describes the direction of the robot's end effector or tool, that is, its orientation, usually expressed in Euler angles or quaternions; Posture information: describes the posture of the robot's end effector or tool relative to a reference coordinate system, that is, its rotation angle. Usually expressed in Euler angles or quaternions; Joint angles: describes the robot's joint angles, that is, the rotation angles of each joint relative to its initial position. These angle values ​​determine the specific shape and movement of the robot; Speed ​​and acceleration: describes the robot's movement speed and acceleration, that is, the movement of the end effector or tool in space.

[0037] Preferably, the local image model can be understood as a collection of image features corresponding to the environmental data under the current robot posture. This model includes various information about the robot's surroundings, such as the shape, size, color, texture and other features of objects, as well as the relationship and layout between them.

[0038] Step S32: determining the movement path of the robot corresponding to the target object based on the environment semantic map.

[0039] In the case of the existence of an environmental semantic map, the corresponding target object on the environmental semantic map is determined based on the acquired environmental semantic map, and the first moving path of the robot is determined based on the position of the target object. When the first moving path is unreachable, the adjusted second moving path is obtained based on the reason why the first moving path is unreachable, and the movement of the robot is determined based on the second moving path.

[0040] Preferably, when an environmental semantic map exists, it also includes: determining a first position of a target area corresponding to a voice instruction based on the acquired voice instruction, determining a second position after execution based on a recognition result of the voice instruction, and determining a corresponding minimum working area in the environmental semantic map based on a distance between the first position and the second position; if there is no obstacle in the minimum working area, determining the distance between the robot and the minimum working area according to the minimum working area, and determining the movement path of the robot according to the environmental semantic map; if there is an obstacle in the minimum working area, determining the overlap between the minimum working area and the obstacle according to the minimum working area, and checking the obstacle information contained in the range centered on the minimum working area until there is no overlap between the minimum working area and the obstacle.

[0041] At this time, the first position and the second position are calculated in advance based on how much the robot needs to move for the current work, and whether there are obstacles in the current minimum working area is determined. When there are no obstacles, it moves directly to the corresponding position to work. If there are obstacles, the current moving route is adjusted according to the size of the minimum working area.

[0042] Step S33, determining the operation instructions of the robot according to the reception status of the voice instructions.

[0043] Preferably, when the voice command received by the robot does not exist, the movable position of the robot is selected based on the environmental data of the operable area, and the movement path of the robot is determined based on the historical recognition data.

[0044] If no voice commands are received, the robot will select a target object that can be executed directly based on its usual working conditions, or choose a relatively open location to wait.

[0045] Preferably, when a voice command is received, the waiting time of the voice command is detected, and the operation command of the robot is determined based on the recognition results of continuous voice commands.

[0046] Preferably, detecting the waiting time of the voice command includes obtaining the issuing time of the voice command, and determining the time interval between receiving different voice commands before and after the issuing of the voice command;

[0047] According to the time interval of the voice instructions, the output information of the currently output voice instruction is selected, including, when the robot executes an operation instruction, if the time interval of adjacent voice instructions exceeds a first preset time, determining the operation instruction executed by the robot based on the most recently received voice instruction.

[0048] The first preset time is set to ensure the timeliness and accuracy of the robot in performing tasks. If the time interval between adjacent voice commands is too long, the robot may ignore some of the intermediate commands and perform operations based on the last command received. This is to avoid the robot being unable to correctly recognize and understand commands due to the user's fast speaking speed or interference in the middle.

[0049] At the same time, when the robot receives a voice command that needs to be executed, the robot will first recognize the voice command, convert it into text information, and recognize the meaning of the text information to identify the current user's intention. After understanding the intention, the robot's work path is planned to determine the execution of the robot's work.

[0050] Preferably, the recognition result based on continuous voice instructions also includes, according to the voice instructions received by the robot in the operation interface, detecting whether there are mismatching operation instructions between the current target operation and the actual operation, and when not, determining the first prompt information corresponding to the input operation instruction based on the operation instruction; after there are mismatching operation instructions, generating a second prompt message, and adjusting the executed operation instruction based on the analysis results of the voice instructions.

[0051] Preferably, when the robot identifies different target objects, the use triggering conditions of the robot are adjusted according to the attribute information corresponding to the target objects;

[0052] The trigger conditions include determining the third prompt information of the robot based on the controllable load of the robot according to a preset voice command cycle, and determining the operation instructions of the robot based on the third prompt information.

[0053] Preferably, when the robot changes its moving direction or position, a second moving path that the robot can move is determined based on the robot posture information, and an operation instruction after the robot moves is determined based on the environmental data of the second moving path.

[0054] The first prompt information is the prompt information generated when the robot operation instruction matches the actual operation. Its function is to confirm whether the robot correctly understands the user's voice instruction and performs the operation according to the instruction. For example, if the user asks the robot to move to a certain position through voice instructions, after the robot correctly executes the instruction, it will issue a confirmation prompt information to inform the user that it has completed the instruction.

[0055] The second prompt message is a prompt message generated when the robot detects that the operation instruction does not match the actual operation. Its function is to inform the user that the robot failed to correctly understand or execute the instruction, and provide some guiding information to help the user re-enter the correct instruction. For example, if the user asks the robot to open an unopened cabinet door, the robot will issue an error prompt message to inform the user that the cabinet door is not open and provide some possible operation suggestions.

[0056] The third prompt information is the prompt information generated when the robot adjusts the use trigger conditions according to the attribute information of the target object. Its function is to provide the conditional restrictions and safety warnings about the robot's operation to ensure that the robot's operation will not cause damage to the target object or the surrounding environment. For example, if the target object is a fragile object, the robot will issue a warning prompt message to inform the user to place the object with care to avoid damage.

[0057] In summary, the first, second and third prompt information are designed to improve the robot's interactive experience and safety. Through these prompt information, the robot can better interact with the user and provide useful operation guidance and safety warnings.

[0058] Preferably, when a new voice instruction is received while an operation instruction is completed, the execution order of the operation instructions is determined based on the priority of the voice instruction content.

[0059] Preferably, after receiving the voice command, the content of the voice command is verified to determine whether the current voice command is executable. For example, different robots work in different areas. If the working area is changed without authorization, it may cause the robot to collide.

[0060] Preferably, when it is determined that the robot has arrived at the working area, the units in the area are identified based on the obtained work area-related map, and the content of the current work that can be performed is recorded. When a voice command is received, the content corresponding to the voice command is executed in sequence according to the interval order of the voice command.

[0061] Preferably, in order to enable the robot to adjust the path during movement according to the received voice message during normal movement, so that the position can be accurately identified after operation, thereby improving the effect of path recognition.

[0062] Specifically, step S3 also includes determining the movement path of the robot according to the speech recognition result.

[0063] First, determine the analysis of voice commands and obtain surrounding environmental data. Based on voice commands and environmental perception, determine the movement path and move in real time according to the calculated path.

[0064] The robot first needs to parse voice commands into executable actions, which includes recognizing key commands such as "forward," "backward," "turn left," and "turn right," as well as possible distance or angle information.

[0065] In order to move safely and efficiently, a robot needs information about its surroundings, which is typically achieved through various sensors such as lidar, cameras, and ultrasonic sensors to detect obstacles, terrain features, etc.

[0066] Based on voice commands and environmental perception data, the robot needs to plan a safe and efficient movement path, which may involve algorithms such as dynamic obstacle avoidance and shortest path calculation. For example, the Dijkstra algorithm or the A* algorithm can help the robot find the shortest path from the current position to the target position.

[0067] The robot uses its built-in navigation controller to guide the robot's movement based on the calculated path, which includes adjusting the robot's speed, steering, etc. to ensure that it moves along the intended path.

[0068] During the robot's movement, the system will monitor environmental changes and the robot's actual position in real time. If necessary, the path will be dynamically adjusted to adapt to the new situation.

[0069] If the user is in the same space as the robot, the robot can also interact with the user through sound, light or other means to inform the user of its current movement status or upcoming actions.

[0070] When the robot successfully reaches the target location or completes the predetermined path, it stops moving and gives feedback or performs other follow-up actions as needed.

[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A mobile robot intelligent control method, characterized in that: include: Step S1, based on a visual sensor disposed on a robot joint, obtaining environmental data of an operable area in the visual sensor; Step S2, based on the environmental data, determining that the current operable area is in an executable state, recognizing that the robot can move and can perform the currently running operation, then determining that the operable area is in an executable state, otherwise determining that the operable area is in a non-executable state; Step S3, identifying the currently sent voice command, determining the operable area based on the voice command, and determining the operation command of the robot based on the recognition result of the target in the operable area; Step S3 also includes the following implementation contents: Step S31, after recognizing the corresponding voice command, constructing an environmental semantic map of the operable area based on the data collected by the robot; Step S32, determining a movement path of the robot corresponding to the target object based on the environment semantic map; Step S33, determining the operation instruction of the robot according to the reception status of the voice instruction; In the case where an environmental semantic map exists, it also includes: determining a first position of a target area corresponding to a voice instruction based on an acquired voice instruction, determining a second position after execution based on a recognition result of the voice instruction, and determining a corresponding minimum working area in the environmental semantic map based on a distance between the first position and the second position; if there is no obstacle in the minimum working area, determining the distance between the robot and the minimum working area based on the minimum working area, and determining the movement path of the robot based on the environmental semantic map; if there is an obstacle in the minimum working area, determining an overlap between the minimum working area and the obstacle based on the minimum working area, and checking obstacle information contained in a range centered on the minimum working area until there is no overlap between the minimum working area and the obstacle.

2. A mobile robot intelligent control method as claimed in claim 1, characterized in that: If the operable area is in an executable state, the mobile device moves according to the current image recognition and executes the operation instruction when reaching the operable area; If the operable area is in a non-executable state, the moving path of the robot is adjusted according to the area information recognized by the image.

3. A mobile robot intelligent control method as claimed in claim 1, characterized in that: Identifying the operable area also includes determining a first position corresponding to the movement of the robot based on the displacement of the robot; the first position indicates the position to which the robotic arm moves after the robot moves; determining a second position corresponding to the first position based on an operation instruction executed by the robot, the second position being used to indicate the position to which the robotic arm moves after the robot performs the operation; and comparing the relationship between the first position and the second position to determine the distance moved before and after the movement of the robotic arm.

4. A mobile robot intelligent control method as claimed in claim 1, characterized in that: In the absence of an environmental semantic map, image features of the environmental data and robot posture information are determined based on the acquired environmental data, image features are weighted fused based on the robot's posture information, a local image model is determined, and an environmental semantic map is determined based on the local image model.

5. The mobile robot intelligent control method according to claim 1, characterized in that: In the case of the existence of an environmental semantic map, the corresponding target object on the environmental semantic map is determined based on the acquired environmental semantic map, and the first moving path of the robot is determined based on the position of the target object. When the first moving path is unreachable, the adjusted second moving path is obtained based on the reason why the first moving path is unreachable, and the movement of the robot is determined based on the second moving path.

6. A mobile robot intelligent control method as claimed in claim 1, characterized in that: When receiving a voice command, the waiting time of the voice command is detected, and the operation command of the robot is determined based on the recognition results of the continuous voice command; Detecting the waiting time of the voice command includes obtaining the time when the voice command is issued, and determining the time interval between receiving different voice commands before and after the voice command is issued; According to the time interval of the voice instructions, the output information of the currently output voice instruction is selected, including, when the robot executes an operation instruction, if the time interval of adjacent voice instructions exceeds a first preset time, determining the operation instruction executed by the robot based on the most recently received voice instruction.

7. A mobile robot intelligent control method as claimed in claim 6, characterized in that: The recognition result based on the continuous voice command also includes, according to the voice command received by the robot on the operation interface, detecting whether there is an operation command that does not match the current target operation and the actual operation, and when there is no such operation command, determining the first prompt information corresponding to the input operation command based on the operation command; After there is an unmatched operation instruction, a second prompt message is generated, and based on the analysis result of the voice instruction, the executed operation instruction is adjusted.

8. A mobile robot intelligent control method as claimed in claim 6, characterized in that: When the robot recognizes different target objects, it adjusts the trigger conditions for the robot's use according to the corresponding attribute information of the target objects; The trigger conditions include determining the third prompt information of the robot based on the controllable load of the robot according to a preset voice command cycle, and determining the operation instructions of the robot based on the third prompt information.

Citation Information

Patent Citations

  • Method and device for building robot work area map, robot and medium

    CN109947109A

  • Mobile laundry assistant robot based on image scanning, recognition and route planning

    CN111002305A

  • Robot intelligent control system

    CN111775151A

  • Intelligent assembly system

    CN116021250A

  • Industrial robot sensing information processing system

    CN117301058A