Terminal object position identification method and system, storage medium and computer program product

Abstract

The invention provides a method for recognizing the position of a tail end object. The method comprises the following steps that position information of visual recognition modules installed on the head of a robot and at least one mechanical arm is obtained according to a current pose signal of the robot; acquiring image data of a target object by using the visual identification module; processing and feature extraction are carried out on the image data, and the position information of a tail end object is calculated by combining the pose information of the robot and the position information of the visual identification module; and according to the calculated position information of the target object, the mechanical arm is controlled to carry out picking operation. The invention also provides a corresponding system, a storage medium and a computer program product, by adopting a plurality of movable visual identification modules, the interference of the shielding object can be avoided, and the position of the tail end object can be accurately obtained.

Description

Technical Field

[0001] The present invention relates to the field of robot interaction technology, and in particular to a method, system, storage medium and computer program product for identifying the position of an end object. Background Art

[0002] In the field of robot interaction technology, especially in natural human-machine interaction, how humanoid robotic arms can learn and reproduce the configuration, end posture and operation trajectory of human arms is a research topic with great application value. In order to achieve this goal, it is crucial to accurately identify the position of the end object. Traditional methods mainly rely on visual perception technology, capture image information through fixed cameras or visual recognition modules, and use computer vision algorithms to detect, identify and locate objects. For example, active depth sensing technologies such as Kinect, as well as multi-eye vision systems, can achieve three-dimensional spatial positioning of objects to a certain extent.

[0003] However, these fixed visual recognition solutions face many challenges in practical applications. Factors such as noise, lighting changes, and occlusion may seriously affect the recognition accuracy, resulting in inaccurate judgment of the object's position. In particular, when the object is blocked by other objects or a person's arm, the object information in the image may be incomplete, making it difficult for the recognition algorithm to accurately obtain the object's location information.

[0004] Therefore, there is an urgent need for an end object position recognition method that can overcome the occlusion problem and improve recognition accuracy. Summary of the invention

[0005] The technical problem to be solved by the present invention is to propose a method, system, storage medium and computer program product for end object position recognition. By adopting multiple movable visual recognition modules, the interference of obstructions can be avoided and the end object position can be accurately obtained.

[0006] As one aspect of the present invention, a method for identifying a position of an end object is provided, which comprises:

[0007] Acquire position information of a visual recognition module installed on the robot head and at least one robotic arm according to the current posture signal of the robot;

[0008] Acquiring image data of a target object using the visual recognition module;

[0009] Processing and feature extraction of image data, combining the posture information of the robot and the position information of the visual recognition module, and calculating the position information of the end object;

[0010] According to the calculated position information of the target object, the robotic arm is controlled to perform the picking operation.

[0011] Wherein, using the visual recognition module to obtain image data of the target object further includes:

[0012] Detect whether the head vision recognition module is blocked. When it is detected that the head vision recognition module is blocked, operate the robotic arm to move to avoid obstacles, and use the vision recognition module on the robotic arm to identify the position of the target object.

[0013] Among them, image data processing and feature extraction include:

[0014] Use deep learning algorithms to preprocess images and detect feature points;

[0015] The image information is processed through feature selection and extraction, feature matching, image processing and feature extraction steps to extract image features in the image data.

[0016] The image data is processed and features are extracted, and the position information of the end object is calculated by combining the posture information of the robot and the position information of the visual recognition module, further comprising:

[0017] The image features are used for motion prediction to obtain target tracking information, and the target tracking information is integrated with image processing and feature extraction to obtain more accurate current image features;

[0018] Comparing the current image feature with the expected feature, calculating the image feature error, and generating a control signal according to the error when the error exceeds a preset error threshold;

[0019] Control the movement of the robot and / or the mechanical arm according to the control signal;

[0020] Calculate and update the robot's posture information in real time according to the robot's dynamic model, and repeat the above steps until the calculated error is within the preset error threshold;

[0021] The position information of the end object is calculated by combining the current posture information of the robot and the position information of the visual recognition module.

[0022] Accordingly, as another aspect of the present invention, there is also provided a terminal object position recognition system, which comprises:

[0023] A posture signal processing module, used for obtaining position information of a visual recognition module installed on the robot head and at least one mechanical arm according to the current posture signal of the robot;

[0024] An image data acquisition module, which uses the visual recognition module to acquire image data of a target object;

[0025] An image processing module processes and extracts features from image data, and calculates the position information of the end object by combining the posture information of the robot and the position information of the visual recognition module;

[0026] The control module controls the robot arm to perform a picking operation according to the calculated position information of the target object.

[0027] Wherein, the image data acquisition module further comprises:

[0028] An occlusion detection unit, used to detect whether the head visual recognition module is blocked;

[0029] The obstacle avoidance processing unit operates the robotic arm to move to avoid obstacles when the occlusion detection unit detects that the head visual recognition module is blocked, and uses the visual recognition module on the robotic arm to identify the position of the target object.

[0030] Wherein, the image processing module includes:

[0031] A preprocessing unit that uses a deep learning algorithm to preprocess images and detect feature points;

[0032] The feature processing unit processes the image information through feature selection and extraction, feature matching, image processing and feature extraction steps to extract image features from the image data.

[0033] Wherein, the image processing module further comprises:

[0034] A motion prediction unit, which uses the image features to perform motion prediction to obtain target tracking information, and integrates the target tracking information with image processing and feature extraction to obtain more accurate current image features;

[0035] a control signal generating unit, which compares the current image feature with the expected feature, calculates the image feature error, and generates a control signal according to the error when the error exceeds a preset error threshold;

[0036] A robot movement control unit, which controls the movement of the robot and / or the movement of the mechanical arm according to the control signal;

[0037] A posture information updating processing unit calculates and updates the posture information of the robot in real time according to the dynamic model of the robot, and controls the movement of the robot according to the motion prediction unit, the control signal generating unit and the robot movement control unit until the calculated error is within a preset error threshold;

[0038] The position information calculation unit calculates the position information of the end object by combining the current posture information of the robot and the position information of the visual recognition module.

[0039] Accordingly, as another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned method are implemented.

[0040] Accordingly, as another aspect of the present invention, a computer program product is provided, comprising computer instructions, wherein the computer instructions instruct a computer device to perform operations corresponding to the aforementioned method.

[0041] Implementing the embodiments of the present invention has the following beneficial effects:

[0042] The present invention provides a method, system, storage medium and computer program product for identifying the position of an end object. By installing movable visual recognition modules on the robot head and mechanical arm (i.e., hand), when the head visual recognition module is blocked, the mechanical arm can be operated to avoid the obstacle, and the position of the end object can be identified by combining the visual recognition modules on the two mechanical arms; in the present invention, multiple movable visual recognition modules can be used to identify the end object, thereby improving the accuracy of identification.

[0043] In the embodiment of the present invention, the position information of the robot is combined, and through algorithms such as forward and inverse kinematics, the position information of the end object can be accurately obtained even under adverse conditions such as occlusion or lighting changes, thereby enhancing the robustness of the system.

[0044] In the embodiment of the present invention, accurate position information can be provided for the robot arm, so that it can better perform tasks such as picking, and has wide application value.

[0045] In summary, the present invention effectively solves the occlusion problem existing in traditional methods through innovative visual recognition methods and system design, improves recognition accuracy and robustness, and provides strong technical support for the development of human-machine collaborative robotic arms. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, without paying creative labor, other drawings obtained based on these drawings still belong to the scope of the present invention.

[0047] Figure 1 A schematic diagram of the main process of an embodiment of a method for identifying the position of an end object provided by the present invention;

[0048] Figure 2 A schematic diagram of the method principle provided by the present invention;

[0049] Figure 3 A schematic diagram of the structure of an embodiment of a terminal object position recognition system provided by the present invention;

[0050] Figure 4 for Figure 3 The structural diagram of the image data acquisition module;

[0051] Figure 5 for Figure 3 Schematic diagram of the structure of the image processing module. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings.

[0053] like Figure 1 FIG. 1 is a schematic diagram showing a main flow chart of an embodiment of a method for identifying the position of an end object provided by the present invention. Figure 2 As shown, in this embodiment, the method at least includes:

[0054] Step S10, obtaining position information of a visual recognition module installed on the robot head and at least one mechanical arm according to the current posture signal of the robot;

[0055] It can be understood that the robot obtains its posture information, including the position and posture in three-dimensional space, through its own sensors (such as gyroscopes, accelerometers, encoders, etc.); and in the present invention, the visual recognition module can be such as a camera, which is included in the head of the robot and one or two robotic arms (hands) of the machine; the position information of the visual recognition module can be calculated through the posture information of the robot; for example, in one example, an industrial robot calculates the three-dimensional spatial position of the camera installed at the end of the robot arm through the joint angle information fed back by its joint encoder, combined with the robot's kinematic model.

[0056] In this embodiment, visual data is collected by installing multiple visual recognition modules on the head and hands of the robot. These modules can move with the movement of the robot arm, thereby avoiding the problem of possible occlusion of fixed cameras or visual recognition modules.

[0057] Step S11, using the visual recognition module to obtain image data of the target object;

[0058] In a specific example, step S11 further includes:

[0059] Detect whether the head vision recognition module is blocked. When it is detected that the head vision recognition module is blocked, operate the robotic arm to move to avoid obstacles, and use the vision recognition module on the robotic arm to identify the position of the target object.

[0060] Step S12, processing and feature extraction of the image data, combining the posture information of the robot and the position information of the visual recognition module, and calculating the position information of the end object;

[0061] In a specific example, in this step, the image data is processed and features are extracted, including:

[0062] Use deep learning algorithms to preprocess images and detect feature points;

[0063] The image information is processed through feature selection and extraction, feature matching, image processing and feature extraction steps to extract image features in the image data.

[0064] It is understood that the visual recognition module captures the image of the target object and extracts the features of the target object through a series of image processing algorithms (such as edge detection, color segmentation, shape recognition, etc.). These features are used to calculate the position of the target object relative to the visual recognition module. For example, in one example, the camera on the head of the robot captures an image of a part, identifies the edge and shape of the part through an image processing algorithm, and then calculates the three-dimensional coordinates of the part relative to the camera.

[0065] In a specific example, the step S12 further includes:

[0066] The image features are used to perform motion prediction to obtain target tracking information, and the target tracking information is integrated with image processing and feature extraction to obtain more accurate current image features. In a specific example, a motion prediction algorithm such as a Kalman filter or a particle filter can be used to predict the position of a target object in a current frame image based on the features of a previous frame image, thereby improving the accuracy of target tracking. For example, in continuous video frames, a robot predicts the position of a moving part in the next frame image through a motion prediction algorithm, thereby more accurately tracking and locating the part.

[0067] Compare the current image features with the expected features, calculate the image feature error, and generate a control signal according to the error when the error exceeds a preset error threshold; for example, in one example, the robot compares the currently captured part features with the expected part features (such as shape, size, color, etc.), calculates the feature error and generates a control signal;

[0068] According to the control signal, the robot movement or / and the mechanical arm movement are controlled. For example, the arm position of the robot can be adjusted according to the control signal so that the parts are more accurately aligned with the assembly position. In a specific example, the robot controller provided in the robot receives the control signal generated by the visual controller and converts it into a driver signal to control the robot's motor or actuator such as a hydraulic cylinder, thereby adjusting the torque or speed of the robot. In a specific example, the robot controller adjusts the current or voltage of the robot arm motor according to the instruction of the visual controller, thereby changing the torque of the arm, so that the arm can more accurately move the parts to the specified position.

[0069] According to the robot's dynamic model, the robot's position and posture information is calculated and updated in real time, and the above steps are repeated until the calculated error is within the preset error threshold; it can be understood that in this step, the robot's dynamic and kinematic models can be used to calculate the robot's position and posture information in real time, and feed it back to the robot control system for updating the position information of the visual recognition module and performing the next step of operation. For example, in one example, the robot calculates the end position and direction of the arm in real time through its internal sensors and dynamic models, and feeds this information back to the control system for updating the position information of the camera and performing subsequent visual recognition tasks.

[0070] The position information of the end object is calculated by combining the current posture information of the robot and the position information of the visual recognition module.

[0071] Step S13, controlling the robot arm to perform a picking operation according to the calculated position information of the target object.

[0072] It is understandable that the embodiment of the present invention not only utilizes the relative position information of the target obtained by the visual recognition module, but also combines the posture information of the robot (such as position, posture, etc.) to accurately obtain the absolute position of the end object through algorithms such as forward and inverse kinematics. This positioning method combined with the posture information of the robot improves the accuracy of positioning, allowing the robot arm to pick up the target object more accurately.

[0073] like Figure 2 As shown, in a specific embodiment of the present invention, the following process may be included:

[0074] First, the robot obtains the position information of the visual recognition module through its own posture signal, and obtains the robot's own coordinate system, the robot's position in three-dimensional space, the robot's posture (Euler angle or quaternion), and the position of the visual recognition module;

[0075] Secondly, the robot obtains the target signal through the visual recognition module, and uses feature selection and extraction, feature matching, image processing and feature extraction to extract the features of the current image and calculate the position information of the end object. In this step, the relative position of the target calculated by the visual recognition module is combined with the absolute position of the target calculated by the robot position; in this step, the visual recognition module installed on the head and hands of the robot collects visual data to extract the position features of the end object to avoid occlusion and interference problems; at the same time, the position of the end object is accurately obtained in combination with the posture information of the robot;

[0076] Then, the data obtained from image processing and feature extraction is used for motion prediction of image features to track the target, and the target tracking information is fed back to feature selection and extraction and feature matching to obtain more accurate current image features.

[0077] In addition, the current image features are compared with the expected features to calculate the image feature error. The vision controller generates a control signal based on the image feature error.

[0078] In addition, the robot controller sends a driver signal according to the control signal to realize the control of the robot torque.

[0079] Then, the robot posture signals obtained by robot dynamics and robot kinematics are fed back to the robot to perform the first step of operation.

[0080] It can be understood that in this solution, the robot head camera obtains the approximate position and posture of the object, moves the robot arm to approach the object, and combines the visual recognition modules of the head and hands to accurately locate the end object, and combines the robot's position and posture information to calculate the coordinates of the end object relative to the robot reference. When the head camera is blocked, the position of the end object can be identified by combining the visual recognition modules on the two robot arms.

[0081] like Figure 3 FIG. 1 is a schematic diagram showing a structure of an embodiment of a terminal object position recognition system provided by the present invention. Figure 4 , Figure 5 As shown, in this embodiment, the end object position recognition system 1 at least includes:

[0082] A posture signal processing module 10, used to obtain position information of a visual recognition module installed on the robot head and at least one mechanical arm according to the current posture signal of the robot;

[0083] An image data acquisition module 11, which uses the visual recognition module to acquire image data of a target object;

[0084] The image processing module 12 processes and extracts features from the image data, and calculates the position information of the end object by combining the posture information of the robot and the position information of the visual recognition module;

[0085] The control module 13 controls the robot arm to perform a picking operation according to the calculated position information of the target object.

[0086] like Figure 4 As shown, the image data acquisition module 11 further includes:

[0087] The occlusion detection unit 110 is used to detect whether the head visual recognition module is blocked;

[0088] The obstacle avoidance processing unit 111 operates the robotic arm to move to avoid the obstacle when the occlusion detection unit detects that the head visual recognition module is blocked, and uses the visual recognition module on the robotic arm to identify the position of the target object.

[0089] like Figure 5 As shown, the image processing module 12 includes:

[0090] A preprocessing unit 120 performs preprocessing and feature point detection on the image using a deep learning algorithm;

[0091] A feature processing unit 121 processes the image information through feature selection and extraction, feature matching, image processing and feature extraction steps to extract image features from the image data;

[0092] The motion prediction unit 122 uses the image features to perform motion prediction to obtain target tracking information, and integrates the target tracking information with image processing and feature extraction to obtain more accurate current image features;

[0093] The control signal generating unit 123 compares the current image feature with the expected feature, calculates the image feature error, and generates a control signal according to the error when the error exceeds a preset error threshold;

[0094] A robot movement control unit 124 controls the movement of the robot and / or the movement of the robotic arm according to the control signal;

[0095] a posture information updating processing unit 125, which calculates and updates the posture information of the robot in real time according to the dynamic model of the robot, and controls the movement of the robot according to the motion prediction unit, the control signal generating unit and the robot movement control unit until the calculated error is within a preset error threshold;

[0096] The position information calculation unit 126 calculates the position information of the end object by combining the current posture information of the robot and the position information of the visual recognition module.

[0097] For more details, please refer to the above Figure 1 to Figure 2 The description is not repeated here.

[0098] Accordingly, as another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned Figure 1 and Figure 2 For more details, please refer to the above Figure 1 to Figure 2 The description is not repeated here.

[0099] Accordingly, as another aspect of the present invention, a computer program product is also provided, comprising computer instructions, wherein the computer instructions instruct a computer device to execute the above-mentioned Figure 1 For more details, please refer to the above Figure 1 to Figure 2 The description is not repeated here.

[0100] The implementation of the embodiments of the present invention has the following beneficial effects:

[0101] The present invention provides a method, system, storage medium and computer program product for identifying the position of an end object. By installing movable visual recognition modules on the head and hands of a robot, when the head visual recognition module is blocked, the robot arm can be operated to avoid obstacles, and the position of the end object can be identified by combining the visual recognition modules on the two robot arms; in the present invention, multiple movable visual recognition modules can be used to identify the end object, thereby improving the accuracy of identification.

[0102] In the embodiment of the present invention, the position information of the robot is combined, and through algorithms such as forward and inverse kinematics, the position information of the end object can be accurately obtained even under adverse conditions such as occlusion or lighting changes, thereby enhancing the robustness of the system.

[0103] In the embodiment of the present invention, accurate position information can be provided for the robot arm, so that it can better perform tasks such as picking, and has wide application value.

[0104] In summary, the present invention effectively solves the occlusion problem existing in traditional methods through innovative visual recognition methods and system design, improves recognition accuracy and robustness, and provides strong technical support for the development of human-machine collaborative robotic arms.

[0105] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0106] The above disclosure is only a preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for identifying the position of an end object, characterized in that: The steps include: Acquire position information of a visual recognition module installed on the robot head and at least one robotic arm according to the current posture signal of the robot; Acquiring image data of a target object using the visual recognition module; Processing and feature extraction of image data, combining the posture information of the robot and the position information of the visual recognition module, and calculating the position information of the end object; According to the calculated position information of the target object, the robotic arm is controlled to perform the picking operation.

2. The method according to claim 1, characterized in that Acquiring image data of a target object using the visual recognition module further includes: Detect whether the head vision recognition module is blocked. When it is detected that the head vision recognition module is blocked, operate the robotic arm to move to avoid obstacles, and use the vision recognition module on the robotic arm to identify the position of the target object.

3. The method according to claim 2, characterized in that Process image data and extract features, including: Use deep learning algorithms to preprocess images and detect feature points; The image information is processed through feature selection and extraction, feature matching, image processing and feature extraction steps to extract image features in the image data.

4. The method according to claim 3, characterized in that Processing the image data and extracting features, combining the posture information of the robot and the position information of the visual recognition module to calculate the position information of the end object, further includes: The image features are used for motion prediction to obtain target tracking information, and the target tracking information is integrated with image processing and feature extraction to obtain more accurate current image features; Comparing the current image feature with the expected feature, calculating the image feature error, and generating a control signal according to the error when the error exceeds a preset error threshold; Control the movement of the robot and / or the mechanical arm according to the control signal; Calculate and update the robot's posture information in real time according to the robot's dynamic model, and repeat the above steps until the calculated error is within the preset error threshold; The position information of the end object is calculated by combining the current posture information of the robot and the position information of the visual recognition module.

5. A terminal object position recognition system, characterized in that: include: A posture signal processing module, used for obtaining position information of a visual recognition module installed on the robot head and at least one mechanical arm according to the current posture signal of the robot; An image data acquisition module, which uses the visual recognition module to acquire image data of a target object; An image processing module processes and extracts features from image data, and calculates the position information of the end object by combining the posture information of the robot and the position information of the visual recognition module; The control module controls the robot arm to perform a picking operation according to the calculated position information of the target object.

6. The system according to claim 5, characterized in that The image data acquisition module further comprises: An occlusion detection unit, used to detect whether the head visual recognition module is blocked; The obstacle avoidance processing unit operates the robotic arm to move to avoid obstacles when the occlusion detection unit detects that the head visual recognition module is blocked, and uses the visual recognition module on the robotic arm to identify the position of the target object.

7. The system according to claim 6, characterized in that The image processing module comprises: A preprocessing unit that uses a deep learning algorithm to preprocess images and detect feature points; The feature processing unit processes the image information through feature selection and extraction, feature matching, image processing and feature extraction steps to extract image features from the image data.

8. The system according to claim 7, characterized in that The image processing module further comprises: A motion prediction unit, which uses the image features to perform motion prediction to obtain target tracking information, and integrates the target tracking information with image processing and feature extraction to obtain more accurate current image features; a control signal generating unit, which compares the current image feature with the expected feature, calculates the image feature error, and generates a control signal according to the error when the error exceeds a preset error threshold; A robot movement control unit, which controls the movement of the robot and / or the movement of the mechanical arm according to the control signal; a posture information updating processing unit, which calculates and updates the posture information of the robot in real time according to the dynamic model of the robot, and controls the movement of the robot according to the motion prediction unit, the control signal generating unit and the robot movement control unit until the calculated error is within a preset error threshold; The position information calculation unit calculates the position information of the end object by combining the current posture information of the robot and the position information of the visual recognition module.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer program product, comprising computer instructions, wherein the computer instructions instruct a computer device to perform operations corresponding to the method according to any one of claims 1 to 4.

Images