Multi-modal fusion sensing robot interaction system

Abstract

The invention provides a multi-modal fusion sensing robot interaction system, and relates to the technical field of multi-modal fusion sensing, and the system comprises an information sensing unit activation module; a target area scanning module; the target position information determining module is used for determining target position information of the to-be-fetched article according to the regional image and the regional point cloud set; the article grabbing module is used for generating an article grabbing path based on the target position information to grab articles; and the article carrying module is used for collecting weight information of the grabbed articles through a pressure sensor, and if the weight information is within a preset weight interval, the grabbed articles are carried to a preset storage area. The technical problems that a traditional carrying robot usually depends on a single sensor for detection, accurate recognition and positioning cannot be effectively achieved in a complex environment, and the object grabbing precision and reliability are insufficient can be solved, more comprehensive and accurate object sensing and grabbing can be achieved, and the object grabbing efficiency is improved. And the article grabbing accuracy and reliability are remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of multimodal fusion perception technology, and in particular to a multimodal fusion perception robot interaction system. Background Art

[0002] Traditional handling robots have been widely used in automated warehouses, logistics and transportation, material handling and other fields, especially playing an important role in tasks such as object handling, sorting and stacking. In order to achieve these tasks, handling robots usually rely on sensor systems for environmental perception and object recognition. Among these systems, the most common sensors include visual sensors (such as RGB cameras, depth cameras) and laser radar (LiDAR).

[0003] However, traditional handling robot systems often rely on a single type of sensor for environmental perception and object identification. This approach may be able to complete the task in simple and stable environments, but often has significant technical problems in complex and dynamic working environments. Summary of the invention

[0004] The purpose of this application is to provide a multimodal fusion perception robot interaction system to solve the technical problems that traditional handling robots usually rely on a single sensor for detection, cannot effectively achieve accurate identification and positioning in complex environments, and have insufficient accuracy and reliability in object grasping.

[0005] In view of the above problems, the present application provides a multimodal fusion perception robot interaction system, including: an information perception unit activation module, used to activate the information perception unit of the perception robot, wherein the perception robot is a handling robot, and the information perception unit includes a visual sensor, a laser radar and a pressure sensor; a target area scanning module, used to collect data on the target area through the visual sensor and the laser radar, and obtain the area image and the area point cloud set; a target position information determination module, used to determine the target position information of the to-be-picked item according to the area image and the area point cloud set; an item grasping module, used to generate an item grasping path based on the target position information, control the perception robot to grasp the item, and obtain the grasped item; an item handling module, used to collect the weight information of the grasped item through the pressure sensor, and if the weight information is in a predetermined weight range, the grasped item is transported to a predetermined storage area.

[0006] The technical solution provided in this application has at least the following technical effects or advantages:

[0007] By activating the information perception unit of the perception robot, wherein the perception robot is a transport robot, and the information perception unit includes a visual sensor, a laser radar and a pressure sensor; then data is collected on the target area through the visual sensor and the laser radar to obtain the regional image and the regional point cloud set; then the target position information of the object to be picked up is determined according to the regional image and the regional point cloud set; further based on the target position information, an object grasping path is generated, and the perception robot is controlled to grasp the object to obtain the grasped object; finally, the weight information of the grasped object is collected through the pressure sensor, and if the weight information is in a predetermined weight range, the grasped object is transported to a predetermined storage area; that is, by combining multiple sensors such as visual sensors, laser radars, pressure sensors, etc., more comprehensive and accurate object perception and grasping can be achieved, significantly improving the accuracy and reliability of object grasping.

[0008] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented according to the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are specifically cited below. It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0010] Figure 1 This is a schematic diagram of the structure of a multi-modal fusion perception robot interaction system for this application;

[0011] Figure 2 This is a schematic diagram of a process for determining target position information of an object to be picked up in a multimodal fusion perception robot interaction system of the present application.

[0012] Description of reference numerals:

[0013] An information sensing unit activation module 11 , a target area scanning module 12 , a target position information determination module 13 , an item grabbing module 14 , and an item transporting module 15 . DETAILED DESCRIPTION

[0014] This application provides a multi-modal fusion perception robot interaction system to solve the technical problems that traditional handling robots usually rely on a single sensor for detection, cannot effectively achieve accurate identification and positioning in complex environments, and have insufficient accuracy and reliability in object grasping. By combining multiple sensors such as visual sensors, laser radars, and pressure sensors, more comprehensive and accurate object perception and grasping can be achieved, significantly improving the accuracy and reliability of object grasping.

[0015] Below, the technical solutions in the present application will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments of the present application. It should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application. It should also be noted that, for the convenience of description, only the parts related to the present application are shown in the accompanying drawings, rather than all of them.

[0016] For examples, please see the attached Figure 1 , the present application provides a multi-modal fusion perception robot interaction system, including:

[0017] An information perception unit activation module 11 is used to activate the information perception unit of the perception robot, wherein the perception robot is a handling robot, and the information perception unit includes a visual sensor, a laser radar and a pressure sensor; a target area scanning module 12 is used to collect data on the target area through the visual sensor and the laser radar, and obtain the area image and the area point cloud set; a target position information determination module 13 is used to determine the target position information of the object to be picked up according to the area image and the area point cloud set; an object grasping module 14 is used to generate an object grasping path based on the target position information, control the perception robot to grasp the object, and obtain the grasped object; an object handling module 15 is used to collect the weight information of the grasped object through the pressure sensor, and if the weight information is in a predetermined weight range, the grasped object is transported to a predetermined storage area.

[0018] Specifically, with the continuous development of automation and intelligent technology, handling robots are increasingly used in logistics, warehousing, factory automation and other fields. However, traditional handling robots usually rely on a single sensor (such as visual sensors or lidar) for environmental perception and object recognition, which may lead to grasping failure or inaccurate recognition in complex and dynamic working environments. In order to solve this technical problem, modern handling robot systems are gradually developing towards multimodal sensor fusion, which improves the system's perception accuracy and grasping reliability by fusing different types of sensor data.

[0019] First, activate the information perception unit of the handling robot, which includes visual sensors, laser radars, and pressure sensors, and can provide different perception information. The visual sensor is used to obtain the appearance information of the object, the laser radar is used to obtain the spatial positioning information of the object, and the pressure sensor is used to monitor the weight and grasping force of the object in real time during the grasping process. Then, the target area is scanned by the laser radar and visual sensor to collect regional images and regional point cloud sets. The regional images generated by the visual sensor provide appearance features for object recognition and positioning, and the point cloud set provided by the laser radar helps the robot obtain the three-dimensional spatial position information of the target object. This process enables the robot to comprehensively analyze the situation of the target area from both visual and depth data.

[0020] Then, based on the collected regional images and regional point cloud sets, the robot determines the target position information of the object to be picked up through algorithm processing, that is, the image and point cloud data are combined. Through data fusion technology, the robot can accurately calculate the position, posture, size and other information of the object, providing an accurate basis for the subsequent grasping path planning. Then, according to the target position information, the robot generates the grasping path of the object. The grasping path takes into account the three-dimensional position of the object, the grasping posture and the spatial layout of the surrounding environment to ensure that the grasping action will not collide with obstacles and can effectively grasp the object. The system controls the perception robot to perform the grasping operation along the generated path to obtain the target object. Finally, after successfully grasping the object, the robot collects the weight information of the grasped object in real time through the pressure sensor. If the weight of the object is within the predetermined weight range, the system determines that the grasping is successful and performs the object handling operation. The pressure sensor ensures that the robot will not cause damage to the object or grasping failure due to excessive or insufficient force, thereby ensuring the reliability and safety of object handling. Finally, the robot moves the grasped object to the predetermined storage area.

[0021] A multimodal fusion perception robot interaction system is provided, comprising: an information perception unit activation module, which is used to activate the information perception unit of the perception robot, wherein the perception robot is a handling robot, and the information perception unit includes a visual sensor, a laser radar and a pressure sensor; a target area scanning module, which is used to collect data on the target area through a visual sensor and a laser radar, and obtain a regional image and a regional point cloud set; a target position information determination module, which is used to determine the target position information of the object to be picked up according to the regional image and the regional point cloud set; an object grabbing module, which is used to generate an object grabbing path based on the target position information, control the perception robot to grab the object, and obtain the grabbed object; an object handling module, which is used to collect the weight information of the grabbed object through a pressure sensor, and if the weight information is in a predetermined weight range, the grabbed object is carried to a predetermined storage area. The technical problem that traditional handling robots usually rely on a single sensor for detection, cannot effectively achieve accurate identification and positioning in a complex environment, and have insufficient object grabbing accuracy and reliability can be solved. By combining multiple sensors such as visual sensors, laser radars, and pressure sensors, more comprehensive and accurate object perception and grabbing can be achieved, and the object grabbing accuracy and reliability can be significantly improved.

[0022] Furthermore, the target area scanning module 12 is also used for:

[0023] The target area is imaged by a visual sensor to obtain an initial area image; the target area is scanned by a laser radar to obtain an initial area point cloud set; according to a predetermined image processing scheme, the initial area image is subjected to image denoising and image enhancement to obtain the area image; the initial area point cloud set is subjected to filtering processing and coordinate transformation to obtain the area point cloud set.

[0024] Specifically, the target area is imaged by a visual sensor to obtain an initial area image, which captures the appearance characteristics of the objects in the target area, such as color, shape, texture and other information. However, due to factors such as ambient lighting changes and noise interference, the initial image may have problems such as blur and noise. The target area is scanned by a laser radar to obtain an initial area point cloud set. The laser radar point cloud data can provide three-dimensional spatial information of the target area (such as the location, size and shape of the object), but due to the limitations of the laser radar (such as sparse point clouds, environmental reflections, etc.), the original point cloud data may also be affected by noise interference or occlusion.

[0025] Next, according to a predetermined image processing scheme, the initial regional image is subjected to image denoising and image enhancement. For example, the noise in the image is removed by a filtering algorithm to reduce the impact of external interference (such as illumination changes, sensor noise, etc.) on the image. Commonly used denoising methods include mean filtering, Gaussian filtering, median filtering, etc.; image enhancement techniques (such as histogram equalization, sharpening, etc.) are used to improve the contrast and clarity of the image so that the edges of objects in the image are more prominent, which is convenient for subsequent object recognition; and the regional image is obtained.

[0026] The initial regional point cloud set is filtered and coordinate transformed. For example, a filtering algorithm is used to remove irrelevant or low-quality point cloud data, such as using statistical outlier removal or voxel grid filtering to reduce the noise of the point cloud and retain useful spatial information. Since the lidar and visual sensors may be installed at different positions and angles, it is necessary to perform coordinate transformation on the point cloud data to unify them into a unified coordinate system. Commonly used coordinate transformation methods include geometric transformations based on rotation matrices and transformation matrices to ensure that the visual data and point cloud data can be accurately aligned. The regional point cloud set is obtained.

[0027] Further, if Figure 2 As shown, the target location information determination module 13 is also used for:

[0028] Acquire attribute characteristics of the item to be picked up, wherein the attribute characteristics include color characteristics, shape characteristics and size parameters; select a frame in the area image according to the color characteristics, shape characteristics and size parameters to determine a target framed area; crop the area image according to the target framed area to obtain a framed image, and determine target position information of the item to be picked up based on the framed image.

[0029] Specifically, the attribute features of the object to be picked up are obtained, wherein the attribute features include color features, shape features and size parameters; then, a frame is selected in the region image based on the obtained color features, shape features and size parameters. The purpose of the frame selection is to mark the position of the object to be picked up from the image based on these attribute features to form a target frame selection area. Through these methods, the position of the object in the image can be accurately located, and a bounding box can be generated to accurately frame the object.

[0030] Then, the regional image is cropped according to the framed area to extract the target part containing the object to be picked up. The cropped image will focus on the target object and its surrounding area and remove irrelevant information, thereby improving the efficiency and accuracy of subsequent processing. Finally, based on the cropped framed image, the target position information of the object to be picked up in the image is determined using image analysis algorithms (such as target detection, key point positioning, feature matching, etc.). This position information usually includes the center position of the object, the spatial position of the object relative to the camera, the posture of the object (such as angle), etc. Combined with the point cloud data obtained by the lidar scanning, the object is located and identified in three-dimensional space. The accuracy of object positioning is further improved by fusing the two-dimensional information provided by the visual image and the three-dimensional information provided by the lidar, especially in complex environments, which can effectively avoid recognition problems caused by occlusion or partial missing.

[0031] Furthermore, the target location information determination module 13 is also used for:

[0032] Contour features are extracted from the framed image to obtain the contour of the object; random fitting is performed in the regional point cloud set according to the object contour to obtain an optimal point cloud fitting result; a regional three-dimensional space of the target area is constructed, coordinate positioning is performed in the regional three-dimensional space according to the optimal point cloud fitting result, and the three-dimensional coordinate data of the object to be picked up is obtained as the target position information.

[0033] Specifically, first, the image is subjected to contour feature extraction. This process uses image processing algorithms (such as Canny edge detection, Hough transform, etc.) to identify and extract the edge contour of the object. Through the extracted contour features, the outer boundary of the object can be obtained, which is crucial for the subsequent positioning of the object and grasping path planning. Then, according to the contour of the object, random fitting is performed in the point cloud set of the region to obtain the optimal point cloud fitting result. Then, according to the contour features of the objects in the region and the data scanned by the laser radar, the three-dimensional space of the target area is constructed. The construction of the three-dimensional space needs to consider the size, shape and other factors that may affect the positioning of the object in the target area (such as the orientation and size of the object); finally, according to the optimal point cloud fitting result, coordinate positioning is performed in the three-dimensional space of the region. This process maps the fitted point cloud data into the three-dimensional space and combines the contour features in the visual information to obtain the accurate position of the object; finally, the robot can obtain the three-dimensional coordinate data of the object to be picked up based on this information, that is, the position of the object in the three-dimensional space (including X, Y, Z coordinates), as the target position information of the object. This method not only relies on visual images and point cloud data, but can also further optimize positioning accuracy through the fusion of the two. The visual sensor provides surface shape information of the object, while the lidar provides depth and spatial position data of the object. Through this fusion of multimodal data, the robot can accurately identify objects and grasp them efficiently in complex environments.

[0034] Furthermore, the target location information determination module 13 is also used for:

[0035] According to the object contour, random fitting is performed in the regional point cloud set to obtain a first point cloud fitting result and a first fitting degree, wherein the first fitting degree is the ratio of the number of point clouds falling within the object contour to the total number of point clouds in the regional point cloud set; according to the object contour, random fitting is performed in the regional point cloud set again to obtain a second point cloud fitting result and a second fitting degree; iterative fitting is continued until a predetermined number of iterations is reached, and the point cloud fitting result with the largest fitting degree is output as the optimal point cloud fitting result.

[0036] Specifically, first, a random fit is performed in the regional point cloud set according to the object outline to obtain a first point cloud fitting result, and a first degree of fit is calculated, wherein the first degree of fit is the ratio of the number of point clouds falling within the object outline to the total number of point clouds in the regional point cloud set. The higher the degree of fit, the more consistent the fitting result is with the object outline. Next, a random fit is performed in the regional point cloud set again according to the object outline to obtain a second point cloud fitting result and a second degree of fit. Then, iterative fitting is continued until a predetermined number of iterations (such as 100 times) is reached, and the point cloud fitting result with the largest degree of fit is output as the optimal point cloud fitting result.

[0037] Furthermore, the object grabbing module 14 is also used for:

[0038] The current position information of the perception robot is read to avoid other perception robots as constraints, and a path is planned according to the current position information and the target position information to generate an object grabbing path; based on the object grabbing path and the target position information, the perception robot is controlled to grab the object to obtain the grabbed object.

[0039] Specifically, in a multi-robot collaborative system, especially an automatic handling robot system, when multiple robots perform tasks in the same environment, they often face path conflicts and collisions. In order to ensure that the robot can grab objects efficiently and accurately while avoiding collisions with other robots, it is necessary to design a dynamic path planning method that can take into account the current position of the robot while avoiding interference from other robots to achieve efficient collaborative work.

[0040] The current position information of the robot is obtained through the position sensor (such as GPS, inertial measurement unit, encoder, etc.) on the perception robot, which includes the real-time position (X, Y, Z coordinates) and posture of the robot; the target object is located by the perception robot to obtain the position information of the target object, which can be obtained by the visual sensor and laser radar combined with point cloud fitting mentioned above. Then, according to the current position information of the perception robot and the target position information of the target object, the path is planned with the constraint of avoiding other perception robots. Commonly used path planning algorithms include A* algorithm, Dijkstra algorithm, RRT (rapid random tree), etc. The appropriate algorithm can be selected according to the complexity and real-time requirements of the actual application. The surrounding environment information is obtained through the sensor in the perception robot (such as laser radar or ultrasonic sensor), and the path is adjusted in real time in combination with the local obstacle avoidance algorithm (such as dynamic window method, artificial potential field method, etc.) to obtain the object grasping path. Finally, based on the object grasping path and target position information, the robot control system accurately controls the movement of the robot so that it travels along the planned path and performs the grasping action when it reaches the position of the target object to obtain the grasped object.

[0041] Furthermore, the article handling module 15 is also used for:

[0042] Read the sample weight data of the item to be taken; configure an error tolerance interval, expand the sample weight data according to the error tolerance interval, and generate the fixed weight interval.

[0043] Specifically, in the automated handling robot system, the accuracy of object grasping not only depends on accurate object positioning, but is also closely related to the object's weight information. Usually, the perception robot obtains the object's weight data through a pressure sensor. However, the sensor's measurement results may be affected by a variety of factors such as environmental factors, sensor accuracy, and object characteristics, resulting in certain errors. In order to ensure the handling robot's grasping process is safe and reliable, it is necessary to set the error tolerance interval based on the sample weight data, thereby generating a suitable fixed weight interval.

[0044] Read the sample weight data of the item to be taken; then configure the error tolerance interval, that is, in order to cope with the influence of factors such as sensor error and external environment changes, it is necessary to configure an error tolerance interval, which is set according to the accuracy requirements in actual applications. The error tolerance interval is usually set as a percentage or a fixed value of the sample weight, for example, the error range is set to ±5%. Then, the sample weight data is expanded according to the error tolerance interval to generate the fixed weight interval.

[0045] Furthermore, the article handling module 15 is also used for:

[0046] After obtaining the grasped object, the storage location information of the predetermined storage area is obtained; based on the storage location information, an object transportation path is generated, and the perception robot is controlled to transport the grasped object to the predetermined storage area.

[0047] Specifically, after obtaining the grasped item, the storage location information of the predetermined storage area (such as the storage area coordinates) is obtained; then, the item transport path is generated based on the storage location information and the robot's current location information; finally, according to the item transport path, the perception robot is controlled to transport the grasped item to the predetermined storage area.

[0048] In summary, the multimodal fusion perception robot interaction system provided by this application has the following technical effects:

[0049] By activating the information perception unit of the perception robot, wherein the perception robot is a transport robot, and the information perception unit includes a visual sensor, a laser radar and a pressure sensor; then data is collected on the target area through the visual sensor and the laser radar to obtain the regional image and the regional point cloud set; then the target position information of the object to be picked up is determined according to the regional image and the regional point cloud set; further based on the target position information, an object grasping path is generated, and the perception robot is controlled to grasp the object to obtain the grasped object; finally, the weight information of the grasped object is collected through the pressure sensor, and if the weight information is in a predetermined weight range, the grasped object is transported to a predetermined storage area; that is, by combining multiple sensors such as visual sensors, laser radars, pressure sensors, etc., more comprehensive and accurate object perception and grasping can be achieved, significantly improving the accuracy and reliability of object grasping.

[0050] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0051] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the present application and its equivalent technology, the present application is also intended to include these modifications and variations.

Claims

1. The multi-modal fusion perception robot interaction system is characterized by: The multimodal fusion perception robot interaction system includes: An information perception unit activation module, used to activate the information perception unit of the perception robot, wherein the perception robot is a handling robot, and the information perception unit includes a visual sensor, a laser radar, and a pressure sensor; The target area scanning module is used to collect data on the target area through visual sensors and laser radars to obtain regional images and regional point cloud sets; A target location information determination module, used to determine the target location information of the object to be picked up according to the regional image and the regional point cloud set; An object grabbing module is used to generate an object grabbing path based on the target position information, control the perception robot to grab the object, and obtain the grabbed object; The article transport module is used to collect the weight information of the grasped article through a pressure sensor, and if the weight information is within a predetermined weight range, transport the grasped article to a predetermined storage area.

2. The multimodal fusion perception robot interaction system according to claim 1, characterized in that: The target area scanning module is also used for: Capture images of the target area through a visual sensor to obtain an initial area image; Scan the target area through laser radar to obtain the initial area point cloud set; According to a predetermined image processing scheme, performing image denoising and image enhancement on the initial regional image to obtain the regional image; The initial regional point cloud set is subjected to filtering processing and coordinate transformation to obtain the regional point cloud set.

3. The multimodal fusion perception robot interaction system according to claim 1, characterized in that: The target location information determination module is also used for: Acquire attribute characteristics of the item to be picked up, wherein the attribute characteristics include color characteristics, shape characteristics and size parameters; Performing a frame selection in the region image according to the color feature, shape feature, and size parameter to determine a target frame selection region; The area image is cropped according to the target framed area to obtain a framed image, and the target position information of the object to be picked up is determined based on the framed image.

4. The multi-modal fusion perception robot interaction system according to claim 3 is characterized in that: The target location information determination module is also used for: Extracting contour features of the framed image to obtain the contour of the object; Performing random fitting on the regional point cloud set according to the object contour to obtain an optimal point cloud fitting result; A regional three-dimensional space of the target area is constructed, coordinate positioning is performed in the regional three-dimensional space according to the optimal point cloud fitting result, and three-dimensional coordinate data of the object to be picked up is obtained as the target position information.

5. The multi-modal fusion perception robot interaction system according to claim 4 is characterized in that: The target location information determination module is also used for: Performing random fitting in the regional point cloud set according to the object contour to obtain a first point cloud fitting result and a first fitting degree, wherein the first fitting degree is a ratio of the number of point clouds falling within the object contour to the total number of point clouds in the regional point cloud set; Performing random fitting in the regional point cloud set again according to the object contour to obtain a second point cloud fitting result and a second fitting degree; Continue iterative fitting until the predetermined number of iterations is reached, and output the point cloud fitting result with the largest fitting degree as the optimal point cloud fitting result.

6. The multi-modal fusion perception robot interaction system according to claim 1, characterized in that: The item grabbing module is also used for: Read the current position information of the perception robot, avoid other perception robots as constraints, perform path planning according to the current position information and target position information, and generate an object grabbing path; Based on the object grasping path and target position information, the perception robot is controlled to grasp the object to obtain the grasped object.

7. The multi-modal fusion perception robot interaction system according to claim 1, characterized in that: The article handling module is also used for: Reading the sample weight data of the item to be taken; An error tolerance interval is configured, and the sample weight data is expanded according to the error tolerance interval to generate the fixed weight interval.

8. The multi-modal fusion perception robot interaction system according to claim 1, characterized in that: The article handling module is also used for: After obtaining the grasped object, obtaining the storage location information of the predetermined storage area; An article transport path is generated based on the storage location information, and the perception robot is controlled to transport the grasped article to a predetermined storage area.

Images