Material box type robot carrying system using 3D visual system and mechanical arm for automatic sorting

Abstract

The invention discloses a workbin type robot carrying system using a 3D visual system and a mechanical arm for automatic sorting, and particularly relates to the field of machine vision and artificial intelligence, the workbin type robot carrying system comprises an order recognition module, a warehouse-in order processing module, a warehouse-out order processing module and a standby module, and all the modules are connected through signals; the order identification module is used for identifying order information, generating an identification signal and sending the identification signal to the warehouse-in order processing module or the warehouse-out order processing module; the warehouse-in order processing module is used for processing the warehouse-in order and sending a processing completion signal to the standby module after processing is completed; the warehouse-out order processing module is used for processing the warehouse-out order and sending a processing completion signal to the standby module after processing is completed. Sorting operation can be directly carried out at the designated goods shelf position, the problem that a multi-layer shuttle vehicle or a mobile robot moves back and forth to the material position and the sorting point is solved, and the automatic storing and taking speed in storage is increased.

Description

Technical Field

[0001] The present invention relates to the fields of machine vision and artificial intelligence, and more specifically, to a material box type robot handling system using a 3D vision system and a mechanical arm for automatic sorting. Background Art

[0002] An automated warehouse system is a system that can automatically store and retrieve materials without direct manual processing. Warehouse automation is the process of using various automated equipment, software systems and technologies to perform operations such as storage, picking, handling, management and monitoring in the warehouse. Its goal is to improve warehouse operating efficiency, reduce costs and improve customer service through automation.

[0003] The multi-layer shuttle is a material handling device used in automated warehousing systems, usually used in large warehouses or distributed logistics centers. Compared with traditional shuttles, multi-layer shuttles can store and retrieve goods in the vertical direction, thereby effectively utilizing the three-dimensional space and improving the efficiency of warehouse logistics. Therefore, how to improve the driving and handling efficiency of multi-layer shuttles is the current focus; the existing multi-layer shuttle trajectory planning technology has the following defects: high computational complexity: traditional algorithms such as Dijkstra algorithm and A* algorithm have a long calculation time when processing large warehouses, especially when real-time response is required, and deadlock occurs as the number of iterations increases. Poor adaptability to dynamic environments: most traditional algorithms such as Dijkstra algorithm and A* algorithm assume that the environment is static when designed. Once the environment changes, the path needs to be recalculated, resulting in low efficiency. Therefore, how to improve the flexibility of trajectory planning results while improving the efficiency of trajectory planning is a difficulty in the prior art. For this purpose, a multi-layer shuttle trajectory planning method and system for unmanned warehousing is provided.

[0004] The Chinese patent "CN118518113A" provides a multi-layer shuttle trajectory planning method for unmanned warehousing, which generates a warehouse visualization image model based on the real-time status information of the warehouse rack and the dynamic shuttle track, and locates the target storage area in the warehouse visualization image model according to the volume of the goods, and then uses the current position of the target goods as the driving starting point and the target storage area as the driving end point. A number of expected driving routes are generated in the warehouse visualization image model, and driving simulation is performed on each expected driving route. The best driving route combination with the shortest time without collision is selected, and a dynamic shuttle track assembly decision is generated. The expected driving route is sent to the shuttle according to the best driving route, and the dynamic shuttle track is dynamically assembled according to the dynamic shuttle track assembly decision until all the goods are transported to the target storage area. It involves the field of shuttle trajectory planning technology, and effectively improves the efficiency of trajectory planning.

[0005] Automated high-bay warehouse is an important part of logistics warehousing. The use of high-bay warehouse equipment can realize the rationalization of warehouse high-rise, automatic access and simple operation. High-bay warehouse makes full use of high-rise space, saves inventory floor space, and significantly improves space utilization compared to ordinary warehouses.

[0006] Traditional stereoscopic warehouses are generally divided into stacker-type stereoscopic warehouses and shuttle-type stereoscopic warehouses. The biggest feature of the shuttle-type stereoscopic warehouse is its high storage density and full utilization of space, but it is impossible to pick up goods at will, and the cost of moving warehouses is very high. The stacker-type stereoscopic warehouse can pick up goods at will, but the storage space is not fully utilized, the storage density is low, and the access efficiency is low. Moreover, for these two types of stereoscopic warehouses, after the project is determined, the storage method and storage location of the automated guided vehicle (AGV) are fixed and cannot be flexibly allocated according to actual needs.

[0007] The Chinese patent "CN110775508A" provides a trackless AGV in-and-out method, which is applied to the AGV trolley in the AGV control system. The AGV control system also includes a stereoscopic warehouse, and each layer of the stereoscopic warehouse is distributed with multiple storage locations, and each type of storage location has a different priority. When entering the warehouse, the warehouse is entered according to the priority of the storage location, which improves the flexibility of the AGV in and out, and also improves the efficiency of the AGV in and out.

[0008] Warehouse automation solutions are developing rapidly both at home and abroad, the technology is maturing, and the application scenarios are gradually expanding. In the field of mobile robot technology for automated warehousing, automated storage and retrieval systems are being developed to improve the flexibility and accuracy of warehousing operations. Research in the direction of artificial intelligence is being conducted to optimize material sorting strategies. However, in the process of sorting order materials, the mobile robot needs to move back and forth between the shelves and the centralized sorting points. In order to reduce the moving time of the mobile robot, a material box robot handling system using a 3D vision system and a robotic arm for sorting is proposed to solve the above problems. Summary of the invention

[0009] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a box-type robot handling system using a 3D vision system and a robotic arm for automatic sorting to solve the problems raised in the above-mentioned background technology.

[0010] To achieve the above object, the present invention provides the following technical solutions:

[0011] A material box type robot handling system using a 3D vision system and a mechanical arm for automatic sorting, comprising an order recognition module, an incoming order processing module, an outgoing order processing module and a standby module, wherein each module is connected by a signal;

[0012] The order recognition module is used to recognize order information and generate a recognition signal to send to the inbound order processing module or the outbound order processing module;

[0013] The incoming order processing module is used to process the incoming order and send a processing completion signal to the standby module after the processing is completed;

[0014] The outbound order processing module is used to process the outbound order and send a processing completion signal to the standby module after the processing is completed;

[0015] The standby module is used for the material box robot to wait for the next order task.

[0016] In a preferred embodiment, the operation of the order identification module specifically includes the following:

[0017] Analyze order details, including material type and quantity;

[0018] According to the urgency of the order, set the outbound priority to optimize the picking sequence.

[0019] In a preferred embodiment, the operation of the incoming order processing module specifically includes the following:

[0020] When the order is an incoming order, the database is queried for empty shelf locations, the material box robot autonomously navigates to the empty shelf location, directly stores the material box at the designated location, and records the completion information.

[0021] In a preferred embodiment, the operation of the outbound order processing module specifically includes the following:

[0022] When the order is an outbound order, the shelf location is confirmed according to the material type in the order, and the optimal movement path is planned. The material box robot autonomously navigates to the empty material box storage area, takes the empty material box, and autonomously navigates to the shelf location specified for the material in the order. The material box robot takes out the material box from the shelf, identifies and locates the material, and sorts the material. After all the materials in the order are sorted, the robot moves to the packaging area for subsequent processing;

[0023] In a preferred embodiment, the bin is scanned by laser and a virtual 3D model of the bin is created for subsequent material classification and accurate grabbing and placement of the robot during sorting;

[0024] Point cloud preprocessing: Use filters to remove noise and abnormal points in the point cloud. If the point cloud data volume is too large, use voxel grid downsampling to reduce the number of points.

[0025] voxelize the preprocessed point cloud data, divide the three-dimensional space into uniform cubic grids by selecting the corresponding voxel size, and assign the points to the corresponding grids;

[0026] A voxel grid is constructed based on the occupancy status and attribute information of each voxel to form a discrete representation of the three-dimensional model. During debugging, the visualization tool in opecv3D is used to display the generated network.

[0027] In a preferred embodiment, the deep learning algorithm PointNet++ is used to realize material recognition and segmentation;

[0028] Use the KNN method to select the neighborhood of each point in the point cloud, extract features from the points in each neighborhood, and generate local features;

[0029] Perform multi-layer perceptron processing on each point to extract local features, and then use average pooling to aggregate local features to ensure insensitivity to the order of points;

[0030] All local features are aggregated to form a global feature representation, which is then further processed for classification or segmentation tasks;

[0031] According to the quantity of materials required in the order, the robot arm grabs the materials in the storage box into the outgoing box and verifies the grabbing results.

[0032] In a preferred embodiment, a hand-eye calibration algorithm is used to calculate the relative position and posture relationship of the camera in the robot arm to achieve precise grasping of the robot;

[0033] Get the relative position and posture relationship of the camera in the robotic arm;

[0034] The transformation matrix from the robot base coordinate system to the camera is marked as

[0035] The transformation matrix from the calibration plate to the camera is marked as

[0036] The transformation matrix from the base coordinates to the end of the robot arm is marked as

[0037] The transformation matrix from the end of the robot to the calibration plate is marked as Since the calibration plate is fixed at the end of the robotic arm, the transformation matrix from the end of the robotic arm to the calibration plate is the same for each set of images;

[0038] For each photo:

[0039]

[0040] Transformation

[0041] In a preferred embodiment, multiple pictures are solved to obtain the equation system AX=XB;

[0042] A, B, and X are all transformation matrices, and the system of equations has n-1 equations, where n is the number of pictures taken;

[0043]

[0044] Break AX=XB down into two equations:

[0045] R A R X =R X R B ;

[0046] (R A -1)T X =T B -R X T A ;

[0047] Solve for R first X , and then solve T X .

[0048] In a preferred embodiment, the operation of the standby module specifically includes the following:

[0049] After the order is processed, the box robot moves to the standby position and waits for the next order task.

[0050] The technical effects and advantages of the present invention are as follows:

[0051] 1. By equipping the mobile robot with a mechanical arm, the mechanical arm and the machine vision result enable the robot to directly perform in-and-out storage operations at the shelf position, reducing the time for the robot to move back and forth and improving storage efficiency;

[0052] 2. In order to realize automatic storage and retrieval of materials, the machine vision algorithm PointNet++ is used to help the robot identify and locate materials to ensure correct sorting of materials. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The schematic diagram of the structure of a material box type robot handling system for automatic sorting using a 3D vision system and a mechanical arm is shown in the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] Example 1

[0056] Figure 1 The present invention provides a material box type robot handling system for automatic sorting using a 3D vision system and a mechanical arm, comprising an order recognition module, an incoming order processing module, an outgoing order processing module and a standby module, wherein each module is connected by a signal;

[0057] The order recognition module is used to recognize order information and generate a recognition signal to send to the inbound order processing module or the outbound order processing module;

[0058] The incoming order processing module is used to process the incoming order and send a processing completion signal to the standby module after the processing is completed;

[0059] The outbound order processing module is used to process the outbound order and send a processing completion signal to the standby module after the processing is completed;

[0060] The standby module is used for the material box robot to wait for the next order task.

[0061] The operation of the order identification module specifically includes the following:

[0062] Analyze order details, including material type and quantity;

[0063] According to the urgency of the order, set the outbound priority to optimize the picking sequence.

[0064] It should be added that by carefully analyzing the material type and quantity in the order, the sorting system can more accurately identify the materials that need to be sorted. This helps to avoid sorting errors caused by unclear material information, such as putting the wrong material in the bin corresponding to the order.

[0065] Knowing the type and quantity of materials required for an order can help warehouses better manage inventory. Warehouse managers can prepare inventory in advance according to order requirements to avoid the situation where insufficient inventory of certain materials affects the order processing progress, and also prevent inventory backlogs.

[0066] Optimizing the sorting order according to the outbound priority can make the entire sorting process more orderly and efficient. Sorters and robots can process orders in order of priority, reducing unnecessary path duplication and time waste, and improving sorting efficiency.

[0067] The operation of the incoming order processing module specifically includes the following:

[0068] When the order is an incoming order, the database is queried for empty shelf locations, the material box robot autonomously navigates to the empty shelf location, directly stores the material box at the designated location, and records the completion information.

[0069] It should be added that by querying the empty shelf locations in the database, the material box robot can quickly determine the storage location of the material box, avoiding the time wasted by manually searching for empty shelves. This fast positioning method can make warehousing operations more efficient, especially in large warehouses with many shelves.

[0070] The robot will record the completion information after storing the material box in the designated location. This precise recording method can ensure the accuracy of inventory information. Warehouse managers can query the storage location and status of materials through the database at any time, which is convenient for inventory counting and management.

[0071] The operation of the outbound order processing module specifically includes the following:

[0072] When the order is an incoming order, the shelf location is confirmed according to the material type in the order, and the optimal movement path is planned. The box robot autonomously navigates to the empty box storage area, takes the empty box, and autonomously navigates to the shelf location specified for the material in the order. The box robot takes out the material box from the shelf, identifies and locates the material, and sorts the material. After all the materials in the order are sorted, the robot moves to the packaging area for subsequent processing;

[0073] It should be added that when the order is an outbound order, the shelf location can be quickly confirmed according to the material type in the order, and the optimal movement path can be planned. This reduces the time of searching for materials, enables the material box robot to quickly reach the target shelf, and greatly improves the speed of material acquisition.

[0074] Get the relative position and posture relationship of the camera in the robotic arm:

[0075] The transformation matrix from the robot base coordinate system to the camera is marked as

[0076] The transformation matrix from the calibration plate to the camera is marked as

[0077] The transformation matrix from the base coordinates to the end of the robot arm is marked as

[0078] The transformation matrix from the end of the robot to the calibration plate is marked as Since the calibration plate is fixed at the end of the robotic arm, the transformation matrix from the end of the robotic arm to the calibration plate is the same for each set of images;

[0079] For each photo:

[0080]

[0081] Transformation

[0082] Solve multiple pictures to get the equation system AX=XB;

[0083] A, B, and X are all transformation matrices, and the system of equations has n-1 equations, where n is the number of pictures taken;

[0084]

[0085] Break AX=XB down into two equations:

[0086] R A R X =R X R B ;

[0087] (R A -1)T X =T B -R X T A ;

[0088] Solve for R first X , and then solve T X .

[0089] Obtaining the relative position and posture relationship of the camera in the robot arm, as well as calculating and calibrating a series of transformation matrices, can allow the robot to know the location of the material more accurately. For example, when sorting materials, accurate hand-eye calibration can ensure that the robot arm accurately grasps the target material, avoiding grasping failure or grasping the wrong material.

[0090] This precise calibration enables the robot system to maintain high grasping accuracy in different working environments and material placement situations. Whether it is the different positions of materials on the shelf or the differences in shapes and sizes of different types of materials, the accurately calibrated robot can better cope with it, thus ensuring the accuracy and stability of the outbound operation.

[0091] Through accurate transformation matrix calculation, the robot arm can perform more optimized motion planning. The robot can move to the target position in the most reasonable path and posture to operate, reduce unnecessary movements and energy consumption, extend the service life of the equipment, and also help improve the operating efficiency and stability of the entire outbound order processing system.

[0092] It should be added that this precise calibration method can improve the accuracy of robot operation. For example, on industrial production lines, robots need to accurately identify and grasp parts. Through accurate hand-eye calibration, robots can reduce grasping errors and improve production efficiency and product quality.

[0093] For different working scenarios and tasks, the robot may need to adjust the position and posture of the camera. By solving this set of equations, the relationship between the camera and the robotic arm can be quickly recalibrated, allowing the robot to adapt to new working environments and task requirements.

[0094] Scan the bin with laser and create a virtual 3D model of the bin for subsequent material classification and accurate robot grasping and placement during sorting.

[0095] Point cloud preprocessing: Use filters to remove noise and abnormal points in the point cloud. If the point cloud data volume is too large, use voxel grid downsampling to reduce the number of points.

[0096] voxelize the preprocessed point cloud data, divide the three-dimensional space into uniform cubic grids by selecting the corresponding voxel size, and assign the points to the corresponding grids;

[0097] A voxel grid is constructed based on the occupancy status and attribute information of each voxel to form a discrete representation of the three-dimensional model. During debugging, the visualization tool in opecv3D is used to display the generated network.

[0098] It should be added that by creating a virtual 3D model of the bin, the robot can obtain the precise position and posture information of the materials in the bin in advance. During sorting, the robot can accurately grasp and place the materials based on this information, reducing grasping errors and inaccurate placement. The construction of the voxel grid and point cloud preprocessing can remove noise and abnormal points, avoiding these interference factors that cause deviations when the robot grasps or places materials.

[0099] The filter operation in point cloud preprocessing can effectively remove noise and abnormal points in the point cloud. Noise points may lead to incorrect judgments on the shape and position of bins and materials. Removing them can improve the accuracy and reliability of the data. When the amount of point cloud data is too large, voxel grid downsampling can reduce the amount of data while retaining key information. This helps to speed up subsequent data processing and avoid waste of computing resources and long processing time due to excessive data volume.

[0100] Accurate sorting operations can reduce losses caused by material damage and misoperation, and reduce operating costs. At the same time, efficient data processing methods can reduce dependence on high-performance hardware and save hardware procurement and maintenance costs.

[0101] Use the deep learning algorithm PointNet++ to achieve material recognition and segmentation;

[0102] Use the KNN method to select the neighborhood of each point in the point cloud, extract features from the points in each neighborhood, and generate local features;

[0103] Perform multi-layer perceptron processing on each point to extract local features, and then use average pooling to aggregate local features to ensure insensitivity to the order of points;

[0104] All local features are aggregated to form a global feature representation, which is then further processed for classification or segmentation tasks;

[0105] According to the quantity of materials required in the order, the robot arm grabs the materials in the storage box into the outgoing box and verifies the grabbing results.

[0106] It should be added that the deep learning algorithm PointNet++ can effectively capture local geometric information in the point cloud and is suitable for complex scenes, which helps to identify materials more accurately. When dealing with materials with diverse shapes, sizes and textures, it can provide more accurate recognition results than traditional methods. The accuracy of feature extraction is further enhanced by selecting neighborhoods and extracting features through the KNN method, as well as multi-layer perceptron processing and average pooling operations, thereby improving the accuracy of material recognition.

[0107] It can significantly improve the speed of material sorting and processing. Fast and accurate identification and grasping can process more orders in a shorter time and improve the overall work efficiency of warehouse operations.

[0108] The operation of the standby module specifically includes the following:

[0109] After the order is processed, the box robot moves to the standby position and waits for the next order task.

[0110] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0111] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0112] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0113] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned embodiments and will not be repeated here.

[0114] In the several embodiments provided in the present application, it should be understood that the disclosed systems and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0115] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0116] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0117] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0118] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0119] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A box-type robot handling system using a 3D vision system and a robotic arm for automatic sorting, characterized in that: It includes an order recognition module, an incoming order processing module, an outgoing order processing module and a standby module, and each module is connected by signals; The order recognition module is used to recognize order information and generate a recognition signal to send to the inbound order processing module or the outbound order processing module; The incoming order processing module is used to process the incoming order and send a processing completion signal to the standby module after the processing is completed; The outbound order processing module is used to process the outbound order and send a processing completion signal to the standby module after the processing is completed; The standby module is used for the material box robot to wait for the next order task.

2. A material box type robot handling system using a 3D vision system and a robotic arm for automatic sorting according to claim 1, characterized in that: The operation of the order identification module specifically includes the following: Analyze order details, including material type and quantity; According to the urgency of the order, set the outbound priority to optimize the picking sequence.

3. The box-type robot handling system for automatic sorting using a 3D vision system and a robotic arm according to claim 1, characterized in that: The operation of the incoming order processing module specifically includes the following: When the order is an incoming order, the database is queried for empty shelf locations, the material box robot autonomously navigates to the empty shelf location, directly stores the material box at the designated location, and records the completion information.

4. The box-type robot handling system for automatic sorting using a 3D vision system and a robotic arm according to claim 1, characterized in that: The operation of the outbound order processing module specifically includes the following: When the order is an incoming order, the shelf location is confirmed according to the material type in the order, and the optimal movement path is planned. The material box robot autonomously navigates to the empty material box storage area, takes the empty material box, and autonomously navigates to the specified shelf location of the material in the order. The material box robot takes out the material box from the shelf, identifies and locates the material, and sorts the material. After all the materials in the order are sorted, the robot moves to the packaging area for subsequent processing.

5. The box-type robot handling system for automatic sorting using a 3D vision system and a robotic arm according to claim 4, characterized in that: Get the relative position and posture relationship of the camera in the robotic arm: The transformation matrix from the robot base coordinate system to the camera is marked as The transformation matrix from the calibration plate to the camera is marked as The transformation matrix from the base coordinates to the end of the robot arm is marked as The transformation matrix from the end of the robot to the calibration plate is marked as Since the calibration plate is fixed at the end of the robotic arm, the transformation matrix from the end of the robotic arm to the calibration plate is the same for each set of images; For each photo: Transformation Solve multiple pictures to get the equation system AX=XB; A, B, and X are all transformation matrices, and the system of equations has n-1 equations, where n is the number of pictures taken; Break AX=XB down into two equations: R A R X =R X R B ; (R A -1)T X =T B -R X T A ; Solve for R first X , and then solve T X .

6. The box-type robot handling system for automatic sorting using a 3D vision system and a robotic arm according to claim 5, characterized in that: Scan the bin with laser and create a virtual 3D model of the bin for subsequent material classification and accurate robot grasping and placement during sorting. Point cloud preprocessing: Use filters to remove noise and abnormal points in the point cloud. If the point cloud data volume is too large, use voxel grid downsampling to reduce the number of points. voxelize the preprocessed point cloud data, divide the three-dimensional space into uniform cubic grids by selecting the corresponding voxel size, and assign the points to the corresponding grids; A voxel grid is constructed based on the occupancy status and attribute information of each voxel to form a discrete representation of the three-dimensional model. During debugging, the visualization tool in opecv3D is used to display the generated network.

7. The box-type robot handling system for automatic sorting using a 3D vision system and a robotic arm according to claim 6, characterized in that: Use the deep learning algorithm PointNet++ to achieve material recognition and segmentation; Use the KNN method to select the neighborhood of each point in the point cloud, extract features from the points in each neighborhood, and generate local features; Perform multi-layer perceptron processing on each point to extract local features, and then use average pooling to aggregate local features to ensure insensitivity to the order of points; All local features are aggregated to form a global feature representation, which is then further processed for classification or segmentation tasks; According to the quantity of materials required in the order, the robot arm grabs the materials in the storage box into the outgoing box and verifies the grabbing results.

8. The box-type robot handling system for automatic sorting using a 3D vision system and a robotic arm according to claim 7, characterized in that: The operation of the standby module specifically includes the following: After the order is processed, the box robot moves to the standby position and waits for the next order task.

Images