Unloading robot system control platform

Abstract

An unloading robot system control platform belongs to the technical field of automatic logistics. The problem that an existing unloading robot system is poor in control flexibility and operability is solved. Comprising an image acquisition and target identification unit, a robot motion planning unit, a robot motion control unit, a data processing and storage unit, a communication module and a visual interaction module. The position and the shape of a target are obtained by collecting image information and point cloud information of the target, a robot system is detected to judge whether the robot system has a fault or not, and the motion posture of the robot at the next moment is planned according to the fault, the position and the shape of the target and a task priority sequence table. And the robot motion control unit controls robot hardware to move by using the planning information. The method is suitable for controlling the automatic logistics technology robot.

Description

Technical Field

[0001] The invention belongs to the technical field of automated logistics. Background Art

[0002] In today's rapidly developing smart manufacturing and logistics industries, the booming e-commerce and express delivery businesses are driving the transformation of the logistics industry. Businesses are increasingly demanding higher levels of logistics efficiency and service quality. Traditional manual unloading methods are no longer able to meet market demands, facing numerous challenges such as rising labor costs, low efficiency, and safety risks. At the same time, with the continuous advancement of technology, the use of automated equipment has gradually become a key means of improving logistics efficiency. Intelligent unloading robots, with their efficiency, flexibility, and precision, can quickly and automatically unload a wide range of products, significantly improving production efficiency and reducing labor costs. The rapid development and application of this technology has made automated unloading robots a key component of modern logistics systems, driving comprehensive upgrades and innovations in the industry.

[0003] The core technologies of automated unloading robot systems can be divided into two major components: hardware and software. These two components complement each other and together determine the overall performance of the system. On the hardware side, intelligent robots are typically equipped with high-precision sensors such as lidar, stereo cameras, and IMUs to achieve real-time perception and positioning of their surroundings. Furthermore, a powerful powertrain and flexible robotic arm design are essential, enabling the robot to adapt to diverse unloading environments and perform precise grasping and handling tasks. On the software side, the system requires multiple functions, including environmental modeling, real-time data processing, task planning and scheduling, and human-machine interaction. Specifically, the control software must be able to process massive amounts of sensor data in real time, generate environmental models, and intelligently allocate and schedule tasks based on these models. Therefore, developing an efficient and stable software control platform is crucial. This platform integrates multiple key technologies to ensure that automated unloading robots can complete their tasks efficiently and accurately.

[0004] The software platform for the unloading robot system should not only provide basic control functions but also advanced features such as real-time monitoring, data analysis, task scheduling, and human-computer interaction. Its design architecture should adopt a modular concept to ensure system flexibility and scalability, allowing technicians to select and combine different functional modules according to specific needs. Each module should have independent functions and be able to seamlessly integrate with other modules to ensure the efficiency and reliability of the overall system operation. However, difficulties that require attention in the design include inter-module compatibility, system response speed, and algorithm optimization. These factors directly affect the stability and real-time performance of the entire system. Therefore, when designing the system architecture, it is necessary to comprehensively consider the collaboration and information sharing between modules to improve the performance of the entire system.

[0005] Domestic research and application of unloading robot system software platforms lags behind. Although many products have emerged on the market, most lack independent innovation in core technologies and still rely on foreign technology. This situation restricts domestic systems in terms of functional scalability and customization, making them unable to meet the complex and ever-changing needs of the industry. Existing unloading robot system software platform technology has the following main shortcomings:

[0006] Lack of unified standards and specifications: Existing unloading robot system software platforms often lack unified development standards and interface specifications, resulting in poor compatibility between different modules and devices. This requires developers to spend a lot of time adapting the system during system integration, reducing system flexibility and scalability and increasing operation and maintenance costs.

[0007] Insufficient modularity: Existing software platforms lack modular design. Functionality overlaps or lacks independence between modules, making it difficult to flexibly configure the system based on specific needs. This design limits technicians' ability to quickly adjust the system based on actual application scenarios, impacting the robot's performance in different operating environments.

[0008] Inadequate human-computer interaction experience: Existing software platforms have shortcomings in human-computer interaction design, often lacking a friendly user interface and convenient operation procedures. This makes it difficult for technicians to respond to situations in real time when using the system, increasing operational difficulty.

[0009] Lack of task scheduling capabilities: Current software platforms have significant shortcomings in task scheduling, often relying on static scheduling strategies rather than dynamic adjustments. This makes it difficult for the system to effectively allocate tasks and optimize resources when faced with complex and changing tasks. Summary of the Invention

[0010] The present invention aims to solve the problems of poor flexibility and operability in the control of existing unloading robot systems, and now provides a control platform for the unloading robot system.

[0011] The unloading robot system control platform of the present invention includes an image acquisition and target recognition unit, a robot motion planning unit, a robot motion control unit, a data processing and storage unit, a communication module and a visual interaction module;

[0012] The image acquisition and target recognition unit is used to collect the image information and point cloud information of the target, identify the target using the collected information, and detect the target position and shape; the target position and shape are sent through the communication module;

[0013] The robot motion planning unit is used to detect the robot system in real time and determine whether there is a fault in the robot system. If there is a fault, the fault information is sent out through the communication module. The position and shape of the target and the task priority sequence table are received through the communication module. The robot's motion posture at the next moment is planned according to the target position and shape, the task priority sequence table and the fault information, and the robot's motion posture at the next moment is sent out through the communication module.

[0014] The visual interaction module provides a port for setting and adjusting the priority of the robot task; the task priority setting and adjustment information is sent out through the communication module;

[0015] The data processing and storage unit receives the task setting and adjustment information through the communication module, generates a task priority sequence table in real time according to the system fault information and the task setting and adjustment information, stores the task priority sequence table and sends it out through the communication module;

[0016] The robot motion control unit receives the robot's next motion posture information through the communication module and controls the robot hardware to move;

[0017] The visual interaction module receives the system's priority sequence list, the robot's next moment motion posture, target position and target shape through the communication module and displays them.

[0018] Furthermore, in the present invention, the image acquisition and target recognition unit includes a depth camera, a pre-processing module and a target detection and recognition module;

[0019] The depth camera is used to collect RGB image information and point cloud information of the target and send the collected information to the preprocessing module;

[0020] The laser radar is used to collect the distance between the target and the robot body and send the collected information to the preprocessing module;

[0021] The preprocessing module preprocesses the received information and sends the preprocessed information to the target detection and recognition module;

[0022] The target detection and recognition module uses the received information to identify the shape and position of the target and sends the shape and position of the target through the communication module.

[0023] Furthermore, in the present invention, the process of the target detection and recognition module using the received information to identify the shape and position of the target is as follows:

[0024] Step 1: Preprocess the point cloud information collected by the depth camera to obtain preprocessed point cloud data;

[0025] Step 2: Use the parameter-limited RANSAC method to extract the plane in the depth direction of the pre-processed point cloud data to obtain the point cloud plane closest to the camera, and then pre-process the point cloud plane; obtain the pre-processed point cloud plane P first_inlier ;

[0026] Step 3: Based on the camera intrinsic parameter matrix K, the point cloud plane P first_inlier Mapped to the two-dimensional pixel coordinates of the RGB image captured by the depth camera 11, the point set S0 is obtained; based on the boundary range of the point set S0, a rectangular area ROI is established, the ROI range is cropped in the RGB image, and the Canny edge detection algorithm is used to extract the boundary point set S1 of the cropped RGB image, and the point set S1 is inversely mapped back to the plane point cloud P first_inlier After filtering out the boundary of the point cloud plane mapped back to the point cloud, we can obtain the point cloud P containing the front surface information of multiple independent cartons. cam ;

[0027] Step 4: Use a clustering segmentation algorithm based on surface growth to cluster the point cloud P containing the front surface information of multiple independent cartons. cam Split into multiple independent point cloud clusters;

[0028] Step 5: Create a minimum cube outside each independent point cloud cluster that can contain all points in the point cloud cluster; use the spatial coordinates and side length of the minimum cube as the spatial position coordinates and outline dimensions of the target.

[0029] Furthermore, in the present invention, in step 3, the boundary is inversely mapped to the plane point cloud P first_inlier After filtering out the boundary point cloud, we obtain the point cloud P containing the front surface information of multiple independent cartons. cam The process is:

[0030] Set the camera center to the point cloud plane P first_inlier The distance is used as the plane depth Distance first_inlier , combined with the camera intrinsic parameter matrix, the two-dimensional boundary point set S1 of the RGB image is inversely mapped back to the camera coordinate system of the three-dimensional point cloud to obtain the three-dimensional point set P boundary , with a three-dimensional point set P boundary Each point in is taken as the origin and the radius r is established b The spherical space of the preprocessed point cloud plane P first_inlier The points whose coordinates are in the spherical space are marked as boundary points. first_inlier Eliminate all boundary points and obtain the point cloud P containing the front surface information of multiple independent cartons cam .

[0031] Furthermore, in the present invention, a clustering segmentation algorithm based on surface growth is used to cluster the point cloud P containing the front surface information of multiple independent cartons. cam The method of segmenting into multiple independent point cloud clusters is:

[0032] Step S41: From point cloud P cam Select any point P i , search for point P i The neighborhood centered on is calculated by calculating the ratio of the minimum eigenvalue of the neighborhood covariance matrix to the sum of the eigenvalues, and the ratio is used as the curvature κ i ;

[0033] At the same time, the covariance matrix of the neighborhood is used to calculate the point P i Normal vector n at the center i , and then calculate the point P i Any point p in the neighborhood j The covariance matrix of the neighborhood centered on point p is then calculated to obtain j Normal vector n at the center j ; Calculate the normal vector n i and normal vector n j Angle;

[0034] Step S42: Set parameter smoothness threshold θ th and curvature threshold κ th ; When θ is satisfied ij <θ th ,κ i <κ th When P i Neighborhood point p j Point P i Class, traverse P i All points in the neighborhood, get the point p i Starting point cloud cluster;

[0035] Step S43: From point cloud P cam Select the unclassified point P k , execute step S41 and step S42 until the point cloud P cam All points are classified, and the point cloud P containing the front surface information of multiple independent cartons is completed. cam Split into multiple independent point cloud clusters.

[0036] Furthermore, in the present invention, the robot motion planning unit includes a robot industrial computer, a motion planning module and a system detection module;

[0037] The system detection module is used to detect the robot system and determine whether there is a fault. If there is a fault, the detected fault information is sent out through the communication module; at the same time, the fault information is transmitted to the robot industrial computer;

[0038] The robot's industrial computer receives the fault information and transmits the system's target task and fault information to the motion planning module;

[0039] The motion planning module uses the system's target task, fault information, target position and shape to plan the robot's motion posture at the next moment, and sends the robot's motion posture planning information at the next moment through the communication module.

[0040] Furthermore, in the present invention, the method in which the motion planning module plans the motion posture of the robot at the next moment is:

[0041] Using the real-time information, the Dijkstra algorithm is used to determine the optimal path for the robot to reach the target from its current position. Then, based on the inverse kinematics algorithm, the motion trajectory of each joint of the robot's mechanical arm is calculated to obtain the robot's motion posture at the next moment.

[0042] Furthermore, in the present invention, the robot motion control unit includes a motion control module and a robot motion controller;

[0043] The motion control module receives the motion posture planning information of the robot at the next moment through the communication module, obtains the posture control information of the robot at the next moment, and sends the obtained information to the robot motion controller;

[0044] The robot motion controller uses the received posture control information to send a motion control signal to the robot control system, receives the motion state of the robot hardware, and sends the motion state information of the robot hardware through the communication module.

[0045] Furthermore, in the present invention, the data processing and storage unit includes a data statistics and analysis module, a task management and statistics module, a database management module and a database;

[0046] The data statistics and analysis module is used to receive fault information, the next moment's robot posture control information, the shape and position of the target, and the motion status information of the robot hardware through the communication module, and send them to the database management module according to the category;

[0047] The task management and statistics module uses a system that supports concurrent execution of multiple tasks and a priority-based task scheduling strategy to generate a priority sequence table, sends the priority sequence table to the database management module, and sends it out through the communication module;

[0048] The database management module is used to send the priority sequence list and robot system fault information, the posture control information of the robot to be received at the next moment, the shape and position of the target, and the motion status information of the robot hardware to the database according to the category;

[0049] The database is used to store the received data.

[0050] Furthermore, the present invention also includes a power management module, which is used to detect the remaining power of the power supply. When the remaining power of the power supply is lower than a preset safety threshold, the power management module sends a warning signal through the communication module, and adjusts the remaining battery power according to the importance of the task to power the hardware corresponding to the high-priority task.

[0051] The present invention designs an integrated and modular software operating platform for the corresponding unloading robot hardware mechanical system. Through modular design, the platform allows technicians to flexibly configure functional modules according to specific needs and quickly implement functional upgrades and adjustments to the system. The platform combines advanced intelligent scheduling algorithms and real-time monitoring technologies to improve the robot's adaptability and task execution efficiency in complex environments. By establishing unified development standards and interface specifications, it promotes interoperability between different devices and software, and promotes the technological progress and market development of domestic unloading robot systems. The present invention can not only improve the overall performance of the unloading robot system, but also provide strong support for the future development of intelligent manufacturing, and help China's independent innovation and development in the field of intelligent logistics. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a block diagram of the system principles of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other in the absence of conflict.

[0054] Specific implementation method 1: refer to Figure 1 Specifically describing this embodiment, the unloading robot system control platform described in this embodiment comprises an image acquisition and target recognition unit 1, a robot motion planning unit 2, a robot motion control unit 3, a data processing and storage unit 4, a communication module 5, and a visual interaction module 6;

[0055] The image acquisition and target recognition unit 1 is used to acquire the image information and point cloud information of the target, identify the target using the acquired information, and detect the target position and shape; the target position and shape are sent out through the communication module 5;

[0056] The robot motion planning unit 2 is used to detect the robot system in real time and determine whether there is a fault in the robot system. If there is a fault, the fault information is sent out through the communication module 5. The position and shape of the target and the task priority sequence table are received through the communication module 5. The robot motion planning unit 2 plans the robot's motion posture at the next moment according to the target position and shape, the task priority sequence table and the fault information, and sends the robot's motion posture at the next moment through the communication module 5.

[0057] The visual interaction module 6 provides a port for setting and adjusting the priority of the robot task; the task priority setting and adjustment information is sent out through the communication module 5;

[0058] The data processing and storage unit 4 receives the task setting and adjustment information through the communication module 5, generates a task priority sequence table in real time according to the system fault information and the task setting and adjustment information, stores the task priority sequence table and sends it out through the communication module 5;

[0059] The robot motion control unit 3 receives the robot's next motion posture information through the communication module 5 and controls the robot hardware to move;

[0060] The visual interaction module 6 receives the system's priority sequence table, the robot's next moment motion posture, target position and target shape through the communication module 5 and displays them.

[0061] Furthermore, in the present invention, the image acquisition and target recognition unit 1 includes a depth camera 11, a pre-processing module 12 and a target detection and recognition module 13;

[0062] The depth camera 11 is used to collect RGB image information and point cloud information of the target and send the collected information to the pre-processing module 12;

[0063] The laser radar (12) is used to collect the distance between the target and the robot body and send the collected information to the pre-processing module 12;

[0064] The preprocessing module 12 preprocesses the received information and sends the preprocessed information to the target detection and recognition module 13;

[0065] The target detection and recognition module 13 uses the received information to recognize the shape and position of the target, and sends the shape and position of the target through the communication module 5.

[0066] Furthermore, in the present invention, the process in which the target detection and recognition module 13 uses the received information to recognize the shape and position of the target is as follows:

[0067] Step 1: pre-process the point cloud information collected by the depth camera 11 to obtain pre-processed point cloud data;

[0068] Step 2: Use the parameter-limited RANSAC method to extract the plane in the depth direction of the pre-processed point cloud data to obtain the point cloud plane closest to the camera, and then pre-process the point cloud plane; obtain the pre-processed point cloud plane P first_inlier ;

[0069] Step 3: Based on the camera intrinsic parameter matrix K, the point cloud plane P first_inlier Mapped to the two-dimensional pixel coordinates of the RGB image captured by the depth camera 11, the point set S0 is obtained; based on the boundary range of the point set S0, a rectangular area ROI is established, the ROI range is cropped in the RGB image, and the Canny edge detection algorithm is used to extract the boundary point set S1 of the cropped RGB image, and the point set S1 is inversely mapped back to the plane point cloud P first_inlier After filtering out the boundary of the point cloud plane mapped back to the point cloud, we can obtain the point cloud P containing the front surface information of multiple independent cartons. cam ;

[0070] Step 4: Use a clustering segmentation algorithm based on surface growth to cluster the point cloud P containing the front surface information of multiple independent cartons. cam Split into multiple independent point cloud clusters;

[0071] Step 5: Create a minimum cube outside each independent point cloud cluster that can contain all points in the point cloud cluster; use the spatial coordinates and side length of the minimum cube as the spatial position coordinates and outline dimensions of the target.

[0072] Furthermore, in the present invention, in step 3, the boundary is inversely mapped to the plane point cloud P first_inlier After filtering out the boundary point cloud, we obtain the point cloud P containing the front surface information of multiple independent cartons. cam The process is:

[0073] Set the camera center to the point cloud plane P first_inlier The distance is used as the plane depth Distance first_inlier , combined with the camera intrinsic parameter matrix, the two-dimensional boundary point set S1 of the RGB image is inversely mapped back to the camera coordinate system of the three-dimensional point cloud to obtain the three-dimensional point set P boundary , with a three-dimensional point set P boundary Each point in is taken as the origin and the radius r is established b The spherical space of the preprocessed point cloud plane P first_inlier The points whose coordinates are in the spherical space are marked as boundary points. first_inlier Eliminate all boundary points and obtain the point cloud P containing the front surface information of multiple independent cartons cam .

[0074] Furthermore, in the present invention, a clustering segmentation algorithm based on surface growth is used to cluster the point cloud P containing the front surface information of multiple independent cartons. cam The method of segmenting into multiple independent point cloud clusters is:

[0075] Step S41: From point cloud P cam Select any point P i , search for point P i The neighborhood centered on is calculated by calculating the ratio of the minimum eigenvalue of the neighborhood covariance matrix to the sum of the eigenvalues, and the ratio is used as the curvature κ i ;

[0076] At the same time, the covariance matrix of the neighborhood is used to calculate the point P i Normal vector n at the center i , and then calculate the point P i Any point p in the neighborhood j The covariance matrix of the neighborhood centered on point p is then calculated to obtain j Normal vector n at the center j ; Calculate the normal vector n i and normal vector n j Angle;

[0077] Step S42: Set parameter smoothness threshold θ th and curvature threshold κ th ; When θ is satisfied ij <θ th ,κ i <κ th When P i Neighborhood point p j Point P i Class, traverse P i All points in the neighborhood, get the point p i Starting point cloud cluster;

[0078] Step S43: From point cloud P cam Select the unclassified point P k , execute step S41 and step S42 until the point cloud P cam All points are classified, and the point cloud P containing the front surface information of multiple independent cartons is completed. cam Split into multiple independent point cloud clusters.

[0079] Furthermore, in the present invention, the robot motion planning unit 2 includes a robot industrial computer 21, a motion planning module 22 and a system detection module 23;

[0080] The system detection module 23 is used to detect the robot system and determine whether there is a fault. If there is a fault, the detected fault information is sent out through the communication module 5; at the same time, the fault information is transmitted to the robot industrial computer 21;

[0081] The robot industrial computer 21 receives the fault information and transmits the system's target task and fault information to the motion planning module 22;

[0082] The motion planning module 22 plans the robot's motion posture at the next moment using the system's target task, fault information, target position and shape, and sends the robot's motion posture planning information at the next moment through the communication module 5.

[0083] Furthermore, in the present invention, the method by which the motion planning module 22 plans the motion posture of the robot at the next moment is:

[0084] Using the real-time information, the Dijkstra algorithm is used to determine the optimal path for the robot to reach the target from its current position. Then, based on the inverse kinematics algorithm, the motion trajectory of each joint of the robot's mechanical arm is calculated to obtain the robot's motion posture at the next moment.

[0085] Furthermore, in the present invention, the robot motion control unit 3 includes a motion control module 31 and a robot motion controller 32;

[0086] The motion control module 31 receives the motion posture planning information of the robot at the next moment through the communication module 5, obtains the posture control information of the robot at the next moment, and sends the obtained information to the robot motion controller 32;

[0087] The robot motion controller 32 uses the received posture control information to send a motion control signal to the robot control system, receives the motion state of the robot hardware, and sends the motion state information of the robot hardware through the communication module 5.

[0088] Furthermore, in the present invention, the data processing and storage unit 4 includes a data statistics and analysis module 41, a task management and statistics module 42, a database management module 43 and a database 44;

[0089] The data statistics and analysis module 41 is used to receive fault information, the next moment robot posture control information, the shape and position of the target, and the motion status information of the robot hardware through the communication module 5, and send them to the database management module 43 according to the category;

[0090] The task management and statistics module 42 generates a priority sequence table using a system that supports concurrent execution of multiple tasks and a priority-based task scheduling strategy, sends the priority sequence table to the database management module, and sends it out through the communication module;

[0091] The database management module 43 is used to send the priority sequence table and robot system fault information, the posture control information of the robot to be received at the next moment, the shape and position of the target, and the motion status information of the robot hardware to the database 44 according to the category;

[0092] The database 44 is used to store the received data.

[0093] In this embodiment, the database is also used to provide a data reading port for the visualization interaction module 6 , and the visualization interaction module 6 reads data via the database management module 43 through the communication module.

[0094] Furthermore, the present invention also includes a power management module, which is used to detect the remaining power of the power supply. When the remaining power of the power supply is lower than a preset safety threshold, the power management module sends a warning signal through the communication module, and adjusts the remaining battery power according to the importance of the task to power the hardware corresponding to the high-priority task.

[0095] The purpose of this invention is to design a software control platform for the corresponding automatic unloading robot system to solve the control requirements and information feedback of the automatic unloading robot in actual work and realize multi-faceted human-computer interaction.

[0096] The software platform will integrate multiple key functions, including system component status monitoring, communication technology, data storage and processing technology, motion control, multi-task management, and human-computer interaction. Through the system architecture design and unification of application program interfaces for key functional modules, the standardization and efficiency of the software platform will be ensured. By establishing a comprehensive database and enabling efficient data access, the platform can provide technicians with a rich graphical interactive interface and convenient operation methods based on the working status of hardware components, algorithm processing of the visual system, and the coordination function of multi-task management. This design not only significantly improves the technician experience, but also effectively meets the expansion of personalized needs, supports technicians' flexible application in different application scenarios, promotes the industrialization of machine vision technology, and provides strong support for the continuous development and innovation in the field of intelligent logistics.

[0097] In general, in the design process of the present invention, first, based on the corresponding automatic unloading robot hardware system structure and work requirements, the functions that the unloading robot system software platform should have are abstracted, including system detection, target detection algorithm, communication control, human-computer interaction, and multi-task management. The difficulties in designing each function are summarized, and the object-oriented design method of modern software engineering is adopted to propose an overall design scheme of the software platform based on the modular component model and detailed design schemes of each functional module. Secondly, based on the layered architecture in modularization, software reconstruction technology and agile development methods of modern software engineering, the present invention completes the design and implementation of seven basic functional modules and underlying algorithm support modules of the software platform, including system detection module, network communication module, motion control module, database management module, task management module, power management module and AR interaction module, and completes the interface design and unit testing of each functional module. In response to the complex problem of information collection and processing of mixed cargo stacks by the unloading robot vision system, the present invention proposes an underlying algorithm structure framework including point cloud processing algorithm module, image processing algorithm module and motion planning algorithm module based on modularization method and basic visual algorithm libraries such as OpenCV and PCL. By programming the underlying visual recognition and positioning algorithms, the algorithm module can recognize and detect different types of image information and mixed stacked object frames of various types. It also achieves decoupling from other functional modules, improving the efficiency of algorithm operation and integration. Finally, the present invention integrates various functional modules, develops a complete unloading robot system software platform, and completes interactive testing and application of the entire system in actual industrial inspection tasks.

[0098] The specific research contents are as follows:

[0099] Software platform requirements analysis and overall solution design

[0100] Based on the corresponding automatic unloading robot hardware system structure and working requirements, the present invention abstracts the functions that the unloading robot system software platform should have, including system detection, target algorithm, human-computer interaction and other aspects, and summarizes the difficulties in designing each function. Based on the object-oriented design method of modern software engineering, the present invention proposes an overall design scheme of the software platform based on the modular component model and the specific design schemes of each functional module.

[0101] The core function of the software platform for the unloading robot system lies in the coordination and control capabilities of the entire robot system, providing standardized and convenient operation solutions, improving the work efficiency of the unloading robot system, and ensuring flexibility to changes in demand to adapt to the expansion of platform functions. The overall structure of the automatic unloading robot system targeted by the present invention covers dozens of hardware and software parts such as industrial cameras, lidars, POE switches, embedded controllers, servo motors, servo drivers, power supplies, communication protocols, PC-side hardware interfaces and software control platforms. Through the analysis of component detection requirements and the grasp of software platform functions, the present invention divides the functional modules of the software platform into seven parts, including the system detection module. Taking these seven core functional modules and the underlying algorithm support modules as the basic units, and the platform performance indicators as the research direction, the specific functional requirements of the unloading robot system software platform are deeply analyzed, and they are met and adjusted in the solution design.

[0102] The present invention is mainly developed using C# and C++ programming languages. These languages ​​are closer to the underlying physical platform, can guarantee module execution efficiency, are compatible with rich class libraries, and are the mainstream languages ​​for current information control and machine vision development. In order to ensure development efficiency and optimization of platform source code, the present invention selects the VC++ integrated development environment under the Windows system as the development environment. In addition, the present invention uses Visual Studio 2017 to develop the class library of the software platform, and uses Unity2022LTS to write the visual interface and test the class library. These two development tools have rich algorithm class libraries and mature development experience and resources, which contribute to the rapid deployment and development of the software platform.

[0103] The software platform design for the unloading robot system is as follows: In the underlying algorithm module, target detection and recognition are implemented based on the OpenCV and PCL basic visual algorithm libraries. RGB image information and point cloud data of mixed material stacks are processed to identify target spatial information, and positioning and grasping are performed using motion planning algorithms. In the basic function module, information is transmitted between sensors and drive elements by rewriting and repackaging the Modbus TCP protocol, using servo drives and communication protocols. Information feedback from various components and the visual system is monitored in real time using a human-computer interaction interface. A unified functional interface is designed with a visual control interface as the core, integrating various functional components to provide technicians with an intuitive, convenient, and efficient visual intelligent operation solution.

[0104] Here, the entire system is divided into architecture and functions according to the overall design plan. Based on the layered architecture model of modular design theory, the present invention divides the entire platform design into four levels, as follows:

[0105] (1) The underlying dependency libraries include OpenCV, PCL and other basic visual algorithm libraries, camera SDK libraries, lidar SDK libraries, Modbus TCP communication protocol libraries, Windows operating system libraries, databases and other underlying resource libraries.

[0106] These are the foundations on which the software platform operates.

[0107] (2) Interface layer function library This layer provides a set of more abstract and general interfaces based on the bottom layer, hiding the complexity of the bottom layer, so that developers at the upper layer can use the bottom layer functions more conveniently. This layer is based on the component design method in modular design and is divided into two categories: bottom-level algorithm support module and basic function module. The bottom-level algorithm module includes three parts: image processing algorithm module, point cloud processing algorithm module and motion planning algorithm module. The basic function module includes seven parts: system detection module, network communication module, motion control module, database management module, task management module, power management module and AR interaction module. The responsibilities and functional boundaries of each sub-module are clarified, and the modules are encapsulated in the form of a link library, leaving only the necessary functional interfaces for the main program. When developing control software, the main program calls the required functions to the corresponding module interface, reads the communication and data between all modules, and performs functional splicing of the overall software system. This method can achieve complete decoupling between functional modules and increase the functional adjustability of the software platform.

[0108] (3) Business Logic Layer In a multi-layer architecture, the business logic layer is usually located between the interface layer and the application layer. The main responsibility of the business logic layer is to implement the business logic of the application. These business logics are defined by the business requirements of the application, such as the ROI area of ​​​​target detection, motor control, multi-task management, etc. The business logic layer relies on the common interface provided by the functional class library to integrate multiple types of hardware and software functions, decoupling the business logic layer from the specific underlying implementation. This layer implements business functions for specific work needs by combining and encapsulating business services.

[0109] (4) Application layer software: This is the top layer of the software platform. It is aimed at end-user technicians and provides an intuitive and easy-to-use technician interface to enable technician interaction and operation. For example, technicians can set the target detection algorithm adjustment parameters, obtain the current status of the power supply and motor, etc. by clicking a button.

[0110] Through a modular, layered architecture, the software is divided into several functionally independent layers, each responsible for specific tasks. This architecture facilitates the independence of each software component, improving development and maintenance efficiency. Furthermore, because each layer interacts with other layers through clearly defined interfaces, individual layers can be easily replaced or updated, enhancing the software's flexibility and scalability.

[0111] Design of basic functional modules of software platform

[0112] Through an overall design, the present invention divides the software platform design into two parts: a basic functional module and an underlying algorithm support module. The basic functional module serves as the basic framework for the software control platform operation and is responsible for implementing the various practical operational functions of the software platform for the unloading robot system.

[0113] Based on the layered architecture model in modularization, software reconstruction technology, agile development and other modern software engineering methods, the present invention designs and implements the basic functional modules of the software platform, including seven parts: system detection module, network communication module, motion control module, database management module, task management module, power management module and AR interaction module, and completes the unit test of each functional module.

[0114] System detection module

[0115] In order to meet the actual working requirements of the unloading robot system, the software platform needs to monitor and control the hardware and software of each part of the robot system in real time. Therefore, the system detection module is a basic component of the software system. The present invention has designed two basic functions for the system detection module: software initialization and hardware device detection. It can also complete detailed functions such as initialization parameter setting of each module of the system, reset and detection of motion control modules, network link detection, hardware detection, connection testing, etc. The specific functions that can be realized by this system detection module are as follows:

[0116] (1) Detect the status of all sensors (such as lidar, depth camera, temperature and humidity sensor, etc.) and perform necessary calibration to ensure the accuracy of sensor data.

[0117] (2) Test the robot's actuators (joint motors, chassis motors, and gripping devices) to ensure they are in the correct initial positions and can respond to signals correctly.

[0118] (3) Initialize all communication interfaces, such as Wi-Fi, industrial bus, etc., to ensure normal communication connections.

[0119] (4) Initialize task management, data management, and human-computer interaction interface, load system configuration, historical tasks and data, and graphical interface elements.

[0120] After the above system detection and initialization, the robot enters the working state.

[0121] Network communication module

[0122] The network communication module is responsible for data transmission and real-time control between the host computer and the embedded controller, as well as between the embedded controller and various actuators and sensors. This module is designed with flexible communication methods and efficient data transmission as its core, ensuring that the system can achieve stable and fast communication and control in various environments and conditions.

[0123] The network communication module supports both wired and wireless communication methods, allowing for flexible communication between the host computer and the embedded controller based on actual needs. Specifically, the module uses the TCP / IP protocol for data transmission, ensuring reliability and scalability across diverse network environments. The TCP / IP protocol not only offers high transmission efficiency but also enables error detection and data retransmission during communication, ensuring data integrity and accuracy.

[0124] Once data reaches the embedded controller, it is exchanged internally via Beckhoff's proprietary ADS communication protocol. The ADS protocol provides fast, real-time communication capabilities for the embedded controller, enabling efficient data transfer between multiple control tasks. The ADS protocol's high real-time performance and low latency ensure that the system can quickly respond to control commands from the host computer and promptly transmit feedback data from sensors and actuators to the host computer.

[0125] Communication between the embedded controller and the motors and actuators utilizes the EtherCAT Industrial Ethernet protocol. EtherCAT is a high-speed, real-time communication protocol widely used in industrial automation, specifically designed for high-precision control of motors and actuators on each axis. Through EtherCAT, the embedded controller can control the motion of each actuator in real time with extremely low latency while simultaneously acquiring feedback from various sensors.

[0126] The network communication module provides a complete data communication link from the host computer to the actuators through the collaborative operation of TCP / IP, ADS, and EtherCAT protocols. The module's design fully guarantees the security, reliability, and real-time performance of data transmission, enabling the system to operate stably in complex industrial environments and achieve efficient and precise unloading task control.

[0127] Motion control module

[0128] The motion control module is a core function of the entire unloading robot system, responsible for precisely controlling the motors and actuators on each axis of the robot and coordinating the completion of complex tasks. Working collaboratively with the host computer and embedded controller, this module ensures the robot can efficiently and accurately complete various motion control tasks in various operating scenarios.

[0129] The host computer is built using the QT technician interface application development framework, providing a rich functional interface for technicians to control and monitor the system. Through this interface, technicians can implement functions such as single-axis motion control, multi-axis linkage, trajectory planning, and teaching. Single-axis motion control allows technicians to independently control the motion of a single axis, while multi-axis linkage coordinates the movements of multiple axes to complete complex motion tasks. The trajectory planning function allows technicians to define complex paths through teaching, and the robot can perform precise operations along the trajectory set by the technician.

[0130] The embedded controller utilizes a Beckhoff embedded control system and runs a proprietary motion control algorithm. This embedded controller's motion control algorithm supports both forward and inverse kinematics and dynamics calculations, ensuring the robot maintains stability and precision during complex motion. The forward kinematics solution determines the end effector position for a given joint position, while the inverse kinematics solution calculates the motion of each joint based on the target end position. The forward and inverse dynamics algorithms account for system mass, inertia, and external forces, ensuring smooth control of each actuator and precise trajectory during motion.

[0131] This module not only controls the actuator's motion but also processes sensor data, providing real-time feedback on the current system status. The embedded controller uses its rapid data processing capabilities to analyze sensor information (such as position, velocity, and acceleration) and adjust the actuator's motion in real time. This closed-loop control mechanism ensures the robot can rapidly respond to changes in the external environment and maintain precise motion control in complex dynamic environments.

[0132] Database management module

[0133] Data Processing Module: Because the simulation system includes A / D data acquisition capabilities, this module is designed to facilitate subsequent data collection and user-friendly graphical display. It filters actual data from sampling and sensor noise. This module primarily includes five components: digital filter selection, filter cutoff wavelength setting, filter bandwidth ratio setting, sampling length selection, and evaluation parameter settings. This module provides packaging support for related services such as intelligent recognition, reducing the storage, utilization, and conversion of large data streams, including providing a discontinuous control behavior algorithm.

[0134] The data storage module is responsible for managing and storing various types of data to support the robot's operation, monitoring, and optimization during different tasks. The data storage module provides the following basic functions: storage parameter settings, data cleaning, task data storage, work video storage, sensor data storage, system log storage, and communication data storage. These functions provide data support for the efficient, stable, and safe operation of the unloading robot, and provide a wealth of historical data and analytical basis for system debugging, optimization, and maintenance. In addition to these basic functions, the data storage module should also provide data values ​​for partial compensation, frame insertion and corresponding frame restoration under certain conditions, and enhance software capabilities for image restoration, image enhancement, and image enhancement in special cases. The module provides technicians with interfaces for storage location, storage time, archiving method, storage time for certain data, and data selection, facilitating customer retrieval. A large amount of interactive data exists between the robot's modules, with different protocols and types. This module facilitates the storage, cleaning, and utilization of large amounts of discrete data, while achieving data-insensitive software performance.

[0135] The data statistics module can classify, summarize, and organize raw data, helping technicians extract useful information and key indicators from the data. This module can perform various statistical analysis operations, such as mean, variance, percentage, frequency distribution, etc., to help technicians understand the characteristics and trends of the data. Reports can be automatically generated according to preset conditions, and statistical results can be provided regularly for convenient viewing and analysis by technicians. The statistics module provides support for technicians' decision-making, helping them make more informed business decisions by displaying key data and indicators. Although it is mainly a statistical function, this module also often includes some basic data visualization functions, such as chart display, to help technicians more intuitively understand statistical results and discover data trends.

[0136] Task management module

[0137] The design and planning of the task management module is a core component of the entire unloading robot software platform, focusing on ensuring the robot can efficiently perform various tasks in the complex and ever-changing unloading environment. First, when planning this module, the present invention adopted a modular and parallel processing design concept for the system architecture. Considering that unloading tasks involve multiple subtasks, such as grasping, moving, and placing, each of which may be performed simultaneously at different time points, the task management module is designed to support the concurrent execution of multiple tasks. To ensure the rationality of task allocation, the task management module adopts a priority-based task scheduling strategy. During the planning phase, the priority requirements of various tasks were thoroughly analyzed. For example, urgent tasks (such as the rapid processing of high-value goods) require a higher priority than routine tasks. Furthermore, the module also reserves the function of dynamically adjusting task priorities. In the event of sudden changes in the external environment (such as the system detecting an obstacle or low battery), the task management module can promptly adjust the execution order of tasks to ensure smooth task execution and improve the fault tolerance of the automatic unloading robot in complex and ever-changing task environments.

[0138] In the planning of the task management module, the present invention places special emphasis on the task monitoring and feedback mechanism. Since the unloading process is a dynamic, real-time operation, the system needs to keep track of the execution status of each task at all times. Therefore, the task management module is designed with a real-time data feedback channel that can collect all key data during the task execution process to the central control system and issue warnings or prompts to technical personnel when necessary. The design of this mechanism enables the task management module to not only operate efficiently without supervision, but also provide technical personnel with necessary support and decision-making basis. In addition, the task management module also has the function of task rollback and reallocation. When a task cannot be completed smoothly due to an emergency, the system can automatically roll back the task to the initial state or reallocate it to other available resources to minimize the impact of task failure on the overall system. The design and implementation of this module also provide a development foundation and more feasibility for using multiple automatic unloading robots simultaneously through a control center to perform task scheduling and resource allocation in actual engineering applications.

[0139] Power Management Module

[0140] To ensure the robot's long-term continuous operation under high loads, the present invention incorporates a power management module specifically designed for the robot's energy system. Given the significant power consumption of robots when performing tasks, the power management module's design focuses on maximizing battery efficiency through intelligent power scheduling and allocation strategies. During the initial design phase of the module, the present invention analyzed the power consumption of various robot tasks and designed different power management strategies based on the task type. For example, for energy-intensive tasks (such as long-distance movement or heavy object handling), the power management module will rationally allocate power based on pre-task power levels to ensure a stable power supply during the task. The power management module's design also includes emergency response capabilities. The present invention specifically incorporates a low-battery warning mechanism and an emergency power switching function. When the system detects that the battery level falls below a preset safety threshold, the module issues an alert and adjusts the remaining power usage based on the task's importance. If the current task is high-priority, the system will extend task execution time by reducing power consumption for non-critical functions. Furthermore, a linkage design exists between the power management module and the task management module. When the system detects insufficient power, the power management module can work together with the task management module to adjust the order of task execution or automatically suspend certain low-priority tasks to ensure that critical tasks can be completed smoothly.

[0141] Visual interaction module

[0142] The visual interaction module is implemented using the AR interaction module. The planning of the AR interaction module focuses more on improving the technician experience, especially in complex working environments, where technicians need to be able to intuitively understand the robot's working status and task progress. The design of this module fully combines the advantages of augmented reality technology. By projecting the data collected by the sensors into the technician's field of view in real time, the technician can intuitively view the robot's current task status, environmental information, and system feedback without relying on traditional graphical interfaces. When planning the AR interaction module, the first thing to focus on is the real-time and accuracy of the data, especially for scenarios that require a quick response, such as obstacle detection or target recognition. The AR interaction module can quickly present this information to technicians in a visual way, helping them make decisions quickly.

[0143] The module's primary function is to help technicians control measuring instruments directly via a mouse or two-axis joystick. Its simple interface allows technicians to follow instructions for performing tasks, along with instructions for operating buttons, making the software easier to use. Furthermore, the module supports both wired and non-wired control modes for human-computer interaction, as well as VR / AR under Unity. This provides technicians with the comprehensive perspective support they need for decision-making in unloading environments, intelligent recognition, and environmental awareness for action operations, providing a more user-friendly and convenient human-computer interaction experience for the devices they use.

[0144] The AR interaction module also pays special attention to the convenience of operation. During the design stage, the present invention introduced a multimodal interaction function, so that technicians can operate the robot not only through the traditional touch screen and keyboard operating system, but also through gesture control and voice commands. Gesture recognition technology allows operators to quickly issue instructions, adjust tasks or pause them when operating from a distance. At the same time, the voice control function provides control of the robot through simple voice commands when both hands are occupied. This multimodal interaction method greatly improves the flexibility and efficiency of operation, especially in emergency situations, operators can quickly intervene in task execution through voice or gestures. In addition, the planning of the AR interaction module also reserves room for future expansion. In the future, more augmented reality functions can be introduced according to the needs of different application scenarios to further enhance the intelligence and operability of the system.

[0145] Design of underlying algorithm support modules for the software platform

[0146] In the overall design of the present invention, the underlying algorithm support module becomes the algorithm core part of the software platform as another design part, and is also the main component of the robot vision system operation. It is responsible for helping the unloading robot body analyze external work information and realize target detection and grasping path planning of mixed material piles.

[0147] Based on the modular approach, the present invention designs and implements the underlying algorithm support module of the software platform, including three parts: image processing algorithm module, point cloud processing algorithm module and motion planning algorithm module, and completes the unit test of each functional module.

[0148] This invention uses a time-of-flight (TOF) depth camera onboard an automated unloading robot to capture real-time RGB image information and 3D point cloud information of the working environment. During the development of this project, an optimized algorithm was designed and developed based on the OpenCV planar vision algorithm library and the PCL spatial point cloud algorithm library. This algorithm uses RGB images and point cloud data as input to detect the surface information of mixed cartons and identify and locate cartons. The image processing and point cloud processing algorithm modules in this invention serve as the foundation and support for this algorithm. These algorithms represent a variety of basic functional algorithms for robotic vision systems in this scenario, selected through practical experiments.

[0149] Image processing algorithm module

[0150] The image processing algorithm module's architecture adopts a modular and scalable design concept to meet the diverse visual tasks of unloading robots in complex operating environments. This module's design not only incorporates optimizations based on traditional computer vision techniques, but also emphasizes code reusability, flexibility, and ease of scalability. In particular, it incorporates in-depth customization and improvements to the optimization and repackaging of OpenCV-related algorithms, ensuring accurate target recognition and positioning during unloading tasks.

[0151] First, the module provides the entire algorithm framework with basic data structures and functional functions for two-dimensional RGB images. These basic data structures, including key image processing elements such as pixels, point coordinates, and rectangular regions, provide the smallest operational unit for all image processing algorithms. These data structures not only represent various types of information in the image but also support precise pixel-level operations on the image, facilitating subsequent algorithm execution. The functional functions in the module revolve around common image processing operations, covering basic image operations (such as reading, displaying, and saving) as well as various image processing algorithm encapsulations, providing fundamental image processing capabilities for the robot vision system.

[0152] From a code reuse perspective, the module's design divides algorithms into two main categories to enhance system scalability and development efficiency. The first category involves image transformation operations, primarily for geometric manipulation and basic image processing, including rotation, translation, and cropping and enlarging of ROIs. The second category comprises image morphological processing algorithms, including filtering, edge detection, dilation, and erosion. These morphological processing algorithms are crucial tools for object detection and image enhancement. Within this module, some algorithms have undergone detailed optimization to better suit the operating environment of the unloading robot. For example, the Canny edge detection algorithm based on OpenCV has been deeply optimized. Because Canny edge detection is susceptible to noise and lighting fluctuations when processing fast-moving objects, the stability and accuracy of edge detection have been improved through thresholding and a multi-level edge extraction method. Key algorithm parameters (such as high and low thresholds) are also interfaced to the main human-machine interface, allowing technicians to adjust edge detection parameters in real time during robot operation to adapt to changes in the operating environment, ensuring accurate object detection.

[0153] In order to enhance the adaptability of the module, the module focuses on optimizing the ROI selection function, allowing technicians or algorithms to crop, enlarge or mark specific areas of the image according to task requirements. The definition of ROI can be manually selected by technicians through a graphical interface, or it can be automatically generated by an algorithm. This flexible area selection function plays an important role in unloading tasks, especially when it is necessary to accurately locate specific parts of the goods (such as labels, barcodes, etc.). The ROI function can greatly improve recognition efficiency. In addition, the module provides predefined ROI templates and a dynamic adjustment interface to support rapid switching between different unloading scenarios, thereby improving the system's work efficiency and accuracy.

[0154] Point cloud processing algorithm module

[0155] Similar to the image processing algorithm module, the architecture of this point cloud processing algorithm module is centered on the geometric structure of the point cloud, supporting the efficient processing of large-scale point cloud data, helping the robot process spatial information, and combining RGB image information to perform mixed pile surface analysis and object identification and positioning.

[0156] This module optimizes the basic data structure of point clouds, enabling accurate storage and rapid access to three-dimensional information such as point coordinates, normals, and colors. Point cloud data processing involves extensive computation, so the design prioritizes computational efficiency, utilizing spatial indexing structures (such as KD trees and octrees) to accelerate point cloud retrieval and processing.

[0157] The main functions include filtering, segmentation, alignment, and geometric transformation of point cloud data. The filtering operation is used to remove invalid data and noise points, improve the quality of the point cloud, and ensure the stability of subsequent processing. Commonly used filtering algorithms such as voxel filtering and statistical filtering can significantly reduce the amount of data while optimizing the density of the point cloud, thereby increasing processing speed. The segmentation operation is used to extract specific target objects, such as goods or obstacles, from the point cloud. The module integrates a variety of segmentation algorithms, including RANSAC plane segmentation and Euclidean clustering, which can effectively identify and separate different objects in the scene and maintain high accuracy even in noisy environments.

[0158] To adapt to dynamic operating environments, the module also supports the processing of real-time point cloud data. Whether it is point cloud data generated in real time or pre-collected data sets, the module can respond quickly and make dynamic adjustments. This means that when the operating scene changes (such as the position of the goods shifts, or new obstacles appear), the system can recalculate the path and plan tasks in real time to ensure that the robot works smoothly. In addition, the module also has the ability to fuse multi-sensor data, fusing data from multiple sensors such as lidar, depth cameras, RGB cameras, etc. to generate more accurate and comprehensive three-dimensional environmental perception information.

[0159] The point cloud processing algorithm module is not only optimized for performance but also boasts high flexibility and scalability. To meet the needs of diverse operational scenarios, the module provides a rich set of interfaces and parameter adjustment capabilities, allowing technicians to dynamically adjust various point cloud processing parameters, such as filter density and segmentation thresholds, based on actual conditions. These adjustments can be made through a human-computer interaction interface, simplifying debugging for technicians and improving the system's adaptability. Furthermore, the module's design allows for a high degree of expansion, enabling the rapid integration of new point cloud processing algorithms or their customization and optimization, ensuring their continued iteration as technology evolves.

[0160] Motion planning algorithm module

[0161] The core task of the motion planning algorithm module is to plan a reasonable and safe motion path for the robot, once the spatial position of the cargo is known, to ensure efficient grasping operations. The design uses the Dijkstra algorithm as the basis for path planning, combined with an inverse kinematics algorithm to calculate the robot's trajectory. This combination of algorithms not only ensures optimal path planning but also ensures the robot's stability and accuracy in complex environments.

[0162] When a cargo unloading robot performs its tasks, the Dijkstra algorithm calculates the weights of each possible path in the environment to find the optimal movement path. This algorithm is particularly suitable for static or semi-dynamic environments, effectively avoiding obstacles and providing the robot with a path that saves the most time and energy. In cargo handling scenarios, the Dijkstra algorithm considers the shortest distance between the robot and the target object while ensuring that the path avoids possible obstacles, ensuring that the robot remains safe and efficient during movement, thereby improving its work efficiency.

[0163] Secondly, the inverse kinematics (IK) algorithm is used to solve the problem of precise movement of the robotic arm in complex environments. The grasping task requires not only the robot to find the optimal path, but also the robotic arm to be able to flexibly reach the specific location of the goods. The inverse kinematics algorithm calculates the reasonable angle and motion trajectory of each joint based on the end position requirements of the robotic arm (that is, the location of the goods). Through the IK algorithm, the robot can approach the goods in a smooth and precise manner without violating its mechanical limitations, avoiding unstable movements of the robotic arm during the grasping process, such as overextension or sudden stops.

[0164] In actual operation, the motion planning module will first determine the optimal path for the robot to reach the target object from its current position through the Dijkstra algorithm, and then calculate the motion trajectory of each joint of the robotic arm based on the inverse kinematics algorithm to ensure that the robotic arm can smoothly grasp the goods. During the grasping process, the module will dynamically adjust the path and grasping force according to sensor feedback to adapt to slight changes in the position of the goods or changes in environmental conditions, ensuring the accuracy and safety of the operation. To enhance the flexibility of the module, the motion planning module also supports real-time adjustment of path planning and motion control. When the environment changes (such as the appearance of new obstacles or the displacement of the goods), the motion trajectory of the robotic arm will be replanned to ensure that the system can always operate efficiently under the premise of safety and stability.

[0165] Software platform integration testing

[0166] Using Unity 2022LTS, we conducted integration testing of the software platform's various functional modules, enabling data transfer and communication between them. We also integrated the automated unloading robot and the software platform to conduct comprehensive unloading testing in real-world industrial scenarios and environmental conditions to verify the platform's reliability and stability in real-world applications.

[0167] The software integrates the control of various hardware components of the unloading robot, real-time status acquisition, the basic functions of multiple vision algorithms, and an intelligent human-machine interaction interface. Through the software platform, the real-time status of the automatic unloading robot's hardware and algorithm processing results can be read, and algorithm parameters and the communication protocols or interfaces used by various hardware components can be modified.

[0168] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in conjunction with other described embodiments.

Claims

1. Unloading robot system control platform, characterized by: Image acquisition and target recognition unit (1), robot motion planning unit (2), robot motion control unit (3), data processing and storage unit (4), communication module (5) and visual interaction module (6); The image acquisition and target recognition unit (1) is used to acquire image information and point cloud information of the target, identify the target using the acquired information, and simultaneously detect the target position and shape; and transmit the target position and shape through the communication module (5); The robot motion planning unit (2) is used to detect the robot system in real time, determine whether the robot system has a fault, and if a fault exists, send the fault information through the communication module (5); receive the position and shape of the target and the task priority sequence table through the communication module (5); plan the robot's motion posture at the next moment according to the target position and shape, the task priority sequence table and the fault information, and send the robot's motion posture at the next moment through the communication module (5); The visual interaction module (6) provides a port for setting and adjusting the priority of the robot task; the task priority setting and adjustment information is sent out through the communication module (5); The data processing and storage unit (4) receives the task setting and adjustment information through the communication module (5), generates a task priority sequence table in real time according to the system fault information and the task setting and adjustment information, stores the task priority sequence table, and sends it out through the communication module (5); The robot motion control unit (3) receives the robot's motion posture information at the next moment through the communication module (5) and controls the robot hardware to move; The visual interaction module (6) receives the system's priority sequence list, the robot's motion posture at the next moment, the target position and the target shape through the communication module (5) and displays them.

2. The unloading robot system control platform according to claim 1, characterized in that: The image acquisition and target recognition unit (1) includes a depth camera (11), a pre-processing module (12) and a target detection and recognition module (13); The depth camera (11) is used to collect RGB image information and point cloud information of the target and send the collected information to the pre-processing module (12); The laser radar (12) is used to collect the distance between the target and the robot body, and send the collected information to the pre-processing module (12); The preprocessing module (12) preprocesses the received information and sends the preprocessed information to the target detection and recognition module (13); The target detection and recognition module (13) uses the received information to recognize the shape and position of the target, and sends the shape and position of the target through the communication module (5).

3. The unloading robot system control platform according to claim 2, characterized in that: The target detection and recognition module (13) uses the received information to identify the shape and position of the target as follows: Step 1: pre-processing the point cloud information collected by the depth camera (11) to obtain pre-processed point cloud data; Step 2: Use the parameter-limited RANSAC method to extract the plane in the depth direction of the pre-processed point cloud data to obtain the point cloud plane closest to the camera, and then pre-process the point cloud plane; obtain the pre-processed point cloud plane P first_inlier ; Step 3: Based on the camera intrinsic parameter matrix K, the point cloud plane P first_inlier Mapping to the two-dimensional pixel coordinates of the RGB image collected by the depth camera (11) to obtain a point set S0; based on the boundary range of the point set S0, a rectangular region ROI is established, the ROI range is cropped in the RGB image, and the Canny edge detection algorithm is used to extract the boundary point set S1 of the cropped RGB image, and the point set S1 is inversely mapped back to the plane point cloud P first_inlier After filtering out the boundary of the point cloud plane mapped back to the point cloud, we can obtain the point cloud P containing the front surface information of multiple independent cartons. cam ; Step 4: Use a clustering segmentation algorithm based on surface growth to cluster the point cloud P containing the front surface information of multiple independent cartons. cam Split into multiple independent point cloud clusters; Step 5: Create a minimum cube outside each independent point cloud cluster that can contain all points in the point cloud cluster; use the spatial coordinates and side length of the minimum cube as the spatial position coordinates and outline dimensions of the target.

4. The unloading robot system control platform according to claim 3, characterized in that: In step 3, the boundary is inversely mapped to the plane point cloud P first_inlier After filtering out the boundary point cloud, we obtain the point cloud P containing the front surface information of multiple independent cartons. cam The process is: Set the camera center to the point cloud plane P first_inlier The distance is used as the plane depth Distance first_inlier , combined with the camera intrinsic parameter matrix, the two-dimensional boundary point set S1 of the RGB image is inversely mapped back to the camera coordinate system of the three-dimensional point cloud to obtain the three-dimensional point set P boundary , with a three-dimensional point set P boundary Each point in is taken as the origin and the radius r is established b The spherical space of the preprocessed point cloud plane P first_inlier The points whose coordinates are in the spherical space are marked as boundary points. first_inlier Eliminate all boundary points and obtain the point cloud P containing the front surface information of multiple independent cartons cam .

5. The unloading robot system control platform according to claim 4, characterized in that: In step 4, a clustering segmentation algorithm based on surface growth is used to cluster the point cloud P containing the front surface information of multiple independent cartons. cam The method of segmenting into multiple independent point cloud clusters is: Step S41: From point cloud P cam Select any point P i , search for point P i The neighborhood centered on is calculated by calculating the ratio of the minimum eigenvalue of the neighborhood covariance matrix to the sum of the eigenvalues, and the ratio is used as the curvature κ i ; At the same time, the covariance matrix of the neighborhood is used to calculate the point P i Normal vector n at the center i , and then calculate the point P i Any point p in the neighborhood j The covariance matrix of the neighborhood centered on point p is then calculated to obtain j Normal vector n at the center j ; Calculate the normal vector n i and normal vector n j The angle between Step S42: Set parameter smoothness threshold θ th and curvature threshold κ th ; When θ is satisfied ij <θ th ,κ i <κ th When P i Neighborhood point p j Point P i Class, traverse P i All points in the neighborhood, get the point p i Starting point cloud cluster; Step S43: From point cloud P cam Select the unclassified point P k , execute step S41 and step S42 until the point cloud P cam All points are classified, and the point cloud P containing the front surface information of multiple independent cartons is completed. cam Split into multiple independent point cloud clusters.

6. The unloading robot system control platform according to claim 1 or 2, characterized in that: The robot motion planning unit (2) includes a robot industrial control computer (21), a motion planning module (22) and a system detection module (23); The system detection module (23) is used to detect the robot system and determine whether there is a fault. If there is a fault, the detected fault information is sent out through the communication module (5); At the same time, the fault information is transmitted to the robot industrial control computer (21); The robot industrial control computer (21) receives the fault information and transmits the system's target task and fault information to the motion planning module (22); The motion planning module (22) plans the motion posture of the robot at the next moment using the system's target task, fault information, target position and shape, and sends the motion posture planning information of the robot at the next moment through the communication module (5).

7. The unloading robot system control platform according to claim 6, characterized in that: The method for the motion planning module (22) to plan the motion posture of the robot at the next moment is: Using the real-time information, the Dijkstra algorithm is used to determine the optimal path for the robot to reach the target from its current position. Then, based on the inverse kinematics algorithm, the motion trajectory of each joint of the robot's mechanical arm is calculated to obtain the robot's motion posture at the next moment.

8. The unloading robot system control platform according to claim 7, characterized in that: The robot motion control unit (3) includes a motion control module (31) and a robot motion controller (32); The motion control module (31) receives the motion posture planning information of the robot at the next moment through the communication module (5), obtains the posture control information of the robot at the next moment, and sends the obtained information to the robot motion controller (32); The robot motion controller (32) uses the received posture control information to send a motion control signal to the robot control system, receives the motion state of the robot hardware, and sends the motion state information of the robot hardware through the communication module (5).

9. The unloading robot system control platform according to claim 8, characterized in that: The data processing and storage unit (4) includes a data statistics and analysis module (41), a task management and statistics module (42), a database management module (43) and a database (44); The data statistics and analysis module (41) is used to receive fault information, the next moment robot posture control information, the shape and position of the target, and the motion state information of the robot hardware through the communication module (5), and send them to the database management module (43) according to the category. The task management and statistics module (42) generates a priority sequence table using a system supporting concurrent execution of multiple tasks and a priority-based task scheduling strategy, sends the priority sequence table to the database management module, and sends it out through the communication module; The database management module (43) is used to send the priority sequence list and robot system fault information, the posture control information of the next moment receiving robot, the shape and position of the target, and the motion state information of the robot hardware to the database (44) according to the category; The database (44) is used to store the received data.

10. The unloading robot system control platform according to claim 1, characterized in that: It also includes a power management module, which is used to detect the remaining power of the power supply. When the remaining power of the power supply is lower than a preset safety threshold, the power management module sends a warning signal through the communication module, and adjusts the remaining battery power to power the hardware corresponding to the high-priority task according to the importance of the task.

Images