Vision-based multi-sensor fusion ball-picking robot device and control method

Abstract

This invention relates to a ball-retrieving robot, and more specifically to an intelligent ball-retrieving robot device and control method based on vision-based multi-sensor fusion. The invention includes: a robot body and a control system. The robot body is equipped with a rotating axis and a rotary motor driving the rotating axis. The robot body has a through hole for the rotating axis to pass through, and a sensor system is fixed to one end of the rotating axis passing through the through hole. The sensor system is connected to the control system. The robot body also has multiple storage compartments. A roller conveyor belt is used to transport balls to the storage compartments. The robot body is equipped with a fan-shaped collector for collecting balls located on the ground. This invention greatly improves the robot's obstacle avoidance and path planning capabilities.

Description

Technical Field

[0001] This invention relates to a ball-collecting robot, and more specifically to an intelligent ball-collecting robot device and control method based on vision-based multi-sensor fusion. Background Technology

[0002] Ball-retrieving robots are primarily used in outdoor or indoor sports fields to replace manual labor in retrieving balls, significantly saving manpower and resources and improving efficiency during sports activities. In everyday ball sports, especially smaller ball sports such as table tennis, tennis, and badminton, retrieving balls is a tedious task that disrupts the flow of the game, greatly reducing the enjoyment of the experience. Therefore, a tool is needed to replace users in this monotonous task. This ball-retrieving robot design not only aligns with the times but also meets people's needs for a high-quality sports experience, enhancing their quality of life. The application of ball-retrieving robots replaces manual labor in retrieving balls, saving human resources.

[0003] Existing technologies offer some ball-retrieving robots, most of which are used outdoors and for picking up single types of balls. They are generally not suitable for large-scale multi-sport venues. Furthermore, traditional methods can lead to misjudgments during the picking process, often resulting in debris being picked up along with the ball, leading to low usability and insufficient picking accuracy. Summary of the Invention

[0004] In view of this, the present invention provides a ball-picking robot device and control method based on vision-based multi-sensor fusion.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A vision-based multi-sensor fusion intelligent ball-picking robot device includes: a robot body and a control system. The robot body is provided with a rotating shaft and a rotating motor that drives the rotating shaft. The robot body is provided with a through hole for the rotating shaft to pass through. A sensor system is fixed at one end of the rotating shaft that passes through the through hole. The sensor system is connected to the control system.

[0007] The robot body is also equipped with multiple storage compartments; a roller conveyor belt is used to transport the spheres to the storage compartments, and the robot body is equipped with a fan-shaped collector for collecting the spheres located on the ground.

[0008] Optionally, the sensor system includes an infrared camera, a high-definition camera, a depth camera, a monocular camera, and an ultrasonic sensor.

[0009] Optionally, the robot body is also equipped with drive wheels and omnidirectional wheels; the drive wheels are connected to the control system.

[0010] Optionally, multiple ultrasonic sensors are provided to detect feedback distance information and plan a path.

[0011] Optionally, the robot body is also equipped with a storage compartment lifting module, which is connected to the control system.

[0012] A vision-based multi-sensor fusion-based intelligent ball-retrieving robot control method, utilizing any one of the vision-based multi-sensor fusion-based intelligent ball-retrieving robot devices described above, includes the following steps:

[0013] Obtain regional information;

[0014] Determine if there is a sphere in the region. If yes, proceed to step S3; otherwise, select the next region.

[0015] Step S3: Determine if the sphere is a preset collection type. If yes, then the area is the object to be picked up, and proceed to step S4.

[0016] Step S4: Information transmission and control system. The control system controls the area where the robot sphere is located, and the robot moves to the target area. Step S5 is then executed.

[0017] Collect spheres;

[0018] The robot determines whether the collection bin exceeds the preset conditions. If it does, the robot returns to the starting point and puts the sphere into the recycling point; otherwise, it continues patrolling.

[0019] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a vision-based multi-sensor fusion ball-picking robot device and control method, which has the following beneficial effects:

[0020] 1) An obstacle avoidance scheme combining vision and ultrasound is adopted. Ultrasonic sensors placed in front of and behind the robot detect obstacle information on the robot's path in real time. Combined with point cloud data of visual images, the robot can more clearly judge the passability of the current planned path, which greatly improves the robot's obstacle avoidance and path planning capabilities. Currently, ball-picking robots generally work by following a preset route.

[0021] 2) The rotating axis can rotate 360 ​​degrees, providing a wider search range for video sequence capture, enabling data monitoring of ball falling on the ground within the current field of view. Furthermore, through data analysis of the control system, it can achieve monitoring of a large area and upload data to a host computer for data sharing.

[0022] 3) Multifunctional use of high-definition cameras: When searching for and identifying balls, the high-definition camera on the robot's head can also be used for the security system in the stadium. The robot's patrol can patrol the entire venue and reach the blind spots of the stadium. It can not only pick up balls in the corners, but also monitor the blind spots of the stadium's cameras at the same time.

[0023] 4) By applying and optimizing target detection algorithms, the misjudgment rate of the robot in collecting balls can be reduced, and the accuracy of robot pose estimation can be improved.

[0024] 5) The design of the information transceiver device can continuously send the robot's real-time location and status information to the host computer.

[0025] 6) The online mechanism allows administrators to observe the robot's status in real time on the host computer, thus responding to emergencies. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0027] Figure 1 This is a system structure block diagram of the robot according to an embodiment of the present invention;

[0028] Figure 2 This is an internal side view of the robot according to an embodiment of the present invention;

[0029] Figure 3 This is a top view of the robot body according to an embodiment of the present invention;

[0030] The components are as follows: 1 is the robot body, 2 is the rotation axis, 3 is the left rotary motor, 4 is the right rotary motor, 5 is the pressure sensor, 6 is the ultrasonic sensor, 7 is the omnidirectional wheel, 8 is the ultrasonic sensor, 9 is the buzzer, 10 is the left drive wheel, 11 is the right drive wheel, 12 is the support rod, 13 is the fan blade, 14 is the positioning module, 15 is the information receiving unit, 16 is the storage compartment, 17 is the conveyor belt, 18 is the IMU, 19 is the main control system, 20 is the sensor system, 21 is the infrared camera, 22 is the high-definition camera, 23 is the monocular camera, 24 is the depth camera, and 25 is the battery compartment. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] This invention discloses a vision-based multi-sensor fusion intelligent ball-collecting robot, such as... Figures 1 to 3 As shown, the robot includes a sensor system 20 arranged vertically and a robot body 1. The robot body has a rotating shaft 2 and a rotating motor 3 that drives the rotating shaft 2. The robot body has three storage compartments 16 for storing three types of balls, and a roller conveyor belt 17 for conveying the balls. The front of the robot has a three-bladed fan-shaped collector 13 for collecting balls located on the ground. The center of the robot body has a through hole for the rotating shaft 2 to pass through. A sensor system is fixed at one end of the rotating shaft 2 that passes through the through hole. The sensor system is equipped with an infrared camera 21, a high-definition camera 22 and a pressure sensor 5.

[0033] The robot body is equipped with a depth camera 26, a monocular camera 25 and an ultrasonic sensor 4 on the front, and two drive wheels and one omnidirectional wheel 7 on the bottom.

[0034] The robot's main body is equipped with a control system 19. An infrared camera 21, a high-definition camera 22, a depth camera 24, a monocular camera 23, a buzzer 9, and a drive wheel are all connected to the main control system 19. The main control system 19 includes an industrial computer and a drive circuit board, which are connected via a serial port for issuing commands to control the intelligent ball-picking robot's activities. The main control system 19 is configured to execute the following steps:

[0035] The system acquires image information from the high-definition camera 22, performs location information detection, and confirms whether there are any scattered balls in the venue.

[0036] The system acquires data collected by the monocular camera 21, obtains region information based on the region recognition results, compares it with preset conditions, determines whether there is a sphere in the region based on the comparison results, and identifies the region containing the sphere as the target region.

[0037] Data from depth camera 24 and monocular camera 23 is acquired to locate the target area, and the drive wheel is controlled to reach the target area.

[0038] The robot also includes ultrasonic sensors, which are mounted on the robot's main body and connected to the main control system 19. Two ultrasonic sensors, 6 and 8, are used to detect and provide feedback distance information for path planning. The two drive wheels are directly driven by stepper hub motors, enabling the ball-picking robot to move forward and backward in any direction. The motors have built-in encoders, and through feedback and speed control, differential speed is used to control the robot's direction of movement. Two ultrasonic sensors, 6, are located at the front and rear of the robot to collect information about obstacles along the movement path and to avoid obstacles when necessary.

[0039] The robot also includes a warehouse lifting module, which is located inside the robot body and connected to the main control system 19.

[0040] The robot also includes an accelerometer and a gyroscope, which together form the IMU18 and are located inside the robot's main body to calculate attitude angles and acceleration in real time.

[0041] The robot also includes a collector 11 and a collector fixing structure 12, which are connected to the main control system 19. The collector fixing structure 12 consists of gears and bearings, which fix the collector blades to the robot body 1. The collector 11 is a device consisting of three blades and is fixed to the collector support structure 12. The blades of the collector 11 are connected to the gears by rods, and the collector 11 is rotated to collect the sphere by a drive motor.

[0042] The robot body is also equipped with a battery compartment 25, located at the bottom of the robot body with an opening at one end. When the robot's battery is low, it will send a message to the host computer via a signal transmission device to replace the battery. The replacement process is quick. After replacement, the robot can immediately leave the station to perform inspection tasks, thus realizing all-weather inspection capabilities.

[0043] Feature points include the sphere's color, shape, and size. Region information is obtained based on the region recognition results, compared with preset conditions, and the comparison result determines whether the region contains a sphere. Specifically, this includes:

[0044] Step S1: Obtain region information based on the region identification results;

[0045] Step S2: Determine if there is a sphere in the area. If yes, proceed to step S3; otherwise, select the next area.

[0046] Step S3: The information collected by the camera is transmitted to the main control system. At the same time, the main control system controls the rotating axis to make the camera rotate 360 ​​degrees to perform image acquisition and intelligent video analysis, thereby determining whether the sphere is a preset collection type. If the size, shape and color of the sphere are within the preset range, it is determined that the area contains a sphere to be picked up, and step S4 is executed.

[0047] Step S4: The main control module transmits information and controls the area where the robot sphere is located. The drive module and positioning system work together to move the robot towards the target area, thus executing step S5.

[0048] The pressure sensor determines whether the collection bin exceeds the preset conditions. If so, if it is confirmed that the collection bin is full, then step S6 is executed.

[0049] Step S5: The robot determines that it has reached the preset position through camera information. The main control system controls the fan blades and columnar conveyor of the ball collection device to start. The fan blades roll the ball into the robot. The ball will fall into the lower warehouse through the gap of the shaft. The shuttlecock will be transported to the upper warehouse by the shaft-shaped conveyor belt.

[0050] Step S6: The robot returns to the starting point. Upon arrival, the door opens and the support rod inside the warehouse rises, allowing the sphere to be placed into the recycling point. Otherwise, it continues patrolling.

[0051] In some embodiments, machine learning and deep learning can also be used. After the robot completes the construction of the regional map, the map data will be shared to a remote computer to obtain detailed map information and the robot's location information in real time. With the deployment of multiple robots, detailed information data of multiple locations can be detected at the same time, so as to better realize cross-domain spherical information collection and control.

[0052] In some embodiments, ball information is captured in real-time by a high-definition camera 22, and ball recognition is achieved using the YOLO (You Only Look Once) object detection algorithm combined with a ball detection model. The detection results are then displayed on a host computer screen. Combined with synchronized camera data, the host computer screen displays the data of the area captured by the robot, as well as the robot's real-time status.

[0053] In some embodiments, this intelligent ball-collecting robot can be customized for different application environments and equipped with various sensor modules according to different needs. It has a wide range of applications, is easy to operate, and users can add modules through the open SDK development package to expand the development of the intelligent robot.

[0054] Autonomous navigation and obstacle avoidance utilize real-time obstacle data acquired by the robot's two ultrasonic sensors (6 and 8) at the front and rear, respectively. This data is fused with environmental point cloud data from depth camera 24 and monocular camera 23, along with the robot's own motion data obtained from IMU 18. An incremental environmental map is created using an extended Kalman filter-based SLAM algorithm, which is then used for self-localization and navigation. An improved A* algorithm based on global planning and local optimization enables incremental mapping and path optimization, facilitating autonomous obstacle avoidance and path planning, allowing for agile movement in complex environments.

[0055] When the ball-picking robot is deployed for automated inspection, it first automatically operates within the current work area, capturing map and obstacle information of the environment using a depth camera 24, a monocular camera 23, and ultrasonic sensors 6 and 8. The main control system 19 generates a detailed grid map, which is then sent to a remote computer via an information receiving unit 15 for remote information sharing. Simultaneously, the robot's location and working status can be remotely viewed. After starting automated inspection, the robot moves autonomously within the designated work area. Simultaneously, the rotary motor 3 controls the rotation axis 2 to rotate, providing a wider field of view for the infrared thermal imaging camera 21 above it. The thermal imaging camera transmits the monitored image data to the control system, which performs numerical analysis to determine if there are any human obstacles. The high-definition camera 22 transmits captured video images to the control system, which analyzes the image data and uses the YOLO target detection algorithm combined with a sphere detection model to determine if the object being tested is a target sphere. The identification result is displayed on the screen 8. The remote computer can also simultaneously receive the screen display via the information receiving device 3. Remote computers can remotely control robots to pick up balls across space. When the monitoring range is larger, multiple ball-picking robots can be configured to divide the area, allowing for more detailed management of large-scale ball-picking operations. A grid map can be used to accurately monitor the robots in each location.

[0056] When the ball-collecting robot detects the presence of a target ball, the area will be marked as the target area. The robot will then proceed to that area. The area features acquired by the depth camera 24 and the monocular camera 23 will be transmitted to the remote computer via the information transmission unit 15. At this time, a marker signal for the target area will appear on the grid map of the remote computer.

[0057] When the ball-collecting robot encounters low battery during operation, it uploads its status and battery swap request information to a remote computer via an information transmission device. The remote computer will then dispatch a nearby robot to take over the work. Simultaneously, the low-battery robot automatically heads to the battery swapping station, where management personnel will perform the battery swapping operation. After entering the battery swapping work area, the robot enters standby mode, and the door of its battery compartment 25 will automatically open, waiting for the battery swapping to be completed. After completing the entire battery swapping operation, the robot leaves the station to continue its work.

[0058] In some applications, the ball-retrieving robot operates 24 / 7. To address this, supplementary lighting can be installed at night in conjunction with the robot's high-definition camera 22, fulfilling security needs in designated areas. Combined with existing surveillance cameras in the area, it achieves comprehensive video surveillance without blind spots. Battery replacement reminders ensure timely charging. Furthermore, the onboard battery can continue to provide power to the robot in emergency power outages.

[0059] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0060] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A vision-based multi-sensor fusion intelligent ball-picking robot device, characterized in that, It includes: a robot body and a control system. The robot body is provided with a rotating shaft and a rotary motor that drives the rotating shaft. The robot body is provided with a through hole for the rotating shaft to pass through. A sensor system is fixed at one end of the rotating shaft that passes through the through hole. The sensor system is connected to the control system. The robot body is also equipped with multiple storage compartments; a roller conveyor belt is used to transport the spheres to the storage compartments, and the robot body is equipped with a fan-shaped collector for collecting the spheres located on the ground; The sensor system includes an infrared camera, a high-definition camera, a depth camera, a monocular camera, and an ultrasonic sensor. The control system (19) is configured to perform the following steps: Obtain image information collected by high-definition camera (22), perform site information detection, and confirm whether there are scattered balls in the site; The data collected by the infrared camera (21) is obtained, and the area information is obtained according to the area recognition result. It is compared with the preset conditions, and the area is determined to have a sphere according to the comparison result. The area containing the sphere is identified as the target area. Acquire data from the depth camera (24) and the monocular camera (23), locate the target area, and control the drive wheel to reach the target area; The robot uses camera information to determine that it has reached the preset position. The control system activates the fan blades and columnar conveyor of the ball collection device. The fan blades roll the ball into the robot, and the ball will fall into the lower warehouse through the gap of the shaft. The shuttlecock will be transported to the upper warehouse by the shaft-shaped conveyor belt. Autonomous navigation and obstacle avoidance are achieved by using two ultrasonic sensors (6, 8) at the front and rear of the robot to acquire obstacle data in real time, integrating environmental point cloud data from the depth camera (24) and monocular camera (23), and combining the robot's own motion data obtained from the IMU (18); an incremental environmental map is created based on the SLAM algorithm of extended Kalman filter, and the robot uses this map to achieve self-localization and navigation; the robot's incremental mapping and path optimization are achieved based on the improved A* algorithm of global planning and local optimization, completing autonomous obstacle avoidance and path planning, and enabling free movement in complex environments; the high-definition camera (22) transmits the captured video images to the control system, the control system analyzes and acquires the image data, and uses the YOLO target detection algorithm combined with the sphere detection model to determine whether the object to be tested is a target sphere.

2. The intelligent ball-picking robot device based on vision multi-sensor fusion according to claim 1, characterized in that, The robot body is also equipped with drive wheels and omnidirectional wheels; the drive wheels are connected to the control system.

3. The intelligent ball-picking robot device based on vision multi-sensor fusion according to claim 2, characterized in that, Multiple ultrasonic sensors are configured to detect feedback distance information and plan the path.

4. The intelligent ball-picking robot device based on vision multi-sensor fusion according to claim 1, characterized in that, The robot body is also equipped with a storage compartment lifting module, which is connected to the control system.

5. A vision-based multi-sensor fusion control method for an intelligent ball-picking robot, characterized in that, The intelligent ball-picking robot device based on vision multi-sensor fusion as described in any one of claims 1-4 includes the following steps: Step S1: Obtain area information; Step 2: Determine if there is a sphere in the region. If yes, proceed to step S3; otherwise, select the next region. Step S3: Determine if the sphere is a preset collection type. If yes, then the area is the object to be picked up, and proceed to step S4. Step S4: Information transmission and control system. The control system controls the area where the robot sphere is located, and the robot moves to the target area. Step S5 is then executed. Step S5: Collect the spheres; Step S6: Determine whether the collection bin exceeds the preset conditions. If yes, the robot returns to the starting point and puts the sphere into the recycling point; otherwise, it continues patrolling.

Images