An image recognition-based data processing method and a robot control method

Abstract

This invention discloses a data processing method and a robot control method based on image recognition, applied to a robot control system, wherein the robot control system includes a data processing terminal and an operation control terminal. This invention divides the computational processing of large amounts of image data into two steps: a data processing step and a robot control step. These two steps are handled separately by two terminals. The data processing terminal executes the data processing method and sends the obtained first data to the operation control terminal, which then executes the robot control method based on the first data sent by the data processing terminal. This approach simultaneously reduces the computational burden on both terminals, improves the computational speed and accuracy of the robot control system, and thus reduces the possibility of recognition errors.

Description

Technical Field

[0001] This invention relates to the field of image data processing, and in particular to a data processing method and a robot control method based on image recognition. Background Technology

[0002] With the improvement of living standards, sports have become a trendy leisure activity, especially ball sports. As ball sports, such as tennis or table tennis, become more popular, small balls dropped during play accumulate over time. If these balls are not retrieved, they can cause ankle sprains, so it's necessary to pick them up regularly. However, retrieving balls is a tedious and time-consuming task, and currently, ball pickers or ball-retrieving devices are commonly used.

[0003] Existing ball-picking devices mostly use STM32 microcontrollers as the main controller, collect image data through a monocular camera, and pick up balls through a robotic arm. The automatic obstacle avoidance function of these ball-picking devices is mostly achieved through ultrasonic modules and human infrared sensors.

[0004] However, existing technologies using STM32 microcontrollers as the main controller can lead to excessive load and insufficient computing power due to the large amount of image data that needs to be recognized and processed, resulting in recognition errors. Summary of the Invention

[0005] This invention provides a data processing method and a robot control method based on image recognition, so as to improve the system's computing power and reduce the possibility of recognition errors.

[0006] To address the aforementioned technical problems, this invention provides a data processing method based on image recognition, applied to a robot control system; wherein the robot control system includes a data processing terminal and an operation control terminal;

[0007] The data processing method includes the following steps:

[0008] The data processing terminal responds to the image processing signal, receives the image data to be processed, performs preliminary image data processing on the image data to be processed to obtain first image data, and performs contour judgment on the first image data. Based on the judgment result, it determines that the object to be processed in the first image data is an object to be grabbed or an object to be avoided.

[0009] If the data processing terminal determines that the judgment result is an object to be grasped, then it performs image depth processing on the first image data to obtain the first positioning data of the object to be grasped, and sends the first positioning data to the operation control terminal so that the operation control terminal performs a grasping operation based on the first positioning data.

[0010] If the data processing terminal determines that the judgment result is an object to be avoided, it sends the first image data to the operation control terminal so that the operation control terminal performs an obstacle avoidance operation based on the first image data.

[0011] The data processing method provided by this invention divides the process of calculating a large amount of image data into two steps and assigns them to two different terminals for processing. The data processing part for image data is handled by the data processing terminal, which is mainly responsible for determining the type of object to be processed and performing different operations on the first image data according to the object type. The operation control terminal is mainly responsible for performing different control operations on the robot according to the different types of data sent by the data processing terminal.

[0012] This invention assigns the image data processing and operation execution parts to two different terminals. A complete robot control system is constructed through data interaction between the two terminals. At the same time, the computational burden of a complete system is distributed between the two terminals. Compared with the existing technology where a single-chip microcomputer is used as the main controller to handle all the computations of the entire control system, the processing method provided by this invention reduces the computational burden on both terminals, improves the computational speed and accuracy of the robot control system, and thus reduces the possibility of recognition errors in the system.

[0013] As a preferred example, the step of performing contour judgment on the first image data and determining the object to be processed in the image data to be processed as an object to be grasped or an object to be avoided based on the judgment result specifically involves:

[0014] The object contour is extracted from the first image data to obtain the first contour;

[0015] The first contour is compared and fitted with a preset model contour, and the object to be processed in the first image data corresponding to the first contour whose fitting result is greater than or equal to the first fitting threshold is determined as the object to be grasped.

[0016] The object to be processed in the first image data corresponding to the first contour whose fitting result is less than the first fitting threshold is identified as the obstacle to be avoided.

[0017] This invention determines the object type of the object to be processed by comparing the contour extracted from the first image data with the preset model contour. The judgment method is simple and direct, so the judgment speed is relatively fast, providing a basis for subsequent different operations based on the judgment result.

[0018] As a preferred example, the step of performing image depth processing on the first image data to obtain the first positioning data of the object to be grasped specifically includes:

[0019] The YOLO-V3 algorithm is used to identify the objects to be captured in the first image data, and the corresponding recognition results are obtained.

[0020] The YOLO-V3 algorithm is invoked to calculate the recognition result and the first image data to determine the first positioning data of the object to be captured;

[0021] At the same time, a preset neural network acceleration algorithm is invoked to accelerate the operation process of the YOLO-V3 algorithm.

[0022] The YOLO-V3 algorithm used in this invention for processing image data employs Feature Pyramid Networks (FPN), which significantly improves the performance of small object detection without substantially increasing the original computational load. Compared with existing technologies using YOLO-V1 or YOLO-V2 algorithms, this algorithm is more efficient in identifying small spherical objects, making it more suitable for recognizing small spherical objects.

[0023] Meanwhile, the computation process is accelerated by a preset neural network acceleration algorithm, which further improves the algorithm's computing power, thereby increasing the computational efficiency of the data processing end and reducing the time required for this deep processing step.

[0024] Accordingly, the present invention also provides a robot control method based on image recognition, applied to a robot control system; wherein the robot control system includes a data processing terminal and an operation control terminal;

[0025] The robot control method includes the following steps:

[0026] The operation control terminal receives first data sent by the data processing terminal and calls a preset recognition method to identify the first data to determine whether the data is first positioning data or first image data; wherein, the first data is determined by the data processing terminal based on the judgment result of the object to be processed in the first image data; the judgment result is determined by the data processing terminal by performing contour judgment on the first image data; the first image data is obtained by the data processing terminal after performing preliminary image data processing on the image data to be processed;

[0027] If the operation control terminal determines that the first data is the first positioning data, then it controls the robot to perform a grasping operation based on the first positioning data;

[0028] If the operation control terminal determines that the first data is the first image data, it controls the robot to perform obstacle avoidance operation based on the first image data.

[0029] The robot control method provided by this invention receives data sent from a data processing terminal through the operation control terminal of the robot control system, identifies the data, and controls the robot to perform different operations based on the different types of data obtained from the identification. Compared with the prior art, the operation control terminal of this invention only needs to control the robot to perform corresponding actions based on the identified data, which reduces the amount of computation it needs to undertake, improves the computing speed of the operation control terminal, and thus improves the reaction speed of the robot control system, thereby improving the sensitivity of the robot.

[0030] As a preferred example, the first data is determined by the data processing terminal based on the judgment result of the object to be processed in the first image data, and the judgment result is determined by the data processing terminal performing contour judgment on the first image data, specifically including:

[0031] The data processing terminal extracts the object contour from the first image data to obtain a first contour.

[0032] The first contour is compared and fitted with a preset model contour. The object to be processed in the first image data corresponding to the first contour whose fitting result is greater than or equal to the first fitting threshold is determined as the object to be grasped, and the first data is determined as the first positioning data.

[0033] The object to be processed in the first image data corresponding to the first contour whose fitting result is less than the first fitting threshold is determined as the obstacle to be avoided, and the first data is determined as the first image data.

[0034] To illustrate the specific source of the data received by the operation control terminal of the robot control system, the present invention also provides a method for performing contour judgment based on first image data and determining the specific content of the first data based on the judgment result.

[0035] As a preferred example, controlling the robot to perform the grasping operation based on the first positioning data specifically includes:

[0036] The first positioning data includes the position data and size data of the object to be grasped;

[0037] Based on the location data in the first positioning data, the robot's mobile device is controlled to move to the location of the object to be grasped, and based on the size data in the first positioning data, the robot's grasping device is controlled to perform a grasping operation on the object to be grasped.

[0038] The operation control terminal of this invention controls the mobile device and the grasping device based on the first positioning data. The operation control terminal only needs to send a movement control signal to the robot's mobile device based on the position data in the first positioning data, and send a grasping control signal to the robot's grasping device based on the size data to end the task. The computational burden is relieved, and the corresponding computational speed is improved.

[0039] As a preferred example, controlling the robot to perform obstacle avoidance operations based on the first image data specifically includes:

[0040] The straight-line distance between the obstacle to be avoided and the robot is determined based on the first image data and the ultrasonic data received from the ultrasonic module.

[0041] Based on the ultrasonic data obtained by the ultrasonic module in real time monitoring, it is determined whether the straight-line distance is less than or equal to a preset first distance threshold. If it is determined that the straight-line distance is less than or equal to the preset first distance threshold, the ultrasonic module is controlled to determine several distance data between the object to be avoided and the robot, and an obstacle avoidance operation is performed based on the several distance data.

[0042] If the straight-line distance is determined to be greater than a preset first distance threshold, the mobile device is controlled to move forward until the straight-line distance is less than or equal to the preset first distance threshold.

[0043] Since the data processing unit of this invention is responsible for depth calculation of the positioning data of the object to be grasped, the complex calculation process required for obstacle avoidance is handled by the operation control unit. When calculating the distance to the object to be avoided, the operation control unit of this invention not only references the first image data but also incorporates ultrasonic data collected by the ultrasonic module to correct the distance, improving the accuracy of the obtained straight-line distance and reducing the possibility of errors in subsequent obstacle avoidance operations based on this straight-line distance.

[0044] As a preferred example, the control of the ultrasonic module determines several distance data points between the object to be avoided and the robot, and performs obstacle avoidance operations based on the several distance data points, specifically including:

[0045] The ultrasonic module is controlled to rotate to the right and to the left by a preset number of angles, and the distance data collected after the rotation is integrated into a distance array.

[0046] Select a first distance data from the distance array, and compare the first distance data with the remaining distance data in the distance array excluding the first distance data, so that the first distance data is the maximum value in the distance array;

[0047] Based on the first angle corresponding to the first distance data, the robot is controlled to perform obstacle avoidance operations on the object to be avoided.

[0048] This invention determines the obstacle avoidance angle by controlling the ultrasonic module to perform a 180-degree scan in front of the robot. It then compares several straight-line distances detected at each angle with each other, and the angle corresponding to the maximum value of the above-mentioned straight-line distances determined by the comparison results is the final obstacle avoidance angle. After determination, the operation control terminal controls the robot's mobile device to rotate and avoid obstacles according to the obstacle avoidance angle.

[0049] This obstacle avoidance method improves the accuracy of the robot's obstacle avoidance operation and is applicable to various types of courts, reducing the interference of the external environment on the robot's obstacle avoidance operation.

[0050] As a preferred example, after the operation control terminal receives the first data sent by the data processing terminal, the method further includes:

[0051] In response to a control trigger signal, the control trigger signal is simultaneously parsed to obtain a first control signal, and the data processing system is controlled according to the first control signal;

[0052] The received first image data is fed back to the server in real time, so that the server sends the first image data to the PC so that the PC can display the first image data.

[0053] This invention enables remote communication and control of robots through the aforementioned server-side forwarding method. Considering the complexity of the robot's operating environment, when the robot is trapped in a dead corner and unable to move, the user can remotely determine the robot's working status and remotely control the robot, thereby improving the user's experience and comfort.

[0054] Accordingly, the present invention also provides a robot control system, which includes an operation control terminal and a data processing terminal;

[0055] The operation control terminal is connected to the data processing terminal;

[0056] The data processing terminal can execute any of the above-described image recognition-based data processing methods;

[0057] The operation control terminal can execute any of the above-described image recognition-based robot control methods. Attached Figure Description

[0058] Figure 1: A flowchart illustrating an embodiment of a data processing method based on image recognition provided by the present invention;

[0059] Figure 2 : A flowchart illustrating an embodiment of steps 101 and 102A provided by the present invention;

[0060] Figure 3 : A flowchart illustrating an embodiment of an image recognition-based robot control method provided by the present invention;

[0061] Figure 4 : A flowchart illustrating an embodiment of step 302B provided by the present invention;

[0062] Figure 5 : A schematic diagram illustrating an embodiment of the specific working process of an ultrasonic module provided by the present invention;

[0063] Figure 6 : A flowchart illustrating one embodiment of step 402A provided by the present invention;

[0064] Figure 7 : A schematic diagram of an embodiment of a robot control system provided by the present invention;

[0065] Figure 8 : A structural schematic diagram of an embodiment of a robot provided by the present invention. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Example 1

[0068] Please refer to Figure 1 This is a flowchart illustrating an embodiment of a data processing method based on image recognition provided by the present invention, applied to a robot control system; wherein the robot control system includes a data processing terminal and an operation control terminal. Figure 1 As shown, the processing method includes steps 101 to 102, wherein step 102 includes steps 102A and 102B:

[0069] Step 101: The data processing terminal responds to the image processing signal, receives the image data to be processed, performs preliminary image data processing on the image data to be processed to obtain first image data, and performs contour judgment on the first image data. Based on the judgment result, it determines that the object to be processed in the first image data is an object to be grabbed or an object to be avoided.

[0070] In this embodiment, the data processing end determines whether the object to be processed in the first image data is an object to be grabbed or an object to be avoided by performing contour judgment on the first image data. This judgment method is simple and direct, and the judgment result provides a reference basis for performing different operations in the future.

[0071] Simultaneously, preliminary image data processing is performed on the image data to be processed, including noise reduction and color and spatial transformation of objects in the image, so that the outline of the processed object can be extracted and used by the system. This embodiment of the invention does not further limit the specific implementation method of this preliminary processing; users can adjust the specific implementation process as needed. In this embodiment, it is preferred to use OpenCV and related algorithms for preliminary processing of the image data to be processed.

[0072] Step 102A: If the data processing terminal determines that the judgment result is an object to be grasped, then the first image data is processed for image depth to obtain the first positioning data of the object to be grasped, and the first positioning data is sent to the operation control terminal so that the operation control terminal performs the grasping operation according to the first positioning data.

[0073] Step 102B: If the data processing terminal determines that the judgment result is an object to be avoided, the first image data is sent to the operation control terminal so that the operation control terminal performs an obstacle avoidance operation based on the first image data.

[0074] In this embodiment, the subsequent execution steps are determined based on the above judgment results, and the calculated first data is sent to the operation execution terminal. The operation execution terminal then determines the control steps to be executed based on the received data content. Therefore, the data processing terminal only needs to handle the data processing part, while the operation control terminal only needs to execute the robot control part. The two terminals each perform their respective functions, and the entire robot control system operates through data interaction.

[0075] In summary, the data processing method provided by this invention divides the calculation process for a large amount of image data into two steps and assigns them to two different terminals for processing. The data processing part for image data is handled by the data processing terminal, which is mainly responsible for determining the type of object to be processed and performing different operations on the first image data according to the object type. The operation control terminal is mainly responsible for performing different control operations on the robot according to the different types of data sent by the data processing terminal.

[0076] This invention assigns the image data processing and operation execution parts to two different terminals. A complete robot control system is constructed through data interaction between the two terminals. At the same time, the computational burden of a complete system is distributed between the two terminals. Compared with the existing technology where a single-chip microcomputer is used as the main controller to handle all the computations of the entire control system, the processing method provided by this invention reduces the computational burden on both terminals, improves the computational speed of the robot control system, and thus reduces the possibility of recognition errors in the system.

[0077] As another example of this embodiment, see Figure 2 , Figure 2 A flowchart illustrating one embodiment of steps 101 and 102A includes steps 201 to 205, wherein step 202 includes steps 202A and 202B.

[0078] Step 201: Extract the object contour from the first image data to obtain a first contour, and compare and fit the first contour with a preset model contour.

[0079] This example involves extracting the contour of the object to be processed from the first image data, comparing and fitting the extracted first contour with a preset model contour in the system, and determining the specific object type based on the fitting result. Users can store model contours in the system in advance according to actual needs and update and modify them, allowing the robot to flexibly adjust its object grasping capabilities.

[0080] Step 202A: The object to be processed in the first image data corresponding to the first contour whose fitting result is greater than or equal to the first fitting threshold is determined as the object to be grasped.

[0081] Step 202B: The object to be processed in the first image data corresponding to the first contour whose fitting result is less than the first fitting threshold is identified as the obstacle to be avoided.

[0082] This example compares and fits the contours extracted from the first image data with a preset model contour, and then compares the fitting result with a preset first fitting threshold to determine the object type of the object to be processed. The judgment method is simple and direct, and the operation process is concise and clear. At the same time, users can adjust the specific value of the first fitting threshold in a timely manner according to the judgment result or the actual scenario to avoid the problem that the threshold is too high, resulting in too few objects being recognized and the accuracy is too low, or the threshold is too low, resulting in too many objects being recognized and the accuracy is too low.

[0083] Step 203A: Use the YOLO-V3 algorithm to identify the object to be captured in the first image data and obtain the corresponding recognition result.

[0084] Step 204A: Call the YOLO-V3 algorithm to calculate the recognition result and the first image data to determine the first positioning data of the object to be captured.

[0085] Step 205A: Simultaneously invoke a preset neural network acceleration algorithm to accelerate the operation process of the YOLO-V3 algorithm.

[0086] The above three steps are the specific implementation methods for performing depth processing on the first image data in step 102A. They include calling the preset YOLO-V3 algorithm to identify the object to be grasped, determining the first positioning data of the object to be grasped based on the identification result and the first image data, and calling the preset neural network acceleration algorithm to accelerate the operation of the YOLO-V3 algorithm.

[0087] The YOLO-V3 algorithm employs Feature Pyramid Networks (FPNs), significantly improving the performance of small object detection with minimal increase in computational cost. Compared to existing technologies using YOLO-V1 or YOLO-V2, this algorithm is more efficient at identifying small spherical objects, making it more suitable for this task. Furthermore, in this example, the YOLO-V3 algorithm also performs global prediction and model building on the object to be grasped in the first image data, improving the accuracy of subsequent grasping operations.

[0088] Meanwhile, the computation process is accelerated by a preset neural network acceleration algorithm, which further improves the algorithm's computing power, thereby increasing the computational efficiency of the data processing end and reducing the time required for this deep processing step.

[0089] This example uses a neural network acceleration algorithm stored in the Intel Gen 2 Neural Compute Stick. This accelerator stick can serve as the computing core for AI applications on IoT devices, capable of handling inference tasks for common deep learning models such as image classification and license plate detection, thus enabling edge computing. Furthermore, the Intel Gen 2 Neural Compute Stick can run directly on a standard USB 3.0 port. Developers only need a laptop and the Intel Gen 2 Neural Compute Stick to quickly get AI and computer vision running; its main function is to enhance the computing power of connected modules.

[0090] Example 2

[0091] See Figure 3 , Figure 3 The flowchart illustrates an embodiment of a robot control method based on image recognition provided by the present invention, including steps 301 to 304, wherein step 302 includes steps 302A and 302B, and the specific details of each step are as follows:

[0092] Step 301: The operation control terminal receives the first data sent by the data processing terminal, and calls a preset recognition method to identify the first data and determine that the first data is the first positioning data or the first image data.

[0093] After receiving the first data sent from the data processing terminal, the operation control terminal of the robot control system can determine the specific data type of the first data by calling a preset recognition method, that is, identify the first data as first positioning data or first image data, and provide a data basis for subsequent control operations.

[0094] Step 302A: If the operation control terminal determines that the first data is the first positioning data, then control the robot to perform a grasping operation according to the first positioning data.

[0095] Step 302B: If the operation control terminal determines that the first data is the first image data, then control the robot to perform obstacle avoidance operation according to the first image data.

[0096] If the first identified data is first positioning data, the robot is controlled to perform a grasping operation; if the first identified data is first image data, the robot is controlled to perform an obstacle avoidance operation. Compared with the prior art, the operation control terminal of the present invention only needs to control the robot to perform corresponding actions based on the identified data, which reduces the amount of computation it needs to undertake, improves the computing speed of the operation control terminal, and thus improves the reaction speed of the robot control system, thereby improving the sensitivity of the robot.

[0097] Step 303A: Control the robot's mobile device to move to the location of the object to be grasped based on the location data in the first positioning data.

[0098] Step 304A: Control the robot's gripping device to perform a gripping operation on the object to be gripped based on the size data in the first positioning data.

[0099] The control system's operation control terminal only needs to send a movement control signal to the robot's mobile device based on the position data in the first positioning data, and a grasping control signal to the robot's grasping device based on the size data to end the control task for this round. After the task of performing depth processing on the first image data to obtain the corresponding first positioning data is handed over to the data processing terminal, the operation control terminal only needs to send the corresponding control signal to the robot based on the identified data type to end the task. Its computational burden is relieved, and the corresponding computational speed will be improved.

[0100] As another example of this embodiment, see Figure 4 , Figure 4 The flowchart of one embodiment of step 302B provided by the present invention is shown, including steps 401 to 402, wherein step 402 includes steps 402A and 402B, and the specific steps are as follows:

[0101] Step 401: Determine the straight-line distance between the obstacle to be avoided and the robot based on the first image data and the ultrasonic data received from the ultrasonic module, and determine whether the straight-line distance is less than or equal to a preset first distance threshold based on the ultrasonic data obtained by the ultrasonic module in real time monitoring.

[0102] In this embodiment, the control system's operation control terminal not only estimates the position data of the object to be avoided by referring to the first image data, but also corrects the estimated position data by referring to the ultrasonic data sent by the ultrasonic module. This ensures that the final determined straight-line distance between the object to be avoided and the robot is more accurate, thereby improving the accuracy of the robot's subsequent obstacle avoidance actions and reducing the possibility of errors arising from subsequent obstacle avoidance operations based on this straight-line distance. This embodiment of the invention does not further limit the first distance threshold; users can adjust it according to the actual scenario requirements. In this example, the first distance threshold is preferably 5 centimeters.

[0103] In this ultrasonic module, when the trigger angle (Trig) is input with a high potential of at least 10µs, it can transmit ultrasonic waves. During the time between the transmission of the ultrasonic wave by the trigger angle (Trig) and the reception of the returned ultrasonic wave by the response angle (Echo), the response angle (Echo) bit remains at a high level. Therefore, the distance to the object being measured can be calculated from the duration of the continuous high-level pulse on the response angle (Echo). Simultaneously, during this period, the ultrasonic module internally emits several 40kHz pulse signals to control the operation of the trigger angle (Trig) and response angle (Echo). The specific implementation idea of ​​the ultrasonic module is as follows... Figure 5 As shown.

[0104] As another example of this embodiment, participants Figure 6 , Figure 6 A flowchart illustrating an embodiment of step 402A provided by the present invention includes steps 601 to 603, each of which is detailed below:

[0105] Step 601: Control the ultrasonic module to rotate to the right and to the left by a preset number of angles, and integrate the distance data collected after the rotation into a distance array.

[0106] Once the robot has traveled to a predetermined distance in front of the obstacle to be avoided, the system will control the ultrasonic module to perform a scanning distance measurement at a maximum angle of 180 degrees in front of the robot. This embodiment of the invention does not further limit the specific number of distance measurement groups; the user can adjust this value as needed. After the ultrasonic module has rotated several times and acquired corresponding distance data at several angles, the system integrates these distance data into a distance array to provide a data basis for subsequently determining the accurate obstacle avoidance angle.

[0107] Step 602: Select and determine a first distance data from the distance array, and compare the first distance data with the remaining distance data in the distance array excluding the first distance data, so that the first distance data is the maximum value in the distance array.

[0108] By comparing each distance data in the integrated distance array with all other distance data in the array except for the selected distance data, the distance data with the largest value in the distance array is obtained. This distance data is then used as the first distance data and transmitted to subsequent obstacle avoidance operations.

[0109] Step 603: Control the robot to perform obstacle avoidance operation on the object to be avoided according to the first angle corresponding to the first distance data.

[0110] If the system determines that there are no obstacles at the angle corresponding to the first distance data, it controls the robot's mobile device to rotate and avoid obstacles based on the first angle corresponding to the first distance data.

[0111] In summary, this obstacle avoidance method improves the accuracy of the robot's obstacle avoidance operation, is applicable to various types of sports fields, and reduces the interference of the external environment on the robot's obstacle avoidance operation.

[0112] As another example of this embodiment, in addition to receiving the first data sent by the data processing terminal, the operation control terminal can also respond to the control trigger signal, parse the control trigger signal to obtain the first control signal, control the data processing system according to the first control signal, and feed back the received first image data to the server in real time, so that the server sends the first image data to the PC so that the PC displays the first image data.

[0113] In summary, this embodiment also achieves remote communication and control of the robot through server-side forwarding. Considering the complexity of the robot's operating environment, when the robot is trapped in a dead corner and cannot move, the user can remotely determine the robot's working status and remotely control the robot, thereby improving the user's experience and comfort.

[0114] In this embodiment, the user preferably transmits control commands via a WeChat mini-program to remotely control the robot. The WeChat mini-program publishes robot control topics through the server, and the operation control terminal subscribes to these topics through the server. Each time the user publishes content in the robot control topic via the WeChat mini-program, the operation control terminal receives the content sent by the user through the robot control topic, and the corresponding control trigger signal is activated.

[0115] Similarly, users can receive image data from the control terminal by subscribing to control feedback topics published by the control terminal, thereby enabling better remote control of the robot.

[0116] Accordingly, in order to better illustrate the working principle and steps of the image recognition-based data processing method and robot control method of the present invention, you may refer to the relevant descriptions above, but not limited to.

[0117] See Figure 7 , Figure 7 This is a schematic diagram of the structure of an embodiment of a robot control system provided by the present invention, as shown below. Figure 7As shown, the robot control system 701 includes a data processing terminal 702A and an operation control terminal 702B, wherein the data processing terminal 702A and the operation control terminal 702B are interconnected, and the data flow between the two is from the data processing terminal 702A to the operation control terminal 702B. Simultaneously, the data processing terminal 702A can execute the aforementioned image recognition-based data processing method, while the operation control terminal can execute the aforementioned image recognition-based robot control method.

[0118] In this embodiment, the data processing terminal 702A of the robot control system 701 is preferably hosted on a Raspberry Pi control motherboard, while the operation control terminal 702B is preferably hosted on an STM32 microcontroller. The image acquisition device for acquiring the image to be processed is preferably a binocular camera. The robot's grasping device is preferably a combination of a brushed motor and a roller. The robot's mobile device is a combination of an L298n drive and a stepper motor. The ultrasonic module is a combination of a servo motor and an HC-SR04 ultrasonic module. The data processing terminal 702A, i.e., the Raspberry Pi chip, is also connected to an OLED screen and an infrared sensor to determine and display the number of grasped objects. The server is preferably an MQTT server. (See above for more details.) Figure 8 , Figure 8 This is a structural schematic diagram of an embodiment of a robot provided by the present invention.

[0119] The gripping device uses a brushed motor to control a roller to grasp objects, while the mobile device uses an L298n drive to move a stepper motor. The ultrasonic module's servo motor drives the HC-SR04 ultrasonic module to rotate at several angles. Simultaneously, the Raspberry Pi communicates with the STM32 microcontroller via serial port, and the STM32 microcontroller also communicates with the server via serial port.

[0120] In this embodiment, an STM32 microcontroller is preferred as the carrier device for the operation control terminal 702B because the STM32 chip has excellent high performance, high real-time performance, low power consumption, and low cost. At the same time, the rich peripherals of STM32 facilitate development. It supports a variety of communication methods, such as IIC, SPI, UART / USART, and CAN, which are commonly used to facilitate the implementation of communication functions.

[0121] The Raspberry Pi is preferred as the carrier device for the data processing terminal 702A because it is equipped with an ARM architecture CPU. The ARM architecture can achieve high execution efficiency, and its low power consumption is very suitable for use in power-sensitive devices. At the same time, the communication chips and protocols embedded in it facilitate the integration of robots in machine vision, making it more powerful than other chips while being smaller in size. Therefore, robots equipped with Raspberry Pi are more agile than those with other chips, realizing a true embedded vision system.

[0122] Binocular cameras are preferred as image acquisition devices for capturing images to be processed because they require less computation and have lower performance requirements than monocular cameras, while producing smaller errors. Therefore, the position of the object to be grasped is more accurate when determined based on the image data acquired by binocular cameras.

[0123] The combination of a brushed motor and a roller is preferred for the robot's gripping device because, compared to a robotic arm, a roller can grip an object in a larger space and is easier to operate, thus improving the robot's gripping efficiency.

[0124] If an MQTT server is preferred as the server-side component, data transmission between the STM32 microcontroller and the PC is achieved via the MQTT protocol. The MQTT server converts received data into JSON format, segments it, and then uploads it, facilitating the posting of control content by the user. Users can access the robot control interface via a WeChat mini-program by entering the server's URL and send control content using preset control buttons defined in control.js.

[0125] In summary, this invention provides a data processing method and a robot control method based on image recognition, applied to a robot control system, wherein the robot control system includes a data processing terminal and an operation control terminal. The method disclosed in this invention divides the computational processing of large amounts of image data into two steps: a data processing step and a robot control step. These two steps are handled separately by two terminals. The data processing terminal executes the data processing method and sends the obtained first data to the operation control terminal, which then executes the robot control method based on the first data sent by the data processing terminal. This approach simultaneously reduces the computational burden on both terminals, improves the computational speed and accuracy of the robot control system, and thus reduces the possibility of recognition errors.

[0126] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A data processing method based on image recognition, characterized in that, Applied to robot control systems; wherein the robot control system includes a data processing terminal and an operation control terminal; The data processing method includes the following steps: The data processing terminal responds to the image processing signal, receives the image data to be processed, performs preliminary image data processing on the image data to be processed to obtain first image data, and performs contour judgment on the first image data. Based on the judgment result, it determines that the object to be processed in the first image data is an object to be grabbed or an object to be avoided. If the data processing terminal determines that the judgment result is an object to be grasped, then it performs image depth processing on the first image data to obtain the first positioning data of the object to be grasped, and sends the first positioning data to the operation control terminal so that the operation control terminal performs a grasping operation based on the first positioning data. If the data processing terminal determines that the judgment result is an object to be avoided, it sends the first image data to the operation control terminal so that the operation control terminal performs an obstacle avoidance operation based on the first image data; the step of making the operation control terminal perform an obstacle avoidance operation based on the first image data includes: So that the operation control terminal can determine the straight-line distance between the obstacle to be avoided and the robot based on the first image data and the ultrasonic data received from the ultrasonic module. This allows the operation control terminal to determine whether the straight-line distance is less than or equal to a preset first distance threshold based on the ultrasonic data obtained by the ultrasonic module in real time. If the straight-line distance is determined to be less than or equal to the preset first distance threshold, the operation control terminal controls the ultrasonic module to determine several distance data between the object to be avoided and the robot, and the operation control terminal performs obstacle avoidance operation based on the several distance data. If it is determined that the straight-line distance is greater than a preset first distance threshold, then the operation control terminal controls the mobile device to move forward until the straight-line distance is less than or equal to the preset first distance threshold; Specifically, the step of having the operation control terminal control the ultrasonic module to determine several distance data points between the object to be avoided and the robot, and having the operation control terminal perform obstacle avoidance operations based on the several distance data points, includes: The operation control terminal controls the ultrasonic module to rotate to the right and to the left by a preset number of angles, and the operation control terminal integrates the distance data acquired after the rotation into a distance array. The operation control terminal selects and determines a first distance data from the distance array, and compares the first distance data with the remaining distance data in the distance array excluding the first distance data, so that the first distance data is the maximum value in the distance array; The operation control terminal controls the robot to perform obstacle avoidance operation on the object to be avoided based on the first angle corresponding to the first distance data.

2. The data processing method based on image recognition as described in claim 1, characterized in that, The step of performing contour judgment on the first image data and determining, based on the judgment result, whether the object to be processed in the image data to be processed is an object to be grasped or an object to be avoided specifically involves: The object contour is extracted from the first image data to obtain the first contour; The first contour is compared and fitted with a preset model contour, and the object to be processed in the first image data corresponding to the first contour whose fitting result is greater than or equal to the first fitting threshold is determined as the object to be grasped. The object to be processed in the first image data corresponding to the first contour whose fitting result is less than the first fitting threshold is identified as the obstacle to be avoided.

3. The data processing method based on image recognition as described in claim 1, characterized in that, The step of performing image depth processing on the first image data to obtain the first positioning data of the object to be grasped specifically includes: The YOLO-V3 algorithm is used to identify the objects to be captured in the first image data, and the corresponding recognition results are obtained. The YOLO-V3 algorithm is invoked to calculate the recognition result and the first image data to determine the first positioning data of the object to be captured; At the same time, a preset neural network acceleration algorithm is invoked to accelerate the operation process of the YOLO-V3 algorithm.

4. A robot control method based on image recognition, characterized in that, Applied to robot control systems; wherein the robot control system includes a data processing terminal and an operation control terminal; The robot control method is implemented based on the image recognition-based data processing method as described in claim 1, and includes the following steps: The operation control terminal receives first data sent by the data processing terminal and calls a preset recognition method to identify the first data to determine whether the first data is first positioning data or first image data; wherein, the first data is determined by the data processing terminal based on the judgment result of the object to be processed in the first image data; the judgment result is determined by the data processing terminal by performing contour judgment on the first image data; the first image data is obtained by the data processing terminal after performing preliminary image data processing on the image data to be processed; If the operation control terminal determines that the first data is the first positioning data, then it controls the robot to perform a grasping operation based on the first positioning data; If the operation control terminal determines that the first data is the first image data, it controls the robot to perform obstacle avoidance operation based on the first image data.

5. The robot control method based on image recognition as described in claim 4, characterized in that, The first data is determined by the data processing terminal based on the judgment result of the object to be processed in the first image data, and the judgment result is determined by the data processing terminal performing contour judgment on the first image data, specifically including: The data processing terminal extracts the object contour from the first image data to obtain a first contour. The first contour is compared and fitted with a preset model contour. The object to be processed in the first image data corresponding to the first contour whose fitting result is greater than or equal to the first fitting threshold is determined as the object to be grasped, and the first data is determined as the first positioning data. The object to be processed in the first image data corresponding to the first contour whose fitting result is less than the first fitting threshold is determined as the obstacle to be avoided, and the first data is determined as the first image data.

6. The robot control method based on image recognition as described in claim 4, characterized in that, The process involves controlling the robot to perform a grasping operation based on the first positioning data. include: The first positioning data includes the position data and size data of the object to be grasped; Based on the location data in the first positioning data, the robot's mobile device is controlled to move to the location of the object to be grasped, and based on the size data in the first positioning data, the robot's grasping device is controlled to perform a grasping operation on the object to be grasped.

7. The robot control method based on image recognition as described in claim 4, characterized in that, After the operation control terminal receives the first data sent by the data processing terminal, it further includes: In response to a control trigger signal, the control trigger signal is simultaneously parsed to obtain a first control signal, and the data processing system is controlled according to the first control signal; The received first image data is fed back to the server in real time, so that the server sends the first image data to the PC so that the PC can display the first image data.

8. A robot control system, characterized in that, The robot control system includes an operation control terminal and a data processing terminal; The operation control terminal is connected to the data processing terminal; The operation control terminal can execute a robot control method based on image recognition as described in any one of claims 4-6.

Images