Visual drive and control integrated controller
By integrating visual detection, servo drive and motion control functions in one CPU, using multi-core CPUs and FPGAs to achieve real-time communication, the complexity and delay problems of traditional robot systems are solved, and the performance and efficiency of robot systems are improved.
Patent Information
- Application Number
- CN202411243984.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-05
- Publication Date
- 2024-11-12
AI Technical Summary
Traditional robot systems design visual inspection, servo drive and motion control as independent modules, resulting in increased system complexity, reduced information transmission delay and control accuracy, and unable to realize real-time information processing.
Design a visual drive-in integrated controller to integrate visual detection, servo drive and motion control functions into one CPU, real-time communication and coordination between modules through inter-core communication, shared memory and Mailbox data structure of multi-core CPUs, and use FPGA controls the drive action.
The structure of the robot system is simplified, complexity is reduced, response speed and operation efficiency is improved, real-time information processing and precise control are realized, and the overall performance of the robot system is improved.
Smart Images

Figure CN118938779A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of robot and automation technology, in particular to a vision-driven-control integrated controller. Background Art
[0002] With the rapid development of robotics technology, visual inspection, servo drive and motion control, as the three key technologies of robotic systems, each play an important role. However, traditional robotic systems often design and implement these functions as independent modules, which not only increases the complexity of the system, but also leads to problems such as information transmission delay and reduced control accuracy. Therefore, developing a controller that can integrate visual inspection, servo drive and motion control functions is of great significance to improving the operating efficiency and performance of robots. Summary of the invention
[0003] In order to overcome the defects in the prior art, the present invention provides a visual-driven-control integrated controller, which integrates the three core functions of visual detection, servo drive and motion control into one, thereby simplifying the structure of the entire robot system, reducing the complexity of the robot system, and improving the overall performance and operating efficiency of the robot system.
[0004] In order to achieve the above object, the technical solution adopted by the present invention is: a visual drive control integrated controller, comprising:
[0005] A CPU, wherein the CPU has a first core and a second core, the first core runs a first system, the first system is used to collect and process image information, obtain visual information, and generate control instructions according to the visual information, and the second core runs a second system, the second system is used to read the control instructions and generate scheduling information according to the control instructions;
[0006] FPGA, the FPGA is connected to the second core via a PCIE bus, the FPGA is used for an external driver, and the FPGA controls the action of the driver connected thereto according to the scheduling information.
[0007] Through the above technical solution, the three core functions of visual inspection, servo drive and motion control required for control are integrated into one CPU to simplify the structure of the entire robot system. Compared with the traditional robot system that installs and connects these three functions as three separate modules, and the communication and coordination between the modules are cumbersome, this application reduces the complexity of the robot system and improves the overall performance and operating efficiency of the robot system.
[0008] Furthermore, the first core and the second core implement data exchange through inter-core communication. Inter-core communication is a critical part of multi-core CPU design, which allows different cores, namely the first core and the second core, to implement data sharing, event notification, and state synchronization based on shared or independent resources.
[0009] Furthermore, a shared memory is provided between the first core and the second core, and the first core can write information into the shared memory for the second core to read. The first core and the second core share a physical memory area, and avoid data conflicts through specific synchronization mechanisms such as locks and semaphores. The first core can write data that needs to be passed to the second core into the shared memory, and the second core reads data from the shared memory through a task scheduling mechanism.
[0010] Furthermore, when the task of the second core needs to exchange information with the task of the first core, the task of the second core can trigger task interruption through a semaphore mechanism, or wait for task interruption of the first core.
[0011] Furthermore, a Mailbox data structure is established between the first core and the second core, and the first system can encapsulate data into messages and send them to the second system through the Mailbox data structure.
[0012] Furthermore, when the second core communicates with the FPGA, the transmitting end outputs two data streams in opposite directions through a differential driver, wherein one data stream carries a first signal and the other data stream carries a complementary signal of the first signal. The receiving end performs differential detection on the two data streams and determines the logical state of the transmission by comparing the voltage difference between the two data streams.
[0013] Furthermore, the first core includes at least two CPU cores, the first core runs the Qiuniu real-time operating system, and the second core includes at least one CPU core, the second core runs the FreeRTOS operating system.
[0014] Furthermore, the first system includes:
[0015] A visual detection module is used to obtain image information, perform feature extraction on the image information, and obtain visual information;
[0016] The motion control module plans the robot's motion trajectory according to the visual information and generates control instructions.
[0017] Furthermore, the second system includes:
[0018] A driving module reads the control instruction and generates scheduling information, identifies the scheduling information through the FPGA and controls the action of a driver connected to the FPGA.
[0019] A vision-driven-control integrated robot comprises a robot body and a vision-driven-control integrated controller as described in any one of claims 1 to 8, wherein the vision-driven-control integrated controller is used to control the movement of the robot body.
[0020] By means of the above technical solution, the beneficial effects of the present invention are as follows:
[0021] In this application, the visual detection module, motion control module and drive module are integrated into the same CPU, that is, the three core functions of visual detection, servo drive and motion control are integrated into one, so as to reduce the complexity of the robot system and simplify the structure of the entire robot system. Compared with the traditional robot system, these three functions are installed and connected as three separate modules, the communication and coordination between the modules are cumbersome, the communication between modules has a large delay, the response speed is slow, and real-time information processing cannot be achieved. However, this application integrates the visual detection module, motion control module and drive module into the same CPU, and the modules communicate through shared memory or Mailbox data structure, which can realize real-time processing and precise control of information, thereby improving the response speed and operating efficiency of the robot and improving the overall performance of the robot system.
[0022] In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0024] Figure 1 Schematic diagram of the overall structure of the visual drive and control integrated controller in an embodiment of the present invention.
[0025] The figure numbers of the above figures are: 1. CPU; 11. First core; 12. Second core; 2. Camera; 3. FPGA; 4. Driver. DETAILED DESCRIPTION
[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0027] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and to distinguish similar objects. There is no order of precedence between the two, and they cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, "plurality" means two or more.
[0028] Example: Combination Figure 1 As shown, embodiment 1 discloses a visual, drive and control integrated controller, including:
[0029] CPU1, the CPU1 has a first core 11 and a second core 12, the first core 11 runs a first system, and the second core 12 runs a second system. The CPU1 is a central processing unit, which is the final execution unit for information processing and program running. The CUP1 used in this application includes six cores, namely core0 to core5, wherein core0 to core4 are the first core 11, and core5 is the second core 12. Data interaction is achieved between the first core 11 and the second core 12 through inter-core communication. Inter-core communication is an extremely critical part of multi-core CPU design, which allows different cores, namely the first core 11 and the second core 12, to achieve data sharing, event notification and status synchronization based on shared or independent resources.
[0030] Inter-core communication between multi-core processors has the following advantages:
[0031] The inter-core communication mechanism between multi-core processors can achieve efficient data transmission, reduce data transmission delays and interruptions, and thus improve the response speed and real-time performance of the controller. This is especially important for robotic applications that require fast response and precise control.
[0032] Through inter-core communication, the controller can fully utilize the parallel processing capabilities of multi-core processors and execute multiple tasks or algorithms simultaneously, thereby improving the overall processing power and computing efficiency.
[0033] The inter-core communication mechanism supports modular design, allowing each part of the controller to be developed and tested independently. This reduces development difficulty and cost, while improving the scalability and maintainability of the system.
[0034] Through inter-core communication, the controller can flexibly handle multiple tasks. Different processor cores can be responsible for different tasks or algorithms, thus achieving parallel processing of tasks and optimal allocation of resources.
[0035] It should be noted that CPUs with different numbers of cores can be selected according to actual needs. The number of CPU cores included in the second core is determined according to the number of driver paths that need to be controlled.
[0036] In this application, the first system is the Necro real-time operating system. The Necro real-time operating system (NECRO) is an industrial real-time operating system independently developed by Guoxun Xinwei, which is mainly used to ensure the efficient and accurate transmission of signals in the system, providing a solid foundation for the real-time perception, decision-making and execution of robots.
[0037] The Qiuniu real-time operating system is provided with a visual detection module and a motion control module. The visual detection module is used to obtain the environmental image collected by the robot's camera 2, and perform feature extraction on the environmental image to obtain the visual information. The motion control module generates a control instruction for manipulating the robot to avoid obstacles based on the visual information. The Qiuniu real-time operating system is built with a task scheduler, which is responsible for allocating the time of the multi-core CPU according to the priority and scheduling strategy of the task. In the drive control, Qiuniu will create special tasks to handle the interaction with the hardware device. These tasks will be scheduled and executed as needed to complete tasks such as data collection and control instruction sending.
[0038] The second system is a FreeRTOS operating system, which is a mini real-time operating system kernel. As a lightweight operating system, the functions include: task management, time management, semaphore, message queue, memory management, recording function, software timer, coroutine, etc. The FreeRTOS operating system is responsible for task scheduling, resource management and interaction with other components in the robot system. The FreeRTOS operating system can receive the control instruction according to the task scheduling mechanism and generate scheduling information according to the control instruction.
[0039] In the present application, a shared memory is provided between the first core 11 and the second core 12. The shared memory is to enable multiple processes to access the same memory space, and is the fastest available IPC form. Other processes can "connect" the same shared memory segment to their own address space. All processes can access the addresses in the shared memory. If a process writes data to this shared memory, the changes made will be immediately seen by other processes that have access to the same shared memory. The use of shared memory greatly reduces the consumption of memory in large-scale data processing. The Prison Bull real-time operating system can write control instructions to the shared memory, and the FreeRTOS operating system reads the control instructions from the shared memory according to the task scheduling mechanism. In the present application, the Prison Bull real-time operating system provides two synchronization mechanisms, mutexes or semaphores, to ensure synchronization and mutual exclusion between processes. When a task needs to exchange information with tasks of other cores, it can trigger task interruption or wait for task interruption of other cores through synchronization mechanisms such as semaphores.
[0040] A Mailbox data structure is also established between the first core 11 and the second core 12. The Mailbox data structure is a communication mechanism that allows messages to be exchanged between processes. Its main feature is that it allows data to be sent to a mailbox in one process and then received by another process. It provides an effective way to transfer data or control information between different tasks. The first system can encapsulate data into messages and send them to the second system through the Mailbox data structure. After receiving the message, the second system performs corresponding operations according to the content of the message and sends feedback information to the first system through the Mailbox data structure.
[0041] The second core 12 is connected to FPGA3 via a PCIe bus. The PCIe bus is a high-performance, scalable serial bus standard. PCIe supports multiple channels, and the number of channels can be configured according to actual needs to match different data transmission rates and bandwidth requirements. The FPGA3 is a field programmable gate array. The FPGA3 is used to connect the various drivers 4 of the robot. The FPGA3 can control the actions of the drivers 4 connected thereto according to the scheduling information. For application scenarios that require high real-time processing, the PCIe communication mechanism can ensure that the data transmission delay between the second core 12 and the FPGA3 is delayed. When the second core and the FPGA3 communicate, the transmitting end outputs two data streams in opposite directions through a differential driver, one of which carries a first signal and the other carries a complementary signal of the first signal. The receiving end performs differential detection on the two data streams and determines the logical state of the transmission by comparing the voltage difference between the two data streams. Differential signal transmission technology effectively improves the anti-interference and stability of data transmission. Differential signal transmission can also reduce the radiation of electromagnetic interference (EMI) and improve the integrity and reliability of signal transmission.
[0042] It should be noted that the robot's camera can adopt a high-resolution camera, and the visual detection module can adopt advanced image processing algorithms such as convolutional neural network or background subtraction method to improve the accuracy and speed of image acquisition and processing.
[0043] Through the above technical solution, in this application, by integrating the visual detection module, motion control module and drive module into the same CPU1, the three core functions of visual detection, servo drive and motion control are integrated into one, so as to reduce the complexity of the robot system and simplify the structure of the entire robot system. Compared with the traditional robot system, these three functions are installed and connected as three separate modules, the communication and coordination between the modules are cumbersome, the communication between modules has a large delay, the response speed is slow, and real-time information processing cannot be achieved. However, this application integrates the visual detection module, motion control module and drive module into the same CPU1, and the modules communicate through shared memory or Mailbox data structure, which can realize real-time processing and precise control of information, thereby improving the response speed and operating efficiency of the robot and improving the overall performance of the robot system.
[0044] Embodiment 2 discloses a vision-driven-controlled integrated robot, comprising a robot body and the vision-driven-controlled integrated controller described above, wherein the vision-driven-controlled integrated controller is used to control the movement of the robot body.
[0045] Wherein, the Qiuniu real-time operating system includes:
[0046] A visual detection module is used to obtain image information, perform feature extraction on the image information, and obtain visual information;
[0047] A motion control module, which plans the robot's motion trajectory according to the visual information and generates control instructions;
[0048] The FreeRTOS operating system reads the control instructions and generates scheduling information through inter-core communication, and finally recognizes the scheduling information through the FPGA and controls the robot's movements.
[0049] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A visual, drive and control integrated controller, characterized in that: include: A CPU, wherein the CPU has a first core and a second core, the first core runs a first system, the first system is used to collect and process image information, obtain visual information, and generate control instructions according to the visual information, and the second core runs a second system, the second system is used to read the control instructions and generate scheduling information according to the control instructions; FPGA, the FPGA is connected to the second core via a PCIE bus, the FPGA is used for an external driver, and the FPGA controls the action of the driver connected thereto according to the scheduling information.
2. The visual, drive and control integrated controller according to claim 1, characterized in that: The first core and the second core implement data exchange through inter-core communication.
3. The visual, drive and control integrated controller according to claim 2, characterized in that: A shared memory is provided between the first core and the second core, and the first core can write information into the shared memory for the second core to read.
4. The visual, drive and control integrated controller according to claim 3, characterized in that: When the task of the second core needs to exchange information with the task of the first core, the task of the second core can trigger task interruption through the semaphore mechanism, or wait for the task interruption of the first core.
5. The visual, drive and control integrated controller according to claim 2, characterized in that: A Mailbox data structure is established between the first core and the second core. The first system can encapsulate data into messages and send them to the second system through the Mailbox data structure.
6. The visual, drive and control integrated controller according to claim 1, characterized in that: When the second core communicates with the FPGA, the transmitting end outputs two data streams in opposite directions through a differential driver, wherein one data stream carries a first signal and the other data stream carries a complementary signal of the first signal. The receiving end performs differential detection on the two data streams and determines the logical state of the transmission by comparing the voltage difference between the two data streams.
7. The visual, drive and control integrated controller according to claim 2, characterized in that: The first core includes at least two CPU cores, and the first core runs the Qiuniu real-time operating system. The second core includes at least one CPU core, and the second core runs the FreeRTOS operating system.
8. The visual, drive and control integrated controller according to claim 7, characterized in that: The first system comprises: A visual detection module is used to obtain image information, perform feature extraction on the image information, and obtain visual information; The motion control module plans the robot's motion trajectory according to the visual information and generates control instructions.
9. The visual, drive and control integrated controller according to claim 8, characterized in that: The second system comprises: A driving module reads the control instruction and generates scheduling information, identifies the scheduling information through the FPGA and controls the action of a driver connected to the FPGA.
10. A visual, drive and control integrated robot, characterized in that: It comprises a robot body and the visual-driven-controlled integrated controller according to any one of claims 1 to 9, wherein the visual-driven-controlled integrated controller is used to control the movement of the robot body.
