Biological robot hybrid control system and method based on visual perception

Abstract

The invention discloses a biological robot hybrid control system and method based on visual perception. The system comprises a virtual scene assembly, a biological behavior acquisition module, a carrier, an upper computer, an environment information acquisition module and a wireless communication module. A real environment is converted into optical flow stimulation, the optical flow stimulation is provided for organisms, behavior information of the organisms is collected to drive mechanical motion, and the function of controlling mechanical motion based on biological behaviors is achieved. According to the application, the carrier can be placed in a special environment or is directly controlled by the upper computer to complete specified movement so as to research the mechanism of organisms. According to the application, the virtual scene, the biological behavior acquisition module, the environment information acquisition module, the carrier and the upper computer are combined modularly, and efficient data transmission and real-time control are realized by using the wireless communication module. According to the modular design, the system development and maintenance complexity is greatly reduced, the system expansibility and the function optimization capability are improved, and high adaptability and flexibility are achieved.

Description

Technical Field

[0001] The present application belongs to the field of insect-machine hybrid control technology, and specifically relates to a bio-robot hybrid control system and method based on visual perception. Background Art

[0002] Cyborg Insects is an interdisciplinary frontier research field that integrates insect biology, robotics and control engineering. This field simulates and utilizes the biological mechanisms of insects to control and optimize robot behavior, giving full play to their advantages in perception, movement patterns and environmental adaptability. This not only promotes basic research on insect behavior and neural mechanisms, but also provides important technical support and application prospects for the development of bio-inspired robots, intelligent control technologies and autonomous systems.

[0003] Organisms with compound eyes usually have visual characteristics such as wide field of view, low resolution, and sensitivity to optical flow. This unique visual characteristic enables them to efficiently complete navigation and behavioral decisions in complex environments. Existing research has mostly focused on simulating the visual mechanisms of compound-eyed organisms and exploring their behavioral laws. However, most studies rely on static light stimulation or preset virtual reality scenes, which makes it difficult to fully reflect the behavioral adaptability of compound-eyed organisms in dynamic natural environments. In addition, these methods usually lack real-time interactive solutions combined with actual behavioral data, which limits their scalability in research and practical applications.

[0004] The patent application number CN201410422437.4 is titled "A LED control system based on CAN bus and bumblebee flight control". The patent proposes a CAN bus-based LED array control system for providing visual stimulation to bumblebees to achieve precise control of their flight behavior. The system consists of a control terminal, a master node, and multiple slave nodes, each of which is responsible for independently controlling a column of LED arrays and communicating with the master node and other slave nodes through the CAN bus. The advantage of this architecture is that it achieves high refresh rate control of the LED array, thereby providing smooth and real-time visual stimulation. However, the complexity of the design increases the hardware cost of the system, and may face the problem of balancing resources and economy in practical applications. Summary of the invention

[0005] The present application provides a bio-robot hybrid control system and method based on visual perception to solve the above-mentioned technical problems.

[0006] In order to solve the above technical problems, a technical solution adopted in this application is: a biological robot hybrid control system based on visual perception, comprising:

[0007] The virtual scene component is used to provide the visual information in the simulated real scene to the organism in the form of optical flow stimulation through the LED dot matrix, and to create a realistic airflow environment for simulated flight;

[0008] The biological behavior acquisition module is located at the upper and lower sides of the biological body, wherein the biological behavior acquisition module is used to collect the behavioral response data of the biological body in the virtual scene component in real time;

[0009] The environmental information acquisition module is used to collect visual information and airflow information in the real environment, process and analyze them, and transmit them to the virtual scene component to construct the virtual scene;

[0010] A carrier, used to carry and move the environmental information collection module and complete the movement corresponding to the biological behavior;

[0011] The host computer is connected to the virtual scene component, the biological behavior acquisition module and the environmental information acquisition module respectively. The host computer is used to collect reaction data, visual information and airflow information, and perform data analysis to obtain integrated information.

[0012] Furthermore, the environmental information collection module includes:

[0013] A high-frequency camera is located in front of the carrier, wherein the high-frequency camera is used to obtain visual image information in front of the carrier;

[0014] The airflow sensor is located in front of the carrier, wherein the airflow sensor is used to collect ambient airflow information.

[0015] Furthermore, the high-frequency camera includes a fisheye lens for simulating the visual characteristics of compound-eye organisms to obtain visual image information.

[0016] Furthermore, the carrier includes:

[0017] A carrier controller, used to obtain the motion information collected by the environmental information collection module through the wireless communication module and control the carrier to complete the corresponding movement;

[0018] The wireless communication module is connected to the host computer, wherein the wireless communication module is used to wirelessly connect the host computer to the carrier controller and the carrier controller.

[0019] Furthermore, the virtual scene components include:

[0020] The ring-shaped LED dot matrix screen is composed of a green light-emitting diode array and has a ring-shaped geometric structure. The ring-shaped LED dot matrix screen is used to simulate the wide field of view characteristics of compound eye organisms and provide optical flow stimulation that is more in line with the visual perception characteristics;

[0021] An LED array controller is connected to the annular LED dot matrix screen, wherein the LED array controller is used to generate dot matrix display data for controlling the annular LED dot matrix screen;

[0022] An annular air flow generator for providing a controlled air flow;

[0023] An air pump controller connected to an air flow sensor, wherein the air pump controller is configured to control the annular air flow generator to generate a corresponding controlled air flow according to the air flow information and the movement direction information.

[0024] Furthermore, the biological behavior acquisition module includes:

[0025] A torque sensor for acquiring the torque signal of the carrier;

[0026] A high-speed camera for acquiring the behavior trajectory signal of the carrier.

[0027] One technical solution adopted in this application is: a hybrid control method for a bio-robot based on visual perception, including:

[0028] Based on the acquired visual information and air flow information, the host computer analyzes and obtains the dot matrix display data and air flow information required for the virtual scene;

[0029] Based on the dot matrix display data and the shown air flow information, drive the air pump controller and the LED array controller to construct a virtual scene;

[0030] Under the virtual scene, real-time acquisition of biological behavior response data;

[0031] Based on the response data, the host computer sends control instructions to the carrier controller through the wireless communication module to control the carrier to complete the movement corresponding to the biological behavior.

[0032] The beneficial effects of this application are: this application converts the real environment into an optical flow stimulus, provides it to the organism and acquires the behavior information of the organism to drive mechanical movement, realizing the function of controlling mechanical movement based on biological behavior. This application can place the carrier in a special environment or directly control the carrier to complete the specified movement through the host computer, providing a new method for the research of biological vision systems and memory functions. This application modularly combines the virtual scene, the biological behavior acquisition module, the environmental information acquisition module, the carrier, and the host computer, and uses the wireless communication module to achieve efficient data transmission and real-time control. The modular design greatly reduces the complexity of system development and maintenance, improves the scalability and function optimization ability of the system, and has high adaptability and flexibility. Brief Description of the Drawings

[0033] Figure 1 It is a schematic structural diagram of an embodiment of the hybrid control system for a bio-robot based on visual perception of this application;

[0034] Figure 2 It is a physical diagram of an embodiment of the hybrid control system for a bio-robot based on visual perception of this application;

[0035] Figure 3 It is a structural block diagram of an embodiment of the visual perception-based bio-robot hybrid control system of the present application;

[0036] Figure 4 is Figure 1 a structural block diagram of an embodiment of the virtual scene component in

[0037] Figure 5 is Figure 1 a structural block diagram of an embodiment of the biological behavior acquisition module in

[0038] Figure 6 is Figure 1 a structural block diagram of an embodiment of the environmental information acquisition module in

[0039] Figure 7 It is a schematic flowchart of an embodiment of the visual perception-based bio-robot hybrid control method of the present application;

[0040] Figure 8 It is the image processing of the direction of the body long axis of the compound eye organism of the present application;

[0041] Fig. 9 It is a relationship diagram of the body long axis and the initial angle of the compound eye organism of the present application;

[0042] Fig.10 It is a trajectory diagram of the carrier movement angle with respect to the rotation angle of the compound eye organism of the present application. Detailed implementation manners

[0043] To make the objectives, technical solutions and advantages of the present invention clearer, the following further describes the present invention in detail with reference to specific embodiments.

[0044] In the following description, many specific details are set forth to facilitate a thorough understanding of the present invention. However, the present invention may be implemented in other ways different from those described herein. Therefore, the present invention is not limited by the limitations of the specific embodiments disclosed below.

[0045] Refer to Figure 1-3 , Figure 1 which is a schematic structural diagram of an embodiment of the visual perception-based bio-robot hybrid control system of the present application. The system includes: a virtual scene component 1, a biological behavior acquisition module 2, an environmental information acquisition module 3, a carrier 4, a host computer 5, and a wireless communication module 42.

[0046] Among them, the virtual scene component 1 is used to provide the visual information in the simulated real scene to the creature 6 in the form of light flow stimulation through the LED dot matrix, and create a realistic airflow environment for simulated flight. The biological behavior acquisition module 2 is located above and below the creature 6; wherein the biological behavior acquisition module 2 is used to collect the behavioral response data of the creature 6 in the virtual scene component 1 in real time. The environmental information acquisition module 3 is used to collect visual information and airflow information in the real environment, and process and analyze and transmit it to the virtual scene component 1 to construct a virtual scene. The carrier 4 is used to carry the environmental information acquisition module and complete the movement corresponding to the behavior of the creature 6. The host computer 5 is connected to the virtual scene component 1, the biological behavior acquisition module 2 and the environmental information acquisition module 3 respectively, wherein the host computer 5 is used to collect reaction data, visual information and airflow information, and perform data analysis, obtain integrated information and send control instructions.

[0047] See also Figure 6 The environmental information acquisition module 3 includes a high-frequency camera 31 and an airflow sensor 32, which are used to collect visual and airflow information in the real environment and transmit it to the host computer 5 for processing and analysis, to obtain the motion information of the carrier 4, and then to construct a virtual scene.

[0048] The carrier 4 is provided with a carrier controller 41 and a wireless communication module 42, wherein the carrier controller 41 is used to obtain the motion information collected by the environmental information collection module 3 through the wireless communication module 42 and control the carrier to complete the corresponding movement. The wireless communication module 42 is connected to the host computer 5, wherein the wireless communication module 42 is used to wirelessly connect the host computer 5 with the above-mentioned environmental information collection module 3 and the carrier controller 41. The present application uses a Wi-Fi module. In other embodiments, the wireless communication module 42 can also use wireless modules such as Bluetooth.

[0049] In this embodiment, the carrier 4 is a small car or robot that can move in two dimensions, front, back, left, and right. The movement directions of the carrier 4 and the creature 6 correspond: when the creature 2 moves forward, the carrier 6 moves forward; when the creature 6 moves backward, the carrier 4 moves backward. The movement trajectory of the creature 6 is simulated by the movement trajectory of the carrier 4. In other embodiments, the carrier 4 can also be a multi-legged robot or a drone, and the specific choice is determined according to the movement mode of the creature 6.

[0050] The high-frequency camera 31 is equipped with a fisheye lens and has an ultra-wide-angle field of view, which can simulate the nearly 360-degree visual characteristics of compound-eye creatures. The airflow sensor 32 is placed in front of the carrier 4 and is responsible for collecting environmental airflow information to simulate a real flight environment.

[0051] See also Figure 4The virtual scene component 1 consists of an LED array controller 11, an annular LED dot matrix screen 12, an air pump controller 13 and an annular airflow generator 14, which is used to provide the visual information in the real scene to the creature 6 in the form of optical flow stimulation through the LED dot matrix, and create a realistic airflow environment for simulating the flying creature 6.

[0052] Among them, the annular LED dot matrix screen 12 is composed of a green light-emitting diode array. The annular geometric structure simulates the wide field of view characteristics of compound eye creatures, and can provide the creature 6 with optical flow stimulation that is more in line with its visual perception characteristics. At the same time, the design of the dot matrix LED meets the low-resolution perception requirements of the creature 6 for optical flow, and can also efficiently transmit environmental information. The LED array controller 11 (this application uses the MAX7219 chip driver) realizes multi-chip cascade control through the SPI bus, and can flexibly and efficiently drive the annular LED dot matrix screen 12. The system refresh rate is as high as 800 Hz, which is significantly higher than the visual refresh rate range of compound eye creatures such as bees, fruit flies, and mantises, ensuring the display smoothness and dynamic response performance of optical flow stimulation. The air pump controller 13 controls the annular airflow generator 14 to generate airflow according to the airflow size and the movement direction information of the carrier 4 collected by the airflow sensor 32 to simulate the real flight environment. The annular airflow generator 14 is used to provide controlled airflow to the carrier 4. The virtual scene also provides a place for exploring how the creature 6 uses airflow and visual signals to achieve direction perception and navigation.

[0053] See also Figure 5 The biological behavior acquisition module 2 is composed of a torque sensor 21 and a high-speed camera 22, which is used to collect the behavioral response data of the compound eye organism 6 under the stimulation of optical flow in real time. Among them, the organism 6 is fixed on the torque sensor 21 by tethered flight, and the high-speed camera 22 is placed below it. Among them, the torque sensor 21 collects torque signals, and the high-speed camera 22 captures the behavior trajectory. The two work together to capture the behavioral changes of the organism 6 under the stimulation of optical flow. The collected data is transmitted to the host computer 5 for processing and analysis, which is used to reveal the perception mechanism of the organism 6 to the optical flow visual stimulation and realize the control of the mechanical motion module 6.

[0054] See also Figure 3, the host computer 5 is connected to the virtual scene component 1, the biological behavior acquisition module 2 and the environmental information acquisition module 3 respectively, wherein the host computer 5 is used to collect reaction data, visual information and airflow information, and perform data analysis to obtain integrated information. Among them, the host computer 5 is used to receive and process the information of the environmental information acquisition module 3 and the biological 6 motion acquisition module to construct a virtual scene and control motion. After receiving the image information of the high-frequency camera 31, the host computer 5 processes the image into the display information of the annular LED dot matrix screen 12, and then drives the LED array controller 11 to control the display of the annular LED dot matrix screen 12 to stimulate the biological 6 through optical flow. After receiving the data of the airflow sensor 32 and the mechanical motion information, the host computer 5 controls the air pump controller 13 to drive the annular airflow generator 14 according to the size and direction of the airflow to simulate the real flight environment. After receiving the data of the biological 6 motion acquisition module, the host computer 5 analyzes the behavioral response of the biological 6 under the stimulation of optical flow, and generates control instructions based on the analysis results. The instructions are transmitted to the mechanical motion module 6 through Wi-Fi and other methods to drive it to complete the corresponding motion control.

[0055] In addition, based on the above system, it can also be used to study the mechanism of organism 6: the carrier 4 can be placed in a special environment or the specified movement can be completed by the host computer 5 to study the perception and behavior mechanism of organism 6 in a specific environment. For example: the carrier 4 is placed in an environment containing obstacles to study how organism 6 avoids obstacles. The carrier 4 can also be directly controlled to pass through obstacles in different ways of movement to study the behavior of organism 6. For another example: the organism 6 can also be controlled to pass through the same scene multiple times to study its memory mechanism. This system not only provides new ideas for the development of bionic robots, but also provides important technical support for the study of visual navigation, memory mechanism and decision-making process of compound eye organism 6.

[0056] See also Figure 7 , Figure 7 1 is a flow chart of an embodiment of a biological robot hybrid control method based on visual perception of the present application. The method comprises:

[0057] Step S1. Based on the collected visual information and airflow information, the host computer analyzes and obtains the dot matrix display data and airflow information required for the virtual scene.

[0058] Specifically, the visual information and airflow information collected by the environmental information collection module 3 and the motion information of the carrier 4 are wirelessly transmitted to the host computer 5 through the Wi-Fi module for processing and displaying dot matrix data and airflow information required for the virtual scene.

[0059] Step S2. Based on the dot matrix display data and airflow information, drive the air pump controller and the LED array controller to construct a virtual scene.

[0060] Specifically, the host computer 5 drives the air pump controller 13 and the LED array controller 11 according to the parsed dot matrix display data and airflow information to build a virtual scene.

[0061] Step S3. In a virtual scene, collect the behavioral response data of the organism in real time.

[0062] Specifically, the motion acquisition module of the organism 6 uses the torque sensor 21 and the high-speed camera 22 to collect the behavior changes of the organism 6 in the virtual scene in real time, and transmits the collected data to the host computer 5 for analysis and processing to obtain reaction data.

[0063] See also Figure 8-9 In this embodiment, the rotation angle of the compound eye organism and the follow-up video of the carrier 4 are obtained, and the direction of the long axis of the body of the compound eye organism 6 is extracted using digital image processing technology.

[0064] The specific steps include:

[0065] (1) Video reading and preprocessing: This includes graying and binarizing the video and using morphological operations such as erosion and dilation to remove noise and obtain the insect body area.

[0066] (2) Ellipse fitting and angle calculation: The region of interest (specific name of the object) in the image is formed into the largest connected region, and an ellipse is fitted to calculate the direction of the insect's body's long axis. In the process of calculating the direction angle, the angle of the insect's body's long axis in the initial state of the experiment is used as the initial angle (i.e., 0° angle), which corresponds to the angle of the subsequent car's forward direction. (In the compound eye biological posture detection video, the horizontal left-90° direction is used as the initial direction)

[0067] (3) Angle smoothing and correction: Smooth angle changes and avoid angle jumps.

[0068] (4) Visualization and real-time display: Fig. 9 The long axis of the compound eye organism's body is adjusted, and the corresponding relationship with the angle is obtained using digital image processing technology, and the angle, center of mass and long axis direction of the insect are marked in real time (that is, the result of digital image processing for each frame of the image).

[0069] For the specific process of obtaining the adjustment angle, refer to Fig.10 The video was shot with an infrared camera (mainly because the use of infrared fill light does not affect the visual information of compound-eyed creatures - for bees, infrared light is similar to black, but it is beneficial to the camera's imaging. The camera can shoot at 60fps. If you need to obtain subtle movements, such as movements of the hind legs, antennae and head, you can use a higher resolution and higher rate camera for shooting).

[0070] According to the above digital image processing process, the real-time deflection angle θ of compound eye organisms during the visual stimulation experiment can be obtained, and the corresponding deflection angular velocity can also be obtained according to the frame rate relationship. .

[0071] In the developed system, in order to regulate and control the movement of the carrier through the behavioral information of compound eye organisms, it is necessary to establish a mutual correspondence to calculate the movement information of the carrier: , , , .

[0072] First, in the initial state of the system, the initial angle (i.e., 0°) of the long axis of the insect body is matched with the default forward direction of the carrier 4, both of which are initial angles. At the same time, the movement speed of the carrier 4 is preset to be constant and , in the Cartesian coordinate system, the velocity component of the carrier 4 in the X-axis direction is ; Velocity component in the Y direction ; According to the carrier control system and biological behavior characteristics, the corresponding relationship between the biological deflection angle and the carrier angle is calibrated through experiments:

[0073] The rotation angle of carrier 4 is: δ = k 1 ×θ (k 1 is a constant)

[0074] The angular velocity of carrier 4 is: ω 2 = k 2 ×ω 1 (k 2 is a constant)

[0075] The carrier 4 in this embodiment is driven by three omnidirectional wheels symmetrically placed at an angle of 120°, which can realize flexible movements such as translation, rotation, forward and backward movement, and angle change. The communication module receives the relevant angular information of the compound eye creature, and the carrier controller combines the preset speed to Analyze and obtain relevant parameters , , , .

[0076] When the relevant data of compound eye biological motion is obtained and converted into the motion parameters of the carrier, the target speed of each of the three wheels a, b, and c of the carrier can be calculated according to the following formula:

[0077]

[0078] in: is the target speed of round A; is the target speed of round B; is the target speed of wheel C; R is the rotation radius of the omnidirectional wheel; α is the angle between the wheel axle and the X axis, and in this structure α is π / 6. By driving each wheel to achieve the predetermined target speed through the carrier controller 41, the movement of the carrier 4 based on the behavioral feedback control of the organism 2 can be realized.

[0079] Step S4. Based on the reaction data, the host computer sends a control instruction to the carrier controller through the wireless communication module to control the carrier to complete the movement corresponding to the biological behavior.

[0080] Specifically, the host computer 5 analyzes and processes the information transmitted by the biological motion acquisition module, generates control instructions, and transmits them to the carrier 4 controller via Wi-Fi to achieve dynamic control of the mechanical vehicle, thereby completing the robot motion control based on the biological 6 behavior.

[0081] This application converts the real environment into optical flow stimulation, provides it to the organism 6, and collects the behavioral information of the organism 6 to drive mechanical movement, thereby realizing the function of inhibiting the behavior of the organism 6 and controlling the mechanical movement. This application places the carrier 4 in a special environment or directly controls the carrier 4 through the host computer 5 to complete the specified movement, providing a new method for the study of the visual system and memory function of the organism 6. The MAX7219 driver chip and SPI bus cascade technology are used to achieve flexible expansion and efficient control of the dot matrix unit, and ensure the real-time and scalability of the dynamic optical flow stimulation of the dot matrix module. The virtual scene, biological behavior acquisition module 2, environmental information acquisition module 3, mechanical motion module 6 and host computer 55 are organically combined in a modular way, and the Wi-Fi module is used to achieve efficient data transmission and real-time control. The modular design greatly reduces the complexity of system development and maintenance, improves the system scalability and function optimization capabilities, and has high adaptability and flexibility.

[0082] The above description is only an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A bio-robot hybrid control system based on visual perception, characterized in that: include: The virtual scene component is used to provide the visual information in the simulated real scene to the organism in the form of optical flow stimulation through the LED dot matrix, and to create a realistic airflow environment for simulated flight; A biological behavior acquisition module, located at the upper and lower sides of the biological, wherein the biological behavior acquisition module is used to collect the behavioral response data of the biological in the virtual scene component in real time; An environmental information acquisition module is used to acquire visual information and airflow information in a real environment, process and analyze the information, and transmit the information to the virtual scene component to construct a virtual scene; A carrier, used to carry the environmental information acquisition module and complete the movement corresponding to the biological behavior; The host computer is connected to the virtual scene component and the biological behavior acquisition module respectively, wherein the host computer is used to collect the behavior data, the visual information and the airflow information, and perform data analysis to obtain integrated information.

2. The system according to claim 1, characterized in that The environmental information collection module includes: A high-frequency camera is located in front of the carrier, wherein the high-frequency camera is used to obtain visual image information in front of the carrier; An airflow sensor is located in front of the carrier, wherein the airflow sensor is used to collect environmental airflow information.

3. The system according to claim 2, characterized in that The high-frequency camera includes a fisheye lens for simulating the visual characteristics of compound-eye organisms to obtain visual image information.

4. The system according to claim 2, characterized in that The carrier comprises: A carrier controller, used to obtain the motion information collected by the environmental information collection module through the wireless communication module and control the carrier to complete the corresponding movement; A wireless communication module is connected to the host computer, wherein the wireless communication module is used to wirelessly connect the host computer to the carrier controller and the carrier controller.

5. The system according to claim 2, characterized in that The virtual scene component includes: The annular LED dot matrix screen is composed of a green light-emitting diode array and has an annular geometric structure, wherein the annular LED dot matrix screen is used to simulate the wide field of view characteristics of compound eye organisms and provide optical flow stimulation that is more in line with visual perception characteristics; An LED array controller connected to the annular LED dot matrix screen, wherein the LED array controller is used to generate dot matrix display data for controlling the annular LED dot matrix screen; an annular airflow generator for providing controlled airflow; An air pump controller is connected to the airflow sensor, wherein the air pump controller is used to control the annular airflow generator to generate a corresponding controlled airflow according to the airflow information and the movement direction information.

6. The system according to claim 1, characterized in that The biological behavior acquisition module includes: A torque sensor, used for collecting a torque signal of the carrier; A high-speed camera is used to collect the behavior trajectory signal of the carrier.

7. A biological robot hybrid control method based on visual perception, characterized in that: include: Based on the collected visual information and airflow information, the host computer analyzes and obtains the dot matrix display data and airflow information required for the virtual scene; Based on the dot matrix display data and the airflow information shown, driving the air pump controller and the LED array controller to construct a virtual scene; In the virtual scene, collecting the behavioral response data of the organism in real time; Based on the reaction data, the host computer sends a control instruction to the carrier through a wireless communication module to control the carrier to complete the movement corresponding to the biological behavior.

Images