A bio-robot hybrid control system and method based on visual perception

Abstract

The present application discloses a bio-robot hybrid control system and method based on visual perception, including a virtual scene component, a biological behavior acquisition module, a carrier, a host computer, an environmental information acquisition module, and a wireless communication module. The present application realizes the function of controlling mechanical motion based on biological behavior by converting the real environment into optical flow stimulation, providing it to the organism and collecting the biological behavior information to drive mechanical motion. The present application can place the carrier in a special environment or directly control the carrier to complete a specified motion through the host computer to study the mechanism of the organism. The present application modularly combines the virtual scene, the biological behavior acquisition module, the environmental information acquisition module, the carrier and the host computer, and uses the wireless communication module to achieve efficient data transmission and real-time control. The modular design greatly reduces the complexity of system development and maintenance, improves the system scalability and function optimization capabilities, and is highly adaptable and flexible.

Description

Technical Field

[0001] The present application belongs to the field of insect-machine hybrid control technology, and specifically relates to a bio-robot hybrid control system and method based on visual perception. Background Art

[0002] Insect-robot hybrid control systems (Cyborg Insects) are a cutting-edge, interdisciplinary research field that integrates insect biology, robotics, and control engineering. This field simulates and leverages insect biological mechanisms to control and optimize robotic behavior, leveraging their strengths in perception, locomotion, and environmental adaptability. This not only advances fundamental research on insect behavior and neural mechanisms but also provides important technical support and application prospects for the development of bio-inspired robotics, intelligent control technologies, and autonomous systems.

[0003] Organisms with compound eyes typically possess visual characteristics such as a wide field of view, low resolution, and sensitivity to optical flow. These unique visual properties enable them to efficiently navigate and make behavioral decisions in complex environments. Existing research has focused on simulating the visual mechanisms of organisms with compound eyes and exploring their behavioral patterns. However, most studies rely on static light stimulation or preset virtual reality scenes, making it difficult to fully reflect the behavioral adaptability of organisms with compound eyes in dynamic natural environments. Furthermore, these methods often lack real-time interactive solutions that can be combined with actual behavioral data, limiting their scalability in research and practical applications.

[0004] Patent application number CN201410422437.4, titled "A CAN-bus-based LED control system and bumblebee flight control," proposes a CAN-bus-based LED array control system for providing visual stimulation to bumblebees and enabling precise control of their flight behavior. The system consists of a control terminal, a master node, and multiple slave nodes. Each slave node independently controls a column of LED arrays and communicates with the master node and other slave nodes via the CAN bus. This architecture offers the advantage of high refresh rate control of the LED array, providing smooth, real-time visual stimulation. However, the complexity of this design increases the system's hardware cost, potentially posing a resource-efficiency trade-off in practical applications. Summary of the Invention

[0005] The present application provides a bio-robot hybrid control system and method based on visual perception to solve the above-mentioned technical problems.

[0006] To solve the above technical problems, the present application adopts a technical solution: a bio-robot hybrid control system based on visual perception, comprising:

[0007] The virtual scene component is used to provide the visual information of the simulated real scene to the organism in the form of optical flow stimulation through the LED dot matrix, and to create a realistic airflow environment for simulated flight;

[0008] Biological behavior acquisition modules are located on the upper and lower sides of the organism, wherein the biological behavior acquisition modules are used to collect the behavioral response data of the organism in the virtual scene component in real time;

[0009] The environmental information acquisition module is used to collect visual information and airflow information in the real environment, process and analyze it, and transmit it to the virtual scene component to construct the virtual scene;

[0010] The carrier is used to carry and move the environmental information acquisition module and complete the movement corresponding to the biological behavior;

[0011] The host computer is connected to the virtual scene component, the biological behavior acquisition module and the environmental information acquisition module respectively. The host computer is used to collect reaction data, visual information and airflow information, and perform data analysis to obtain integrated information.

[0012] Furthermore, the environmental information collection module includes:

[0013] A high-frequency camera is located in front of the carrier, wherein the high-frequency camera is used to obtain visual image information in front of the carrier;

[0014] The airflow sensor is located in front of the carrier, wherein the airflow sensor is used to collect ambient airflow information.

[0015] Furthermore, the high-frequency camera includes a fisheye lens for simulating the visual characteristics of compound-eye organisms to obtain visual image information.

[0016] Furthermore, the carrier includes:

[0017] The carrier controller is used to obtain the motion information collected by the environmental information collection module through the wireless communication module and control the carrier to complete the corresponding movement;

[0018] The wireless communication module is connected to the host computer, wherein the wireless communication module is used to wirelessly connect the host computer and the carrier controller and the carrier controller.

[0019] Furthermore, the virtual scene components include:

[0020] The ring-shaped LED dot matrix screen is composed of a green light-emitting diode array and has a ring-shaped geometric structure. The ring-shaped LED dot matrix screen is used to simulate the wide field of view characteristics of compound eye organisms and provide optical flow stimulation that is more in line with visual perception characteristics.

[0021] An LED array controller is connected to the annular LED dot matrix screen, wherein the LED array controller is used to generate dot matrix display data for controlling the annular LED dot matrix screen;

[0022] an annular airflow generator for providing controlled airflow;

[0023] The air pump controller is connected to the air flow sensor, wherein the air pump controller is used to control the annular air flow generator to generate a corresponding controlled air flow according to the air flow information and the movement direction information.

[0024] Furthermore, the bio-behavior collection module includes:

[0025] Torque sensor, used to collect the torque signal of the carrier;

[0026] High-speed camera, used to collect the behavior trajectory signal of the carrier.

[0027] A technical solution adopted in this application is: a bio-robot hybrid control method based on visual perception, including:

[0028] Based on the collected visual information and airflow information, the host computer analyzes and obtains the dot matrix display data and airflow information required for the virtual scene;

[0029] Based on the dot matrix display data and the airflow information shown, the air pump controller and the LED array controller are driven to construct a virtual scene;

[0030] Collect biological behavioral response data in real time in a virtual scene;

[0031] Based on the response data, the host computer sends control instructions to the carrier controller through the wireless communication module to control the carrier to complete the movement corresponding to the biological behavior.

[0032] The beneficial effects of the present application are as follows: the present application realizes the function of controlling mechanical motion based on biological behavior by converting the real environment into optical flow stimulation, providing it to the organism and collecting the biological behavior information to drive mechanical motion. The present application can place the carrier in a special environment or directly control the carrier through the host computer to complete the specified motion, providing a new method for the study of biological visual systems and memory functions. The present application modularly combines virtual scenes, biological behavior acquisition modules, environmental information acquisition modules, carriers and host computers, and uses wireless communication modules to achieve efficient data transmission and real-time control. The modular design greatly reduces the complexity of system development and maintenance, improves the system scalability and function optimization capabilities, and is highly adaptable and flexible. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a structural diagram of an embodiment of a bio-robot hybrid control system based on visual perception of the present application;

[0034] Figure 2 This is a physical diagram of an embodiment of the bio-robotic hybrid control system based on visual perception of the present application;

[0035] Figure 3 This is a structural block diagram of an embodiment of a bio-robot hybrid control system based on visual perception of the present application;

[0036] Figure 4 yes Figure 1 A structural block diagram of an embodiment of a virtual scene component in FIG.

[0037] Figure 5 yes Figure 1 A structural block diagram of an embodiment of a biological behavior acquisition module in FIG.

[0038] Figure 6 yes Figure 1 A structural block diagram of an embodiment of an environmental information acquisition module in FIG.

[0039] Figure 7 This is a flow chart of an embodiment of a bio-robot hybrid control method based on visual perception of the present application;

[0040] Figure 8 It is the image processing of the direction of the long axis of the body of the compound eye organism of the present application;

[0041] Figure 9 This is a diagram showing the relationship between the long axis of the compound eye organism and the initial angle;

[0042] Figure 10 This is a trajectory diagram of the carrier movement angle of the present application following the rotation angle of the compound eye organism. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to specific embodiments.

[0044] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0045] See Figure 1-3 , Figure 1 This is a structural diagram of an embodiment of the bio-robot hybrid control system based on visual perception of the present application. The system includes: a virtual scene component 1, a biological behavior acquisition module 2, an environmental information acquisition module 3, a carrier 4, a host computer 5 and a wireless communication module 42.

[0046] Among them, the virtual scene component 1 is used to provide the visual information in the simulated real scene to the creature 6 in the form of optical flow stimulation through the LED dot matrix, and create a realistic airflow environment for simulated flight. The biological behavior acquisition module 2 is located above and below the creature 6; wherein, the biological behavior acquisition module 2 is used to collect the behavioral response data of the creature 6 in the virtual scene component 1 in real time. The environmental information acquisition module 3 is used to collect visual information and airflow information in the real environment, and process and analyze and transmit it to the virtual scene component 1 to construct a virtual scene. The carrier 4 is used to carry the environmental information acquisition module and complete the movement corresponding to the behavior of the creature 6. The host computer 5 is connected to the virtual scene component 1, the biological behavior acquisition module 2 and the environmental information acquisition module 3 respectively, wherein the host computer 5 is used to collect reaction data, visual information and airflow information, and perform data analysis, obtain integrated information and send control instructions.

[0047] See Figure 6 The environmental information acquisition module 3 includes a high-frequency camera 31 and an airflow sensor 32, which are used to collect visual and airflow information in the real environment and transmit it to the host computer 5 for processing and analysis, to obtain the motion information of the carrier 4, and then to construct a virtual scene.

[0048] The carrier 4 is provided with a carrier controller 41 and a wireless communication module 42. The carrier controller 41 is used to obtain the motion information collected by the environmental information acquisition module 3 through the wireless communication module 42 and control the carrier to complete the corresponding movement. The wireless communication module 42 is connected to the host computer 5, wherein the wireless communication module 42 is used to wirelessly connect the host computer 5 to the above-mentioned environmental information acquisition module 3 and the carrier controller 41. In this application, a Wi-Fi module is used. In other embodiments, the wireless communication module 42 can also use a wireless module such as Bluetooth.

[0049] In this embodiment, carrier 4 is a small vehicle or robot capable of two-dimensional movement, forward, backward, and left and right. The movement directions of carrier 4 and organism 6 correspond: if organism 2 moves forward, carrier 6 moves forward; if organism 6 moves backward, carrier 4 moves backward. The trajectory of organism 6 is simulated by the motion of carrier 4. In other embodiments, carrier 4 can also be a multi-legged robot or drone, the specific choice being determined based on the movement pattern of organism 6.

[0050] The high-frequency camera 31 is equipped with a fisheye lens, which has an ultra-wide-angle field of view and can simulate the nearly 360-degree vision characteristics of compound-eye creatures. The airflow sensor 32 is placed in front of the carrier 4 and is responsible for collecting ambient airflow information to simulate a real flight environment.

[0051] See Figure 4The virtual scene component 1 consists of an LED array controller 11, an annular LED dot matrix screen 12, an air pump controller 13 and an annular airflow generator 14, which is used to provide the visual information in the real scene to the creature 6 in the form of optical flow stimulation through the LED dot matrix, and create a realistic airflow environment for the simulated flying creature 6.

[0052] The annular LED dot matrix screen 12 is composed of an array of green light-emitting diodes. Its circular geometry simulates the wide field of view of compound-eyed creatures, providing optic flow stimulation more consistent with the creature's visual perception characteristics. The dot matrix LED design also meets the creature's low-resolution optical flow perception requirements while efficiently transmitting environmental information. The LED array controller 11 (in this application, a MAX7219 chip driver) utilizes an SPI bus to achieve multi-chip cascade control, enabling flexible and efficient driving of the annular LED dot matrix screen 12. The system boasts a refresh rate of up to 800 Hz, significantly exceeding the visual refresh rate range of compound-eyed creatures such as bees, fruit flies, and mantises, ensuring smooth and dynamic responsive display of optic flow stimulation. The air pump controller 13 controls the annular airflow generator 14 based on the airflow magnitude and motion direction of the vehicle 4, as measured by the airflow sensor 32, to simulate a realistic flight environment. The annular airflow generator 14 provides controlled airflow to the vehicle 4. The virtual scene also provides a venue for exploring how the creature 6 uses airflow in combination with visual signals to achieve directional perception and navigation.

[0053] See Figure 5 The biological behavior acquisition module 2 is composed of a torque sensor 21 and a high-speed camera 22, which is used to collect behavioral response data of the compound eye organism 6 under optical flow stimulation in real time. The organism 6 is fixed on the torque sensor 21 by tethered flight, and the high-speed camera 22 is placed below it. The torque sensor 21 collects torque signals and the high-speed camera 22 captures the behavioral trajectory. The two work together to capture the behavioral changes of the organism 6 under optical flow stimulation. The collected data is transmitted to the host computer 5 for processing and analysis, which is used to reveal the perception mechanism of the organism 6 to the optical flow visual stimulation and realize the control of the mechanical motion module 6.

[0054] See Figure 3The host computer 5 is connected to the virtual scene component 1, the biological behavior acquisition module 2, and the environmental information acquisition module 3, respectively. The host computer 5 is used to collect reaction data, visual information, and airflow information, and perform data analysis to obtain integrated information. Specifically, the host computer 5 receives and processes information from the environmental information acquisition module 3 and the biological motion acquisition module 6 to construct the virtual scene and control movement. After receiving image information from the high-frequency camera 31, the host computer 5 processes the image into display information for the annular LED dot matrix screen 12. It then drives the LED array controller 11 to control the display of the annular LED dot matrix screen 12, thereby stimulating the biological 6 with optical flow. After receiving data from the airflow sensor 32 and mechanical motion information, the host computer 5 controls the air pump controller 13 to drive the annular airflow generator 14 based on the airflow size and direction to simulate a real flight environment. After receiving data from the biological 6 motion acquisition module, the host computer 5 analyzes the biological 6's behavioral responses to the optical flow stimulation and generates control instructions based on the analysis results. These instructions are transmitted to the mechanical motion module 6 via Wi-Fi or other means, driving it to complete the corresponding motion control.

[0055] Furthermore, this system can be used to study the mechanisms of organisms 6: By placing carrier 4 in a specific environment or requiring a host computer 5 to perform a specified motion, the perception and behavior of organisms 6 in specific environments can be studied. For example, carrier 4 can be placed in an environment containing obstacles to study how organisms 6 avoid them. Carrier 4 can also be directly controlled to navigate obstacles using different motion patterns to study the behavior of organisms 6. Another example is the ability to control organisms 6 to repeatedly navigate the same scene to study their memory mechanisms. This system not only provides new insights into the development of biomimetic robots but also offers important technical support for the study of visual navigation, memory mechanisms, and decision-making processes in compound-eyed organisms 6.

[0056] See Figure 7 , Figure 7 This is a flow chart of an embodiment of the bio-robot hybrid control method based on visual perception of the present application. The method includes:

[0057] Step S1. Based on the collected visual information and airflow information, the host computer analyzes and obtains the dot matrix display data and airflow information required for the virtual scene.

[0058] Specifically, the visual information and airflow information collected by the environmental information collection module 3 and the motion information of the carrier 4 are wirelessly transmitted to the host computer 5 through the Wi-Fi module for processing and forming dot matrix display data and airflow information required for the virtual scene.

[0059] Step S2. Based on the dot matrix display data and airflow information, drive the air pump controller and the LED array controller to construct a virtual scene.

[0060] Specifically, the host computer 5 drives the air pump controller 13 and the LED array controller 11 according to the parsed dot matrix display data and airflow information to build a virtual scene.

[0061] Step S3: In the virtual scene, collect the behavioral response data of the organism in real time.

[0062] Specifically, the motion acquisition module of the organism 6 uses the torque sensor 21 and the high-speed camera 22 to collect the behavioral changes of the organism 6 in the virtual scene in real time, and transmits the collected data to the host computer 5 for analysis and processing to obtain reaction data.

[0063] See Figure 8-9 In this embodiment, the rotation angle of the compound-eye creature and the follow-up video of the carrier 4 are obtained, and the direction of the long axis of the body of the compound-eye creature 6 is extracted using digital image processing technology.

[0064] The specific steps include:

[0065] (1) Video reading and preprocessing: This includes grayscale and binarization of the video and the use of morphological operations such as corrosion and dilation to remove noise and obtain the insect body area.

[0066] (2) Ellipse fitting and angle calculation: The region of interest (specifically the object) in the image is formed into the largest connected region, and an ellipse is fitted to calculate the orientation of the insect's body's long axis. In the process of calculating the orientation angle, the angle of the insect's body's long axis at the initial state of the experiment is used as the initial angle (i.e., 0° angle), which corresponds to the angle of the subsequent vehicle's forward direction. (In the compound eye biological posture detection video, the horizontal left-90° direction is used as the initial direction)

[0067] (3) Angle smoothing and correction: Smooth angle changes and avoid angle jumps.

[0068] (4) Visualization and real-time display: Figure 9 The long axis of the compound eye organism's body is adjusted, and the corresponding relationship with the angle is obtained using digital image processing technology, and the angle, center of mass and long axis direction of the ellipse of the insect are displayed in real time (that is, the result of digital image processing for each frame of the image).

[0069] For the specific process of obtaining the adjustment angle, refer to Figure 10 The video was shot with an infrared camera (mainly because the use of infrared fill light does not affect the visual information of compound-eyed creatures - for bees, infrared light is like black, but it is conducive to camera imaging. The camera can shoot at 60fps. If you need to obtain subtle movements, such as movements of the hind legs, antennae and head, you can use a higher resolution and higher speed camera for shooting).

[0070] According to the above digital image processing process, the real-time deflection angle θ of compound eye organisms during the visual stimulation experiment can be obtained, and the corresponding deflection angular velocity can also be obtained according to the frame rate relationship. .

[0071] In the developed system, in order to regulate and control the movement of the carrier through the behavioral information of compound eye organisms, it is necessary to establish a mutual correspondence to calculate the movement information of the carrier: , , , .

[0072] First, in the initial state of the system, the initial angle of the long axis of the insect body (i.e., 0°) is matched with the default forward direction of the carrier 4, both of which are initial angles. At the same time, the movement speed of the carrier 4 is preset to be constant and , in the Cartesian coordinate system, the velocity component of the carrier 4 in the X-axis direction is ; Velocity component in the Y-axis direction According to the carrier control system and biological behavior characteristics, the corresponding relationship between the biological deflection angle and the carrier angle is calibrated through experiments:

[0073] The rotation angle of the carrier 4 is: δ = k1×θ (k1 is a constant)

[0074] The angular velocity of carrier 4 is: ω2 = k2×ω1 (k2 is a constant)

[0075] The carrier 4 in this embodiment is driven by three omnidirectional wheels placed symmetrically at an angle of 120 degrees, which can achieve flexible movements such as translation, rotation, forward and backward, and angle changes. The communication module receives the relevant angular information of the compound eye creature and uses the carrier controller combined with the preset speed to Parse and obtain relevant parameters , , , .

[0076] Once the relevant data on compound eye movement is obtained and converted into the motion parameters of the carrier, the target speeds of the three wheels a, b, and c of the carrier can be calculated according to the following formula:

[0077]

[0078] in: is the target speed of round A; is the target speed of round B; is the target speed of wheel C; R is the rotation radius of the omnidirectional wheel; α is the angle between the wheel axle and the X-axis, which in this configuration is π / 6. By driving each wheel to achieve a predetermined target speed through vehicle controller 41, vehicle 4 can be controlled based on the behavioral feedback of organism 2.

[0079] Step S4. Based on the response data, the host computer sends a control instruction to the carrier controller through the wireless communication module to control the carrier to complete the movement corresponding to the biological behavior.

[0080] Specifically, the host computer 5 analyzes and processes the information input by the biological motion acquisition module, generates control instructions, and transmits them to the carrier 4 controller via Wi-Fi to achieve dynamic control of the mechanical vehicle, thereby completing the robot motion control based on the biological 6 behavior.

[0081] This application converts the real environment into optical flow stimulation, provides it to the organism 6, and collects the behavioral information of the organism 6 to drive mechanical movement, thereby achieving the function of suppressing the behavior of the organism 6 and controlling the mechanical movement. This application places the carrier 4 in a special environment or directly controls the carrier 4 through the host computer 5 to complete the specified movement, providing a new method for the study of the visual system and memory function of the organism 6. The MAX7219 driver chip and SPI bus cascade technology are used to achieve flexible expansion and efficient control of the dot matrix unit, ensuring the real-time and scalability of the dynamic optical flow stimulation of the dot matrix module. The virtual scene, biological behavior acquisition module 2, environmental information acquisition module 3, mechanical motion module 6 and host computer 55 are organically combined in a modular manner, and the Wi-Fi module is used to achieve efficient data transmission and real-time control. The modular design greatly reduces the complexity of system development and maintenance, improves the system scalability and function optimization capabilities, and is highly adaptable and flexible.

[0082] The above description is merely an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A bio-robot hybrid control system based on visual perception, characterized in that: include: The virtual scene component is used to provide the visual information of the simulated real scene to the organism in the form of optical flow stimulation through the LED dot matrix, and to create a realistic airflow environment for simulated flight; Biological behavior acquisition modules are located on the upper and lower sides of the organism, wherein the biological behavior acquisition modules are used to collect the behavioral response data of the organism in the virtual scene component in real time; An environmental information acquisition module is used to collect visual information and airflow information in the real environment, process and analyze the information, and transmit it to the virtual scene component to construct a virtual scene; A carrier, used to carry the environmental information acquisition module and complete the movement corresponding to the biological behavior; The host computer is connected to the virtual scene component and the biological behavior acquisition module respectively, wherein the host computer is used to collect the behavior data, the visual information and the airflow information, and perform data analysis to obtain integrated information.

2. The system according to claim 1, wherein: The environmental information collection module includes: a high-frequency camera, located in front of the carrier, wherein the high-frequency camera is used to obtain visual image information in front of the carrier; An airflow sensor is located in front of the carrier, wherein the airflow sensor is used to collect ambient airflow information.

3. The system according to claim 2, characterized in that The high-frequency camera includes a fisheye lens for simulating the visual characteristics of compound-eye organisms to obtain visual image information.

4. The system according to claim 2, wherein: The carrier comprises: A carrier controller, configured to obtain the motion information collected by the environmental information collection module through a wireless communication module and control the carrier to complete corresponding movement; A wireless communication module is connected to the host computer, wherein the wireless communication module is used to wirelessly connect the host computer and the carrier controller and the carrier controller.

5. The system according to claim 2, wherein: The virtual scene component includes: The annular LED dot matrix screen is composed of an array of green light-emitting diodes and has a circular geometric structure. The annular LED dot matrix screen is used to simulate the wide field of view characteristics of compound eye organisms and provide optical flow stimulation that is more in line with visual perception characteristics. An LED array controller connected to the annular LED dot matrix screen, wherein the LED array controller is used to generate dot matrix display data for controlling the annular LED dot matrix screen; an annular airflow generator for providing controlled airflow; An air pump controller is connected to the airflow sensor, wherein the air pump controller is used to control the annular airflow generator to generate a corresponding controlled airflow according to the airflow information and the movement direction information.

6. The system according to claim 1, wherein: The biological behavior acquisition module includes: A torque sensor, used for collecting a torque signal of the carrier; A high-speed camera is used to collect the behavior trajectory signal of the carrier.

7. A control method based on the system according to any one of claims 1 to 6, characterized in that: include: Based on the collected visual information and airflow information, the host computer analyzes and obtains the dot matrix display data and airflow information required for the virtual scene; Based on the dot matrix display data and the airflow information, driving the air pump controller and the LED array controller to construct a virtual scene; In the virtual scene, collecting behavioral response data of the organism in real time; Based on the reaction data, the host computer sends a control instruction to the carrier through the wireless communication module to control the carrier to complete the movement corresponding to the biological behavior.

Images