Motion control system and method based on multiple bionic visual neurons

Through a variety of bionic visual neuron systems combining lobular giant motion detector LGMD, directional selective neuron DSN and tri-state hysteresis neuron TSHN, the misjudgment problem of single neurons in complex scenarios is solved, and the ground microautonomy is achieved Mobile robots’ efficient obstacle avoidance capabilities.

CN119007285APending Publication Date: 2024-11-22GUANGZHOU UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411043106.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing visual motion trends to identify neural networks have many false alarms in complex and changing light scenes. Single neurons respond inaccurately to multiple stimuli. The calculation direction selective neurons will introduce process noise, resulting in micro-autonomous movement on the ground. The obstacle avoidance task is inefficient during robot movement.

Method used

A variety of bionic visual neuron systems are adopted, including the lobular giant motion detector LGMD and direction-selective neuron DSN, and the obstruction warning information is calculated through LGMD, the DSN computer robot translates the direction, and the tri-state lag neuron TSHN is used to schedule motion trends. Combined with a monopole recursive filter to reduce noise, optimize neuronal output priorities for improved robustness.

Benefits of technology

It realizes the rapid extraction and integration of multiple visual motion trends in a dynamic environment, automatically triggers behavioral responses, simple structure, efficient calculations, strong robustness, effectively completes obstacle avoidance tasks of ground micro autonomous mobile robots, reduces memory usage, and saves resources. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007285A_ABST
    Figure CN119007285A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a motion control system and method based on multiple bionic visual neurons, and the system comprises an image collection module which is used for collecting an environment video in real time, and obtaining image information; the visual detection module is used for calculating and outputting a visual motion trend through LGMD and DSN according to the acquired image information, and the visual motion trend comprises early warning information and a translation direction of an obstacle object; and the motion execution module comprises a three-state hysteretic neuron TSHN and is used for carrying out scheduling through the set neuron output priority and the visual motion trend, outputting corresponding motion instructions in sequence and controlling the robot to move. By adopting the embodiment of the invention, various visual motion trends can be quickly extracted and fused from a dynamic environment, and corresponding behavior responses are automatically triggered according to different visual motion scenes, so that visual motion sensing neurons in different modes are correctly scheduled and allocated, and an obstacle avoidance task of a robot is completed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This document relates to the field of motion information processing technology, and in particular to a motion control system and method based on multiple bionic visual neurons. Background Art

[0002] Optical flow is an effective strategy for robot collision avoidance. However, when implementing the robot's collision avoidance and navigation functions, it is necessary to deal with the large amount of optical flow calculation, insufficient precision and certainty, and other issues. The deep learning direction is excluded from embedded systems due to the large model, large number of samples required, and complex deployment. The contact time (TTC) algorithm cannot be used due to the limitation of the sensing principle. Therefore, after comprehensively considering the cost, efficiency, size and portability of ground micro autonomous mobile robots, the biological likelihood visual neural network in the feature extraction algorithm direction has become the first choice for implementing multi-scene collision detection tasks in embedded systems.

[0003] However, the existing visual motion trend discrimination neural networks, including LGMD1, LGMD2 and DSNs, and their improved solutions have three main shortcomings: (1) A single neuron responds to multiple stimuli. Although the bio-inspired lateral inhibitory neuron giant motion detector (LGMD1 and LGMD2) can show very high selectivity in response to stimuli produced by specific features when used as a single neuron, experiments have shown that the encoding utility of a single neuron for external stimuli is limited. One of the reasons is that a single neuron is sensitive to multiple stimuli, so the output response may be mixed and lack accurate directionality; (2) The robustness of a single neuron to multiple scenes and parameters is poor. Although a single neuron has high accuracy and efficiency in identifying obstacles in specific scenes, it has a high number of false positives and warnings in the face of complex and changing light scenes, left-right translational motion, and rotational motion-dominated ground robot scenes. There is a lack of distinction and screening for different scenes, and the model itself has many parameters and poor robustness. During the movement of a ground micro autonomous mobile robot, translation, rotation, forward and backward are its main movement modes. It is unrealistic to use only a single visual motion perception neuron model for work and experiments in complex scenes; (3) Direction-selective neurons will introduce process noise when used in embedded systems. A direct way to calculate direction-selective neurons is to directly calculate the change values ​​in its two fields of view to select the left and right directions. However, this method is not stable because noise is introduced during the calculation of DSN. Summary of the invention

[0004] One or more embodiments of this specification provide a motion control system based on multiple bionic visual neurons, including:

[0005] Image acquisition module: used to collect environmental video in real time through the camera carried by the robot to obtain image information;

[0006] Visual detection module: including lobular giant motion detector LGMD and direction selective neuron DSN, which are used to calculate and output visual motion trends through LGMD and DSN according to the acquired image information; wherein, the warning information of obstacle objects is obtained through LGMD calculation, and the translation direction of the robot is calculated through DSN;

[0007] Motion execution module: includes three-state hysteresis neuron TSHN, which is used to schedule the output motion trend through the set neuron output priority, select the motion trend in turn to output the corresponding motion command, and control the robot movement.

[0008] Furthermore, the visual motion trends include: safe motion; collision approach motion; collision approach motion caused by a dark object; leftward translation motion; and rightward translation motion.

[0009] Furthermore, the visual detection module is specifically used for:

[0010] The image information is processed by LGMD, and the brightness change of the corresponding pixel in the input image is obtained by the photoreceptor, and the brightness change result is input into the polarity channel. In each polarity channel, two different inhibition I cell results are generated according to the change of inhibition strength and inhibition range, and then two different local S cell results are generated. The difference between the two local S cells is used as the final output of each polarity channel. The information of the two channels is integrated through S cells to obtain the early warning information of the obstacle object, and the approach is judged by multiple stimulations within the window time.

[0011] Furthermore, the visual detection module includes lobular giant motion detectors LGMD1 and LGMD2, and outputs the visual motion trend of the collision approaching motion through LGMD1 and outputs the visual motion trend of the collision approaching motion caused by the dark object through LGMD2. The specific method is as follows:

[0012] The grayscale value of the corresponding pixel position in the input image is obtained through multiple photoreceptors in the first layer of the LGMD network, and the change difference between adjacent sequence frames is calculated to obtain the brightness change result, and the brightness increment flows into the ON path, while the brightness decrement flows into the OFF path;

[0013] In E-unit cells, the channel information flowing in through the ON channel and the OFF channel is preprocessed, and the excitatory output of the channels with different polarities in the P-unit cells is selectively inherited; I-unit cells, i.e. inhibitory neurons, inherit the inhibitory output of the channels with different polarities in the P-unit cells; LGMD2 has a stronger inhibitory effect in all ON channels, forming a biased ON pathway;

[0014] In the S unit cell, i.e. the local integration cell layer, two different local S cell results generated from the E and I unit cell layers are collected, which are the results of the Son and Soff channels respectively;

[0015] Determine whether it is the LGMD1 or LGMD2 neuron network. If the LGMD1 neuron is turned on, calculate the output of the FFI layer. If the LGMD2 neuron pathway is turned on, calculate the output of the S-layer neuron cells on the On bias pathway.

[0016] The output of the S-layer unit cell from the off pathway is calculated, and the information from the On pathway in the S-layer unit cell is integrated with the information from the Off pathway through superlinear operations to finally form a complete S-layer unit cell output.

[0017] Furthermore, the LGMD1 or LGMD2 neural network is judged. If the LGMD1 neuron is turned on, the output of the FFI layer is calculated; if the LGMD2 neuron pathway is turned on, the output of the S layer neuron cell on the On bias pathway is calculated:

[0018] If one of the LGMD1 or LGMD2 algorithms is turned on, the output of the S-layer unit neuron is convolved with the specified weight, and the output and output threshold of the G-layer unit neuron are integrated to achieve adaptive threshold calculation; otherwise, the response threshold of the G-layer unit neuron is directly calculated.

[0019] Furthermore, the direction selective neurons DSN include DSNL and DSNR, which are used to output visual motion trends of leftward translational motion or rightward translational motion, and the visual detection module also includes a further unit for:

[0020] By calculating the difference of DSN, the visual motion trend of the robot's leftward translational motion or rightward translational motion is obtained.

[0021] Furthermore, the motion execution module is specifically used for:

[0022] The difference between DSNL and DSNR is evaluated by the three-state hysteresis neuron TSHN, and when the difference output is stable, it is judged that translational movement is performed;

[0023] The direction of translation movement is determined based on the preset threshold. When the absolute value of the input intensity is greater than the preset offset threshold, the left or right translation movement is selected; when the input intensity drops below the offset threshold and crosses to the other side, it returns to the center state; when the difference value is between the two thresholds, the original output is kept unchanged.

[0024] Furthermore, the outputs of the DSNL and DSNR are filtered by a single-pole recursive filter.

[0025] Furthermore, the neuron output priority is specifically: DSNR>DSNL>TSHN>LGMD2>LGMD1.

[0026] One or more embodiments of this specification provide a motion control method based on multiple bionic visual neurons, including:

[0027] The camera carried by the robot collects environmental video in real time to obtain image information;

[0028] According to the acquired image information, the visual motion trend is calculated and output through LGMD and DSN; among them, the warning information of obstacle objects is obtained through LGMD calculation, and the translation direction of the robot is calculated through DSN;

[0029] It includes a three-state hysteresis neuron TSHN, which is used to schedule the output motion trend through the set neuron output priority, select the motion trend in turn to output the corresponding motion instruction, and control the robot movement.

[0030] By adopting the embodiments of the present invention, it is possible to quickly extract and integrate multiple visual motion trends from a dynamic environment, and automatically trigger corresponding behavioral responses according to different visual motion scenarios; the present invention has a simple structure, efficient calculation, and strong robustness, and can be easily applied to other autonomous mobile platforms, effectively reducing the memory usage between different single neurons, saving resource overhead, while maintaining the performance of the network itself, and enabling the visual motion perception neurons in different modes to be correctly scheduled and allocated, effectively completing the obstacle avoidance task of the ground micro autonomous mobile robot.

[0031] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0033] Figure 1 A schematic diagram of the composition of a motion control system based on multiple bionic visual neurons provided in one or more embodiments of this specification;

[0034] Figure 2 A flowchart of a motion control method based on multiple bionic visual neurons provided for one or more embodiments of this specification. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0036] System Example

[0037] According to an embodiment of the present invention, a motion control system based on multiple bionic visual neurons is provided. Figure 1 A schematic diagram of a motion control system based on multiple bionic visual neurons provided in one or more embodiments of this specification, such as Figure 1 As shown, the motion control system based on multiple bionic visual neurons according to an embodiment of the present invention specifically includes:

[0038] Image acquisition module 10: used to collect environmental video in real time through the camera carried by the robot to obtain image information.

[0039] Visual detection module 12: includes lobular giant motion detector LGMD and direction selective neuron DSN, which is used to calculate and output visual motion trend through LGMD and DSN according to the acquired image information; wherein, warning information of obstacle objects is obtained through LGMD calculation, and the translation direction of the robot is calculated through DSN.

[0040] Visual motion trends include: safe motion; collision approach motion; collision approach motion caused by a dark object; left translation motion; right translation motion.

[0041] The visual inspection module 12 is specifically used for:

[0042] The LGMD network is used to process the image information input from the camera mounted on the robot, and the brightness change of the corresponding pixel in the input image stream is obtained through the photoreceptor, and the brightness change result is input into the polarity channel. In each polarity channel, two different inhibition I cell results are generated according to the changes in the inhibition strength and inhibition range, and then two different local S cell results are generated. The difference between the two local S cells is used as the final output of each polarity channel. The information of the two channels is integrated through the S cells to obtain the warning information of the obstacle object, and finally the approach is judged by multiple stimulations within the window time. The visual detection module includes the lobular giant motion detector LGMD1 and LGMD2. The visual motion trend of the collision approaching motion is output through LGMD1, and the visual motion trend of the collision approaching motion caused by the dark object is output through LGMD2. The specific method is as follows:

[0043] Receive image information from the camera input, and process the input image stream from the camera through the Extended-LGMD1 and Extended-LGMD2 networks. Multiple photoreceptors in the first layer of the Extended-LGMD network configured by BVNN capture the grayscale values ​​of the corresponding pixel positions in the input image stream, and calculate the change difference between adjacent sequence frames to obtain the brightness change results to form photonic unit cells. The increase in brightness flows into the ON path, while the decrease in brightness flows into the OFF path;

[0044] In the excitatory unit cells (excitatory neurons), the channel information flowing in through the ON channel and the OFF channel is preprocessed, and the excitatory output of the different polarity channels in the photonic unit cells is selectively inherited. The inhibitory unit cells (inhibitory neurons) inherit the inhibitory output of the different polarity channels in the photonic unit cells. In order to achieve the specific selectivity of LGMD2, a stronger inhibitory effect is proposed in all ON channels to form a biased ON pathway;

[0045] In the Combination unit cell (local integration cell layer), two different local combination cell results generated from the Excitatory and Inhibitory unit cell layers are collected, which are the results of the Con and Coff channels respectively;

[0046] Determine whether it is the LGMD1 or LGMD2 neural network. If the LGMD1 neuron is turned on, calculate the output of the feedforward inhibition layer; if the LGMD2 neuron pathway is turned on, calculate the output of the Combination layer neuron cells on the On bias pathway;

[0047] Calculate the output of the Combination layer unit cell from the Off pathway, and integrate the information from the On pathway and the Off pathway in the Combination layer unit cell through superlinear operations, and finally form a complete Combination layer unit cell output.

[0048] For the judgment of LGMD1 or LGMD2 neural network, if one of the LGMD1 or LGMD2 algorithms is turned on, the output of the Combination layer unit neuron is convolved with the specified weight, and the output and output threshold of the Integration layer unit neuron are integrated to realize adaptive threshold calculation; otherwise, the response threshold of the Integration layer unit neuron is directly calculated.

[0049] The direction selective neurons DSN include DSNL and DSNR, which are used to output the visual motion trend of leftward translational motion or rightward translational motion. The visual detection module 12 is further used to:

[0050] A pair of base neuron prototype BVNNs configured as DSNL and DSNR are used to process the image information input from the robot's camera and calculate the difference between direction selective neurons (DSNs) to obtain the decision of whether the robot should translate to the left or to the right as a whole.

[0051] The motion execution module 14 includes a three-state hysteresis neuron TSHN, which is used to schedule the output motion trends through the set neuron output priority, select the motion trends in turn to output corresponding motion instructions, and control the robot movement.

[0052] In order to reduce noise, the difference value dTSHN between a pair of DSNL and DSNR is evaluated by a three-state hysteresis neuron TSHN, and translational motion is judged when the difference value output is stable. The specific steps for evaluating the difference value dTSHN between DSNL and DSNR are as follows:

[0053] First, the outputs of DSNL and DSNR are filtered through a single-pole recursive filter to reduce noise. When dTSHN has a stable output, a reliable judgment of translational movement is made. The three-state hysteresis neuron TSHN has three constant set thresholds. The three thresholds form three outputs, corresponding to left translation, right translation, and no translation, which are represented by -1, 1, and 0, respectively.

[0054] The direction of translational movement is determined based on the preset threshold. When the absolute value of the input intensity is greater than the absolute value of the preset offset threshold, the left or right translational movement is selected according to the actual sign of the input intensity, and the offset threshold is 1; when the input intensity drops below the offset threshold and the sign of the input intensity value changes, it will return to the center state; when the difference value is between the two thresholds, the neuron keeps the original output unchanged.

[0055] The priority preemption mechanism is used to adjust the order of input neurons, and the five neurons LGMD1, LGMD2, DSNL, DSNR, and TSHN that identify visual motion trends are set to specific priorities for connecting presynaptic neurons. Neurons with high priorities will inhibit the output of neurons with lower priorities to ensure that only one output is valid at any time. In this embodiment, LGMD1 and LGMD2 are set to be responsible for detecting the movement of approaching objects, so "movement" has a higher priority than "safety"; LGMD2 can provide further recognition of dark approaching objects; the translational movement (left or right) decision given by TSHN will terminate the detection of collision approaching movement, so the neuron output priority is set to: DSNR>DSNL>TSHN>LGMD2>LGMD1.

[0056] The beneficial effects of the present invention are as follows:

[0057] By adopting the embodiments of the present invention, it is possible to quickly extract and integrate multiple visual motion trends from a dynamic environment, and automatically trigger corresponding behavioral responses according to different visual motion scenarios; the present invention has a simple structure, efficient calculation, and strong robustness, and can be easily applied to other autonomous mobile platforms, effectively reducing the memory usage between different single neurons, saving resource overhead, while maintaining the performance of the network itself, and enabling the visual motion perception neurons in different modes to be correctly scheduled and allocated, effectively completing the obstacle avoidance task of the ground micro autonomous mobile robot.

[0058] Method Embodiment

[0059] According to an embodiment of the present invention, a motion control method based on multiple bionic visual neurons is provided. Figure 2 A flowchart of a motion control method based on multiple bionic visual neurons is provided for one or more embodiments of this specification, such as Figure 2 As shown, the motion control method based on multiple bionic visual neurons according to an embodiment of the present invention specifically includes:

[0060] S1. Collect environmental video in real time through the camera carried by the robot to obtain image information;

[0061] S2. According to the acquired image information, the visual motion trend is calculated and output through LGMD and DSN; wherein, the warning information of the obstacle object is obtained through LGMD calculation, and the translation direction of the robot is calculated through DSN;

[0062] S3. includes a three-state hysteresis neuron TSHN, which is used to schedule the output motion trend through the set neuron output priority, select the motion trend in turn to output the corresponding motion instruction, and control the robot movement.

[0063] The embodiment of the present invention is a method embodiment corresponding to the above-mentioned system embodiment. The specific operation of each step can be understood by referring to the description of the system embodiment, which will not be repeated here.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A motion control system based on multiple bionic visual neurons, characterized in that: include: Image acquisition module: used to collect environmental video in real time through the camera carried by the robot to obtain image information; Visual detection module: including lobular giant motion detector LGMD and direction selective neuron DSN, which are used to calculate and output visual motion trends through LGMD and DSN according to the acquired image information; wherein, the warning information of obstacle objects is obtained through LGMD calculation, and the translation direction of the robot is calculated through DSN; Motion execution module: includes three-state hysteresis neuron TSHN, which is used to schedule the output motion trend through the set neuron output priority, select the motion trend in turn to output the corresponding motion command, and control the robot movement.

2. The system according to claim 1, characterized in that The visual motion trends include: safe motion; collision approach motion; collision approach motion caused by a dark object; leftward translation motion; and rightward translation motion.

3. The system according to claim 2, characterized in that The visual detection module is specifically used for: The image information is processed by LGMD, and the brightness change of the corresponding pixel in the input image is obtained by the photoreceptor, and the brightness change result is input into the polarity channel. In each polarity channel, two different inhibition I cell results are generated according to the change of inhibition strength and inhibition range, and then two different local S cell results are generated. The difference between the two local S cells is used as the final output of each polarity channel. The information of the two channels is integrated through S cells to obtain the early warning information of the obstacle object, and the approach is judged by multiple stimulations within the window time.

4. The system according to claim 3, characterized in that The visual detection module includes lobular giant motion detectors LGMD1 and LGMD2, and outputs the visual motion trend of the collision approaching motion through LGMD1 and the visual motion trend of the collision approaching motion caused by the dark object through LGMD2. The specific method is as follows: The grayscale value of the corresponding pixel position in the input image is obtained through multiple photoreceptors in the first layer of the LGMD network, and the change difference between adjacent sequence frames is calculated to obtain the brightness change result, and the brightness increment flows into the ON path, while the brightness decrement flows into the OFF path; In E-unit cells, the channel information flowing in through the ON channel and the OFF channel is preprocessed, and the excitatory output of channels with different polarities in P-unit cells is selectively inherited; I-unit cells, i.e. inhibitory neurons, inherit the inhibitory output of different polarity channels in P-unit cells; LGMD2 has a stronger inhibitory effect among all ON channels, forming a biased ON pathway; In the S unit cell, i.e. the local integration cell layer, two different local S cell results generated from the E and I unit cell layers are collected, which are the results of the Son and Soff channels respectively; Determine whether the LGMD1 or LGMD2 neuron network is on. If the LGMD1 neuron is on, calculate the output of the FFI layer. If the LGMD2 neuron pathway is on, the output of the S-layer neuron cell on the On bias pathway is calculated; The output of the S-layer unit cell from the off pathway is calculated, and the information from the On pathway in the S-layer unit cell is integrated with the information from the Off pathway through superlinear operations to finally form a complete S-layer unit cell output.

5. The system according to claim 4, characterized in that The LGMD1 or LGMD2 neural network is judged. If the LGMD1 neuron is turned on, the output of the FFI layer is calculated; if the LGMD2 neuron pathway is turned on, the output of the S layer neuron cell on the On bias pathway is calculated: If one of the LGMD1 or LGMD2 algorithms is turned on, the output of the S-layer unit neuron is convolved with the specified weight, and the output and output threshold of the G-layer unit neuron are integrated to achieve adaptive threshold calculation; otherwise, the response threshold of the G-layer unit neuron is directly calculated.

6. The system according to claim 5, characterized in that The direction selective neurons DSN include DSNL and DSNR, which are used to output visual motion trends of leftward translational motion or rightward translational motion. The visual detection module also includes a module for further: By calculating the difference of DSN, the visual motion trend of the robot's leftward or rightward translational motion is obtained.

7. The system according to claim 6, characterized in that The motion execution module is specifically used for: The difference between DSNL and DSNR is evaluated by the three-state hysteresis neuron TSHN, and when the difference output is stable, it is judged that translational movement is performed; The direction of translation movement is determined based on the preset threshold. When the absolute value of the input intensity is greater than the preset offset threshold, the left or right translation movement is selected; when the input intensity drops below the offset threshold and crosses to the other side, it returns to the center state; when the difference value is between the two thresholds, the original output is kept unchanged.

8. The system according to claim 7, characterized in that The outputs of the DSNL and DSNR are filtered by a single-pole recursive filter.

9. The system according to claim 6, characterized in that The neuron output priority is specifically: DSNR>DSNL>TSHN>LGMD2>LGMD1.

10. A motion control method based on multiple bionic visual neurons, characterized in that: include: The camera carried by the robot collects environmental video in real time to obtain image information; According to the acquired image information, the visual motion trend is calculated and output through LGMD and DSN; among them, the warning information of obstacle objects is obtained through LGMD calculation, and the translation direction of the robot is calculated through DSN; It includes a three-state hysteresis neuron TSHN, which is used to schedule the output motion trend through the set neuron output priority, select the motion trend in turn to output the corresponding motion instruction, and control the robot movement.