Fall prevention control method and device based on event camera and pulse neural network

Abstract

The application belongs to the technical field of robot safety control, and discloses a fall-prevention control method and device based on an event camera and a spiking neural network. The method comprises the following steps: collecting a brightness change event stream of a work surface by using an event camera; inputting the event stream into a spiking neural network SNN deployed on a neuromorphic processor, extracting local optical flow features through a space-time convolution layer, judging the overall displacement trend through a global motion pooling layer, and outputting a slip warning pulse; determining a slip risk index according to the slip warning pulse; when the slip risk index exceeds a safety threshold, directly triggering an emergency brake or an adsorption enhancement action of a motor through a hardware reflex channel independent of a main control CPU. The above-mentioned method compresses the end-to-end response delay to the microsecond level, solves the problem of robot out-of-control falling caused by the perception delay and CPU scheduling delay in the traditional fall-prevention control, and is suitable for harsh working environments such as high altitude, large slope, strong light / dark light alternation, etc.

Description

Technical Field

[0001] This invention relates to the field of robot safety control technology, and in particular to a fall prevention control method and device based on an event camera and a pulse neural network. Background Technology

[0002] In high-altitude and steep-slope operations at hydropower stations, fall protection is the lifeline of robots. Traditional fall protection control solutions typically employ inertial measurement units (IMUs), visual odometry (VO), or force sensors. IMUs detect weightlessness or severe vibrations using accelerometers, but by the time a signal is detected, slippage has often already occurred and is difficult to reverse. Visual odometry calculates displacement by comparing consecutive frames, but is limited by camera frame rates, has high latency, and is prone to losing feature points during rapid movement or sudden changes in lighting (such as strong light reflection or darkness). Force sensors detect force on the feet, but are susceptible to interference from uneven ground, leading to false alarms. All of these solutions suffer from high sensing latency, susceptibility to environmental interference, and excessive end-to-end latency due to CPU scheduling, making it difficult to achieve microsecond-level fall protection response.

[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this invention is to provide a fall prevention control method and device based on an event camera and a spiking neural network, which aims to solve the technical problem of robot loss of control and falling due to perception delay and CPU scheduling delay in existing fall prevention control methods.

[0005] To achieve the above objectives, the present invention provides a fall prevention control method based on an event camera and a spiking neural network, the fall prevention control method based on an event camera and a spiking neural network comprising the following steps: The event camera deployed on the robot collects brightness change information of the working surface in real time and generates asynchronous event stream data. Each event in the event stream data contains at least pixel coordinates, timestamps and brightness change polarity. The event stream data is input in real time to a pre-constructed spiking neural network (SNN), which is deployed on a neuromorphic processor and includes at least one spatiotemporal convolutional layer and one global motion pooling layer. The spatiotemporal convolutional layer is used to perform spatiotemporal domain convolution operations on the input event stream data to extract optical flow pulse features that characterize local motion trends. The global motion pooling layer is used to aggregate and analyze the optical flow pulse features to determine whether there is an overall displacement trend that meets the conditions within the entire field of view. Based on the determination result, a slip alarm pulse is output. The slip risk index is determined based on the slip alarm pulse; When the slip risk index exceeds the preset safety threshold, the emergency braking pin or adsorption enhancement pin of the motor driver is triggered through the hardware reflection path, so that the robot performs an emergency stop lock-up or adsorption force enhancement action. The hardware reflection path operates independently of the task scheduling queue of the robot's main control CPU.

[0006] In one embodiment, the event camera employs a logarithmic light intensity change detection mechanism, whereby each pixel independently monitors brightness changes. When the brightness change exceeds a preset logarithmic threshold, an event is generated. The event is represented by coordinates, time, and polarity, where polarity represents brightening or darkening. The event camera has a temporal resolution of less than 1 microsecond and a dynamic range of not less than 120dB.

[0007] In one embodiment, the neurons of the SNN network adopt the leak-integration-fire (LIF) model, and the equation corresponding to the membrane potential change of the LIF neuron is:

[0008] in, Let be the membrane potential of the LIF neuron at time t. The membrane time constant is The rate of change of membrane potential, Let R be the film resistance. For synaptic weights, The input pulse sequence.

[0009] In one embodiment, the synaptic weights of the SNN network are trained unsupervised using the pulse time-dependent plasticity (STDP) rule, which adjusts the weights based on the time difference between the presynaptic pulse and the postsynaptic pulse: if the presynaptic pulse occurs before the postsynaptic pulse, the weight is increased; if the presynaptic pulse occurs after the postsynaptic pulse, the weight is decreased.

[0010] In one embodiment, the slip risk index The calculation formula is:

[0011] in, For the SNN output layer at time... pulse sequence, Let be the integral variable, representing any moment within the time window. The preset time window length.

[0012] In one embodiment, the hardware reflection path simultaneously triggers the following actions: An emergency stop pulse is sent to the hub motor driver to lock the hub motor and apply reverse braking torque. Send an overload enable signal to the vacuum adsorption motor driver to increase the adsorption power to 120% to 150% of the rated power, and maintain it for no more than 1 second; Send a locking command to the robot's posture adjustment mechanism to prevent the joint degrees of freedom from moving passively during the sliding process.

[0013] In one embodiment, the method further includes: After triggering the anti-fall action, the slip risk index is continuously monitored. When the slip risk index is lower than the safety threshold for at least two consecutive time windows, a recovery signal is output through the hardware reflection path to notify the robot's main control CPU to take over control and gradually reduce the adsorption power to the normal operation mode.

[0014] Furthermore, to achieve the above objectives, the present invention also proposes a fall protection control device based on an event camera and a spiking neural network. This fall protection control device is applied to the fall protection control method based on an event camera and a spiking neural network described above. The device includes: The dynamic visual perception front-end module is used to collect brightness change information of the working surface in real time using an event camera deployed on the robot, and generate asynchronous event stream data. Each event in the event stream data includes at least pixel coordinates, timestamps, and brightness change polarity. The neural reflex controller module is used to input the event stream data into a pre-constructed spiking neural network (SNN) in real time. The SNN is deployed on a neuromorphic processor and includes at least one spatiotemporal convolutional layer and one global motion pooling layer. The neural reflex controller module is used to perform spatiotemporal domain convolution operations on the input event stream data using the spatiotemporal convolution layer to extract optical flow pulse features that characterize local motion trends, and to perform aggregation analysis on the optical flow pulse features using the global motion pooling layer to determine whether there is an overall displacement trend that meets the conditions within the entire field of view, and to output a slip alarm pulse based on the determination result. The neural reflex controller module is used to determine the slip risk index based on the slip alarm pulse; The reflection arc control module is used to trigger the emergency braking pin or adsorption enhancement pin of the motor driver through the hardware reflection path when the slip risk index exceeds the preset safety threshold, so that the robot can perform an emergency stop lock-up or adsorption force enhancement action. The hardware reflection path runs independently of the task scheduling queue of the robot's main control CPU.

[0015] Furthermore, to achieve the above objectives, the present invention also proposes a fall protection control device based on an event camera and a spiking neural network. The fall protection control device based on an event camera and a spiking neural network includes: a memory, a processor, and a fall protection control program based on an event camera and a spiking neural network stored in the memory and executable on the processor. The fall protection control program based on an event camera and a spiking neural network is configured to implement the steps of the fall protection control method based on an event camera and a spiking neural network as described above.

[0016] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a drop prevention control program based on an event camera and a spiking neural network. When the drop prevention control program based on an event camera and a spiking neural network is executed by a processor, it implements the steps of the drop prevention control method based on an event camera and a spiking neural network as described above.

[0017] This invention utilizes an event camera to collect event streams of brightness changes on the working surface; the event stream is input into a spiking neural network (SNN) deployed on a neuromorphic processor, where local optical flow features are extracted through spatiotemporal convolutional layers, and a global motion pooling layer determines the overall displacement trend and outputs a slip alarm pulse; a slip risk index is determined based on the slip alarm pulse; when the slip risk index exceeds a safety threshold, an emergency braking of the motor or an adsorption enhancement action is directly triggered through a hardware reflection path independent of the main control CPU. This method compresses the end-to-end response latency to the microsecond level, solving the problem of robot loss of control and falls caused by perception delays and CPU scheduling delays in traditional fall prevention control. It is suitable for harsh working environments such as high altitudes, steep slopes, and alternating strong / dark light conditions. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the first embodiment of the anti-fall control method based on an event camera and a pulse neural network according to the present invention. Figure 2 This is a structural block diagram of the first embodiment of the anti-fall control device based on an event camera and a spiking neural network of the present invention.

[0019] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0021] This invention provides a fall prevention control method based on an event camera and a pulse neural network, referring to... Figure 1 , Figure 1This is a flowchart illustrating the first embodiment of the anti-fall control method based on an event camera and a pulse neural network according to the present invention.

[0022] In this embodiment, the fall prevention control method based on event camera and spiking neural network includes the following steps: Step S10: Use the event camera deployed on the robot to collect real-time brightness change information of the working surface and generate asynchronous event stream data.

[0023] In this embodiment, the execution subject is a drop prevention control device based on an event camera and a pulse neural network. The drop prevention control device based on an event camera and a pulse neural network has functions such as data processing, data communication, and program execution. The drop prevention control device based on an event camera and a pulse neural network can be a computer terminal device or other network device, or other devices with similar functions. This embodiment does not limit the scope of the application.

[0024] It is important to note that in high-altitude and steep-slope operations at hydropower stations, fall protection is the lifeline of robots. Traditional fall protection control solutions typically employ inertial measurement units (IMUs), visual odometry (VO), or force sensors. IMUs detect weightlessness or severe vibrations using accelerometers, but by the time a signal is detected, slippage has often already occurred and is difficult to reverse. Visual odometry calculates displacement by comparing consecutive frames, but is limited by camera frame rates, has high latency, and is prone to losing feature points during rapid movement or sudden changes in lighting (such as strong light reflection or darkness). Force sensors detect force on the feet, but are susceptible to interference from uneven ground, leading to false alarms. All of these solutions suffer from high sensing latency, susceptibility to environmental interference, and excessive end-to-end latency due to CPU scheduling, making it difficult to achieve microsecond-level fall protection response.

[0025] To address the aforementioned technical issues, this embodiment utilizes an event camera to collect event streams of brightness changes on the work surface. These event streams are then input into a spiking neural network (SNN) deployed on a neuromorphic processor. Local optical flow features are extracted through a spatiotemporal convolutional layer, and a global motion pooling layer determines the overall displacement trend and outputs a slip alarm pulse. A slip risk index is determined based on the slip alarm pulse. When the slip risk index exceeds a safety threshold, an emergency braking of the motor or an adsorption enhancement action is directly triggered via a hardware reflection path independent of the main control CPU. This method compresses the end-to-end response latency to the microsecond level, solving the problem of robot loss of control and falls caused by perception and CPU scheduling delays in traditional fall prevention control. It is suitable for harsh working environments such as high altitudes, steep slopes, and alternating strong / dark light conditions. Specifically, it can be implemented as follows.

[0026] It should be noted that each event in the event stream data includes at least pixel coordinates, a timestamp, and the polarity of the brightness change. The event camera uses a logarithmic light intensity change detection mechanism, where each pixel independently monitors brightness changes. When the brightness change exceeds a preset logarithmic threshold, an event is generated. The event is represented by coordinates, time, and polarity, where polarity represents brightening or darkening, respectively. The event camera has a temporal resolution of less than 1 microsecond and a dynamic range of no less than 120dB.

[0027] In the specific implementation, the event generation model works as follows: each pixel of the event camera independently monitors logarithmic light intensity changes, and when the change exceeds a threshold... Event triggered at time :

[0028] in, This indicates polarity (brightening or darkening). This mechanism ensures that the system responds only to dynamic changes (such as slip), with data redundancy approaching zero. Step S20: Input the event stream data into the pre-constructed spiking neural network in real time.

[0029] It should be noted that the SNN network is deployed on a neuromorphic processor and includes at least one spatiotemporal convolutional layer and one global motion pooling layer. The neurons in the SNN network employ the leak-integration-fire (LIF) model, and the equation corresponding to the membrane potential change of the LIF neuron is:

[0030] in, Let be the membrane potential of the LIF neuron at time t. The membrane time constant is The rate of change of membrane potential, Let R be the film resistance. For synaptic weights, The input pulse sequence.

[0031] In one embodiment, the synaptic weights of the SNN network are trained unsupervised using the pulse time-dependent plasticity (STDP) rule. The STDP rule adjusts the weights based on the time difference between the presynaptic pulse and the postsynaptic pulse: if the presynaptic pulse occurs before the postsynaptic pulse, the weight is increased; if the presynaptic pulse occurs later than the postsynaptic pulse, the weight is decreased. Through STDP, the SNN can autonomously learn the spatiotemporal patterns of slip events.

[0032] Step S30: The spatiotemporal convolutional layer is used to perform spatiotemporal domain convolution operations on the input event stream data to extract optical flow pulse features that characterize local motion trends. The global motion pooling layer is used to aggregate and analyze the optical flow pulse features to determine whether there is an overall displacement trend that meets the conditions within the entire field of view. Based on the judgment result, a slip alarm pulse is output.

[0033] In the specific implementation, the spatiotemporal convolutional layer uses separable spatiotemporal convolutional kernels (first temporal one-dimensional convolution, then spatial two-dimensional convolution) to extract local motion vectors in the event stream. The global motion pooling layer calculates the orientation histogram of motion vectors of all pixels. When the proportion of pixels in a certain orientation interval to the total number of pixels exceeds a preset consistency threshold, it is determined that there is an overall slippage trend, and a slippage alarm pulse is output.

[0034] For example, when the robot slides down the wall due to insufficient adhesion, all texture edges in the image generate downward motion events. The global motion pooling layer detects that more than 80% of the pixels move downward (with the consistency threshold set to 75%), thus outputting a high-frequency slip alarm pulse.

[0035] Step S40: Determine the slip risk index based on the slip alarm pulse.

[0036] In this embodiment, the slip risk index The calculation formula is:

[0037] in, For the SNN output layer at time... pulse sequence, Let be the integral variable, representing any moment within the time window. The preset time window length.

[0038] Step S50: When the slip risk index exceeds the preset safety threshold, the emergency braking pin or adsorption enhancement pin of the motor driver is triggered through the hardware reflection path to make the robot perform an emergency stop lock-up or adsorption force enhancement action.

[0039] In its implementation, the hardware reflection path directly connects the output pin of the neuromorphic processor to the hardware interrupt pin of the motor driver. At this time, the FPGA outputs a high-level pulse, bypassing the interrupt controller of the main CPU, and directly triggers emergency braking or adsorption enhancement. Specifically, the device reflection path simultaneously triggers the following actions: An emergency stop pulse is sent to the hub motor driver to lock the hub motor and apply reverse braking torque. Send an overload enable signal to the vacuum adsorption motor driver to increase the adsorption power to 120% to 150% of the rated power, and maintain it for no more than 1 second; Send a locking command to the robot's posture adjustment mechanism to prevent the joint degrees of freedom from moving passively during the sliding process.

[0040] Furthermore, after triggering the anti-fall action, the slip risk index is continuously monitored; when $R_{slip}(t)$ is below the safety threshold for at least two consecutive time windows, a recovery signal is output through the hardware reflection path to notify the main control CPU to take over control and gradually reduce the adsorption power to the normal operation mode.

[0041] Furthermore, this embodiment will be fully described using a specific application scenario: high-altitude inspection of the concrete panel of a large hydropower station dam in Yunnan. This embodiment demonstrates the robot's fall prevention operation on a concrete panel at a height of 200 meters with a steep slope (1:1.4, approximately 35°). The environment features gusts of wind (level 6), dry moss on the surface, and complex lighting conditions (alternating between sun and shadow). Scene initialization: The robot crawls along the horizontal seam on the dam surface using a vacuum adsorption module. The event camera is aimed at the wall beneath its feet, in a "silent listening" state, outputting an extremely low event rate (only background noise). Sudden slippage: A strong gust of wind blows from the side, and the adsorption force on the robot's left foot momentarily weakens, resulting in a slight displacement (approximately 0.1 mm). The DVS pixel captures the instantaneous movement of the texture edge, bursting into a cluster of high-frequency event streams within 10 microseconds. Pulse processing and decision-making: The event stream is input into the SNN network. The spatiotemporal convolutional layer extracts local motion vectors, and the global motion pooling layer detects that all pixels move in the same direction (downward). The slippage alarm pulse frequency momentarily exceeds the safety threshold. The entire calculation is completed within the FPGA, taking only 50 microseconds. Reflective execution: The hardware reflection path directly pulls the Emergency_Stop pin of the motor driver high; the adsorption motor power is instantly overclocked from 60% to 120%; the hub motor applies a reverse torque to lock the wheel system. The robot is locked when the slip distance is less than 0.5mm, preventing a runaway fall caused by kinetic energy accumulation. Recovery: After the wind weakens and the slip event disappears, the SNN output frequency decreases. The hardware reflection path outputs a recovery signal, the main control CPU takes over control, adjusts the posture, and continues operation. In this embodiment, the event camera's parameter time resolution is <1µs, and the dynamic range is >120dB; the SNN training uses the STDP rule; the slip risk index calculation, multiple execution actions, and recovery logic are as described above. The synergistic effect of these features achieves a microsecond-level anti-fall response, significantly superior to existing technologies.

[0042] In this embodiment, an event camera is used to collect event streams of brightness changes on the working surface. These event streams are then input into a spiking neural network (SNN) deployed on a neuromorphic processor. Local optical flow features are extracted through a spatiotemporal convolutional layer, and a global motion pooling layer determines the overall displacement trend and outputs a slip alarm pulse. A slip risk index is determined based on the slip alarm pulse. When the slip risk index exceeds a safety threshold, an emergency braking of the motor or an adsorption enhancement action is directly triggered through a hardware reflection path independent of the main control CPU. This method compresses the end-to-end response latency to the microsecond level, solving the problem of robot loss of control and falls caused by perception and CPU scheduling delays in traditional fall prevention control. It is suitable for harsh working environments such as high altitudes, steep slopes, and alternating strong / dark light conditions.

[0043] Furthermore, this embodiment of the invention also proposes a storage medium storing a drop prevention control program based on an event camera and a spiking neural network. When the drop prevention control program based on an event camera and a spiking neural network is executed by a processor, it implements the steps of the drop prevention control method based on an event camera and a spiking neural network as described above.

[0044] Reference Figure 2 , Figure 2 This is a structural block diagram of the first embodiment of the anti-fall control device based on an event camera and a spiking neural network of the present invention.

[0045] like Figure 2 As shown, the anti-fall control device based on an event camera and a pulse neural network proposed in this embodiment of the invention includes: The dynamic visual perception front-end module 10 is used to collect brightness change information of the working surface in real time using an event camera deployed on the robot, and generate asynchronous event stream data. Each event in the event stream data includes at least pixel coordinates, timestamps and brightness change polarity. The neural reflex controller module 20 is used to input the event stream data into a pre-constructed spiking neural network in real time. The SNN network is deployed on a neuromorphic processor and includes at least one spatiotemporal convolutional layer and one global motion pooling layer. The neural reflex controller module 20 is used to perform spatiotemporal domain convolution operations on the input event stream data using the spatiotemporal convolution layer to extract optical flow pulse features that characterize local motion trends, and to perform aggregation analysis on the optical flow pulse features using the global motion pooling layer to determine whether there is an overall displacement trend that meets the conditions within the entire field of view, and to output a slip alarm pulse based on the judgment result. The neural reflex controller module 20 is used to determine the slip risk index based on the slip alarm pulse; The reflection arc control module 30 is used to trigger the emergency braking pin or adsorption enhancement pin of the motor driver through the hardware reflection path when the slip risk index exceeds the preset safety threshold, so that the robot can perform an emergency stop lock-up or adsorption force enhancement action. The hardware reflection path runs independently of the task scheduling queue of the robot's main control CPU.

[0046] In this embodiment, an event camera is used to collect event streams of brightness changes on the working surface. These event streams are then input into a spiking neural network (SNN) deployed on a neuromorphic processor. Local optical flow features are extracted through a spatiotemporal convolutional layer, and a global motion pooling layer determines the overall displacement trend and outputs a slip alarm pulse. A slip risk index is determined based on the slip alarm pulse. When the slip risk index exceeds a safety threshold, an emergency braking of the motor or an adsorption enhancement action is directly triggered through a hardware reflection path independent of the main control CPU. This method compresses the end-to-end response latency to the microsecond level, solving the problem of robot loss of control and falls caused by perception and CPU scheduling delays in traditional fall prevention control. It is suitable for harsh working environments such as high altitudes, steep slopes, and alternating strong / dark light conditions.

[0047] This application embodiment also provides a fall protection control device based on an event camera and a spiking neural network, including a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other through the communication bus. The memory is used to store the fall protection control program based on the event camera and the spiking neural network. When the processor executes the program stored in the memory, it implements the above-mentioned fall protection control method based on the event camera and the spiking neural network.

[0048] The communication bus mentioned in the aforementioned drop protection control device based on event cameras and pulse neural networks can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0049] The communication interface is used for communication between the aforementioned anti-fall control device based on event camera and spiking neural network and other devices.

[0050] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0051] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0052] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0053] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0054] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0055] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0056] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0057] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0058] In addition, for technical details not described in detail in this embodiment, please refer to the anti-fall control method based on event camera and pulse neural network provided in any embodiment of the present invention, which will not be repeated here.

[0059] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0060] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0061] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0062] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

[0063] It is understood that the system provided in the embodiments of the present invention corresponds to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

Claims

1. A fall prevention control method based on an event camera and a spiking neural network, characterized in that, The fall prevention control method based on event camera and pulse neural network includes: The event camera deployed on the robot collects brightness change information of the working surface in real time and generates asynchronous event stream data. Each event in the event stream data contains at least pixel coordinates, timestamps and brightness change polarity. The event stream data is input in real time to a pre-constructed spiking neural network (SNN), which is deployed on a neuromorphic processor and includes at least one spatiotemporal convolutional layer and one global motion pooling layer. The spatiotemporal convolutional layer is used to perform spatiotemporal domain convolution operations on the input event stream data to extract optical flow pulse features that characterize local motion trends. The global motion pooling layer is used to aggregate and analyze the optical flow pulse features to determine whether there is an overall displacement trend that meets the conditions within the entire field of view. Based on the determination result, a slip alarm pulse is output. The slip risk index is determined based on the slip alarm pulse; When the slip risk index exceeds the preset safety threshold, the emergency braking pin or adsorption enhancement pin of the motor driver is triggered through the hardware reflection path, so that the robot performs an emergency stop lock-up or adsorption force enhancement action. The hardware reflection path operates independently of the task scheduling queue of the robot's main control CPU.

2. The fall prevention control method based on event camera and pulse neural network as described in claim 1, characterized in that, The event camera employs a logarithmic light intensity change detection mechanism, where each pixel independently monitors brightness changes. When the brightness change exceeds a preset logarithmic threshold, an event is generated. The event is represented by coordinates, time, and polarity, where polarity represents brightening or darkening. The event camera has a temporal resolution of less than 1 microsecond and a dynamic range of no less than 120dB.

3. The fall prevention control method based on event camera and pulse neural network as described in claim 1, characterized in that, The neurons in the SNN network adopt the leak-integration-fire (LIF) model, and the equation corresponding to the membrane potential change of the LIF neuron is: in, Let be the membrane potential of the LIF neuron at time t. The membrane time constant is The rate of change of membrane potential, Let R be the film resistance. For synaptic weights, The input pulse sequence.

4. The fall prevention control method based on event camera and pulse neural network as described in claim 3, characterized in that, The synaptic weights of the SNN network are trained unsupervised using the pulse time-dependent plasticity (STDP) rule. The STDP rule adjusts the weights based on the time difference between the presynaptic pulse and the postsynaptic pulse: if the presynaptic pulse occurs before the postsynaptic pulse, the weight is increased; if the presynaptic pulse occurs after the postsynaptic pulse, the weight is decreased.

5. The fall prevention control method based on an event camera and a pulse neural network as described in claim 1, characterized in that, Slip Risk Index The calculation formula is: in, For the SNN output layer at time... pulse sequence, Let be the integral variable, representing any moment within the time window. The preset time window length.

6. The fall prevention control method based on event camera and pulse neural network as described in claim 1, characterized in that, The hardware reflection path simultaneously triggers the following actions: An emergency stop pulse is sent to the hub motor driver to lock the hub motor and apply reverse braking torque. Send an overload enable signal to the vacuum adsorption motor driver to increase the adsorption power to 120% to 150% of the rated power, and maintain it for no more than 1 second; Send a locking command to the robot's posture adjustment mechanism to prevent the joint degrees of freedom from moving passively during the sliding process.

7. The fall prevention control method based on an event camera and a pulse neural network as described in claim 1, characterized in that, The method further includes: After triggering the anti-fall action, the slip risk index is continuously monitored. When the slip risk index is lower than the safety threshold for at least two consecutive time windows, a recovery signal is output through the hardware reflection path to notify the robot's main control CPU to take over control and gradually reduce the adsorption power to the normal operation mode.

8. A fall protection control device based on an event camera and a pulse neural network, characterized in that, The drop prevention control device based on an event camera and a pulse neural network is applied to the drop prevention control method based on an event camera and a pulse neural network as described in any one of claims 1 to 7, wherein the device comprises: The dynamic visual perception front-end module is used to collect brightness change information of the working surface in real time using an event camera deployed on the robot, and generate asynchronous event stream data. Each event in the event stream data includes at least pixel coordinates, timestamps, and brightness change polarity. The neural reflex controller module is used to input the event stream data into a pre-constructed spiking neural network (SNN) in real time. The SNN is deployed on a neuromorphic processor and includes at least one spatiotemporal convolutional layer and one global motion pooling layer. The neural reflex controller module is used to perform spatiotemporal domain convolution operations on the input event stream data using the spatiotemporal convolution layer to extract optical flow pulse features that characterize local motion trends, and to perform aggregation analysis on the optical flow pulse features using the global motion pooling layer to determine whether there is an overall displacement trend that meets the conditions within the entire field of view, and to output a slip alarm pulse based on the determination result. The neural reflex controller module is used to determine the slip risk index based on the slip alarm pulse; The reflection arc control module is used to trigger the emergency braking pin or adsorption enhancement pin of the motor driver through the hardware reflection path when the slip risk index exceeds the preset safety threshold, so that the robot can perform an emergency stop lock-up or adsorption force enhancement action. The hardware reflection path runs independently of the task scheduling queue of the robot's main control CPU.

9. A fall protection control device based on an event camera and a pulse neural network, characterized in that, The drop prevention control device based on event camera and spiking neural network includes: a memory, a processor, and a drop prevention control program based on event camera and spiking neural network stored in the memory and executable on the processor. The drop prevention control program based on event camera and spiking neural network is configured to implement the steps of the drop prevention control method based on event camera and spiking neural network as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a fall protection control program based on an event camera and a spiking neural network. When the event camera and spiking neural network fall protection control program is executed by the processor, it implements the steps of the fall protection control method based on an event camera and a spiking neural network as described in any one of claims 1 to 7.

Images