Perception processing system based on hybrid sensor

Abstract

The invention relates to a perception processing system based on a hybrid sensor, and the system comprises a frame sensor which is used for capturing a scene image at a fixed rate, and generating frame image data; the event sensor is used for capturing scene change information based on event driving and generating event data; the image preprocessing module is used for synchronizing data transmitted by the frame sensor and the event sensor; the in-memory computing chip is used for deploying a frame processing network and an event processing network respectively, extracting image features from the frame image data through the frame processing network, and extracting dynamic changes of a scene from the event data through the event processing network; the controller is used for controlling data collected by the frame sensor and the event sensor to serve as input respectively, and the data are sent to the corresponding frame processing network and the event processing network after being synchronized by the image preprocessing module. Compared with the prior art, the method not only can monitor static objects, but also can quickly recognize dynamic objects, and has high sensitivity and high processing efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image sensors, in particular to a perception processing system based on a hybrid sensor. BACKGROUND

[0002] With the explosion of artificial intelligence application scenarios, the demand for data, algorithms and computing power of artificial intelligence is growing. In the traditional information perception processing flow, data collection and processing are two separate processes, and the massive data transmission between the two processes leads to a large amount of energy consumption of the system and limits the speed of system decision-making. In order to solve this problem, it is necessary to integrate a processor unit with high efficiency, high computing power and low power consumption directly into a sensing device, so that the sensing device front end has the functions of sensing, storage and computing at the same time, i.e. a sensing-storage-computing integrated system.

[0003] Image sensors, as very common perception signal front ends in life, can be roughly divided into two categories: one is the traditional frame-based sensor, and the other is the new event-based sensor. Frame-based sensors capture the entire scene image at a fixed rate, which can provide rich scene information including color, texture and motion, and generate continuous video streams. This type of sensor is very common and is widely used in video surveillance, smartphones, and most modern cars for advanced driver assistance systems (ADAS). However, they may miss some instantaneous changes due to the limitation of frame rate, or produce dynamic blur in high-speed motion, which makes frame-based sensors more suitable for slow-changing static scenes.

[0004] Unlike frame-based sensors, event-based sensors do not work at a fixed frame rate. They use an asynchronous approach and only generate data when the brightness of a pixel in the scene changes beyond a certain threshold. This "event-driven" mechanism enables them to capture scene changes with extremely high temporal resolution, making them ideal for detecting fast motion or working in environments with dramatic changes in lighting conditions. Whether it is a frame-based sensor or an event-based sensor, the delay caused by uploading the data collected by them to the cloud for processing is relatively high, which makes it necessary to process these data on the edge side in real time. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art and provide a high-sensitivity perception processing system based on a hybrid sensor that can monitor static objects and quickly identify dynamic objects.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] A sensing and processing system based on hybrid sensors includes a frame sensor, an event sensor, an image preprocessing module, an in-memory computing chip, and a controller.

[0008] The frame sensor is used to capture scene images at a fixed rate and generate frame image data;

[0009] The event sensor is used to capture scene change information based on event-driven mechanisms and generate event data;

[0010] The image preprocessing module is used to synchronize the data transmitted by the frame sensor and the event sensor;

[0011] The in-memory computing chip is used to deploy a frame processing network and an event processing network respectively. The frame processing network extracts image features from frame image data, and the event processing network extracts dynamic changes of the scene from event data.

[0012] The controller is connected to the frame sensor, event sensor, image preprocessing module and in-memory computing chip respectively. It is used to control the data collected by the frame sensor and event sensor as input, and after being synchronized by the image preprocessing module, it is sent to the corresponding frame processing network and event processing network.

[0013] Furthermore, the frame sensor and the event sensor are integrated into a hybrid sensor, which performs hardware-level synchronization settings when it is activated, so that the frame sensor and the event sensor share the same clock source or synchronization signal.

[0014] Furthermore, both the frame image data and the event data are timestamped. The image preprocessing module compares the timestamps of the event data with the timestamps of the frame image data to determine which frame image data each event data is related to, thereby achieving synchronization between the frame image data and the event data.

[0015] Furthermore, the frame processing network is a convolutional neural network.

[0016] Furthermore, the event processing network is a GNN network.

[0017] Furthermore, the image features extracted from the frame image data include color, texture, shape, and spatial information.

[0018] Furthermore, in the process of extracting dynamic changes of the scene from event data, the event processing network also combines image features extracted by the frame processing network for processing.

[0019] Furthermore, the frame processing network and the event processing network are trained respectively based on the data collected by the frame sensor and the event sensor.

[0020] Furthermore, the frame sensor is used to sense image data in a static scene.

[0021] Furthermore, the event sensor is used to detect fast-moving objects or environments with drastic changes in lighting conditions.

[0022] Compared with the prior art, the present invention has the following advantages:

[0023] (1) This invention uses both frame sensors and event sensors as inputs to the system's sensing end. It aims to utilize the advantages of frame sensors in static scenes and the characteristics of event sensors in dynamic scenes with high speed changes. Furthermore, it utilizes an in-memory computing chip to deploy a neural network that processes the data from these sensors, thereby realizing a highly sensitive sensing and processing system that can both monitor static objects and quickly identify dynamic objects.

[0024] (2) Based on the characteristics of the data collected by the two types of sensors, this invention trains two artificial neural networks respectively. For example, convolutional neural networks are suitable for perceiving and extracting object information in static scenes and can be used for processing data collected by frame-based cameras; another example is the asynchronous processing capability and efficient processing capability of GNN for sparse data, which can be used for processing asynchronous and sparse data collected by event-based cameras. By deploying these two neural networks trained according to their respective data characteristics on the CIM chip, the processing efficiency is further improved.

[0025] (3) The present invention adopts this hybrid sensor and separate data processing method, which can avoid the disadvantage of frame-based sensors in processing fast motion or dynamic changes, and can also avoid the disadvantage of event-based sensors in capturing slow changes. At the same time, it reduces the total amount of data to be transmitted and the amount of data calculation, improves processing efficiency, and realizes high-sensitivity perception processing. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the overall architecture of a sensing and processing system based on a hybrid sensor provided in an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram illustrating the generation and synchronization of frame data and event data from a hybrid sensor provided in an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram illustrating the collaborative operation of two networks deployed on a CIM chip according to an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram illustrating the equivalent acquisition effect of a hybrid sensor-based sensing processing system and a traditional frame sensor in an embodiment of the present invention.

[0030] Figure 5 This is a schematic diagram comparing the data transmission volume of a hybrid sensor-based sensing processing system and a traditional frame sensor in an embodiment of the present invention.

[0031] Figure 6 This is a schematic diagram comparing the computational load of a hybrid sensor-based sensing processing system and a traditional frame sensor in an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0033] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0034] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0035] Example 1

[0036] like Figure 1 As shown, this embodiment provides a sensing and processing system based on a hybrid sensor, including a frame sensor, an event sensor, an image preprocessing module, an in-memory computing chip, and a controller;

[0037] A frame sensor is used to capture scene images at a fixed rate and generate frame image data;

[0038] Event sensors are used to capture scene change information based on event-driven processes and generate event data;

[0039] The image preprocessing module is used to synchronize the data transmitted by the frame sensor and the event sensor;

[0040] The Computing In Memory (CIM) chip is used to deploy a frame processing network and an event processing network respectively. The frame processing network extracts image features from frame image data, and the event processing network extracts dynamic changes of the scene from event data.

[0041] The controller is connected to the frame sensor, event sensor, image preprocessing module, and in-memory computing chip respectively. It controls the data collected by the frame sensor and event sensor to be used as inputs, and after being synchronized by the image preprocessing module, they are sent to the corresponding frame processing network and event processing network.

[0042] Specifically, the frame sensor captures images of the entire scene at a fixed rate to perceive image data in static scenes where changes are slow.

[0043] Event sensors utilize their "event-driven" mechanism to capture scene change information with extremely high temporal resolution, enabling them to detect fast-moving objects or play a role in dynamic scenes such as environments with drastic changes in lighting conditions.

[0044] The image preprocessing module is used to synchronize the two types of data, ensuring that their timestamps correspond.

[0045] The CIM chip deploys two neural networks to process the data collected by the two types of cameras respectively, thereby improving processing efficiency.

[0046] The controller's role is to ensure that data captured by frame-based cameras and data captured by event-based cameras are used as inputs to the corresponding neural networks, thus ensuring error-free operation.

[0047] By employing this hybrid sensor approach and processing data separately, we can avoid the limitations of frame-based sensors in handling fast-moving or dynamically changing scenes, as well as the inability of event-based sensors to capture slow changes. This approach also improves processing efficiency and enables highly sensitive perception processing.

[0048] The advantages of deploying two neural networks in a CIM chip:

[0049] Utilizing the CIM chip accelerates edge computing capabilities. Furthermore, based on the characteristics of the data acquired by the two types of sensors, two artificial neural networks are trained separately. For example, convolutional neural networks (CNNs), suitable for perceiving and extracting object information in static scenes, can be used for processing data acquired by frame-based cameras. On the other hand, the asynchronous processing capabilities and efficient handling of sparse data by generative neural networks (GNNs) can be used for processing asynchronous and sparse data acquired by event-based cameras. By deploying these two neural networks, trained according to their respective data characteristics, on the CIM chip, processing efficiency is further improved.

[0050] Working principle:

[0051] This invention combines frame sensors and event sensors with a CIM chip to form a highly efficient real-time vision processing system. During operation, the system first captures data using two types of sensors: the frame sensor provides a continuous stream of images, while the event sensor generates data only when scene brightness changes. Under the control of a controller, this data is synchronized by a preprocessing module and then fed into the CIM chip, where two dedicated neural networks are deployed to process these two different data types respectively. Due to the application of in-memory computing technology, the data transfer time between memory and the processor is minimized, thereby reducing latency. Ultimately, the system can achieve accurate perception and real-time analysis of objects in both dynamic and static scenes while maintaining low latency and low bandwidth requirements.

[0052] The time synchronization process in this solution is as follows:

[0053] The frame sensor and event sensor are integrated into a hybrid sensor, which performs hardware-level synchronization settings upon startup to ensure that the two types of sensors share the same clock source or synchronization signal.

[0054] like Figure 2 As shown, when the frame sensor captures a frame of image, that frame is marked with a timestamp, such as a start time of 0, and the next frame is 1 / fps (frames per second). Due to its asynchronous nature, the event sensor generates event data with precise timestamps whenever pixel brightness changes. Subsequently, the image preprocessing module will process the event timestamps generated by the event sensor (such as...) Figure 2 The event is compared with the frame timestamp (t0) to determine which frames the event is related to, thus achieving synchronization.

[0055] like Figure 3 As shown, the collaborative working process of the two networks deployed on the CIM chip in this solution is as follows:

[0056] To achieve efficient visual processing, two neural networks were trained to process frame image data and event data, respectively. These two neural networks were trained separately based on the characteristics of the data acquired by the two sensors. When the frame sensor captures an image, the frame processing network begins to work, extracting image features from the frame image data. These features typically include color, texture, shape, and spatial information, which are very useful for understanding static scenes and slowly changing environments. Simultaneously, if the event sensor generates event data, the event processing network can process this asynchronous and sparse event data stream, capturing dynamic changes in the scene.

[0057] In some cases, image features from frame data can be fed into the event processing network to achieve better dynamic change capture.

[0058] likeFigure 4 The image shown illustrates the effect of using a hybrid sensor to acquire image data, which is equivalent to the acquisition effect of a traditional frame sensor at a high frame rate.

[0059] like Figure 5 As shown, when the latency effect of using a hybrid sensor is the same as that of using a high frame rate sensor, the perception effect of both on moving and static objects is basically the same. However, the total amount of data that the hybrid sensor needs to transmit is lower than that of the high frame rate sensor, and the required transmission rate is also lower.

[0060] like Figure 6 As shown, when the latency effect of using a hybrid sensor is the same as that of using a high frame rate sensor, the recognition and processing effects of both for moving and static objects are basically the same. However, the computational load in the hybrid sensor processing system is lower than that in the high frame rate sensor.

[0061] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A sensing and processing system based on hybrid sensors, characterized in that, It includes a frame sensor, an event sensor, an image preprocessing module, an in-memory computing chip, and a controller; The frame sensor is used to capture scene images at a fixed rate and generate frame image data; The event sensor is used to capture scene change information based on event-driven mechanisms and generate event data; The image preprocessing module is used to synchronize the data transmitted by the frame sensor and the event sensor; The in-memory computing chip is used to deploy a frame processing network and an event processing network respectively. The frame processing network extracts image features from frame image data, and the event processing network extracts dynamic changes of the scene from event data. The controller is connected to the frame sensor, event sensor, image preprocessing module and in-memory computing chip respectively. It is used to control the data collected by the frame sensor and event sensor as input, and after being synchronized by the image preprocessing module, it is sent to the corresponding frame processing network and event processing network.

2. The sensing and processing system based on a hybrid sensor according to claim 1, characterized in that, The frame sensor and event sensor are integrated into a hybrid sensor. When the hybrid sensor is activated, hardware-level synchronization is performed so that the frame sensor and event sensor share the same clock source or synchronization signal.

3. The sensing and processing system based on a hybrid sensor according to claim 1, characterized in that, Both the frame image data and the event data are timestamped. The image preprocessing module compares the timestamp of the event data with the timestamp of the frame image data to determine which frame image data each event data is related to, thereby achieving synchronization between the frame image data and the event data.

4. The sensing and processing system based on a hybrid sensor according to claim 1, characterized in that, The frame processing network is a convolutional neural network.

5. A sensing and processing system based on a hybrid sensor according to claim 1, characterized in that, The event processing network is a GNN network.

6. The sensing and processing system based on a hybrid sensor according to claim 1, characterized in that, Image features extracted from frame image data include color, texture, shape, and spatial information.

7. A sensing and processing system based on a hybrid sensor according to claim 1, characterized in that, The event processing network extracts dynamic changes in the scene from event data, and also processes the image features extracted by the frame processing network.

8. A sensing and processing system based on a hybrid sensor according to claim 1, characterized in that, The frame processing network and the event processing network are trained respectively based on the data collected by the frame sensor and the event sensor.

9. A sensing and processing system based on a hybrid sensor according to claim 1, characterized in that, The frame sensor is used to sense image data in static scenes.

10. A sensing processing system based on a hybrid sensor according to claim 1, characterized in that, The event sensor is used to detect fast-moving objects or environments with drastic changes in lighting conditions.

Images