A full-sky cloud imaging system based on neuromorphic vision sensor

Abstract

The present invention discloses a full-sky cloud imaging system based on a neuromorphic visual sensor, comprising a full-sky imager, a sunshade belt, a data acquisition module, and a data calculation module; the data acquisition module comprises a hemispherical mirror and an event-driven sensor for acquiring sky-related data; the full-sky imager uses an event-driven sensor for shooting, the event-driven sensor being arranged on a support arm of the full-sky imager and located above the center of the hemispherical mirror; the sunshade belt covers the surface of the hemispherical mirror; and the data acquisition module is connected to the data calculation module via a transmission module. The present invention has higher temporal resolution and dynamic range, and can more accurately identify and track targets by monitoring the movement of objects in the image. It can also measure cloud cover and cloud speed, ultimately achieving full-sky cloud cover and cloud movement speed measurement and forecasting of the probability of clouds blocking the sun.

Description

Technical Field

[0001] The present invention relates to the technical field of target recognition and tracking, and in particular to an all-sky cloud imaging system based on a neuromorphic vision sensor. Background Art

[0002] Visual analysis of motion has been a hot topic in the field of computer vision in recent years. It detects, identifies, and tracks targets from image sequences and understands and describes their behavior. It has extremely wide applications in smart healthcare, industrial quality inspection, robot tracking and navigation, sports events, mechanical testing and analysis, intelligent transportation, and military visual guidance.

[0003] Traditional image sensors output image information at a fixed frequency based on frame rates and exposure times. This results in large data volumes, high processing latency, high noise, and low dynamic range for back-end machine vision processing. Traditional cameras are also susceptible to changes in scene light intensity. While these shortcomings have been significantly improved in processing speed and target recognition accuracy through subsequent image algorithm enhancements, they still suffer from large errors, high computational complexity, and high power consumption when processing complex environments. This hinders effective results for ultra-short-term forecasting needs. Existing cloud cover recognition instruments mostly use traditional vision sensors that output image information at a fixed frequency. For tasks requiring 24 / 7 monitoring, the data volume is large and the processing speed required is high. Summary of the Invention

[0004] The present invention provides a full-sky cloud imaging system based on a neuromorphic visual sensor to solve the problems raised in the above background technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A full-sky cloud imaging system based on a neuromorphic visual sensor, comprising a full-sky imager, a sunshade, a data acquisition module, and a data calculation module;

[0007] The data acquisition module includes a hemispherical mirror and an event-driven sensor for acquiring sky-related data;

[0008] The all-sky imager uses an event-driven sensor for shooting, and the event-driven sensor is arranged on the support arm of the all-sky imager, and the event-driven sensor is located above the center of the hemispherical mirror; the solar shading belt covers the surface of the hemispherical mirror; the data acquisition module is connected to the data calculation module through the transmission module.

[0009] As a further improvement of the present technical solution: the sky-related data includes the outline of clouds in the sky, the direction of movement of clouds, the distribution of clouds, and the position information of the sun.

[0010] As a further improvement of the present technical solution: the data operation module includes a first operation module and a second operation module, wherein the first operation module takes the data stream of the event-driven sensor as input and calculates and identifies the target; the second operation module takes the data of target identification of the first operation module as input to obtain the motion trajectory prediction of the target.

[0011] As a further improvement of the present technical solution: the first operation module includes filtering and denoising and a convolutional neural network. The filtering and denoising adopts the homomorphic filtering method to remove noise in the data stream and increase the image contrast; the convolutional neural network performs target recognition on the filtered and denoised image through a 6-layer structure.

[0012] As a further improvement of the present technical solution: the second operation module includes a feature extractor and a motion model constructed by a Kalman filter. The feature extractor takes the data of the first operation module as input, obtains the similarity between the target features, and obtains the degree of feature matching between the target objects in the two previous and subsequent data streams by measuring the similarity; the motion model constructed by the Kalman filter crops and extracts the local portraits of all targets in the detected target objects to predict the motion trajectory, and accurately associates the targets with each other by combining the Kalman filter motion model and the appearance model of the deep features. When the detection result is successfully associated with the tracking trajectory, the identity identification code ID of the target corresponding to the previous data stream is assigned to the target object with a high matching degree in the next data stream, and then the status of the target object is updated.

[0013] As a further improvement of this technical solution: extracting appearance features and motion features through a single multi-border detection algorithm.

[0014] As a further improvement of this technical solution: the calculation of similarity combines the appearance model and the motion model to achieve the optimal allocation function of the detection results and the tracker prediction results.

[0015] As a further improvement of this technical solution: all event data in the system include the coordinate position and timestamp of the triggering event.

[0016] As a further improvement of this technical solution: it can be used to track a piece of cloud;

[0017] The method for tracking a cloud is to calculate the cloud's movement direction and speed based on the optical flow method, and to obtain the time that the cloud blocks the sunlight near the sun, thereby predicting the impact of the cloud on solar radiation power generation.

[0018] As a further improvement of this technical solution: it can be used to track multiple clouds;

[0019] The method for tracking multiple clouds is: multi-target tracking, which enables real-time prediction of multiple cloud-blocking events to obtain the probability of the sun being blocked. By directly measuring the cloud movement speed with the help of neuromorphic visual sensors, further prediction of cloud-blocking events can be made.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] The present invention applies neuromorphic visual sensors to atmospheric cloud detection. Because the event-driven sensor has a large dynamic range, data can be collected at night. The invention has the ability to observe in all weather conditions. Compared with traditional observation, the device has higher temporal resolution and dynamic range. It can identify and track targets more accurately by monitoring the movement of objects in the image, and the data volume is small, and the processing speed requirement is low. At the same time, it can also measure cloud cover and cloud speed, ultimately realizing the measurement of cloud cover and cloud movement speed in the entire sky and the prediction of the probability of clouds blocking the sun.

[0022] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and to implement it according to the contents of the description, the following preferred embodiments of the present invention are described in detail with reference to the accompanying drawings. The specific implementation methods of the present invention are given in detail by the following embodiments and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0024] Figure 1 Schematic diagram of the all-sky cloud imaging system based on neuromorphic vision sensor proposed in this invention. DETAILED DESCRIPTION

[0025] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not intended to limit the scope of the present invention. The following paragraphs describe the present invention in more detail by way of example with reference to the accompanying drawings. The advantages and features of the present invention will become more apparent from the following description and claims. It should be noted that the drawings are all in a very simplified form and are not in exact proportions. They are only used to facilitate and clearly illustrate the purpose of the embodiments of the present invention.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0027] See also Figure 1 ,In an embodiment of the present invention, a full sky cloud imaging system based on a neuromorphic visual sensor includes a full sky imager, a sun shading belt, a data acquisition module, and a data computing module;

[0028] The data acquisition module includes a hemispherical mirror and an event-driven sensor, which is used to obtain sky-related data. Sky-related data includes the outline of clouds in the sky, the direction of cloud movement, the distribution of clouds, and the position of the sun. It should be noted that:

[0029] Event-driven sensors only respond to moving objects, and clouds in the sky are rarely still. Therefore, when there are no clouds, specific algorithms will not generate a large number of images, thus reducing storage efficiency.

[0030] The features or advantages of the neuromorphic visual sensor of the present invention are:

[0031] 1) Low storage requirements and strong real-time performance. Traditional cameras can only take pictures at fixed intervals due to storage limitations, while neuromorphic vision sensors can take pictures in real time, and no moving objects in the sky will trigger an image.

[0032] 2) Strong environmental adaptability, high dynamic range, and can effectively suppress the impact of sun spots on the algorithm

[0033] 3) Expanding the detection capabilities of existing devices, it is possible to directly calculate the movement speed of the cloud based on a single sensor with pixel-level accuracy.

[0034] The all-sky imager uses an event-driven sensor for shooting. The event-driven sensor is installed on the support arm of the all-sky imager and is located above the center of the hemispherical mirror; the solar shading belt covers the surface of the hemispherical mirror; the data acquisition module is connected to the data calculation module through the transmission module.

[0035] Specifically, the data operation module includes a first operation module and a second operation module. The first operation module uses the data stream of the event-driven sensor as input to calculate and identify the target; the second operation module uses the target identification data of the first operation module as input to obtain the target's motion trajectory prediction.

[0036] It should be noted that the first operation module includes filtering and denoising and convolutional neural networks. Filtering and denoising uses homomorphic filtering to remove noise from the data stream and increase image contrast. The convolutional neural network uses a 6-layer structure to perform target recognition on the filtered and denoised image.

[0037] The second operation module includes a feature extractor and a motion model constructed by Kalman filtering. The feature extractor uses the data of the first operation module as input to obtain the similarity between the target features, and obtains the degree of feature matching between the target objects in the two data streams by measuring the similarity. The motion model constructed by Kalman filtering crops and extracts the local portraits of all targets in the detected target objects to predict the motion trajectory, and accurately associates the targets with each other by combining the Kalman filter motion model and the appearance model of the deep features. When the detection result is successfully associated with the tracking trajectory, the identity identification code ID of the target corresponding to the previous data stream is assigned to the target object with a high matching degree in the next data stream, and then the status of the target object is updated.

[0038] The appearance features and motion features are extracted through a single multi-border detection algorithm.

[0039] The calculation of the similarity mentioned above combines the appearance model and the motion model to achieve the optimal allocation function of the detection results and the tracker prediction results.

[0040] Specifically, all event data in the system include the coordinate position and time stamp of the triggering event.

[0041] Specifically, the method for tracking a cloud is to calculate the cloud's movement direction and speed based on the optical flow method, and then determine the time the cloud blocks the sunlight near the sun, thereby predicting the cloud's impact on solar radiation power generation.

[0042] The method for tracking multiple clouds is: multi-target tracking, which enables real-time prediction of multiple clouds blocking the sun and obtains the probability of the sun being blocked.

[0043] The main data generated on a clear day is the movement of the sun. This error can be processed by noise reduction or calculation of the solar zenith angle to determine that there are no clouds at this time.

[0044] The working principle of the present invention is:

[0045] The present invention provides a cloud identification and tracking technology based on event-driven sensors. The present invention applies neuromorphic visual sensors to atmospheric cloud detection. Because the dynamic range of event-driven sensors is large, data can be collected at night. The invention has the ability to observe in all weather conditions. Compared with traditional observations, the device has higher temporal resolution and dynamic range. It can identify and track targets more accurately by monitoring the movement of objects in the image. At the same time, it can also measure cloud cover and cloud speed, ultimately realizing the measurement of cloud cover and cloud movement speed in the entire sky and the prediction of the probability of clouds blocking the sun.

[0046] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any ordinary technician in this industry can smoothly implement the present invention as shown in the drawings and described above. However, any equivalent changes, modifications and evolutions made by technicians familiar with this profession without departing from the scope of the technical solution of the present invention using the technical content disclosed above are all equivalent embodiments of the present invention. At the same time, any equivalent changes, modifications and evolutions made to the above embodiments based on the essential technology of the present invention are still within the scope of protection of the technical solution of the present invention.

Claims

1. A full-sky cloud imaging system based on neuromorphic visual sensors, characterized in that: It includes a full sky imager, a sun shading belt, a data acquisition module and a data calculation module; The data acquisition module includes a hemispherical mirror and an event-driven sensor for acquiring sky-related data; The all-sky imager uses an event-driven sensor for shooting, and the event-driven sensor is arranged on a support arm of the all-sky imager, and the event-driven sensor is located above the center of the hemispherical mirror; the sunshade belt covers the surface of the hemispherical mirror; the data acquisition module is connected to the data calculation module through the transmission module; The data operation module includes a first operation module and a second operation module, wherein the first operation module uses the data stream of the event-driven sensor as input to calculate and identify the target; the second operation module uses the target identification data of the first operation module as input to obtain the target's motion trajectory prediction; The first operation module includes filtering and denoising and a convolutional neural network. The filtering and denoising adopts a homomorphic filtering method to remove noise in the data stream and increase image contrast. The convolutional neural network uses a 6-layer structure to perform target recognition on the filtered and denoised image. The second operation module includes a feature extractor and a motion model constructed by Kalman filtering. The feature extractor uses the data of the first operation module as input to obtain the similarity between the target features, and obtains the degree of feature matching between the target objects in the two previous and subsequent data streams by measuring the similarity; the motion model constructed by Kalman filtering crops and extracts local portraits of all targets in the detected target objects to predict the motion trajectory, and accurately associates the targets with each other by combining the Kalman filter motion model and the appearance model of the deep features. When the detection result is successfully associated with the tracking trajectory, the identity identification code ID of the target corresponding to the previous data stream is assigned to the target object with a high matching degree in the next data stream, and then the status of the target object is updated.

2. The all-sky cloud imaging system based on neuromorphic visual sensors according to claim 1, characterized in that: The sky-related data includes the outline of clouds in the sky, the direction of cloud movement, the distribution of clouds, and the position information of the sun.

3. The all-sky cloud imaging system based on neuromorphic visual sensors according to claim 1, characterized in that: The appearance features and motion features are extracted through a single multi-border detection algorithm.

4. The all-sky cloud imaging system based on neuromorphic visual sensors according to claim 3, characterized in that: The similarity calculation combines the appearance model and the motion model to achieve the optimal allocation function between the detection results and the tracker prediction results.

5. The all-sky cloud imaging system based on neuromorphic visual sensors according to claim 4, characterized in that: All event data in this system include the coordinate location and time stamp of the triggering event.

6. The all-sky cloud imaging system based on neuromorphic visual sensors according to claim 5, characterized in that: Can be used to track a cloud; The method for tracking a cloud is to calculate the cloud's movement direction and speed based on the optical flow method, and to obtain the time that the cloud blocks the sunlight near the sun, thereby predicting the impact of the cloud on solar radiation power generation.

7. The all-sky cloud imaging system based on neuromorphic visual sensors according to claim 6, characterized in that: Can be used to track multiple clouds; The method for tracking multiple clouds is: multi-target tracking, which enables real-time prediction of multiple clouds blocking the sun and obtains the probability of the sun being blocked.

Images