Actuator-based high-speed item registration system

By adopting an event-based visual sensor and processor system in warehouse management, the labor-intensive and untimely data update problems caused by manual operations are solved, and the automation and speed of item registration and classification are realized, and efficiency and accuracy.

CN118922844APending Publication Date: 2024-11-08SONY SEMICON SOLUTIONS CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380029707.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-30
Filing Date
2023-03-09
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In existing warehouse management, tasks such as item search and inventory inventory rely on manual operations, resulting in labor-intensive, time-consuming and high error rates and accident risks, and data updates are not timely and cannot effectively respond to complex needs.

Method used

A system that uses an event-based vision sensor (EVS) coupled with the processor is used to perform vision-based registration tasks, and optimize item registration and classification to achieve automation and high-speed by inferring the consistency of quality metrics and task output.

Benefits of technology

It improves the efficiency and accuracy of item registration and classification, reduces labor costs, realizes real-time data updates and secure automated operations, and is suitable for complex warehouse management tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118922844A_ABST
    Figure CN118922844A_ABST
Patent Text Reader

Abstract

A system (28, 29) includes circuitry configured to perform a vision-based enrollment task, the circuitry (28) coupled with an event-based vision sensor (26) and configured to adapt to an inferred quality (24) derived from an output of the event-based vision sensor (26) and / or to a consistency (34) of a task output (33) of the vision-based enrollment task.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure is generally in the field of computer vision. Background Art

[0002] Computer vision is concerned with how computers can gain high-level understanding from digital images or videos. From an engineering perspective, it attempts to understand and automate tasks that the human visual system can accomplish.

[0003] As the demands of the industrial equipment business become increasingly complex and diverse, the use of sensing to extract necessary information from images captured by cameras continues to grow, requiring more efficient data acquisition.

[0004] Currently, most warehouse management tasks, such as item searches, cycle counting and stock taking, are performed manually and are therefore labour-intensive. Manual stock taking is an example of an extremely time-consuming process. This activity requires human resources, who also require pallet trucks or forklifts. At the same time, it involves a high error rate and a non-trivial risk of accidents. The fundamental shortage of skilled workers and the increasing time pressure lead to costly mistakes. In addition, the manually collected data is often no longer up-to-date after a short time. Therefore, traditional methods are labour-intensive and not scalable. Summary of the invention

[0005] According to a first aspect, the present disclosure provides a system comprising a circuit configured to perform a vision-based registration task, the circuit being coupled to an event-based vision sensor and configured to adapt to a quality of inference derived from an output of the event-based vision sensor and / or to a consistency of a task output of the vision-based registration task.

[0006] According to another aspect, the present disclosure provides an EVS-based sensing subsystem, including circuitry configured to perform a vision-based registration task, the subsystem including an event-based vision sensor and circuitry configured to output an inference quality metric to a processor of the vision-based registration task system.

[0007] According to another aspect, the present disclosure provides a computer-implemented method for performing a vision-based registration task, the method comprising: obtaining the output of an event-based vision sensor, and adapting the vision-based registration task system to the inference quality derived from the output of the event-based vision sensor and / or to the task output of the vision-based registration task performed by the vision-based registration task system.

[0008] According to yet another aspect, the present disclosure provides a program comprising instructions, which are configured to execute the above method when operated by a processor.

[0009] Further aspects are set out in the dependent claims, the following description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Embodiments are explained by way of example with reference to the accompanying drawings, in which:

[0011] Figure 1 An example of warehouse inventory is shown;

[0012] Figure 2a Schematically illustrates an embodiment of an actuator-based optical registration system with an EVS-based sensing subsystem;

[0013] Figure 2b Schematically illustrates an embodiment of an actuator-based optical registration system having an EVS-based sensing subsystem including a conventional camera;

[0014] Figure 3 A flow chart showing a tag detection process performed in an actuator-based optical registration system with an EVS-based sensing subsystem;

[0015] Figure 4a and Figure 4b A possible implementation of an image quality metric that ensures that the reconstructed image contrast is significant is shown;

[0016] As an example, Figure 5 Shown are a sharp image (upper left image) and the presence of corresponding high frequency components in its Fourier transform (upper right image), as well as a blurred image (lower left image) and the corresponding small amplitude values ​​in the Fourier domain (lower right image);

[0017] Figure 6a The data flow obtained from the EVS sensor is schematically shown.

[0018] Figure 6b An example of accumulating events obtained from an EVS sensor in a channel is shown;

[0019] Figure 6c An example of time-bin interpolation of events obtained from an EVS sensor is shown;

[0020] Figure 6d An example of segmenting events obtained from an EVS sensor along a polarity dimension is shown;

[0021] Figure 7 A flow chart showing a tag detection process with tag presence triggering performed in an actuator-based optical registration system with an EVS-based sensing subsystem;

[0022] Figure 8A flow chart showing a tag detection process with detection and positioning of tags in an actuator-based optical registration system with an EVS-based sensing subsystem;

[0023] Fig. 9 shows the actuator-based optical registration system of FIG. 2 with an EVS-based sensing subsystem employed in an item registration task involving a conveyor belt;

[0024] Fig.10 Schematically illustrates an embodiment of an actuator-based optical registration system with an EVS-based sensing subsystem implemented as a scanning drone;

[0025] Fig.11 shows the actuator-based optical registration system of FIG. 4 with an EVS-based sensing subsystem employed in a warehouse scenario; and

[0026] Fig.12 The actuator-based optical registration system of FIG. 2 with an EVS-based sensing subsystem is shown employed in high-speed waste sorting. DETAILED DESCRIPTION

[0027] In reference Figures 1 to 12 Before describing in detail the embodiments of the present invention, a general description is given.

[0028] An embodiment discloses a system comprising a circuit configured to perform a vision-based registration task, the circuit being coupled to an event-based vision sensor and configured to adapt to a quality of inferences derived from an output of the event-based vision sensor and / or to a consistency of a task output of the vision-based registration task.

[0029] Vision-based registration tasks may include, for example, scanning item tags, object classification or registration, high-speed item counting, high-speed waste sorting, item defect monitoring, or extensions of any of the above, hereinafter summarized as "vision-based registration tasks".

[0030] Employing an actuator-based optical registration system with an EVS-based sensing subsystem may, for example, facilitate performing vision-based registration tasks in a high-speed manner.

[0031] Inference quality and confidence metrics that are adaptive to the objects being classified.

[0032] The circuit may include a processor. The processor may be, for example, a processor specialized for a specific task, such as a tensor processing unit, an image signal processor, or a field programmable gate array, but is not limited to these types of processors. The data processing may be performed, for example, by a processing unit of the sensing subsystem or by a processing unit included in an existing processing pipeline of the actuator system. The circuit or processor may also be configured to implement a neural network, such as a CNN or a DNN, etc.

[0033] The circuit may include memory, storage devices, input components, output components, light emitting diodes, etc., speakers, etc., interfaces, etc., as is generally known for electronic devices. In addition, the circuit may also include sensors for sensing still image or video image data, for sensing fingerprints, for sensing environmental parameters, etc.

[0034] An event-based vision sensor may, for example, be configured to observe a scene for which a vision-based registration task is to be performed.

[0035] The output of the sensor subsystem can be used, for example, for any combination of these applications: performing object registration, performing obstacle detection and avoidance, providing feedback to actuator control loops.

[0036] An advantage of this system, compared to deploying a sensing unit with a conventional imaging sensor that is decoupled from control of the actuator, may be the ability to find a state in which the entire system operates optimally with respect to the factor to be optimized (e.g., efficiency or throughput).

[0037] The circuit may, for example, be configured to determine a quality metric from the output of the event-based vision sensor, the quality metric being used in a feedback loop to control the actuator and thereby influence the quality of content sensed by the event-based vision sensor.

[0038] A quality metric can be anything that describes quality.

[0039] This motion generated by the actuator may affect whether and how well the vision-based registration task can be performed, and may therefore affect, for example, classification certainty.

[0040] Depending on an embodiment of the system, the system itself can be moved, or an object can be moved in its position relative to the system by an actuator that takes control input based on the inferred quality derived from the output of an event-based vision sensor and / or the consistency of the task output.

[0041] The object may move, for example, on a conveyor belt that takes control inputs based on the quality of inferences derived from the output of an event-based vision sensor and / or the consistency of the task output.

[0042] The actuator-based optical registration system may include an EVS-based sensing subsystem including an event-based vision sensor.

[0043] The EVS-based sensing subsystem may include circuitry for performing at least a portion of the vision-based registration task. The circuitry may, for example, include a processing unit that includes a processor dedicated to a particular task, such as a tensor processing unit, an image signal processor, or a field programmable gate array. The EVS-based sensing subsystem may also include circuitry, such as a central processing unit assisted by a graphics processing unit and a random access memory.

[0044] The system may be moved, or an object may be moved, by an actuator in terms of the position of the object relative to the system, which actuator takes control input based on inference quality derived from output of an event-based vision sensor and / or consistency of task output.

[0045] The system may include an EVS-based sensing subsystem including an event-based vision sensor, and wherein the EVS-based sensor subsystem uses events measured by the EVS sensor to reconstruct item tags and / or navigation cues.

[0046] The system may include an EVS-based sensing subsystem including an event-based vision sensor, and wherein the EVS-based sensor subsystem is configured to output an inferred quality metric.

[0047] The inferred quality may be derived, for example, from an image-based quality metric or a label-based quality metric.

[0048] The system may include a conveyor belt coupled to an EVS-based sensing subsystem, the movement of the conveyor belt being adapted to the quality of inferences derived from output of the event-based vision sensor and / or to the consistency of the task output.

[0049] For example, embodiments foresee mounting an actuator-based optical registration system with an EVS-based sensing subsystem on a conveyor belt for high-speed waste sorting in a recycling plant. Recyclable waste needs to be sorted, e.g., glass, plastic, parts containing metal, etc. This can be done manually or with an image-based vision system using conventional cameras, and is therefore a slow or cumbersome process.

[0050] The system may include an EVS-based sensing subsystem implemented on a scanning drone or robot, the movement of the scanning drone or robot being adapted to the quality of inferences derived from the output of an event-based vision sensor and / or to the consistency of the task output.

[0051] Using an EVS-based sensing subsystem can increase the robustness of vision-based positioning methods. As a result, automated robotic systems can complete tasks faster and more safely, which is an important part when scaling up to large numbers of items to be registered / processed in warehouse applications. For power limitations, autonomous systems can increase the number of tasks completed before the battery is exhausted, saving costs and time. A high degree of automation opens up the possibility of use during idle time. For example, the solution can be used when no one is left in the warehouse. Already established processes and workflows remain unchanged.

[0052] The item tag may be selected from a barcode, a QR code, and an APR tag, and the vision-based registration task may, for example, include a tag detection process.

[0053] The circuitry may be configured to transform an event representation of an event obtained from an event-based vision sensor.

[0054] The circuitry may also be configured to perform image reconstruction based on the event stream obtained by the event-based vision sensor.

[0055] Furthermore, the circuit can be configured to perform a specified task on the reconstructed image to obtain a task output.

[0056] In other embodiments, the circuit is configured to operate directly on the event stream without reconstructing the image from the event stream. This can be achieved, for example, using a spiking neural network (SNN).

[0057] Performing the specified task may, for example, include performing vision-based registration tasks, such as image label decoding, object classification, or 3D registration. This can be solved using existing and established algorithms directly on the reconstructed image or in direct event input. The task output 33 may, for example, be an image label, QR code, object marker, or point cloud that has been detected while performing the task. For example, a neural network-based algorithm may be used to scan a QR code based on contrast change information in the reconstructed image.

[0058] The circuitry may be configured to perform inference of image quality based on the reconstructed image to obtain an image quality metric.

[0059] The image quality metric may for example be a no-reference metric, which states that the reconstructed image will not be compared with a ground truth or absolute intensity image. The metric may for example be used to detect for example the presence of a label or the quality of the reconstructed image.

[0060] The circuitry may be configured to perform a consistency check on the task output to obtain a consistency metric.

[0061] A consistency metric may be anything that describes the consistency of a task output. A consistency metric may be, for example, a confidence value, such as a code confidence.

[0062] The circuitry may be configured to execute a control loop based on the image quality metric and / or the consistency metric.

[0063] For example, EVS-based feedback to the navigation control loop may be beneficial because the system does not have to stop in order to take a blur-free picture.

[0064] For example, the process may implement a control loop in which a vision-based registration task system uses an image quality metric in a feedback loop to control an actuator and thereby affect the quality of the content being sensed in the scene. For example, the control loop may act on an actuator of an external or internal system and attempt to optimize to increase the image quality metric.

[0065] The circuitry may be configured to infer the likelihood that the current frame contains an image tag, and / or the circuitry may be configured to locate the tag.

[0066] The absence of a tag is information that can be fed into a control loop to, for example, increase the moving speed of, for example, a conveyor belt, since no object is currently in the field of view of the camera subsystem.

[0067] Neural networks can be used, but not necessarily, to solve the problem of tag detection and localization.

[0068] Embodiments also disclose an EVS-based sensing subsystem, including circuitry configured to perform a vision-based registration task, the subsystem including an event-based vision sensor and circuitry configured to output an inference quality metric to a processor of the vision-based registration task system.

[0069] Embodiments also disclose a computer-implemented method for performing a vision-based registration task, the method comprising: obtaining an output of an event-based vision sensor, and adapting a vision-based registration task system to an inference quality derived from the output of the event-based vision sensor and / or to a task output of a vision-based registration task performed by the vision-based registration task system. The method may include all of the processes described herein.

[0070] The embodiment also discloses a program including instructions, which are configured to execute the above method when operated by a processor.

[0071] Assuming the system is coupled with a moving sensing unit or actuator of the object of interest, this would allow optimization for the best movement speed (derived from the image quality metric), where, for example, image labels can still be detected and recognized, but throughput is also increased compared to ordinary sensors that are limited in this area. Typically, objects on a conveyor belt move at a fixed speed, slow enough to ensure that there is no motion blur.

[0072] Event-based Vision Sensor (EVS)

[0073] Conventional cameras (e.g., those found in smartphones) function by acquiring a complete image of the entire scene at regular intervals at a specific frame rate, which is done by exposing all pixels of the image at the same time. However, with this technique, moving objects cannot be detected until the onboard computer has analyzed all pixels. With the frame-based approach used by conventional cameras, the entire image is output at specific intervals determined by the frame rate. Conventional camera frame rates are low and require good lighting conditions. Vision systems using conventional cameras or depth sensors are accurate (up to 5cm), but not fast. On the other hand, RF-based positioning technologies suffer from low accuracy (10-30cm).

[0074] For conventional cameras, the faster the sensor or object moves, the lower the SNR (signal-to-noise ratio) in the acquired image. Movement during the exposure time causes motion blur, obscuring, for example, tags to be detected and identified.

[0075] In contrast, event-based vision sensors (hereinafter also referred to as EVS sensors or simply EVS) utilize an event-based approach that asynchronously detects changes in pixel light brightness and outputs data with pixel position and time information, thereby achieving high-speed, low-latency data output. That is, the EVS sensor records changes in contrast with a very high temporal resolution. The EVS sensor has low latency (in microseconds) and a high dynamic range. The EVS sensor provides a much higher "frame rate" than traditional vision systems. Therefore, the EVS sensor is more robust to motion blur in adverse lighting scenes.

[0076] EVS sensors respond to brightness changes in the scene asynchronously and independently for each pixel. Pixels that do not detect a brightness change remain silent. When a pixel's brightness changes by more than a threshold, the camera sends an event, which is transmitted from the chip containing the location, time, and polarity of the change. Events are transmitted from the pixel array out of the camera, typically by using an Address Event Representation (AER) readout, using a shared digital output bus.

[0077] When the EVS sensor records changes in intensity (temporal contrast steps), no movement produces a fairly low SNR because it is difficult to separate the information from the background noise. The faster the object or camera moves, the higher the SNR, until other limits (e.g., bandwidth limitations) are reached.

[0078] In addition, the SNR of EVS also depends on the underlying texture of the region of interest. Flat (white) areas generate almost no events, regardless of motion, while contrast-rich areas generate a large number of events. Therefore, EVS is well suited for object label registration tasks.

[0079] The output of the EVS is a variable data stream of digital events, where each event represents a change in brightness of a predetermined magnitude at a pixel at a specific time. Compared to conventional cameras, the EVS sensor generates a sparse stream of events so that only a small fraction of all pixels in the image need to be processed by the onboard computer, significantly speeding up computations.

[0080] Inventory Registration

[0081] Figure 1 An example of warehouse inventory is shown. Inventory items 12 of the warehouse are located in pallet racks 11. The inventory items 12 have visual codes (e.g., QR codes or bar codes). Warehouses usually maintain a large number of such pallet racks 11 to store inventory.

[0082] According to an embodiment described in more detail below, automated drones are deployed to take over warehouse inventory applications, such as item search, inventory auditing, object classification / registration, cycle counting and inventory counting, and automated quality assurance. The drones are tasked with flying over the shelves, recording tags and other relevant location and object data, and automatically processing this information. Drones for warehouse inventory applications are equipped with cameras and fly over all shelves at different heights throughout the warehouse. The drones typically stop briefly at an item location to take a photo of a QR code to register the product. For example, automated drones can take over such warehouse inventory applications during off-hours.

[0083] Other applications where drones could be applied in a similar way are QR codes or quality registration on conveyor belts in manufacturing, drone inspection of fields in agriculture, spare parts delivery or sorting of objects on conveyor belts for sorting, for example, for sorting recyclable waste in recycling.

[0084] Drones can use conventional cameras and visual SLAM technology (simultaneous localization and mapping) to accomplish the above tasks. Although visual SLAM is very accurate, it requires wide aisles, and autonomous systems are usually only flown during off-hours to avoid accidents with workers or other obstacles. The use of SLAM algorithms enables drones to first build a map of the environment while determining their own location. This allows drones to navigate in unknown terrain without GPS or Wi-Fi.

[0085] Actuator-based optical registration system with EVS-based sensing subsystem

[0086] Embodiments described in more detail propose coupling an actuator-based optical registration system with an event-based vision sensor (EVS). Specifically, embodiments provide a subsystem comprising an event-based vision sensor (EVS) connected to a processing system running an algorithm. The subsystem is attached to any existing actuator-based optical registration system. Thus, embodiments provide a system for automatic high-speed item registration.

[0087] Embodiments describe performing vision-based object registration tasks in a high-speed manner, where the system itself can be moved by an actuator, or an object can be moved by an actuator at the position of the object relative to the system, and the actuator obtains its control input based on the quality or confidence of performing the registration task.

[0088] Figure 2a An embodiment of an actuator-based optical registration system with an EVS-based sensing subsystem is schematically shown. A vision-based registration task system 21 is configured to perform a registration task involving inventory items in a scene 22. The vision-based registration task system 21 also includes a sensing subsystem 23, which includes an EVS sensor 26 configured to observe the scene 22. The sensing subsystem 23 includes a processing unit 25 that communicates with the EVS sensor 26. The processing unit 25 may include a processor dedicated to a specific task, such as a tensor processing unit (TPU), an image signal processor (ISP), or a field programmable gate array (FPGA), but is not limited to these types of processors. According to an alternative embodiment, the sensing subsystem includes a central processing unit (CPU) assisted by a graphics processing unit (GPU) and a random access memory (RAM).

[0089] The sensing subsystem 23 uses the events measured by the EVS sensors 26 to reconstruct the item labels and any navigation cues, while outputting an inferred quality metric 24. That is, with the help of the EVS sensors 26, the processing unit 25 of the sensing subsystem 23 is able to output information related to, for example, item labels to be scanned (e.g., QR codes, APR tags), object classification or registration (3D scanning), high-speed item counting, item defect monitoring, or an extension of any of the above, which is summarized as "vision-based registration tasks" below. Based on the information obtained from the EVS sensors 26, the processing unit 25 of the sensing subsystem 23 determines the quality metric 24. This quality metric 24 is used in a feedback loop by the vision-based registration task system 21 to control the actuator 29 (e.g., a motor) and thus affect the quality of the content sensed in the scene 22. The sensing subsystem 23 is configured to output information related to the scan / inference quality or confidence metric and navigation cues (e.g., the system's position, posture, and any obstacles along the trajectory).

[0090] The output of the sensor subsystem can be used, for example, for any combination of these applications: performing object registration, performing obstacle detection and avoidance, providing feedback to actuator control loops.

[0091] exist Figure 2a In the embodiment of the present invention, the event-based visual sensor 26 operates according to the above-mentioned EVS principle. However, it should be noted that the event-based visual sensor 26 as used in the following embodiments can also be a hybrid sensor including conventional pixels (whose output is, for example, RGB or grayscale) and pixels that operate asynchronously according to the above-mentioned EVS principle. Such a hybrid sensor can operate according to the principle of time multiplexing (for example, some sensors have pixels that can change the operation mode and capture events or intensities) or according to the principle of spatial multiplexing (some sensors have both EVS and conventional pixels so that they can capture both information at the same time).

[0092] To solve vision-based registration tasks such as scanning item tags (e.g., QR codes), events are usually preprocessed first when the EVS senses intensity changes in reflectivity from objects on a logarithmic scale. Rule-based algorithms and learned mappings (neural network-based methods) are well described in the literature (see, for example, Zelin Zhang et al., "Image Reconstruction from Events. Why learn it?", Computer Science ArXiv, 2021; C. Scheerlinck et al., "Fast Image Reconstruction with an Event Camera", IEEE Winter Conference on Applications of Computer Vision, 2020). Most of these techniques aim to reconstruct images based on events before detecting and parsing tag information. In the presented embodiment, a neural network-based method will be used because it has been shown to be fast and accurate. The image reconstruction and tag decoding steps can be preceded by a fast tag localization algorithm that acts directly on the event.

[0093] The inference quality can be derived from an image-based quality metric (e.g., PSNR) or a tag-based quality metric (e.g., cross-reference if the tag is correctly registered and present in the database). When the image quality is not below a specified threshold, control feedback to the actuator system can be given, for example to increase the movement speed of the actuator (conveyor, robot). Note that any movement caused by the actuators in the system (moving reference frame) relative to the scene containing the tag (fixed reference frame) will affect whether and how well the sensor can perceive the tag, thereby affecting the classification certainty or inference quality, respectively.

[0094] NN-based algorithms can be used to infer navigation cues, thus detecting obstacles with very low latency. For all NN-based algorithms, there is no restriction on dedicated processing units, but it may prove helpful for inference speed. Events are converted into a convenient representation as input to the neural network. The output here can be bounding box coordinates including the position and size of the detected object.

[0095] exist Figure 2a In the example of , data processing is performed by the processing unit 25 of the sensing subsystem 23, and the actuation / control output is provided directly by the processing unit 25 of the sensing subsystem 23.

[0096] In alternative embodiments, if sufficient computing resources are available, data processing may also be incorporated into the existing processing pipeline of the actuator system (e.g., into the onboard processing unit 28), such as Figure 2b shown.

[0097] In addition, conventional cameras (e.g., RGB or grayscale) can be used in parallel with EVS-based sensors within an EVS-based sensing subsystem, such as Figure 2b shown.

[0098] Figure 2b An embodiment of an actuator-based optical registration system with an EVS-based sensing subsystem including an onboard processing unit and a conventional camera is schematically illustrated. Figure 2b The vision-based registration task system 21 and Figure 2a The difference in the implementation in is that Figure 2b The vision-based registration task system 21 includes an onboard processing unit 28 in communication with a conventional camera 27 and an actuator / motor 29. The sensing subsystem 23 is in communication with the onboard processing unit 28 of the vision-based registration task system 21. Figure 2b In an embodiment, data processing is performed by an onboard processing unit 28 of the actuator-based optical registration system, and actuation / control outputs are provided by an onboard processing unit 28 of the actuator-based optical registration system.

[0099] It should also be noted that Figure 2a , Figure 2b and Figure 3 The scene 22 in refers to a field (fixed coordinate system) that also contains the tags to be scanned, while the vision-based registration task system 21 depicts a moving reference system whose position and attitude can be changed by actuators. This movement affects whether and how well a tag can be perceived, and therefore affects the classification certainty of the tag.

[0100] Alternatively, the camera system can be rigidly mounted (in a fixed coordinate system) and the items containing the tags can be moved in a mobile reference system relative to the fixed coordinate system of the camera system. This is described in more detail in connection with the conveyor belt system with an EVS-based sensing subsystem described below.

[0101] Figure 3A flow chart of a tag detection process performed in an actuator-based optical registration system with an EVS-based sensing subsystem is shown. At 301, an event is acquired from the EVS sensor 26. At 302, the event representation of the event obtained from the EVS sensor 26 is transformed. One possible representation of the asynchronous and continuous event data stream is to convert it into a 2D frame using the pixel position of each event and its timestamp as the value of the pixel. Another option is to convert the incoming events in a certain time window or a fixed number of events into a 3D volume. The x-dimension and y-dimension represent the location of the event, while the third axis represents the arrival time of the event. This discretized 3D volume is also called a voxel grid or event grid. The representation transformation is adapted to the required input shape of the subsequent algorithm or neural network architecture. However, it should be noted that transforming the event data is optional. For example, if, for example, a spiking neural network is used, then transforming the event data may not be necessary. Associated with the following Figures 6a to 6d Additional details regarding this transformation of event streams are described.

[0102] The transformed event representation as obtained in 302 is fed to an algorithm or neural network 31. At 303, the algorithm or neural network 31 runs inference on the transformed event representation to obtain a reconstructed image 32. For example, one approach is to first reconstruct an intensity image from the temporal contrast changes (events), which can be done algorithmically by integrating the intensity changes over time or by using a CNN (convolutional neural network) based approach. Running inference on one event input can reconstruct multiple subsequent images based on the event representation. For example, a neural network based algorithm can be used to reconstruct the image.

[0103] At 304, the specified task is performed on the reconstructed image 32 to obtain a task output 33. Performing the specified task may, for example, include performing a vision-based registration task, such as image label decoding, object classification, or 3D registration. This can be solved using existing and established algorithms directly on the reconstructed image or in direct event input. The task output 33 may, for example, be an image label, a QR code, an object marker, or a point cloud that has been detected while performing the task. For example, a neural network-based algorithm may be used to scan a QR code based on contrast change information in the reconstructed image.

[0104] At 305, an inference of the image quality is performed based on the reconstructed image 32 to obtain an image quality metric 24. The image quality metric may be, for example, a no-reference metric, which means that the reconstructed image 32 will not be compared with a ground truth or absolute intensity image (since this setup does not require a conventional imaging sensor). This metric may then be used, for example, to detect the presence of, for example, a label or the quality (contrast) of the reconstructed image 32.

[0105] At 306, a consistency check is performed on the task output 33 to obtain a consistency metric 34 (e.g., code confidence). Given a certain confidence in the presence of an item tag / code from the image quality metric (by attempting to detect the code in the current frame, where a frame refers to an event that is transformed into a representation suitable as an input to a subsequent algorithm), additional information can be generated: the detected code does not (not) exist in the potential database, or the code detection failed. The consistency metric 34 can be fed to the control loop together with the image quality metric 24 to determine whether the actuator 29 should slow down or increase the object movement.

[0106] At 307, a control signal 36 is generated based on the consistency metric 34 and the image quality metric 24, and the control signal 36 is fed to an actuator 29 of an external or internal system (e.g., a conveyor or a robot motor). At 308, the actuator 29 moves the system based on the control signal 36 received from the generated control signal 307 to affect the scene. This process implements a control loop (feedback loop) in which the vision-based registration task system 21 uses the image quality metric 24 in the feedback loop to control the actuator 29 and thereby affect the quality of the content being sensed in the scene 22. For example, the control loop acting on the actuator 29 of the external or internal system will attempt to optimize to increase the image quality metric 24. For example, if a potential target (code / label) is not detected, the conveyor moves faster because there may be no events in the field of view of the camera system. Another strategy may be to try to increase the speed while maintaining image quality and successfully reading the code to dynamically optimize the throughput of the vision-based registration task.

[0107] exist Figure 3 In an embodiment, an algorithm or neural network 31 performs inference on the event representation to produce a reconstructed image 32. However, this process is optional. It is conceivable that the task can be solved directly based on the event ( Figure 3 304), such as detecting and identifying tags without reconstructing the intensity image.

[0108] Figure 4a and Figure 4b A possible implementation of an image quality metric that ensures that the reconstructed image contrast is apparent (since barcodes, QR codes, April tags, etc. are combinations of black and white geometric shapes) is shown. In this example, inferring the image quality at 305 includes determining a histogram of image intensity. The image quality metric 24 is represented by a histogram of image intensity. The abscissa of the histogram shows intensity bins from 0 to 255, while the ordinate shows the cumulative number of pixel intensities for each corresponding intensity bin. The histogram of image intensity shows two intensity peaks around the dark and bright values ​​of the code, as well as a fairly centered and narrow distribution for a flat and poorly contrasted image. Figure 4aA rather flat image without a QR code is shown. Figure 4b An image with obvious contrast in which a code label is present is shown. Therefore, the detection of QR codes in digital images can be based on histogram similarity. For example, an algorithmic approach is to calculate the RMS contrast, which involves looking at the standard deviation of the pixel intensity of the histogram (for Figure 4a is low and for Figure 4b is high).

[0109] Another example of determining image quality metrics is to look at the 2D frequency domain of pixel intensity. An image containing code / image labels will have different frequency peaks in the frequency domain.

[0110] Another option for no-reference image quality metrics is to evaluate the reconstructed image in the frequency domain by applying a Fourier transform to the reconstructed image. This allows confirmation of the sharpness of the image by looking at the magnitude of the values. The lack of high frequency components indicates that the image intensity does not change rapidly along the gradient and therefore does not have sharp transitions (which is preferred for label / code detection tasks).

[0111] As an example, Figure 5 A sharp image (upper left image) and the corresponding presence of high frequency components in its Fourier transform (upper right image) as well as a blurred image (lower left image) and the corresponding small amplitude values ​​in the Fourier domain (lower right image) are shown.

[0112] Other options for determining a no-reference image quality metric may include adding a conventional image sensor that records images at a lower frequency but provides a reference for other image quality metrics, such as the Structural Similarity Index metric (SSIM) or Peak Signal-to-Noise Ratio (PSNR) or more human perception metrics, such as the VGG loss (VGG = Visual Geometry Group) or the LPIPS loss (LPIPS = Learned Perceptual Patch Similarity).

[0113] Figures 6a to 6d It shows that it can be Figure 3 Example of an event representation transformation performed at 302 of . In order to process sparse and continuous event data streams obtained from EVS sensors, the sensor raw data is transformed. One possibility is to use a specialized representation called a voxel grid (or event grid) that represents events in a fixed grid that can be input into a standard network. A voxel grid is a 3D volume in which events are segmented into specific layers / channels according to their timestamps. A voxel grid (event grid) can be created by a combination of the following methods. A predetermined number of events or fixed time windows are defined, and based on these time windows, time is segmented into channels (time bins). The event pixel values ​​are then interpolated based on the exact arrival timestamps between two channels. Optionally or alternatively, there is the possibility to segment event values ​​along a polar dimension.

[0114] Figure 6a The data flow obtained from the EVS sensor is schematically shown. The three-dimensional graph is plotted at time t 0 and t 1 The abscissa of the three-dimensional graph shows the time at which the event was captured. The ordinate x* and the depth axis y show the pixel position at which the event was captured.

[0115] Figure 6b An example of accumulating events obtained from an EVS sensor in channels is shown. A predetermined number of events or a fixed time window is defined. Thus, time is divided into a predetermined number of channels (time bins), here for example 5 channels.

[0116] Figure 6c An example of time-bin interpolation of events obtained from an EVS sensor is shown. The event pixel value is interpolated based on its exact arrival timestamp between two channels.

[0117] Figure 6d An example of segmenting events obtained from an EVS sensor along the polarity dimension is shown. Events obtained from the sensor event stream have different polarities. The polarity can be represented, for example, as a 1-bit polarity p of a brightness change that decodes to a brightness increase or brightness decrease. Figure 6d In the example of , all events with positive polarity (increase in brightness) are grouped together, and all events with negative polarity (decrease in brightness) are grouped together.

[0118] Figure 7 A flow chart of a tag detection process with tag presence triggering performed in an actuator-based optical registration system with an EVS-based sensing subsystem is shown. Figure 7 Steps 701, 702, 703 and 704 in correspond to Figure 3 , and therefore, will not be described in detail for the sake of brevity. Similarly, Figure 7 Steps 706, 707, 708 and 709 in correspond to Figure 3 Steps 304, 306, 307 and 308 in , and therefore are not described again. However, Figure 7 The process shown is different from Figure 3 The process shown, because Figure 7 The process shown includes an additional step 705 of inferring the likelihood that the current frame contains an image tag. At 704, the same method as in Figure 3In a similar manner as at 305 in , an image quality metric 24 is derived by performing an inference of the image quality based on the reconstructed image 32. At 705, based on the image quality metric 24 obtained in 704, it is determined whether the reconstructed image contains a label. The rest of the pipeline is triggered only by the presence of such a label. If it is determined at 705 that the image contains a label, the process continues with solving task 706 with the aid of the neural network 35. On the other hand, if it is determined at 705 that the image does not contain a label, the process returns to step 701. That is, the absence of a label is information that can be fed into a control loop to, for example, increase the moving speed of, for example, a conveyor belt, because no object is currently in the field of view of the camera subsystem.

[0119] Figure 8 A flow chart of a tag detection process with detection and positioning of tags in an actuator-based optical registration system with an EVS-based sensing subsystem is shown. Figure 8 Steps 801, 804, 805, 806, 807, 808, 809 and 810 in correspond to Figure 3 , and therefore, will not be described in detail for the sake of brevity.

[0120] However, Figure 8 The process shown is similar to Figure 3 The process shown differs in that Figure 8 The process shown includes an additional step 802 after step 801. At 801, events are acquired from the EVS sensor 26. The acquired events are fed to the algorithm or neural network 38. At 802, the algorithm or neural network 38 runs inference on the acquired events to locate tags. If it is determined at 803 that tags are present, then at 804, the event representation of the events obtained from the EVS sensor 26 is transformed similar to Figure 3 On the other hand, if it is determined at 803 that no tag exists, the process returns to step 801. A neural network may be used, but does not necessarily have to be used, to solve the detection and positioning of tags.

[0121] Algorithmically, the detection of a barcode or item tag can include counting the number of positive and negative events. If the number exceeds a certain threshold, the rest of the pipeline is triggered.

[0122] Another rule-based approach is to detect corner points directly on the event stream. If a certain density of corner points is detected in an event cluster, then the region with potential item labels can be assumed.

[0123] Spiking neural networks are also a potential neural network-based approach because they are designed to process asynchronous data streams, such as events. Spiking neural networks can be used to detect and locate code tags.

[0124] Conveyor belt system with EVS-based sensing subsystem

[0125] Embodiments incorporating conveyor-based systems are described below.

[0126] Fig. 9 The actuator-based optical registration system of FIG. 2 with an EVS-based sensing subsystem is shown employed in an item registration task involving a conveyor belt. A vision-based registration task system 21 (e.g., as described above in FIG. 2 ) is configured to perform an object classification task within a scene 22 including an object 12 with an item tag 43 on a conveyor belt 41. The vision-based registration task system 21 includes a sensing subsystem 23 having an EVS sensor that observes the scene 22. The sensing subsystem 23 communicates with an onboard processing unit 28 of the vision-based registration task system 21. The output of the sensing subsystem 23 is used by the vision-based registration task system 21 in a feedback loop to control (via an actuator control interface 29 of the vision-based registration task system 21) a motor 42 of the conveyor belt 41, and thereby affect the quality of what is sensed by the vision-based registration task system 21 in the scene 22. The motor 42 drives the conveyor belt 41 and thereby determines the speed at which objects move on the conveyor belt 41.

[0127] For example, if the sensing subsystem 23 of the vision-based registration task system 21 determines that a fast and high-quality scan of an item tag 43 has been captured by the EVS sensor of the sensing subsystem 23, it can infer that the conveyor belt 41 can move faster without negatively affecting image acquisition.

[0128] By providing the high speed subsystem described above, item labels 43 (e.g., barcodes or QR codes) can be scanned at much higher speeds without the need for an external light source. The subsystem gives feedback on the quality of the scanned label and whether it was not successfully registered due to wear or defective markings. This feedback can be passed to the actuator 32 of the main system, which allows the conveyor speed to be set dynamically. When it is determined that the scanned label quality is sufficient, the speed and thus the throughput of the entire factory or warehouse can be increased.

[0129] It should be noted that in addition to the EVS sensors, the actuator-based optical registration system of FIG. 2 with an EVS-based sensing subsystem may also apply a conventional frame-based camera mounted on a track with a possible flash or external light source to improve the camera's SNR.

[0130] EVS-based sensing subsystem on a scanning drone

[0131] Another embodiment is characterized by an EVS-based sensing subsystem on a scanning drone. Using the technology provided by the embodiment, drones can more easily complete indoor navigation in warehouses.

[0132] The EVS-based sensing subsystem on a scanning drone may, for example, feature a forward-facing camera. The drone moves laterally from item to item across multiple shelves in a large warehouse. With only a forward-facing camera, the drone would need to stop quickly for each item to ensure an unobstructed image of the scanned tag.

[0133] Fig.10 Schematically illustrated is an embodiment of an actuator-based optical registration system with an EVS-based sensing subsystem implemented as a scanning drone. Fig.10 The vision-based registration task system 21 is similar to the vision-based registration task system 21 of FIG. 2 . Fig.10 The vision-based registration task system 21 is configured to perform a registration task involving inventory items in a scene 22. The vision-based registration task system 21 includes an onboard processing unit 28 in communication with a conventional camera 27 and an actuator 29. The vision-based registration task system 21 also includes a sensing subsystem 23 in communication with the onboard processing unit 28 of the vision-based registration task system 21. The sensing subsystem 23 is implemented on a scanning drone. The sensing subsystem 23 deploys three EVS sensors 52, 53, 54, one forward 52 and two side 53, 54.

[0134] 2, the sensing subsystem 23 includes a processing unit 25 in communication with an EVS sensor 26. The sensor subsystem 23 uses events measured by the EVS sensor 26 to perform a vision-based registration task, as described in more detail above in connection with FIG2. Based on information obtained from the EVS sensor 26, the processing unit 25 of the sensing subsystem 23 determines a quality metric 24. The quality metric 24 is used by the vision-based registration task system 21 in a feedback loop to control an actuator 29 and, therefore, affect the quality of the content being sensed in the scene 22.

[0135] Fig.11 Shows the use of Fig.10An actuator-based optical registration system with an EVS-based sensing subsystem. Scene 22 includes a plurality of inventory items 12 of a warehouse located within a pallet rack 11. The inventory items 12 have visual codes (e.g., QR codes or bar codes). The EVS-based sensing subsystem is implemented on a scanning drone 51. The scanning drone 51 features three EVS cameras, a forward camera 52 and two side cameras 53, 54. The forward EVS camera 52 provides obstacle avoidance and additional high-speed navigation cues, which allows the drone to be deployed during working hours rather than just when the warehouse is empty. When the event encodes all motion information, the side EVS camera is configured to continuously detect and scan item tags or QR codes without stopping the drone. By feeding back the image quality, the flight speed of the drone 51 can be adjusted, for example. Therefore, the sensing subsystem (23 in FIG. 2) can allow a much higher throughput of the drone 51, scanning more items 12 in a shorter time, thereby contributing to reducing downtime due to the need to charge the battery.

[0136] like Fig.10 and Fig.11 As shown, EVS sensors can be deployed to register all products while flying over the shelves at high speed. Using EVS sensors, drones do not have to maintain a stable position when registering barcodes, while motion blur occurs in the case of conventional cameras. This allows for rapid registration of items or labels (e.g., QR codes). In addition, EVS allows for the detection of obstacles in the trajectory. Specifically, EVS sensors can be used for autonomous registration of moving items by drones, where rapid motion and challenging lighting conditions are common. Drones require less space and less precise motion planning to avoid obstacles. In addition, the application of EVS sensors may require drones to be charged less frequently, allowing them to complete their paths faster, which saves cost and electricity. The proposed autonomous system is less limited by batteries and their onboard vision systems in terms of range and flight time in terms of completing its mission. That is, according to an embodiment, the warehouse scanning coverage can be increased.

[0137] On top Fig.10 and Fig.11 In the embodiment of the present invention, the sensing subsystem based on EVS implemented on a scanning drone is shown. However, it should be noted that in alternative embodiments, the sensing subsystem based on EVS can also be implemented on a robot or the like.

[0138] High-speed waste sorting

[0139] Yet another embodiment foresees mounting an actuator-based optical registration system with an EVS-based sensing subsystem on a conveyor belt for high-speed waste sorting in a recycling plant. Recyclable waste needs to be sorted, e.g., glass, plastic, parts containing metal, etc. This can be done manually or with an image-based vision system using conventional cameras, and is therefore a slow or cumbersome process.

[0140] Employing an actuator-based optical registration system with an EVS-based sensing subsystem as described in more detail below, high-speed waste sorting is facilitated and adaptive to inferred quality and confidence metrics of the sorted objects.

[0141] Fig.12 The actuator-based optical registration system of FIG. 2 with an EVS-based sensing subsystem is shown employed in high-speed waste sorting.

[0142] The scene 62 includes a plurality of waste items 61a, 61b, 61c located on a conveyor belt 63. The waste items 61a, 61b, 61c are of different types. The waste item 61a is of a first type, as represented by a square. The waste item 61b is of a second type, as represented by a circle. The waste item 61c is of a third type, as represented by a triangle. The waste items 61a, 61b, 61c may be of different sizes. An item tag scanning system 64 having an EVS-based scanning subsystem is arranged near the conveyor belt 63. The item tag scanning system 64 may be implemented, for example, as described in FIG. 2 above. The item tag scanning system 64 featuring an EVS sensor is configured to perform object classification tasks on a conveyor belt 41 within the scene 62. The item tag scanning system 64 includes a sensing subsystem (see 23 in FIG. 2 ) having an EVS sensor that observes the scene 62. The sensing subsystem of the item tag scanning system 64 communicates with an onboard processing unit (see 28 in FIG. 2 ) of the item tag scanning system 64. The output of the sensing subsystem is used in a feedback loop by the item tag scanning system 64 to control the motor 65 of the conveyor belt 63 and thus influence the quality of what is sensed in the scene 62 by the item tag scanning system 64. The motor 65 drives the conveyor belt 63 and thus determines the speed at which the waste items 61a, 61b, 61c move on the conveyor belt 63. The system may, for example, be mounted face down on the conveyor belt 63, with the picked up recycling waste moving forward and being sorted by a robotic arm (not shown in FIG6 ).

[0143] Depending on the classification confidence, the conveyor belt can be accelerated or decelerated, thereby increasing throughput. For example, if the sensing subsystem of the item tag scanning system 64 determines that a fast and high-quality scan of the waste items 61a, 61b, 61c has been captured by the EVS sensor of the item tag scanning system 64, it can be inferred that the conveyor belt 63 can move faster without negatively affecting the image acquisition. This can allow the waste sorting process to be significantly accelerated. The feedback loop makes the system adaptive to the inferred quality and confidence metrics of the objects being classified.

[0144] Note that this technique can also be configured as follows:

[0145] (1) A system (25, 28, 29) comprising circuitry configured to perform a vision-based registration task, the circuitry (28) being coupled to an event-based vision sensor (26) and configured to adapt to an inference quality (24) derived from an output of the event-based vision sensor (26) and / or to a consistency (34) of a task output (33) of the vision-based registration task.

[0146] (2) The system of (1), wherein the circuit is configured to find a state in which the system optimally performs the vision-based registration task.

[0147] (3) A system according to (1) or (2), wherein the circuit is configured to determine a quality metric (24) based on the output of the event-based vision sensor, and the quality metric (24) is used in a feedback loop to control the actuator (29) and thereby affect the quality of the content sensed by the event-based vision sensor (26).

[0148] (4) A system according to any one of (1) to (3), wherein the system itself can be moved by an actuator (29), or an object can be moved by an actuator (29) in the position of the object relative to the system, and the actuator obtains control input based on the inferred quality (24) derived from the output of the event-based vision sensor (26) and / or the consistency (34) of the task output (33).

[0149] (5) A system according to any one of (1) to (4), wherein the actuator-based optical registration system includes an EVS-based sensing subsystem (23), which includes an event-based visual sensor (26).

[0150] (6) A system according to any one of (1) to (5), wherein the system (51) is moved by an actuator (29), or an object (41, 63) is moved by an actuator (29) in a position of the object (41, 63) relative to the system, and the actuator obtains a control input (36) based on an inferred quality (24) derived from the output of an event-based vision sensor (26) and / or a consistency (34) of the task output (33).

[0151] (7) A system according to any one of (1) to (6), comprising an EVS-based sensing subsystem (23), the EVS-based sensing subsystem (23) including an event-based visual sensor (26), and wherein the EVS-based sensor subsystem (23) uses events measured by the EVS sensor (26) to reconstruct object tags and / or navigation cues.

[0152] (8) A system according to any one of (1) to (7), comprising an EVS-based sensing subsystem (23), the EVS-based sensing subsystem (23) comprising an event-based visual sensor (26), and wherein the EVS-based sensor subsystem (23) is configured to output an inference quality metric (24).

[0153] (9) A system according to any one of (1) to (8), comprising a conveyor belt (41) coupled to an EVS-based sensing subsystem (23), wherein movement of the conveyor belt (41) is adapted to an inferred quality (24) derived from output of an event-based visual sensor (26) and / or to a consistency (34) of a task output (33).

[0154] (10) A system according to any one of (1) to (9), wherein the EVS-based sensing subsystem is implemented on a scanning drone (51) or a robot, and the movement of the scanning drone (51) or the robot is adapted to the inference quality (24) derived from the output of the event-based visual sensor (26) and / or to the consistency (34) of the task output (33).

[0155] (11) The system of any one of (1) to (10), wherein the item tag is selected from a barcode, a QR code, and an APR tag, and wherein the vision-based registration task includes a tag detection process.

[0156] (12) The system of any one of (1) to (11), wherein the circuit is configured to transform (302) an event representation of an event obtained from an event-based vision sensor (26).

[0157] (13) The system of any one of (1) to (12), wherein the circuit is configured to perform image reconstruction (303) based on an event stream obtained by an event-based vision sensor (26).

[0158] (14) A system according to any one of (1) to (13), wherein the circuit is configured to perform (304) a specified task on the reconstructed image (32) to obtain a task output (33).

[0159] (15) A system according to any one of (1) to (14), wherein the circuit is configured to perform (305) inference of image quality based on the reconstructed image (32) to obtain an image quality metric (24).

[0160] (16) A system according to any one of (1) to (15), wherein the circuit is configured to perform (306) a consistency check on the task output (33) to obtain a consistency metric (34).

[0161] (17) A system according to any one of (1) to (16), wherein the circuit is configured to execute the control loop based on the image quality metric (24) and / or the consistency metric (34).

[0162] (18) A system according to any one of (1) to (17), wherein the circuit is configured to infer (705) the likelihood that the current frame contains an image tag, and / or the circuit is configured to locate the tag.

[0163] (19) An EVS-based sensing subsystem comprising circuitry configured to perform a vision-based registration task, the subsystem comprising an event-based vision sensor (26) and circuitry configured to output an inferred quality metric (24) to a processor (28) of a vision-based registration task system (21).

[0164] (20) A computer-implemented method for performing a vision-based registration task, the method comprising: obtaining (301) the output of an event-based vision sensor (26), and adapting a vision-based registration task system (21) to an inference quality (24) derived from the output of the event-based vision sensor (26) and / or to a task output (33) of a vision-based registration task performed by the vision-based registration task system (21).

[0165] (21) A program comprising instructions, the instructions being configured to execute the method according to (20) when operated by a processor.

[0166] Reference numerals

[0167] 11 Pallet rack

[0168] 12 Inventory Items

[0169] 21Vision-based registration task system

[0170] 22, 62 scenes

[0171] 23 Sensing subsystem

[0172] 24 Image Quality Metrics

[0173] 25 processing units

[0174] 26EVS Sensor

[0175] 27 Camera

[0176] 28 onboard processing units

[0177] 29 Actuator

[0178] 32 reconstructed images

[0179] 34Consistency Metrics

[0180] 31, 35, 38 Neural Network (NN)

[0181] 36 control signals

[0182] 41, 63 conveyor belt

[0183] 42, 65 conveyor belt control motor

[0184] 43 Item Tags

[0185] 51 UAV

[0186] 52 UAV forward facing EVS camera

[0187] 53, 54 Side-facing EVS camera of drone

[0188] 61a, 61b, 61c Waste items

[0189] 64 Item Tag Scanning System

Claims

1. A system comprising circuitry configured to perform a vision-based registration task, the circuitry being coupled to an event-based vision sensor and configured to adapt to a quality of inferences derived from an output of the event-based vision sensor and / or to a consistency of a task output of the vision-based registration task.

2. The system according to claim 1, wherein: The circuitry is configured to find a state in which the system optimally performs the vision-based registration task.

3. The system according to claim 1, wherein: The circuit is configured to determine a quality metric based on the output of the event-based vision sensor, the quality metric being used in a feedback loop to control an actuator and thereby affect the quality of content sensed by the event-based vision sensor.

4. The system according to claim 1, wherein: The system itself can be moved by an actuator, or an object can be moved by an actuator in terms of the position of the object relative to the system, the actuator taking control input based on the inference quality derived from the output of the event-based vision sensor and / or the consistency of the task output.

5. The system according to claim 1, wherein: The actuator-based optical registration system includes an EVS-based sensing subsystem, which includes the event-based vision sensor.

6. The system according to claim 1, wherein: The system is moved by an actuator, or an object is moved by an actuator at a position of the object relative to the system, the actuator taking control input based on the inference quality derived from the output of the event-based vision sensor and / or the consistency of the task output.

7. The system of claim 1, comprising an EVS-based sensing subsystem, the EVS-based sensing subsystem comprising the event-based vision sensor, and wherein, The EVS-based sensor subsystem uses events measured by the EVS sensors to reconstruct item tags and / or navigation cues.

8. The system of claim 1, comprising an EVS-based sensing subsystem, the EVS-based sensing subsystem comprising the event-based vision sensor, and wherein, The EVS-based sensor subsystem is configured to output an inferred quality metric.

9. The system of claim 1 , comprising a conveyor belt coupled to an EVS-based sensing subsystem, the movement of the conveyor belt being adapted to the quality of inferences derived from the output of the event-based vision sensor and / or to the consistency of task output.

10. The system according to claim 1, wherein: The EVS-based sensing subsystem is implemented on a scanning drone or robot whose movement is adapted to the quality of inferences derived from the output of the event-based vision sensor and / or to the consistency of task output.

11. The system according to claim 1, wherein: The item tag is selected from a barcode, a QR code, and an APR tag, and wherein the vision-based registration task includes a tag detection process.

12. The system of claim 1, wherein: The circuit is configured to transform an event representation of an event obtained from the event-based vision sensor.

13. The system of claim 1, wherein: The circuit is configured to perform image reconstruction based on a stream of events obtained by the event-based vision sensor.

14. The system of claim 1, wherein: The circuit is configured to perform a specified task on the reconstructed image to obtain a task output.

15. The system of claim 1, wherein: The circuit is configured to perform an inference of image quality based on the reconstructed image to obtain an image quality metric.

16. The system of claim 1, wherein: The circuit is configured to perform a consistency check on the task output to obtain a consistency metric.

17. The system of claim 1, wherein: The circuit is configured to execute a control loop based on an image quality metric and / or a consistency metric.

18. The system of claim 1, wherein: The circuit is configured to infer the likelihood that the current frame contains an image tag, and / or the circuit is configured to locate the tag.

19. An EVS-based sensing subsystem comprising circuitry configured to perform a vision-based registration task, the subsystem comprising an event-based vision sensor and circuitry configured to output an inference quality metric to a processor of the vision-based registration task system.

20. A computer-implemented method for performing a vision-based registration task, the method comprising: An output of an event-based vision sensor is acquired and a vision-based registration task system is adapted to a quality of inferences derived from the output of the event-based vision sensor and / or to a task output of the vision-based registration task performed by the vision-based registration task system.

21. A program comprising instructions configured to perform the method according to claim 20 when operated by a processor.