An event image output method, device, equipment and readable storage medium

Abstract

The present application provides an event image output method, apparatus, device and readable storage medium. The method includes: when the number of frames of event data included in the buffered event stream reaches the target number of frames N, for each new frame of event data generated thereafter, performing an OR operation on the new event data and the binary values at the same pixel positions of the N-1 frames of event data before the new event data to obtain merged event data; mapping the target pixel positions where the binary values in the new event data are the first value to the merged event data, and counting the number of first values of the first pixel array including the target pixel positions in the merged event data; performing denoising processing on the binary values at the target pixel positions in the merged event data based on the number of first values to obtain denoised event data; and outputting an event image based on the denoised event data. Thus, while not reducing the frame rate, the imaging quality of the event image is improved, and it is beneficial to reduce the computational amount of the denoising operation.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and particularly to an event image output method, apparatus, device, and readable storage medium. Background Art

[0002] With the continuous development of science and technology, computer vision technology has become increasingly mature. The emergence of event cameras has attracted more and more attention in the field of vision. It simulates the human retina and responds to pixel point pulses of brightness changes caused by motion. Therefore, it can capture the brightness changes of a scene at an extremely high frame rate, record events at specific time points and specific positions in an image, and form an event stream rather than a frame stream, thus solving problems such as information redundancy, large data storage, and real-time processing of traditional cameras.

[0003] However, the event cameras provided in the related art do not have asynchronous sensors. Therefore, the event images captured by the event cameras usually contain a large amount of noise. And noise filtering of event images is one of the important means to ensure image quality. The existing event image denoising technologies generally have defects such as complex operations, large delays, and low frame rates, and the denoising effect in practical applications is poor. Summary of the Invention

[0004] The embodiments of the present application provide an event image output method, apparatus, device, and readable storage medium, which can at least solve the problems of complex operations, large delays, and low frame rates existing in the event image denoising technology provided in the related art.

[0005] The first aspect of the embodiments of the present application provides an event image output method, which is applied to an event camera and includes:

[0006] When the number of frames of event data included in the buffered event stream reaches the target number of frames N, for each new frame of event data generated thereafter, perform an OR operation on the binary values of the same pixel positions of the new event data and the N-1 frames of event data before the new event data to obtain merged event data; wherein, the binary value includes a first value and a second value, the first value represents that an event occurs at the pixel, and the second value represents that no event occurs at the pixel;

[0007] Map the target pixel positions in the new event data where the binary values are the first value to the merged event data, and count the number of first values of the first pixel array including the target pixel positions in the merged event data;

[0008] Perform denoising processing on the binary values of the target pixel positions in the merged event data based on the number of first values to obtain denoised event data;

[0009] Output an event image based on the denoised event data.

[0010] In a second aspect of the embodiments of the present application, an event image output device is provided, which is applied to an event camera and includes:

[0011] A merging module, configured to, when the number of frames of event data included in the buffered event stream reaches the target number of frames N, perform an OR operation on the binary values at the same pixel positions of the new event data and the N-1 frames of event data before the new event data for each subsequent frame of new event data, to obtain merged event data; wherein, the binary values include a first value and a second value, the first value represents that an event occurs at a pixel, and the second value represents that no event occurs at a pixel;

[0012] A statistics module, configured to map the target pixel positions where the binary values in the new event data are the first value to the merged event data, and count the number of first values of a first pixel array including the target pixel positions in the merged event data;

[0013] A denoising module, configured to perform denoising processing on the binary values at the target pixel positions in the merged event data based on the number of first values, to obtain denoised event data;

[0014] An output module, configured to output an event image based on the denoised event data.

[0015] In a third aspect of the embodiments of the present application, a terminal device is provided, including: a memory and a processor, wherein the processor is configured to execute a computer program stored on the memory, and when the processor executes the computer program, implement the steps in the event image output method provided in the first aspect of the embodiments of the present application.

[0016] In a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, implement the steps in the event image output method provided in the first aspect of the embodiments of the present application.

[0017] As can be seen from the above, according to the event image output method, device, equipment and readable storage medium provided by the solution of the present application, when the number of frames of event data included in the buffer event stream reaches the target number of frames N, for each new frame of event data generated thereafter, perform an OR operation on the binary values at the same pixel positions of the new event data and the N-1 frames of event data before the new event data to obtain merged event data; map the target pixel positions where the binary values in the new event data are the first value to the merged event data, and count the number of first values of the first pixel array including the target pixel positions in the merged event data; perform denoising processing on the binary values of the target pixel positions in the merged event data based on the number of first values to obtain denoised event data; output an event image based on the denoised event data. Thus, while not reducing the frame rate, the imaging quality of the event image is improved, and it is beneficial to reduce the computational amount of the denoising operation and the image output delay. Description of the Drawings

[0018] Figure 1 It is a schematic diagram of the basic process of an event image output method provided by the first embodiment of the present application;

[0019] Figures 2a to 2d They are schematic diagrams of 4 frames of event data output in sequence from time t1 to time t4 provided by the first embodiment of the present application, Figure 2e It is a schematic diagram of the merged event data provided by the first embodiment of the present application;

[0020] Figure 3 It is a schematic diagram of the first pixel array provided by the first embodiment of the present application;

[0021] Figure 4 It is a schematic diagram of the denoised event data provided by the first embodiment of the present application;

[0022] Figure 5 It is a schematic diagram of the third pixel array provided by the first embodiment of the present application;

[0023] Figure 6 It is a schematic diagram of the refined process of an event image output method provided by the second embodiment of the present application;

[0024] Figure 7 It is a schematic diagram of the program modules of the event image output device provided by the third embodiment of the present application;

[0025] Figure 8 It is a schematic diagram of the structure of the terminal device provided by the fourth embodiment of the present application. Detailed Embodiments

[0026] To make the objectives, features, and advantages of the present application more apparent and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0027] In the description of the embodiments of the present application, it should be understood that the orientation or positional relationship indicated by the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the embodiments of the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0028] In addition, the terms "first" and "second" are only used for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present application, the meaning of "a plurality" is two or more, unless otherwise specifically defined.

[0029] In the embodiments of the present application, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed", etc. should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or integrated; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the communication inside two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to specific circumstances.

[0030] The above are only the preferred embodiments of the present application and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

[0031] An event-based camera is a new type of image sensor. The image sensor includes a pixel array composed of a plurality of pixels, and each pixel operates independently. Only when the brightness change of a certain pixel reaches a certain threshold does it output an event.

[0032] It should be noted that each pixel sensor in the pixel array of the event camera is an integrated circuit. In this integrated circuit, a photodiode can be integrated with a capacitor for accumulating charges. The photodiode generates a photocurrent in response to the incident light intensity, and a real-time voltage is generated accordingly based on the photocurrent.

[0033] Then, after subtracting each real-time voltage from a preset reference voltage, the voltage differences are respectively compared with a preset voltage threshold range, and binary vectors for each pixel are generated correspondingly according to the comparison results. The binary vectors are used to represent whether the optical signal becomes stronger, weaker, or remains unchanged. It should be noted that when the voltage difference exceeds the voltage threshold range, the generated binary vector is [1, 0] or [0, 1]. Among them, [1, 0] indicates that the pixel generates an UP event, and [0, 1] indicates that the pixel generates a DN event. These two binary vectors correspond to the same binary value 1. When the voltage difference does not exceed the voltage threshold range, the generated binary vector is [0, 0], indicating that the pixel does not generate an event, and this binary vector corresponds to the binary value 0.

[0034] Finally, the event polarities of the pixels with events are attached with pixel coordinate position information and timestamp information to generate event data, and an event image is output based on the obtained event data.

[0035] To solve the problems of complex operation, large delay, and low frame rate existing in the event image denoising technology provided in the related art, the first embodiment of this application provides an event image output method, which is applied to an event camera, as Figure 1 This is a schematic diagram of the basic process of the event image output method provided in this embodiment. The event image output method includes the following steps:

[0036] Step 101: When the number of frames of event data included in the buffered event stream reaches the target number of frames N, for each new frame of event data generated thereafter, an OR operation is performed on the new event data and the binary values at the same pixel positions of the N - 1 frames of event data before the new event data to obtain merged event data.

[0037] Specifically, in this embodiment, the value of N is a positive integer greater than 1. The binary values include a first value and a second value. The first value represents that an event occurs at the pixel, and the second value represents that no event occurs at the pixel. Preferably, the first value is 1 and the second value is 0. In this embodiment, when performing a logical OR operation on the binary values at the same pixel positions of different event data, if at least one of the multiple binary values at the same pixel position is 1, the OR operation result of the multiple binary values at the same pixel position is 1. Then, the values of the OR operation results are assigned to all pixel positions in the overall pixel array of the image sensor to obtain merged event data.

[0038] It should be noted that the target number of frames in this embodiment is the total number of sequentially continuous event data frames required for outputting event images. In practical applications, starting from the first frame of event data, when the number of continuously buffered event data frames reaches the target number of frames, if a new frame of event data is generated, the first frame of event data is removed, and then the other event data is merged. Similarly, when a new frame of event data is continuously generated, the new first frame of event data in the buffered event stream, that is, the original second frame of event data, is also removed, and then the other event data is merged, and so on until no new event data is generated.

[0039] In this embodiment, an exemplary description is made with the target number of frames being 3. For example, Figures 2a to 2d are schematic diagrams of 4 frames of event data sequentially output from time t1 to time t4. The event data 201 output at time t1 is the first frame of event data, and the event data 204 output at time t4 is the new event data. When executing the event data merging process of this embodiment, first remove the first frame of event data, that is, 201, and then perform a logical OR operation on the three frames of event data 202, 203, and 204 to obtain Figure 2e the merged event data shown. It should be noted that in Figures 2a to 2e , A, B, D, and E are exemplary pixel positions where events occur, and C is an exemplary pixel position where no event occurs.

[0040] In an implementation manner of this embodiment, the method further includes: obtaining the frequency of the noise signal and the frame rate of the image sensor of the event camera; calculating the target number of frames based on the frequency and the frame rate.

[0041] Specifically, in practical applications, the number of frames of event data to be merged can be flexibly selected according to actual needs, such as 3 frames, 5 frames, 10 frames, etc. Preferably, the number of frames N merged in this embodiment can consider the frequency f of the noise and the frame rate F of the image sensor. The frame number calculation formula can be expressed as N = F / f. Assuming that the noise is 50HZ and the frame rate is 150fps, then N is 3 frames, so that it can be ensured that one period of the noise is included in the merged number of frames.

[0042] Step 102: Map the target pixel positions with the first value of each binary value in the new event data to the merged event data, and count the number of first values of the first pixel array including the target pixel positions in the merged event data.

[0043] Specifically, in this embodiment, taking the pixel position where an event occurs in the new event data as a reference, the position of the pixel to be denoised is located in the merged event data. That is, for each pixel position with a first value in the new event data, it is mapped to the merged event data, and the same pixel position is located in the merged event data. Then, the overall event generation situation in the area where this pixel position is located is statistically analyzed, that is, the sum of all first values in the pixel array composed of this pixel position and its neighboring pixel positions is statistically analyzed. It should be understood that the first pixel array in this embodiment is composed of the target pixel position and its neighboring pixel positions. In practical applications, this pixel array can be a pixel array with a regular shape or an irregular shape, and the neighborhood size can also be flexibly set according to the actual application scenario. This embodiment does not make a unique limitation on this.

[0044] In an implementation manner of this embodiment, the step of statistically analyzing the number of first values of the first pixel array including the target pixel position in the merged event data includes: in the overall pixel array corresponding to the merged event data, a first pixel array with a target size is delimited with the target pixel position as the center; the number of first values of the first pixel array is statistically analyzed.

[0045] This embodiment takes the above Figure 2e A pixel position in it as an example for illustration. For example, Figure 3 is a schematic diagram of a first pixel array provided by this embodiment. Preferably, a pixel array is regularly delimited with the target pixel position as the center. Figure 3 The size of the pixel array in it is 3*3, and it can be flexibly adjusted according to the actual application in practical applications. The number of first values in this pixel array is 6.

[0046] Step 103: Denoise the binary value of the target pixel position in the merged event data based on the number of first values to obtain denoised event data.

[0047] Specifically, in this embodiment, the number of first values is compared with a preset first quantity threshold. When the number of first values is less than the first quantity threshold, the binary value of the target pixel position in the merged event data is set to a second value to obtain denoised event data.

[0048] Continuing with the foregoing example, assuming that the first quantity threshold is 5, since the first value quantity of the first pixel array at the A pixel position is 6, which is greater than this threshold, it indicates that the pixel position to be denoised is not a noise point, and the event it generates is a valid event. On the contrary, if the first value quantity is less than the first quantity threshold, it is determined that the pixel position to be denoised is a noise point, and the event it generates is a false event, and it should be removed, that is, the original first value is set to the second value, that is, the binary value 1 is reset to 0. For all pixel positions with events generated where the value is the first value in the new event data, the above operations are performed at the corresponding pixel positions of the merged event data, and all event pixels with the total number of neighborhood events less than the threshold are filtered to obtain denoised event data.

[0049] In an implementation manner of this embodiment, after the step of denoising the binary value of the target pixel position in the merged event data based on the first value quantity, it further includes: increasing and adjusting the array size of the first pixel array, and then returning to the step of counting the first value quantity of the first pixel array including the target pixel position in the merged event data.

[0050] Specifically, in order to improve the denoising quality, this embodiment can perform denoising on the event data multiple times, that is, after completing the previous denoising of the merged event data, re - delimit the first pixel array for the target pixel positions that have not been filtered in the merged event data. The size of the re - delimited first pixel array is larger than the array size of the previous denoising. For example, the array size delimited for the previous denoising is 3*3, and the array size delimited for the next denoising can be 5*5, and then the denoising process is executed again. In practical applications, the number of repeated denoising can be flexibly set according to the actual application scenario, and this embodiment does not make a unique limitation on this.

[0051] In another implementation manner of this embodiment, before the step of outputting an event image based on the denoised event data, it further includes: dividing the overall pixel array of the denoised event data into multiple second pixel arrays; counting the number of first values in each second pixel array, and assigning the number of first values as the decimal value of the corresponding second pixel array to obtain combined event data in units of second pixel arrays; summing all the decimal values of the third pixel array including the second pixel array in the combined event data; when the summation result is less than the second quantity threshold, setting the binary values of all pixel positions of the second pixel array in the denoised event data to the second value to obtain the final denoised event data.

[0052] Specifically, in order to improve the denoising quality, this embodiment is implemented by using the method of multiple denoising. Among them, in the first denoising process, denoising processing is performed in units of individual pixel positions, while in the second denoising process, the overall pixel array of the denoised event data is further divided into multiple second pixel arrays, as Figure 4The figure shows a schematic diagram of denoised event data provided by this embodiment. The overall pixel array of the denoised event data 206 includes 9*9 single-pixel positions (such as A in the figure). Then, taking the second pixel array with a size of 3*3 (such as 2061 in the figure) as a unit, the overall pixel array is divided into 3*3 second pixel arrays. Next, for each second pixel array, the total number of binary values of all its pixel positions with the first value is counted to obtain the corresponding decimal value for each second pixel array. For Figure 4 In the second pixel array 2061, the total number of the first values of all its pixel positions is 6, so the decimal value of this second pixel array is 6. Further, taking the second pixel array as the denoising unit, the sum of the decimal values of the third pixel array (this pixel array includes the second pixel array to be denoised and its adjacent second pixel arrays) including the second pixel array to be denoised and the nearby second pixel arrays is counted, and the sum result is compared with a preset second quantity threshold. If it is less than the second quantity threshold, it is determined that the second pixel array to be denoised is a noise block, and all the pixel positions it includes are noise points. Thus, the binary values of all pixel positions in this second pixel array are set to the second value, filtering out all noise events in this noise block and completing the secondary denoising.

[0053] As Figure 5 The figure shows a schematic diagram of the third pixel array provided by this embodiment. This third pixel array is composed of the second pixel array to be denoised and its neighboring second pixel arrays. In this embodiment, take Figure 4 The second pixel array 2061 in it as the second pixel array to be denoised. The second pixel array 2061 and the two neighboring second pixel arrays 2062 and 2063 below and to its right form the third pixel array. Among them, the decimal value of the second pixel array 2061 is 6, the decimal value of the second pixel array 2062 is 3, and the decimal value of the second pixel array 2063 is 6. The sum of their decimal values is 15. If the second quantity threshold is 12, the sum of the decimals of the third pixel array where the second pixel array to be denoised is located is greater than the second quantity threshold, indicating that the events generated in the second pixel array are valid events, and its events are retained. That is, in this embodiment, after the first fine-grained denoising, coarse-grained denoising is further performed, which can effectively filter out some blocky noises and improve the denoising quality.

[0054] Step 104: Output an event image based on the denoised event data.

[0055] Specifically, in this embodiment, the target number of frames is used as the unit for frame merging to output denoised event data. In practical applications, the number of frames of event data in the continuously generated event stream may be much larger than the target number of frames. Then, multiple pieces of denoised event data will be generated. In this embodiment, multiple event images can be respectively output based on each piece of denoised event data, or a single event image can be output by combining multiple frames of denoised event data. It should be noted that different from the related art where an event image is generated only based on the current frame of event data, each piece of denoised event data in this embodiment actually includes the event information of the current frame and multiple frames of event data of the previous frames. Therefore, the event image generated based on the denoised event data can improve the imaging quality of the event image without reducing the frame rate.

[0056] In an implementation manner of this embodiment, the step of outputting an event image based on the denoised event data includes: generating event images respectively based on the denoised event data at different times; fusing all the event images and then outputting them.

[0057] Specifically, in this embodiment, an event image can be generated respectively for the denoised event data obtained at different times, and then all the event images are fused to obtain the finally required output event image. Thus, an event image with better integrity and higher resolution can be output.

[0058] Based on the technical solution of the embodiment of the present application above, when the number of frames of event data included in the buffered event stream reaches the target number of frames N, for each new frame of event data generated thereafter, perform an OR operation on the binary values at the same pixel positions of the new event data and the N - 1 frames of event data before the new event data to obtain merged event data; map the target pixel positions where the binary values in the new event data are the first value to the merged event data, and count the number of first values of the first pixel array including the target pixel positions in the merged event data; perform denoising processing on the binary values at the target pixel positions in the merged event data based on the number of first values to obtain denoised event data; output an event image based on the denoised event data. Thus, the imaging quality of the event image is improved without reducing the frame rate, and it is beneficial to reduce the computational amount of the denoising operation.

[0059] Figure 6 The method in... is a refined event image output method provided by the second embodiment of the present application. This event image output method includes:

[0060] Step 601, when the number of frames of event data included in the buffered event stream reaches the target number of frames N, for each new frame of event data generated thereafter, perform an OR operation on the binary values at the same pixel positions of the new event data and the N - 1 frames of event data before the new event data to obtain merged event data.

[0061] In this embodiment, the binary value includes a first value and a second value. The first value indicates that an event occurs in the pixel, and the second value indicates that no event occurs in the pixel.

[0062] Step 602: Map the target pixel positions with the first value in the new event data to the merged event data.

[0063] Step 603: In the overall pixel array corresponding to the merged event data, delimit a unit pixel array with a target size centered on the target pixel position, and count the number of first values in the unit pixel array.

[0064] Specifically, the number of first values in this embodiment is also the sum of the first values that appear at all pixel positions in the unit pixel array.

[0065] Step 604: Compare the number of first values with a preset number threshold.

[0066] Step 605: When the number of first values is less than the number threshold, set the binary value at the target pixel position in the merged event data to the second value to obtain the denoised event data.

[0067] In this embodiment, if the number of first values is greater than the number threshold, it means that the pixel position to be denoised is not a noise point, and the event generated by it is a valid event. Maintain its original event polarity, that is, the value of the original binary value. On the contrary, if the number of first values is less than the number threshold, it is determined that the pixel position to be denoised is a noise point, and the event generated by it is a false event, and it should be removed.

[0068] Step 606: Increase and adjust the array size of the unit pixel array. Then return to step 403.

[0069] Specifically, after the first denoising is completed in this embodiment, the size of the original unit pixel array is increased and adjusted, and then the denoised event data obtained for the first time is denoised again to obtain event data with higher denoising quality, and then step 607 is executed.

[0070] Step 607: Generate event images based on the denoised event data at different times, and fuse and output all the event images.

[0071] Specifically, for the event stream obtained in a single imaging cycle of the event camera, in this embodiment, a denoised event data is generated for each event data whose frame number is after the target frame number in the event stream, and then an event image is generated for each different denoised event data. Finally, all the event images are fused to obtain the final required output event image, so that an event image with better integrity and higher resolution can be output.

[0072] It should be understood that the sequence numbers of the steps in this embodiment do not indicate the order of execution of the steps. The order of execution of each step should be determined by its function and internal logic, and should not uniquely limit the implementation process of the embodiments of this application.

[0073] Figure 7 An event image output device provided in the third embodiment of this application. This event image output device can be used to implement the event image output method in the foregoing embodiments. As Figure 7 shown, this event image output device mainly includes:

[0074] A merging module 701, configured to, when the number of frames of event data included in the buffered event stream reaches the target number of frames N, perform an OR operation on the binary values at the same pixel positions of the new event data and the N-1 frames of event data before the new event data for each subsequent frame of new event data, to obtain merged event data; wherein, the binary values include a first value and a second value, the first value represents that an event occurs at the pixel, and the second value represents that no event occurs at the pixel;

[0075] A statistics module 702, configured to map the target pixel positions where the binary values in the new event data are the first value to the merged event data, and count the number of first values of the first pixel array including the target pixel positions in the merged event data;

[0076] A denoising module 703, configured to perform denoising processing on the binary values at the target pixel positions in the merged event data based on the number of first values, to obtain denoised event data;

[0077] An output module 704, configured to output an event image based on the denoised event data.

[0078] In some implementation manners of this embodiment, when the statistics module executes the function of counting the number of first values of the first pixel array including the target pixel positions in the merged event data, it is specifically configured to: in the overall pixel array corresponding to the merged event data, delimit a first pixel array with a target size centered on the target pixel position; count the number of first values of the first pixel array.

[0079] In some implementation manners of this embodiment, the denoising module is specifically configured to: compare the number of first values with a preset first quantity threshold; when the number of first values is less than the first quantity threshold, set the binary value at the target pixel position in the merged event data to the second value, to obtain denoised event data.

[0080] In some implementation manners of this embodiment, this event image output device further includes: a calculation module, configured to: obtain the frequency of the noise signal and the frame rate of the image sensor of the event camera; calculate the target number of frames based on the frequency and the frame rate.

[0081] In some embodiments of the present embodiment, the event image output device further includes: an adjustment module, configured to: perform an increase adjustment on the array size of the first pixel array. Correspondingly, the statistics module is further configured to, after the adjustment module performs an increase adjustment on the array size of the first pixel array, re-execute its function of counting the number of first values of the first pixel array including the target pixel position in the statistical merged event data.

[0082] In some embodiments of the present embodiment, the statistics module is further configured to: divide the overall pixel array of the denoised event data into a plurality of second pixel arrays; count the number of first values in each second pixel array, assign the number of first values as the decimal value of the corresponding second pixel array, to obtain combined event data in units of the second pixel array; sum all the decimal values of the third pixel array including the second pixel array in the combined event data. The denoising module is further configured to: when the sum result is less than the second quantity threshold, set the binary values of all pixel positions of the second pixel array in the denoised event data to the second value, to obtain the final denoised event data.

[0083] In some embodiments of the present embodiment, the output module is specifically configured to: generate event images respectively based on the denoised event data at different times; fuse and output all the event images.

[0084] It should be noted that the event image output methods in the first and second embodiments can both be implemented based on the event image output device provided in the present embodiment. Those of ordinary skill in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the event image output device described in the present embodiment can refer to the corresponding process in the foregoing method embodiments, and will not be elaborated herein.

[0085] According to the event image output device provided in the present embodiment, when the number of frames of event data included in the buffered event stream reaches the target number of frames N, for each new frame of event data generated thereafter, perform an OR operation on the new event data and the binary values at the same pixel positions of the N - 1 frames of event data before the new event data, to obtain merged event data; map the target pixel positions with the binary value of the first value in the new event data to the merged event data, count the number of first values of the first pixel array including the target pixel position in the merged event data; perform denoising processing on the binary values of the target pixel positions in the merged event data based on the number of first values, to obtain denoised event data; output an event image based on the denoised event data. Thereby, while not reducing the frame rate, the imaging quality of the event image is improved, and it is beneficial to reduce the computational amount of the denoising operation.

[0086] Figure 8A terminal device provided for the fourth embodiment of the present application. This terminal device can be used to implement the event image output method in the foregoing embodiments, and mainly includes:

[0087] A memory 801, a processor 802, and a computer program 803 stored on the memory 801 and executable on the processor 802. The memory 801 and the processor 802 are communicatively connected. When the processor 802 executes the computer program 803, the method in the first or second foregoing embodiment is implemented. Among them, the number of processors can be one or more.

[0088] The memory 801 can be a high-speed random access memory (RAM), or a non-volatile memory, such as a disk memory. The memory 801 is used to store executable program codes, and the processor 802 is coupled to the memory 801.

[0089] Furthermore, an embodiment of the present application also provides a computer-readable storage medium. This computer-readable storage medium can be disposed in the electronic devices in the foregoing embodiments. This computer-readable storage medium can be the memory in the foregoing Figure 8 illustrated embodiments.

[0090] A computer program is stored on this computer-readable storage medium. When this program is executed by a processor, the event image output method in the foregoing embodiments is implemented. Furthermore, this computer-readable storage medium can also be various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk, or an optical disc that can store program codes.

[0091] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there can be other division methods. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or modules can be electrical, mechanical, or other forms.

[0092] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical modules, that is, they can be located in one place, or they can be distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0093] In addition, in each embodiment of the present application, each functional module can be integrated into one processing module, or each module can exist physically alone, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0094] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present application. The aforementioned readable storage medium includes: various media such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0095] It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0096] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0097] The above is the description of the event image output method, device, equipment, and readable storage medium provided by the present application. For those skilled in the art, according to the idea of the embodiments of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. An event image output method, applied to an event camera, characterized in that, the event image output method includes: When the number of frames of event data included in the buffered event stream reaches the target number of frames N, for each new frame of event data generated thereafter, perform an OR operation on the new event data and the binary values at the same pixel positions of the N-1 frames of event data before the new event data to obtain merged event data; wherein, the binary values include a first value and a second value, the first value represents that an event occurs at the pixel, and the second value represents that no event occurs at the pixel; Map the target pixel positions in the new event data where the binary values are the first value to the merged event data, and count the number of first values in the first pixel array including the target pixel positions in the merged event data; Perform denoising processing on the binary values at the target pixel positions in the merged event data based on the number of first values to obtain denoised event data; Output an event image based on the denoised event data.

2. The event image output method according to claim 1, characterized in that, the step of counting the number of first values in the first pixel array including the target pixel positions in the merged event data includes: In the overall pixel array corresponding to the merged event data, define a first pixel array with a target size centered on the target pixel position; Count the number of first values in the first pixel array.

3. The event image output method according to claim 1, characterized in that, the step of performing denoising processing on the binary values at the target pixel positions in the merged event data based on the number of first values to obtain denoised event data includes: Compare the number of first values with a preset first quantity threshold; When the number of first values is less than the first quantity threshold, set the binary value at the target pixel position in the merged event data to the second value to obtain denoised event data.

4. The event image output method according to claim 1, characterized in that, further includes: Obtain the frequency of the noise signal and the frame rate of the image sensor of the event camera; Calculate the target number of frames based on the frequency and the frame rate.

5. The event image output method according to claim 1, characterized in that, after the step of performing denoising processing on the binary values at the target pixel positions in the merged event data based on the number of first values, further includes: Increase the array size of the first pixel array, and then return to the step of counting the number of first values in the first pixel array including the target pixel positions in the merged event data.

6. The event image output method according to claim 3, characterized in that, before the step of outputting an event image based on the denoised event data, further includes: Divide the overall pixel array of the denoised event data into multiple second pixel arrays; Count the number of the first values in each of the second pixel arrays, assign the number of the first values as the decimal value of the corresponding second pixel array, and obtain combined event data in units of the second pixel arrays; Sum all the decimal values corresponding to a third pixel array in the combined event data; wherein, the third pixel array includes the second pixel array to be denoised and its adjacent second pixel arrays; When the sum result is less than a second quantity threshold, set all the binary values at all pixel positions corresponding to the second pixel array to be denoised in the denoised event data to the second value, so as to obtain final denoised event data.

7. The event image output method according to any one of claims 1 to 6, wherein, the step of outputting an event image based on the denoised event data includes: generating event images respectively based on the denoised event data at different times; fusing all the event images and then outputting the fused image.

8. An event image output device applied to an event camera, wherein, the event image output device includes: a merging module, configured to, when the number of frames of event data included in a buffered event stream reaches a target number of frames N, perform an OR operation on the new event data and the binary values at the same pixel positions of the N-1 frames of event data before the new event data for each subsequent frame of new event data, so as to obtain merged event data; wherein, the binary values include a first value and a second value, the first value represents that an event occurs at a pixel, and the second value represents that no event occurs at a pixel; a counting module, configured to map the target pixel positions where the binary values in the new event data are the first value to the merged event data, and count the number of first values of a first pixel array including the target pixel positions in the merged event data; a denoising module, configured to perform denoising processing on the binary values at the target pixel positions in the merged event data based on the number of first values, so as to obtain denoised event data; an output module, configured to output an event image based on the denoised event data.

9. A terminal device, wherein, it includes a memory and a processor, and: the processor is configured to execute a computer program stored on the memory; when the processor executes the computer program, the steps in the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, on which a computer program is stored, wherein, when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

Images