Image parallax calculation method and device based on dynamic vision sensor

Abstract

The application discloses a kind of based on dynamic visual sensor's image disparity calculation method and device, wherein, method includes obtaining binocular dynamic visual sensor data;According to sliding time window, binocular dynamic visual sensor data is carried out data reconstruction processing, and reconstruction data is obtained;Reconstruction data is carried out cost matching processing, and initial disparity is obtained;Initial disparity is carried out smooth optimization processing, and aggregated disparity is obtained;According to filter window, aggregated disparity is carried out median filter processing, and the purpose disparity is obtained.The embodiment of the application carries out image disparity calculation based on dynamic visual sensor, and passes through the calculation efficiency, can be widely applied in computer vision technical field.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and apparatus for calculating image parallax based on a dynamic vision sensor. Background Technology

[0002] Currently, image disparity calculation technology has played a core role in many important application scenarios, such as object recognition, scene 3D modeling, and assisted driving. Using binocular stereo matching on highways can effectively replace radar in obtaining disparity information from vehicle images, offering advantages such as low cost, dense data, and visual characteristics. However, traditional binocular cameras suffer from large data volumes, long computation times, and low temporal resolution, resulting in low efficiency in calculating disparity for images of high-speed moving vehicles. In summary, the technical problems existing in related technologies urgently need to be solved. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide an image disparity calculation method and apparatus based on a dynamic vision sensor, so as to improve the calculation efficiency of image disparity.

[0004] On one hand, the present invention provides an image disparity calculation method based on a dynamic vision sensor, comprising:

[0005] Acquire data from binocular dynamic vision sensors;

[0006] The binocular dynamic vision sensor data is reconstructed according to a sliding time window to obtain reconstructed data.

[0007] The reconstructed data is subjected to cost matching processing to obtain the initial disparity;

[0008] The initial disparity is smoothed and optimized to obtain the aggregated disparity;

[0009] The aggregated disparity is subjected to median filtering based on the filtering window to obtain the target disparity.

[0010] Optionally, acquiring binocular dynamic vision sensor data includes:

[0011] Acquire binocular dynamic vision sensor images using a dynamic vision sensor;

[0012] The binocular dynamic vision sensor image is transmitted based on the low-voltage differential signal, and the binocular dynamic vision sensor data is obtained through serial-to-parallel conversion processing.

[0013] Optionally, the step of performing data reconstruction processing on the binocular dynamic vision sensor data according to a sliding time window to obtain reconstructed data includes:

[0014] Acquire the binocular dynamic vision sensor data, which includes event time, event location, and event polarity;

[0015] The binocular dynamic vision sensor data is sorted in ascending order according to the event time based on the event location to obtain sorted data.

[0016] The sorted data is processed by vector construction based on the sliding time window to obtain an event vector;

[0017] The binocular dynamic vision sensor data is reconstructed based on the event vector to obtain reconstructed data, which includes window initial events, event positions, and window event vectors.

[0018] Optionally, the reconstructed data is subjected to cost matching processing to obtain an initial disparity, including:

[0019] The reconstructed data is subjected to matching cost calculation to obtain the cost space;

[0020] The initial parallax is obtained by selecting and processing the cost space according to the winner-takes-all strategy.

[0021] Optionally, the smoothing optimization process performed on the initial disparity to obtain the aggregated disparity includes:

[0022] Get the support window;

[0023] Based on the support window, the initial disparity is subjected to cost aggregation processing to obtain an aggregated disparity set;

[0024] The parallax with the lowest energy is selected from the aggregated parallax set based on the support window and determined as the aggregated parallax.

[0025] Optionally, the step of performing median filtering on the aggregated disparity according to the filtering window to obtain the target disparity includes:

[0026] Get the parallax window;

[0027] The occurrence frequency of the aggregated parallax is counted based on the parallax window to obtain a parallax statistics table;

[0028] The median value of the disparity statistics table is selected to obtain the target disparity.

[0029] On the other hand, embodiments of the present invention also provide an image disparity calculation device based on a dynamic vision sensor, comprising:

[0030] The first module is used to acquire data from the binocular dynamic vision sensor.

[0031] The second module is used to perform data reconstruction processing on the binocular dynamic vision sensor data according to the sliding time window to obtain reconstructed data.

[0032] The third module is used to perform cost matching processing on the reconstructed data to obtain the initial disparity;

[0033] The fourth module is used to perform smoothing optimization on the initial disparity to obtain aggregated disparity;

[0034] The fifth module is used to perform median filtering on the aggregated disparity according to the filtering window to obtain the target disparity.

[0035] Optionally, the first module is used to acquire binocular dynamic vision sensor data, including:

[0036] The first unit is used to acquire binocular dynamic vision sensor images through a dynamic vision sensor.

[0037] The second unit is used to transmit the image from the binocular dynamic vision sensor according to the low-voltage differential signal, and to obtain the binocular dynamic vision sensor data through serial-to-parallel conversion processing.

[0038] Optionally, the second module is used to perform data reconstruction processing on the binocular dynamic vision sensor data according to a sliding time window to obtain reconstructed data, including:

[0039] The third unit is used to acquire the binocular dynamic vision sensor data, which includes event time, event location, and event polarity.

[0040] The fourth unit is used to sort the binocular dynamic vision sensor data in ascending order according to the event time based on the event location, so as to obtain sorted data;

[0041] The fifth unit is used to perform vector construction processing on the sorted data according to the sliding time window to obtain an event vector;

[0042] The sixth unit is used to reconstruct the binocular dynamic vision sensor data based on the event vector to obtain reconstructed data, which includes window initial event, event position and window event vector.

[0043] Optionally, the third module is used to perform cost matching processing on the reconstructed data to obtain an initial disparity, including:

[0044] The seventh unit is used to perform matching cost calculation on the reconstructed data to obtain the cost space;

[0045] The eighth unit is used to select and process the cost space according to the winner-take-all strategy to obtain the initial parallax.

[0046] Optionally, the fourth module is used to perform smoothing optimization processing on the initial disparity to obtain aggregated disparity, including:

[0047] Unit 9 is used to obtain the support window;

[0048] The tenth unit is used to perform cost aggregation processing on the initial disparity based on the support window to obtain an aggregated disparity set;

[0049] The eleventh unit is used to select the parallax with the minimum energy from the aggregated parallax set according to the support window and determine it as the aggregated parallax.

[0050] Optionally, the fifth module is configured to perform median filtering on the aggregated disparity according to the filtering window to obtain the target disparity, including:

[0051] Unit 12 is used to obtain the parallax window;

[0052] The thirteenth unit is used to count the number of occurrences of the aggregated parallax based on the parallax window, and obtain a parallax statistics table;

[0053] The fourteenth unit is used to perform median selection processing on the disparity statistics table to obtain the target disparity.

[0054] On the other hand, embodiments of the present invention also disclose an electronic device, including a processor and a memory;

[0055] The memory is used to store programs;

[0056] The processor executes the program to implement the method described above.

[0057] On the other hand, embodiments of the present invention also disclose a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0058] On the other hand, embodiments of the present invention also disclose a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0059] Compared with existing technologies, the present invention, employing the above technical solution, has the following technical effects: The embodiments of the present invention acquire binocular dynamic vision sensor data; perform data reconstruction processing on the binocular dynamic vision sensor data according to a sliding time window to obtain reconstructed data; perform cost matching processing on the reconstructed data to obtain initial disparity; perform smoothing optimization processing on the initial disparity to obtain aggregated disparity; and perform median filtering processing on the aggregated disparity according to a filtering window to obtain the target disparity. The embodiments of the present invention improve the calculation efficiency of image disparity by using dynamic vision sensor data that is sparse, has low latency, high temporal resolution, and a large dynamic range. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart of an image disparity calculation method based on a dynamic vision sensor provided in an embodiment of this application;

[0062] Figure 2 This is a schematic diagram of the structure of an image parallax calculation device based on a dynamic vision sensor provided in an embodiment of this application;

[0063] Figure 3 This is a schematic diagram of the structure of an image parallax calculation device based on a dynamic vision sensor provided in an embodiment of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0065] First, let's analyze some of the terms used in this application:

[0066] Parallax refers to the difference in direction that occurs when observing the same target from two points that are a certain distance apart.

[0067] A Dynamic Vision Sensor (DVS) outputs the event detected by the current pixel, specifically whether the change in the sensor's brightness reaches a threshold. Its event output is a triplet, including the event location, polarity, and timestamp information. Compared to traditional vision sensors, it features outputting only motion events, data sparsity, low latency, high temporal resolution, and a large dynamic range.

[0068] Low Voltage Differential Signaling (LVDS) is a differential signaling technology characterized by low power consumption, low bit error rate, low crosstalk, and low radiation.

[0069] In related technologies, binocular stereo matching can effectively replace radar to obtain vehicle depth information on highways. Its advantages include low cost, dense data, and visual characteristics. However, traditional binocular cameras have low efficiency in calculating image parallax due to their large data volume, long computation time, and low temporal resolution, making it difficult to detect and track high-speed moving vehicles.

[0070] Reference Figure 1 This invention provides a method for calculating image disparity based on a dynamic vision sensor, comprising:

[0071] S101. Acquire data from the binocular dynamic vision sensor;

[0072] S102. Perform data reconstruction processing on the binocular dynamic vision sensor data according to the sliding time window to obtain reconstructed data;

[0073] S103. Perform cost matching processing on the reconstructed data to obtain the initial disparity;

[0074] S104. Perform smoothing optimization on the initial disparity to obtain aggregated disparity;

[0075] S105. Perform median filtering on the aggregated disparity according to the filtering window to obtain the target disparity.

[0076] In this embodiment of the invention, binocular dynamic vision sensor data is first acquired. This data can be obtained by processing images acquired by binocular cameras into the dynamic vision sensor, or directly by using a binocular dynamic vision sensor (DVS). Specifically, two DVSs capture the same scene simultaneously, and the output events form the left and right eye images, respectively. Next, the binocular dynamic vision sensor data is reconstructed using a time-sliding window to extract richer features. Cost matching is then performed on the reconstructed data to obtain the initial disparity. This initial disparity is then optimized and smoothed, specifically through aggregation processing to obtain the aggregated disparity. Finally, a filtering window is generated by the row buffer for median filtering of the disparity. Median filtering is then applied to the aggregated disparity to obtain the target disparity of the image. This embodiment can be applied to vehicle detection scenarios such as highways and urban roads. The disparity calculation method in this embodiment can be used to detect and track high-speed moving vehicles.

[0077] As a further preferred embodiment, in step S101 above, acquiring binocular dynamic vision sensor data includes:

[0078] Acquire binocular dynamic vision sensor images using a dynamic vision sensor;

[0079] The binocular dynamic vision sensor image is transmitted based on the low-voltage differential signal, and the binocular dynamic vision sensor data is obtained through serial-to-parallel conversion processing.

[0080] In this embodiment of the invention, the Dynamic Vision Sensor (DVS), also known as an event camera, can acquire images from a binocular dynamic vision sensor. This embodiment uses Low Voltage Differential Signaling (LVDS) for data transmission. Specifically, an FPGA development board is directly connected to the LVDS interface of the DVS, and a high-frequency to low-frequency conversion method is used for serial-to-parallel conversion. Specifically, an FPGA adapter board is used as the connection interface at the output of the DVS. The FPGA adapter board uses a 20MHz main frequency to convert the DVS output data from serial to parallel data, and inputs it into the data reconstruction module in the form of an event stream.

[0081] As a further preferred embodiment, in step S102 above, the step of performing data reconstruction processing on the binocular dynamic vision sensor data according to the sliding time window to obtain reconstructed data includes:

[0082] Acquire the binocular dynamic vision sensor data, which includes event time, event location, and event polarity;

[0083] The binocular dynamic vision sensor data is sorted in ascending order according to the event time based on the event location to obtain sorted data.

[0084] The sorted data is processed by vector construction based on the sliding time window to obtain an event vector;

[0085] The binocular dynamic vision sensor data is reconstructed based on the event vector to obtain reconstructed data, which includes window initial events, event positions, and window event vectors.

[0086] In this embodiment of the invention, data reconstruction is performed based on a sliding time window to extract richer features. In the event camera, each event in the binocular dynamic vision sensor data is denoted as e(t,x,y,p), where t represents the time of the event, x and y represent the location of the event, and p represents the polarity of the event. In this embodiment, time t0 is used as the initial moment of the current sliding time window. To reconstruct the data for the sliding time window, the initial time and window size can be set according to the actual situation. In a feasible embodiment, the initial time is set to the earliest event time in the binocular dynamic vision sensor data, and the window size is set to 5 seconds. Specifically, firstly, the event position (x0, y0) corresponding to the initial event at time t0 is obtained. Based on the event position, the binocular dynamic vision sensor data, i.e., the events, are sorted in ascending order according to time to obtain sorted data. In this embodiment, a working window of [H, W] can also be reconstructed. This working window is a storage space of size [H, W] pre-allocated in memory. The working window has the same resolution as the DVS sensor, where H is the height of the DVS output resolution and W is the width of the DVS output resolution. In this embodiment, the length of a single storage unit is the upper limit of the event, which is set to 128 bits wide. It is conceivable that in this embodiment, the sorted data is stored in the corresponding position in memory. The sorted data in memory is processed into an event vector based on the sliding time window. Specifically, the sorted data is processed into an event vector by constructing a vector from the data in memory. Received events are stored in the corresponding memory locations according to their coordinates. The binocular dynamic vision sensor data is reconstructed based on the event vectors to obtain reconstructed data, which includes the window initial event, event position, and window event vector. The data format changes from e(t,x,y,p) to e(t0,x,y,v), where v is... A vector of events over a time period, where t0 represents the initial event of the current working window.

[0087] As a further preferred embodiment, in step S103 above, cost matching processing is performed on the reconstructed data to obtain the initial disparity, including:

[0088] The reconstructed data is subjected to matching cost calculation to obtain the cost space;

[0089] The initial parallax is obtained by selecting and processing the cost space according to the winner-takes-all strategy.

[0090] In this embodiment of the invention, matching the reconstructed data, i.e., the reconstructed event data, on the window specifically includes: calculating the matching cost, the calculation method of which is as follows:

[0091] C(e,d)=∑ win g(v l ,v r );

[0092] In the formula, C(e,d) represents the cost space; e represents the reconstructed data; d represents the alternative cost, ranging from [0,64], with one alternative cost calculated for each alternative disparity; win represents the size of the sliding time window; g(v l ,v r ) is used to calculate the cosine similarity of event vectors.

[0093] Next, the winner-takes-all strategy is used on the cost space to obtain the initial parallax:

[0094]

[0095] In the formula, d init The initial disparity is calculated.

[0096] As a further preferred embodiment, in step S104 above, the smoothing and optimization processing of the initial disparity to obtain the aggregated disparity includes:

[0097] Get the support window;

[0098] Based on the support window, the initial disparity is subjected to cost aggregation processing to obtain an aggregated disparity set;

[0099] The parallax with the lowest energy is selected from the aggregated parallax set based on the support window and determined as the aggregated parallax.

[0100] In this embodiment of the invention, a support window is first obtained. This support window is selected based on the working window. Specifically, a support window of size [w,h] is selected on the working window. Since the spatial position of the road surface and vehicles in a highway is relatively long and narrow, w needs to be much larger than h in [w,h]. In this invention, [w,h] = [31,9]. The initial disparity obtained from the support window is optimized and smoothed, and a cost aggregation function is constructed for each initial disparity within the support window.

[0101]

[0102] In the formula, E(d) init (i) represents the aggregation cost, SR represents the support window, and ||d init (i)-d init (j)||2 is the L2 norm for calculating the two initial costs.

[0103] The winner-takes-all strategy is applied again to the obtained aggregation cost to obtain an aggregated disparity set. The disparity with the lowest energy in the support window is selected as the aggregated disparity.

[0104] d agg =argmin 0≤i≤SR E(d init (i));

[0105] In the formula, d agg This is the calculated aggregated disparity.

[0106] As a further preferred embodiment, in step S105 above, the step of performing median filtering on the aggregated disparity according to the filtering window to obtain the target disparity includes:

[0107] Get the parallax window;

[0108] The occurrence frequency of the aggregated parallax is counted based on the parallax window to obtain a parallax statistics table;

[0109] The median value of the disparity statistics table is selected to obtain the target disparity.

[0110] In this embodiment of the invention, median filtering is performed on the obtained aggregated disparity. The median filtering calculation specifically includes disparity window generation, disparity statistics, and disparity selection. In disparity window generation, a disparity window is first generated by the line buffers for median filtering. Specifically, a line buffer identical to the working window W is constructed using registers. The number of line buffers is the same as the high K of the median filtering. The output of the previous line buffer is used as the input of the next line buffer. Each clock cycle, one disparity is input to the first line buffer. After W×K cycles, a K×K register is used to sequentially receive the output of each line buffer from the previous K cycles. Therefore, obtaining the median-filtered window requires W×K+K clock cycles. Next, the disparity statistics function counts the number of times each disparity appears within the window. This is done by initializing a theoretical array with the same length as the number of all possible disparities. The array stores the count of each disparity's occurrence. For each input data point, the count is compared sequentially with the theoretical disparity. If the count equals the current disparity, the corresponding number in the array is incremented by 1. If not, the count is passed to the next comparator and compared with the next theoretical disparity, and so on, until all comparisons are complete. This count is then mapped to a table. Because the disparity amplitude is small, this table is relatively small. Theoretically, disparity statistics should count all disparities within the disparity window each time. However, the sliding window used during window selection creates a large overlapping area between adjacent disparity windows, leading to redundant statistics and resource waste. To address this, when selecting disparities for statistics, only newly input disparities are added to the statistics, while newly output disparities are removed. In one feasible embodiment, the present invention uses a 13×13 filtering window. Only 13 additions and 13 subtractions are needed to calculate all disparities in the disparity statistics, which greatly reduces resource consumption. Finally, a disparity statistics table is maintained through registers. By updating and selecting from this table, a median value is obtained as the final output.

[0111] On the other hand, refer to Figure 2 This invention also provides an image parallax calculation device based on a dynamic vision sensor, comprising:

[0112] The first module 201 is used to acquire data from the binocular dynamic vision sensor.

[0113] The second module 202 is used to perform data reconstruction processing on the binocular dynamic vision sensor data according to the sliding time window to obtain reconstructed data;

[0114] The third module 203 is used to perform cost matching processing on the reconstructed data to obtain the initial disparity;

[0115] The fourth module 204 is used to perform smoothing optimization processing on the initial disparity to obtain aggregated disparity;

[0116] The fifth module 205 is used to perform median filtering on the aggregated disparity according to the filtering window to obtain the target disparity.

[0117] As a further preferred embodiment, the first module is used to acquire binocular dynamic vision sensor data, including:

[0118] The first unit is used to acquire binocular dynamic vision sensor images through a dynamic vision sensor.

[0119] The second unit is used to transmit the image from the binocular dynamic vision sensor according to the low-voltage differential signal, and to obtain the binocular dynamic vision sensor data through serial-to-parallel conversion processing.

[0120] As a further preferred embodiment, the second module is used to perform data reconstruction processing on the binocular dynamic vision sensor data according to a sliding time window to obtain reconstructed data, including:

[0121] The third unit is used to acquire the binocular dynamic vision sensor data, which includes event time, event location, and event polarity.

[0122] The fourth unit is used to sort the binocular dynamic vision sensor data in ascending order according to the event time based on the event location, so as to obtain sorted data;

[0123] The fifth unit is used to perform vector construction processing on the sorted data according to the sliding time window to obtain an event vector;

[0124] The sixth unit is used to reconstruct the binocular dynamic vision sensor data based on the event vector to obtain reconstructed data, which includes window initial event, event position and window event vector.

[0125] As a further preferred embodiment, the third module is used to perform cost matching processing on the reconstructed data to obtain an initial disparity, including:

[0126] The seventh unit is used to perform matching cost calculation on the reconstructed data to obtain the cost space;

[0127] The eighth unit is used to select and process the cost space according to the winner-take-all strategy to obtain the initial parallax.

[0128] As a further preferred embodiment, the fourth module is used to perform smoothing optimization processing on the initial disparity to obtain aggregated disparity, including:

[0129] Unit 9 is used to obtain the support window;

[0130] The tenth unit is used to perform cost aggregation processing on the initial disparity based on the support window to obtain an aggregated disparity set;

[0131] The eleventh unit is used to select the parallax with the minimum energy from the aggregated parallax set according to the support window and determine it as the aggregated parallax.

[0132] As a further preferred embodiment, the fifth module is used to perform median filtering on the aggregated disparity according to the filtering window to obtain the target disparity, including:

[0133] Unit 12 is used to obtain the parallax window;

[0134] The thirteenth unit is used to count the number of occurrences of the aggregated parallax based on the parallax window, and obtain a parallax statistics table;

[0135] The fourteenth unit is used to perform median selection processing on the disparity statistics table to obtain the target disparity.

[0136] It is understood that the contents of the above-described embodiments of the image disparity calculation method based on dynamic vision sensors are all applicable to the embodiments of this device. The specific functions implemented by the embodiments of this device are the same as those of the above-described embodiments of the image disparity calculation method based on dynamic vision sensors, and the beneficial effects achieved are also the same as those achieved by the above-described embodiments of the image disparity calculation method based on dynamic vision sensors.

[0137] Reference Figure 3 This invention also provides an electronic device, including a processor 301 and a memory 302; the memory is used to store a program; the processor executes the program to implement the method described above.

[0138] and Figure 1 Corresponding to the method described above, embodiments of the present invention also provide a computer-readable storage medium storing a program that is executed by a processor to implement the method described above.

[0139] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0140] In related technologies, traditional binocular cameras are characterized by large data volume, long computing time, and low temporal resolution, making it difficult to detect and track high-speed moving vehicles.

[0141] In summary, the embodiments of the present invention have the following advantages: The embodiments of the present invention use a DVS sensor as data input, and the obtained optical flow and disparity correspond to motion data, featuring only outputting motion events, data sparsity, low latency, high temporal resolution, and a large dynamic range. Using a DVS-based scene flow can improve the efficiency of image disparity calculation, effectively enabling real-time monitoring of highways, and eliminating data noise interference caused by changes in illumination.

[0142] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0143] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0144] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0145] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0146] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0147] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0148] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0149] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0150] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for calculating image disparity based on a dynamic vision sensor, characterized in that, The method includes: Acquire data from binocular dynamic vision sensors; The binocular dynamic vision sensor data is reconstructed according to a sliding time window to obtain reconstructed data. The reconstructed data is subjected to cost matching processing to obtain the initial disparity; The initial disparity is smoothed and optimized to obtain the aggregated disparity; The aggregated disparity is subjected to median filtering based on the filtering window to obtain the target disparity; The process of smoothing and optimizing the initial disparity to obtain the aggregated disparity includes: Select the supporting window based on the working window; Based on the cost aggregation function, the initial disparity is subjected to cost aggregation processing according to the support window to obtain an aggregated disparity set; The expression for the cost aggregation function is: ； In the formula, As the aggregation cost, SR serves as the support window. To calculate the L2 norm of the two initial costs; Based on the support window, the parallax with the lowest energy is selected from the aggregated parallax set and determined as the aggregated parallax. The step of reconstructing the binocular dynamic vision sensor data according to a sliding time window to obtain reconstructed data includes: Acquire the binocular dynamic vision sensor data, which includes event time, event location, and event polarity; The binocular dynamic vision sensor data is sorted in ascending order according to the event time based on the event location to obtain sorted data. The sorted data is processed using the sliding time window to construct a vector, resulting in an event vector. The binocular dynamic vision sensor data is reconstructed based on the event vector to obtain reconstructed data, which includes window initial events, event positions, and window event vectors.

2. The method according to claim 1, characterized in that, The acquisition of binocular dynamic vision sensor data includes: Acquire binocular dynamic vision sensor images using a dynamic vision sensor; The binocular dynamic vision sensor image is transmitted based on the low-voltage differential signal, and the binocular dynamic vision sensor data is obtained through serial-to-parallel conversion processing.

3. The method according to claim 1, characterized in that, The reconstructed data is subjected to cost matching processing to obtain the initial disparity, including: The reconstructed data is subjected to matching cost calculation to obtain the cost space; The initial parallax is obtained by selecting and processing the cost space according to the winner-takes-all strategy.

4. The method according to claim 1, characterized in that, The step of performing median filtering on the aggregated disparity according to the filtering window to obtain the target disparity includes: Get the parallax window; The occurrence frequency of the aggregated parallax is counted based on the parallax window to obtain a parallax statistics table; The median value of the disparity statistics table is selected to obtain the target disparity.

5. An image parallax calculation device based on a dynamic vision sensor, characterized in that, The device includes: The first module is used to acquire data from the binocular dynamic vision sensor. The second module is used to perform data reconstruction processing on the binocular dynamic vision sensor data according to the sliding time window to obtain reconstructed data. The third module is used to perform cost matching processing on the reconstructed data to obtain the initial disparity; The fourth module is used to perform smoothing optimization on the initial disparity to obtain aggregated disparity; The fifth module is used to perform median filtering on the aggregated disparity according to the filtering window to obtain the target disparity; The fourth module is used to perform smoothing optimization processing on the initial disparity to obtain aggregated disparity, including: Select the supporting window based on the working window; Based on the cost aggregation function, the initial disparity is subjected to cost aggregation processing according to the support window to obtain an aggregated disparity set; The expression for the cost aggregation function is: ； In the formula, As the aggregation cost, SR serves as the support window. To calculate the L2 norm of the two initial costs; Based on the support window, the parallax with the lowest energy is selected from the aggregated parallax set and determined as the aggregated parallax. The second module is used to perform data reconstruction processing on the binocular dynamic vision sensor data according to a sliding time window to obtain reconstructed data, including: Acquire the binocular dynamic vision sensor data, which includes event time, event location, and event polarity; The binocular dynamic vision sensor data is sorted in ascending order according to the event time based on the event location to obtain sorted data. The sorted data is processed using the sliding time window to construct a vector, resulting in an event vector. The binocular dynamic vision sensor data is reconstructed based on the event vector to obtain reconstructed data, which includes window initial events, event positions, and window event vectors.

6. The apparatus according to claim 5, characterized in that, The first module is used to acquire binocular dynamic vision sensor data, including: The first unit is used to acquire binocular dynamic vision sensor images through a dynamic vision sensor. The second unit is used to transmit the image from the binocular dynamic vision sensor according to the low-voltage differential signal, and to obtain the binocular dynamic vision sensor data through serial-to-parallel conversion processing.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor; The memory is used to store programs; The processor executes the program to implement the method of any one of claims 1 to 4.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.

Images