Upsampling method, device, equipment and storage medium for space-time pulse signal

Abstract

The present application discloses a method, device, equipment and storage medium for upsampling a spatiotemporal pulse signal, the method comprising: collecting the spatiotemporal pulse signal of the target object; estimating the motion trajectory of the pulse signal, and determining the main pulse signal and the noise signal according to the number of pulses on the motion trajectory; upsampling the main pulse signal and the noise signal respectively to obtain the upsampled main pulse signal and the noise signal; merging the upsampled main pulse signal and the noise signal to generate the final upsampled signal. According to the method for upsampling a spatiotemporal pulse signal provided in the embodiment of the present application, a sparse pulse signal can be upsampled into a dense pulse signal, so that the upsampled pulse signal is reliable and has a basis, effectively solving the problem caused by the sparsity of the spatiotemporal pulse signal, providing more effective data for visual tasks based on pulse signals, and improving the performance of downstream tasks based on pulses.

Description

Technical Field

[0001] The present invention relates to the field of signal processing technology, and in particular to an up-sampling method, device, equipment and storage medium for a spatiotemporal pulse signal. Background Art

[0002] Neuromorphic vision sensors are a new type of biomimetic retinal camera. Unlike traditional frame-based cameras that generate complete images at fixed time intervals, neuromorphic cameras only generate signals when they detect brightness changes in the scene or when the accumulated absolute value of brightness exceeds a set threshold. This sampling mechanism gives neuromorphic vision sensors the advantages of high temporal resolution, high dynamic range, and low data redundancy, and they are increasingly used in the field of computer vision, such as autonomous driving, image reconstruction, and target detection. Although neuromorphic vision sensors have high temporal resolution and can capture high-speed moving objects, in slow or low-brightness scenes, only a few pulse signals are usually generated within a small time interval. The number of effective pulses directly affects the performance of tasks based on pulse signals, such as image reconstruction, target detection, and recognition tasks.

[0003] For slow scenes, the slow movement of objects makes it difficult for brightness changes in a short period of time to exceed the preset threshold, so only a small number of pulse signals will be generated. In low-brightness scenes, it is difficult for the camera to capture dynamic information and more noise is generated. Since these scenes can only provide sparse valid events and contain noise, the performance of event-based tasks is poor. For example, for image reconstruction tasks, the limited information provided by sparse events makes it impossible for the reconstructed image to express fine textures and details. For target detection tasks, the lack of key information in the original events makes it difficult to detect targets.

[0004] Therefore, how to solve the problems caused by the sparsity of pulse signals, provide more effective signals for pulse-based downstream tasks and improve task performance is an urgent and meaningful technical problem to be solved. Summary of the invention

[0005] Embodiments of the present application provide a method, device, equipment and storage medium for upsampling a spatiotemporal pulse signal. The purpose is to generate a dense upsampled pulse signal from a sparse pulse signal, thereby highlighting the main pulse signal of the moving subject and suppressing the generation of noise. This method can generate more effective and reliable data and can improve the performance of downstream pulse-based tasks. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary section is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0006] In a first aspect, an embodiment of the present application provides a method for upsampling a spatiotemporal pulse signal, comprising:

[0007] Collecting the spatiotemporal pulse signals of the target object;

[0008] Estimate the motion trajectory of the pulse signal, and determine the main pulse signal and the noise signal according to the number of pulses on the motion trajectory;

[0009] Upsampling the main pulse signal and the noise signal respectively to obtain the upsampled main pulse signal and the noise signal;

[0010] The upsampled main pulse signal and the noise signal are combined to generate the final upsampled signal.

[0011] In an optional embodiment, collecting the spatiotemporal pulse signal of the target object includes:

[0012] Collecting the spatiotemporal pulse signals of the target object through a dynamic vision sensor; or,

[0013] The spatiotemporal pulse signals of the target object are collected through an ultra-high-speed full-time visual sensor.

[0014] In an optional embodiment, estimating the motion trajectory of the pulse signal includes:

[0015] Estimate the motion trajectory of the pulse signal by using a contrast maximization algorithm; or,

[0016] Estimate the motion trajectory of the pulse signal by minimizing the average timestamp algorithm; or,

[0017] The pulse signal is converted into a two-dimensional image, and the two-dimensional image is input into a preset first neural network model to obtain the motion trajectory of the pulse signal.

[0018] In an optional embodiment, determining the main pulse signal and the noise signal according to the number of pulses on the motion trajectory includes:

[0019] When the number of pulses on the motion trajectory is greater than a preset pulse number threshold, the pulse signal corresponding to the motion trajectory is the main pulse signal of the target object;

[0020] When the number of pulses on the motion trajectory is less than or equal to a preset pulse number threshold, the pulse signal corresponding to the motion trajectory is a noise signal of the target object.

[0021] In an optional embodiment, upsampling the main pulse signal and the noise signal respectively to obtain the upsampled main pulse signal and the noise signal includes:

[0022] Up-sampling the main pulse signal and the noise signal respectively through the point process model to obtain the up-sampled main pulse signal and the noise signal; or,

[0023] The main pulse signal and the noise signal are upsampled respectively through a preset second neural network model to obtain the upsampled main pulse signal and the noise signal.

[0024] In an optional embodiment, upsampling the main pulse signal and the noise signal respectively through a point process model to obtain the upsampled main pulse signal and the noise signal includes:

[0025] The main pulse signal is upsampled by using a Hawkes point process model to obtain an upsampled main pulse signal;

[0026] The noise signal is upsampled through the self-correction point process model to obtain an upsampled noise signal.

[0027] In a second aspect, an embodiment of the present application provides an upsampling device for a spatiotemporal pulse signal, comprising:

[0028] An acquisition module, used for acquiring the spatiotemporal pulse signals of the target object;

[0029] A motion estimation module is used to estimate the motion trajectory of the pulse signal and determine the main pulse signal and the noise signal according to the number of pulses on the motion trajectory;

[0030] An up-sampling module is used to up-sample the main pulse signal and the noise signal respectively to obtain the up-sampled main pulse signal and the noise signal;

[0031] The merging module is used to merge the up-sampled main pulse signal and the noise signal to generate a final up-sampled signal.

[0032] In an optional embodiment, the motion estimation module includes:

[0033] a motion estimation unit, configured to estimate the motion trajectory of the impulse signal by using a contrast maximization algorithm; or,

[0034] Estimate the motion trajectory of the pulse signal by minimizing the average timestamp algorithm; or,

[0035] The pulse signal is converted into a two-dimensional image, and the two-dimensional image is input into a preset first neural network model to obtain a motion trajectory of the pulse signal;

[0036] It also includes a pulse signal classification unit, which is used to classify the pulse signal corresponding to the motion trajectory as the main pulse signal of the target object when the number of pulses on the motion trajectory is greater than a preset pulse number threshold;

[0037] When the number of pulses on the motion trajectory is less than or equal to a preset pulse number threshold, the pulse signal corresponding to the motion trajectory is a noise signal of the target object.

[0038] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory storing program instructions, wherein the processor is configured to execute the upsampling method of the space-time pulse signal provided in the above embodiment when executing the program instructions.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having computer-readable instructions stored thereon, and the computer-readable instructions are executed by a processor to implement a method for upsampling a space-time pulse signal provided in the above embodiment.

[0040] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0041] According to the upsampling method of spatiotemporal pulse signals provided in the embodiment of the present application, sparse pulse signals can be upsampled into dense pulse signals, thereby highlighting the main pulse signals of the moving subject and suppressing the generation of noise. The upsampled pulse signals are reliable and reliable, effectively solving the problems caused by the sparsity of spatiotemporal pulse signals, providing more effective data for visual tasks based on pulse signals, and improving the performance of downstream tasks based on pulses.

[0042] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0044] Figure 1 is a flow chart of a method for upsampling a spatiotemporal pulse signal according to an exemplary embodiment;

[0045] Figure 2 is a schematic diagram of an upsampling method of a spatiotemporal pulse signal according to an exemplary embodiment;

[0046] Figure 3 is a schematic diagram showing a method of collecting pulse signals based on a neuromorphic vision sensor according to an exemplary embodiment;

[0047] Figure 4 is a schematic diagram of motion trajectory estimation of a pulse signal according to an exemplary embodiment;

[0048] Figure 5 is a schematic diagram showing upsampling based on a point process model according to an exemplary embodiment;

[0049] Figure 6 is a schematic diagram showing an effect of performing task processing according to an up-sampled signal according to an exemplary embodiment;

[0050] Figure 7 is a structural schematic diagram of an up-sampling device for a spatiotemporal pulse signal according to an exemplary embodiment;

[0051] Figure 8 is a schematic structural diagram of an electronic device according to an exemplary embodiment;

[0052] Fig. 9 It is a schematic diagram of a computer storage medium according to an exemplary embodiment. DETAILED DESCRIPTION

[0053] The following description and the drawings sufficiently illustrate specific embodiments of the invention to enable those skilled in the art to practice them.

[0054] It should be clear that the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are only examples of systems and methods consistent with some aspects of the present invention as detailed in the attached claims.

[0056] At present, the sparsity of pulse signals of neuromorphic vision sensors in special scenarios is a daunting challenge for existing pulse-based tasks. In order to fundamentally solve the problem caused by the sparsity of pulse signals, the embodiment of the present application provides a method for upsampling spatiotemporal pulse signals.

[0057] First, the correct motion trajectory of the pulse signal is estimated by the motion estimation algorithm. Then, the main signal and noise are distinguished by the number of pulses on each trajectory. If the number of pulses on the trajectory is greater than the set threshold, the signal on the trajectory is considered to be the main pulse signal, otherwise it is noise. Finally, along the motion trajectory of the pulse, the two types of signals are modeled separately by the upsampling method, and the upsampling of the pulse signal is realized, and the final dense upsampled signal can be obtained.

[0058] The upsampling method based on spatiotemporal pulse signals provided by the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. Figure 1, the method specifically includes the following steps.

[0059] S101 collects the spatiotemporal pulse signal of the target object.

[0060] Among them, the target object is the object to be photographed, and the pulse signal is a discrete point set that is sparse in time and space, which records and represents the light intensity information in the scene in the form of pulses. The spatiotemporal pulse signal of the target object is collected by a neuromorphic vision sensor, which is a visual sensor that imitates the mechanism of retinal ganglion cells emitting pulses and retinal photoreceptor cells being sensitive to brightness changes. The emitted spatiotemporal pulse signal is a spatiotemporal sparse pulse signal, which has the advantages of high temporal resolution, high dynamic range, and low power consumption compared to traditional fixed frame rate cameras, and has huge market application potential in the fields of autonomous driving, robot vision navigation and positioning, etc.

[0061] In an optional embodiment, the neuromorphic vision sensor includes a dynamic vision sensor, which collects the spatiotemporal pulse signal of the target object. The dynamic vision sensor uses a differential sampling mechanism to sense the change of light intensity in the scene and generates an asynchronous pulse signal, which has the advantages of high temporal resolution, high dynamic range and low power consumption. It can overcome the defects of traditional cameras such as motion blur in high-speed scenes, overexposure in light-sensitive scenes, and unclear imaging in weak-exposure scenes. Dynamic vision sensors include but are not limited to: DVS, DAVIS, ATIS, Celex, etc.

[0062] In an optional embodiment, the neuromorphic vision sensor includes an ultra-high-speed full-time vision sensor, which collects the spatiotemporal pulse signals of the target object. The ultra-high-speed full-time vision sensor senses the absolute light intensity in the scene with an integral sampling mechanism, and emits synchronized pulse array signals at regular intervals, with the characteristics of high temporal resolution, clear texture and high dynamic range. It can overcome the high-speed motion blur of traditional cameras and has the ability of high dynamic imaging.

[0063] Figure 3 is a schematic diagram showing a method of collecting pulse signals based on a neuromorphic vision sensor according to an exemplary embodiment, such as Figure 3 As shown, the target object, that is, the object to be photographed, is sampled by a neuromorphic vision sensor to obtain a spatiotemporal pulse signal of the collected object. The pulse signal collected by the neuromorphic vision sensor is a pulse signal emitted and recorded under the stimulation of scene light intensity, and is a sparse discrete point set in the spatiotemporal domain.

[0064] S102 estimates the motion trajectory of the pulse signal, and determines the main pulse signal and the noise signal according to the number of pulses on the motion trajectory.

[0065] The collected spatiotemporal pulse signal is subjected to motion estimation to obtain the motion trajectory of the pulse signal. In one possible implementation, a traditional motion estimation algorithm designed for the special data format of the spatiotemporal pulse signal can be used, such as estimating the motion trajectory of the pulse signal by a contrast maximization algorithm or estimating the motion trajectory of the pulse signal by a minimization average timestamp algorithm.

[0066] Optionally, a deep learning method can also be used to estimate the motion trajectory of the pulse signal, for example, the pulse signal is converted into a two-dimensional image, and the two-dimensional image is input into a preset first neural network model to obtain the motion trajectory of the pulse signal. The preset first neural network model can be an EV-FlowNet neural network, an E-RAFT neural network, or a Spike-FlowNet neural network.

[0067] The trajectory of the pulse signal is obtained by motion estimation, that is, the motion trajectory obtains the motion direction in a short period of time, and the motion directions within a continuous period are pieced together to form the pulse motion trajectory of this period of time.

[0068] Furthermore, the main pulse signal and the noise signal are determined according to the number of pulses on the motion trajectory, wherein the main pulse signal is the spatiotemporal pulse signal sampled by the object due to the motion, and the noise signal is the isolated and meaningless signal generated by the instability of the ambient light or the problem of the neuromorphic vision sensor circuit itself.

[0069] In an optional embodiment, when the number of pulses on the motion trajectory is greater than a preset pulse number threshold, the pulse signal corresponding to the motion trajectory is the main pulse signal of the target object; when the number of pulses on the motion trajectory is less than or equal to the preset pulse number threshold, the pulse signal corresponding to the motion trajectory is the noise signal of the target object.

[0070] The preset pulse number threshold may be a fixed threshold, which is set according to the statistical results of specific data, or an adaptive threshold, which is adjusted at any time according to the motion characteristics of pulse signals in different scenes. The embodiment of the present application does not specifically limit the value of the threshold, which can be set according to actual conditions.

[0071] Figure 4 is a schematic diagram of motion trajectory estimation of a pulse signal according to an exemplary embodiment; Figure 4 As shown, the motion trajectory estimated along the correct trajectory and the motion trajectory estimated along the wrong trajectory are obtained, and the number of pulses of each trajectory is counted respectively. The trajectory pulse number greater than the threshold is the main pulse, and the trajectory pulse number less than or equal to the threshold is noise.

[0072] S103 upsamples the main pulse signal and the noise signal respectively to obtain the upsampled main pulse signal and the noise signal.

[0073] In an optional embodiment, the main pulse signal and the noise signal are upsampled respectively to obtain the upsampled main pulse signal and the noise signal, including: upsampling the main pulse signal and the noise signal respectively through a point process model to obtain the upsampled main pulse signal and the noise signal.

[0074] For example, the main pulse signal is upsampled through the Hawkes point process model to obtain the upsampled main pulse signal. In the Hawkes point process, each historical signal will have a positive effect on the current signal. The magnitude of this effect is determined by the length of time between the historical signal and the current signal. The longer the historical signal is, the smaller the impact on the current signal. The Hawkes point process is suitable for upsampling the main pulse signal, and can refer to the distribution of historical pulse signals to generate more reliable and effective main pulse signals.

[0075] Furthermore, the noise signal is upsampled through the self-correction point process model to obtain the upsampled noise signal. In the self-correction point process, the probability of noise occurrence will slowly increase over time, but this probability will decrease when new noise is generated. The self-correction point process is suitable for upsampling noise events to ensure that too much noise is not upsampled.

[0076] In an optional embodiment, the main pulse signal and the noise signal are upsampled respectively by a preset second neural network model to obtain an upsampled main pulse signal and a noise signal. The preset second neural network model may be a recurrent neural network, a sequence-to-sequence neural network model, a generative adversarial network, etc. A sparse pulse signal is used as input, and the pulse signal is upsampled using the preset second neural network model to output a dense upsampled pulse signal.

[0077] Figure 5 is a schematic diagram showing upsampling based on a point process model according to an exemplary embodiment; Figure 5 As shown, the main pulse signal is upsampled through the Hawkes point process model, the noise signal is upsampled through the self-correction point process model, and the upsampled main pulse and noise signals are merged to obtain the upsampled pulse signal. It can be seen that the upsampled pulse signal is denser.

[0078] S104 combines the up-sampled main pulse signal and the noise signal to generate a final up-sampled signal.

[0079] After obtaining the upsampled main pulse signal and noise signal, add the two together to merge into the final upsampled signal. This method upsamples the sparse pulse signal into a dense pulse signal, thereby highlighting the main pulse signal of the moving subject and suppressing the generation of noise. It can generate more effective and reliable data, serve downstream tasks, and improve the performance of corresponding tasks.

[0080] Furthermore, the upsampled dense pulse signal is input into downstream tasks based on pulse signals, such as image reconstruction, target detection, target recognition, etc. based on the upsampled signal. The effectiveness of the upsampled pulse signal and the improvement effect of upsampling on downstream tasks are verified. Figure 6 As shown, the original space-time pulse signal is upsampled to obtain a dense upsampled space-time pulse signal, and the original space-time pulse signal and the upsampled space-time pulse signal are respectively input into the downstream object detection task or image reconstruction task. It can be clearly seen that the reconstructed image and object signal detection based on the upsampled space-time pulse signal are clearer and the performance is better.

[0081] In order to facilitate understanding of the upsampling method of the spatiotemporal pulse signal provided in the embodiment of the present application, the following is a Figure 2 Provide explanation.

[0082] like Figure 2 As shown, the method includes collecting the spatiotemporal pulse signal of the moving subject, estimating the motion trajectory of the collected pulse signal, counting the number of pulses on each motion trajectory, and comparing it with a preset threshold. If it is greater than the threshold, it is considered to be the main pulse signal of the moving subject, otherwise it is a noise signal. Then the main pulse signal and the noise signal are upsampled respectively, for example, the main pulse signal is upsampled through the Hawkes point process model to obtain the upsampled main pulse signal. The noise signal is upsampled through the self-correction point process model to obtain the upsampled noise signal. The upsampled main pulse signal and the noise signal are merged to generate the final upsampled signal. The final upsampled pulse signal is input into the downstream task for image reconstruction, target detection, target tracking, etc.

[0083] According to the upsampling method of spatiotemporal pulse signals provided in the embodiment of the present application, sparse pulse signals can be upsampled into dense pulse signals, thereby highlighting the main pulse signals of the moving subject and suppressing the generation of noise. The upsampled pulse signals are reliable and reliable, effectively solving the problems caused by the sparsity of spatiotemporal pulse signals, providing more effective data for visual tasks based on pulse signals, and improving the performance of downstream tasks based on pulses.

[0084] The present application also provides a device for upsampling a space-time pulse signal, which is used to perform the method for upsampling a space-time pulse signal in the above embodiment. Figure 7 As shown, the device comprises:

[0085] The acquisition module 701 is used to acquire the spatiotemporal pulse signal of the target object;

[0086] A motion estimation module 702 is used to estimate the motion trajectory of the pulse signal and determine the main pulse signal and the noise signal according to the number of pulses on the motion trajectory;

[0087] An up-sampling module 703 is used to up-sample the main pulse signal and the noise signal respectively to obtain the up-sampled main pulse signal and the noise signal;

[0088] The merging module 704 is used to merge the up-sampled main pulse signal and the noise signal to generate a final up-sampled signal.

[0089] In an optional embodiment, the motion estimation module includes:

[0090] a motion estimation unit, configured to estimate the motion trajectory of the impulse signal by using a contrast maximization algorithm; or,

[0091] Estimate the motion trajectory of the pulse signal by minimizing the average timestamp algorithm; or,

[0092] The pulse signal is converted into a two-dimensional image, and the two-dimensional image is input into a preset first neural network model to obtain a motion trajectory of the pulse signal;

[0093] It also includes a pulse signal classification unit, which is used to classify the pulse signal corresponding to the motion trajectory as the main pulse signal of the target object when the number of pulses on the motion trajectory is greater than a preset pulse number threshold;

[0094] When the number of pulses on the motion trajectory is less than or equal to a preset pulse number threshold, the pulse signal corresponding to the motion trajectory is a noise signal of the target object.

[0095] It should be noted that the up-sampling device for space-time pulse signals provided in the above-mentioned embodiment only uses the division of the above-mentioned functional modules as an example when executing the up-sampling method for space-time pulse signals. In practical applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the up-sampling device for space-time pulse signals provided in the above-mentioned embodiment and the up-sampling method embodiment for space-time pulse signals belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0096] An embodiment of the present application also provides an electronic device corresponding to the up-sampling method of the space-time pulse signal provided in the above embodiment, so as to execute the above-mentioned up-sampling method of the space-time pulse signal.

[0097] Please refer to Figure 8 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 8 As shown, the electronic device includes: a processor 800, a memory 801, a bus 802 and a communication interface 803, and the processor 800, the communication interface 803 and the memory 801 are connected via the bus 802; the memory 801 stores a computer program that can be run on the processor 800, and when the processor 800 runs the computer program, it executes the upsampling method of the space-time pulse signal provided in any of the aforementioned embodiments of the present application.

[0098] The memory 801 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 803 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0099] The bus 802 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 801 is used to store programs, and the processor 800 executes the program after receiving the execution instruction. The up-sampling method of the spatiotemporal pulse signal disclosed in any implementation of the embodiment of the present application may be applied to the processor 800, or implemented by the processor 800.

[0100] The processor 800 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 800. The above processor 800 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 801, and the processor 800 reads the information in the memory 801 and completes the steps of the above method in combination with its hardware.

[0101] The electronic device provided in the embodiment of the present application and the up-sampling method of the spatiotemporal pulse signal provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0102] The present application also provides a computer-readable storage medium corresponding to the upsampling method of the spatiotemporal pulse signal provided in the above embodiment. Fig. 9 The computer-readable storage medium shown is a CD 900 on which a computer program (ie, a program product) is stored. When the computer program is executed by a processor, the upsampling method of the spatiotemporal pulse signal provided in any of the aforementioned embodiments will be executed.

[0103] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0104] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the upsampling method of the spatiotemporal pulse signal provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0105] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0106] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the attached claims.

Claims

1. A method for upsampling a spatiotemporal pulse signal, characterized in that ,include: Collecting the spatiotemporal pulse signal of the target object; wherein the target object is the object to be photographed, and the spatiotemporal pulse signal is a discrete point set that is sparse in time and space, and records and represents the light intensity information in the scene in the form of pulses; Estimate the motion trajectory of the pulse signal, and determine the main pulse signal and the noise signal according to the number of pulses on the motion trajectory; including: when the number of pulses on the motion trajectory is greater than a preset pulse number threshold, the pulse signal corresponding to the motion trajectory is the main pulse signal of the target object; when the number of pulses on the motion trajectory is less than or equal to the preset pulse number threshold, the pulse signal corresponding to the motion trajectory is the noise signal of the target object; Upsampling the main pulse signal and the noise signal respectively to obtain an upsampled main pulse signal and a noise signal; The up-sampled main pulse signal and the noise signal are combined to generate a final up-sampled signal.

2. The method according to claim 1, characterized in that: Collect the spatiotemporal pulse signals of the target object, including: The spatiotemporal pulse signal of the target object is collected by a dynamic vision sensor; the dynamic vision sensor includes: DVS, DAVIS, ATIS, Celex.

3. The method according to claim 1, characterized in that Estimating the motion trajectory of the pulse signal, comprising: Estimate the motion trajectory of the pulse signal by using a contrast maximization algorithm; or, Estimate the motion trajectory of the pulse signal by minimizing the average timestamp algorithm; or, The pulse signal is converted into a two-dimensional image, and the two-dimensional image is input into a preset first neural network model to obtain a motion trajectory of the pulse signal, wherein the preset first neural network model adopts an EV-FlowNet neural network, an E-RAFT neural network or a Spike-FlowNet neural network.

4. The method according to claim 1, characterized in that Upsampling the main pulse signal and the noise signal respectively to obtain the upsampled main pulse signal and the noise signal, including: Up-sampling the main pulse signal and the noise signal respectively through a point process model to obtain an up-sampled main pulse signal and a noise signal; or, The main pulse signal and the noise signal are upsampled respectively through a preset second neural network model to obtain an upsampled main pulse signal and a noise signal; wherein the preset second neural network model includes a recurrent neural network, a sequence-to-sequence neural network model, and a generative adversarial network.

5. The method according to claim 4, characterized in that The main pulse signal and the noise signal are respectively upsampled by a point process model to obtain the upsampled main pulse signal and the noise signal, including: Upsampling the main pulse signal through a Hawkes point process model to obtain an upsampled main pulse signal; The noise signal is upsampled through a self-correction point process model to obtain an upsampled noise signal.

6. An upsampling device for a spatiotemporal pulse signal, characterized in that: include: An acquisition module is used to acquire the spatiotemporal pulse signal of the target object; wherein the target object is the object to be photographed, and the spatiotemporal pulse signal is a discrete point set that is sparse in time and space, and records and represents the light intensity information in the scene in the form of pulses; A motion estimation module, used for estimating the motion trajectory of the pulse signal, and determining the main pulse signal and the noise signal according to the number of pulses on the motion trajectory; the motion estimation module includes a pulse signal classification unit, used for when the number of pulses on the motion trajectory is greater than a preset pulse number threshold, the pulse signal corresponding to the motion trajectory is the main pulse signal of the target object; when the number of pulses on the motion trajectory is less than or equal to the preset pulse number threshold, the pulse signal corresponding to the motion trajectory is the noise signal of the target object; An up-sampling module, used to up-sample the main pulse signal and the noise signal respectively to obtain the up-sampled main pulse signal and the noise signal; The merging module is used to merge the up-sampled main pulse signal and the noise signal to generate a final up-sampled signal.

7. The device according to claim 6, characterized in that The motion estimation module comprises: a motion estimation unit, configured to estimate the motion trajectory of the pulse signal by using a contrast maximization algorithm; or, Estimate the motion trajectory of the pulse signal by minimizing the average timestamp algorithm; or, The pulse signal is converted into a two-dimensional image, and the two-dimensional image is input into a preset first neural network model to obtain a motion trajectory of the pulse signal; wherein the preset first neural network model adopts an EV-FlowNet neural network, an E-RAFT neural network or a Spike-FlowNet neural network.

8. An electronic device, characterized in that: It comprises a processor and a memory storing program instructions, wherein the processor is configured to execute the up-sampling method of the spatiotemporal pulse signal as claimed in any one of claims 1 to 5 when executing the program instructions.

9. A computer-readable medium, characterized in that Computer-readable instructions are stored thereon, and the computer-readable instructions are executed by a processor to implement a method for up-sampling a spatiotemporal pulse signal as described in any one of claims 1 to 5.

Images