Signal processing method, signal processing apparatus, signal processor, device, and medium

Abstract

The present disclosure provides a signal processing method, a signal processing apparatus, a signal processor, a device, and a medium. The method comprises: separately sampling a preset target by means of a first sampler and a second sampler, to obtain multiple first sampling signals of a first sampling frequency and multiple second sampling signals of a second sampling frequency; for a prediction moment, performing signal prediction on the basis of a historical output signal, to determine a predicted sampling signal at the prediction moment, the historical output signal comprising at least one sampling output signal before the prediction moment; on the basis of the predicted sampling signal at the prediction moment and the second sampling signal at the prediction moment, determining an error comparison result; and, on the basis of the error comparison result, selecting and outputting a sampling output signal from among the first sampling signals and the second sampling signals.

Description

Signal processing method, signal processing device, signal processor, device and medium Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular, to a signal processing method, a signal processing device, a signal processor, an electronic device, and a computer-readable storage medium. Background Art

[0002] A key foundation of modern signal processing is the Shannon-Nyquist sampling theorem, which states that for a band-limited signal, if its sampling frequency is twice the signal bandwidth, the original signal can be reconstructed without distortion.

[0003] Summary of the Invention

[0004] The embodiments of the present disclosure provide a signal processing method, a signal processing device, a signal processor, an electronic device, and a computer-readable storage medium, which are suitable for signal acquisition in scenarios such as frequency instability. On the one hand, they can reduce sampling noise and data redundancy, and on the other hand, they can ensure the quality of signal reconstruction.

[0005] In a first aspect, the present disclosure provides a signal processing method, which includes: sampling a preset target by a first sampler and a second sampler respectively to obtain multiple first sampling signals of a first sampling frequency and multiple second sampling signals of a second sampling frequency, wherein the first sampling frequency of the first sampler is less than the second sampling frequency of the second sampler; for a predicted moment, performing signal prediction based on a historical output signal to determine a predicted sampling signal at the predicted moment, wherein the historical output signal includes at least one sampling output signal before the predicted moment; determining an error comparison result based on the predicted sampling signal at the predicted moment and the second sampling signal at the predicted moment; and selecting and outputting a sampling output signal from the first sampling signal and the second sampling signal based on the error comparison result.

[0006] In a second aspect, the present disclosure provides a signal processing device, which includes: a sampling module, which is used to sample a preset target through a first sampler and a second sampler, respectively, to obtain multiple first sampling signals of a first sampling frequency and multiple second sampling signals of a second sampling frequency, wherein the first sampling frequency of the first sampler is less than the second sampling frequency of the second sampler; a prediction module, which is used to perform signal prediction based on historical output signals for a prediction moment, and determine a predicted sampling signal at the prediction moment, wherein the historical output signal includes at least one sampling output signal before the prediction moment; a determination module, which is used to determine an error comparison result based on the predicted sampling signal at the prediction moment and the second sampling signal at the prediction moment; and an output module, which is used to select and output a sampling output signal from the first sampling signal and the second sampling signal based on the error comparison result.

[0007] In a third aspect, the present disclosure provides a signal processor comprising: at least one first sampler, a second sampler and a signal processing device; wherein the signal processing device is used to execute the signal processing method described in any one of the embodiments of the present disclosure.

[0008] In a fourth aspect, the present application provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor so that the at least one processor can execute the signal processing method described in any one of the embodiments of the present disclosure.

[0009] In a fifth aspect, the present application provides a computer-readable storage medium on which a computer program is stored, wherein the computer program implements the signal processing method described in any one of the embodiments of the present disclosure when executed by a processor / processing core.

[0010] In a sixth aspect, an embodiment of the present disclosure provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements any one of the signal processing methods in the embodiments of the present disclosure.

[0011] In the embodiments provided by the present disclosure, a preset target is sampled by a first sampler and a second sampler, respectively, to obtain a plurality of first sampling signals of a first sampling frequency and a plurality of second sampling signals of a second sampling frequency, wherein the first sampling frequency of the first sampler is less than the second sampling frequency of the second sampler; for a prediction moment, a signal prediction is performed based on a historical output signal to determine a prediction sampling signal at the prediction moment, wherein the historical output signal includes at least one sampling output signal before the prediction moment; an error comparison result is determined based on the prediction sampling signal at the prediction moment and the second sampling signal at the prediction moment; and based on the error comparison result, a sampling output signal is selected from the first sampling signal and the second sampling signal and output.

[0012] It can be seen from this that, considering that the frequency of the preset target may be variable, if only a fixed sampling frequency is used to sample the preset target, it may not be possible to take into account both the sampling effect and the redundancy of the sampled output data. Therefore, the embodiment of the present disclosure uses at least part of the historical output signal to perform signal prediction, and compares the predicted sampling signal with the actual high-frequency sampling signal corresponding to the prediction moment to obtain an error comparison result. The error comparison result can be used to determine whether the predicted sampling signal is relatively accurate. When the error comparison result shows that the predicted sampling signal is relatively accurate, the first sampling signal with a lower sampling frequency can be outputted externally. When the error comparison result shows that the accuracy of the predicted sampling signal is low, the second sampling signal with a higher sampling frequency can be outputted externally. In this way, when the preset target is at a low frequency, the first sampling signal with a lower sampling frequency can be outputted, and when the preset target is at a high frequency, the second sampling signal with a higher sampling frequency can be outputted. Accordingly, when performing signal reconstruction, for low-frequency situations, relatively accurate signal reconstruction can be achieved using only the first sampling signal. Since the sampling frequency is low, the sampling noise is correspondingly low, and there is less redundant data. For high-frequency situations, the second sampling signal can be used for signal reconstruction. Since the sampling frequency of the second sampling signal is high, a reconstruction result with higher accuracy can be obtained. In summary, the embodiments of the present disclosure are suitable for signal acquisition in scenarios such as frequency instability. On the one hand, it can reduce sampling noise and data redundancy, and on the other hand, it can also ensure the quality of signal reconstruction.

[0013] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG1 is a flowchart of a signal processing method provided by an embodiment of the present disclosure.

[0015] FIG2 is a schematic diagram of a pixel provided by an embodiment of the present disclosure.

[0016] FIG3 is a schematic diagram of a pixel array provided by an embodiment of the present disclosure.

[0017] FIG4 is a schematic diagram of a signal processing method provided by an embodiment of the present disclosure.

[0018] FIG5 is a schematic diagram of a signal processing method provided by an embodiment of the present disclosure.

[0019] FIG6 is a schematic diagram of a signal processing method provided by an embodiment of the present disclosure.

[0020] FIG7 is a schematic diagram of a signal processing method provided by an embodiment of the present disclosure.

[0021] FIG8 is a schematic diagram of a sampling signal and a sampling output signal provided by an embodiment of the present disclosure.

[0022] FIG9 is a flow chart of a signal processing method provided in an embodiment of the present disclosure.

[0023] FIG10 is a block diagram of a signal processing device provided by an embodiment of the present disclosure.

[0024] FIG11 is a block diagram of a signal processor provided by an embodiment of the present disclosure.

[0025] FIG12 is a block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The present disclosure will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the present disclosure and are not intended to limit the present disclosure. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present disclosure, not all structures.

[0027] In the related art, in most sampling scenarios, the frequency of the sampled signal is usually much smaller than the sampling frequency. In this way, the quality of signal reconstruction can be improved to a certain extent. For scenarios where the frequency of the sampled signal is variable, there may be situations where the signal frequency is less than the sampling frequency, the signal frequency is equal to the sampling frequency, and the signal frequency is greater than the sampling frequency, which may cause problems of oversampling (the sampling frequency is much greater than the signal frequency) and undersampling (the sampling frequency is less than the signal frequency). For the oversampling problem, although the signal can be reconstructed well, a large amount of redundant data is usually generated, thereby increasing the amount of data processing; for the undersampling problem, it may lead to data loss, and the quality of signal reconstruction cannot be guaranteed (for example, the illusion of the wheel hub being reversed).

[0028] It can be seen from this that the relevant technology has not proposed a reasonable sampling method that can effectively reduce the redundancy of the sampled data while taking into account the signal reconstruction quality when sampling is performed using this sampling method.

[0029] In view of this, embodiments of the present disclosure provide a signal processing method, a signal processing device, a signal processor, an electronic device, and a computer-readable storage medium.

[0030] In an embodiment of the present disclosure, a preset target is sampled by a first sampler and a second sampler, respectively, to obtain a plurality of first sampling signals of a first sampling frequency and a plurality of second sampling signals of a second sampling frequency, wherein the first sampling frequency of the first sampler is less than the second sampling frequency of the second sampler; for a prediction moment, a signal prediction is performed based on a historical output signal to determine a prediction sampling signal at the prediction moment, wherein the historical output signal includes at least one sampling output signal before the prediction moment; an error comparison result is determined based on the prediction sampling signal at the prediction moment and the second sampling signal at the prediction moment; and based on the error comparison result, a sampling output signal is selected from the first sampling signal and the second sampling signal and output.

[0031] It can be seen from this that, considering that the frequency of the preset target may be variable, if only a fixed sampling frequency is used to sample the preset target, it may not be possible to take into account both the sampling effect and the redundancy of the sampled output data. Therefore, the embodiment of the present disclosure uses at least part of the historical output signal to perform signal prediction, and compares the predicted sampling signal with the actual high-frequency sampling signal corresponding to the prediction moment to obtain an error comparison result. The error comparison result can be used to determine whether the predicted sampling signal is relatively accurate. When the error comparison result shows that the predicted sampling signal is relatively accurate, the first sampling signal with a lower sampling frequency can be outputted externally. When the error comparison result shows that the accuracy of the predicted sampling signal is low, the second sampling signal with a higher sampling frequency can be outputted externally. In this way, when the preset target is at a low frequency, the first sampling signal with a lower sampling frequency can be outputted, and when the preset target is at a high frequency, the second sampling signal with a higher sampling frequency can be outputted. Accordingly, when performing signal reconstruction, for low-frequency scenarios, relatively accurate signal reconstruction can be achieved using only the first sampling signal. Since the sampling frequency is low, the sampling noise is correspondingly low, and there is less redundant data. For high-frequency scenarios, the second sampling signal can be used for signal reconstruction. Since the sampling frequency of the second sampling signal is higher, a more accurate reconstruction result can be obtained. In summary, the disclosed embodiments are suitable for signal acquisition in scenarios such as frequency instability. On the one hand, they can reduce sampling noise and data redundancy, and on the other hand, they can ensure the quality of signal reconstruction.

[0032] A first aspect of an embodiment of the present disclosure provides a signal processing method.

[0033] Figure 1 is a flow chart of a signal processing method provided by an embodiment of the present disclosure. Referring to Figure 1 , the signal processing method may include the following steps.

[0034] In step S11, the preset target is sampled by the first sampler and the second sampler respectively to obtain multiple first sampling signals of the first sampling frequency and multiple second sampling signals of the second sampling frequency, and the first sampling frequency of the first sampler is lower than the second sampling frequency of the second sampler.

[0035] In step S12, for the prediction moment, signal prediction is performed based on the historical output signal to determine the predicted sampling signal at the prediction moment, where the historical output signal includes at least one sampling output signal before the prediction moment.

[0036] In step S13, an error comparison result is determined based on the predicted sampling signal at the predicted time and the second sampling signal at the predicted time.

[0037] In step S14 , based on the error comparison result, a sampling output signal is selected from the first sampling signal and the second sampling signal and output.

[0038] It can be seen from this that in the embodiment of the present disclosure, during the signal sampling process, a first sampler with a relatively low sampling frequency and a second sampler with a relatively high sampling frequency are provided, and both sample the preset target according to their respective sampling frequencies; when outputting the sampling signal, the sampling signals of all samplers (i.e., the first sampler and the second sampler) are not all output, but rather it is determined based on certain conditions which sampling signals can be output as sampling output signals and which sampling signals will not be added to the sampling output signals for output. In this way, on the one hand, it can be ensured that the sampling signal is relatively sufficient, and the problem of data loss due to insufficient sampling will not be caused, thereby ensuring the quality of signal reconstruction; on the other hand, it can also reduce the output of redundant sampling signals, thereby achieving high-quality signal reconstruction with a higher data utilization rate.

[0039] In some optional implementations, the preset target is the object to be sampled, which may correspond to one-dimensional data (for example, audio data), two-dimensional data (for example, two-dimensional image), three-dimensional data (for example, three-dimensional image with depth information), or even nD (n-Dimensions) data (for example, multi-dimensional point cloud data). The embodiments of the present disclosure are not limited to this.

[0040] In some optional implementations, the first sampler and the second sampler are both samplers for sampling preset targets, and the first sampler and the second sampler can collect different types of signals. Accordingly, the first sampling signal and the second sampling signal belong to different types of sampling signals, respectively.

[0041] In some optional implementations, if the preset target is in audio format, the first sampler may be configured to collect a sound intensity signal, and accordingly, the first sampled signal includes a sound intensity signal. The second sampler may be configured to collect a pitch signal, and accordingly, the second sampled signal includes a pitch signal. Sound intensity reflects the strength of a sound and is typically determined by its amplitude, while pitch reflects the highness of a sound and is typically related to its frequency.

[0042] In some optional implementations, if the preset target is in the form of an image, the first sampler may be used to collect color signals, and accordingly, the first sampling signal includes a color sampling signal; the second sampler may be used to collect light intensity signals, and accordingly, the second sampling signal may include an absolute light intensity signal and / or a relative light intensity signal. The color sampling signal is a signal about color (for example, the color sampling signal may include an RGB sampling signal, where RGB corresponds to red, green, and blue, respectively), the absolute light intensity signal is a signal reflecting the absolute value of the light intensity at the sampling moment, and the relative light intensity signal is a signal reflecting the change in light intensity. Moreover, the change in light intensity may refer to the change in light intensity in the spatial dimension or the change in light intensity in the temporal dimension.

[0043] Exemplarily, the first sampling signal includes a color sampling signal, and the second sampling signal includes a spatiotemporal differential sampling signal. The spatiotemporal differential sampling signal includes a time differential sub-signal and a spatial differential sub-signal. The time differential sub-signal represents the difference in light intensity between two adjacent second sampling moments in a sampling region corresponding to a preset target, while the spatial differential sub-signal represents the difference in light intensity between the sampling region and adjacent sampling regions at the same second sampling moment. In other words, the color sampling signal is a sampling signal obtained by color sampling the sampling region corresponding to the preset target, while the spatiotemporal differential sampling signal reflects the difference in light intensity within the sampling region from both the temporal and spatial dimensions.

[0044] For example, at the first sampling time t11, the first sampling signal collected by the first sampler includes a color sampling signal of the sampling area Q1 corresponding to the preset target, and the color sampling signal includes a red sampling component, a green sampling component, and a blue sampling component.

[0045] For example, at the second sampling time t21, the second sampling signal collected by the second sampler includes a spatiotemporal differential sampling signal of the sampling area Q1 corresponding to the preset target. The spatial differential sub-signal in the spatiotemporal differential sampling signal can reflect the light intensity difference information between the sampling area Q1 and the adjacent sampling areas Q2, Q3, Q4 and Q5 at the second sampling time t21. The temporal differential sub-signal in the spatiotemporal differential sampling signal can reflect the light intensity difference information between the sampling area Q1 at the second sampling time t21 and the previous second sampling time t20, where 1 / (t21-t20)=f2, f2 is the second sampling frequency of the second sampler, and Q2, Q3, Q4, and Q5 are adjacent sampling areas located above, below, left, and right of Q1, respectively.

[0046] In some optional implementations, in step S11, sampling a preset target by a first sampler and a second sampler, respectively, to obtain multiple first sampling signals at a first sampling frequency and multiple second sampling signals at a second sampling frequency, includes: performing color sampling on the preset target by the first sampler to obtain multiple first sampling signals at the first sampling frequency, wherein the sampling area corresponding to the preset target includes multiple pixels, and the first sampling signal includes multiple color sampling signals corresponding to the multiple pixels; and performing light intensity differential sampling on the preset target by the second sampler to obtain multiple second sampling signals at a second sampling frequency, wherein the second sampling signal includes multiple spatiotemporal differential sampling signals corresponding to the multiple pixels. Light intensity differential sampling refers to collecting light intensity change signals of the sampling area from both the temporal and spatial dimensions.

[0047] As can be seen, when sampling any pixel, the first sampler can acquire the pixel's color sampling signal, while the second sampler can acquire the pixel's spatiotemporal differential sampling signal. Furthermore, a sampling output signal can be subsequently selected from the first and second sampling signals through signal prediction, error comparison, and other methods, and output externally, so that the sampling output signal sequence corresponding to the preset target may include both the color sampling signal and the light intensity differential sampling signal.

[0048] It should be noted that for the color sampling signal, it is necessary to collect the color signal of each sampling area of ​​the preset target, and the color signal is usually a non-zero value. Therefore, if the color sampling signal is represented in a matrix form, most of the values ​​in the corresponding color matrix are non-zero values.

[0049] For the spatiotemporal differential sampling signal, whether in the time dimension or the space dimension, what is collected is the light intensity difference signal. Considering that the change of the preset target is continuous and is a gradual process, therefore, in the two adjacent second sampling moments, for the same sampling area of ​​the preset target, the light intensity change may be small or there is no light intensity change, which is reflected in the time difference sub-signal, showing that most of the time difference sub-signals are zero, and only a small number of sampling areas may have non-zero values ​​(non-zero values ​​represent that the light intensity of the corresponding sampling area has changed between the two adjacent second sampling moments). Similarly, within the preset target, the light intensity of adjacent positions is usually relatively close. Therefore, for any second sampling moment, most of the spatial difference sub-signals of the preset target are also zero, and only a small number of sampling areas may have non-zero values ​​(non-zero values ​​represent that there is a difference in light intensity between the current sampling area and the adjacent sampling area). If the above-mentioned time difference sub-signals and spatial difference sub-signals are represented in matrix form, the corresponding time difference matrix and spatial difference matrix are sparse matrices.

[0050] Furthermore, if the sampling area of ​​the preset target is represented as N×N, the corresponding color sampling signal can be represented as W1 N×N , the spatiotemporal differential sampling signal can be characterized as W2 N×N =(W21 N×N , W22 N×N ), among which W21 N×N represents the time difference molecular signal, W22 N×N Represents the spatial differential subsignal. If the same data accuracy is used for numerical representation, then due to W1 N×N is a non-sparse matrix, so W1 N×N The amount of data may be large, W21 N×N and W22 N×N Both are sparse matrices, so W2 N×N The amount of data is relatively small.

[0051] Assume that the data size of the color sampling signal obtained by each color signal acquisition is a1 (that is, assume W1 N×N The data volume is a1), if the first sampling frequency is f1, the second sampling frequency is f2 (f1<f2), and the first sampler and the second sampler are both used to collect color signals, then in the time period T, the total data volume of all sampling signals is (T×f1)×a1+(T×f2)×a1, and the data volume of the sampled output signals ranges from [(T×f1)×a1, (T×f2)×a1], wherein the data volume range of (T×f1)×a1 indicates that the sampled output signals are all first sampling signals, and the data volume range of (T×f2)×a1 indicates that the sampled output signals are all second sampling signals.

[0052] Assume that the data size of the space-time differential signal obtained by each space-time differential sampling is a2 (that is, assume W2 N×N The data volume is a2, and a2 is much smaller than a1), and the first sampler is used to collect color signals, and the second sampler is used to collect spatiotemporal differential signals. Then, within the time period T, the total data volume of all sampling signals is (T×f1)×a1+(T×f2)×a2, and the data volume of the sampling output signal ranges from [(T×f2)×a2, (T×f1)×a1]. Among them, when the data volume range is (T×f2)×a2, it indicates that the sampling output signals are all second sampling signals, and when the data volume range is (T×f1)×a1, it indicates that the sampling output signals are all first sampling signals.

[0053] Since a2 is much smaller than a1, by comparing (T×f1)×a1+(T×f2)×a1 and (T×f1)×a1+(T×f2)×a2, we can see that in the second sampling method (i.e., the first sampler collects the color signal and the second sampler collects the light intensity difference signal), the total data volume of the sampled signal is relatively small. Therefore, adopting the second sampling method can effectively reduce the amount of sampled data and alleviate the pressure on the sampled data cache. Similarly, by comparing [(T×f1)×a1, (T×f2)×a1] and [(T×f2)×a2, (T×f1)×a1], we can see that the data volume of the sampled output signal of the second sampling method is smaller than that of the sampled output signal of the first sampling method.

[0054] In summary, in the embodiment of the present disclosure, by using a first sampler with a lower sampling frequency to collect color signals with a relatively large data volume, and using a second sampler with a higher sampling frequency to collect spatiotemporal differential signals with a relatively small data volume, the amount of sampled data and the amount of sampled output signals can be further reduced, thereby effectively reducing data redundancy while also meeting higher signal reproduction quality requirements.

[0055] From this, we can see that by collecting different types of signals at different sampling frequencies, on the one hand, it can ensure the adequacy of the sampling data and effectively alleviate the low reconstruction quality caused by insufficient sampling. On the other hand, it can also reduce the amount of sampling data and reduce the output of redundant sampling data.

[0056] It should be noted that since the second sampler performs light intensity differential sampling on the preset target, the second sampler is equivalent to collecting the light intensity change information of the preset target, and usually, a light intensity change event will only occur when the light intensity of the preset target changes, thereby generating the corresponding spatiotemporal differential sampling signal. When sampling the color signal, global color sampling is required for each sampling moment. It can be seen that different sampling methods can effectively improve perception efficiency.

[0057] In some optional implementations, the second sampling signal includes a spatial difference sub-signal and a time difference sub-signal. Accordingly, the preset target is subjected to light intensity differential sampling by the second sampler to obtain multiple second sampling signals of the second sampling frequency, including: the preset target is subjected to light intensity differential sampling by the second sampler to obtain an excitation light intensity signal and multiple inhibition light intensity signals corresponding to each pixel at the current second sampling moment, the excitation light intensity signal is a signal obtained by sampling the excitation type photosensitive unit in the pixel, and the inhibition light intensity signal is a signal obtained by sampling the inhibition type photosensitive unit in the pixel; based on the multiple inhibition light intensity signals, the average inhibition light intensity signal corresponding to each pixel at the current second sampling moment is determined; based on the first difference value between the excitation light intensity signal of each pixel at the current second sampling moment and the corresponding average inhibition light intensity signal, the spatial difference sub-signal corresponding to each pixel at the current second sampling moment is obtained; based on the second difference value between the excitation light intensity signal of each pixel at the current second sampling moment and the excitation light intensity signal of each pixel at the previous second sampling moment, the time difference sub-signal corresponding to each pixel at the current second sampling moment is obtained.

[0058] It can be seen from this that a pixel may include an excitatory photosensitive unit and at least one inhibitory photosensitive unit. When the second sampler is sampling, it collects an excitatory light intensity signal for the excitatory photosensitive unit and an inhibitory light intensity signal for the inhibitory photosensitive unit. Moreover, based on the first difference value between the excitatory light intensity signal and the average inhibitory light intensity signal of the inhibitory photosensitive unit, the spatial difference sub-signal corresponding to the pixel at the current second sampling moment can be obtained; based on the second difference value between the excitatory light intensity signal of the pixel at the current second sampling moment and the excitatory light intensity signal of the pixel at the previous second sampling moment, the time difference sub-signal corresponding to the pixel at the current second sampling moment can be obtained.

[0059] Figure 2 is a schematic diagram of a pixel provided by an embodiment of the present disclosure. Referring to Figure 2, a pixel is arrayed with multiple units, including four inhibitory photosensitive units at the four corners of the pixel, one excitatory photosensitive unit at the center of the pixel, and four color units at other locations.

[0060] During signal acquisition, the four color units are sampled based on the first sampler to obtain multiple color sampling signals of the first sampling frequency. At the same time, the inhibitory photosensitive unit and the four excitatory photosensitive units are sampled respectively based on the second sampler to obtain an excitatory light intensity signal and four inhibitory light intensity signals of the second sampling frequency.

[0061] For example, for the second sampling time t, if the corresponding exciting light intensity signal is x1, and the four inhibiting light intensity signals are y1, y2, y3 and y4, the average inhibiting light intensity signal y can be obtained based on the mean calculation. 均值 =(y1+y2+y3+y4) / 4, and determine the spatial differential sub-signal s1=x1-y 均值 . Further, if the sampling interval of the second sampling moment is ΔT, and assuming that the exciting light intensity signal at the previous second sampling moment t-ΔT is x2, then the time difference molecular signal s2 corresponding to the second sampling moment t = x1-x2. Among them, the time difference molecular signal s2 is a signed value. When its value is a positive number, it means that the light intensity at the current second sampling moment is higher than the light intensity at the previous second sampling moment. Conversely, if s2 is a negative number, it means that the light intensity at the current second sampling moment is lower than the light intensity at the previous second sampling moment.

[0062] By alternately arranging excitatory photosensitive units and inhibitory photosensitive units in this array, the ability to perceive changes in light intensity of light signals can be improved, thereby collecting sampling signals with richer and more accurate light intensity representation information.

[0063] In some optional implementations, in order to simulate the gap junction visual mode of the human eye, a connection switch can be set between the excitatory photosensitive unit and the inhibitory photosensitive unit of the pixel. When the ambient light is strong, the excitatory photosensitive unit is connected to fewer inhibitory photosensitive units by adjusting the opening and closing of the connection switch (for example, the excitatory photosensitive unit is only connected to one inhibitory photosensitive unit). When the ambient light is weak, the excitatory photosensitive unit is connected to more inhibitory photosensitive units by adjusting the opening and closing of the connection switch (for example, the excitatory photosensitive unit is connected to all four inhibitory photosensitive units).

[0064] In some optional implementations, before performing light intensity differential sampling on a preset target through a second sampler to obtain an excitation light intensity signal and multiple inhibition light intensity signals corresponding to each pixel at the current second sampling moment, the method also includes: obtaining the connection status between the excitation-type photosensitive unit and the multiple inhibition-type photosensitive units in each pixel; performing light intensity differential sampling on the preset target through the second sampler to obtain an excitation light intensity signal and multiple inhibition light intensity signals corresponding to each pixel at the current second sampling moment; based on the connection status corresponding to each pixel, performing light intensity differential sampling on the preset target through the second sampler to obtain an excitation light intensity signal and at least one effective inhibition light intensity signal corresponding to each pixel at the current second sampling moment, the inhibition light intensity signal is a signal obtained by sampling the inhibition photosensitive unit in the pixel whose connection status with the excitation-type photosensitive unit is open.

[0065] In other words, before performing light intensity differential sampling, the second sampler needs to clearly identify which inhibitory photosensitive units and excitatory photosensitive units are in the on state and which inhibitory photosensitive units and excitatory photosensitive units are in the off state. Then, when performing light intensity differential sampling, only the inhibitory photosensitive units in the on state are sampled to obtain the corresponding effective inhibition light intensity signal. Alternatively, the second sampler can also sample all inhibitory photosensitive units, but only use the effective inhibition light intensity signal when participating in the subsequent light intensity differential calculation.

[0066] In some optional implementations, a plurality of pixels may be arranged in an array to form a pixel array, and at least one inhibitory photosensitive unit may be shared between two adjacent pixels.

[0067] Figure 3 is a schematic diagram of a pixel array provided by an embodiment of the present disclosure. Referring to Figure 3 , the pixel array is composed of a plurality of pixel arrays arranged.

[0068] As shown in Figure 3, the first pixel in the upper left corner of the pixel array is labeled pixel 1, and the second adjacent pixel is labeled pixel 2. The suppression-type photosensitive unit located in row 1, column 3 of the pixel array and the suppression-type photosensitive unit located in row 3, column 3 belong to both the two suppression-type photosensitive units on the right side of pixel 1 and the two suppression-type photosensitive units on the left side of pixel 2. In other words, pixel 1 and pixel 2 share these two suppression-type photosensitive units. The other pixels are similar and will not be described in detail here.

[0069] This pixel arrangement enables the reuse of suppressive photosensitive units and improves the pixel fill factor, thereby enhancing the accuracy of the sampled signal. Furthermore, when designing chips such as sensors, the above pixel array layout and the signal sampling and output methods used by the data collector for the pixel array can be used as a reference to achieve chips with high sensing efficiency and better reconstruction results.

[0070] In some optional implementations, the first sampler and the second sampler may sample the same type of signal.

[0071] Exemplarily, the first sampler is used to perform color sampling on a sampling area corresponding to a preset target based on a first sampling frequency, thereby obtaining multiple color sampling signals of the first sampling frequency, and the second sampler is used to perform color sampling on a sampling area corresponding to a preset target based on a second sampling frequency, thereby obtaining multiple color sampling signals of the second sampling frequency.

[0072] Exemplarily, the first sampler is used to perform light intensity differential sampling on the sampling area corresponding to the preset target based on a first sampling frequency, thereby obtaining multiple space-time differential sampling signals of the first sampling frequency, and the second sampler is used to perform light intensity differential sampling on the sampling area corresponding to the preset target based on a second sampling frequency, thereby obtaining multiple space-time differential sampling signals of the second sampling frequency.

[0073] It should be noted that the above description of the first sampling signal and the second sampling signal is merely an example, and the embodiments of the present disclosure do not limit this.

[0074] In some optional implementations, after obtaining the first sampling signal and the second sampling signal, it is not determined which sampling signal to output. Therefore, signal prediction, error comparison and signal output are performed respectively in steps S12, S13 and S14, so as to clearly determine whether the first sampling signal or the second sampling signal is added to the sampling output signal for external output.

[0075] In some optional implementations, in step S12, for the predicted moment, signal prediction can be performed based on the historical output signal, so as to obtain the predicted sampling signal at the predicted moment, wherein the historical output signal includes the sampling output signal before the predicted moment, which may be one or more sampling output signals before the predicted moment, and the one or more sampling output signals may include only the first sampling signal, may include only the second sampling signal, or may include both the first sampling signal and the second sampling signal, and the embodiments of the present disclosure are not limited to this.

[0076] In some optional implementations, the prediction moment is determined based on any one or more of preset prediction requirements, experience, statistical data, simulation results, and other information. The interval between two adjacent prediction moments can be the same or different, and the embodiments of the present disclosure do not limit this.

[0077] Exemplarily, a prediction moment may be determined at every preset time period, and signal prediction may be performed at the prediction moment.

[0078] For example, at time t1, the i-th signal prediction is performed (i.e., time t1 is the prediction time), at time t1+T, the i+1-th signal prediction is performed, ..., and at time t1+nT, the i+n-th signal prediction is performed, where T is a preset time period, i is an integer greater than or equal to 1, and n is also an integer greater than or equal to 1.

[0079] For example, signal prediction may be performed when it is detected that the reconstructed signal quality is poor.

[0080] For example, if the accuracy of the signal reconstruction result obtained at time t1 is lower than a preset accuracy threshold, signal prediction is performed at time t2, where time t2 is a time relatively close to time t1.

[0081] It should be noted that no matter at which prediction moment the signal prediction is performed, the historical output signals used include the sampled output signals output in the previous period adjacent to the prediction moment. These adjacent sampled output signals can more accurately reflect the signal sampling conditions in the historical time period, thereby obtaining more accurate signal prediction results.

[0082] For example, if the prediction time is t1, the historical output signal in the time period from t0 to t1 can be used for signal prediction, where t0 is less than t1. Furthermore, if there are multiple prediction times between t0 and t1 (i.e., the previous prediction time is later than t0), the corresponding historical output signal may contain both the first sampling signal and the second sampling signal. If there is no prediction time between t0 and t1 (i.e., the previous prediction time is earlier than t0), the corresponding historical output signal may contain only the first sampling signal or only the second sampling signal.

[0083] It should be noted that, when performing signal prediction, generally, if the selected historical output signal corresponds to a longer time period, the signal prediction result may be relatively accurate. If the selected historical output signal corresponds to a shorter time period, the signal prediction result may be less accurate. The length of the historical output signal time period selected for signal prediction can be determined based on experience, statistical data, simulation data, and prediction accuracy requirements, and is not limited in this regard in the present embodiment.

[0084] In some optional implementations, signal prediction is performed based on historical output signals to determine a predicted sampling signal at a prediction moment, including: inputting the historical output signal into a preset predictor to perform signal prediction to obtain a predicted sampling signal at the prediction moment.

[0085] In some optional implementations, the preset predictor may be a traditional predictor or a predictor based on a neural network, for example, a linear predictor, a predictor based on a generative neural network (GNN), etc.

[0086] After obtaining the predicted sampling signal, the predicted sampling signal at the predicted moment and the second sampling signal at the predicted moment can be compared in step S13 to determine an error comparison result between the two. The second sampling signal at the predicted moment refers to a signal obtained by sampling the preset target by the second sampler at the predicted moment.

[0087] It should be noted that, since the second sampler is a high-speed sampler with a high sampling frequency and a short sampling interval, it is considered that the second sampling moment of the second sampler can "cover" any predicted moment. In other words, it is considered that at each predicted moment, the second sampler performs the corresponding signal sampling process, and thus the second sampling signal at each predicted moment can be obtained. Alternatively, it can be said that for each predicted moment, there is a second sampling moment that is relatively close to the predicted moment. Since the time interval Δt between the predicted moment and the second sampling moment is short, the preset target is unlikely to change within this time interval Δt, or even if it changes, the change is negligible. Therefore, it is considered that the signal collected at the second sampling moment is the same as the signal collected at the predicted moment, and the second sampling signal collected at the second sampling moment is used as the second sampling signal at the predicted moment.

[0088] In some optional implementations, the error comparison result may be a result obtained by performing a difference operation between the predicted sampling signal at the prediction moment and the second sampling signal.

[0089] Exemplarily, the error comparison result includes a second signal difference between the predicted sampling signal at the predicted time and the second sampling signal at the predicted time. The second signal difference may be an absolute value or a signed value, which is not limited in the present embodiment.

[0090] In some optional implementations, based on the error comparison result, a sampling output signal is selected and output from the first sampling signal and the second sampling signal, including: when the error comparison result indicates that the difference between the second signal is less than or equal to a preset error threshold, the first sampling signal is selected and output as the sampling output signal; when the error comparison result indicates that the difference between the second signal is greater than the preset error threshold, the second sampling signal is selected and output as the sampling output signal.

[0091] It can be seen from this that as long as the error comparison result is low, the first sampling signal with a lower sampling frequency can be output as the sampling output signal; as long as the error comparison result is high, the second sampling signal with a higher sampling frequency can be output as the sampling output signal.

[0092] In some optional implementations, the first sampling signal is a color sampling signal, and the second sampling signal is a spatiotemporal differential sampling signal. Therefore, the two belong to different types of signals and may not be directly compared with the predicted sampling signal to obtain an error comparison result. Some signal conversion and other processing need to be performed first to compare signals of the same type.

[0093] Exemplarily, the first sampling signal is a color sampling signal, the second sampling signal is a spatiotemporal differential sampling signal, and the predicted sampling signal is a predicted color signal; accordingly, the error comparison result is determined based on the predicted sampling signal at the prediction moment and the second sampling signal at the prediction moment, including: reconstructing the color signal based on the spatiotemporal differential sampling signal to obtain a reconstructed color signal corresponding to the prediction moment; calculating the first signal difference between the reconstructed color signal and the predicted sampling signal to obtain an error comparison result; based on the error comparison result, selecting and outputting a sampling output signal from the first sampling signal and the second sampling signal, including: when the error comparison result indicates that the first signal difference is less than or equal to a preset error threshold, selecting and outputting the first sampling signal as the sampling output signal; when the error comparison result indicates that the first signal difference is greater than the preset error threshold, selecting and outputting the second sampling signal as the sampling output signal.

[0094] As can be seen from this, the predicted sampling signal is a color signal. Therefore, when determining the error comparison result, it is necessary to first reconstruct the color signal based on at least one spatiotemporal differential sampling signal to obtain a reconstructed color signal at the predicted time. This allows the predicted sampling signal to be compared with the sampling signal at the predicted time to obtain a first signal difference. Furthermore, when determining the sampled output signal, the first sampling signal is output when the first signal difference is small, and the second sampling signal is output when the first signal difference is large.

[0095] In some optional implementations, the preset error threshold can be a threshold determined based on sampling noise. This is because, considering that the sampling signal may produce errors due to the influence of noise during the sampling process, a preset error threshold can be set based on the sampling noise. The preset error threshold is equivalent to an error band (error bond). As long as the error value does not exceed the range of the error band, the sampling signal is considered to be accurate. Applied to the predicted sampling signal and the second sampling signal, that is, for the same prediction moment, as long as the signal difference between the predicted sampling signal and the second sampling signal is less than or equal to the preset error threshold, the predicted sampling signal is considered to be relatively accurate, indicating that a relatively accurate signal prediction can be achieved based on the historical output signal. Therefore, using the first sampling signal with a lower frequency can meet the signal reconstruction requirements. Conversely, if the signal difference between the predicted sampling signal and the second sampling signal is greater than the preset error threshold, the predicted sampling signal is considered to have low accuracy, indicating that a relatively accurate signal prediction cannot be achieved based on the historical output signal. Therefore, the second sampling signal with a higher frequency needs to be used as the sampling output signal to meet the signal reconstruction requirements.

[0096] The signal processing method of the embodiment of the present disclosure will be described below in conjunction with FIG4 .

[0097] FIG4 is a schematic diagram of a signal processing method provided by an embodiment of the present disclosure. As shown in FIG4 , there are three situations for the historical output signal: in the first situation, the historical output signal includes only the first sampling signal (for example, at least one first sampling signal), indicating that during the historical period corresponding to the historical output signal, the first sampling signal of the first sampler is used as the sampling output signal; in the second situation, the historical output signal includes only the second sampling signal (for example, at least one second sampling signal), indicating that during the historical period corresponding to the historical output signal, the second sampling signal of the second sampler is used as the sampling output signal; in the third situation, the historical output signal includes both the first sampling signal and the second sampling signal (for example, at least one first sampling signal and at least one second sampling signal), indicating that during the historical period corresponding to the historical output signal, there was a period of time when the first sampling signal of the first sampler was used as the sampling output signal, and a period of time when the second sampling signal of the second sampler was used as the sampling output signal.

[0098] Furthermore, for the predicted moment, signal prediction is performed through historical output signals to obtain a predicted sampling signal at the predicted moment, and the predicted sampling signal is compared with the second sampling signal at the predicted moment to obtain an error comparison result. The error comparison result includes a signal difference, which can be either a first signal difference or a second signal difference.

[0099] In the first case, if the signal difference is less than or equal to the preset error threshold, the first sampling signal continues to be output; if the signal difference is greater than the preset error threshold, the second sampling signal is output instead. This is because during the historical output signal stage, the first sampling signal is output by the first sampler. Therefore, after the prediction moment, if the error is small, the first sampling signal can still be output by the first sampler. If the error is large, the second sampling signal needs to be output by the second sampler as the sampling output signal.

[0100] In the second case, since the second sampling signal is used as the sampled output signal for a period of time before the predicted time, it is generally considered that the probability of the signal difference being greater than the preset error threshold is low, and therefore this case is not considered. If the error comparison result is that the signal difference is less than or equal to the preset error threshold, the first sampling signal is output instead.

[0101] For the third case, if the signal difference is less than or equal to the preset error threshold, the first sampling signal continues to be output, or the output of the second sampling signal is switched to the output of the first sampling signal, that is, the sampling output signal output for at least a period of time after the prediction moment is the first sampling signal; if the signal difference is greater than the preset error threshold, the output is switched to the second sampling signal, or the output of the first sampling signal is switched to the output of the second sampling signal, that is, the sampling output signal output for at least a period of time after the prediction moment is the second sampling signal. This is because in the historical output signal stage, the first sampling signal may be output by the first sampler, or the second sampling signal may be output by the second sampler. Therefore, after the prediction moment, if the error is small and the first sampling signal was output at the moment before the prediction moment, the first sampling signal can continue to be output; if the error is small and the second sampling signal was output at the moment before the prediction moment, an adjustment is required, from outputting the second sampling signal to outputting the first sampling signal. Similarly, after the prediction moment, if the error is large and the second sampling signal was output at the moment before the prediction moment, then the second sampling signal can continue to be output; if the error is large and the first sampling signal was output at the moment before the prediction moment, then it is necessary to adjust from outputting the first sampling signal to outputting the second sampling signal.

[0102] In some optional implementations, determining whether the signal difference is greater than a preset error threshold may be implemented by a discriminator.

[0103] In some optional implementations, the discriminator can be a traditional discriminator or a discriminator built based on a neural network. For example, the discriminator includes a conditional discriminator or a discriminator built based on a Visual Geometry Group (VGG) network.

[0104] It should be noted that the above preset predictors and discriminators are merely examples, and the embodiments of the present disclosure do not limit this.

[0105] Figure 5 is a schematic diagram of a signal processing method provided by an embodiment of the present disclosure. Referring to Figure 5 , a first sampler samples a preset target at a first sampling frequency f1 to obtain a first sampling signal at the first sampling frequency f1. A second sampler samples the preset target at a second sampling frequency f2 to obtain a second sampling signal at the second sampling frequency f2 (f2 > f1). After a period of data processing, a corresponding sampled output signal is obtained.

[0106] For any prediction moment, the historical output signal includes at least one sampled output signal before the prediction moment. The historical output signal is input into the predictor as input data. The predictor performs signal prediction based on the historical output signal to obtain a predicted sampling signal corresponding to the prediction moment. The predicted sampling signal reflects the result of predicting the sampling signal at the prediction moment through the historical sampling output signal.

[0107] The second sampling signal corresponding to the prediction moment and the predicted sampling signal at the prediction moment are input into the discriminator. The error comparison result is obtained through processing by the discriminator, and the sampling output signal can be selected from the first sampling signal and the second sampling signal according to the error comparison result and outputted externally.

[0108] Exemplarily, when the error comparison result represents that the signal difference is greater than a preset error threshold, the second sampling signal is added to the sampling output signal and outputted externally; when the judgment result represents that the signal difference is less than or equal to the preset error threshold, the first sampling signal is added to the sampling output signal and outputted externally.

[0109] For example, for the i-th prediction moment, a signal prediction is performed based on the historical output signal to obtain the i-th prediction sampling signal (i.e., the prediction sampling signal for the i-th prediction moment). The i-th prediction sampling signal is compared with the second sampling signal at the i-th prediction moment to obtain an error comparison result for the i-th prediction moment. If the error comparison result indicates a small error, the first sampling signal is output as the sampling output signal starting from the i-th prediction moment. If the error comparison result indicates a large error, the second sampling signal is output as the sampling output signal starting from the i-th prediction moment. The above process is repeated until the i+1-th prediction moment is reached, thereby determining the sampling output signal between the i+1-th prediction moment and the i+2-th prediction moment. Similarly, the sampling output signal output during the period from each prediction moment to the next prediction moment can be determined.

[0110] FIG6 is a schematic diagram of a signal processing method provided by an embodiment of the present disclosure. Referring to FIG6 , a first sampler samples a preset target at a first sampling frequency f1 to obtain a color sampling signal at the first sampling frequency f1. A second sampler samples the preset target at a second sampling frequency f2 to obtain a spatiotemporal differential sampling signal at the second sampling frequency f2 (f2>f1). After a period of processing, a historical output signal is obtained, which may include at least one first sampling signal and / or at least one second sampling signal.

[0111] For each prediction moment, the historical output signal is input into the predictor for signal prediction, resulting in a predicted sampling signal for the prediction moment. This predicted sampling signal is a color signal. Simultaneously, the color signal is reconstructed using at least one spatiotemporal differential sampling signal prior to the prediction moment, resulting in a reconstructed color signal corresponding to the prediction moment. The predicted sampling signal and the reconstructed color signal are input into the discriminator for a difference operation to obtain a signal difference, which is then compared with a preset error threshold to obtain an error comparison result. Based on the error comparison result, a sampling output signal can be selected from the first sampling signal and the second sampling signal and output externally, thereby obtaining a new sampling output signal. By analogy and repeating the above signal processing process, a more accurate sampling output signal with less redundancy for the prediction target can be obtained.

[0112] In some optional implementations, the first sampling signal includes a color sampling signal, and the second sampling signal includes a spatiotemporal differential sampling signal. Accordingly, in step S12, a spatiotemporal differential signal prediction can be performed based on the historical output signal at the predicted time to obtain a predicted sampling signal at the predicted time. This predicted sampling signal is a spatiotemporal differential signal. Because the second sampling signal is a spatiotemporal differential sampling signal, a signal difference can be obtained by directly comparing the predicted sampling signal with the second sampling signal at the predicted time. The sampled output signal to be output is then determined based on the signal difference and a preset error threshold.

[0113] It can be seen from this that when predicting the signal, the time-space differential sampling signal can be directly predicted to obtain a predicted sampling signal of the same type as the second sampling signal, and the predicted sampling signal can be directly compared with the corresponding second sampling signal.

[0114] FIG7 is a schematic diagram of a signal processing method provided by an embodiment of the present disclosure. Referring to FIG7 , a first sampler samples a preset target at a first sampling frequency f1 to obtain a color sampling signal at the first sampling frequency f1, and a second sampler samples the preset target at a second sampling frequency f2 to obtain a spatiotemporal differential sampling signal at the second sampling frequency f2 (f2>f1).

[0115] After a period of processing, a historical output signal is obtained, and the historical output signal may include at least one first sampling signal and / or at least one second sampling signal.

[0116] At each prediction moment, the historical output signal is input into the predictor for signal prediction, resulting in a predicted sampling signal for the prediction moment. This predicted sampling signal is a spatiotemporal difference signal. This predicted sampling signal and the spatiotemporal difference sampling signal at the prediction moment are input into the discriminator for a difference operation, resulting in a signal difference. This signal difference is then compared with a preset error threshold to obtain an error comparison result. Based on the error comparison result, a sampled output signal is selected from the first and second sampled signals and output, thereby obtaining a new sampled output signal. Repeating this signal processing process in this way can yield a more accurate sampled output signal for the predicted target with minimal redundancy.

[0117] It can be seen from this that in the above implementation process, the time-space difference signal can be directly predicted, so there is no need to reconstruct the color signal, and the time-space difference signal can be directly compared.

[0118] In some optional implementations, after step S14, the method may further include: performing signal reconstruction based on the sampled output signal to obtain a signal reconstruction result corresponding to a preset target. The signal reconstruction result corresponds to the signal type of the preset target. For example, if the preset target is a sound signal, the signal reconstruction result is also a sound signal; for another example, if the preset target is an image signal, the signal reconstruction result is also an image signal.

[0119] In some optional implementations, the sampled output signal can form a signal sequence, which may include a color sampling signal and may also include a time-space differential sampling signal. In this case, when performing signal reconstruction to obtain a reconstruction result, a color reconstruction result can be obtained, and the color reconstruction result is expressed as a colored image, or a light intensity reconstruction result can be obtained, and the color reconstruction result is expressed as a grayscale image. The embodiments of the present disclosure are not limited to this.

[0120] Exemplarily, for the signal sequence corresponding to the sampled output signal, color signal reconstruction can be performed based on the spatiotemporal differential sampling signal (i.e., the second sampling signal) therein to obtain a reconstructed color signal corresponding to the spatiotemporal differential sampling signal. Based on this, the types of all sampling signals in the entire signal sequence are color signals, and the reconstruction result obtained by reconstruction based on the signal reconstruction method is a color reconstruction result, which can be one or more colored images.

[0121] Exemplarily, for the signal sequence corresponding to the sampled output signal, a light intensity differential operation is performed based on the color sampling signal (i.e., the first sampling signal) therein to obtain a corresponding space-time differential signal. Based on this, all sampling signals in the entire signal sequence are of the space-time differential signal type, and the reconstruction result obtained by reconstruction based on the signal reconstruction method is a light intensity reconstruction result, which can be one or more grayscale images.

[0122] It should be noted that for the preset target, when the frequency of the preset target is low, the proportion of low-frequency sampling signals in the corresponding sampling output signal is high, based on which redundant data can be effectively reduced; when the frequency of the preset target is high, the proportion of high-frequency sampling signals in the corresponding sampling output signal is high, so that the data of the preset target can be more completely retained through the high-frequency sampling signal, ensuring the high quality of signal reconstruction.

[0123] In some optional implementations, it is assumed that the preset target is a one-dimensional signal g(t), and the main components of the signal g(t) are distributed in the frequency range of [-ω, +ω], and all its possible frequency components are distributed in the frequency range of [-Ω, +Ω], where ω>0, Ω>>ω.

[0124] Furthermore, the first sampling signal can be set to sω=g(t)×comb(t / ω), and the second sampling signal can be set to sΩ=g(t)×comb(t / Ω); wherein comb() represents a comb function, which can sample the signal g(t) at time t / ω and time t / ω.

[0125] The algorithm code corresponding to the above sampling process can be represented as follows:

[0126] initial state:k=0 / # Initialization state parameter k is 0# /

[0127] input:magnitude-time function g(t) / # represents the signal to be sampled as an amplitude-time function g(t)# /

[0128] buffer:s c [m]=none,m=0 / #Set the buffer in which the sequence s corresponding to the sampled output signal is stored c [m], m represents the serial number of the sampled output signal# /

[0129] while:sampling

[0130] sampling primitives c:s ω [k]=∫g(t)×δ(tk / ω)dt, push s ω[k]to buffer s c [m], m+=1 / # represents that g(t) is sampled by the first sampler at the first sampling frequency k / ω, the first sampling signal is added to the sampled output signal, and the sequence number m# / of the sampled output signal is updated.

[0131] while t∈[k / ω,k+1 / ω) / #within the sampling interval of the first sampler# /

[0132] sampling primitives C:s Ω [k] = ∫g(t) × δ(tk / Ω)dt / # represents the sampling of g(t) by the second sampler at the second sampling frequency k / Ω# /

[0133] the generative model Gθpredicate new observation:s' Ω [k]=Gθ(s c [m]) / #Gθ represents the prediction function. The prediction function Gθ is used for the second sampling moment, and the historical output signal s is obtained. c [m] performs signal prediction and obtains the predicted sampling signal s' Ω [k]# /

[0134] if||s' Ω [k]-s Ω [k]||1<error bond

[0135] No output / #If the difference between the predicted sampling signal and the second sampling signal is less than the preset error threshold, the sampling signal is not output externally# /

[0136] else

[0137] output s Ω [k], push s Ω [k]to buffer s c [m], m+=1 / # If the difference between the predicted sampling signal and the second sampling signal is greater than or equal to the preset error threshold, the second sampling signal s Ω [k] is added to the sampled output signal and the sequence number m# / of the sampled output signal is updated.

[0138] k+=1 / #Update state parameter k and enter the next round of sampling# /

[0139] End:output a nonuniform sampling sequences s c[m] / #After completing the signal acquisition, a signal sequence including multiple sampled output signals can be obtained# /

[0140] Here, / #…# / is a code explanation. It should be noted that the first sampling frequency 1 / ω and the second sampling frequency 1 / Ω may satisfy a certain multiple relationship. Therefore, the signal collected at any first sampling time is essentially both the first sampling signal and the second sampling signal. Since the sampled output signal is either the first sampling signal or the second sampling signal, the sampled signal at the first sampling time will be output as the sampled output signal in either case. Therefore, output determination is only required for the sampled signals at sampling times other than the first sampling time.

[0141] It should also be noted that in the above algorithm code example, the second sampling moment is used as the prediction moment for signal prediction. That is, for any second sampling moment, signal prediction is performed to obtain a predicted sampling signal, which is compared with the actual second sampling signal at the second sampling moment. If the signal difference between the two is less than or equal to the preset error threshold, it is considered that the predicted sampling signal is relatively accurate. Therefore, only the first sampling signal is output, which is manifested as not using the second sampling signal at the second sampling moment as the sampling output signal. On the contrary, if the signal difference between the two is greater than the preset error threshold, it is considered that the predicted sampling signal is not accurate enough. Therefore, it is necessary to output the second sampling signal at the second sampling moment as the sampling output signal. In other words, using the second sampling moment as the prediction moment is an optional implementation method of the embodiment of the present disclosure, which can also reduce the redundancy of the sampling output data while ensuring the quality of signal reconstruction.

[0142] If expanded to a 2D scene, taking image sampling as an example, assuming that the preset target is a preset image, in some optional implementation methods, the preset image can be color sampled at a first sampling frequency by a first sampler, and the obtained first sampling signal is added to the sampling output signal for output. At the same time, the preset image is color sampled at a second sampling frequency by a second sampler. Combined with signal prediction and error threshold judgment, the second sampling signal to be output is determined, and finally a signal sequence including multiple sampling output signals can be obtained, and the multiple sampling output signals in the signal sequence all belong to color sampling signals.

[0143] In some optional implementations, a first sampler may be used to perform color sampling on a preset image, resulting in a first sampling signal corresponding to a color sampling signal. A second sampler may be used to perform spatiotemporal differential sampling on the preset image, resulting in a second sampling signal representing a spatiotemporal differential sampling signal representing light intensity variation information. Specifically, a color sampling signal is output at a first sampling instant, and a spatiotemporal differential sampling signal is output at certain second sampling instants (e.g., the second sampling signal is added to the sampled output signal when a preset output condition is met). In the resulting sampled output signal sequence, when the frequency of the preset image changes slowly, all or most of the output sampling signals are color sampling signals. When the frequency of the preset image changes rapidly, the output sampling signals include a greater number of spatiotemporal differential sampling signals.

[0144] According to the statistical data of a large number of natural images, the natural image g(x,t) has 1 / f in the time and space domain. 2 The power spectrum of the image satisfies the following conditions: G(u,v)<O(1 / (u 2 ·v 2 )), where G(u,v) is the frequency domain representation of g(x,t), and O() represents the complexity function.

[0145] Furthermore, assuming that the band-limited interval of g(x,t) is {[-A,+A],[-B,+B]}, and its main frequency components are distributed in {[-a,+a],[-b,+b]}, through efficiency calculation, it can be inferred that its perception efficiency is improved by M×P times, where M=A / a and P=B / b. If extended to the nD scenario, the perception efficiency is exponentially improved (e.g., M n )promote.

[0146] FIG8 is a schematic diagram of a sampling signal and a sampling output signal provided by an embodiment of the present disclosure.

[0147] As shown in FIG8 , for the signal to be sampled g(t), when the first sampler is used to sample at a first sampling frequency, multiple first sampling signals as shown in FIG8(a) can be obtained (for example, four first sampling signals corresponding to the first sampling times t11, t12, t13, and t14). At the same time, based on the second sampler, g(t) is sampled at a second sampling frequency. In addition to sampling at the above-mentioned times t11, t12, t13, and t14, sampling is also performed between two adjacent first sampling signals, thereby obtaining multiple second sampling signals. For example, between t11 and t12, the second sampler performs a sampling at the second sampling times t21, t22, t23, and t24, respectively, to obtain four corresponding second sampling signals. Sampling is performed in a similar manner in other time periods. Therefore, there are four second sampling signals between two adjacent first sampling signals.

[0148] As mentioned above, each first sampling signal belongs to both the first sampling signal and the second sampling signal, so it can be directly added to the sampling output signal.

[0149] For each second sampling moment between two adjacent first sampling moments, a signal prediction is performed based on the historical output signal to obtain a predicted sampling signal for each second sampling moment. Furthermore, based on a preset error threshold, each second sampling signal is used as a reference value, and upper and lower limits are set according to the preset error threshold. This allows for the determination of an error band (error bond) for each second sampling signal. For example, if the value of the second sampling signal is x and the preset error threshold is thr, the error band's corresponding value range is (x-thr, x+thr).

[0150] For any of the above-mentioned second sampling moments, if its predicted sampling signal falls within its error band, it can be determined that the signal difference between the predicted sampling signal at the second sampling moment and the actual second sampling signal is less than the preset error threshold. Therefore, the sampling output signal outputted externally is determined to be the first sampling signal, and the second sampling signal is not added to the sampling output signal, that is, the sampling signal is not outputted externally at the second sampling moment; conversely, if the predicted sampling signal does not fall within the error band, it can be determined that the signal difference between the predicted sampling signal at the second sampling moment and the actual second sampling signal is greater than or equal to the preset error threshold. Therefore, the sampling output signal outputted externally is determined to be the second sampling signal, and the second sampling signal is added to the sampling output signal, that is, the second sampling signal is outputted externally at the second sampling moment.

[0151] Take t21, t22, t23, and t24 as examples for illustration. As shown in Figure 8(b), at predicted time t21, the predicted sampling signal falls within the error band of the second sampling signal. It can be determined that the error comparison result is small, so the first sampling signal is selected as the output sampling signal. Based on this, as shown in Figure 8(c), there is no output sampling signal at the second sampling time t21; similarly, the predicted sampling signals at the second sampling times t22, t23, and t24 fall within the error band of their respective second sampling signals. Therefore, there is no sampled output signal between t11 and t12.

[0152] Take t25, t26, t27, and t28 as examples for illustration. As shown in Figure 8(b), at the predicted time t25, the predicted sampling signal does not fall within the error band of the second sampling signal. It can be determined that the error comparison result is large, and therefore, the second sampling signal is selected as the output sampling signal. Based on this, as shown in Figure 8(c), the second sampling signal at the second sampling time t25 is output as the sampling output signal. Similarly, at the predicted times t26 and t28, the corresponding second sampling signals are also output as sampling output signals. However, at the predicted time t27, since the predicted sampling signal falls within the error band of the second sampling signal, no sampling signal is output at the predicted time t27.

[0153] In some optional implementations, multiple first samplers may be provided, and one of them may be selected as a low-speed sampler for signal sampling according to needs.

[0154] In some optional implementations, there are multiple first samplers, and different first samplers correspond to different first sampling frequencies. Accordingly, based on the error comparison result, a sampling output signal is selected from the first sampling signal and the second sampling signal and outputted, including: when the error comparison result indicates that the signal difference is less than or equal to a preset error threshold, determining the historical sampler corresponding to the historical output signal; when the historical sampler is the first sampler, selecting and outputting the first sampling signal of the historical sampler as the sampling output signal; or, when the historical sampler is the first sampler and the first sampling frequency corresponding to the historical sampler is not the minimum first sampling frequency, determining a first target sampler having a first sampling frequency less than that of the historical sampler, and determining the first sampling signal of the first target sampler as the sampling output signal for external output. The historical sampler is the sampler corresponding to the sampling output signal at one or more moments before the predicted moment.

[0155] Exemplarily, multiple first samplers are arranged in ascending order according to the first sampling frequency, and it is assumed that the historical sampler is the i-th first sampler, i>1; accordingly, if the error comparison result is that the signal difference is less than or equal to the preset error threshold, the first sampling signal of the i-th first sampler can be selected as the sampling output signal, or the i+1-th first sampler can be selected as the first target sampler, and the first sampling signal of the first target sampler can be used as the sampling output signal.

[0156] As can be seen, when the error comparison result determines that the error is small and the historical sampler is the first sampler, the sampling requirement can be met with a lower first sampling signal. Therefore, the first sampling signal of the historical sampler can continue to be used as the sampling output signal for external output after the prediction time. Alternatively, a first sampler with a lower first sampling frequency can be selected as the first target sampler, and the first sampling signal of the first target sampler can be output as the sampling output signal to further reduce the redundancy of the sampled output data.

[0157] It should be noted that in some optional implementations, when there are multiple first samplers, in the initial stage, it may not be clear which first sampler can meet the sampling requirements. Therefore, multiple first samplers can be used to work continuously for a period of time at the same time. When the continuous working time exceeds the preset working time threshold and a certain number of first sampling signals are accumulated, the first sampling signals of each first sampler can be used to perform signal prediction, and the first sampler that meets the preset error threshold can be screened out, and the first sampling signal of one of them can be selected as the subsequent sampling output signal. When the prediction moment is reached, the subsequent sampling output signal can be determined based on the corresponding signal processing method. Furthermore, if multiple first samplers that meet the preset error threshold are screened out in the initial stage, the first sampler with the lowest first sampling frequency can be screened out, and the first sampling signal of the first sampler can be used as the sampling output signal.

[0158] In some optional implementations, when there are multiple first samplers and different first samplers correspond to different first sampling frequencies, in the initial stage, a first sampling signal is randomly selected from the multiple first samplers as the sampling output signal, or the second sampling signal of the second sampler is used as the sampling output signal, and signal prediction is performed based on the sampling output signal to obtain a predicted sampling signal, and an error comparison result is determined by the predicted sampling signal and the corresponding second sampling signal, thereby determining a subsequent sampling output signal.

[0159] In some optional implementations, the first sampling frequency of the first sampler is an adjustable frequency, and the first sampling frequency can be adjusted in a timely manner according to demand to meet the sampling demand as much as possible.

[0160] In some optional implementations, based on the error comparison result, a sampling output signal is selected and output from the first sampling signal and the second sampling signal, including: when the error comparison result indicates that the signal difference is less than or equal to a preset error threshold, determining a historical sampler corresponding to the historical output signal; when the historical sampler is the first sampler, determining the first sampling signal of the first sampler as the sampling output signal and outputting it externally, or, reducing the first sampling frequency of the first sampler according to the first preset frequency adjustment interval, and determining the first sampling signal collected based on the reduced first sampling frequency as the sampling output signal and outputting it externally.

[0161] In some optional implementations, based on the error comparison result, a sampling output signal is selected and output from the first sampling signal and the second sampling signal, including: when the error comparison result indicates that the signal difference is greater than a preset error threshold, the first sampling frequency of the first sampler is increased according to the second preset frequency adjustment interval, and the first sampling signal collected based on the increased first sampling frequency is determined as the sampling output signal for external output.

[0162] Among them, the first preset frequency adjustment interval is used to represent the value of the first sampling frequency that is reduced once, and the second preset frequency adjustment interval is used to represent the value of the first sampling frequency that is increased once, and the first preset frequency adjustment interval and the second preset frequency adjustment interval can be the same or different, and the embodiment of the present disclosure does not limit this.

[0163] In summary, when the first sampling frequency of the first sampler is adjustable, if the error is small and the sampling output signal before the prediction moment is the first sampling signal, and the first sampling signal corresponds to the first sampling frequency f11, then it can be determined that the first sampling frequency f11 is a sampling frequency that can meet the sampling requirements. Subsequently, the first sampling signal with the first sampling frequency f11 can continue to be used as the sampling output signal, or the first sampling frequency can be appropriately reduced, and f11-△f1 is used as the new first sampling frequency, and the first sampling signal corresponding to the first sampling frequency is used as the sampling output signal, where △f1 is the first preset frequency adjustment interval. Conversely, if the error is large, it means that the sampling frequency of the previous sampling output signal is low. Therefore, f11+△f2 can be used as the new first sampling frequency, and the first sampling signal corresponding to the first sampling frequency can be used as the sampling output signal, where △f2 is the second preset frequency adjustment interval.

[0164] It should be noted that the sampling methods of the first sampler and the second sampler in the embodiment of the present disclosure can be applied to specific scenarios such as detection scenarios and posterior scenarios.

[0165] In some optional implementations, the method may also include: performing signal reconstruction based on multiple first sampling signals of the first sampler within the target time period to obtain a first reconstructed signal corresponding to a preset target; performing signal reconstruction based on multiple second sampling signals of the second sampler within the target time period to obtain a first reconstructed reference signal corresponding to the preset target; and determining an accuracy detection result of the first sampler based on first difference information between the first reconstructed signal and the first reconstructed reference signal.

[0166] It can be seen from this that the second sampler can be used to detect data errors in the first sampler. The data error here refers to the data error in the signal reconstruction based on the first sampling signal of the first sampler, not the error in the first sampling signal of the first sampler itself.

[0167] In some optional implementations, the second sampler performs signal sampling based on a discontinuous working state; accordingly, the method may also include: performing signal reconstruction based on multiple second sampling signals of the second sampler within the working period to obtain a second reconstructed reference signal corresponding to a preset target; performing signal reconstruction based on multiple first sampling signals of the first sampler within the working period to obtain a second reconstructed signal corresponding to the preset target; and determining the a posteriori detection result of the first sampler based on second difference information between the second reconstructed signal and the second reconstructed reference signal.

[0168] For example, when some high-speed samplers (for example, high-speed cameras, etc.) cannot work continuously, the above implementation method can be used to perform a posteriori detection on the first sampler, and the a posteriori detection results can be used to determine whether the first sampler can meet the sampling requirements.

[0169] FIG9 is a flow chart of a signal processing method provided by an embodiment of the present disclosure. Referring to FIG9 , the signal processing method may include the following steps.

[0170] Step S901 : performing color sampling on a preset target by a first sampler to obtain a plurality of first sampling signals of a first sampling frequency, wherein the first sampling signals include a plurality of color sampling signals corresponding to a plurality of pixels.

[0171] Step S902 : performing light intensity differential sampling on a preset target through a second sampler to obtain a plurality of second sampling signals of a second sampling frequency, wherein the second sampling signals include a plurality of spatiotemporal differential sampling signals corresponding to a plurality of pixels.

[0172] Step S903 : For the prediction moment, signal prediction is performed based on the historical output signal to determine the prediction sampling signal corresponding to the prediction moment.

[0173] Step S904 , based on a deep learning method, reconstructing the color signal according to the spatiotemporal differential signal at the prediction moment to obtain a reconstructed color signal corresponding to the prediction moment.

[0174] Step S905 , calculating the signal difference between the reconstructed color signal corresponding to the prediction moment and the predicted sampling signal.

[0175] Step S906: compare the signal difference with a preset error threshold.

[0176] Step S907 : When the signal difference is greater than the preset error threshold, the second sampling signal is selected as the sampling output signal to be outputted externally.

[0177] Step S908 : When the signal difference is less than or equal to the preset error threshold, the first sampling signal is selected as the sampling output signal to be outputted externally.

[0178] Step S909 , performing signal reconstruction according to the sampled output signal to obtain a reconstruction result corresponding to a preset target.

[0179] It should be noted that when reconstructing a signal for a preset target, the signal used is a sampled output signal. In this sampled output signal, if the frequency of the preset target itself changes slowly, the sampled output signal is typically a first sampled signal; if the frequency of the preset target itself changes rapidly, the sampled output signal is typically a second sampled signal. This ensures that the frequency of the sampled output signal closely matches the changing frequency of the preset target itself. This approach not only achieves a higher-quality reconstruction result, but also effectively reduces redundancy in the sampled output data.

[0180] It should be understood that the above embodiment can also be used in combination with any other embodiment of the present disclosure. The above embodiment is only a specific example of the present disclosure, and is not intended to limit the scope of protection of the present disclosure.

[0181] A second aspect of the embodiments of the present disclosure provides a signal processing device, which is a corresponding device for implementing the signal processing method provided by the above embodiments of the present disclosure. The device can be implemented in software and / or hardware and can generally be integrated into an electronic device.

[0182] FIG10 is a block diagram of a signal processing device provided by an embodiment of the present disclosure. Referring to FIG10 , the signal processing device 1000 may include the following modules.

[0183] The sampling module 1001 is used to sample a preset target through a first sampler and a second sampler respectively to obtain multiple first sampling signals of a first sampling frequency and multiple second sampling signals of a second sampling frequency, wherein the first sampling frequency of the first sampler is less than the second sampling frequency of the second sampler.

[0184] The prediction module 1002 is configured to perform signal prediction based on historical output signals at a prediction moment and determine a predicted sampled signal at the prediction moment, where the historical output signals include at least one sampled output signal before the prediction moment.

[0185] The determination module 1003 is configured to determine an error comparison result according to the predicted sampling signal at the prediction moment and the second sampling signal at the prediction moment.

[0186] The output module 1004 is configured to select and output a sampled output signal from the first sampled signal and the second sampled signal based on the error comparison result.

[0187] In the embodiment provided by the present disclosure, a sampling module is used to sample a preset target through a first sampler and a second sampler, respectively, to obtain multiple first sampling signals of a first sampling frequency and multiple second sampling signals of a second sampling frequency, wherein the first sampling frequency of the first sampler is less than the second sampling frequency of the second sampler; a prediction module is used to perform signal prediction based on historical output signals at a prediction moment, and determine a predicted sampling signal at the prediction moment, wherein the historical output signals include at least one sampling output signal before the prediction moment; a determination module is used to determine an error comparison result based on the predicted sampling signal at the prediction moment and the second sampling signal at the prediction moment; and an output module is used to select and output a sampling output signal from the first sampling signal and the second sampling signal based on the error comparison result.

[0188] It can be seen from this that, considering that the frequency of the preset target may be variable, if only a fixed sampling frequency is used to sample the preset target, it may not be possible to take into account both the sampling effect and the redundancy of the sampled output data. Therefore, the embodiment of the present disclosure uses at least part of the historical output signal to perform signal prediction, and compares the predicted sampling signal with the actual high-frequency sampling signal corresponding to the prediction moment to obtain an error comparison result. The error comparison result can be used to determine whether the predicted sampling signal is relatively accurate. When the error comparison result shows that the predicted sampling signal is relatively accurate, the first sampling signal with a lower sampling frequency can be outputted externally. When the error comparison result shows that the accuracy of the predicted sampling signal is low, the second sampling signal with a higher sampling frequency can be outputted externally. In this way, when the preset target is at a low frequency, the first sampling signal with a lower sampling frequency can be outputted, and when the preset target is at a high frequency, the second sampling signal with a higher sampling frequency can be outputted. Accordingly, when performing signal reconstruction, for low-frequency situations, relatively accurate signal reconstruction can be achieved using only the first sampling signal. Since the sampling frequency is low, the sampling noise is correspondingly low, and there is less redundant data. For high-frequency situations, the second sampling signal can be used for signal reconstruction. Since the sampling frequency of the second sampling signal is high, a reconstruction result with higher accuracy can be obtained. In summary, the embodiments of the present disclosure are suitable for signal acquisition in scenarios such as frequency instability. On the one hand, it can reduce sampling noise and data redundancy, and on the other hand, it can also ensure the quality of signal reconstruction.

[0189] A third aspect of the embodiments of the present disclosure provides a signal processor, which is a processor that implements the signal processing method provided by the above embodiments of the present disclosure. The processor can be implemented in software and / or hardware and can generally be integrated into an electronic device.

[0190] Figure 11 is a block diagram of a signal processor provided by an embodiment of the present disclosure. Referring to Figure 11 , the signal processor 1100 includes at least one first sampler 1101, a second sampler 1102, and a signal processing device 1103; wherein the signal processing device 1103 is used to perform any signal processing method of the embodiments of the present disclosure.

[0191] In some optional implementations, the signal processor may further include a predictor and a discriminator (not shown in the figure), wherein the predictor is used to perform signal prediction based on historical output signals to determine a predicted sampling signal at a prediction moment, and the discriminator is used to perform error comparison between the predicted sampling signal at the prediction moment and the second sampling signal at the prediction moment to select a sampling output signal from the first sampling signal and the second sampling signal.

[0192] In addition, the present disclosure also provides an electronic device, a computer-readable storage medium, and a computer program product.

[0193] FIG12 is a block diagram of an electronic device provided by an embodiment of the present disclosure.

[0194] 12 , an embodiment of the present disclosure provides an electronic device, comprising: at least one processor 1201; at least one memory 1202; and one or more I / O interfaces 1203 connected between the processor 1201 and the memory 1202; wherein the memory 1202 stores one or more computer programs that can be executed by the at least one processor 1201, and the one or more computer programs are executed by the at least one processor 1201 so that the at least one processor 1201 can execute the signal processing method of the embodiment of the present disclosure.

[0195] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor / processing core, implements the signal processing method of the present disclosure. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0196] An embodiment of the present disclosure also provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the signal processing method of the embodiment of the present disclosure.

[0197] It will be understood by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium).

[0198] As is well known to those skilled in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as is well known to those skilled in the art, communication media typically contains computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0199] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0200] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0201] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0202] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0203] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0204] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0205] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0206] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for purposes of limitation. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly indicated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A signal processing method, wherein: The method comprises: The preset target is sampled by a first sampler and a second sampler respectively to obtain a plurality of first sampling signals of a first sampling frequency and a plurality of second sampling signals of a second sampling frequency, wherein the first sampling frequency of the first sampler is less than the second sampling frequency of the second sampler; For a prediction moment, signal prediction is performed based on a historical output signal to determine a prediction sampling signal at the prediction moment, wherein the historical output signal includes at least one sampling output signal before the prediction moment; Determining an error comparison result according to the predicted sampling signal at the predicted time and the second sampling signal at the predicted time; Based on the error comparison result, a sampling output signal is selected from the first sampling signal and the second sampling signal and outputted.

2. The signal processing method according to claim 1, wherein: The first sampling signal includes a color sampling signal, and the second sampling signal includes a spatiotemporal differential sampling signal; Among them, the spatiotemporal differential sampling signal includes a time difference sub-signal and a space difference sub-signal. The time difference sub-signal is used to characterize the light intensity difference information between the sampling area corresponding to the preset target at two adjacent second sampling moments, and the space difference sub-signal is used to characterize the light intensity difference information between the sampling area and the adjacent sampling area at the same second sampling moment.

3. The signal processing method according to claim 2, wherein: The method of sampling the preset target by the first sampler and the second sampler to obtain a plurality of first sampling signals of the first sampling frequency and a plurality of second sampling signals of the second sampling frequency comprises: Performing color sampling on the preset target by the first sampler to obtain a plurality of first sampling signals of the first sampling frequency, wherein the sampling area corresponding to the preset target includes a plurality of pixels, and the first sampling signal includes a plurality of color sampling signals corresponding to the plurality of pixels; The light intensity differential sampling of the preset target is performed by the second sampler to obtain a plurality of second sampling signals of the second sampling frequency, wherein the second sampling signals include a plurality of spatiotemporal differential sampling signals corresponding to the plurality of pixels.

4. The signal processing method according to claim 3, wherein: The second sampling signal includes a spatial differential sub-signal and a temporal differential sub-signal, and the second sampler performs light intensity differential sampling on the preset target to obtain a plurality of second sampling signals of the second sampling frequency, including: The preset target is subjected to light intensity differential sampling by the second sampler to obtain an exciting light intensity signal and a plurality of inhibiting light intensity signals corresponding to each pixel at the current second sampling moment, wherein the exciting light intensity signal is a signal obtained by sampling the exciting photosensitive unit in the pixel, and the inhibiting light intensity signal is a signal obtained by sampling the inhibiting photosensitive unit in the pixel; Determine, according to the multiple suppressed light intensity signals, an average suppressed light intensity signal corresponding to each of the pixels at the current second sampling moment; Obtaining a spatial differential sub-signal corresponding to each of the pixels at the current second sampling moment according to a first difference value between the exciting light intensity signal of each of the pixels at the current second sampling moment and the corresponding average inhibiting light intensity signal; According to the second difference value between the excitation light intensity signal of each pixel at the current second sampling moment and the excitation light intensity signal of each pixel at the previous second sampling moment, the time difference sub-signal corresponding to each pixel at the current second sampling moment is obtained.

5. The signal processing method according to claim 4, wherein: Before performing light intensity differential sampling on the preset target by the second sampler to obtain an exciting light intensity signal and a plurality of inhibiting light intensity signals corresponding to each pixel at the current second sampling moment, the method further includes: Acquiring the connection status between the excitatory photosensitive unit and the plurality of inhibitory photosensitive units in each of the pixels; The preset target is subjected to light intensity differential sampling by the second sampler to obtain an exciting light intensity signal and a plurality of inhibiting light intensity signals corresponding to each pixel at the current second sampling moment; Based on the connection status corresponding to each of the pixels, the preset target is sampled for light intensity differential by the second sampler to obtain an excitation light intensity signal corresponding to each of the pixels at the current second sampling moment and at least one effective inhibition light intensity signal, wherein the inhibition light intensity signal is a signal obtained by sampling an inhibition photosensitive unit in the pixel whose connection status with the excitation photosensitive unit is an open connection.

6. The signal processing method according to claim 4, wherein: A plurality of the pixel arrays are arranged to form a pixel array, and at least one of the inhibition-type photosensitive units is shared between two adjacent pixels.

7. The signal processing method according to claim 2, wherein: The predicted sampling signal is a predicted color signal; The step of determining the error comparison result according to the predicted sampling signal at the predicted time and the second sampling signal at the predicted time comprises: Reconstructing a color signal according to the spatiotemporal differential sampling signal to obtain a reconstructed color signal corresponding to the predicted time; Calculating a first signal difference between the reconstructed color signal and the predicted sampling signal to obtain the error comparison result; The selecting and outputting a sampling output signal from the first sampling signal and the second sampling signal based on the error comparison result comprises: When the error comparison result indicates that the first signal difference is less than or equal to a preset error threshold, selecting and outputting the first sampling signal as a sampling output signal; When the error comparison result indicates that the first signal difference is greater than the preset error threshold, the second sampling signal is selected and output as a sampling output signal.

8. The signal processing method according to claim 1, wherein: The error comparison result includes a second signal difference between the predicted sampling signal at the predicted time and the second sampling signal at the predicted time; The selecting and outputting a sampling output signal from the first sampling signal and the second sampling signal based on the error comparison result comprises: When the error comparison result indicates that the second signal difference is less than or equal to a preset error threshold, selecting and outputting the first sampling signal as a sampling output signal; When the error comparison result indicates that the second signal difference is greater than the preset error threshold, the second sampling signal is selected and output as a sampling output signal.

9. The signal processing method according to claim 7 or 8, wherein: There are multiple first samplers, and different first samplers correspond to different first sampling frequencies respectively; The selecting and outputting a sampling output signal from the first sampling signal and the second sampling signal based on the error comparison result comprises: In the case where the error comparison result represents that the signal difference is less than or equal to a preset error threshold, determining a historical sampler corresponding to the historical output signal; When the historical sampler is the first sampler, the first sampling signal of the historical sampler is selected and output as the sampling output signal, or, when the historical sampler is the first sampler and the first sampling frequency corresponding to the historical sampler is not the minimum first sampling frequency, it is determined that the first sampling frequency is less than the first target sampler of the historical sampler, and the first sampling signal of the first target sampler is determined as the sampling output signal to be output externally.

10. The signal processing method according to claim 7 or 8, wherein: The first sampling frequency of the first sampler is an adjustable frequency; The selecting and outputting a sampling output signal from the first sampling signal and the second sampling signal based on the error comparison result comprises: In the case where the error comparison result represents that the signal difference is less than or equal to a preset error threshold, determining a historical sampler corresponding to the historical output signal; When the historical sampler is the first sampler, the first sampling signal of the first sampler is determined as a sampling output signal to be output externally, or the first sampling frequency of the first sampler is reduced according to a first preset frequency adjustment interval, and the first sampling signal collected based on the reduced first sampling frequency is determined as a sampling output signal to be output externally.

11. The signal processing method according to claim 10, wherein: The selecting and outputting a sampling output signal from the first sampling signal and the second sampling signal based on the error comparison result comprises: When the error comparison result indicates that the signal difference is greater than a preset error threshold, the first sampling frequency of the first sampler is increased according to a second preset frequency adjustment interval, and the first sampling signal collected based on the increased first sampling frequency is determined as a sampling output signal for external output.

12. The signal processing method according to claim 1, wherein: The method further comprises: Performing signal reconstruction based on a plurality of first sampling signals of the first sampler within a target time period to obtain a first reconstructed signal corresponding to the preset target; Performing signal reconstruction based on a plurality of second sampling signals of the second sampler within the target time period to obtain a first reconstructed reference signal corresponding to the preset target; Based on first difference information between the first reconstructed signal and the first reconstructed reference signal, an accuracy detection result of the first sampler is determined.

13. The signal processing method according to claim 1, wherein: The second sampler performs signal sampling based on a discontinuous working state; the method further includes: Performing signal reconstruction based on a plurality of second sampling signals of the second sampler in a working period to obtain a second reconstructed reference signal corresponding to the preset target; Performing signal reconstruction based on a plurality of first sampling signals of the first sampler within the working period to obtain a second reconstructed signal corresponding to the preset target; Based on second difference information between the second reconstructed signal and the second reconstructed reference signal, a posteriori detection result of the first sampler is determined.

14. The signal processing method according to any one of claims 1 to 7, wherein: The signal prediction is performed according to the historical output signal to determine the predicted sampling signal at the prediction moment, including: The historical output signal is input into a preset predictor for signal prediction to obtain a predicted sampling signal at the prediction moment, wherein the preset predictor includes a predictor constructed based on a neural network.

15. A signal processing device, wherein: The device comprises: A sampling module, configured to sample a preset target through a first sampler and a second sampler respectively, to obtain a plurality of first sampling signals of a first sampling frequency and a plurality of second sampling signals of a second sampling frequency, wherein a first sampling frequency of the first sampler is less than a second sampling frequency of the second sampler; A prediction module, configured to perform signal prediction based on historical output signals at a prediction moment, and determine a prediction sampling signal at the prediction moment, wherein the historical output signals include at least one sampling output signal before the prediction moment; A determination module, configured to determine an error comparison result according to the predicted sampling signal at the predicted time and the second sampling signal at the predicted time; An output module is used to select and output a sampling output signal from the first sampling signal and the second sampling signal based on the error comparison result.

16. A signal processor, wherein: The signal processor comprises at least a first sampler, a second sampler and a signal processing device; The signal processing device is used to execute the signal processing method as described in any one of claims 1-14.

17. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the signal processing method according to any one of claims 1 to 14.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the signal processing method according to any one of claims 1 to 14.

19. A computer program product comprising computer readable code, or a non-volatile computer readable storage medium carrying computer readable code, wherein: When the computer readable code is executed in a processor of an electronic device, the processor in the electronic device executes a signal processing method according to any one of claims 1 to 14.

Images