Pulse coding method, image acquisition method, device and medium

Abstract

The present application discloses a pulse coding method, an image acquisition method, a device and a medium. The pulse coding method includes: performing photoelectric conversion on the incident light to obtain the charge corresponding to each pixel in the image sensor; continuously accumulating the charge of each pixel to form an accumulated voltage; whenever the accumulated voltage reaches a preset voltage threshold, clearing the accumulated charge and generating a pulse signal; continuously accumulating the pulse signal and counting the number of accumulated pulse signals; at the pulse readout moment, according to the accumulated number N of the current counter, selecting the symbol representing the maximum integer M less than or equal to N from the preset pulse symbol set; outputting the pulse coding symbol corresponding to M and updating the state of the counter. The pulse coding method of the present application can improve the ability to express visual signals with a high dynamic range, improve the ability to simultaneously perform texture depiction and motion capture on highlight areas and dark areas in the scene, and ultimately improve the imaging effect of high dynamic range visual scenes.

Description

Technical Field

[0001] The present application relates to the field of signal processing technology, and in particular to a pulse encoding method for dynamic optical scene visual information, an image acquisition method, an image sensor, an image acquisition device, an electronic device and a computer-readable storage medium. Background Art

[0002] Traditional digital cameras usually shoot images at a fixed frame rate, and each frame of the image is generated in the following way: within a certain exposure time window, each pixel of the image sensor performs photoelectric conversion and charge accumulation on the incident light. After the exposure is completed, all the charges accumulated in each pixel are converted to digital to obtain the total amount of light for the pixel, and a fixed number of bits is used for quantization. In the related technology, a single pulse is used to record the dynamic change process of the incident light intensity at high speed. The intensity of the light is mainly reflected by the spacing of the pulse symbols. In scenes with a very large dynamic range, it is difficult to effectively take into account both bright and dark areas. Summary of the invention

[0003] The purpose of the present application is to provide a pulse coding method, an image acquisition method, an image sensor, an image acquisition device, an electronic device and a computer-readable storage medium. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0004] According to one aspect of an embodiment of the present application, a pulse coding method is provided, which is applied to an image sensor; the method comprises:

[0005] Performing photoelectric conversion on incident light to obtain charges corresponding to each pixel in the image sensor;

[0006] Continuously accumulating the charges of each pixel to form an accumulated voltage;

[0007] Whenever the accumulated voltage reaches a preset voltage threshold, the accumulated charge is cleared and a pulse signal is generated;

[0008] Accumulating the pulse signals and counting the number of the accumulated pulse signals;

[0009] At each pulse information reading moment driven by the clock signal, according to the accumulated number N of the current counter, a symbol representing the largest integer M less than or equal to N is selected from a preset pulse symbol set;

[0010] Output the pulse code symbol corresponding to M and update the status of the counter.

[0011] In some embodiments of the present application, the pulse signal is encoded in a multi-value pulse form, including:

[0012] Setting a variety of pulse symbols to represent different charge amounts;

[0013] The pulse signal is converted into a string of pulse code symbols including 0 and the plurality of pulse symbols.

[0014] In some embodiments of the present application, the setting is used to represent a plurality of pulse symbols with different charge amounts, including:

[0015] Setting a plurality of pulse symbols;

[0016] Using a preset positive integer set to set the charge amount represented by each pulse symbol;

[0017] The length of the unsigned binary representation of each positive integer in the preset positive integer set does not exceed 2 bits to the power of a preset positive integer, and each of the unsigned binary representations, except for the highest non-zero bit and a preset number of bits immediately following the highest non-zero bit, may contain non-zero values, while all other lower bits are 0.

[0018] In some embodiments of the present application, the pulse code corresponding to each positive integer in the preset positive integer set includes a non-zero flag code, a highest bit code and a refinement code.

[0019] In some embodiments of the present application, converting the pulse signal into a string of pulse coding symbols including 0 and the multiple pulse symbols includes:

[0020] determining the charge of each pixel according to the pulse signal;

[0021] Representing the charge of each pixel as a corresponding pulse symbol;

[0022] All the pulse symbols are spliced ​​in a pixel scanning order to obtain pulse coding symbols in a binary data stream form.

[0023] In some embodiments of the present application, the method further includes:

[0024] After the pulse code symbol is output, the counted quantity is updated.

[0025] According to another aspect of an embodiment of the present application, a pulse coding symbol-based image acquisition method is provided, comprising:

[0026] Decoding the pulse code symbol from the image sensor to obtain a pulse signal;

[0027] Acquiring the charge amount of each pixel of the image sensor according to the pulse signal;

[0028] Acquire the brightness of each pixel according to the charge amount;

[0029] The pixels with determined brightness are stitched together to obtain the corresponding image.

[0030] In some embodiments of the present application, obtaining the brightness of each pixel according to the charge amount includes:

[0031] The brightness of each pixel in the preset time window is calculated according to the charge amount of each pixel corresponding to the pulse signal in the preset time window.

[0032] According to another aspect of an embodiment of the present application, there is provided an image sensor, comprising a photoelectric converter, an integrator, a comparator, a pulse counter, and a coding output circuit connected in sequence;

[0033] The photoelectric converter is used to perform photoelectric conversion on the incident light to obtain charges corresponding to each pixel in the image sensor;

[0034] The integrator is used to continuously accumulate the charges of each pixel to form an accumulated voltage;

[0035] The comparator is used to clear the accumulated charge and generate a pulse signal whenever the accumulated voltage reaches a preset voltage threshold;

[0036] The pulse counter is used to accumulate the pulse signals and count the number of the accumulated pulse signals;

[0037] The coding output circuit is used for selecting one of the pulse signals to be coded in a multi-value pulse form and outputting a pulse coding symbol when the counted number reaches a preset number threshold;

[0038] The counter state updating circuit calculates the remaining pulse number according to the accumulated pulse number and the encoded pulse number, and updates the counter state according to the remaining pulse number.

[0039] According to another aspect of an embodiment of the present application, there is provided an image acquisition device based on a pulse coding symbol, comprising:

[0040] A decoding module, used for decoding the pulse coding symbol from the image sensor to obtain a pulse signal;

[0041] A charge acquisition module, used for acquiring the charge of each pixel of the image sensor according to the pulse signal;

[0042] A pixel brightness acquisition module, used for acquiring the brightness of each pixel according to the charge amount;

[0043] The splicing module is used to splice pixels of determined brightness to obtain a corresponding image.

[0044] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any one of the above methods.

[0045] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and the program is executed by a processor to implement any of the above methods.

[0046] The technical solution provided by one aspect of the embodiments of the present application may have the following beneficial effects:

[0047] The pulse coding method provided in the embodiment of the present application performs photoelectric conversion on the incident light to obtain charges corresponding to each pixel in the image sensor, accumulates the charges of each pixel to form an accumulated voltage, and when the accumulated voltage reaches a preset voltage threshold, clears the accumulated charges and generates a pulse signal, accumulates the pulse signals, and counts the number of accumulated pulse signals. When the clock-controlled pulse information is read, one of the pulse signals is selected to be encoded in the form of a multi-value pulse, and a pulse coding symbol is output. This can more effectively record the texture and motion information of the highlight and dark areas in the high dynamic range visual scene, improve the expression ability of the high dynamic range visual signal, improve the ability to simultaneously perform texture characterization and motion capture on the highlight and dark areas in the scene, and ultimately improve the imaging effect of the high dynamic range visual scene, thereby overcoming the technical defects of the related technology that the high-speed moving objects cannot be effectively imaged, resulting in blurred imaging of the high-speed moving objects.

[0048] Other features and advantages of the present application will be described in the subsequent description, and some of them will become obvious from the description, or some of them can be inferred or determined unambiguously from the description, or can be understood by implementing the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 A schematic diagram of the working mechanism of a pulse vision sensor in related technology is shown.

[0051] Figure 2 A schematic diagram showing a single-value pulse representation of a visual signal in the related art is shown.

[0052] Figure 3 A schematic structural diagram of an image sensor according to an embodiment of the present application is shown.

[0053] Figure 4 A flow chart of a pulse encoding method according to an embodiment of the present application is shown.

[0054] Figure 5 A schematic diagram of a multi-value pulse representation of a visual signal in one embodiment of the present application is shown.

[0055] Figure 6 A schematic diagram showing the highest non-zero bit and several bits below the MSB of a binary unsigned integer representation in a specific example of the present application is shown.

[0056] Figure 7 A schematic diagram showing the range of integers included in a specific example of the present application.

[0057] Figure 8 A schematic diagram of an electronic device according to an embodiment of the present application is shown.

[0058] Fig. 9 A schematic diagram of a computer-readable storage medium according to an embodiment of the present application is shown.

[0059] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0061] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0062] In recent years, the neuronal connection structure of the biological fovea and the integral-release model of ganglion cells have provided new insights into visual sampling. In related technologies, a pulse imaging model including photoreceptors, integrators, and threshold comparators has been proposed through simulation and abstraction of the fovea. The pulse imaging method represents visual information in the form of a binary pulse array, which can continuously record the change process of light intensity. There is no concept of exposure time window, which breaks through the limitations of traditional cameras and can capture and record high-speed motion.

[0063] The working mode of the camera based on the imaging model is: each pixel of the image sensor accumulates photoelectric charge independently and continuously; when the accumulated charge of a certain pixel reaches a certain threshold, the pixel is recorded to emit a pulse (recorded in the pulse state), the charge of the pixel is immediately cleared and the charge accumulation process is re-entered; the charge accumulation, pulse emission, charge clearing and re-entering the accumulation state of each pixel are independent of each other. The digital circuit of the camera reads and outputs the pulse state of each pixel at a very high clock frequency (for example, 40,000 Hz or higher). When a pixel generates a pulse in the past reading interval, it reads 1, otherwise it reads 0, and the pulse state is cleared after each reading. This imaging mode does not directly record an image, but records the arrival process of light information (photons) with extremely high time resolution, which has unique advantages in recording high-speed motion scenes. The image sensor uses the photoelectric conversion function of the photoelectric device to convert the light image on the photosensitive surface into an electrical signal that is proportional to the light image.

[0064] Different object points in an optical scene may have different light intensities, and the speed at which the image sensor pixel accumulates charge may be very different. In a high dynamic range visual scene, the speed at which charge accumulates on different pixels may differ by 100 to 1000 times, or even higher. In the imaging model, the difference in light intensity on each pixel can be reflected by the pulse frequency recorded by the image sensor. In the basic visual signal pulse representation method, each pulse represents the same amount of photons / charge. In order to avoid overflow of the representation range of the visual signal, the comparison threshold of the accumulated charge on the pixel is likely to be set to the maximum number of charges that a single pixel can accumulate in a pulse reading cycle. In this case, the pulse generated by the pixel with the strongest illumination has a pulse emission frequency of approximately one pulse per reading cycle; the pixel with an illumination intensity of 1 / n of the brightest pixel has a pulse emission frequency of approximately one pulse per n reading cycles. This easily leads to a long period of visual information recording in the dark area, and the information in the recording period may be severely aliased, resulting in insufficient effective recording of motion information and texture changes in the dark area.

[0065] In actual application scenarios, most areas containing meaningful texture information may belong to medium light conditions. In order to better depict visual texture information under medium light intensity, the camera may set the voltage threshold for controlling pulse emission to correspond to a certain medium light intensity. In this case, the light intensity information recorded in the high-brightness area reaches saturation, which is similar to the overexposure effect in traditional cameras.

[0066] In the pulsed visual signal representation model of the related art, the arrival of photons is characterized by a pulse sequence. The specific working mode of the related art is as follows: Figure 1 As shown, each pixel on the image sensor continuously converts incident photons into electric charges, and accumulates the charges in the integration component; the voltage of the accumulated charges is continuously compared with a preset reference voltage in the comparator; if at a certain moment, the accumulated voltage reaches the threshold voltage, the integrator is reset (cleared to release all charges) and a pulse emission is recorded (the pulse flag is set to 1); if the threshold voltage is not reached, charge accumulation will continue; the pulse flag (0 or 1) recorded on each pixel will be periodically read out and reset to 0 under the control of a high-speed clock signal.

[0067] like Figure 2 As shown in the figure, in this working mode, each pulse represents an equal amount of photons, and different light intensities are characterized by different pulse frequencies or spacings. The advantage of this pulse recording method is that it can depict the dynamic changes of the incident light signal in the visual scene in a fine-grained manner, and can effectively depict high-speed motion processes. However, the current limitation of this method is that it is difficult to effectively take into account the light intensity characterization of extremely bright areas and extremely dark areas.

[0068] In view of the limitations and defects in the related art, an embodiment of the present application provides a pulse coding method, which is applied to an image sensor to realize the coding representation of visual signals with a high dynamic range. The pulse coding method proposed in the embodiment of the present application introduces multiple pulse types to represent different photon accumulations or charge accumulations, forming a multi-value pulse form.

[0069] like Figure 3 As shown, the above-mentioned image sensor may include a photoelectric converter, an integrator, a comparator, a pulse counter and a coding output circuit connected in sequence.

[0070] like Figure 4 As shown, in some embodiments, the pulse encoding method includes:

[0071] S10, performing photoelectric conversion on the incident light to obtain charges corresponding to each pixel in the image sensor.

[0072] The photoelectric converter of the image sensor receives incident light, performs photoelectric conversion on the incident light, converts the incident light signal into charges corresponding to each pixel in the image sensor, and then inputs the charges into the integrator.

[0073] S20, continuously accumulating the charges of each pixel to form an accumulated voltage.

[0074] The integrator receives the charge from the photosensor and continuously accumulates the charge to form an accumulated voltage.

[0075] S30 , whenever the accumulated voltage reaches a preset voltage threshold, the accumulated charge is cleared and a pulse signal is generated.

[0076] The comparator compares the accumulated voltage with a preset voltage threshold. When the accumulated voltage reaches the preset voltage threshold, the comparator inputs a clear signal to the integrator to clear the charge accumulated by the integrator and generates a pulse signal, which is input to the pulse counter.

[0077] S40, accumulating pulse signals, and counting the number of accumulated pulse signals.

[0078] The pulse counter accumulates the received pulse signals and counts the number of accumulated pulse signals.

[0079] S50. At the moment of reading the pulse information driven by the clock, according to the accumulated number N of the current counter, select the symbol representing the largest integer M less than or equal to N from the preset pulse symbol set; output the pulse code symbol corresponding to M, and update the state of the counter.

[0080] At the moment of reading the pulse information driven by the clock, one of the pulse signals is selected to be encoded in the form of a multi-value pulse, and a pulse encoding symbol is output.

[0081] At the moment of reading the pulse information driven by the clock, the encoding output circuit selects to encode one of the pulse signals in the form of a multi-value pulse and outputs a pulse encoding symbol.

[0082] Specifically, the pulse signal is encoded in a multi-value pulse form, including: setting a plurality of pulse symbols for representing different charge amounts; and converting the pulse signal into a string of pulse coding symbols including 0 and a plurality of pulse symbols.

[0083] Setting a plurality of pulse symbols for representing different charge amounts includes: setting a plurality of pulse symbols; using a preset positive integer set to set the charge amount represented by each pulse symbol; wherein the length of the unsigned binary representation of each positive integer in the preset positive integer set does not exceed 2 bits to the power of a preset positive integer, and each unsigned binary representation, except for the highest non-zero bit and a preset number of bits immediately following the highest non-zero bit, may contain non-zero values, and all other lower bits are 0. The pulse code corresponding to each positive integer in the preset positive integer set includes a non-zero flag code, a highest bit code, and a refinement code.

[0084] The non-zero flag code is used to indicate whether the current pulse symbol is 0. If it is 0, the non-zero flag code is "0", and if it is any other pulse symbol, the non-zero flag code is "1". For symbol 0, the highest bit code and refinement code are no longer needed, but for non-zero pulse symbols, there are highest bit codes and refinement codes. The highest bit code describes the bit position of the MSB bit (represented by K binary bits), and the refinement code describes the m bits below the MSB bit. Each non-zero pulse symbol is encoded using a total of K+m+1 bits.

[0085] Specifically, converting a pulse signal into a string of pulse coding symbols including 0 and multiple pulse symbols may include: determining the charge of each pixel based on the pulse signal; representing the charge of each pixel as a corresponding pulse symbol; and splicing all the pulse symbols in a pixel scanning order to obtain pulse coding symbols in the form of a binary data stream.

[0086] In some implementations, the pulse encoding method of this embodiment may further include:

[0087] After the pulse code symbol is output, the counted number is updated, that is, the number of accumulated pulse signals is updated.

[0088] In the pulse imaging representation method of the related art, the illumination process recorded by an image sensor pixel is converted into a binary sequence, such as ( Figure 2 (shown)

[0089] 0,1,0,0,1,0,1,0,0,1,0,1,0,0,1,0,0,1,0,…

[0090] Where 0 represents that the pixel has not accumulated to reach the threshold, and 1 represents that the accumulation has reached the threshold. In the method proposed in the embodiment of the present application, the illumination process recorded by an image sensor pixel is converted into a multi-value sequence, for example (such as Figure 5 (shown)

[0091] A,0,0,A,C,D,B,B,0,0,0,A,C,B,B,A,…

[0092] The symbol 0 represents that the pixel has not accumulated to the threshold, that is, no pulse is generated; and the non-zero symbol (A, B, C, D, E, F, ..., etc.) represents that the accumulation has reached above the threshold, and may actually reach multiple times of the threshold. The specific number of times the threshold is represented by the pulse symbol (A, B, C, D, E, etc.).

[0093] The choice of symbols contained in the pulse symbol set Ω = {A, B, C, ...} has an important impact on the efficiency of visual information encoding. The simplest choice is to let A represent 1 times the threshold, B represent 2 times the threshold, C represent 3 times the threshold, D represent 4 times the threshold, E represent 5 times the threshold, F represent 6 times the threshold, and so on. In this case, if fixed-length encoding is used for the pulse symbol set Ω, then the K-bit symbol can only support 2 K The dynamic range of the threshold is times that of the threshold, indicating that the capability is relatively limited. For example, to support about 10 3 A threshold ratio of 10 times requires 10 bits of pulse symbol coding, while supporting about 10 6 A threshold ratio of times requires 20 bits of pulse symbol encoding, which is very costly in terms of code length.

[0094] In order to take into account both the range and efficiency of representation, the number of photocharges corresponding to the pulse symbols (A, B, C, D, E, etc.) can be selected in an exponential manner, for example, A is set to represent 1 times the threshold, B represents 2 times the threshold, C represents 4 times the threshold, D represents 8 times the threshold, E represents 16 times the threshold, F represents 32 times the threshold, and so on. Here, 2 is used as the step multiple, and other step multiples can also be used in actual use. In the pulse representation described above, the number of symbols contained in the pulse symbol set Ω = {A, B, C, ...} is set according to the dynamic range that needs to be supported, and can generally be set to 2^K, and encoded and represented by K bits. When the dynamic range is not particularly large, Ω can be set to contain 4 pulse symbols (using 2-bit encoding) or 8 pulse symbols (using 3-bit encoding), which can support 8 times the threshold and 128 times the threshold respectively. If the dynamic range is larger, it can be considered to set Ω to contain 16 pulse symbols (using 4-bit encoding to support 32768 times the threshold).

[0095] In the coding representation method of the visual signal of this embodiment, the pulse symbol is a multi-value symbol rather than a binary symbol, so the image sensor needs to accumulate the incident light charge that has not been coded and count the pulse emission, and the counting result determines which pulse symbol is transmitted each time the pulse is read. Due to the discontinuous design of the threshold multiple represented by the pulse symbol set Ω = {A, B, C, ...}, only part of the information in the counter is read out, and the remaining count quantity still needs to be retained in the counter, and the new pulse triggered by the comparator in the subsequent comparison process is combined with the count accumulation to determine the time point of the next pulse reading for reading.

[0096] The working process of the pulse image sensor based on the above idea is as follows: Figure 3 As shown. Photoelectric reception, charge accumulation, threshold comparison, charge clearing and resetting parts and Figure 1 are the same. The difference is that when the comparator determines that the voltage of the current accumulated charge exceeds the reference voltage threshold, the comparator will control the integrator to clear the charge immediately on the one hand, and trigger a pulse on the other hand, passing the signal to the pulse counter for pulse counting. The counter will record the number of all pulses triggered up to the current moment until the next pulse reading moment. The pulse reading circuit will select a pulse symbol for encoding output according to the state of the pulse counter (counting number) and update the state of the counter accordingly. The pulse symbol sent out by the encoding represents the amount of photocharge corresponding to several times the reference voltage. The update of the counter state is to subtract the amount of pulses sent out by the encoding, and retain the number of pulses that have not been sent out by the encoding. For the design of Ω={A、B、C、D、E、…}={1、2、4、8、16、…}, the update of the counter after each pulse reading is actually to reset the non-zero highest bit of the counter to zero, and all lower bits are retained. This can be achieved through bit operations.

[0097] In order to better balance the dynamic range and accuracy of the representation, the number of photocharges represented by each symbol in the pulse symbol set Ω = {A, B, C, D, E, ...} can be selected in the following manner. First, define the integer set:

[0098] Ω(K, m) = {positive integer n|n satisfies the following conditions (1) and (2)}

[0099] The condition (1) is That is, the available length of n does not exceed 2 K The condition (2) is: for the binary unsigned integer representation of n, only the most significant non-zero bit (MSB) and the m bits below the MSB are retained, and the remaining bits are cleared, and the resulting integer and n remain unchanged. The definition of the MSB bit and the m bits below the MSB bit is as follows Figure 6 As shown. The definition of Ω(K, m) is as follows Figure 7 shown.

[0100] In some embodiments, the maximum number of charges that can be accumulated by a single pixel in a single reading cycle can be determined according to the speed at which the sensor reads out the pulse symbol sequence, and then the required parameter K is calculated by the following method:

[0101]

[0102] For example, when the value of K is 3, pulses with intensities that differ by nearly 2^8-1=255 times can be supported. When the value of K is 3, pulses with intensities that differ by nearly 2^16-1=35535 times can be supported.

[0103] In some embodiments, the minimum allowable phase difference multiple α of the number of photocharges represented by adjacent pulse symbols in the pulse symbol set can be determined according to the requirements of signal recording accuracy, and then the required parameter m can be calculated by the following method:

[0104]

[0105] For example, when the photocharge multiples corresponding to adjacent pulse symbols need to be set to 2, 1.5, 1.25, and 1.125, the values ​​of the parameter m can be calculated to be 0, 1, 2, and 3, respectively.

[0106] For example, Ω(3,1) includes all integers that can be represented within 8 bits (1≤n≤255) and only have non-zero bits in the MSB and MSB-1 bits, that is, Ω(3,1)={1,2,3,4,6,8,12,16,24,32,48,64,96,128,192}. And Ω(3,2) includes all integers that can be represented within 8 bits (1≤n≤255) and only have non-zero bits in the MSB, MSB-1, and MSB-2 bits, that is, Ω(3,2)={1,2,3,4,5,6,7,8,10,12,14,16,20,24,28,32,40,48,56,64,80,96,112,128,160,192,224}.

[0107] It is agreed that each symbol (A, B, C, D, E, ..., etc.) in the pulse symbol set Ω corresponds to the threshold value of the corresponding digital multiple in Ω(K, m) in order. For example, when Ω(3, 1) is used, A, B, C, D, E, F represent 1 times, 2 times, 3 times, 4 times, 6 times, 8 times the threshold value, and so on; when Ω(3, 2) is used, A, B, C, D, E, F represent 1 times, 2 times, 3 times, 4 times, 5 times, 6 times the threshold value, and so on.

[0108] In the multi-value pulse representation of visual information proposed in the present application, the number of symbols contained in the pulse symbol set Ω={A, B, C, ...} (i.e., the size of the integer set Ω(K, m)) is set according to the dynamic range to be supported and the light intensity recording accuracy to be supported. When the dynamic range is not particularly large, K=2 or K=3 can be set. If the dynamic range is larger, K=4 or a larger value can be considered. When the light intensity recording accuracy requirement is low, m=0 or m=1 can be taken. When the requirement is high, m=2 or m=3 can be taken.

[0109] The amount of charge represented by each pulse symbol is set by a positive integer set Ω(K, m). The set Ω(K, m) contains all positive integers n that meet the following two conditions: (1) the length of the unsigned binary integer representation of n does not exceed 2^K bits; (2) the unsigned binary integer representation of n may contain 1 only in the highest non-zero bit (MSB bit) and the m bits below the MSB bit, and all other bits are 0. The parameters K and m can be selected according to actual needs.

[0110] The above multi-value pulse sequence is coded and represented. The binary symbol "0" is used to represent 0 in the multi-value pulse sequence, and the pulse symbols A, B, C, etc. in the sequence can be represented by binary "1" + pulse symbol fixed length code according to the definition of the symbol set Ω(K, m). The fixed length code of the pulse symbol describes the integer n corresponding to Ω(K, m), which consists of two parts. The first part describes the bit position of the MSB bit (requires K binary bits to represent), and the second part describes the m bits below the MSB bit. Therefore, each pulse symbol is coded and represented using a total of K+m+1 bits, as shown in Tables 1 to 3.

[0111] Table 1: Meaning and corresponding encoding of the pulse symbol set using Ω(3,0)

[0112]

[0113] Table 2: Meaning and corresponding encoding of the pulse symbol set using Ω(3,1)

[0114]

[0115]

[0116] Table 3: Meaning and corresponding encoding of the pulse symbol set using Ω(3,2)

[0117]

[0118]

[0119] Since the threshold multiples represented by the symbol set Ω are not continuous integers, the accumulated charge amount not represented by the current pulse symbol should be retained and incorporated into the accumulated amount of the next pulse reading. It should be pointed out that the scheme designed in this application is actually a double-layer accumulation scheme, and the part below the threshold voltage is accumulated by the integrator in the charge domain, which can ensure full-time continuous integration without losing photoelectric information. The part above the threshold voltage is represented by the counted pulses, which are accumulated and maintained in the digital domain by the pulse counter, where the high-bit part is output through the pulse coding symbol when the pulse is read, and the remaining pulses are retained in the pulse counter and counted together with the newly triggered pulses in the next read cycle, waiting to be read out. Such a double-layer accumulation mode strongly supports high dynamic range and low-cost coding representation.

[0120] The pulse coding method of the embodiment of the present application utilizes a bionic image sensor to continuously accumulate charge for each pixel, and immediately clears the charge and counts the pulses when a certain preset threshold is reached, and encodes and represents the visual dynamic change signal in the form of a multi-value pulse, and updates the state of the pulse counter according to the counter state and the output pulse code. This can more effectively record the texture and motion information of the highlight and dark areas in the high dynamic range visual scene, improve the expression ability of the high dynamic range visual signal, improve the ability to simultaneously perform texture characterization and motion capture on the highlight and dark areas in the scene, and ultimately improve the imaging effect of the high dynamic range visual scene, thereby overcoming the technical defects of the related technology that the high-speed moving objects cannot be effectively imaged, resulting in blurred imaging of the high-speed moving objects.

[0121] The pixels of the sensor independently perform continuous photoelectric conversion and charge accumulation. The accumulated charge is continuously compared with a preset threshold. When the accumulated charge reaches the threshold, a basic pulse is recorded, the accumulated charge is immediately released, and subsequent charge accumulation continues. The sensor sets a pulse counter N (integer) for each pixel to represent the number of basic pulses that have not been read out of the current pixel. Whenever the above basic pulse is generated, the value of counter N increases by 1. The sensor periodically queries the counter status, outputs a multi-value pulse symbol according to the status of the counter, and modifies the value in the counter. The sensor outputs the multi-value pulse symbol after binary encoding.

[0122] The sensor determines the binary code of the output pulse symbol in the following manner: first, it outputs a bit 0 or 1 according to whether the value of N is zero (output 0 when N=0, otherwise output 1); when N is not zero, it continues to output the position code of the highest non-zero bit (MSB bit) in the binary representation of N (using K bits), and then outputs the binary bits of m bits below the MSB bit in the binary representation of N.

[0123] The sensor modifies the pulse counter N in the following manner: whenever a pulse symbol is output, the highest non-zero bit (MSB bit) of the counter N and the m bits below the MSB bit are cleared.

[0124] The sensor outputs the binary information of the visual signal in the following way: at each pulse frame output time point, the multi-value pulse symbol generated by each pixel is encoded, and the pulse binary information of all pixels is spliced ​​in the pixel scanning order to form a binary data stream, which is output through the data interface after byte padding.

[0125] In the embodiment of the present application, a specific method for selecting the number of charges represented by the multi-value pulse is adopted, and a corresponding encoding method is set, and a method for counting pulses and updating a pulse counter in the image sensor is also set.

[0126] A specific example method of an embodiment of the present application includes the following steps:

[0127] Step 1: The pixels of the image sensor independently perform continuous photoelectric conversion and charge accumulation, and the accumulated charge is continuously compared with a preset threshold voltage. Whenever the accumulated charge reaches the threshold, a basic pulse is recorded and generated, the accumulated charge is immediately released, and the next round of charge accumulation, comparison, and pulse triggering continues.

[0128] Step 2: The pulse counter records the pulse state (integer N) of each pixel, which specifically represents the number of basic pulses that have not been read out of the current pixel. Whenever a basic pulse is generated in step 1, the value of counter N increases by 1. The image sensor periodically reads the state of the pulse counter and outputs the pulse symbol corresponding to the current counter state according to a specific encoding method. When the pulse information is read out, the pulse counter updates or resets its state under certain conditions.

[0129] Step 3. The output rule of the pulse symbol is as follows: if the value of N is zero, the pulse symbol 0 is output; if the value of N is not zero, the pulse symbol representing the largest integer not exceeding N in the symbol set Ω is output. For example, assuming that the symbol set Ω(3,1) is used, if N=10, F is output (F represents 8 times the threshold, i.e., 8 basic pulses), and if N=15, G is output (G represents 12 times the threshold, i.e., 12 basic pulses). Whenever a pulse symbol is output, the counter N is decremented, and the decremented value is the number of basic pulses represented by the output pulse symbol. For example, assuming that the symbol set Ω(3,1) is used, if N=10 is read, the pulse symbol F is output, and the counter N is reduced by 8, becoming N=2; if N=15 is read, the pulse symbol G is output, and the counter N is reduced by 12, becoming N=3.

[0130] Step 4: The image sensor circuit converts the pulse symbol sequence in step 3 into a binary data stream for output. The specific mapping mode of each symbol is as described above (as shown in Tables 1 to 3).

[0131] Step 5. The state update rule of the pulse counter in the image sensor is as follows: Assuming that the value of the counter before reading is N, when reading, the integer corresponding to the pulse symbol representing the largest integer not exceeding N in the symbol set Ω is M, then the counter state is updated to NM. The actual operation mode is to reset the highest non-zero bit of the counter and the following m second highest bits.

[0132] Step 6: The camera's signal processing module decodes and processes the pulse symbol data stream output by the image sensor to reconstruct the dynamic visual signal recorded by the image sensor.

[0133] The proposed pulse coding method was simulated, and it was verified that the multi-value pulse form coding method of the embodiment of the present application can more effectively characterize the texture and motion in the high dynamic range visual scene, and can better take into account the visual information recording of the highlight area and the dark area.

[0134] Another embodiment of the present application provides an image acquisition method based on pulse code symbols, including: decoding pulse code symbols from an image sensor to obtain a pulse signal; obtaining the charge amount of each pixel of the image sensor according to the pulse signal; obtaining the brightness of each pixel according to the charge amount; splicing each pixel with a determined brightness to obtain a corresponding image. Among them, obtaining the brightness of each pixel according to the charge amount may include: calculating the brightness of each pixel in a preset time window according to the charge amount of each pixel corresponding to the pulse signal in the preset time window.

[0135] Another embodiment of the present application provides an image sensor, comprising a photoelectric converter, an integrator, a comparator, a pulse counter, and a coding output circuit connected in sequence;

[0136] A photoelectric converter, used for performing photoelectric conversion on the incident light to obtain a charge corresponding to each pixel in the image sensor;

[0137] an integrator, used for continuously accumulating the charges of each pixel to form an accumulated voltage;

[0138] a comparator, configured to clear the accumulated charge and generate a pulse signal whenever the accumulated voltage reaches a preset voltage threshold;

[0139] A pulse counter, used for accumulating the pulse signals and counting the number of the accumulated pulse signals;

[0140] A coding output circuit, used for selecting one of the pulse signals to be coded in a multi-value pulse form and outputting a pulse coding symbol when the counted number reaches a preset number threshold;

[0141] The counter state updating circuit calculates the remaining pulse number according to the accumulated pulse number and the encoded pulse number, and updates the counter state according to the remaining pulse number.

[0142] Another embodiment of the present application provides an image acquisition device based on pulse coding symbols, comprising:

[0143] A decoding module, used for decoding the pulse coding symbol from the image sensor to obtain a pulse signal;

[0144] A charge acquisition module, used for acquiring the charge of each pixel of the image sensor according to the pulse signal;

[0145] A pixel brightness acquisition module, used for acquiring the brightness of each pixel according to the charge amount;

[0146] The splicing module is used to splice pixels of determined brightness to obtain a corresponding image.

[0147] like Figure 8 As shown, the electronic device 10 may include: a processor 100, a memory 101, a bus 102 and a communication interface 103, and the processor 100, the communication interface 103 and the memory 101 are connected via the bus 102; the memory 101 stores a computer program that can be run on the processor 100, and when the processor 100 runs the computer program, it executes the method provided in any of the aforementioned embodiments of the present application.

[0148] The memory 101 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0149] The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 101 is used to store a program, and the processor 100 executes the program after receiving an execution instruction. The method disclosed in any implementation of the aforementioned embodiment of the present application may be applied to the processor 100, or implemented by the processor 100.

[0150] The processor 100 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 100 or an instruction in the form of software. The above processor 100 may be a general-purpose processor, which may include a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101 and completes the steps of the above method in combination with its hardware.

[0151] The electronic device provided in the embodiment of the present application and the method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, operated or implemented by them.

[0152] The present application also provides a computer-readable storage medium corresponding to the method provided in the above embodiment. Fig. 9 As shown, the computer-readable storage medium shown is a CD 20 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the method provided by any of the aforementioned embodiments will be executed.

[0153] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0154] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0155] It should be noted that:

[0156] The term "module" is not intended to be limited to a specific physical form. Depending on the specific application, a module can be implemented as hardware, firmware, software, and / or a combination thereof. In addition, different modules can share common components or even be implemented by the same components. There may or may not be clear boundaries between different modules.

[0157] The algorithm and display provided herein are not inherently related to any specific computer, virtual device or other equipment. Various general devices can also be used together with examples based on this. According to the above description, it is obvious to construct the structure required for this type of device. In addition, the application is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the application described herein, and the description made to specific languages ​​above is for the purpose of disclosing the best mode of implementation of the application.

[0158] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0159] The above-mentioned embodiments only express the implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A pulse coding method, characterized in that: Applied to an image sensor; the method comprises: Performing photoelectric conversion on incident light to obtain charges corresponding to each pixel in the image sensor; Continuously accumulating the charges of each pixel to form an accumulated voltage; Whenever the accumulated voltage reaches a preset voltage threshold, the accumulated charge is cleared and a pulse signal is generated; Accumulating the pulse signals and counting the number of the accumulated pulse signals; At the pulse information reading moment, according to the accumulated number N of the current counter, a symbol representing the largest integer M less than or equal to N is selected from a preset pulse symbol set; Output the pulse code symbol corresponding to M and update the status of the counter; The pulse code symbol is obtained by encoding the pulse signal in a multi-value pulse form.

2. The method according to claim 1, characterized in that: The pulse signal is encoded in a multi-value pulse form, comprising: Setting a variety of pulse symbols to represent different charge amounts; The pulse signal is converted into a string of pulse code symbols including 0 and the plurality of pulse symbols.

3. The method according to claim 2, characterized in that The settings are used to represent a variety of pulse symbols with different charge amounts, including: Setting a plurality of pulse symbols; Using a preset positive integer set to set the charge amount represented by each pulse symbol; The length of the unsigned binary representation of each positive integer in the preset positive integer set does not exceed 2 bits to the power of a preset positive integer, and each of the unsigned binary representations, except for the highest non-zero bit and a preset number of bits immediately following the highest non-zero bit, are allowed to contain non-zero values, while all other lower bits are 0.

4. The method according to claim 3, characterized in that The pulse code corresponding to each positive integer in the preset positive integer set includes a non-zero flag code, a highest bit code and a refinement code.

5. The method according to claim 2, characterized in that: The step of converting the pulse signal into a string of pulse coding symbols including 0 and the plurality of pulse symbols comprises: determining the charge of each pixel according to the pulse signal; Representing the charge of each pixel as a corresponding pulse symbol; All the pulse symbols are spliced ​​in a pixel scanning order to obtain pulse coding symbols in a binary data stream form.

6. The method according to claim 1, characterized in that The method further comprises: After the pulse code symbol is output, the counted quantity is updated.

7. A pulse coding symbol based image acquisition method, characterized in that: The pulse code symbol is obtained by the method described in any one of claims 1-6; The image acquisition method comprises: Decoding the pulse code symbol from the image sensor to obtain a pulse signal; Acquiring the charge amount of each pixel of the image sensor according to the pulse signal; Acquire the brightness of each pixel according to the charge amount; Splice the pixels with determined brightness to obtain the corresponding image; The pulse code symbol is obtained by encoding the pulse signal in a multi-value pulse form.

8. The method according to claim 7, characterized in that The obtaining the brightness of each pixel according to the charge amount includes: The brightness of each pixel in the preset time window is calculated according to the charge amount of each pixel corresponding to the pulse signal in the preset time window.

9. An image sensor, characterized in that: It includes a photoelectric converter, an integrator, a comparator, a pulse counter and a coding output circuit connected in sequence; The photoelectric converter is used to perform photoelectric conversion on the incident light to obtain charges corresponding to each pixel in the image sensor; The integrator is used to continuously accumulate the charges of each pixel to form an accumulated voltage; The comparator is used to clear the accumulated charge and generate a pulse signal whenever the accumulated voltage reaches a preset voltage threshold; The pulse counter is used to accumulate the pulse signals and count the number of the accumulated pulse signals; The coding output circuit is used to select the applicable multi-value pulse coding corresponding to the number of pulses accumulated by the current pixel at the reading time controlled by the clock, and output the coding symbol of the pulse; The counter state updating circuit calculates the remaining number of pulses according to the accumulated number of pulses and the number corresponding to the encoded pulse symbols, and updates the counter state according to the remaining number of pulses; The coding symbol of the pulse is obtained by encoding the pulse signal in a multi-value pulse form.

10. An image acquisition device based on pulse coding symbols, characterized in that: The pulse code symbol is obtained by the method described in any one of claims 1 to 6; the image acquisition device comprises: A decoding module, used for decoding the pulse coding symbol from the image sensor to obtain a pulse signal; A charge acquisition module, used for acquiring the charge of each pixel of the image sensor according to the pulse signal; A pixel brightness acquisition module, used for acquiring the brightness of each pixel according to the charge amount; A splicing module, used for splicing pixels of determined brightness to obtain a corresponding image; The pulse code symbol is obtained by encoding the pulse signal in a multi-value pulse form.

11. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method according to any one of claims 1 to 8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 8.

Images