A pulse camera data encoding method for monitoring scenes

Abstract

The present invention discloses a pulse camera data encoding method for monitoring scenes, which belongs to the field of information encoding technology, and includes the following steps: based on a pre-trained CNN pulse array decoder, extracting a light intensity array with time and space domain noise removed from the original pulse array; using the light intensity array to generate pixel-level motion information, combining the light intensity array domain motion mask array, using a motion adaptive pulse encoder to generate a pulse array for background denoising; using the maximum pulse interval limit coefficient, using the first pulse interval aggregation encoder to aggregate and encode the denoised pulses; using the entropy encoder to aggregate and entropy encode the code stream elements generated to generate a binary file. The present invention effectively removes the influence of noise information on pulse camera data encoding, enhances the adaptability of the algorithm to different scene illumination, has good robustness, and improves the efficiency of pulse camera data encoding and decoding in monitoring scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of information coding, and in particular relates to a pulse camera data coding method for monitoring scenes. Background Art

[0002] Compared with traditional frame-based cameras and emerging dynamic vision sensors (DVS), the ultra-high temporal resolution of pulse cameras and the generation of pulses in both static background areas and moving areas give them significant advantages in recording high-speed moving targets, but they also bring huge data redundancy.

[0003] The characteristic of the monitoring scene is that the background and the image acquisition device are relatively static, which can produce a good video encoding effect. The pulse array data collected by the pulse camera contains spatial noise and temporal noise affected by the sensor manufacturing process, as well as flicker noise caused by the sensor's own asynchronous reset synchronous readout mechanism. Flicker noise will change with the intensity of external light. The stronger the light intensity, the more obvious the flicker noise. The existing pulse array coding algorithm does not specifically process the noise signal, resulting in poor pulse array compression efficiency, and the algorithm is greatly affected by the scene lighting, and the robustness of the coding algorithm is poor. Summary of the invention

[0004] The purpose of the present invention is to provide a pulse camera data encoding method for monitoring scenarios, so as to solve the problems that the pulse camera data encoding and decoding in existing monitoring scenarios does not specifically process noise signals, is greatly affected by scene lighting, and has poor robustness of the encoding algorithm.

[0005] In order to achieve the above object, the technical solution of the present invention is as follows:

[0006] The present invention relates to a pulse camera data encoding method for monitoring scenes, which comprises the following steps:

[0007] S1. Extract the light intensity information of the original pulse array through the CNN pulse array decoder to obtain the light intensity array;

[0008] S2. Extract motion information from the light intensity array and use the motion information adaptive pulse encoder to generate a pulse array after background denoising. The specific steps are:

[0009] S2.1. Extract pixel-level motion information from the light intensity array and obtain a motion mask matrix;

[0010] S2.2. The motion adaptive pulse encoder uses the light intensity array and the extracted corresponding motion information mask array to perform pulse encoding to generate a pulse array after background denoising;

[0011] S3. Use the first pulse interval aggregation encoder to encode the denoised pulse array, and then entropy encode the encoded data to obtain a binary code stream.

[0012] Preferably, the specific steps of S1 are: inputting an original pulse array with a time length of T, using a sliding window with a step size of 1 to input a number of continuous pulse array fragments into the CNN pulse decoder, and then converting the pulse array into a light intensity matrix, and removing the spatiotemporal noise of the original pulse array, and extracting the light intensity information in the original pulse array.

[0013] Preferably, the CNN pulse array decoder used in S1 adopts a U-Net network structure to extract the light intensity matrix.

[0014] The number of input channels of the U-Net network structure is 41, and each time a pulse array fragment of 41 consecutive time units is input. The U-Net network structure includes 2 downsampling layers, 2 bottom layers and 2 upsampling layers. The number of channels of the two downsampling layers is 64 and 128 respectively, the number of channels of the two bottom layers is 256, the number of channels of the two upsampling layers is 128 and 64 respectively, and the number of output channels of the U-Net network structure is 1.

[0015] Preferably, the specific operation process of the sliding window in S1 is: for an original pulse array with a length of T, a full-zero pulse array with a length of N is supplemented at both ends of the original pulse array to form a synthetic array, starting from time 0 as the starting point of the sliding window, and inputting several continuous pulse array fragments into the CNN pulse decoder.

[0016] Preferably, the motion mask matrix in S2.1 is expressed by equation (1)-equation (2):

[0017] (1),

[0018] (2),

[0019] in, L ( x , y , t n )for t n The light intensity matrix at each moment, L ( x , y , t n-1 )for t n The light intensity matrix at the previous moment, D ( x , y , tn ) represents the change in light intensity before and after the pixel, f is the motion judgment threshold, M ( x , y , t n )for t n The motion mask matrix at time N is t 0 , the corresponding motion matrix M ( x , y , t 0) is an all-zero matrix.

[0020] Preferably, the adaptive pulse encoder used in S2.2 is expressed by equation (3) to equation (5):

[0021] (3),

[0022] (4),

[0023] (5),

[0024] in, A ac ( x , y , t )for t time( x , y ) The accumulated value of the pulse integrator corresponding to the position is updated, A pre ( x , y , t )for t time( x , y ) The accumulated value of the pulse integrator at the previous moment corresponding to the position, L ( x , y , t )for t time( x , y ) position corresponding to the light intensity, c is the brightness value scaling factor, c is the gamma factor, i is the pulse integrator release threshold, K is the maximum pulse interval limit coefficient, i / KThe dark current input in the pulse integrator is used to limit the maximum pulse interval of the output pulse train to K , S dn ( x , y , t )for t time( x , y ) The pulse emission value corresponding to the pixel position, A ( x , y , t +1) pre for t +1 Moment( x , y ) position corresponding to the pulse accumulation value.

[0025] Preferably, the S3 uses a first pulse interval aggregation encoder to encode the denoised pulse array, outputs 4 types of coded code stream elements, namely, the inherent attributes of the pulse array, the first pulse interval, the aggregated pulse interval and the code stream terminator, and then performs entropy coding on the coded code stream elements to obtain a binary code stream;

[0026] The inherent properties of the pulse array include: a pulse array plane height property, a pulse array plane width property, a pulse array time domain length property, and a maximum pulse interval limit coefficient;

[0027] The output method of the first pulse interval, aggregated pulse interval and code stream terminator is as follows: according to the time domain priority principle, the one-dimensional pulse sequence in each pixel position in the denoised pulse array is extracted in turn, and the pulse interval is calculated. If the current pulse interval is not equal to the previous pulse interval, it is the first pulse interval, and this pulse interval is output. If it is equal to, the number of consecutive identical pulse intervals is increased by one until the first pulse interval is encountered again or the pulse sequence encoding ends, and the number of superimposed pulse intervals is output, indicating the encoding of the aggregated pulse interval, and the above steps are repeated to process each pixel position until the pulse array processing is completed, and the code stream terminator is output.

[0028] Preferably, the entropy coding adopts a combination of Huffman coding and adaptive context binary arithmetic coding to finally encode the code stream elements in the first pulse interval aggregation encoder into a binary file.

[0029] Preferably, the specific steps of the entropy coding include:

[0030] S3.1. Use the binary arithmetic decoder and Huffman decoder with adaptive context to decode the binary code stream file;

[0031] S3.2. Read the fixed byte length of the pulse array inherent properties from the entropy decoding file to obtain the pulse array inherent properties of the pulse array to be decoded;

[0032] S3.3. Read a single pulse interval and determine whether it is greater than the maximum pulse interval limit coefficient and the end symbol of the code stream. If it is less than the coefficient, it is represented as the first pulse interval, and it is directly decoded into the corresponding pulse sequence. If it is greater than the maximum pulse interval limit coefficient, it is represented as an aggregated pulse interval, and the maximum pulse interval limit coefficient is subtracted from it to obtain the number of continuous pulse intervals, and the previous pulse interval is obtained, and the decoding is performed according to the previous pulse interval of the number of continuous pulse intervals;

[0033] S3.4. Return to S3.2 until the end symbol of the code stream is detected, the decoding of the entire pulse sequence is completed, and the decoded one-dimensional pulse sequence is divided according to the inherent properties of the pulse array to synthesize the pulse array format.

[0034] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:

[0035] The present invention relates to a pulse camera data encoding method for monitoring scenes. The method is based on a pre-trained CNN pulse array decoder, extracts light intensity information from an original pulse array, generates pixel-level motion information from a light intensity array, uses a motion adaptive pulse encoder to pulse-code the light intensity array and the extracted motion information mask array, removes noise information contained in background pixels while retaining high-speed motion details, combines a maximum pulse interval limit coefficient in the motion adaptive pulse encoder, uses a first pulse interval aggregation encoder to aggregate and encode denoised pulses, and finally uses an entropy encoder to encode code stream elements into a binary file. Meanwhile, a corresponding efficient decoding algorithm is proposed, which effectively removes the influence of noise information on pulse encoding, enhances the adaptability of the algorithm to different scene illuminations, has good robustness, and improves the pulse camera data encoding and decoding efficiency in monitoring scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a pulse camera data encoding method for monitoring scenarios provided by the present invention.

[0037] Figure 2 A schematic diagram of the structure of a CNN pulse array decoder provided by the present invention.

[0038] Figure 3 This is a schematic diagram of the first pulse interval aggregation coding method provided by the present invention. DETAILED DESCRIPTION

[0039] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with examples. The following examples are used to illustrate the present invention but are not used to limit the scope of the present invention.

[0040] Reference Figure 1 As shown, the pulse camera data encoding method for monitoring scenes involved in the present invention comprises the following steps:

[0041] S1. Extract the light intensity information of the original pulse array through the CNN pulse array decoder to obtain the light intensity array, specifically: input the original pulse array with a time length of T, use a sliding window with a step size of 1 to input continuous pulse array fragments to the pre-trained CNN pulse decoder, and then convert the pulse array into a light intensity matrix, that is, convert the 1-bit encoded pulse array into an 8-bit encoded light intensity matrix, and remove the spatiotemporal noise of the original pulse array to extract the light intensity information in the original pulse array;

[0042] In this embodiment, the CNN pulse decoder network adopts Figure 2 The U-Net network structure shown in the figure can realize the extraction of light intensity information of the original pulse array. The original pulse array is essentially a three-dimensional binary symbol array, which can be expressed as S ( x , y , t ),in x , y represent the pixel positions in the sensor, t Represents time information, S ( x , y , t ) can only take the value of 0 or 1, representing the emission status of the pulse integrator at the corresponding position at the current moment. Since the pulse array does not directly represent the light intensity, it can be regarded as a 1-bit encoding of the light intensity. Extracting 8 bits from the pulse sequence to express the light intensity information can be compared to the decoding process of the pulse array, and in this process the temporal and spatial noise of the original pulse sequence is removed.

[0043] The U-Net network structure, the number of input channels C in =41, that is, each time a pulse array segment of 41 consecutive time units is input S ( x , y , t ) 41The U-Net network structure consists of 2 downsampling layers (downsampling layer 1 and downsampling layer 2), 2 bottom layers and 2 upsampling layers (upsampling layer 1 and upsampling layer 2). The number of channels of the two downsampling layers is 64 and 128 respectively, the number of channels of the two bottom layers is 256, the number of channels of the two upsampling layers is 128 and 64 respectively, and the number of output channels is C out is 1, that is, the extracted light intensity matrix L ( x , y ). In particular, the feature fusion in the network adopts a skip connection strategy, concatenating the feature map of the corresponding downsampling layer in the encoder with the feature map of the current upsampling layer in the channel dimension, and adopts the ReLU activation function - nonlinearity is introduced by setting all negative values ​​to zero. Each convolutional layer is followed by an InstanceNorm2d layer for instance normalization. The Tanh activation function maps the output value to the interval [-1,1].

[0044] The loss function of the CNN pulse decoder network satisfies the following formula:

[0045] (6)

[0046] (7),

[0047] (8),

[0048] in, L L2 The real light intensity y and the decoded light intensity The mean square error between is used to measure the difference at the pixel level. N is the number of pixels, i Indicates the pixel at the corresponding position of the real light intensity and the decoded light intensity, L VGG The pre-trained VGG19 network is used to calculate the difference in feature maps. M is the number of feature maps, F j ( y ) is the real light intensity matrix obtained by the VGG19 network at a certain layer j feature map, is the feature map corresponding to the decoded light intensity matrix, l L2 and l VGG is a hyperparameter.

[0049] If the input raw pulse array S (x , y , t ) T Length is T , the array is synthesized by completing the full zero pulse array with a length of 20 in both ends of the array S ( x , y , t ) T+40 , starting from time 0 as the starting point of the sliding window, using the CNN pulse array decoder to generate a light intensity array of the same length T L ( x , y , t ) T .

[0050] S2. Although the CNN pulse decoder can remove most of the temporal and spatial noise, it is still necessary to further remove the remaining noise information through the motion information of the pixels. This step extracts the motion information from the light intensity array and uses the motion information adaptive pulse encoder to generate the pulse array after background denoising;

[0051] The specific steps are:

[0052] S2.1. Extract pixel-level motion information from the light intensity array and obtain the motion mask matrix. M ( x , y , t ) T The calculation satisfies the following formula:

[0053] (1),

[0054] (2),

[0055] in, L ( x , y , t n )for t n The light intensity matrix at each moment, L ( x , y , t n-1 )for t n The light intensity matrix at the previous moment, D ( x , y , t n ) represents the change in light intensity before and after the pixel, fis the motion judgment threshold, M ( x , y , t n )for t n The motion mask matrix at time N is t 0 , the corresponding motion matrix M ( x , y , t 0) is an all-zero matrix.

[0056] S2.2. The motion adaptive pulse encoder uses the light intensity array and the extracted corresponding motion information mask array for pulse encoding to generate a pulse array after background denoising: After generating the light intensity matrix and the motion mask matrix, they are input into the motion adaptive pulse encoder to generate a pulse array after background denoising. The motion adaptive pulse encoder satisfies the following formula:

[0057] (3),

[0058] (4),

[0059] (5),

[0060] in, A ac ( x , y , t )for t time( x , y ) The accumulated value of the pulse integrator corresponding to the position is updated, A pre ( x , y , t )for t time( x , y ) The accumulated value of the pulse integrator at the previous moment corresponding to the position, L ( x , y , t )for t time( x , y ) position corresponding to the light intensity, c is the brightness value scaling factor, c is the gamma factor, i is the pulse integrator release threshold, K is the maximum pulse interval limit coefficient, i / K It can be regarded as the dark current input in the pulse integrator, which can limit the maximum pulse interval of the output pulse train to K , in order to solve the problem of too large pulse intervals in some scenes with poor lighting or sensor failure. S dn ( x , y , t )for t time( x , y ) The pulse emission value corresponding to the pixel position. When the pulse integrator accumulated value after update is greater than the pulse emission threshold, the value 1 is used to indicate pulse emission, and 0 is used to indicate no pulse emission. A ( x , y , t +1) pre for t +1 is the pulse cumulative value corresponding to the position at time, that is, the pulse integrator cumulative value after the pulse is issued at time. If the pulse is not issued, the cumulative value is directly assigned to the cumulative value after the time is updated. If the pulse is issued, different reset strategies are adopted for the integrator according to the motion information of the pixel, that is, the static background pixels are directly reset to 0 potential to better remove noise. For moving pixels, the cumulative value of the integrator is reset to the resting potential to retain the motion details of high-speed moving objects.

[0061] S3. Use the first pulse interval aggregation encoder to encode the denoised pulse array, and then perform entropy coding on the encoded data to finally obtain a binary code stream.

[0062] In this embodiment, after obtaining the pulse array with background denoising, S dn ( x , y , t ), since the maximum pulse interval is limited to K , the pulse array is efficiently encoded using the first pulse interval aggregation encoder. The first pulse interval aggregation encoder first outputs the inherent attribute elements of the pulse array, including the pulse array plane height attribute H , Pulse array plane width properties W , pulse array time domain length attribute T , Maximum pulse interval limit coefficient K , each element is expressed using a fixed number of bytes.

[0063] The pulse interval contains both the emission status information of the pulse integrator and the time information of the pulse emission. Since the light intensity in the background area is constant, the corresponding pulse interval is also equal. This characteristic can be used to aggregate the pulse interval. According to the time domain priority principle, the denoised pulse array is extracted in sequence. S dn ( x , y , t ) in each pixel position N ( x ,y)={ n 1, n 2, n 3… n T} T , calculate the pulse interval, if the current pulse interval ISI now Interval with the previous pulse ISI pre If the first pulse interval is not equal to ISI now and ISI pre Equal, for the same number of consecutive pulse intervals M Add one until the first pulse interval is encountered again or the pulse sequence encoding ends, and output the number of pulse intervals after superposition K + M , which means encoding the continuous pulse intervals, continuously processing each pixel position until the pulse array is processed and the code stream end mark is output.

[0064] To limit the number of consecutive pulse intervals M The count size is K + M The size of the codeword is when the number of consecutive pulse intervals is greater than a certain set value. r , the output can be K + M elements, and then M Clear to zero, and continue to count the consecutive identical pulse intervals. The code stream elements output by the first pulse interval aggregation encoder are entropy encoded using a combination of Huffman coding and adaptive context binary arithmetic coding, and finally encoded into a binary file. The specific steps are as follows: Figure 3 As shown, including:

[0065] S3.1. Use the binary arithmetic decoder and Huffman decoder with adaptive context to decode the binary code stream file;

[0066] S3.2. Read the fixed byte length pulse array intrinsic attribute from the entropy decoding file to obtain the pulse array plane height attribute of the pulse array to be decodedH , Pulse array plane width properties W , pulse array time domain length attribute T , Maximum pulse interval limit coefficient K ;

[0067] S3.3. Reading a single pulse interval ISI n , determine whether it is greater than the maximum pulse interval limit coefficient K and the end symbol of the code stream, if it is less than the coefficient K , which is represented by the first pulse interval, is directly decoded into the corresponding pulse sequence. If it is greater than the coefficient K , which is the aggregate pulse interval, is subtracted K Get the number of consecutive pulse intervals M , and get the last pulse interval ISI n-1 , according to M indivual ISI n-1 Pulse intervals are decoded;

[0068] S3.4. Return to S3.2 until the end symbol of the code stream is detected, the decoding of the entire pulse sequence is completed, and the decoded one-dimensional pulse sequence is divided according to the inherent properties of the pulse array to synthesize the pulse array format.

[0069] Compared with the existing pulse data encoding and decoding algorithms, the present invention independently processes the noise in the pulse data, uses a relatively simple CNN network architecture and extracts motion information in the pulse data, and designs a motion adaptive pulse encoder that combines light intensity information and motion information. It can not only effectively remove the noise information in the background pixels, but also well retain the details of high-speed moving objects, and effectively remove the influence of noise information on pulse coding. From the perspective of pulse coding, the present invention utilizes the maximum pulse interval limit and the invariance of the pulse interval in the background pixels to propose a simple and effective first pulse interval encoder, which enhances the adaptability of the algorithm to different scene illuminations, has good robustness, and improves the pulse camera data encoding and decoding efficiency in monitoring scenarios.

[0070] The present invention is described in detail above in conjunction with the embodiments, but the contents described are only preferred embodiments of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made according to the scope of application of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A pulse camera data encoding method for monitoring scenes, characterized in that: It contains the following steps: S1. Extract the light intensity information of the original pulse array through the CNN pulse array decoder to obtain the light intensity array; S2. Extract motion information from the light intensity array and use the motion information adaptive pulse encoder to generate a pulse array after background denoising. The specific steps are: S2.

1. Extract pixel-level motion information from the light intensity array to obtain a motion mask matrix, which is represented by equations (1) and (2): （1）， （2）， in, L ( x , y , t n )for t n The light intensity matrix at each moment, L ( x , y , t n-1 )for t n The light intensity matrix at the previous moment, D ( x , y , t n ) represents the change in light intensity before and after the pixel, φ is the motion judgment threshold, M ( x , y , t n )for t n The motion mask matrix at time N is t 0 , the corresponding motion matrix M ( x , y , t 0) is an all-zero matrix; S2.

2. The motion adaptive pulse encoder uses the light intensity array and the extracted corresponding motion information mask array to perform pulse encoding to generate a pulse array after background denoising. The adaptive pulse encoder used is expressed by equations (3) to (5): （3）， （4）， （5）， in, A ac ( x , y , t )for t time( x , y ) The accumulated value of the pulse integrator corresponding to the position is updated, A pre ( x , y , t )for t time( x , y ) The accumulated value of the pulse integrator at the previous moment corresponding to the position, L ( x , y , t )for t time( x , y ) position corresponding to the light intensity, c is the brightness value scaling factor, γ is the gamma factor, θ is the pulse integrator release threshold, K is the maximum pulse interval limit coefficient, θ / K The dark current input in the pulse integrator is used to limit the maximum pulse interval of the output pulse train to K , S dn ( x , y , t )for t time( x , y ) The pulse emission value corresponding to the pixel position, A ( x , y , t +1) pre for t +1 Moment( x , y ) The pulse accumulation value corresponding to the position; S3. Encode the denoised pulse array using the first pulse interval aggregation encoder, output 4 types of coded code stream elements, namely, the inherent attributes of the pulse array, the first pulse interval, the aggregated pulse interval and the code stream end mark, and then perform entropy coding on the coded code stream elements to obtain a binary code stream, and then perform entropy coding on the coded data to obtain a binary code stream; The inherent properties of the pulse array include: a pulse array plane height property, a pulse array plane width property, a pulse array time domain length property, and a maximum pulse interval limit coefficient; The output method of the first pulse interval, aggregated pulse interval and code stream terminator is as follows: according to the time domain priority principle, the one-dimensional pulse sequence in each pixel position in the denoised pulse array is extracted in turn, and the pulse interval is calculated. If the current pulse interval is not equal to the previous pulse interval, it is the first pulse interval, and this pulse interval is output. If it is equal to, the number of consecutive identical pulse intervals is increased by one until the first pulse interval is encountered again or the pulse sequence encoding ends, and the number of superimposed pulse intervals is output, indicating the encoding of the aggregated pulse interval, and the above steps are repeated to process each pixel position until the pulse array processing is completed, and the code stream terminator is output.

2. The pulse camera data encoding method for monitoring scenes according to claim 1 is characterized in that: The specific steps of S1 are: inputting an original pulse array with a time length of T, using a sliding window with a step size of 1 to input a number of continuous pulse array fragments into the CNN pulse decoder, and then converting the pulse array into a light intensity matrix, and removing the spatiotemporal noise of the original pulse array, and extracting the light intensity information in the original pulse array.

3. The pulse camera data encoding method for monitoring scenes according to claim 2 is characterized in that: The CNN pulse array decoder used in S1 adopts a U-Net network structure to extract the light intensity matrix.

4. The pulse camera data encoding method for monitoring scenes according to claim 2, characterized in that: The specific operation process of the sliding window in S1 is as follows: for an original pulse array with a length of T, a full-zero pulse array with a length of N is supplemented at both ends of the original pulse array to form a synthetic array, and starting from time 0 as the starting point of the sliding window, several continuous pulse array segments are input into the CNN pulse decoder.

5. The pulse camera data encoding method for monitoring scenes according to claim 1, characterized in that: The entropy coding adopts a combination of Huffman coding and adaptive context binary arithmetic coding to finally encode the code stream elements in the first pulse interval aggregation encoder into a binary file.

6. The pulse camera data encoding method for monitoring scenes according to claim 1, characterized in that: The specific steps of the entropy coding include: S3.

1. Use the binary arithmetic decoder and Huffman decoder with adaptive context to decode the binary code stream file; S3.

2. Read the fixed byte length of the pulse array inherent properties from the entropy decoding file to obtain the pulse array inherent properties of the pulse array to be decoded; S3.

3. Read a single pulse interval and determine whether it is greater than the maximum pulse interval limit coefficient and the end symbol of the code stream. If it is less than the coefficient, it is represented as the first pulse interval, and it is directly decoded into the corresponding pulse sequence. If it is greater than the maximum pulse interval limit coefficient, it is represented as an aggregated pulse interval, and the maximum pulse interval limit coefficient is subtracted from it to obtain the number of continuous pulse intervals, and the previous pulse interval is obtained, and the decoding is performed according to the previous pulse interval of the number of continuous pulse intervals; S3.

4. Return to S3.2 until the end symbol of the code stream is detected, the decoding of the entire pulse sequence is completed, and the decoded one-dimensional pulse sequence is divided according to the inherent properties of the pulse array to synthesize the pulse array format.

Images