Pulse data compression method, device, electronic device and medium based on neural network

Abstract

The present application discloses a pulse data compression method, device, electronic device and medium based on a neural network. In the present application, a pulse sequence to be processed can be obtained, and based on the long-short time-course feature fusion module of the pulse compression model, the reconstructed scene information corresponding to the pulse sequence to be processed can be obtained; after the reconstructed scene information is compressed into a code stream to be processed using an image encoder, the code stream to be processed is decoded to obtain decoded scene information; through the accumulation and release mechanism of biological neurons, the decoded scene information is converted into a decoded pulse sequence. By applying the technical solution of the present application, a method can be implemented in which a scene is reconstructed for a pulse sequence using a neural network model, and then the reconstructed scene information is encoded to achieve indirect compression of the pulse sequence. Thereby achieving the purpose of constructing an efficient and compact representation of pulse data while ensuring the information fidelity of the pulse data.

Description

Technical Field

[0001] The present application relates to digital signal processing technology, and in particular to a pulse data compression method, device, electronic device and medium based on a neural network. Background Art

[0002] Deep learning has continuously achieved breakthroughs in traditional computer vision tasks, surpassing the state-of-the-art VVC standard for image coding. However, pulse data, due to its binary nature and poor spatiotemporal correlation, cannot be directly applied to existing deep learning-based image and video coding frameworks.

[0003] Among them, the existing pulse data encoding method converts the original pulse into the pulse interval (Inter Spike Interval, ISI), and then divides the ISI sequence into blocks in the time and space domain. The ISI value range within each block is small. The ISI block can be efficiently represented by encoding the mean and the residual of each point in the block, thereby representing the entire ISI sequence and restoring the pulse sequence.

[0004] However, because this method's block partitioning is manually designed, it suffers from high coding loss for sequences with intense motion. Therefore, finding a solution for efficiently and compactly representing pulse data while minimizing the fidelity of the reconstructed data has become a challenge for those skilled in the art. Summary of the Invention

[0005] The embodiments of the present application provide a neural network-based pulse data compression method, device, electronic device and medium. The embodiments of the present application are used to solve the problem in the related art that there is no efficient pulse data compression.

[0006] According to one aspect of the embodiments of the present application, a neural network-based pulse data compression method is provided, the method comprising:

[0007] Obtaining a pulse sequence to be processed, and obtaining reconstructed scene information corresponding to the pulse sequence to be processed based on a long-short time feature fusion module of a pulse compression model;

[0008] After compressing the reconstructed scene information into a code stream to be processed using an image encoder, decoding the code stream to be processed to obtain decoded scene information;

[0009] The decoded scene information is converted into a decoded pulse sequence through the accumulation and firing mechanism of biological neurons.

[0010] Optionally, in another embodiment of the method of the present application, the long-short time course feature fusion module based on the pulse compression model obtains the reconstructed scene information corresponding to the pulse sequence to be processed, including:

[0011] Dividing the pulse sequence to be processed into a plurality of pulse blocks, wherein the pulse blocks have the same length and intersect with each other;

[0012] Inputting the plurality of pulse blocks into the short-term feature fusion module of the pulse compression model to obtain initial reconstructed scene information;

[0013] The initial reconstructed scene information is input into the long-term feature fusion module of the pulse compression model to obtain the reconstructed scene information with a fineness higher than that of the initial reconstructed scene information.

[0014] Optionally, in another embodiment of the above method of the present application, inputting the multiple pulse blocks into the short-term feature fusion module of the pulse compression model to obtain initial reconstructed scene information includes:

[0015] Through the mixed-scale time domain filtering module, the time domain features of each pulse block at different scales are extracted;

[0016] Using the channel self-attention module, the time domain features at different scales are fused to obtain fused features;

[0017] The spatiotemporal information of the fused features is integrated and gradiented using multiple spatiotemporal residual modules to obtain the preliminary reconstructed scene.

[0018] Optionally, in another embodiment of the above method of the present application, inputting the initial reconstructed scene information into the long-term feature fusion module of the pulse compression model to obtain the reconstructed scene information having a finer degree than the initial reconstructed scene information includes:

[0019] Sampling the initial reconstructed scene information to obtain the initial reconstructed scene information at multiple different spatial scales; and obtaining target reconstructed scene information at multiple different spatial scales, wherein the target reconstructed scene information is scene information having the same scene type as the initial reconstructed scene information;

[0020] Using a channel self-attention module, feature alignment is performed on the initial reconstructed scene information and the target reconstructed scene information at each spatial scale to obtain a plurality of aligned features;

[0021] The multiple aligned features are fused using a mutual attention mechanism module to obtain the reconstructed scene information.

[0022] Optionally, in another embodiment of the above method of the present application, compressing the reconstructed scene information into a code stream to be processed by using an image encoder includes:

[0023] Extracting motion information of the pulse sequence to be processed through a convolutional long short-term memory (ConvLSTM) module and converting the motion information into an attention map; and extracting shallow feature information of the reconstructed scene information using the pulse compression model;

[0024] Performing feature encoding on the attention map and the shallow feature information after weighting to obtain encoded features;

[0025] The coding feature is input into an entropy encoder to obtain the code stream to be processed.

[0026] Optionally, in another embodiment of the method of the present application, the over-convolution long short-term memory (ConvLSTM) module extracts motion information of the pulse sequence to be processed and converts the motion information into an attention map, including:

[0027] Dividing the pulse sequence to be processed into a preceding subsequence and a succeeding subsequence according to a time domain order;

[0028] Inputting the preamble subsequence into the ConvLSTM module to extract the first motion information of the pulse sequence to be processed; and

[0029] Performing time-domain flipping processing on the subsequent subsequence to obtain a flipped subsequence, and inputting the flipped subsequence into the ConvLSTM module to extract the second motion information of the pulse sequence to be processed;

[0030] The first motion information and the second motion information are integrated and extracted to obtain the attention map.

[0031] Optionally, in another embodiment of the method of the present application, decoding the code stream to be processed to obtain decoded scene information includes:

[0032] Decoding the code stream to be processed using an entropy decoder to obtain a reconstructed feature;

[0033] Feature decoding is performed on the reconstructed features to obtain the decoded scene information.

[0034] According to another aspect of the embodiments of the present application, a neural network-based pulse data compression device is provided, comprising:

[0035] an acquisition module configured to acquire a pulse sequence to be processed and obtain reconstructed scene information corresponding to the pulse sequence to be processed based on a long- and short-time feature fusion module of a pulse compression model;

[0036] a processing module configured to compress the reconstructed scene information into a code stream to be processed using an image encoder, and then decode the code stream to be processed to obtain decoded scene information;

[0037] The conversion module is configured to convert the decoded scene information into a decoded pulse sequence through the accumulation and firing mechanism of biological neurons.

[0038] According to another aspect of the embodiments of the present application, an electronic device is provided, including:

[0039] a memory for storing executable instructions; and

[0040] A display is used to execute the executable instructions with the memory to complete the operation of any of the above-mentioned neural network-based pulse data compression methods.

[0041] According to another aspect of an embodiment of the present application, a computing device readable storage medium is provided for storing instructions readable by the computing device, wherein the instructions, when executed, perform the operations of any of the above-mentioned neural network-based pulse data compression methods.

[0042] In the present application, a pulse sequence to be processed can be obtained, and based on the long-short time-course feature fusion module of the pulse compression model, the reconstructed scene information corresponding to the pulse sequence to be processed can be obtained; after the reconstructed scene information is compressed into a code stream to be processed using an image encoder, the code stream to be processed is decoded to obtain decoded scene information; through the accumulation and release mechanism of biological neurons, the decoded scene information is converted into a decoded pulse sequence. By applying the technical solution of the present application, a method can be implemented to reconstruct the scene of a pulse sequence using a neural network model, and then the reconstructed scene information is encoded to achieve indirect compression of the pulse sequence. Thereby achieving the purpose of constructing an efficient and compact representation of pulse data while ensuring the information fidelity of the pulse data.

[0043] The technical solution of the present application is further described in detail below using multiple embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and, together with the description, serve to explain the principles of the present application.

[0045] The present application can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0046] Figure 1A schematic diagram of a system architecture for neural network-based pulse data compression provided by an embodiment of the present application is shown;

[0047] Figure 2 A system architecture diagram of a neural network-based pulse data compression system provided by an embodiment of the present application is shown;

[0048] Figure 3 A schematic diagram of the overall process of neural network-based pulse data compression provided by an embodiment of the present application is shown;

[0049] Figure 4 A schematic diagram of a process for generating an attention map provided by an embodiment of the present application is shown;

[0050] Figure 5 A schematic structural diagram of an electronic device provided in one embodiment of the present application is shown;

[0051] Figure 6 A schematic structural diagram of an electronic device provided in one embodiment of the present application is shown;

[0052] Figure 7 A schematic diagram of a storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0053] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present application.

[0054] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0055] The following description of at least one exemplary embodiment is merely illustrative in nature and is not intended to limit the present disclosure, its application, or uses.

[0056] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0057] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0058] In addition, the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in this field can implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0059] It should be noted that all directional indications in the embodiments of the present application (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0060] The following combination Figures 1-4 The following describes a method for performing neural network-based pulse data compression according to an exemplary embodiment of the present application. It should be noted that the following application scenarios are merely provided to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0061] Furthermore, the present application also proposes a neural network-based pulse data compression method, device, traveling vehicle and medium.

[0062] Figure 1 The flowchart of a pulse data compression method based on a neural network according to an embodiment of the present application is schematically shown. Figure 1 As shown, the method includes:

[0063] S101, obtaining a pulse sequence to be processed, and obtaining reconstructed scene information corresponding to the pulse sequence to be processed based on a long-short time feature fusion module of a pulse compression model.

[0064] S102 , compressing the reconstructed scene information into a code stream to be processed by using an image encoder, and decoding the code stream to be processed to obtain decoded scene information.

[0065] S103, through the accumulation and release mechanism of biological neurons, the decoded scene information is converted into a decoding pulse sequence.

[0066] In related technologies, deep learning has continuously made breakthroughs in traditional computer vision tasks, and its performance in image coding tasks has surpassed the most advanced traditional image standard VVC.

[0067] Furthermore, for pulse data, due to its characteristics such as binary values ​​and poor spatiotemporal correlation, the existing deep learning-based image and video coding framework cannot be directly used.

[0068] Instead, existing pulse data encoding methods usually convert the original pulse into an inter-spike interval (ISI), and then divide the ISI sequence into blocks in the time and space domain. The ISI value range within each block is small, and the ISI block can be efficiently represented by encoding the mean and the residual of each point in the block, thereby representing the entire ISI sequence and restoring the pulse sequence.

[0069] However, since the block division method of this method is manually designed, it also has the disadvantage of high coding loss for sequences with more intense motion.

[0070] Based on the above problems, an embodiment of the present application proposes a pulse data compression method based on a neural network. The solution is to first reconstruct the pulse sequence into scene information, then compress the scene information into a code stream using an image encoder, and then decode the corresponding decoded scene information from the code stream, so that the accumulation-release mechanism can be used to simulate the decoded scene information to generate a compressed pulse sequence.

[0071] In one way, Figure 2 As shown, this application proposes a pulse compression model based on a neural network to perform three-stage processing on pulse data, thereby achieving a technical solution for compressing pulse data. Specifically, it includes:

[0072] Phase 1:

[0073] In the embodiment of the present application, the pulse sequence to be processed can be divided into multiple pulse blocks of equal length and intersecting with each other. These multiple pulse blocks are then input into different branches of the pulse compression model. This allows the initial reconstruction of the scene to be obtained through the short-term feature fusion module, and then, after alignment of this information, the long-term feature fusion module is used to obtain more refined reconstruction of the scene.

[0074] Phase 2:

[0075] The embodiment of the present application can extract features from the reconstructed scene information to obtain shallow feature information. It also obtains an attention map for characterizing motion information in the pulse sequence information to be processed, and then performs feature encoding after weighting the two. The encoded features are then input into an entropy encoder to convert them into a bitstream to be processed.

[0076] Furthermore, in the embodiment of the present application, an entropy decoder may decode the code stream to be processed into reconstructed features, and then obtain decoded scene information through feature decoding.

[0077] Phase 3:

[0078] Through the accumulation-release mechanism of biological neurons, the decoding scene sequence is converted into a decoding pulse sequence.

[0079] In the present application, a pulse sequence to be processed can be obtained, and based on the long-short time-course feature fusion module of the pulse compression model, the reconstructed scene information corresponding to the pulse sequence to be processed can be obtained; after the reconstructed scene information is compressed into a code stream to be processed using an image encoder, the code stream to be processed is decoded to obtain decoded scene information; through the accumulation and release mechanism of biological neurons, the decoded scene information is converted into a decoded pulse sequence. By applying the technical solution of the present application, a method can be implemented to reconstruct the scene of a pulse sequence using a neural network model, and then the reconstructed scene information is encoded to achieve indirect compression of the pulse sequence. Thereby achieving the purpose of constructing an efficient and compact representation of pulse data while ensuring the information fidelity of the pulse data.

[0080] Optionally, in another embodiment of the method of the present application, the long-short time course feature fusion module based on the pulse compression model obtains the reconstructed scene information corresponding to the pulse sequence to be processed, including:

[0081] Dividing the pulse sequence to be processed into a plurality of pulse blocks, wherein the pulse blocks have the same length and intersect with each other;

[0082] Inputting the plurality of pulse blocks into the short-term feature fusion module of the pulse compression model to obtain initial reconstructed scene information;

[0083] The initial reconstructed scene information is input into the long-term feature fusion module of the pulse compression model to obtain the reconstructed scene information with a fineness higher than that of the initial reconstructed scene information.

[0084] Optionally, in another embodiment of the above method of the present application, inputting the multiple pulse blocks into the short-term feature fusion module of the pulse compression model to obtain initial reconstructed scene information includes:

[0085] Through the mixed-scale time domain filtering module, the time domain features of each pulse block at different scales are extracted;

[0086] Using the channel self-attention module, the time domain features at different scales are fused to obtain fused features;

[0087] The spatiotemporal information of the fused features is integrated and gradiented using multiple spatiotemporal residual modules to obtain the preliminary reconstructed scene.

[0088] Optionally, in another embodiment of the above method of the present application, inputting the initial reconstructed scene information into the long-term feature fusion module of the pulse compression model to obtain the reconstructed scene information having a finer degree than the initial reconstructed scene information includes:

[0089] Sampling the initial reconstructed scene information to obtain the initial reconstructed scene information at multiple different spatial scales; and obtaining target reconstructed scene information at multiple different spatial scales, wherein the target reconstructed scene information is scene information having the same scene type as the initial reconstructed scene information;

[0090] Using a channel self-attention module, feature alignment is performed on the initial reconstructed scene information and the target reconstructed scene information at each spatial scale to obtain a plurality of aligned features;

[0091] The multiple aligned features are fused using a mutual attention mechanism module to obtain the reconstructed scene information.

[0092] Optionally, in another embodiment of the above method of the present application, compressing the reconstructed scene information into a code stream to be processed by using an image encoder includes:

[0093] Extracting motion information of the pulse sequence to be processed through a convolutional long short-term memory (ConvLSTM) module and converting the motion information into an attention map; and extracting shallow feature information of the reconstructed scene information using the pulse compression model;

[0094] Performing feature encoding on the attention map and the shallow feature information after weighting to obtain encoded features;

[0095] The coding feature is input into an entropy encoder to obtain the code stream to be processed.

[0096] Optionally, in another embodiment of the method of the present application, the over-convolution long short-term memory (ConvLSTM) module extracts motion information of the pulse sequence to be processed and converts the motion information into an attention map, including:

[0097] Dividing the pulse sequence to be processed into a preceding subsequence and a succeeding subsequence according to a time domain order;

[0098] Inputting the preamble subsequence into the ConvLSTM module to extract the first motion information of the pulse sequence to be processed; and

[0099] Performing time-domain flipping processing on the subsequent subsequence to obtain a flipped subsequence, and inputting the flipped subsequence into the ConvLSTM module to extract the second motion information of the pulse sequence to be processed;

[0100] The first motion information and the second motion information are integrated and extracted to obtain the attention map.

[0101] Optionally, in another embodiment of the method of the present application, decoding the code stream to be processed to obtain decoded scene information includes:

[0102] Decoding the code stream to be processed using an entropy decoder to obtain a reconstructed feature;

[0103] Feature decoding is performed on the reconstructed features to obtain the decoded scene information.

[0104] In one way, Figure 3 FIG. 1 shows a schematic diagram of the overall process of a neural network-based pulse data compression method provided by an embodiment of the present application, which includes:

[0105] Step 1: Obtain a pulse sequence to be processed and divide the pulse sequence to be processed into multiple pulse blocks.

[0106] The pulse blocks have the same length and intersect with each other.

[0107] Step 2: Input multiple pulse blocks into the short-term feature fusion module of the pulse compression model to obtain the initial reconstructed scene information.

[0108] In one approach, an embodiment of the present application can first utilize a short-term feature fusion module, use mixed-scale time domain filtering to extract the time domain features of each pulse block at different scales, and then utilize a channel self-attention module to fuse features of different scales, and integrate and gradient the spatiotemporal domain information through multiple spatiotemporal residual modules to obtain the initial reconstructed scene information.

[0109] Step 3: Input the initial reconstructed scene information into the long-term feature fusion module of the pulse compression model to obtain reconstructed scene information with higher precision than the initial reconstructed scene information.

[0110] In one approach, the embodiments of the present application can utilize a long-term feature fusion module to downsample the initial reconstructed scene information and the reconstructed target scene to multiple spatial scales, align the features of each scale based on a deformable convolutional neural network (DCN), and use this to guide the alignment of the upper scale. The aligned features are obtained and fused using a mutual attention mechanism to ultimately obtain more refined reconstructed scene information.

[0111] Step 4: Divide the pulse sequence to be processed into a preceding subsequence and a succeeding subsequence according to the time domain order.

[0112] In one approach, considering that pulse data is often used to record high-speed motion scene information, the present embodiment can utilize a Convolutional Long Short-Term Memory (ConvLSTM) network to extract the motion information contained therein and generate an attention map to weight the features to be encoded, so that the model can more efficiently handle the blur and distortion caused by motion.

[0113] Furthermore, since the time domain features of the pulse sequence are simpler, the embodiments of this application do not use convolution operations to extract its time domain features. Instead, the ConvLSTM module with a small number of parameters is used to extract the characteristics of the spatiotemporal domain information and integrate this information into an attention map to guide the efficient compression of the reconstructed scene.

[0114] Step 5: Input the preceding subsequence into the ConvLSTM module to extract the first motion information of the pulse sequence to be processed; and perform time domain flipping processing on the subsequent subsequence to obtain a flipped subsequence, and input the flipped subsequence into the ConvLSTM module to extract the second motion information of the pulse sequence to be processed.

[0115] In one approach, the present application proposes a bidirectional attention module for compressing pulse trains. Figure 4 As shown, in the embodiment of the present application, the pulse sequence to be processed can be first divided into a preceding subsequence and a subsequent subsequence in the time domain.

[0116] Furthermore, the preceding subsequence is fed into a ConvLSTM module to extract the primary motion information contained therein. The subsequent subsequence is then time-flipped and, after obtaining the flipped subsequence, is also fed into a ConvLSTM module to extract the secondary motion information contained therein. This allows the motion information of each subsequence to be integrated and extracted to generate an attention map, thereby guiding the efficient compression of the reconstructed scene.

[0117] Step 6: Integrate and extract the first motion information and the second motion information to obtain an attention map.

[0118] Step 7: Use the pulse compression model to extract shallow feature information of the reconstructed scene information, and perform feature encoding on the attention map and the shallow feature information after weighting to obtain the encoded features.

[0119] Step 8: Use an entropy decoder to decode the bitstream to be processed to obtain reconstructed features, and perform feature decoding on the reconstructed features to obtain decoded scene information.

[0120] Step 9: Convert the decoded scene information into a decoded pulse sequence through the accumulation and release mechanism of biological neurons.

[0121] In the present application, a pulse sequence to be processed can be obtained, and based on the long-short time-course feature fusion module of the pulse compression model, the reconstructed scene information corresponding to the pulse sequence to be processed can be obtained; after the reconstructed scene information is compressed into a code stream to be processed using an image encoder, the code stream to be processed is decoded to obtain decoded scene information; through the accumulation and release mechanism of biological neurons, the decoded scene information is converted into a decoded pulse sequence. By applying the technical solution of the present application, a method can be implemented to reconstruct the scene of a pulse sequence using a neural network model, and then the reconstructed scene information is encoded to achieve indirect compression of the pulse sequence. Thereby achieving the purpose of constructing an efficient and compact representation of pulse data while ensuring the information fidelity of the pulse data.

[0122] In another embodiment of the present application, Figure 5 As shown, the present application also provides a pulse data compression device based on a neural network. It includes:

[0123] An acquisition module 201 is configured to acquire a pulse sequence to be processed and obtain reconstructed scene information corresponding to the pulse sequence to be processed based on a long- and short-time feature fusion module of a pulse compression model;

[0124] The processing module 202 is configured to compress the reconstructed scene information into a code stream to be processed using an image encoder, and then decode the code stream to be processed to obtain decoded scene information;

[0125] The conversion module 203 is configured to convert the decoded scene information into a decoded pulse sequence through the accumulation and firing mechanism of biological neurons.

[0126] In the present application, a pulse sequence to be processed can be obtained, and based on the long-short time-course feature fusion module of the pulse compression model, the reconstructed scene information corresponding to the pulse sequence to be processed can be obtained; after the reconstructed scene information is compressed into a code stream to be processed using an image encoder, the code stream to be processed is decoded to obtain decoded scene information; through the accumulation and release mechanism of biological neurons, the decoded scene information is converted into a decoded pulse sequence. By applying the technical solution of the present application, a method can be implemented to reconstruct the scene of a pulse sequence using a neural network model, and then the reconstructed scene information is encoded to achieve indirect compression of the pulse sequence. Thereby achieving the purpose of constructing an efficient and compact representation of pulse data while ensuring the information fidelity of the pulse data.

[0127] In another embodiment of the present application, the processing module 202 is configured to:

[0128] Dividing the pulse sequence to be processed into a plurality of pulse blocks, wherein the pulse blocks have the same length and intersect with each other;

[0129] Inputting the plurality of pulse blocks into the short-term feature fusion module of the pulse compression model to obtain initial reconstructed scene information;

[0130] The initial reconstructed scene information is input into the long-term feature fusion module of the pulse compression model to obtain the reconstructed scene information with a fineness higher than that of the initial reconstructed scene information.

[0131] In another embodiment of the present application, the processing module 202 is configured to:

[0132] Through the mixed-scale time domain filtering module, the time domain features of each pulse block at different scales are extracted;

[0133] Using the channel self-attention module, the time domain features at different scales are fused to obtain fused features;

[0134] The spatiotemporal information of the fused features is integrated and gradiented using multiple spatiotemporal residual modules to obtain the preliminary reconstructed scene.

[0135] In another embodiment of the present application, the processing module 202 is configured to:

[0136] Sampling the initial reconstructed scene information to obtain the initial reconstructed scene information at multiple different spatial scales; and obtaining target reconstructed scene information at multiple different spatial scales, wherein the target reconstructed scene information is scene information having the same scene type as the initial reconstructed scene information;

[0137] Using a channel self-attention module, feature alignment is performed on the initial reconstructed scene information and the target reconstructed scene information at each spatial scale to obtain a plurality of aligned features;

[0138] The multiple aligned features are fused using a mutual attention mechanism module to obtain the reconstructed scene information.

[0139] In another embodiment of the present application, the processing module 202 is configured to:

[0140] Extracting motion information of the pulse sequence to be processed through a convolutional long short-term memory (ConvLSTM) module and converting the motion information into an attention map; and extracting shallow feature information of the reconstructed scene information using the pulse compression model;

[0141] Performing feature encoding on the attention map and the shallow feature information after weighting to obtain encoded features;

[0142] The coding feature is input into an entropy encoder to obtain the code stream to be processed.

[0143] In another embodiment of the present application, the processing module 202 is configured to:

[0144] Dividing the pulse sequence to be processed into a preceding subsequence and a succeeding subsequence according to a time domain order;

[0145] Inputting the preamble subsequence into the ConvLSTM module to extract the first motion information of the pulse sequence to be processed; and

[0146] Performing time-domain flipping processing on the subsequent subsequence to obtain a flipped subsequence, and inputting the flipped subsequence into the ConvLSTM module to extract the second motion information of the pulse sequence to be processed;

[0147] The first motion information and the second motion information are integrated and extracted to obtain the attention map.

[0148] In another embodiment of the present application, the processing module 202 is configured to:

[0149] Decoding the code stream to be processed using an entropy decoder to obtain a reconstructed feature;

[0150] Feature decoding is performed on the reconstructed features to obtain the decoded scene information.

[0151] The present application also provides an electronic device to perform the above-mentioned neural network-based pulse data compression method. Figure 6 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 6 As shown, the electronic device 3 includes: a processor 300, a memory 301, a bus 302 and a communication interface 303, and the processor 300, the communication interface 303 and the memory 301 are connected via the bus 302; the memory 301 stores a computer program that can be run on the processor 300, and when the processor 300 runs the computer program, it executes the neural network-based pulse data compression method provided in any of the aforementioned embodiments of the present application.

[0152] The memory 301 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The communication connection between the device network element and at least one other network element is achieved through at least one communication interface 303 (which may be wired or wireless), and may use the Internet, a wide area network, a local area network, a metropolitan area network, etc.

[0153] The bus 302 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 301 is used to store programs. Upon receiving an execution instruction, the processor 300 executes the program. The video transmission method disclosed in any of the aforementioned embodiments of the present application may be applied to or implemented by the processor 300.

[0154] The processor 300 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 300 or by software instructions. The above processor 300 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 301 , and the processor 300 reads the information in the memory 301 and completes the steps of the above method in combination with its hardware.

[0155] The electronic device provided in the embodiment of the present application and the neural network-based pulse data compression method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented by them.

[0156] The present application also provides a computer-readable storage medium corresponding to the neural network-based pulse data compression method provided in the above embodiment. Figure 7 The computer-readable storage medium shown is a CD 40 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the video transmission method provided by any of the aforementioned embodiments is executed.

[0157] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0158] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the video transmission method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0159] It should be noted that:

[0160] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0161] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed application requires more features than the features expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present application.

[0162] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of this application and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0163] The above description is merely a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A pulse data compression method based on neural network, characterized in that: The method comprises: Obtaining a pulse sequence to be processed, and obtaining reconstructed scene information corresponding to the pulse sequence to be processed based on a long-short time feature fusion module of a pulse compression model; After compressing the reconstructed scene information into a code stream to be processed using an image encoder, decoding the code stream to be processed to obtain decoded scene information; The decoding scene information is converted into a decoding pulse sequence through the accumulation and firing mechanism of biological neurons; The long- and short-time feature fusion module based on the pulse compression model obtains the reconstructed scene information corresponding to the pulse sequence to be processed, including: Dividing the pulse sequence to be processed into a plurality of pulse blocks, wherein the pulse blocks have the same length and intersect with each other; Inputting the plurality of pulse blocks into the short-term feature fusion module of the pulse compression model to obtain initial reconstructed scene information; Inputting the initial reconstructed scene information into the long-term feature fusion module of the pulse compression model to obtain the reconstructed scene information having a finer degree than the initial reconstructed scene information; Inputting the plurality of pulse blocks into the short-term feature fusion module of the pulse compression model to obtain initial reconstructed scene information includes: Through the mixed-scale time domain filtering module, the time domain features of each pulse block at different scales are extracted; Using the channel self-attention module, the time domain features at different scales are fused to obtain fused features; Using multiple spatiotemporal residual modules to integrate and gradient the spatiotemporal information of the fusion features to obtain the initial reconstructed scene; Inputting the initial reconstructed scene information into the long-term feature fusion module of the pulse compression model to obtain the reconstructed scene information having a finer degree than the initial reconstructed scene information includes: Sampling the initial reconstructed scene information to obtain the initial reconstructed scene information at multiple different spatial scales; and obtaining target reconstructed scene information at multiple different spatial scales, wherein the target reconstructed scene information is scene information having the same scene type as the initial reconstructed scene information; Using a channel self-attention module, feature alignment is performed on the initial reconstructed scene information and the target reconstructed scene information at each spatial scale to obtain a plurality of aligned features; The multiple aligned features are fused using a mutual attention mechanism module to obtain the reconstructed scene information.

2. The method according to claim 1, wherein The compressing the reconstructed scene information into a code stream to be processed by using an image encoder includes: Extracting motion information of the pulse sequence to be processed through a convolutional long short-term memory (ConvLSTM) module and converting the motion information into an attention map; and extracting shallow feature information of the reconstructed scene information using the pulse compression model; Performing feature encoding on the attention map and the shallow feature information after weighting to obtain encoded features; The coding feature is input into an entropy encoder to obtain the code stream to be processed.

3. The method according to claim 2, wherein The convolutional long short-term memory (ConvLSTM) module extracts motion information of the pulse sequence to be processed and converts the motion information into an attention map, including: Dividing the pulse sequence to be processed into a preceding subsequence and a succeeding subsequence according to a time domain order; Inputting the preamble subsequence into the ConvLSTM module to extract the first motion information of the pulse sequence to be processed; and Performing time-domain flipping processing on the subsequent subsequence to obtain a flipped subsequence, and inputting the flipped subsequence into the ConvLSTM module to extract the second motion information of the pulse sequence to be processed; The first motion information and the second motion information are integrated and extracted to obtain the attention map.

4. The method according to claim 1, wherein The decoding process of the to-be-processed code stream to obtain decoding scene information includes: Decoding the code stream to be processed using an entropy decoder to obtain a reconstructed feature; Feature decoding is performed on the reconstructed features to obtain the decoded scene information.

5. A pulse data compression device based on a neural network, implemented by the method of any one of claims 1 to 4, characterized in that: include: an acquisition module configured to acquire a pulse sequence to be processed and obtain reconstructed scene information corresponding to the pulse sequence to be processed based on a long- and short-time feature fusion module of a pulse compression model; a processing module configured to compress the reconstructed scene information into a code stream to be processed using an image encoder, and then decode the code stream to be processed to obtain decoded scene information; The conversion module is configured to convert the decoded scene information into a decoded pulse sequence through the accumulation and firing mechanism of biological neurons.

6. An electronic device, characterized in that: include: a memory for storing executable instructions; as well as, A processor is used to execute the executable instructions with the memory to complete the operation of the neural network-based pulse data compression method according to any one of claims 1 to 4.

7. A computing device readable storage medium for storing instructions readable by a computing device, characterized in that: When the instructions are executed, the operations of the neural network-based pulse data compression method according to any one of claims 1 to 4 are performed.

Images