Image processing method and system

Abstract

The invention discloses an image processing method and system. The method comprises the following steps: acquiring M pieces of original image data and pulse data; carrying out discarding preprocessing on the M pieces of original image data to obtain N pieces of processed image data; according to the time sequence of the timestamps, the processed image data and the pulse data are transmitted in parallel through two transmission channels; recovering the discarded processed image data according to the processed image data and the pulse data obtained through transmission; and inputting the pulse data into the pulse neural network for pulse processing to obtain pulse result data, and comparing the pulse result data with the processed image data to adjust parameters of the pulse neural network. Before transmission, the original image data is discarded and preprocessed, key information of the image data is reserved, and the redundancy of a transmission channel is reduced; data transmission errors can be avoided by adopting two transmission channels according to the time sequence of the timestamps; and after the transmission is finished, the discarded processed image data is recovered, and the integrity of the data is ensured.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method and system. Background Art

[0002] Brain-like computing is a computing method that imitates the working principle of the human brain's nervous system. It aims to develop fast, reliable, and low-power computing technology, hoping to study the working mechanism of the human brain and simulate a robot with the ability to think and learn like humans. Spiking Neural Network (SNN) is known as the "third generation of neural network". Compared with the currently popular numerical deep learning neural network, the spiking neural network has more brain-like characteristics in terms of connection mode, information processing mechanism, and synaptic weight learning method. It is an important research direction of brain-like computing.

[0003] Pulse neural networks process pulse sequences, and there are generally two ways to obtain pulse sequences: one is to convert images into pulse sequences through some encoding method (such as threshold encoding, frequency encoding, Gaussian difference method, etc.); the other is derived from neuromorphic cameras. Neuromorphic cameras are special cameras that integrate unique image perception and communication methods into hardware systems to achieve special device designs. The pixels of this camera can produce pulse responses to changes in light intensity, similar to the working method of retinal neurons in biological visual systems. The pulse signals it outputs can be directly used as input to pulse neural networks and are the direct source of pulse sequences.

[0004] Existing neuromorphic cameras usually include a camera module, a pulse sensing module and a communication module. The camera module is used to take pictures to obtain specific images, and the pulse sensing module is used to generate pulse data according to specific image sensing; the communication module is used to transmit data with the pulse neural network through a channel. However, there is a large amount of communication information redundancy in the channel transmission of image data and pulse data, resulting in waste of resources and increased transmission costs. Summary of the invention

[0005] The present invention provides an image processing method and system for solving the technical problem that a large amount of communication information redundancy exists in the channel transmission of image data and pulse data, resulting in waste of resources and increased transmission cost.

[0006] A first aspect of the present invention provides an image processing method, the method comprising:

[0007] Acquire M raw image data and pulse data;

[0008] Performing a discard preprocessing on the M original image data to obtain N processed image data; wherein M and N are both positive integers, and M>N;

[0009] According to the time sequence of the timestamps, the processed image data and the pulse data are respectively transmitted in parallel and intermittently through two transmission channels;

[0010] restoring the discarded processed image data according to the transmitted processed image data and the pulse data;

[0011] The pulse data is input into the pulse neural network for pulse processing to obtain pulse result data; the processed image data is compared and judged with the pulse result data to obtain a judgment result to adjust the parameters of the pulse neural network.

[0012] More specifically, the step of acquiring M original image data and pulse data includes:

[0013] Acquire M raw image data through the camera according to a preset shooting interval;

[0014] The M original image data are converted into M pulse data by using a frame difference method.

[0015] More specifically, the step of discarding the M original image data to obtain N processed image data includes:

[0016] Comparing the overlap between the original image data and the pulse data, and cropping the original image data;

[0017] By discarding the M cropped image data, N processed image data are obtained.

[0018] More specifically, the step of discarding the M cropped image data to obtain N processed image data includes:

[0019] sorting the M cropped image data based on timestamps of the M cropped image data;

[0020] According to a preset discarding interval number, the sorted M cropped image data are grouped to obtain multiple groups of image data;

[0021] The first cropped image data in each group of image data is retained, and the remaining cropped image data are discarded, so as to obtain N processed image data.

[0022] A second aspect of the present invention further provides an image processing system, the system comprising:

[0023] An acquisition module, used for acquiring M original image data and pulse data;

[0024] A preprocessing module, used for discarding and preprocessing the M original image data to obtain N processed image data; wherein M and N are both positive integers, and M>N;

[0025] A transmission module, used for aligning the processed image data and the pulse data according to the timestamp, and transmitting the processed image data and the pulse data in parallel and intermittently through two transmission channels respectively;

[0026] A recovery module, used for recovering the discarded processed image data according to the transmitted processed image data and the pulse data;

[0027] The pulse processing module is used to input the pulse data into the pulse neural network for pulse processing to obtain pulse result data; compare and distinguish the processed image data with the pulse result data to obtain the distinction result to adjust the parameters of the pulse neural network.

[0028] More specifically, the acquisition module includes:

[0029] The camera submodule is used to obtain M raw image data through a camera according to a preset shooting interval;

[0030] The conversion submodule is used to convert the M original image data into M pulse data by using a frame difference method.

[0031] More specifically, the preprocessing module includes:

[0032] A cropping submodule, used for comparing the overlap between the original image data and the pulse data, and cropping the original image data;

[0033] The discarding submodule is used to obtain N processed image data by discarding the M cropped image data.

[0034] More specifically, the discarding submodule is specifically used to sort the M cropped image data based on the timestamps of the M cropped image data; group the sorted M cropped image data according to a preset discarding interval number to obtain multiple groups of image data; retain the first cropped image data in each group of image data, and discard the remaining cropped image data to obtain N processed image data.

[0035] More specifically, the recovery module is specifically used to, if only pulse data is received in the current time step, add the processed image data received in the previous time step to the pulse data, and recover and save them as the processed image data of the current time step.

[0036] It can be seen from the above technical solutions that the present invention has the following advantages:

[0037] The present invention provides an image processing method and system, wherein the method comprises: acquiring M original image data and pulse data; performing discard preprocessing on the M original image data to obtain N processed image data; transmitting the processed image data and pulse data in parallel and at intervals through two transmission channels respectively according to the time sequence of timestamps; restoring the discarded processed image data according to the transmitted processed image data and pulse data; inputting the pulse data into a pulse neural network for pulse processing to obtain pulse result data; comparing and distinguishing the processed image data and the pulse result data to obtain a distinction result to adjust the parameters of the pulse neural network.

[0038] The present invention discards the original image data before data transmission, retains the key information of the image data, effectively reduces the amount of data transmitted through the channel and reduces the transmission cost; at the same time, the processed image data and pulse data are transmitted in the time sequence of the timestamps through two transmission channels, which can avoid data transmission errors; after the transmission is completed, the discarded processed image data is restored according to the processed image data and pulse data obtained by the transmission, thereby ensuring the integrity of the processed image data, and solving the technical problem that there is a large amount of communication information redundancy in the existing channel transmission of image data and pulse data, resulting in waste of resources and increased transmission costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 A flowchart of an image processing method provided by an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of parallel transmission of original image data and pulse data provided by an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of parallel transmission of image data and pulse data after clipping and discarding provided by an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of restoring discarded processed image data provided by an embodiment of the present invention;

[0044] Figure 5 A structural block diagram of an image processing system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The embodiments of the present invention provide an image processing method and system for solving the technical problem that a large amount of communication information redundancy exists in the existing channel transmission of image data and pulse data, resulting in waste of resources and increased transmission cost.

[0046] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] See also Figure 1 The first aspect of the present invention provides an image processing method, the method comprising:

[0048] Step 101, obtaining M original image data and pulse data.

[0049] It should be noted that M raw image data can be obtained through the camera of the neuromorphic camera according to a preset shooting interval, and then the M raw image data can be converted into M pulse data (pulse sequence) using the frame difference method using the pulse sensing module of the neuromorphic camera; specifically, the frame difference method is used to process the images continuously captured by the camera, and the grayscale value difference of each pixel point of the original image is calculated to generate pulse data.

[0050] In particular, it is necessary to record the time stamp of the original image data and the corresponding pulse data when they were taken.

[0051] Step 102, performing discard preprocessing on the M original image data to obtain N processed image data; wherein M and N are both positive integers, and M>N.

[0052] In order to reduce the transmission redundancy of information, this step crops and discards the original image data to retain only the key data, thereby improving the transmission rate of large-scale image transmission, avoiding resource waste and reducing transmission costs.

[0053] Specifically, this step includes the following sub-steps:

[0054] Sub-step 1021, comparing the overlap between the original image data and the pulse data, and cropping the original image data.

[0055] It should be noted that in actual transmission, it is found that there is a lot of overlap between the pulse data and the original image data, resulting in information redundancy, which is not conducive to large-scale image transmission. Therefore, the original image data can be cropped to a certain extent based on the overlap between the original image data and the pulse data.

[0056] Sub-step 1022, obtaining N processed image data by discarding the M cropped image data.

[0057] In this sub-step, firstly, based on the timestamps of the M cropped image data (i.e., the M processed image data), the M cropped image data are sorted; then, according to a preset discarding interval number, the sorted M cropped image data are grouped to obtain multiple groups of image data; finally, the first cropped image data in each group of image data is retained, and the remaining cropped image data are discarded to obtain N processed image data.

[0058] Exemplarily, assuming that M is 99 and the preset discarding interval number is 4, the preset discarding interval number plus one is used as the number of each group of image data (i.e., 5). Then, a total of 19 groups of image data can be divided, of which the 1st to 18th groups each include five cropped image data, and the last group includes four cropped image data, so 19 processed image data can be obtained.

[0059] It is understandable that the preset discard interval number can be adjusted according to the importance of the image data and the real-time status of the subsequent transmission channel (such as the data capacity of the transmission channel); in addition, the preset discard interval number cannot be set too large, because the external environment will cause noise interference to the transmission channel, which will have a certain degree of impact on the processing of image data and pulse data, and will affect the quality of subsequent image data reconstruction.

[0060] Step 103 , according to the time sequence of the timestamps, the processed image data and the pulse data are respectively transmitted in parallel and intermittently through two transmission channels.

[0061] It should be noted that the image data and pulse data are transmitted and processed through two transmission channels. Figure 3 , the image data and pulse data are transmitted intermittently according to the time sequence of the timestamps, or in other words, the processed image data is aligned with the pulse data according to the timestamps. Since the pulse data is complete and the processed image data is discarded, the accuracy of the subsequent processed image data and pulse data transmission can be guaranteed.

[0062] For example, Figure 2 and Figure 3 As shown:

[0063] Figure 2The case where the original image data and the pulse data without discarding preprocessing are transmitted in parallel through the channel is shown; Figure 2 In the process, since the original image data is not discarded preprocessed, there is a lot of communication information redundancy on its transmission channel. For example, the timestamps of pulse data 1 and original image data 1 are the same. During transmission, pulse data 1 and original image data 1 are transmitted in parallel through transmission channel 1 and transmission channel 2 respectively. is the transmission interval, then after After that, pulse data 2 and original image data 2 are transmitted simultaneously through transmission channel 1 and transmission channel 2 respectively, and so on. However, compared with pulse data, original image data carries more data information, and has higher requirements on the bandwidth and transmission rate of the transmission channel, and there is a lot of communication information redundancy on the transmission channel.

[0064] Based on this, the present invention has made further improvements, such as Figure 3 As shown, Figure 3 It shows the case where the original image data and pulse data with discard preprocessing are transmitted in parallel through the channel; Figure 3 In the process, the original image data is firstly cropped to obtain the processed image data. There are originally four processed image data between processed image data 1 and processed image data 6. In order to alleviate the transmission pressure of the image transmission channel, the preset discarding interval number is 4 as an example. At this time, the processed image data between processed image data 1 and processed image data 6 are discarded, thereby effectively reducing the amount of image data in the subsequent image transmission process.

[0065] During transmission, pulse data 1 and processed image data 1 are transmitted in parallel through transmission channel 1 and transmission channel 2 respectively. is the transmission interval, in 5 During this period, pulse data 2-5 are transmitted in sequence through transmission channel 1, and after 5 After that, the pulse data 6 and the processed image data 6 are respectively transmitted in parallel through the transmission channel 1 and the transmission channel 2. The present invention reduces the data volume of the image transmission channel and improves the transmission efficiency by cutting and discarding the original image data, while retaining enough processed image data to meet the application requirements. In addition, the transmission interval can be adjusted according to the transmission pressure of the actual channel.

[0066] Step 104: restore the discarded MN processed image data according to the transmitted processed image data and pulse data.

[0067] It is understandable that in order not to affect the subsequent comparison results of the pulse neural network on the processed image data and the pulse result data, it is necessary to ensure the integrity and quality of the received processed image data. Therefore, in this step, it is necessary to restore the discarded MN processed image data to avoid information loss of the processed image data and affect the subsequent discrimination results.

[0068] Specifically, if only pulse data is received in the current time step, the processed image data received in the previous time step is added to the pulse data, and restored and saved as the processed image data of the current time step, so that the processed image data discarded in step 103 can be restored.

[0069] For example, Figure 4 As shown, assuming that only pulse data 2 is received in the current time step, the processed image data 1 received in the previous time step is added to the pulse data 2 to restore the processed image data 2 of the current time step. Similarly, if only pulse data 3 is still received in the next time step, the processed image data 2 restored in the previous time step is added to the pulse data 3 of the next time step to restore the processed image data 3 of the next time step.

[0070] Among them, the above-mentioned addition operation specifically includes: converting the pulse data into a numerical matrix of the same dimension as the processed image data, and then adding elements at each pixel position of the processed image data, that is, adding the corresponding value in the numerical matrix of the pulse data to the pixel point of the processed image data, thereby restoring the processed image data.

[0071] It is understandable that in order to reduce the noise interference generated during the restoration process, the restored processed image data can be further optimized with the help of reconstruction technology, such as refining the image details of the processed image data through sparse coding or generative adversarial networks.

[0072] Step 105, inputting the pulse data into the pulse neural network for pulse processing to obtain pulse result data; comparing and distinguishing the processed image data with the pulse result data to obtain a distinction result to adjust the parameters of the pulse neural network.

[0073] It should be noted that the pulse neural network includes a coding layer, which performs Poisson coding on the gray value difference data of the pulse data in the form of frequency, thereby completing the conversion from numerical value to pulse form.

[0074] In addition, the pulse neural network also includes a discriminator, which divides the discrimination results of the processed image data and the pulse result data into light frost, moderate frost and heavy frost. Specifically, the discriminator selects the processed image data as a data sample for training the set threshold of the discriminator, that is, two thresholds K1 and K2; when the frost calculation value of the pulse result data is less than K1, it is considered to be light frost, the frost calculation value of the pulse result data is greater than K1 and less than K2, it is considered to be moderate frost, and the frost calculation value of the pulse result data is greater than K2, it is considered to be heavy frost; the parameters of the pulse neural network are adjusted according to different degrees of frost, so that the effectiveness of the pulse neural network can be verified, thereby ensuring the accuracy of the pulse neural network processing data.

[0075] See also Figure 5 The second aspect of the present invention further provides an image processing system, the system comprising:

[0076] An acquisition module 201 is used to acquire M original image data and pulse data;

[0077] A preprocessing module 202 is used to discard and preprocess the M original image data to obtain N processed image data; wherein M and N are both positive integers, and M>N;

[0078] A transmission module 203, configured to transmit the processed image data and the pulse data in parallel and intermittently through two transmission channels respectively according to the time sequence of the timestamps;

[0079] A recovery module 204, configured to recover the discarded processed image data based on the transmitted processed image data and pulse data;

[0080] The pulse processing module (pulse sensing module) 205 is used to input the pulse data into the pulse neural network for pulse processing to obtain pulse result data; compare and distinguish the processed image data with the pulse result data to obtain the judgment result to adjust the parameters of the pulse neural network.

[0081] It should be noted that the image processing system includes a neuromorphic camera and an image storage module 206, and the neuromorphic camera includes an acquisition module, a preprocessing module and a transmission module; wherein the preprocessing module can be connected to the transmission module through a mipi interface; the image storage module is connected to the recovery module for storing and processing image data; at the same time, the pulse processing module is connected to the image storage module.

[0082] More specifically, the acquisition module 201 includes:

[0083] The camera submodule is used to obtain M raw image data through a camera according to a preset shooting interval;

[0084] The conversion submodule is used to convert M original image data into M pulse data by using a frame difference method.

[0085] More specifically, the pre-processing module 202 includes:

[0086] A cropping submodule, used to compare the coincidence between the original image data and the pulse data, and to crop the original image data;

[0087] The discarding submodule is used to obtain N processed image data by discarding the M cropped image data.

[0088] More specifically, the discarding submodule is specifically used to sort the M cropped image data based on the timestamps of the M cropped image data; group the sorted M cropped image data according to a preset discarding interval number to obtain multiple groups of image data; retain the first cropped image data in each group of image data, and discard the remaining cropped image data to obtain N processed image data.

[0089] More specifically, the recovery module 204 is specifically used to, if only pulse data is received in the current time step, add the processed image data received in the previous time step to the pulse data, and recover and save them as the processed image data of the current time step.

[0090] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0091] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0092] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0093] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0094] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0095] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire M raw image data and pulse data; Performing a discard preprocessing on the M original image data to obtain N processed image data; wherein M and N are both positive integers, and M>N; According to the time sequence of the timestamps, the processed image data and the pulse data are respectively transmitted in parallel and intermittently through two transmission channels; restoring the discarded processed image data according to the transmitted processed image data and the pulse data; The pulse data is input into the pulse neural network for pulse processing to obtain pulse result data; the processed image data is compared and judged with the pulse result data to obtain a judgment result to adjust the parameters of the pulse neural network.

2. The image processing method according to claim 1, characterized in that: The step of acquiring M original image data and pulse data comprises: Acquire M raw image data through the camera according to a preset shooting interval; The M original image data are converted into M pulse data by using a frame difference method.

3. The image processing method according to claim 1, characterized in that: The step of discarding the M original image data to obtain N processed image data comprises: Comparing the overlap between the original image data and the pulse data, and cropping the original image data; By discarding the M cropped image data, N processed image data are obtained.

4. The image processing method according to claim 3, characterized in that: The step of discarding the M cropped image data to obtain N processed image data includes: sorting the M cropped image data based on timestamps of the M cropped image data; According to a preset discarding interval number, the sorted M cropped image data are grouped to obtain multiple groups of image data; The first cropped image data in each group of image data is retained, and the remaining cropped image data are discarded, so as to obtain N processed image data.

5. The image processing method according to claim 4, characterized in that: The step of restoring the discarded processed image data based on the processed image data and the pulse data obtained through transmission includes: if only pulse data is received in the current time step, adding the processed image data received in the previous time step to the pulse data, and restoring and saving them as the processed image data of the current time step.

6. An image processing system, characterized in that: The system comprises: An acquisition module, used for acquiring M original image data and pulse data; A preprocessing module, used for discarding and preprocessing the M original image data to obtain N processed image data; wherein M and N are both positive integers, and M>N; A transmission module, used for transmitting the processed image data and the pulse data in parallel and intermittently through two transmission channels respectively according to the time sequence of the timestamps; A recovery module, used for recovering the discarded processed image data according to the transmitted processed image data and the pulse data; The pulse processing module is used to input the pulse data into the pulse neural network for pulse processing to obtain pulse result data; compare and distinguish the processed image data with the pulse result data to obtain the distinction result to adjust the parameters of the pulse neural network.

7. The image processing system according to claim 6, characterized in that: The acquisition module comprises: The camera submodule is used to obtain M raw image data through a camera according to a preset shooting interval; The conversion submodule is used to convert the M original image data into M pulse data by using a frame difference method.

8. The image processing system according to claim 6, characterized in that: The preprocessing module comprises: A cropping submodule, used for comparing the overlap between the original image data and the pulse data, and cropping the original image data; The discarding submodule is used to obtain N processed image data by discarding the M cropped image data.

9. The image processing system according to claim 8, characterized in that: The discarding submodule is specifically used to sort the M cropped image data based on the timestamps of the M cropped image data; group the sorted M cropped image data according to a preset discarding interval number to obtain multiple groups of image data; retain the first cropped image data in each group of image data, and discard the remaining cropped image data to obtain N processed image data.

10. The image processing system according to claim 6, characterized in that: The recovery module is specifically used to, if only pulse data is received in the current time step, add the processed image data received in the previous time step to the pulse data, and recover and save them as the processed image data of the current time step.

Images