A computing perception method, system, device and storage medium

Abstract

The present invention relates to a computational perception method, system, device and storage medium, the method comprising: designing and training a coding-decoding deep neural network with a weight contribution calculation module in a coding subnetwork, and obtaining a coding modulation mask sequence corresponding to the coding-decoding deep neural network; selecting the first N coding modulation masks in the coding modulation mask sequence; using the selected mask to code-modulate a target image, and using a single-pixel detector to collect coupled data after coding modulation to obtain single-pixel measurement data corresponding to different codes; inputting the measurement data into a decoding subnetwork of a pre-selected trained coding-decoding deep neural network to obtain an image perception result of the target image. The technical solution provided by the present invention can improve the accuracy of image reconstruction and semantic perception by performing image reconstruction and semantic perception of the target image based on the measurement data obtained from the selected coding modulation mask.

Description

Technical Field

[0001] The present invention relates to the field of image computing perception, and in particular to a computing perception method, system, device and storage medium. Background Art

[0002] With the development of deep learning technology and the improvement of hardware platform computing power, machine vision has gradually replaced human decision-making in various aspects of life and realized automated production in many industries. However, with the continuous progress of society, simple tasks such as low-level feature extraction and feature matching can no longer meet people's daily needs. Therefore, with the development of optics and computer fields, the work of machine vision is increasingly biased towards large-scale, high-resolution image task processing. Although these changes are in line with the development of society, they bring greater computational complexity and algorithm difficulty, and greatly increase the demand for hardware platforms such as imaging devices and computing resources. In specific situations such as small information collection and low computing power of existing platforms, we often do not need or have the conditions to adopt these existing large-scale frameworks to achieve perception tasks.

[0003] In the development history of machine vision, since the 21st century, compressed sensing methods have attracted widespread attention due to their advantages in storage and transmission, such as good encryption and extremely low information bandwidth. Subsequently, people have proposed a variety of sparse imaging algorithms. Its basic principle is to use the sparse characteristics of the signal, randomly sample the discrete samples of the signal under the condition of far lower than the Nyquist sampling rate, and then use a variety of algorithms based on one-dimensional coupled data to obtain reconstructed images or high-level semantic perception results. Traditional solution methods include gradient descent, conjugate gradient descent, alternating projection and other algorithms. For example, Zibang Zhang et al. modulated the scene using a sinusoidal mask and used the inverse Fourier transform to transform the obtained spectrum to achieve image reconstruction. With the development of artificial intelligence technology, network learning methods have also been introduced into the reconstruction of sparse data. Yang et al. proposed a new deep learning architecture ADMM CSNet for high-quality image reconstruction under sparse sampling. Palangi et al. based on the inter-channel dependency of sparse signals, calculated the conditional probability of each entry in each vector, and achieved high-performance reconstruction tasks based on the LSTM network. However, in traditional perception methods, people often use traditional methods such as Hadamard mask, sinusoidal mask, random binary mask, etc. to design masks to modulate scene light. However, in complex and changeable scenes in reality, these traditional coded modulation masks with fixed values ​​often have difficulty in extracting target information well, making it difficult to further obtain reconstructed images or high-level semantic perception results.

[0004] In view of the above shortcomings, in order to optimize light modulation, Suhas Lohit et al. proposed to use the coding modulation mask as part of the neural network through training to obtain the optimized coding modulation mask. Hao Fu et al. proposed the perception architecture of "coding network-decoding network", which regards the mask modulation process as the convolution layer of the neural network, and the value of the mask is the weight of the convolution layer. Although this network-based method can better extract scene information and achieve higher-precision perception, it is necessary to adjust some network parameters and retrain the network for tasks at different sampling rates. The steps are complicated and redundant, and the calculation is large and time-consuming. Summary of the invention

[0005] The present application provides a computational perception method, system, device and storage medium to at least solve the technical problems in the related art of inaccurate image reconstruction and semantic perception, large amount of calculation and long time consumption.

[0006] The first aspect of the present application provides a computing perception method, the method comprising:

[0007] Design and train a coding-decoding deep neural network with a weight contribution calculation module in the coding subnetwork, and obtain a coding modulation mask sequence corresponding to the coding-decoding deep neural network;

[0008] Selecting first N coding modulation masks in the coding modulation mask sequence from the pre-acquired coding modulation mask sequence based on the perception requirements of the target image;

[0009] The target image is coded and modulated using the selected mask, and coupled data after coding and modulation is collected using a single-pixel detector to obtain single-pixel measurement data corresponding to different codes;

[0010] The measurement data is input into a decoding subnetwork of a pre-selected trained encoding-decoding deep neural network to obtain an image perception result of the target image.

[0011] A second aspect of the present application provides a computing perception system, characterized in that the system includes:

[0012] Design a training module for designing and training a coding-decoding deep neural network having a weight contribution calculation module in the coding subnetwork, and obtaining a coding modulation mask sequence corresponding to the coding-decoding deep neural network;

[0013] A selection module, configured to select the first N coding modulation masks in the coding modulation mask sequence from the pre-acquired coding modulation mask sequence based on the perception requirements of the target image;

[0014] A measurement data module, used to code and modulate the target image using the selected mask, and collect the coupled data after coding and modulation using a single-pixel detector to obtain single-pixel measurement data corresponding to different codes;

[0015] The perception module is used to input the measurement data into a decoding subnetwork of a pre-selected trained encoding-decoding deep neural network to obtain an image perception result of the target image.

[0016] The third aspect of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the prediction method of the first aspect of the present application is implemented.

[0017] The fourth aspect of the present application provides a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the prediction method of the first aspect of the present application is implemented.

[0018] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0019] The present invention provides a computational perception method, system, device and storage medium, the method comprising: designing and training a coding-decoding deep neural network with a weight contribution calculation module in a coding subnetwork, and obtaining a coding modulation mask sequence corresponding to the coding-decoding deep neural network; selecting the first N coding modulation masks in the coding modulation mask sequence; using the selected mask to code-modulate the target image, and using a single-pixel detector to collect the coupled data after coding modulation to obtain single-pixel measurement data corresponding to different codes; inputting the measurement data into a decoding subnetwork of a pre-selected trained coding-decoding deep neural network to obtain an image perception result of the target image. The technical solution provided by the present invention can improve the accuracy of image reconstruction and semantic perception by performing image reconstruction and semantic perception based on the measurement data obtained from the selected coding modulation mask, while reducing the amount of calculation and saving time.

[0020] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0022] Figure 1 is a flow chart of a computational perception method provided according to an embodiment of the present application;

[0023] Figure 2 It is a structural diagram of a coding-decoding deep neural network in a computational perception method provided according to an embodiment of the present application;

[0024] Figure 3 It is a diagram of a mask sequence sorting and extraction process in a computational perception method provided according to an embodiment of the present application;

[0025] Figure 4 It is a structural diagram of a computing perception system provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0027] The present application proposes a computational perception method, system, device and storage medium, the method comprising: designing and training a coding-decoding deep neural network with a weight contribution calculation module in a coding subnetwork, and obtaining a coding modulation mask sequence corresponding to the coding-decoding deep neural network; selecting the first N coding modulation masks in the coding modulation mask sequence; using the selected mask to code-modulate the target image, and using a single-pixel detector to collect the coupled data after coding modulation to obtain single-pixel measurement data corresponding to different codes; inputting the measurement data into the decoding subnetwork of the pre-selected trained coding-decoding deep neural network to obtain the image perception result of the target image. The technical solution provided by the present invention can improve the accuracy of image reconstruction and semantic perception by performing image reconstruction and semantic perception based on the measurement data obtained from the selected coding modulation mask, while the amount of calculation is small, saving time and calculation time.

[0028] Example 1

[0029] Figure 1 A flowchart of a computing perception method provided by an embodiment of the present disclosure, such as Figure 1 As shown, the method includes:

[0030] Step 1: Design and train a coding-decoding deep neural network with a weight contribution calculation module in the coding subnetwork, and obtain a coding modulation mask sequence corresponding to the coding-decoding deep neural network.

[0031] It should be noted that the encoding subnetwork is used to obtain the spatial light coding modulation mask, the weight contribution calculation module is used to obtain the contribution of different coding modulation masks to the perception accuracy, and the decoding subnetwork is used to obtain the reconstructed image or high-level semantic perception results from the modulated acquired data.

[0032] In an embodiment of the present disclosure, the training process of the encoding subnetwork in the encoding-decoding deep neural network includes:

[0033] Using a pre-acquired image set and a gradient descent algorithm, the encoding subnetwork in the encoding-decoding deep neural network is iteratively trained until the encoding subnetwork converges, thereby obtaining a trained encoding subnetwork in the encoding-decoding deep neural network;

[0034] Among them, the structural design of the encoding subnetwork in the encoding-decoding deep neural network includes: convolutional layer, fully connected layer and activation layer.

[0035] For example, the encoding part uses a coded modulation mask to encode and encrypt the scene information, and the encoding network is used to learn an optimal mask sequence so that it can extract scene feature information with a greater probability, thereby facilitating subsequent high-performance reconstruction or semantic perception tasks based on single-pixel detection values. The structural design of the encoding network includes but is not limited to convolutional layers, fully connected layers, activation layers, etc.; wherein the scene information includes: classification problems of handwritten digits, clothing classification, face recognition, animal recognition, etc.

[0036] In an embodiment of the present disclosure, the training process of the decoding subnetwork in the pre-trained encoding-decoding deep neural network includes:

[0037] Using the obtained single-pixel measurement data of different codes and using a gradient descent algorithm, iteratively train a decoding subnetwork in a coding-decoding deep neural network until the decoding subnetwork converges, thereby obtaining a trained decoding subnetwork in the coding-decoding deep neural network;

[0038] In the embodiments of the present disclosure, the structural design of the decoding subnetwork in the encoding-decoding deep neural network includes but is not limited to: a neural network based on an ultra-deep convolutional layer or a neural network based on a composite model scaling method.

[0039] For example, the input of the decoding subnetwork is the one-dimensional measurement value modulated by the encoding subnetwork, and the output is the semantic perception or reconstruction result. The loss function is consistent with the encoding subnetwork, and the network is basically a gradient descent algorithm; the decoding subnetwork is mainly used to realize visual tasks such as image reconstruction and semantic perception from one-dimensional coupling information. The decoding subnetwork can be based on the following structures: neural network based on ultra-deep convolutional layer (VGG), neural network based on composite model scaling method (EfficientNet), etc. Among them, (a) neural network based on ultra-deep convolutional layer (VGG), VGG network is a very deep neural network, and uses multiple very small convolution kernels in the whole network and combines them with the input on each pixel. This network can be applied to a wide range of tasks and data sets, and its performance may be better than that of image processing methods built on lower depth networks. (b) neural network based on composite model scaling method (EfficientNet), EfficientNet is a recognition network based on a multi-dimensional mixed model scaling method. This network takes into account both speed and accuracy, and is also the best network framework that meets the three measurement indicators of network depth, network width, and image resolution.

[0040] It should be noted that in this step, according to the task requirements, it is necessary to extract the number at the corresponding sampling rate from the sorted mask sequence, use it as the weight value of the convolution layer, and fix the convolution layer weight, and continue to train the subsequent network until convergence. Finally, the network will output the imaging-free perception result of the target scene under the mask modulation optimized by our network.

[0041] In the disclosed embodiment, the weight contribution calculation module is used to obtain the contribution degree of different coding modulation masks to the perception accuracy;

[0042] Among them, the network design of the weight contribution calculation module includes but is not limited to: SENet structure.

[0043] For example, by introducing a weight contribution calculation module in the coding subnetwork, the intrinsic connection between each channel can be learned, and the contribution of different convolution channels to the network can be analyzed, so as to calculate the weight of each coding modulation mask. The network design of the weight contribution calculation module includes but is not limited to SENet, etc. SENet is short for "Squeeze-and-Excitation". This architecture focuses on the relationship between channels. It can selectively emphasize information features based on global information and suppress less useful features. The architecture automatically adjusts the feature response of the channel by learning the degree of mutual dependence between channels. In the disclosed embodiment, we can use special structures such as SENet to learn the relationship between each mask, so as to obtain the contribution weights of different masks in the feature extraction process.

[0044] It should be noted that based on the above encoding subnetwork, weight contribution calculation module and decoding subnetwork, the designed encoding-decoding deep neural network is as follows Figure 2 As shown, the weight calculation module in the figure is the weight contribution calculation module in the article.

[0045] In an embodiment of the present disclosure, the step of obtaining a coding modulation mask sequence corresponding to the coding-decoding deep neural network includes:

[0046] The weight contribution calculation module is used to obtain the contribution of different coding modulation masks to the perception accuracy;

[0047] The contribution of the different coding and modulation masks to the perception accuracy is arranged in descending order to form a coding and modulation mask sequence corresponding to the coding-decoding deep neural network.

[0048] For example, Figure 3 As shown, the specific values ​​of each mask and the weight, i.e., contribution, of each coding modulation mask are extracted and rearranged in descending order of contribution.

[0049] Step 2: Select the first N coding modulation masks in the coding modulation mask sequence from the pre-acquired coding modulation mask sequence based on the perception requirements of the target image.

[0050] In the disclosed embodiment, the number of the N coding modulation masks is determined based on the sampling rate of the target image.

[0051] For example, for a 28*28 image, the first step of the encoding-decoding network training is the mask with a sampling rate of 1, that is, the number is 28*28*1=784. We extract the network parameters to obtain the best 784 masks and their corresponding contribution weights (the value of the mask is the convolution kernel of the convolution layer of the encoding subnetwork, and the weight is the parameter in the weight calculation module such as SENet), and sort the masks from large to small according to the weight. Corresponding to the physical modulation process, for example, if the required sampling rate is 0.1, only 28*28*0.1=78 masks with the largest weight need to be extracted to achieve efficient semantic perception or reconstruction.

[0052] Step 3: Code-modulate the target image using the selected mask, and use a single-pixel detector to collect the coupled data after the code modulation to obtain single-pixel measurement data corresponding to different codes.

[0053] Step 4: Input the measurement data into a decoding subnetwork of a pre-selected trained encoding-decoding deep neural network to obtain an image perception result of the target image.

[0054] In summary, the computational perception method provided by the embodiment of the present disclosure performs image reconstruction and semantic perception of the target image based on the measurement data obtained by the selected coding modulation mask, which can improve the accuracy of image reconstruction and semantic perception, while reducing the amount of calculation and saving calculation time.

[0055] Example 2

[0056] Figure 4 A structural diagram of a computing perception system provided by an embodiment of the present disclosure, such as Figure 4 As shown, the system comprises:

[0057] Design a training module for designing and training a coding-decoding deep neural network having a weight contribution calculation module in the coding subnetwork, and obtaining a coding modulation mask sequence corresponding to the coding-decoding deep neural network;

[0058] A selection module, configured to select the first N coding modulation masks in the coding modulation mask sequence from the pre-acquired coding modulation mask sequence based on the perception requirements of the target image;

[0059] A measurement data module, used to code and modulate the target image using the selected mask, and collect the coupled data after coding and modulation using a single-pixel detector to obtain single-pixel measurement data corresponding to different codes;

[0060] The perception module is used to input the measurement data into a decoding subnetwork of a pre-selected trained encoding-decoding deep neural network to obtain an image perception result of the target image.

[0061] In an embodiment of the present disclosure, the training process of the encoding subnetwork in the encoding-decoding deep neural network includes:

[0062] Using a pre-acquired image set and a gradient descent algorithm, the encoding subnetwork in the encoding-decoding deep neural network is iteratively trained until the encoding subnetwork converges, thereby obtaining a trained encoding subnetwork in the encoding-decoding deep neural network;

[0063] Among them, the structural design of the encoding subnetwork in the encoding-decoding deep neural network includes: convolutional layer, fully connected layer and activation layer.

[0064] In an embodiment of the present disclosure, the training process of the decoding subnetwork in the pre-trained encoding-decoding deep neural network includes:

[0065] Using the obtained single-pixel measurement data of different codes and using a gradient descent algorithm, iteratively train a decoding subnetwork in a coding-decoding deep neural network until the decoding subnetwork converges, thereby obtaining a trained decoding subnetwork in the coding-decoding deep neural network;

[0066] The structural design of the decoding subnetwork in the encoding-decoding deep neural network includes but is not limited to: a neural network based on an ultra-deep convolutional layer or a neural network based on a composite model scaling method.

[0067] In the disclosed embodiment, the weight contribution calculation module is used to obtain the contribution degree of different coding modulation masks to the perception accuracy;

[0068] Among them, the network design of the weight contribution calculation module includes but is not limited to: SENet structure.

[0069] In an embodiment of the present disclosure, the step of obtaining a coding modulation mask sequence corresponding to the coding-decoding deep neural network includes:

[0070] The weight contribution calculation module is used to obtain the contribution of different coding modulation masks to the perception accuracy;

[0071] The contribution of the different coding and modulation masks to the perception accuracy is arranged in descending order to form a coding and modulation mask sequence corresponding to the coding-decoding deep neural network.

[0072] In the disclosed embodiment, the number of the N coding modulation masks is determined based on the sampling rate of the target image.

[0073] In summary, a computational perception system provided by an embodiment of the present disclosure includes: a design training module, a selection module, a measurement data module and a perception module. The system performs image reconstruction and semantic perception of a target image based on the measurement data obtained by the selected coding modulation mask, which can improve the accuracy of image reconstruction and semantic perception, while having a small amount of calculation, saving calculation time.

[0074] Example 3

[0075] In order to implement the above embodiments, the present disclosure also provides a computer device.

[0076] The computer device provided in this embodiment includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method in Embodiment 1 is implemented.

[0077] Example 4

[0078] In order to implement the above embodiments, the present disclosure also proposes a non-transitory computer-readable storage medium.

[0079] The computer device provided in this embodiment stores a computer program, and when the computer program is executed by a processor, the method in Embodiment 1 is implemented.

[0080] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0081] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0082] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A computational perception method, characterized in that: The method comprises: Design and train a coding-decoding deep neural network with a weight contribution calculation module in the coding subnetwork, and obtain a coding modulation mask sequence corresponding to the coding-decoding deep neural network; Selecting first N coding modulation masks in the coding modulation mask sequence from the pre-acquired coding modulation mask sequence based on the perception requirements of the target image; The target image is coded and modulated using the selected mask, and coupled data after coding and modulation is collected using a single-pixel detector to obtain single-pixel measurement data corresponding to different codes; The measurement data is input into a decoding subnetwork of a pre-selected trained encoding-decoding deep neural network to obtain an image perception result of the target image.

2. The method according to claim 1, characterized in that The training process of the encoding subnetwork in the encoding-decoding deep neural network includes: Using a pre-acquired image set and a gradient descent algorithm, the encoding subnetwork in the encoding-decoding deep neural network is iteratively trained until the encoding subnetwork converges, thereby obtaining a trained encoding subnetwork in the encoding-decoding deep neural network; Among them, the structural design of the encoding subnetwork in the encoding-decoding deep neural network includes: convolutional layer, fully connected layer and activation layer.

3. The method according to claim 1, characterized in that The training process of the decoding subnetwork in the pre-trained encoding-decoding deep neural network includes: Using the obtained single-pixel measurement data of different codes and using a gradient descent algorithm, iteratively train a decoding subnetwork in a coding-decoding deep neural network until the decoding subnetwork converges, thereby obtaining a trained decoding subnetwork in the coding-decoding deep neural network; The structural design of the decoding subnetwork in the encoding-decoding deep neural network includes: a neural network based on an ultra-deep convolutional layer or a neural network based on a composite model scaling method.

4. The method according to claim 1, characterized in that The weight contribution calculation module is used to obtain the contribution of different coding modulation masks to the perception accuracy; Among them, the network design of the weight contribution calculation module includes: SENet structure.

5. The method according to claim 4, characterized in that The obtaining of a coding modulation mask sequence corresponding to the coding-decoding deep neural network includes: The weight contribution calculation module is used to obtain the contribution of different coding modulation masks to the perception accuracy; The contribution of the different coding and modulation masks to the perception accuracy is arranged in descending order to form a coding and modulation mask sequence corresponding to the coding-decoding deep neural network.

6. The method according to claim 1, characterized in that The number of the N coding modulation masks is determined based on the sampling rate of the target image.

7. A computing perception system, characterized in that: The system comprises: Design a training module for designing and training a coding-decoding deep neural network having a weight contribution calculation module in the coding subnetwork, and obtaining a coding modulation mask sequence corresponding to the coding-decoding deep neural network; A selection module, configured to select the first N coding modulation masks in the coding modulation mask sequence from the pre-acquired coding modulation mask sequence based on the perception requirements of the target image; A measurement data module, used to code and modulate the target image using the selected mask, and collect the coupled data after coding and modulation using a single-pixel detector to obtain single-pixel measurement data corresponding to different codes; The perception module is used to input the measurement data into a decoding subnetwork of a pre-selected trained encoding-decoding deep neural network to obtain an image perception result of the target image.

8. The system according to claim 7, characterized in that The number of the N coding modulation masks is determined based on the sampling rate of the target image.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images