Multispectral recognition device and method based on super-structure surface, medium and equipment

Abstract

The embodiment of the invention discloses a multispectral recognition device and method based on a metasurface, a medium and equipment, and the device comprises a sub-aperture imaging lens group which is used for collecting the multispectral information of a target object, carrying out the sub-aperture convergence, and outputting the sub-aperture convergent light in a parallel light form; the metasurface modulation device is used for generating a preset number of metasurface atoms through the micro-nano structure metasurface, performing multispectral modulation on the aperture-divided convergent light according to the metasurface atoms, and obtaining modulated multispectral information and unmodulated spectral information; and the charge-coupled device is used for collecting the modulated multispectral information and the unmodulated spectral information to obtain a multispectral image and a full-band image. And the target detection module is used for carrying out softmax normalization processing after carrying out feature fusion on the multispectral image and the full-band image, and outputting a final target identification result. By combining the super-structure surface multispectral imaging under the deep learning method, the imaging equipment is miniaturized, and the target recognition accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of spectral imaging and target detection, and in particular to a multi-spectral recognition device, method, medium and equipment based on a metasurface. Background Art

[0002] With the development of anti-detection camouflage technology, traditional single-band imaging detection methods have become increasingly difficult to meet the detection needs of camouflaged targets in complex environments. Therefore, it is urgent to carry out research on multi-spectral target detection technology. By analyzing the differences in characteristic radiation between various targets and backgrounds in different spectral bands, all-day detection of camouflaged targets in complex environments can be achieved.

[0003] Multispectral imaging technology uses the differences in spectral characteristics between the target and the background and artificial interference to distinguish between real targets, background and artificial interference, which can greatly improve the target recognition ability. With the continuous development of multispectral technology, multispectral imaging technology can also effectively identify targets under the conditions of missing geometric or motion information of space targets. The reason is that each pixel of multispectral data provides spectral information of multiple narrow bands, which form a complete and continuous spectral curve. Different space targets have various differences in their spectral curves. Some information that cannot be detected in a wide band can be reflected in a specific band, thereby only realizing the recognition of specific targets. Summary of the invention

[0004] Based on this, it is necessary to propose a multispectral recognition device, method, medium and equipment based on metasurface to address the above problems.

[0005] A multispectral recognition device based on a metasurface, the device comprising:

[0006] The sub-aperture imaging lens group is used to collect multi-spectral information of the target object, and perform sub-aperture convergence on the multi-spectral information to output sub-aperture converged light in the form of parallel light.

[0007] A metasurface modulation device is used to generate a preset number of metasurface atoms through a micro-nanostructure metasurface, perform multi-spectral modulation on the sub-aperture focused light according to the preset number of metasurface atoms, and obtain modulated multi-spectral information and unmodulated spectral information.

[0008] The charge coupled device is used to collect the modulated multi-spectral information and the unmodulated spectral information to obtain the multi-spectral image and the full-band image.

[0009] The target detection module is used to perform feature fusion on the multispectral image and the full-band image, obtain a fusion pooling result, perform softmax normalization processing on the fusion pooling result, and output a final target recognition result.

[0010] The divided-aperture imaging lens group specifically includes: a beam reduction lens and a microlens array;

[0011] The beam reduction lens is used to collect multi-spectral information of the target object.

[0012] The microlens array is used to perform aperture convergence on the multi-spectral information and output aperture converged light in the form of parallel light.

[0013] A multispectral recognition method based on a metasurface, the method comprising:

[0014] The multispectral information of the target object is collected, and the multispectral information is converged to output the sub-aperture converged light in the form of parallel light.

[0015] A preset number of metasurface atoms are generated through a micro-nanostructure metasurface, and multi-spectral modulation is performed on the sub-aperture focused light according to the preset number of metasurface atoms to obtain modulated multi-spectral information and unmodulated spectral information.

[0016] The modulated multispectral information and the unmodulated spectral information are collected to obtain a multispectral image and a full-band image.

[0017] After feature fusion of the multispectral image and the full-band image, a fusion pooling result is obtained, and softmax normalization processing is performed on the fusion pooling result to output a final target recognition result.

[0018] The method of generating a preset number of metasurface atoms through a micro-nanostructure metasurface, performing multi-spectral modulation on the sub-aperture focused light according to the preset number of metasurface atoms, and obtaining modulated multi-spectral information and unmodulated spectral information specifically includes:

[0019] Randomly initialize the binary encoding vector.

[0020] A predicted spectrum vector is obtained according to the binary coding vector.

[0021] If it is determined that the predicted spectral vector satisfies a preset spectral condition, atomic shape parameters corresponding to the predicted spectral vector are obtained, and a preset number of metasurface atoms are determined according to the atomic shape parameters.

[0022] The sub-aperture focused light is subjected to multi-spectral modulation according to the preset number of metasurface atoms to obtain modulated multi-spectral information and unmodulated spectral information.

[0023] The step of obtaining a predicted spectrum vector according to the binary coding vector specifically includes:

[0024] The binary encoded vector is decoded into a corresponding shape vector and a parameter vector.

[0025] The shape vector obtains a mean vector and a variance vector according to the first three-layer convolution layer and the first linear rectification function activation layer.

[0026] The mean vector and the variance vector are added, and the atomic shape image is obtained through the second three-layer convolution layer and the second linear rectification function activation layer.

[0027] A predicted spectrum vector is obtained according to the parameter vector and the atomic shape image.

[0028] The step of obtaining a predicted spectral vector according to the parameter vector and the atomic shape image specifically includes:

[0029] After the parameter vector is fully connected, a first feature map is obtained through dimensionality transformation.

[0030] The atomic shape image is activated according to the third convolutional layer and the third linear rectification function layer to obtain a second feature map.

[0031] The first feature map and the second feature map are spliced ​​in the third dimension to obtain a feature splicing map.

[0032] The feature splicing graph obtains a feature fusion graph according to the fourth three-layer convolution layer and the fourth linear rectification function activation layer.

[0033] The feature fusion graph is fully connected to obtain a predicted spectrum vector.

[0034] The step of obtaining a predicted spectrum vector according to the binary coding vector further specifically includes:

[0035] If it is determined that the predicted spectrum vector does not satisfy the preset spectrum condition, the predicted spectrum vector is obtained again according to the binary code vector until the predicted spectrum vector satisfies the preset spectrum condition.

[0036] Among them, after the multispectral image and the full-band image are feature-fused, a fusion pooling result is obtained, the fusion pooling result is subjected to softmax normalization processing, and the final target recognition result is output, which specifically includes:

[0037] The multispectral images are stitched in the third dimension to obtain a three-dimensional multispectrum.

[0038] The three-dimensional multi-spectrum is subjected to feature fusion through a dual attention residual block to obtain a fused feature map.

[0039] The full-band image is feature fused through the dual attention residual block to obtain a full-band feature map.

[0040] The fused feature map and the full-band feature map are respectively encoded into key-value mapping pairs to obtain a fused feature value map, a fused feature key map, a full-band feature value map and a full-band feature key map.

[0041] The fused eigenvalue map and the full-band eigenvalue map are subjected to feature fusion to obtain a fused value map.

[0042] The fused feature key graph and the full-band feature key graph are subjected to feature fusion to obtain a fused key graph.

[0043] The fusion value map and the fusion key map are respectively subjected to softmax normalization, and the normalized fusion value map and fusion key map are summed to obtain the final feature fusion result.

[0044] Perform context-aware feature aggregation on the final feature fusion result, select a preset number of resolutions to perform parallel pooling operations, and obtain a corresponding preset number of pooling results.

[0045] The preset number of pooling results are fused through the attention mechanism to obtain a fused pooling result.

[0046] The fusion pooling result is subjected to softmax normalization processing, and the final target recognition result is output.

[0047] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the method described above.

[0048] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0049] The embodiments of the present invention have the following beneficial effects:

[0050] The present invention adopts a sub-aperture imaging lens group and a sub-aperture design, which can simultaneously collect multi-spectral information of the target object, capture richer spectral details and output them in the form of parallel light, providing high-quality input for the modulation of the subsequent metasurface modulation device. Furthermore, a preset number of metasurface atoms are generated by using a micro-nanostructure metasurface, and these metasurface atoms have the ability to accurately control light waves. By multi-spectral modulation of the sub-aperture converging light, not only the modulated multi-spectral information can be obtained, but also the unmodulated spectral information can be retained, thereby realizing flexible processing and subdivision modulation of the spectral information. The charge-coupled device accurately collects modulated multi-spectral information and unmodulated spectral information to generate multi-spectral images and full-band images. Finally, in the target detection module, the multi-spectral image and the full-band image are feature fused based on deep learning, the fusion pooling result is obtained, the fusion pooling result is softmax normalized, and the final target recognition result is output, thereby improving the target recognition accuracy. By combining the metasurface multi-spectral imaging under the deep learning method, the imaging device is miniaturized and the spectral features are used to improve the target recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] in:

[0053] Figure 1 A schematic structural diagram of an embodiment of a multi-spectral recognition device based on a metasurface provided by the present invention;

[0054] Figure 2 A schematic structural diagram of another embodiment of a multi-spectral recognition device based on a metasurface provided by the present invention;

[0055] Figure 3 A schematic diagram of a process flow of an embodiment of a method for manufacturing a micro-nanostructure super-surface provided by the present invention;

[0056] Figure 4 A schematic diagram of multi-spectral imaging of different regions on a target surface of a charge coupled device provided by the present invention;

[0057] Figure 5 A schematic diagram of the network structure of an embodiment of a target detection module provided by the present invention;

[0058] Figure 6 A schematic diagram of a process flow of an embodiment of a multispectral recognition method based on a metasurface provided by the present invention;

[0059] Figure 7 A schematic flow chart of another embodiment of a multi-spectral recognition method based on a metasurface provided by the present invention;

[0060] Figure 8 A schematic flow chart of another embodiment of a multi-spectral recognition method based on a metasurface provided by the present invention;

[0061] Fig. 9 A schematic structural diagram of an embodiment of the device provided by the present invention;

[0062] Fig.10 A schematic structural diagram of an embodiment of the medium provided by the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] like Figure 1 As shown, Figure 1 A schematic diagram of a structure of an embodiment of a multispectral recognition device based on a metasurface provided by the present invention. A multispectral recognition device based on a metasurface, the device comprising:

[0065] The sub-aperture imaging lens group 1 is used to collect multi-spectral information of the target object, and perform sub-aperture convergence on the multi-spectral information to output sub-aperture converged light in the form of parallel light.

[0066] For example, in combination with participation Figure 2 , Figure 2 A schematic diagram of the structure of another embodiment of a multispectral recognition device based on a metasurface provided by the present invention. The aperture-splitting imaging lens group 1 includes: a focus reduction lens 11 and a microlens array 12. The focus reduction lens 11 is used to collect multispectral information of a target object; the microlens array 12 is used to perform aperture-splitting convergence on the multispectral information and output aperture-splitting converged light in the form of parallel light.

[0067] It should be noted that the resolution of the microlens array 11 is a×b, where a is the number of rows and b is the number of columns. The microlens array 11 outputs a×b block areas of sub-aperture focused light.

[0068] The metasurface modulation device 2 is used to generate a preset number of metasurface atoms through a micro-nanostructure metasurface, perform multi-spectral modulation on the aperture-converging light according to the preset number of metasurface atoms, and obtain modulated multi-spectral information and unmodulated spectral information.

[0069] For example, the metasurface modulation device 2 uses a micro-nanostructured metasurface, the substrate material of which is SiO2, and the nanocolumn material is Si3N4. Figure 3 , Figure 3 A flow chart of an embodiment of a method for making a micro-nano structure super surface provided by the present invention. The method for making a micro-nano structure super surface is as follows: randomly initialize a 1×240 binary code vector; decode the randomly initialized binary code vector into a corresponding 1×220 shape vector and a 1×20 parameter vector through a trained decoder; obtain a 1×20 mean vector and a 1×20 variance vector from the 1×220 shape vector in a network composed of the first three convolution layers with a convolution kernel size of 3×3 per layer and the first linear rectification function activation layer; add the mean vector and the variance vector, and obtain a 64×64 atomic shape image through the second three convolution layers with a convolution kernel size of 3×3 per layer and the second linear rectification function activation layer.

[0070] Furthermore, the atomic shape image is input into the trained encoder to output the predicted spectral vector. Specifically, after the parameter vector with a dimension of 1×20 is input into the fully connected layer composed of dimensions of 1×64 and 1×4096, the first feature map with a dimension of 8×8×64 is obtained through dimension transformation; the atomic shape image with a dimension of 64×64 is obtained according to the third three-layer convolution layer with a convolution kernel size of 3×3 per layer and the third linear rectification function activation layer, and the second feature map with a dimension of 8×8×64 is obtained; the first feature map and the second feature map are spliced ​​in the third dimension to obtain a feature splicing map with a dimension of 8×8×128; the feature splicing map is obtained according to the fourth three-layer convolution layer and the fourth linear rectification function activation layer to obtain a feature fusion map; the feature fusion map is input into the fully connected layer composed of dimensions of 1×256 and 1×301 for full connection to obtain a predicted spectral vector with a dimension of 1×301.

[0071] Further, it is determined whether the output predicted spectral vector meets the preset spectral condition. If it is determined that the predicted spectral vector meets the preset spectral condition, the atomic shape parameters corresponding to the predicted spectral vector are obtained, and 24 metasurface atoms are determined according to the atomic shape parameters. If it is determined that the predicted spectral vector does not meet the preset spectral condition, the decoder is optimized, and the predicted spectral vector is obtained again according to the binary coding vector until the predicted spectral vector meets the preset spectral condition.

[0072] Furthermore, spectral modulation is performed through 24 metasurface atoms to achieve a×b-1 types of spectral modulation and obtain modulated multi-spectral information. Since spectral modulation will cause spectral information loss, the last channel is not spectrally modulated, and all light is passed through as unmodulated spectral information.

[0073] The charge coupled device 3 is used to collect modulated multi-spectral information and unmodulated spectral information to obtain multi-spectral images and full-band images.

[0074] Exemplarily, the resolution of the charge coupled device 3 is M×N, where M is the number of rows and N is the number of columns. The modulated multispectral information and the unmodulated spectral information are collected by the charge coupled device 3 to obtain a×b-1 multispectral images and a full-band image. Figure 4 , Figure 4 A schematic diagram of regional multispectral imaging on a charge coupled device target surface provided by the present invention, obtaining a×b-1 multispectral images and one full-band image, that is, a multispectral image of 24 channels and a full-band image of the last channel.

[0075] The target detection module 4 is used to perform feature fusion on the multispectral image and the full-band image, obtain the fusion pooling result, perform softmax normalization on the fusion pooling result, and output the final target recognition result.

[0076] For example, referring to Figure 5 , Figure 5 The network structure diagram of the target detection module of the present invention is shown in Figure 1. The a×b-1 multispectral images acquired by the charge coupled device 3 are spliced ​​in the third dimension to obtain a three-dimensional multispectral image. The size of the three-dimensional multispectral image is

[0077] Furthermore, the spliced ​​three-dimensional multi-spectrum is subjected to feature fusion through an image feature integration module composed of three dual attention residual blocks to obtain a fused feature map. The last full-band image acquired by the charge coupled device 3 is subjected to feature fusion through an image feature integration module composed of three dual attention residual blocks to obtain a full-band feature map. Among them, the full-band image size is

[0078] Furthermore, the fused feature map and the full-band feature map are respectively encoded into key-value mapping pairs F_v, F_k, G_v, G_k through a deep encoder, where F_v is a fused feature value map, F_k is a fused feature key map, G_v is a full-band feature value map, and G_k is a full-band feature key map.

[0079] Furthermore, by measuring the similarities and differences between the multispectral image features and the full-band image features, the features of the two are fused to obtain the fusion value graph and the fusion key graph, as shown in the following formula:

[0080] V=f(F_v,G_v)

[0081] K=f(F_k,G_k)

[0082] Among them, V is the fusion value graph and K is the fusion key graph.

[0083] Furthermore, the fusion value graph and the fusion key graph are respectively subjected to softmax normalization, and the normalized results are summed to obtain the final feature fusion result. The final feature fusion result is shown in the following formula:

[0084] F2=softmax(V)+softmax(K)

[0085] Among them, F2 is the final feature fusion result.

[0086] Furthermore, the final feature fusion result is passed through the context-aware feature aggregation module, and three resolutions of 4, 8, and 12 are selected for parallel pooling operations on the final feature fusion result to obtain three pooling results Favg_4, Favg_8, and Favg_12.

[0087] It should be noted that a larger resolution (such as 12) tends to focus on local information and can be used for small targets, while a smaller resolution (such as 4) captures the overall image information and is conducive to the recognition of larger targets.

[0088] Furthermore, the three pooling results are fused through the attention mechanism to obtain the fused pooling result. The fused pooling result is subjected to softmax normalization and the final target recognition result is output.

[0089] It should be noted that the present invention adopts a sub-aperture imaging lens group and a sub-aperture design, which can simultaneously collect multi-spectral information of the target object, capture richer spectral details and output them in the form of parallel light, providing high-quality input for the modulation of the subsequent metasurface modulation device. Furthermore, a preset number of metasurface atoms are generated by using a micro-nanostructure metasurface, and these metasurface atoms have the ability to accurately control light waves. By multi-spectral modulation of the sub-aperture converging light, not only the modulated multi-spectral information can be obtained, but also the unmodulated spectral information can be retained, thereby realizing flexible processing and subdivision modulation of the spectral information. The charge-coupled device accurately collects modulated multi-spectral information and unmodulated spectral information to generate multi-spectral images and full-band images. Finally, the target detection module performs feature fusion on the multi-spectral image and the full-band image based on deep learning, obtains the fusion pooling result, performs softmax normalization on the fusion pooling result, outputs the final target recognition result, and improves the target recognition accuracy. By combining the metasurface multi-spectral imaging under the deep learning method, the imaging device is miniaturized and the spectral features are used to improve the target recognition accuracy.

[0090] like Figure 6 As shown, Figure 6 A schematic diagram of a flow chart of an embodiment of a multispectral recognition method based on a metasurface provided by the present invention. A multispectral recognition method based on a metasurface, the method comprising:

[0091] S101: Collect multispectral information of a target object, and converge the multispectral information to output multi-aperture converged light in the form of parallel light.

[0092] Exemplarily, the multi-spectral information of the target object is collected by the sub-aperture imaging lens group 1, and the multi-spectral information is subjected to sub-aperture convergence to output the sub-aperture converged light in the form of parallel light. Specifically, the sub-aperture imaging lens group 1 includes: a focus reduction lens 11 and a microlens array 12. The multi-spectral information of the target object is collected by the focus reduction lens 11, and the multi-spectral information is subjected to sub-aperture convergence to output the sub-aperture converged light in the form of parallel light through the microlens array 12.

[0093] S102: Generate a preset number of metasurface atoms through a micro-nanostructure metasurface, perform multi-spectral modulation on the aperture-converging light according to the preset number of metasurface atoms, and obtain modulated multi-spectral information and unmodulated spectral information.

[0094] Exemplarily, a binary encoding vector is randomly initialized; a predicted spectral vector is obtained according to the binary encoding vector; if it is determined that the predicted spectral vector meets the preset spectral condition, the atomic shape parameters corresponding to the predicted spectral vector are obtained, and a preset number of metasurface atoms are determined according to the atomic shape parameters; multi-spectral modulation is performed on the aperture focusing light according to the preset number of metasurface atoms to obtain modulated multi-spectral information and unmodulated spectral information. If it is determined that the predicted spectral vector does not meet the preset spectral condition, the predicted spectral vector is obtained again according to the binary encoding vector until the predicted spectral vector meets the preset spectral condition.

[0095] Specifically, when obtaining the predicted spectral vector according to the binary coding vector, the binary coding vector is first decoded into the corresponding shape vector and parameter vector; the shape vector is activated by the first three-layer convolution layer and the first linear rectification function to obtain the mean vector and variance vector; the mean vector and the variance vector are added, and the atomic shape image is obtained through the second three-layer convolution layer and the second linear rectification function activation layer. After the parameter vector is fully connected, the first feature map is obtained through dimensional transformation; the atomic shape image is activated by the third three-layer convolution layer and the third linear rectification function activation layer to obtain the second feature map; the first feature map and the second feature map are spliced ​​in the third dimension to obtain the feature splicing map; the feature splicing map is activated by the fourth three-layer convolution layer and the fourth linear rectification function activation layer to obtain the feature fusion map; the feature fusion map is fully connected to obtain the predicted spectral vector.

[0096] S103: Collect modulated multispectral information and unmodulated spectral information to obtain a multispectral image and a full-band image.

[0097] Exemplarily, the modulated multi-spectral information and the unmodulated spectral information are collected by the charge coupled device 3 to obtain the multi-spectral image and the full-band image.

[0098] S104: After feature fusion of the multispectral image and the full-band image, a fusion pooling result is obtained, softmax normalization is performed on the fusion pooling result, and a final target recognition result is output.

[0099] Exemplarily, the multispectral image is spliced ​​in the third dimension to obtain a three-dimensional multispectrum; the three-dimensional multispectrum is feature fused through a dual attention residual block to obtain a fused feature map; the full-band image is feature fused through a dual attention residual block to obtain a full-band feature map; the fused feature map and the full-band feature map are respectively encoded into key-value mapping pairs to obtain a fused eigenvalue map, a fused feature key map, a full-band eigenvalue map and a full-band feature key map; the fused eigenvalue map and the full-band eigenvalue map are feature fused to obtain a fused value map; the fused feature key map and the full-band feature key map are feature fused The features are fused to obtain a fused key graph; the fused value graph and the fused key graph are respectively subjected to softmax normalization processing, and the normalized fused value graph and the fused key graph are summed to obtain the final feature fusion result; context-aware feature aggregation is performed on the final feature fusion result, and a preset number of resolutions are selected for parallel pooling operations to obtain the corresponding preset number of pooling results; the preset number of pooling results are fused through the attention mechanism to obtain a fused pooling result; the fused pooling result is subjected to softmax normalization processing, and the final target recognition result is output.

[0100] From the above description, it can be seen that the present invention simultaneously collects multi-spectral information of the target object, captures richer spectral details and outputs them in the form of parallel light. Furthermore, a preset number of metasurface atoms are generated by using a micro-nanostructure metasurface, and multi-spectral modulation is performed on the aperture focusing light through a preset number of metasurface atoms, which can not only obtain the modulated multi-spectral information, but also retain the unmodulated spectral information, thereby realizing flexible processing and subdivision modulation of the spectral information, and then collecting the modulated multi-spectral information and the unmodulated spectral information, generating multi-spectral images and full-band images, performing feature fusion on the multi-spectral images and the full-band images, obtaining the fusion pooling result, performing softmax normalization on the fusion pooling result, outputting the final target recognition result, and improving the target recognition accuracy.

[0101] like Figure 7 As shown, Figure 7 A schematic flow chart of another embodiment of a multispectral recognition method based on a metasurface provided by the present invention. A multispectral recognition method based on a metasurface, the method comprising:

[0102] S201: Collect multispectral information of the target object, and converge the multispectral information to output multi-aperture converged light in the form of parallel light.

[0103] It should be noted that step S201 Figure 6 This has been discussed in detail in the implementation scenario shown and will not be repeated here.

[0104] S202: Randomly initialize a binary encoding vector.

[0105] S203: Decode the binary coded vector into a corresponding shape vector and a parameter vector.

[0106] For example, referring to Figure 2 , a binary encoding vector with a randomly initialized dimension of 1×240 is decoded into a corresponding shape vector with a dimension of 1×220 and a parameter vector with a dimension of 1×20 through the trained decoder.

[0107] S204: The shape vector obtains a mean vector and a variance vector according to the first three convolutional layers and the first linear rectification function activation layer.

[0108] Exemplarily, a shape vector with a dimension of 1×220 is transformed into a network consisting of the first three convolutional layers with a convolution kernel size of 3×3 in each layer and the first linear rectification function activation layer to obtain a mean vector with a dimension of 1×20 and a variance vector with a dimension of 1×20.

[0109] S205: The mean vector and the variance vector are added, and the atomic shape image is obtained through the second three-layer convolution layer and the second linear rectification function activation layer.

[0110] Exemplarily, the mean vector and the variance vector are added, and through the second three convolution layers with each convolution kernel size of 3×3 and the second linear rectification function activation layer, an atomic shape image with a dimension of 64×64 is obtained.

[0111] S206: After the parameter vector is fully connected, a first feature map is obtained through dimensionality transformation.

[0112] S207: The atomic shape image is activated according to the third convolutional layer and the third linear rectification function layer to obtain a second feature map.

[0113] Exemplarily, after a parameter vector with a dimension of 1×20 is input into a fully connected layer composed of dimensions of 1×64 and 1×4096, a first feature map with a dimension of 8×8×64 is obtained through dimensional transformation; the atomic shape image with a dimension of 64×64 is converted into a second feature map with a dimension of 8×8×64 according to the third convolution layer with a convolution kernel size of 3×3 and the third linear rectification function activation layer.

[0114] S208: The first feature map and the second feature map are spliced ​​in a third dimension to obtain a feature splicing map.

[0115] S209: The feature concatenation map obtains a feature fusion map according to the fourth three-layer convolution layer and the fourth linear rectification function activation layer.

[0116] S210: Perform full connection on the feature fusion graph to obtain a predicted spectrum vector.

[0117] Exemplarily, the first feature map and the second feature map are spliced ​​in the third dimension to obtain a feature splicing map with a dimension of 8×8×128; the feature splicing map is activated according to the fourth convolutional layer and the fourth linear rectification function layer to obtain a feature fusion map; the feature fusion map is input into a fully connected layer composed of dimensions of 1×256 and 1×301 for full connection to obtain a predicted spectral vector with a dimension of 1×301.

[0118] S211: Determine whether the predicted spectrum vector meets a preset spectrum condition.

[0119] S2111: If it is determined that the predicted spectrum vector does not satisfy the preset spectrum condition, the predicted spectrum vector is obtained again according to the binary coding vector until the predicted spectrum vector satisfies the preset spectrum condition.

[0120] S2112: If it is determined that the predicted spectral vector meets a preset spectral condition, then the atomic shape parameters corresponding to the predicted spectral vector are obtained, and a preset number of metasurface atoms are determined according to the atomic shape parameters.

[0121] Exemplarily, it is determined whether the output predicted spectral vector meets the preset spectral condition. If it is determined that the predicted spectral vector meets the preset spectral condition, the atomic shape parameters corresponding to the predicted spectral vector are obtained, and 24 metasurface atoms are determined according to the atomic shape parameters. If it is determined that the predicted spectral vector does not meet the preset spectral condition, the decoder is optimized, and the predicted spectral vector is obtained again according to the binary coding vector until the predicted spectral vector meets the preset spectral condition.

[0122] S212: Perform multi-spectral modulation on the aperture-converging light according to a preset number of metasurface atoms to obtain modulated multi-spectral information and unmodulated spectral information.

[0123] For example, spectral modulation is performed by 24 metasurface atoms to achieve a×b-1 types of spectral modulation and obtain modulated multi-spectral information. Since spectral modulation will cause spectral information loss, the last channel is not spectrally modulated to achieve full light passing as unmodulated spectral information.

[0124] S213: Collect modulated multispectral information and unmodulated spectral information to obtain a multispectral image and a full-band image.

[0125] Exemplarily, the modulated multispectral information and the unmodulated spectral information are collected by the charge coupled device 3 to obtain a×b-1 multispectral images and one full-band image.

[0126] S214: After feature fusion of the multispectral image and the full-band image, a fusion pooling result is obtained, softmax normalization is performed on the fusion pooling result, and a final target recognition result is output.

[0127] It should be noted that step S214 Figure 6 This has been discussed in detail in the implementation scenario shown and will not be repeated here.

[0128] like Figure 8 As shown, Figure 8 A schematic flow chart of another embodiment of a multispectral recognition method based on a metasurface provided by the present invention. A multispectral recognition method based on a metasurface, the method comprising:

[0129] S301: Perform third-dimensional stitching on the multispectral images to obtain a three-dimensional multispectrum.

[0130] S302: The three-dimensional multi-spectrum is subjected to feature fusion through a dual attention residual block to obtain a fused feature map.

[0131] For example, referring to Figure 5 , the a×b-1 multispectral images acquired by the charge coupled device 3 are spliced ​​in the third dimension to obtain a three-dimensional multispectral. Among them, the size of the three-dimensional multispectral is The spliced ​​three-dimensional multi-spectrum is fused through an image feature integration module composed of three double attention residual blocks to obtain a fused feature map.

[0132] S303: The full-band image is feature fused through a dual attention residual block to obtain a full-band feature map.

[0133] Exemplarily, the last full-band image acquired by the charge coupled device 3 is subjected to feature fusion through an image feature integration module composed of three dual attention residual blocks to obtain a full-band feature map. Among them, the full-band image size is

[0134] S304: The fused feature map and the full-band feature map are respectively encoded into key-value mapping pairs to obtain a fused feature value map, a fused feature key map, a full-band feature value map, and a full-band feature key map.

[0135] Exemplarily, the fused feature map and the full-band feature map are respectively encoded into key-value mapping pairs F_v, F_k, G_v, G_k through a deep encoder, where F_v is a fused feature value map, F_k is a fused feature key map, G_v is a full-band feature value map, and G_k is a full-band feature key map.

[0136] S305: Fusing the eigenvalue map and the full-band eigenvalue map to perform feature fusion and obtain a fused value map.

[0137] S306: Fusing the feature key graph and the full-band feature key graph to perform feature fusion and obtain a fused key graph.

[0138] Exemplarily, by measuring the similarities and differences between the multispectral image features and the full-band image features, the features of the two are fused to obtain a fusion value graph and a fusion key graph, as shown in the following formula:

[0139] V=f(F_v,G_v)

[0140] K=f(F_k,G_k)

[0141] Among them, V is the fusion value graph and K is the fusion key graph.

[0142] S307: The fusion value map and the fusion key map are respectively subjected to softmax normalization processing, and the normalized fusion value map and fusion key map are summed to obtain the final feature fusion result.

[0143] Exemplarily, the fusion value graph and the fusion key graph are respectively subjected to softmax normalization, and the normalized results are summed to obtain the final feature fusion result. The final feature fusion result is shown in the following formula:

[0144] F2=softmax(V)+softmax(K)

[0145] Among them, F2 is the final feature fusion result.

[0146] S308: Perform context-aware feature aggregation on the final feature fusion result, select a preset number of resolutions to perform parallel pooling operations, and obtain a corresponding preset number of pooling results.

[0147] Exemplarily, the final feature fusion result is passed through a context-aware feature aggregation module, and three resolutions of 4, 8, and 12 are selected for parallel pooling operations on the final feature fusion result to obtain three pooling results Favg_4, Favg_8, and Favg_12.

[0148] S309: Fusing a preset number of pooling results through an attention mechanism to obtain a fused pooling result.

[0149] S310: Perform softmax normalization processing on the fusion pooling result and output the final target recognition result.

[0150] like Fig. 9 As shown, Fig. 9 The device 20 includes a memory 21 and a processor 22. The memory 21 stores a computer program, and the processor 22 executes the computer program when working to implement the following. Figure 6 , Figure 7 and Figure 8 The method shown.

[0151] The specific technical details of a multispectral recognition method based on a metasurface implemented when the terminal 20 executes a computer program have been discussed in detail in the aforementioned method steps, so they will not be repeated here.

[0152] like Fig.10 As shown, Fig.10 The structure diagram of an embodiment of the medium provided by the present invention is shown in FIG. The medium 30 stores at least one computer program 31, and the computer program 31 is executed by the processor 22 to implement the following Figure 6 , Figure 7 and Figure 8 In one embodiment, the medium 30 may be a storage chip, a hard disk, a mobile hard disk, a USB flash drive, an optical disk, or other readable and writable storage tools, or a server, etc.

[0153] The above describes specific embodiments of the present specification, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0154] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0155] The apparatus, device, non-volatile computer-readable storage medium and method provided in the embodiments of this specification correspond to each other, and therefore, the apparatus, device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the corresponding apparatus, device, and non-volatile computer storage medium will not be repeated here.

[0156] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0157] For the convenience of description, the above device is described by being divided into various units according to their functions and described separately. Of course, when implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware. It should be understood by those skilled in the art that this specification embodiment can be provided as a method, system, or computer program product. Therefore, this specification embodiment can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification embodiment can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0158] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0159] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0161] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0162] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0163] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0164] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0165] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0166] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0167] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A multispectral recognition device based on a metasurface, characterized in that: The device comprises: A sub-aperture imaging lens group is used to collect multi-spectral information of the target object, and perform sub-aperture convergence on the multi-spectral information to output sub-aperture converged light in the form of parallel light; A metasurface modulation device, used to generate a preset number of metasurface atoms through a micro-nanostructure metasurface, perform multi-spectral modulation on the sub-aperture focused light according to the preset number of metasurface atoms, and obtain modulated multi-spectral information and unmodulated spectral information; A charge coupled device, used for collecting the modulated multi-spectral information and the unmodulated spectral information, and obtaining a multi-spectral image and a full-band image; The target detection module is used to perform feature fusion on the multispectral image and the full-band image, obtain a fusion pooling result, perform softmax normalization processing on the fusion pooling result, and output a final target recognition result.

2. The multispectral recognition device based on metasurface according to claim 1, characterized in that: The sub-aperture imaging lens group specifically includes: a beam reduction lens and a microlens array; The beam reduction lens is used to collect multispectral information of the target object; The microlens array is used to perform aperture convergence on the multi-spectral information and output aperture converged light in the form of parallel light.

3. A multispectral recognition method based on metasurface, characterized in that: The method comprises: Collecting multispectral information of the target object, and converging the multispectral information to output aperture-converged light in the form of parallel light; Generate a preset number of metasurface atoms through a micro-nanostructure metasurface, perform multi-spectral modulation on the sub-aperture focused light according to the preset number of metasurface atoms, and obtain modulated multi-spectral information and unmodulated spectral information; Collecting the modulated multispectral information and the unmodulated spectral information to obtain a multispectral image and a full-band image; After feature fusion of the multispectral image and the full-band image, a fusion pooling result is obtained, and softmax normalization processing is performed on the fusion pooling result to output a final target recognition result.

4. The multispectral recognition method based on metasurface according to claim 3, characterized in that: The method of generating a preset number of metasurface atoms through a micro-nanostructure metasurface, performing multi-spectral modulation on the sub-aperture focused light according to the preset number of metasurface atoms, and obtaining modulated multi-spectral information and unmodulated spectral information specifically includes: Randomly initialize the binary encoding vector; Obtaining a predicted spectrum vector according to the binary coding vector; Determining that the predicted spectral vector satisfies a preset spectral condition, obtaining an atomic shape parameter corresponding to the predicted spectral vector, and determining a preset number of metasurface atoms according to the atomic shape parameter; The sub-aperture focused light is subjected to multi-spectral modulation according to the preset number of metasurface atoms to obtain modulated multi-spectral information and unmodulated spectral information.

5. The multispectral recognition method based on metasurface according to claim 4, characterized in that: The step of obtaining a predicted spectrum vector according to the binary coding vector specifically includes: Decoding the binary coded vector into a corresponding shape vector and a parameter vector; The shape vector obtains a mean vector and a variance vector according to the first three convolutional layers and the first linear rectification function activation layer; The mean vector and the variance vector are added, and the atomic shape image is obtained through the second three-layer convolution layer and the second linear rectification function activation layer; A predicted spectrum vector is obtained according to the parameter vector and the atomic shape image.

6. The multispectral recognition method based on metasurface according to claim 5, characterized in that: The step of obtaining a predicted spectral vector according to the parameter vector and the atomic shape image specifically includes: After the parameter vector is fully connected, a first feature map is obtained by dimensional transformation; The atomic shape image is activated according to a third convolutional layer and a third linear rectification function layer to obtain a second feature map; The first feature map and the second feature map are spliced ​​in a third dimension to obtain a feature splicing map; The feature splicing graph obtains a feature fusion graph according to the fourth three-layer convolution layer and the fourth linear rectification function activation layer; The feature fusion graph is fully connected to obtain a predicted spectrum vector.

7. A multispectral recognition method based on a metasurface according to any one of claims 4 to 6, characterized in that: The step of obtaining a predicted spectrum vector according to the binary coding vector further specifically includes: If it is determined that the predicted spectrum vector does not satisfy the preset spectrum condition, the predicted spectrum vector is obtained again according to the binary code vector until the predicted spectrum vector satisfies the preset spectrum condition.

8. The multispectral recognition method based on metasurface according to claim 7, characterized in that: After the multispectral image and the full-band image are subjected to feature fusion, a fusion pooling result is obtained, the fusion pooling result is subjected to softmax normalization processing, and a final target recognition result is output, which specifically includes: Performing third-dimensional stitching on the multispectral images to obtain a three-dimensional multispectrum; The three-dimensional multi-spectrum is subjected to feature fusion through a dual attention residual block to obtain a fused feature map; The full-band image is feature fused through the dual attention residual block to obtain a full-band feature map; The fused feature map and the full-band feature map are respectively encoded into key-value mapping pairs, and a fused feature value map, a fused feature key map, a full-band feature value map and a full-band feature key map are obtained; The fused eigenvalue map and the full-band eigenvalue map are subjected to feature fusion to obtain a fused value map; The fused feature key graph and the full-band feature key graph are subjected to feature fusion to obtain a fused key graph; The fusion value map and the fusion key map are respectively subjected to softmax normalization, and the normalized fusion value map and fusion key map are summed to obtain the final feature fusion result; Performing context-aware feature aggregation on the final feature fusion result, selecting a preset number of resolutions to perform parallel pooling operations, and obtaining a corresponding preset number of pooling results; Fusing the preset number of pooling results through an attention mechanism to obtain a fused pooling result; The fusion pooling result is subjected to softmax normalization processing, and the final target recognition result is output.

9. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 3 to 8.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 3 to 8.

Images