Multichannel privacy imaging device and method and apparatus for designing the same

Abstract

The disclosure provides a multi-channel privacy imaging device and a design method and device thereof, and relates to the fields of artificial intelligence and optical imaging. The design method can include: obtaining initial structure parameters of each optimization object respectively; wherein the multi-channel privacy imaging device includes: N kinds of regulation units arranged according to a preset period, N is a positive integer greater than 1, the N kinds of regulation units are determined as optimization objects respectively, and different optimization objects are used for modulating light of different wavelengths; obtaining training samples, and performing joint optimization on each optimization object according to the training samples and the initial structure parameters of each optimization object to obtain target structure parameters of each optimization object. The scheme disclosed in the disclosure can improve the overall performance of the multi-channel privacy imaging device.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the fields of artificial intelligence and optical imaging, in particular to micro-nano optics, computational imaging, deep learning and privacy protection technologies, and especially to a multi-channel privacy imaging device and a design method and apparatus thereof. BACKGROUND

[0002] With the popularization of intelligent sensing technology, it brings convenience, but also causes increasingly serious personal privacy leakage risk. For this purpose, a privacy imaging device can be used for imaging. At present, the privacy imaging device is mainly a single-channel privacy imaging device, which focuses on a single wavelength (such as 940nm), which will cause the need for additional light in insufficient ambient light such as daytime, and introduce power consumption and heat dissipation problems, and the ability of single wavelength to capture information is limited, which has great limitations in application. SUMMARY

[0003] The present disclosure provides a multi-channel privacy imaging device and a design method and apparatus thereof.

[0004] A design method of a multi-channel privacy imaging device, comprising:

[0005] Respectively acquiring initial structure parameters of each optimization object; wherein the multi-channel privacy imaging device comprises N kinds of regulation units arranged according to a preset period, N is a positive integer greater than 1, the N kinds of regulation units are respectively determined as the optimization objects, and different optimization objects are respectively used for modulating light of different wavelengths;

[0006] Acquiring training samples, and according to the training samples and the initial structure parameters, jointly optimizing each optimization object to obtain target structure parameters of each optimization object.

[0007] A design apparatus of a multi-channel privacy imaging device, comprising: a first processing module and a second processing module;

[0008] The first processing module is configured to respectively acquire initial structure parameters of each optimization object; wherein the multi-channel privacy imaging device comprises N kinds of regulation units arranged according to a preset period, N is a positive integer greater than 1, the N kinds of regulation units are respectively determined as the optimization objects, and different optimization objects are respectively used for modulating light of different wavelengths;

[0009] The second processing module is configured to acquire training samples, and according to the training samples and the initial structure parameters, jointly optimize each optimization object to obtain target structure parameters of each optimization object.

[0010] A multi-channel privacy imaging device, comprising:

[0011] N types of control units are arranged according to a preset period, where N is a positive integer greater than 1. The N types of control units are used to modulate light of different wavelengths. The structural parameters of the N types of control units are the target structural parameters determined according to the design method of the multi-channel privacy imaging device described above.

[0012] An electronic device, comprising:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0016] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0017] A computer program product includes a computer program / instructions that, when executed by a processor, implement the method described above.

[0018] It should be understood that the descriptions in this section are not intended to identify key or essential features of the embodiments of this disclosure, nor are they intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0020] Figure 1 This is a flowchart illustrating an embodiment of the design method for the multi-channel privacy imaging device disclosed herein;

[0021] Figure 2 A schematic diagram showing the structural parameters of sublattice L1 and sublattice L2 of this disclosure;

[0022] Figure 3 This is a schematic diagram of the power spectrum distribution after direct imaging without privacy protection.

[0023] Figure 4 This is a schematic diagram of the power spectrum distribution after privacy-preserving imaging in accordance with this disclosure.

[0024] Figure 5 This is a schematic diagram of the composition structure of an embodiment of the multi-channel privacy imaging device of this disclosure;

[0025] Figure 6This is a schematic diagram of the metasurface structure in the multi-channel privacy imaging device of this disclosure, which includes sublattice L1 and sublattice L2.

[0026] Figure 7 A schematic block diagram of an electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0027] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0028] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0029] Figure 1 This is a flowchart illustrating an embodiment of the design method for the multi-channel privacy imaging device described in this disclosure. Figure 1 As shown, the specific implementation methods are as follows.

[0030] In step 101, the initial structural parameters of each optimization object are obtained respectively; wherein, the multi-channel privacy imaging device includes: N types of control units arranged according to a preset period, where N is a positive integer greater than 1, and the N types of control units are respectively determined as optimization objects, and different optimization objects are used to modulate light of different wavelengths.

[0031] In step 102, training samples are obtained, and each optimization object is jointly optimized based on the training samples and each initial structural parameter to obtain the target structural parameters of each optimization object.

[0032] The multi-channel privacy imaging device described in the above-described method embodiments can effectively fuse different wavelengths, achieving information complementarity and all-weather operation, thereby avoiding power consumption and heat dissipation issues. Furthermore, it can be applied in various scenarios, exhibiting wide applicability. Additionally, by jointly optimizing the target structural parameters of various control units, the performance competition between different wavelength channels can be automatically balanced, optimizing the structural configuration from a global perspective and improving the overall performance of the multi-channel privacy imaging device.

[0033] The multi-channel privacy imaging device described in this disclosure may include N types of control units arranged according to a preset period, where N is a positive integer greater than 1, and the specific value is determined according to actual needs. Different control units can be used to modulate light of different wavelengths, and different wavelengths represent different channels.

[0034] In practical applications, the multi-channel privacy imaging device can be a multi-channel metasurface privacy imaging device. A metasurface is an artificial layered material with a size smaller than or approximately equal to the wavelength, which can be considered as a two-dimensional counterpart of a metamaterial. It can achieve the control of electromagnetic wave polarization, phase, amplitude, frequency, propagation mode, and other characteristics through subwavelength metastructural units on its surface, realizing beam shaping, beam deflection, superlensing, superholography, optical rotation, anti-reflection, and anti-reflection properties. Accordingly, the multi-channel metasurface privacy imaging device may include a metasurface, on which N seed lattices (i.e., control units) are arranged according to a preset period. Assuming N is 2, the metasurface includes at least one sub-lattice L1 and at least one sub-lattice L2. Sublattice L1 is used to modulate light with wavelength λ1, and sublattice L2 is used to modulate light with wavelength λ2. λ1 can be 650nm (visible light) and λ2 can be 940nm (near infrared). Wavelength decoupling can be achieved by separating sublattice L1 and sublattice L2 in physical space. The specific number of sublattice L1 and sublattice L2 can also be determined according to actual needs.

[0035] Several micro- and nanostructures can be arranged within the sublattice L1, and the arrangement period of these micro- and nanostructures can satisfy... Several micro / nano structures can also be arranged within the sublattice L2, and the arrangement period of these micro / nano structures can satisfy... , This represents the effective refractive index of the corresponding sublattice. Its value ranges from [0.6, 1.2], aiming to strike a balance between suppressing higher-order diffraction and ensuring design freedom.

[0036] Accordingly, the structural parameters of each optimized object refer to the geometric parameters of the micro / nano structures in each sublattice. For example, the structural parameters of sublattice L1 can be expressed as follows: The structural parameters of the sublattice L2 can be expressed as: , , , These represent the length, width, and rotation angle of the micro / nano structure, respectively. For example... Figure 2 As shown, Figure 2 This is a schematic diagram of the structural parameters of sublattice L1 and sublattice L2 as described in this disclosure. The left side represents sublattice L1, and the right side represents sublattice L2.

[0037] In some embodiments of this disclosure, each optimized object may be implemented using different materials, and / or, polarization multiplexing may be used between the optimized objects.

[0038] For example, sublattices L1 and L2 can be stacked using hybrid materials. Sublattice L1 can be made of titanium dioxide (TiO2). TiO2 has a high refractive index and low absorption loss in the visible light band, making it an ideal visible light metasurface material. Sublattice L2 can be made of amorphous silicon (a-Si). a-Si has a high refractive index and low absorption loss in the near-infrared band, making it a commonly used infrared metasurface material. Hybrid material stacking refers to using different materials on the same plane or employing a multilayer structure so that each wavelength of light is processed by its most suitable material, thereby improving processing efficiency.

[0039] Furthermore, sublattices L1 and L2 can employ polarization multiplexing. For example, λ1 can use a PB geometric phase channel (Pancharatnam-Berry phase), while λ2 can use a polarization-preserving propagation phase channel to improve band orthogonality and crosstalk immunity. In other words, for channel λ1, the PB geometric phase can be used so that when left-circularly polarized light (LCP) is incident, its cross-polarization component carries a 2φ phase. This cross-polarization component refers to the right-circularly polarized light (RCP) emitted, providing a non-resonant and efficient phase modulation method. Simultaneously, since it only acts on the cross-polarization component, it can naturally separate from the other channel in polarization. For channel λ2, the micro / nano structure itself can act as a waveguide, where light propagates and generates a resonant phase shift. This phase shift is related to the dimensions (length and width) of the micro / nano structure but independent of the rotation angle, thus maintaining the incident polarization (same-polarized emission). Correspondingly, by combining the λ1 and λ2 channels, polarization dimensions can be reused. One channel processes cross-polarized light, while the other channel processes co-polarized light, thereby greatly improving the orthogonality of the two bands and fundamentally reducing crosstalk.

[0040] In the scheme described in this disclosure, structural parameters of each optimization object can also be optimized. For example, the initial structural parameters of each optimization object can be obtained separately, then training samples can be obtained, and the optimization objects can be jointly optimized based on the training samples and the initial structural parameters to obtain the target structural parameters of each optimization object.

[0041] Accordingly, through four dimensions—spatial separation (such as sublattice L1 and sublattice L2), material selection (such as TiO2 and a-Si), polarization multiplexing, and structural parameter optimization—we can jointly ensure that multi-channel privacy imaging devices can achieve independent, efficient, and low-crosstalk optical field modulation for different wavelengths.

[0042] Furthermore, the Optical Transfer Function (OTF) describes how an optical system transmits information at different spatial frequencies (i.e., the level of detail in an image). Ideally, the modulus of the OTF for imaging should be 1 across all achievable frequencies, perfectly preserving all information. The scheme described in this disclosure aims to achieve a preset target OTF, namely, high-fidelity transmission of low-frequency information such as the general outline and smooth areas of an image (low-frequency fidelity), while strongly suppressing high-frequency information such as fine edges, textures, and identifiable details like faces (high-frequency suppression). Different channels can correspond to different target OTFs (preset).

[0043] Accordingly, the desired final image is one that is "understandable as to what happened," but cannot identify who or what specific details are being made, thus achieving privacy protection.

[0044] It should be noted that in the scheme described in this disclosure, each optimized object can be implemented using different materials or the same material. Structurally, each optimized object can also be implemented without any multiplexing method, in which case each optimized object corresponds to an independent optical path and is processed separately. Of course, multiplexing methods such as polarization multiplexing can also be used as needed. Specifically, the materials and multiplexing methods can be selected and combined according to actual needs.

[0045] In some embodiments of this disclosure, different optimized objects may also be refractive optical elements, diffractive optical elements, or scattering medium elements. Refractive optical elements include, but are not limited to, lenses or prisms made of materials such as optical glass, optical plastics, and optical crystals. Diffractive optical elements include, but are not limited to, two-step or multi-step diffractive optical elements, gratings, Dammann gratings, metasurfaces, holograms, phase masks, intensity masks, and spatial light modulators. Scattering medium elements include, but are not limited to, frosted glass.

[0046] In some embodiments of this disclosure, the multi-channel metasurface privacy imaging device may further include a sensing unit for receiving modulated optical signals and converting them into electrical signals to generate modulated images. The sensing unit may be a complementary metal-oxide-semiconductor (CMOS) photosensitive element, a charge-coupled device (CCD) photosensitive element, or an array photodetector, etc.

[0047] In addition, each control unit can have a corresponding filter unit on its light-emitting side, and the passband center wavelength of each filter unit can be the same as the operating wavelength of its corresponding control unit. The operating wavelength of the control unit is the wavelength it is responsible for processing. Thus, the incident light is first modulated to its operating wavelength by the control unit, and then the filter unit precisely selects the light of that wavelength while filtering out stray light of non-operating wavelengths, thereby reducing imaging interference. All filter units can be integrated into a single filter.

[0048] In addition, the N types of control units can also share a single multi-passband filter unit, and the center wavelength of each passband of the multi-passband filter unit corresponds one-to-one with the operating wavelength of the N types of control units.

[0049] In some embodiments of this disclosure, when optimizing the structural parameters of each optimization object, the initial structural parameters of each optimization object can be obtained first. Specifically, each optimization object can be optimized independently to obtain its initial structural parameters.

[0050] Independent optimization can quickly and efficiently determine good initial solutions for different optimization objects, thus laying a good foundation for subsequent joint optimization.

[0051] There are no restrictions on how to optimize each optimization object independently, or, to speed up the processing flow, preset, corresponding empirical values ​​can be determined as the initial structural parameters for each optimization object.

[0052] Subsequently, training samples can be obtained, and joint optimization can be performed on each optimization object based on the training samples and the initial structural parameters of each optimization object. In some embodiments of this disclosure, the training samples may include: training images and ground truth labels, where the ground truth labels are the task processing results corresponding to the training images. The method for joint optimization of each optimization object may include: generating a modulation image corresponding to the training image based on the current structural parameters of each optimization object; determining the task loss based on the modulation image and the ground truth labels; obtaining the intra-channel performance loss corresponding to each optimization object and obtaining the inter-channel performance loss common to each optimization object; determining the comprehensive loss based on the task loss, intra-channel performance loss, and inter-channel performance loss; and updating the structural parameters of each optimization object based on the comprehensive loss.

[0053] The training images in the training samples can be clear images captured using an infinite-resolution camera. Preferably, they need to cover different application scenarios, such as indoors, outdoors, streets, and vehicle interiors, and need to have spectral diversity, such as including objects with different spectral reflectance characteristics in the training images. Furthermore, the task processing results can be generated using automatic and / or manual annotation methods.

[0054] In practical applications, the joint optimization may include multiple rounds of training performed sequentially until a preset optimization termination condition (convergence condition) is met. During each training round, a modulated image corresponding to the training image in the training samples can be generated based on the current structural parameters (latest updated structural parameters) of each optimization object. The task loss can then be determined based on the modulated image and the true labels in the training samples. Furthermore, the intra-channel performance loss for each optimization object and the inter-channel performance loss common to all optimization objects can be obtained separately. The comprehensive loss can then be determined by combining the task loss, intra-channel performance loss, and inter-channel performance loss.

[0055] By combining task loss, intra-channel performance loss, and inter-channel performance loss to determine the comprehensive loss, multi-objective collaborative optimization can be achieved. Moreover, this multi-dimensional loss design method not only considers the independent performance of each channel, but also the common performance between channels, thereby achieving global optimization at the system level and ensuring the comprehensive performance and robustness of multi-channel privacy imaging devices in complex application scenarios.

[0056] In some embodiments of this disclosure, the optical response corresponding to each optimization object can be determined according to the current structural parameters of each optimization object. Then, the point spread function (PSF) corresponding to each optimization object can be determined according to the optical response. Finally, a modulated image can be generated according to the training image and each point spread function.

[0057] In practical applications, a differentiable electromagnetic forward model can be established. This model can be implemented using techniques such as rigorous coupled-wave analysis (RCWA), adjoint finite-difference time-domain (ADT), or multipole moment surrogate network (MPN). Based on this model, given the current structural parameters of any optimization object, the corresponding optical response, such as the transmission coefficient, can be generated. The transmission coefficient represents the complex amplitude of the transmitted light field at the corresponding wavelength. Changing the current structural parameters will also change the corresponding transmission coefficient. Based on the transmission coefficient, the point spread function and optical transfer function corresponding to the optimized object can be calculated in a predetermined manner.

[0058] The process of generating modulated images can be realized through simulation. Moreover, the imaging process can be modeled as a convolution process: the training image is convolved with a point spread function to obtain the modulated image.

[0059] That is:

[0060] (1)

[0061] Where I represents the training image, This represents the point spread function corresponding to a certain optimization object. This represents the optical transfer function corresponding to the object being optimized. The result of the inverse Fourier transform is , This represents the intermediate modulation image corresponding to the optimized object.

[0062] For different optimization targets, the corresponding results can be obtained according to the method in formula (1). This can then be understood as referring to each The images are added together to obtain the final modulated image, such as a blurred image that has been privacy-processed after metasurface imaging.

[0063] Understandably, modulated images can also be obtained using a physical multi-channel metasurface privacy imaging device. Simultaneously, a programmable metasurface can be employed to configure the structural parameters of each optimized object (i.e., sublattice) according to training requirements.

[0064] The task loss can then be determined based on the modulated image and the true labels in the training samples. In some embodiments of this disclosure, the modulated image can be input into the downstream task model to obtain the output task processing result, and the output task processing result can be determined as the predicted label, thereby determining the task loss based on the true label and the predicted label.

[0065] For example, the downstream task model can refer to a face recognition model, object detection model, etc., depending on the application scenario of the scheme described in this disclosure. After inputting the modulated image into the downstream task model, the task processing result output by the downstream task model can be obtained, and the output task processing result can be determined as the predicted label. Then, the error between the true label and the predicted label can be calculated through methods such as cross-entropy loss, and the error can be determined as the task loss. Accordingly, the gradient of the task loss can be backpropagated to the structural parameters of each optimization object through differentiable modeling, so that it can be optimized according to the "task". The downstream task model can be pre-trained and does not perform parameter updates during the joint optimization process.

[0066] In addition, in some embodiments of this disclosure, the method for obtaining the channel performance loss corresponding to each optimization object may include: for any optimization object, determining the optical transfer function corresponding to the optimization object based on the corresponding optical response, determining the bandwidth power ratio based on the optical transfer function, determining the image constraint information based on the modulated image, determining the privacy fidelity loss based on the bandwidth power ratio and the image constraint information, determining the light intensity based on the optical response, determining the efficiency loss by integrating the light intensity on a preset transverse wave vector, and determining the privacy fidelity loss and efficiency loss as the channel performance loss corresponding to the optimization object.

[0067] Specifically, in some embodiments of this disclosure, the method for determining the band power ratio may include: acquiring high-frequency power and low-frequency power respectively, wherein the high-frequency power is the sum of the power spectra of frequency components that are higher than or equal to a preset frequency threshold, and the low-frequency power is the sum of the power spectra of frequency components that are lower than the frequency threshold, and determining the ratio of high-frequency power to low-frequency power as the band power ratio.

[0068] That is:

[0069] Bandwidth power ratio = / (2)

[0070] Where f represents frequency, This represents a preset frequency threshold, the specific value of which can be determined according to actual needs. It is used to distinguish between low-frequency information that needs to be retained and high-frequency information that needs to be suppressed. This represents the sum of the power spectra of all frequency components above or equal to the frequency threshold at wavelength λ (the wavelength corresponding to the target object, such as λ1 or λ2). It represents the total energy of the "details" in the image that need to be suppressed. This represents the sum of the power spectra of all frequency components below a frequency threshold. It signifies the total energy of the "non-details" in the image that needs to be preserved. Ideally, the bandwidth power ratio should be as small as possible, such as less than a certain constant threshold. (less than 1), in order to achieve the purpose of high frequency suppression and low frequency fidelity preservation.

[0071] In the scheme described in this disclosure, in addition to utilizing the bandwidth power ratio, when calculating the privacy fidelity loss, image constraint information is further combined. For example, edge detection operators such as Canny can be used to process the modulated image, and the processing result is used as image constraint information to enhance high-frequency information such as edges. This allows the subject edges to be preserved as much as possible while suppressing high-frequency details, thus preventing excessive image blurring.

[0072] Accordingly, we can have:

[0073] (3)

[0074] in, Represents image constraint information. This indicates a loss of privacy.

[0075] In addition, the light intensity can be determined based on the optical response, and the efficiency loss can be determined by integrating the light intensity over a preset transverse wave vector.

[0076] That is:

[0077] (4)

[0078] in, This indicates the preset integration region. This represents the preset transverse wave vector, different Corresponding to light waves propagating at different angles, Indicates the transmission coefficient. Indicates light intensity. This indicates efficiency loss, used to ensure maximum light energy transmitted through the metasurface and to avoid designing a metasurface that is too dark.

[0079] For each optimization object, the corresponding privacy fidelity loss and efficiency loss can be determined in the manner described above. Both privacy fidelity loss and efficiency loss are intra-channel performance losses. In addition, the inter-channel performance loss common to each optimization object can also be obtained.

[0080] In some embodiments of this disclosure, the point spread function loss corresponding to each optimization object can be obtained, and the influence information of the structural parameter changes of each optimization object on other optimization objects can be obtained. Then, the crosstalk penalty loss can be determined according to each point spread loss function and each influence information, and the crosstalk penalty loss is determined as the inter-channel performance loss.

[0081] That is:

[0082] (5)

[0083] Taking the optimization objects including sublattice L1 and sublattice L2 as an example, This represents the point spread function loss, used to ensure the imaging quality of each channel itself. It can be understood as This is the sum of the point spread function losses corresponding to sublattice L1 and sublattice L2, respectively. The point spread function loss can be calculated based on the point spread function calculated from the transmission coefficient and the preset target optical transfer function, etc. This indicates the degree to which changes in the structural parameters of sublattice L2 affect sublattice L1. Specifically, it refers to the rate of change of the transmission coefficient of sublattice L1 when the structural parameters of sublattice L2 change. This indicates the degree to which changes in the structural parameters of sublattice L1 affect sublattice L2. Specifically, it refers to the rate of change of the transmission coefficient of sublattice L2 when the structural parameters of sublattice L1 change. Ideally, if the two channels are completely independent and have no crosstalk, then this rate of change should be 0. This indicates the penalty loss for crosstalk.

[0084] This can be called a crosstalk penalty term, which can be used as part of the crosstalk penalty loss, forcing the algorithm to find a set of structural parameters. , This makes the transmission coefficient corresponding to sublattice L1 only for... Sensitive, and to Insensitive, and vice versa, thus making the two channels as independent as possible.

[0085] After obtaining the task loss, intra-channel performance loss, and inter-channel performance loss respectively, the comprehensive loss can be determined by combining these losses. The structural parameters of each optimization object can be updated according to the comprehensive loss to obtain the target structural parameters of each optimization object.

[0086] In some embodiments of this disclosure, the comprehensive loss can be determined based on the weight coefficients corresponding to the task loss, intra-channel performance loss, inter-channel performance loss, and preset different losses. Then, the structural parameters of each optimization object can be updated according to the principle of minimizing the comprehensive loss.

[0087] That is:

[0088] (6)

[0089] Taking an optimization object that includes sublattice L1 and sublattice L2 as an example, where L represents the overall loss, Indicates mission loss. Each table represents a preset weighting coefficient, used to adjust the importance of different losses in the overall loss. The specific values ​​can be determined according to actual needs.

[0090] Through the above processing, the scheme described in this disclosure realizes an end-to-end design method, which can connect the design of physical devices with subsequent computational tasks (downstream task models) through differentiable modeling, and can use a comprehensive loss containing multiple objectives to guide optimization, and finally find the required target structural parameters so that the optical transfer function corresponding to each optimization object is close to its respective target optical transfer function, with maximum efficiency and minimum crosstalk, thereby automatically designing a multi-channel privacy imaging device that can achieve the predetermined imaging effect.

[0091] Furthermore, when optimizing each optimization object independently, the training samples used for each object can include training images and ground truth labels. Additionally, the task loss, privacy-preserving loss, and efficiency loss can be determined by combining these with the downstream task model. A comprehensive loss can then be calculated based on this comprehensive loss, allowing for the updating of the structural parameters of the optimization object. It can be seen that, compared to joint optimization, the comprehensive loss of independent optimization removes the crosstalk penalty loss.

[0092] After obtaining the target structural parameters of each optimized object through joint optimization, a multi-channel privacy imaging device can be fabricated based on the target structural parameters. This fabricated multi-channel privacy imaging device can then be applied to real-world scenarios to obtain privacy-preserving images. Specifically, the multi-channel privacy imaging device can physically modulate the light field at the front end of the imaging system to simultaneously process light of specific wavelengths (such as 650nm and 940nm), resulting in a visually "naturally blurred" imaging result. This preserves low-frequency information for downstream task models to process, while actively suppressing high-frequency information to achieve privacy protection, thus achieving a balance between privacy protection and useful information extraction at the source.

[0093] The scenarios may include fall detection, access control / attendance, anti-peeping for human-machine interfaces in conference rooms / vehicles, near-eye displays and augmented reality (AR) windows, near-infrared (NIR) biometrics countermeasures and visible / readable guidance, etc. Figure 3 This is a schematic diagram of the power spectrum distribution after direct imaging without privacy protection. Figure 4 This is a schematic diagram of the power spectrum distribution after privacy-preserving imaging in accordance with the method described in this disclosure. Both images are taken of the same scene. It can be seen that high-frequency information is significantly suppressed after using the method described in this disclosure.

[0094] It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0095] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0096] Figure 5 This is a schematic diagram of the structural composition of an embodiment 500 of the multi-channel privacy imaging device design device described in this disclosure. Figure 5 As shown, it includes: a first processing module 501 and a second processing module 502.

[0097] The first processing module 501 is used to acquire the initial structural parameters of each optimization object; wherein, the multi-channel privacy imaging device includes: N types of control units arranged according to a preset period, where N is a positive integer greater than 1, and the N types of control units are respectively determined as the optimization objects, and different optimization objects are used to modulate light of different wavelengths.

[0098] The second processing module 502 is used to acquire training samples and perform joint optimization on each optimization object based on the training samples and each initial structural parameter to obtain the target structural parameters of each optimization object.

[0099] In some embodiments of this disclosure, each optimized object may be implemented using different materials, and / or, polarization multiplexing may be used between the optimized objects.

[0100] In addition, in some embodiments of this disclosure, the first processing module 501 can independently optimize each optimization object to obtain the initial structural parameters of each optimization object.

[0101] Subsequently, the second processing module 502 can acquire training samples and perform joint optimization on each optimization object based on the training samples and the initial structural parameters of each optimization object. In some embodiments of this disclosure, the training samples may include: training images and real labels, where the real labels are the task processing results corresponding to the training images. The joint optimization of each optimization object by the second processing module 502 may include: generating a modulation image corresponding to the training image based on the current structural parameters of each optimization object; determining the task loss based on the modulation image and the real labels; acquiring the intra-channel performance loss corresponding to each optimization object and the inter-channel performance loss common to each optimization object; determining the comprehensive loss based on the task loss, intra-channel performance loss, and inter-channel performance loss; and updating the structural parameters of each optimization object based on the comprehensive loss.

[0102] In some embodiments of this disclosure, the second processing module 502 can determine the optical response corresponding to each optimization object based on the current structural parameters of each optimization object, and then determine the point spread function corresponding to each optimization object based on the optical response, and then generate a modulation image based on the training image and each point spread function.

[0103] In addition, in some embodiments of this disclosure, the second processing module 502 can input the modulated image into the downstream task model to obtain the output task processing result, and can determine the output task processing result as the predicted label, and then determine the task loss based on the real label and the predicted label.

[0104] In some embodiments of this disclosure, the second processing module 502 may obtain the channel performance loss corresponding to each optimization object in the following ways: for any optimization object, determine the optical transfer function corresponding to the optimization object based on the corresponding optical response, determine the bandwidth power ratio based on the optical transfer function, determine the image constraint information based on the modulated image, determine the privacy fidelity loss based on the bandwidth power ratio and the image constraint information, determine the light intensity based on the optical response, determine the efficiency loss by integrating the light intensity on a preset transverse wave vector, and determine the privacy fidelity loss and efficiency loss as the channel performance loss corresponding to the optimization object.

[0105] Specifically, in some embodiments of this disclosure, the second processing module 502 may determine the band power ratio by: acquiring high-frequency power and low-frequency power respectively, wherein the high-frequency power is the sum of the power spectra of frequency components that are higher than or equal to a preset frequency threshold, and the low-frequency power is the sum of the power spectra of frequency components that are lower than the frequency threshold, and determining the ratio of high-frequency power to low-frequency power as the band power ratio.

[0106] In some embodiments of this disclosure, the second processing module 502 can obtain the point spread function loss corresponding to each optimization object, and can obtain the influence information of the structural parameter changes of each optimization object on other optimization objects. Then, it can determine the crosstalk penalty loss based on each point spread loss function and each influence information, and determine the crosstalk penalty loss as the inter-channel performance loss.

[0107] After obtaining the task loss, intra-channel performance loss, and inter-channel performance loss respectively, the second processing module 502 can determine the comprehensive loss by combining these losses, and can update the structural parameters of each optimization object according to the comprehensive loss to obtain the target structural parameters of each optimization object. In some embodiments of this disclosure, the second processing module 502 can determine the comprehensive loss according to the weight coefficients corresponding to the task loss, intra-channel performance loss, inter-channel performance loss, and preset different losses, and then update the structural parameters of each optimization object according to the principle of minimizing the comprehensive loss.

[0108] This disclosure also discloses a multi-channel privacy imaging device, including: N types of control units arranged according to a preset period, where N is a positive integer greater than 1. The N types of control units are respectively used to modulate light of different wavelengths, and the structural parameters of the N control units are respectively arranged according to... Figure 1 The target structural parameters are determined by the method shown.

[0109] Figure 6 This is a schematic diagram of the metasurface structure in the multi-channel privacy imaging device described in this disclosure, including sublattice L1 and sublattice L2. (See diagram for reference.) Figure 6 As shown, two types of control units (sublattice L1 and sublattice L2) are arranged on the metasurface according to a preset period. Sublattice L1 is used to control wavelengths of... (e.g., 650 nm) light is modulated, and the sublattice L2 is used to modulate the wavelength. (e.g., 940nm) light is modulated.

[0110] The specific workflow of each of the above device embodiments can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0111] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0112] Figure 7A schematic block diagram of an electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0113] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the electronic device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0114] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of displays, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0115] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the methods described herein by any other suitable means (e.g., by means of firmware).

[0116] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0118] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0119] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0120] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0121] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0122] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0123] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A design method for a multi-channel privacy imaging device, characterized in that, include: The initial structural parameters of each optimization object are obtained respectively; wherein, the multi-channel privacy imaging device includes: N types of control units arranged according to a preset period, where N is a positive integer greater than 1, and the N types of control units are respectively determined as the optimization objects, and different optimization objects are used to modulate light of different wavelengths; Acquire training samples, which include training images and ground truth labels, where the ground truth labels are the task processing results corresponding to the training images. Perform joint optimization on each optimization object, including: generating a modulation image corresponding to the training image based on the current structural parameters of each optimization object; determining the task loss based on the modulation image and the ground truth labels; acquiring the intra-channel performance loss corresponding to each optimization object and acquiring the inter-channel performance loss common to all optimization objects; determining a comprehensive loss based on the task loss, the intra-channel performance loss, and the inter-channel performance loss; updating the structural parameters of each optimization object based on the comprehensive loss to obtain the target structural parameters of each optimization object. The process of obtaining the channel performance loss corresponding to any optimization object includes: determining the optical transfer function based on the optical response of the optimization object; determining the bandwidth power ratio based on the optical transfer function and determining the image constraint information based on the modulated image; determining the privacy fidelity loss based on the bandwidth power ratio and the image constraint information; determining the light intensity based on the optical response and determining the efficiency loss by integrating the light intensity on a preset transverse wave vector; and determining the privacy fidelity loss and the efficiency loss as the channel performance loss corresponding to the optimization object, wherein the bandwidth power ratio is the ratio of high-frequency power to low-frequency power.

2. The method according to claim 1, characterized in that, Each of the control units is provided with a filter unit on its light-emitting side, and the passband center wavelength of the filter unit is consistent with the operating wavelength of the corresponding control unit.

3. The method according to claim 1, characterized in that, The process of generating the modulated image corresponding to the training image includes: Based on the current structural parameters of each optimization object, the optical response corresponding to each optimization object is determined respectively; The point spread function corresponding to each optimization object is determined based on the optical response. The modulated image is generated based on the training image and the spread function at each point.

4. The method according to claim 1, characterized in that, The step of determining the task loss based on the modulated image and the real label includes: The modulated image is input into the downstream task model to obtain the output task processing result, and the output task processing result is determined as the predicted label; The task loss is determined based on the actual label and the predicted label.

5. The method according to claim 1, characterized in that, The high-frequency power is the sum of the power spectra of frequency components that are higher than or equal to a preset frequency threshold, and the low-frequency power is the sum of the power spectra of frequency components that are lower than the frequency threshold.

6. The method according to claim 3, characterized in that, The process of obtaining the common inter-channel performance loss for each optimization object includes: The point spread function loss for each optimization object is obtained, and the impact of changes in the structural parameters of each optimization object on other optimization objects is also obtained. Based on the diffusion loss function at each point and the information on the degree of influence, the crosstalk penalty loss is determined, and the crosstalk penalty loss is defined as the inter-channel performance loss.

7. The method according to claim 1, characterized in that, The step of determining the comprehensive loss based on the task loss, the intra-channel performance loss, and the inter-channel performance loss, and updating the structural parameters of each optimization object based on the comprehensive loss, includes: The comprehensive loss is determined based on the weighting coefficients corresponding to the task loss, the intra-channel performance loss, the inter-channel performance loss, and the preset different losses. Based on the principle of minimizing the overall loss, the structural parameters of each optimization object are updated respectively.

8. A design device for a multi-channel privacy imaging device, characterized in that, include: First processing module and second processing module; The first processing module is used to obtain the initial structural parameters of each optimization object respectively; wherein, the multi-channel privacy imaging device includes: N types of control units arranged according to a preset period, where N is a positive integer greater than 1, and the N types of control units are respectively determined as the optimization objects, and different optimization objects are used to modulate light of different wavelengths respectively; The second processing module is used to acquire training samples, which include training images and ground truth labels, where the ground truth labels are the task processing results corresponding to the training images; and to perform joint optimization on each optimization object, including: generating a modulation image corresponding to the training image based on the current structural parameters of each optimization object; determining the task loss based on the modulation image and the ground truth labels; acquiring the intra-channel performance loss corresponding to each optimization object and acquiring the inter-channel performance loss common to all optimization objects; determining a comprehensive loss based on the task loss, the intra-channel performance loss, and the inter-channel performance loss; and updating the structural parameters of each optimization object based on the comprehensive loss. The target structural parameters of each optimization object are obtained; wherein, obtaining the channel performance loss corresponding to any optimization object includes: determining the optical transfer function based on the optical response corresponding to the optimization object; determining the bandwidth power ratio based on the optical transfer function, and determining the image constraint information based on the modulation image; determining the privacy fidelity loss based on the bandwidth power ratio and the image constraint information; determining the light intensity based on the optical response, and determining the efficiency loss by integrating the light intensity on a preset transverse wave vector; determining the privacy fidelity loss and the efficiency loss as the channel performance loss corresponding to the optimization object, wherein the bandwidth power ratio is the ratio of high-frequency power to low-frequency power.

9. A multi-channel privacy imaging device, characterized in that, include: N types of control units are arranged according to a preset period, where N is a positive integer greater than 1. The N types of control units are used to modulate light of different wavelengths. The structural parameters of the N types of control units are the target structural parameters determined by the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

12. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the method of any one of claims 1-7.

Images