Joint training method, device and system for imaging module and task processing model

Abstract

The invention provides a joint training method, device and system for an imaging module and a task processing model, and relates to the fields of optical sensing, computational imaging and the like. The joint training method comprises the following steps: generating an output image corresponding to a target object by using an imaging module; wherein the imaging module comprises a light source, a first optical modulation element and a photosensitive element, the first optical modulation element is used for modulating a light beam emitted by the light source and projecting the modulated light beam to a target object, and the photosensitive element is used for imaging based on reflected light of the target object to obtain an output image; generating a task processing result corresponding to the output image by using the task processing model, and determining the task processing result as a prediction label; and determining comprehensive loss according to the prediction label and the output image, and updating configuration parameters of the first optical modulation element and model parameters of the task processing model according to the comprehensive loss. By applying the scheme of the disclosure, the risk of privacy disclosure and the like can be reduced.

Description

Technical Field

[0001] This disclosure relates to fields such as optical sensing and computational imaging, and particularly to technologies such as metasurface phase modulation, near-infrared dot matrix projection, three-dimensional depth measurement and privacy-preserving imaging, especially to joint training methods, devices and systems for imaging modules and task processing models. Background Technology

[0002] When performing tasks such as gesture recognition, posture recognition, and face recognition, it is usually necessary to acquire the corresponding images first. However, direct imaging will expose facial features and high-frequency texture information, which poses a significant risk of privacy leakage. Summary of the Invention

[0003] This disclosure provides a method, apparatus, and system for joint training of an imaging module and a task processing model.

[0004] A joint training method for an imaging module and a task processing model, comprising: An imaging module is used to generate an output image corresponding to a target object. The imaging module includes a light source, a first optical modulation element, and a photosensitive element. The first optical modulation element is used to modulate the light beam emitted by the light source and project the modulated light beam onto the target object. The photosensitive element is used to perform imaging based on the reflected light from the target object to obtain the output image. The task processing result corresponding to the output image is generated using the task processing model and determined as the predicted label; The comprehensive loss is determined based on the predicted label and the output image, and the configuration parameters of the first optical modulation element and the model parameters of the task processing model are updated based on the comprehensive loss.

[0005] A joint training device for an imaging module and a task processing model includes: an image generation module, a label generation module, and a parameter update module; The image generation module is used to generate an output image corresponding to the target object using the imaging module; the imaging module includes: a light source, a first optical modulation element and a photosensitive element, the first optical modulation element is used to modulate the light beam emitted by the light source and project the modulated light beam onto the target object, and the photosensitive element is used to perform imaging based on the reflected light of the target object to obtain the output image; The label generation module is used to generate the task processing result corresponding to the output image using the task processing model, and determine it as the predicted label; The parameter update module is used to determine the comprehensive loss based on the predicted label and the output image, and update the configuration parameters of the first optical modulation element and the model parameters of the task processing model based on the comprehensive loss.

[0006] A task processing system includes: an imaging module and a task processing model; The imaging module includes: a light source, a first optical modulation element, and a photosensitive element; the first optical modulation element is used to modulate the light beam emitted by the light source and project the modulated light beam onto the target object; the photosensitive element is used to perform imaging based on the reflected light from the target object to obtain a target image. The task processing model is used to generate the task processing result corresponding to the target image. The configuration parameters of the first optical modulation element and the model parameters of the task processing model are obtained through joint training.

[0007] An electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0008] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 A flowchart illustrating an embodiment of the joint training method for the imaging module and the task processing model described in this disclosure; Figure 2 This is a schematic diagram of the composition structure of the first embodiment 200 of the task processing system described in this disclosure; Figure 3 This is a schematic diagram of the composition structure of the second embodiment 300 of the task processing system described in this disclosure; Figure 4 This is a schematic diagram of the composition structure of the third embodiment 400 of the task processing system described in this disclosure; Figure 5 This is a schematic diagram of the composition structure of the fourth embodiment 500 of the task processing system described in this disclosure; Figure 6 This is a schematic diagram of the composition structure of Embodiment 600 of the joint training device for the imaging module and the task processing model described in this disclosure; Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0011] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0012] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0013] Figure 1 This is a flowchart illustrating an embodiment of the joint training method for the imaging module and the task processing model described in this disclosure. Figure 1 As shown, the specific implementation methods are as follows.

[0014] In step 101, an output image corresponding to the target object is generated using an imaging module. The imaging module includes a light source, a first optical modulation element, and a photosensitive element. The first optical modulation element is used to modulate the light beam emitted by the light source and project the modulated light beam onto the target object. The photosensitive element is used to image the target object based on the reflected light to obtain the output image.

[0015] In step 102, the task processing model is used to generate the task processing result corresponding to the output image and determine it as the prediction label.

[0016] In step 103, the comprehensive loss is determined based on the predicted label and the output image, and the configuration parameters of the first optical modulation element and the model parameters of the task processing model are updated based on the comprehensive loss.

[0017] By employing the scheme described in the above method embodiments, the incident light can be actively modulated using the first optical modulation element, thereby blurring the output image generated by the photosensitive element at the optical front end. This can suppress high-frequency textures, thus reducing the risk of privacy leakage. Moreover, the blurring is physical blurring, which is irreversible and can be bypassed by reversible algorithms, thus achieving hardware-level privacy protection. In addition, the first optical modulation element can be jointly trained end-to-end with the task processing model, which can collaboratively optimize the physical modulation and algorithm model. This allows the imaging module to automatically learn how to hide privacy information to the maximum extent at the optical front end while preserving effective features for the back-end algorithm model as much as possible, achieving an effective balance between privacy protection and task performance.

[0018] In some embodiments of this disclosure, the light source may include: a controllable lattice mode vertical-cavity surface-emitting laser (VCSEL) light source, and the first optical modulation element may include: a first metasurface lens.

[0019] Controllable dot matrix mode VCSEL light source can have 50 A 50 or variable resolution dot matrix with programmable dot size, spacing, and pattern, and can be linked with a metasurface point spread function (PSF) to form a computable imaging code.

[0020] The metasurface lens described in this disclosure can also be called a metasurface phase-controlled lens. Through phase engineering, the PSF can be flexibly designed, and high-frequency textures can be suppressed to achieve physical blur. At the same time, the recognizability of structured light spots can be taken into account, and parameters such as the size, shape (e.g., ring, speckle, single peak, multi-peak), blur intensity and direction of the light spot can be adjusted in real time.

[0021] Furthermore, metasurface lenses are optical elements based on artificial micro / nano structures. Their surfaces are composed of a large number of subwavelength-scale nanounits arranged in a specific aperiodic pattern, each capable of localized phase modulation of incident light. Through overall design, they can precisely disrupt and recombine the Gaussian laser wavefront emitted by a controllable lattice mode VCSEL light source into a specific disordered light field. Their core function is not to produce clear imaging, but rather to actively create random speckle patterns at the optical front end, ensuring that subsequent photosensitive elements can only receive the encoded scattered light, thus achieving privacy protection.

[0022] Accordingly, the first optical modulation element can be positioned after the light source to modulate (pre-modulate and encode) the light beam emitted by the light source. This is equivalent to transforming a regular light spot (such as a Gaussian light spot) into a complex, random, and irregular random speckle, and then illuminating the target object. Consequently, the target object reflects this encoded, detailless reflected light, thus ultimately forming a blurred image on the photosensitive element.

[0023] The photosensitive element can be a complementary metal-oxide-semiconductor (CMOS) image sensor, which can image based on the reflected light from the target object to obtain the desired output image. Facial details and other information can be removed from the output image, but information such as movement, posture, and skeleton will be preserved.

[0024] As can be seen, the imaging module described in this disclosure adopts a modular design approach. In practical applications, the light source, the first optical modulation element, and the photosensitive element can be flexibly integrated into one unit or deployed independently as needed.

[0025] Furthermore, the output image can be input into the task processing model to obtain the output task processing result, and the task processing result can be determined as the predicted label.

[0026] The task processing model can be a neural network model (CCN) used to perform tasks such as gesture recognition, posture recognition, or face recognition, and there are no restrictions on the specific model structure.

[0027] Based on the predicted labels and output images, the overall loss can be determined, and the configuration parameters of the first optical modulation element and the model parameters of the task processing model can be updated according to the overall loss.

[0028] In some embodiments of this disclosure, the task loss can be determined based on the real label and predicted label corresponding to the input image. The real label is the real task processing result corresponding to the input image. The input image and the output image correspond to the same imaging scene, and the input image is a clear original image. The privacy loss can be determined based on the output image and the input image. Then, the comprehensive loss can be determined based on the task loss and the privacy loss.

[0029] In other words, the input image contains clear original image content, and the output image is essentially a blurred version of the original image content.

[0030] There are no restrictions on how the task loss is determined. For example, cross-entropy loss or mean squared error (MSE) loss can be used to calculate the task loss based on the true label and the predicted label.

[0031] In addition to task loss, privacy loss can be determined based on the output and input images, and then a comprehensive loss can be determined based on the task loss and privacy loss. Introducing privacy loss as a component of the comprehensive loss provides a clear privacy protection optimization objective for the entire end-to-end joint training process, thereby further improving training efficiency and training results.

[0032] For example, it could be: L = Ltask + Lprivacy; (1) Where L represents the overall loss, Ltask represents the task loss, and Lprivacy represents the privacy loss.

[0033] In addition, in some embodiments of this disclosure, the method of determining the privacy loss based on the output image and the input image may include: obtaining the sharpness loss between the output image and the input image, obtaining the frequency domain loss of the output image, obtaining the mask ratio loss of the output image, and determining the privacy loss based on the sharpness loss, the frequency domain loss and the mask ratio loss.

[0034] In some embodiments of this disclosure, the method of obtaining the mask ratio loss of the output image may include: for each pixel in the output image, the following processing is performed respectively: in response to determining that the light intensity of each pixel within a predetermined area where the pixel is located is greater than a first threshold, the pixel is determined as a valid pixel; otherwise, the pixel is determined as a mask pixel, and the mask ratio loss is determined according to the number of mask pixels.

[0035] If: (2) Where Iout represents the output image, Iclear represents the input image, LowPass represents the low-frequency components in the output image, HighPass represents the high-frequency components in the output image, ε represents the stability parameter, λ1, λ2, and λ3 represent the coefficients corresponding to sharpness loss, frequency domain loss, and mask scaling loss, respectively, and the specific values ​​of λ1, λ2, λ3, and ε can be determined according to actual needs, |||2 represents the L2 norm, |||1 represents the L1 norm, and N represents the total number of pixels included in the input / output image.

[0036] Sharpness loss can be used to penalize the similarity between the output and input images. Frequency domain loss is used to encourage the output image to retain low-frequency components (typically corresponding to contour information with less leakage of detail) and suppress high-frequency components. Low-frequency and high-frequency components can be distinguished and obtained in the frequency domain based on their spatial frequency after performing a Fourier transform on the output image.

[0037] mask_ratio = number of mask pixels / total number of pixels in the output image; (3) For each pixel in the output image, it can be divided into valid pixels or mask pixels. A valid pixel is one that meets the following requirement: the light intensity of all pixels within a predetermined area containing the valid pixel is greater than a first threshold. The specific value of the first threshold can be determined according to actual needs. Furthermore, the specific size of the predetermined area can also be determined according to actual needs; for example, a 3... 3. The area range of size.

[0038] In some embodiments of this disclosure, in addition to the light source, the first optical modulation element, and the photosensitive element, the imaging module may also include a second optical modulation element. The second optical modulation element can be used to modulate the reflected light from the target object and project it onto the photosensitive element. Accordingly, after obtaining the overall loss, the configuration parameters of the second optical modulation element can be updated according to the overall loss. That is, the configuration parameters of the first optical modulation element, the configuration parameters of the second optical modulation element, and the model parameters of the task processing model can be updated according to the overall loss.

[0039] If a metasurface lens is added in front of the photosensitive element, the reflected light from the target object will be modulated by the metasurface lens before being projected onto the photosensitive element. This allows for secondary modulation and optimization of the reflected light, thereby further improving the blurring effect of the output image and enhancing privacy protection.

[0040] In some embodiments of this disclosure, in addition to the light source, the first optical modulation element, and the photosensitive element, the imaging module may also include a beam splitting component and a depth sensor. The beam splitting component can be used to change the propagation path of the reflected light from the target object, thereby splitting the reflected light from the target object to the photosensitive element and the depth sensor, such as obtaining a first echo light signal and a second echo light signal, and propagating the first echo light signal to the photosensitive element and the second echo light signal to the depth sensor. The depth sensor can be used to generate depth information corresponding to the target object. Accordingly, when generating the task processing result corresponding to the output image using the task processing model, the output image and the depth information can be combined to generate the task processing result using the task processing model.

[0041] The beam-splitting component can be a beam-splitting prism, polarizer, metasurface, grating, or dichroic mirror, etc. Beam splitting does not change the blurring nature of the output image. The depth sensor can be a single-photon avalanche diode (SPAD), which can obtain high-precision depth information by measuring the time of flight of the reflections from the lattice, and can be spatially aligned with the photosensitive element (through the common optical path after beam splitting).

[0042] In some embodiments of this disclosure, the beam splitting component is configured to physically separate the light beam based on wavelength or polarization. Specifically, the beam splitting component guides a portion of the wavelength or polarization of light reflected back from the target object to a depth sensor for time-of-flight measurement, and another portion to a photosensitive element for two-dimensional image capture, thereby achieving synchronous and independent acquisition of depth and texture information.

[0043] By introducing a beam splitter and a depth sensor, the system's privacy protection capabilities can be enhanced from "two-dimensional spatial ambiguity" to "three-dimensional spatiotemporal encryption," thereby obtaining more accurate, reliable, and three-dimensional information-rich perception capabilities, and improving the accuracy of the acquired task processing results.

[0044] In some embodiments of this disclosure, depth loss can also be determined based on the depth information corresponding to the target object and the depth information corresponding to the input image. Correspondingly, a comprehensive loss can be determined based on task loss, privacy loss and depth loss, such as by adding the three together to determine the comprehensive loss.

[0045] For example, depth loss can be calculated using the MSE loss calculation method: (4) Where N represents the total number of pixels in the input / output image, Ldepth represents the depth loss, Zy represents the depth value (z coordinate) of the pixel in the depth information corresponding to the target object, and Zr represents the depth value of the pixel at the same position in the depth information corresponding to the input image.

[0046] By introducing depth loss, three-dimensional geometric constraints can be added to the entire end-to-end joint training process, and the task processing model is forced to use and trust the depth information provided by the depth sensor when understanding the task, rather than relying solely on the blurred two-dimensional image. This allows the system to maintain or even improve the accuracy of its perception of the three-dimensional shape, position and motion of the target object while achieving physical-level privacy protection.

[0047] In some embodiments of this disclosure, in addition to the light source, the first optical modulation element, and the photosensitive element, the imaging module may also include: a second optical modulation element, a beam splitting component, and a depth sensor. The second optical modulation element can be used to modulate the reflected light from the target object and project it onto the beam splitting component. The beam splitting component can be used to split the projected light beam to the photosensitive element and the depth sensor. The depth sensor can be used to generate depth information corresponding to the target object. Accordingly, when generating the task processing result corresponding to the output image using the task processing model, the task processing result can be generated using the task processing model based on the output image and the depth information.

[0048] In addition, the depth loss can be determined based on the depth information corresponding to the target object and the depth information corresponding to the input image. Correspondingly, the comprehensive loss can be determined based on the task loss, privacy loss and depth loss. For example, the sum of the three can be determined as the comprehensive loss. Then, the configuration parameters of the first optical modulation element, the configuration parameters of the second optical modulation element and the model parameters of the task processing model can be updated based on the comprehensive loss.

[0049] Regardless of the components included in the imaging module, the joint training process can terminate when a predetermined termination condition is met. This predetermined termination condition could refer to reaching a predetermined number of training epochs or loss convergence, among other things. Accordingly, a first optical modulation element and a second optical modulation element can be fabricated based on the configuration parameters determined through joint training, and the model parameters of the task processing model can be fixed to the model parameters determined through joint training.

[0050] It should be noted that, for the sake of simplicity, the foregoing method embodiments are described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0051] The above is an introduction to the method embodiments. The following system and device embodiments will further illustrate the solution described in this disclosure.

[0052] Figure 2 This is a schematic diagram of the structural composition of the first embodiment 200 of the task processing system described in this disclosure. Figure 2 As shown, it includes: imaging module 21 and task processing model 22.

[0053] The imaging module 21 may include a light source 211, a first optical modulation element 212, and a photosensitive element 213. The first optical modulation element 212 is used to modulate the light beam emitted by the light source 211 and project the modulated light beam onto the target object. The photosensitive element 213 is used to perform imaging based on the reflected light from the target object to obtain a target image.

[0054] Task processing model 22 is used to generate task processing results corresponding to the target image. The configuration parameters of the first optical modulation element 212 and the model parameters of task processing model 22 are obtained through joint training. The joint training method may be... Figure 1 The joint training method shown in the figure.

[0055] Figure 3This is a schematic diagram of the structural composition of the second embodiment 300 of the task processing system described in this disclosure. Figure 3 As shown, it includes: imaging module 21 and task processing model 22.

[0056] The imaging module 21 may include: a light source 211, a first optical modulation element 212, a photosensitive element 213, and a second optical modulation element 214.

[0057] It can be seen that, compared to Figure 2 The embodiment shown, Figure 3 The embodiment shown further includes a second optical modulation element 214, which can be used to modulate the reflected light from the target object and project it onto the photosensitive element 213. The configuration parameters of the second optical modulation element 214 can also be obtained by training through the joint training method.

[0058] Figure 4 This is a schematic diagram of the structural composition of the third embodiment 400 of the task processing system described in this disclosure. Figure 4 As shown, it includes: imaging module 21 and task processing model 22.

[0059] The imaging module 21 may include: a light source 211, a first optical modulation element 212, a photosensitive element 213, a beam splitting assembly 215, and a depth sensor 216.

[0060] It can be seen that, compared to Figure 2 The embodiment shown, Figure 4 The illustrated embodiment further includes a beam splitter 215 and a depth sensor 216. The beam splitter 215 can be used to split the reflected light from the target object into the photosensitive element 213 and the depth sensor 216, and the depth sensor 216 can be used to generate depth information corresponding to the target object. Accordingly, the task processing model 22 can generate a task processing result based on the target image and the depth information.

[0061] Figure 5 This is a schematic diagram of the structural composition of the fourth embodiment 500 of the task processing system described in this disclosure. Figure 5 As shown, it includes: imaging module 21 and task processing model 22.

[0062] The imaging module 21 may include: a light source 211, a first optical modulation element 212, a photosensitive element 213, a second optical modulation element 214, a beam splitting assembly 215, and a depth sensor 216.

[0063] It can be seen that, compared to Figure 2 The embodiment shown, Figure 5The illustrated embodiment further includes a second optical modulation element 214, a beam splitter 215, and a depth sensor 216. The second optical modulation element 214 modulates the reflected light from the target object and projects it onto the beam splitter 215. The configuration parameters of the second optical modulation element 214 can be obtained through a joint training method. The beam splitter 215 splits the projected light beam to a photosensitive element 213 and a depth sensor 216. The depth sensor 216 generates depth information corresponding to the target object. Accordingly, the task processing model 22 generates a task processing result based on the target image and the depth information.

[0064] In practical applications, the light source 211 can be a controllable dot matrix VCSEL light source, the first optical modulation element 212 can be a first metasurface lens, the photosensitive element 213 can be a CMOS image sensor, the second optical modulation element 214 can be a second metasurface lens, the beam splitting component 215 can be a dichroic mirror, a polarizing beam splitter, or a miniature cubic beam splitter, and the depth sensor 216 can be a SPAD.

[0065] As can be seen, the solution described in this disclosure provides a composite imaging method that combines metasurface physical fuzzy imaging, dot matrix light source, CMOS image capture, and SPAD ranging. This method can simultaneously achieve privacy-preserving two-dimensional imaging (fuzzy) and high-precision three-dimensional depth measurement, thereby meeting the needs of different scenarios, such as human detection in security, smart homes, and privacy computing devices (e.g., bathrooms, bedrooms, and medical settings). Moreover, the entire system is small in size, highly integrated, and easy to deploy and maintain.

[0066] This disclosure also discloses a joint training device for the imaging module and the task processing model. Accordingly, Figure 6 This is a schematic diagram of the structural composition of an embodiment 600 of the joint training device for the imaging module and the task processing model described in this disclosure. Figure 6 As shown, it includes: an image generation module 601, a label generation module 602, and a parameter update module 603.

[0067] The image generation module 601 is used to generate an output image corresponding to the target object using the imaging module. The imaging module includes a light source, a first optical modulation element, and a photosensitive element. The first optical modulation element is used to modulate the light beam emitted by the light source and project the modulated light beam onto the target object. The photosensitive element is used to perform imaging based on the reflected light of the target object to obtain the output image.

[0068] The label generation module 602 is used to generate the task processing result corresponding to the output image using the task processing model and determine it as the predicted label.

[0069] The parameter update module 603 is used to determine the comprehensive loss based on the predicted label and the output image, and update the configuration parameters of the first optical modulation element and the model parameters of the task processing model based on the comprehensive loss.

[0070] In practical applications, the imaging module may further include other components, such as... Figures 3-5 As shown in the figure, the specific workflows of the image generation module 601, the label generation module 602, and the parameter update module 603 can be referred to the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0071] It should be noted that the images and task processing results in the embodiments described in this disclosure are not targeted at any specific user and are not intended to reflect the personal information of any specific user. The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solutions of this disclosure all comply with relevant laws and regulations and do not violate public order and good morals.

[0072] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0073] Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0074] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the electronic device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0075] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of displays, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0076] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the methods described herein by any other suitable means (e.g., by means of firmware).

[0077] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0078] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0079] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0080] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0081] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0082] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0083] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0084] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A joint training method for an imaging module and a task processing model, characterized in that, include: The imaging module is used to generate an output image corresponding to the target object. The imaging module includes a light source, a first optical modulation element, and a photosensitive element. The first optical modulation element is used to modulate the light beam emitted by the light source and project the modulated light beam onto the target object. The photosensitive element is used to perform imaging based on the reflected light from the target object to obtain the output image. The task processing result corresponding to the output image is generated using the task processing model and determined as the predicted label; The comprehensive loss is determined based on the predicted label and the output image, and the configuration parameters of the first optical modulation element and the model parameters of the task processing model are updated based on the comprehensive loss.

2. The method according to claim 1, characterized in that, The determination of the comprehensive loss based on the predicted label and the output image includes: The task loss is determined based on the real label corresponding to the input image and the predicted label. The real label is the real task processing result corresponding to the input image. The input image and the output image correspond to the same imaging scene, and the input image is a clear original image. The privacy loss is determined based on the output image and the input image; The combined loss is determined based on the task loss and the privacy loss.

3. The method according to claim 2, characterized in that, The determination of privacy loss based on the output image and the input image includes: The sharpness loss between the output image and the input image is obtained; Obtain the frequency domain loss of the output image; Obtain the mask scaling loss of the output image; The privacy loss is determined based on the resolution loss, the frequency domain loss, and the masking ratio loss.

4. The method according to claim 3, characterized in that, The mask scaling loss for obtaining the output image includes: For each pixel in the output image, the following processing is performed: in response to determining that the light intensity of each pixel within a predetermined area where the pixel is located is greater than a first threshold, the pixel is determined as a valid pixel; otherwise, the pixel is determined as a mask pixel. The mask ratio loss is determined based on the number of mask pixels.

5. The method according to claim 1, characterized in that, The imaging module further includes a second optical modulation element, which modulates the reflected light from the target object and projects it onto the photosensitive element. The method further includes updating the configuration parameters of the second optical modulation element based on the overall loss.

6. The method according to claim 2, characterized in that, The imaging module also includes: a beam splitter and a depth sensor; The beam-splitting component is used to split the reflected light from the target object into the photosensitive element and the depth sensor, and the depth sensor is used to generate depth information corresponding to the target object; The step of generating the task processing result corresponding to the output image using the task processing model includes: generating the task processing result using the task processing model based on the output image and the depth information.

7. The method according to claim 2, characterized in that, The imaging module also includes: a second optical modulation element, a beam splitter, and a depth sensor; The second optical modulation element is used to modulate the reflected light from the target object and project it onto the beam splitting component. The beam splitting component is used to split the projected light beam into the photosensitive element and the depth sensor. The depth sensor is used to generate depth information corresponding to the target object. The step of generating the task processing result corresponding to the output image using the task processing model includes: generating the task processing result using the task processing model based on the output image and the depth information; The method further includes updating the configuration parameters of the second optical modulation element based on the overall loss.

8. The method according to claim 6 or 7, characterized in that, The method further includes: determining a depth loss based on the depth information corresponding to the target object and the depth information corresponding to the input image; The determination of the comprehensive loss based on the task loss and the privacy loss includes: determining the comprehensive loss based on the task loss, the privacy loss, and the depth loss.

9. A joint training device for an imaging module and a task processing model, characterized in that, include: Image generation module, label generation module, and parameter update module; The image generation module is used to generate an output image corresponding to the target object using the imaging module; The imaging module includes a light source, a first optical modulation element, and a photosensitive element. The first optical modulation element is used to modulate the light beam emitted by the light source and project the modulated light beam onto the target object. The photosensitive element is used to perform imaging based on the reflected light from the target object to obtain the output image. The label generation module is used to generate the task processing result corresponding to the output image using the task processing model, and determine it as the predicted label; The parameter update module is used to determine the comprehensive loss based on the predicted label and the output image, and update the configuration parameters of the first optical modulation element and the model parameters of the task processing model based on the comprehensive loss.

10. A task processing system, characterized in that, include: Imaging module and task processing model; The imaging module includes: a light source, a first optical modulation element, and a photosensitive element; The first optical modulation element is used to modulate the light beam emitted by the light source and project the modulated light beam onto the target object; the photosensitive element is used to perform imaging based on the reflected light from the target object to obtain a target image. The task processing model is used to generate the task processing result corresponding to the target image. The configuration parameters of the first optical modulation element and the model parameters of the task processing model are obtained through joint training.

11. The task processing system according to claim 10, characterized in that, The imaging module further includes a second optical modulation element, used to modulate the reflected light from the target object and project it onto the photosensitive element. The configuration parameters of the second optical modulation element are obtained through the joint training method.

12. The task processing system according to claim 10, characterized in that, The imaging module also includes: a beam splitter and a depth sensor; The beam splitting component is used to split the reflected light from the target object into the photosensitive element and the depth sensor; The depth sensor is used to generate depth information corresponding to the target object; The task processing model generates the task processing result based on the target image and the depth information.

13. The task processing system according to claim 10, characterized in that, The imaging module also includes: a second optical modulation element, a beam splitter, and a depth sensor; The second optical modulation element is used to modulate the reflected light from the target object and project it onto the beam splitter. The configuration parameters of the second optical modulation element are obtained by training through the joint training method. The beam splitting component is used to split the projected light beam into the photosensitive element and the depth sensor; The depth sensor is used to generate depth information corresponding to the target object; The task processing model generates the task processing result based on the target image and the depth information.

14. The task processing system according to claim 13, characterized in that, The light source includes: a controllable lattice mode vertical cavity surface-emitting laser light source; And / or, the first optical modulation element includes: a first metasurface lens; And / or, the second optical modulation element includes: a second metasurface lens; And / or, the depth sensor includes: a single-photon avalanche diode.

15. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

16. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

Images