Joint training and image processing method, device and system

Abstract

The invention provides a joint training and image processing method, device and system, and relates to the fields of artificial intelligence, optical imaging and the like. The method comprises the steps that a training sample is acquired, the training sample comprises a training image and a real label, and the real label is a foreground detection result corresponding to the training image; obtaining a modulation result after the training image is modulated by the first optical assembly, and determining an input image according to the modulation result; inputting the input image into the neural network model to obtain an output foreground detection result, and determining the foreground detection result as a prediction label; obtaining a foreground enhancement evaluation result corresponding to the input image, and determining comprehensive loss according to the prediction label, the real label and the foreground enhancement evaluation result; and updating structural parameters of the first optical component and / or model parameters of the neural network model according to the comprehensive loss. By applying the scheme, the task processing precision and the like can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the fields of artificial intelligence and optical imaging, in particular to technologies such as metasurface optics, computational imaging and deep learning, and more particularly to a joint training and image processing method, device and system. BACKGROUND

[0002] In a traditional machine vision system, the original image captured by a camera often contains a large amount of background interference information, and then a complex feature extraction and foreground detection need to be performed by relying on a convolutional neural network (CNN) model, which not only increases the computational overhead of the CNN model, but also leads to a decrease in robustness in a complex lighting or strong interference environment. SUMMARY

[0003] The present disclosure provides a joint training and image processing method, device and system.

[0004] A joint training method of an optical component and a neural network model, comprising: obtaining a training sample, wherein the training sample comprises a training image and a true label, and the true label is a foreground detection result corresponding to the training image; obtaining a modulation result of the training image after being modulated by a first optical component, and determining an input image according to the modulation result; inputting the input image into a neural network model to obtain an output foreground detection result, and determining a predicted label; obtaining a foreground enhancement evaluation result corresponding to the input image, and determining a comprehensive loss according to the predicted label, the true label and the foreground enhancement evaluation result; updating a structure parameter of the first optical component and / or a model parameter of the neural network model according to the comprehensive loss.

[0005] An image processing method, comprising: generating a first image corresponding to a target region by using a first optical component; predicting the first image by using a neural network model to obtain a foreground detection result, wherein a structure parameter used by the first optical component and a model parameter of the neural network model are obtained by joint training according to the method.

[0006] A joint training device of an optical component and a neural network model, comprising: a sample obtaining module, an input determining module, a label obtaining module, a loss determining module and a parameter updating module. The sample acquisition module is configured to acquire a training sample, the training sample including a training image and a true label, the true label being a foreground detection result corresponding to the training image. The input determination module is configured to acquire a modulation result of the training image after the training image is modulated by a first optical assembly, and determine an input image according to the modulation result. The label acquisition module is configured to input the input image into a neural network model to obtain an output foreground detection result, and determine the output foreground detection result as a predicted label. The loss determination module is configured to acquire a foreground enhancement evaluation result corresponding to the input image, and determine a comprehensive loss according to the predicted label, the true label, and the foreground enhancement evaluation result. The parameter update module is configured to update a structural parameter of the first optical assembly and / or a model parameter of the neural network model according to the comprehensive loss.

[0007] An image processing system includes a first optical assembly and a neural network model. The first optical assembly is configured to generate a first image corresponding to a target region. The neural network model is configured to predict the first image to obtain a foreground detection result, the structural parameter of the first optical assembly and the model parameter of the neural network model being obtained after being jointly trained according to the method.

[0008] An electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0009] A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to enable a computer to perform the method described above.

[0010] A computer program product includes computer programs / instructions, the computer programs / instructions being executed by a processor to implement the method described above.

[0011] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings are used to better understand the present scheme and do not constitute a limitation on the present disclosure. Among them: Figure 1 The flow chart of the joint training method of the optical component and the neural network model according to the present disclosure; Figure 2 The flow chart of the image processing method according to the present disclosure; Figure 3 The composition structure schematic diagram of the joint training device of the optical component and the neural network model according to the present disclosure; Figure 4 The composition structure schematic diagram of the first embodiment of the image processing system according to the present disclosure; Figure 5 The composition structure schematic diagram of the second embodiment of the image processing system according to the present disclosure; Figure 6 A schematic block diagram of an electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0013] The exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, the description below omits the description of well-known functions and structures.

[0014] In addition, it should be understood that the term "and / or" herein merely describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents an "or" relationship between the associated objects before and after it.

[0015] As a new type of artificial micro-nano optical structure, the metasurface can control the light field through sub-wavelength units to realize spatial frequency selective transmission, depth selective focusing and polarization sensitive modulation, etc. If the metasurface can be designed in combination with the deep learning network, the foreground enhancement can be completed at the optical front end (physical front end), which will significantly improve the foreground detection performance.

[0016] Therefore, the present scheme proposes a joint training method of an optical component (such as a metasurface) and a neural network model, so as to realize foreground enhancement detection.

[0017] Figure 1 The flow chart of the joint training method of the optical component and the neural network model according to the present disclosure. As Figure 1The embodiments are shown below.

[0018] In step 101, a training sample is obtained, the training sample including a training image and a true label, the true label being a foreground detection result corresponding to the training image.

[0019] In step 102, a modulation result of the training image after modulation by the first optical assembly is obtained, and an input image is determined according to the modulation result.

[0020] In step 103, the input image is input into the neural network model to obtain an output foreground detection result, which is determined as a predicted label.

[0021] In step 104, a foreground enhancement evaluation result corresponding to the input image is obtained, and a comprehensive loss is determined according to the predicted label, the true label, and the foreground enhancement evaluation result.

[0022] In step 105, the structure parameters of the first optical assembly and / or the model parameters of the neural network model are updated according to the comprehensive loss.

[0023] The scheme of the above method embodiment can realize end-to-end joint training of the first optical assembly and the neural network model, that is, the structure parameters of the first optical assembly and the model parameters of the neural network model can be simultaneously optimized by using a back propagation algorithm according to the determined comprehensive loss, so that the imaging and detection processes are integrated to form an optical-algorithm collaborative optimization framework. Accordingly, through joint training, foreground enhancement and background suppression can be realized at the optical front end, thereby significantly improving the accuracy and robustness of task processing, and reducing the computational overhead of the neural network model, which has broad application prospects in the fields of security monitoring, smart home, industrial detection, etc.

[0024] The scheme of the present disclosure does not limit how to generate the training sample. Each training sample can include a training image and a corresponding true label. When the training sample is used to jointly train the first optical assembly and the neural network model, the modulation result of the incident light field corresponding to the training image in the training sample after modulation by the first optical assembly can be obtained, and the input image can be determined according to the modulation result and the light field propagation modeling method. It can be considered that the training image and the incident light field are one-to-one corresponding, and based on the training image, its corresponding incident light field can be determined by a predetermined method.

[0025] In some embodiments of the present disclosure, the first optical assembly can include at least one of at least one first metasurface, at least one first refractive optical element, at least one first diffractive optical element, and at least one first scattering medium element.

[0026] Correspondingly, when updating the structural parameters of the first optical assembly according to the comprehensive loss, specifically, updating the structural parameters of one or more of the first metasurface, the first refractive optical element, the first diffractive optical element, and the first scattering medium element.

[0027] The first optical assembly can further include a photosensitive element. The photosensitive element can be a complementary metal oxide semiconductor (CMOS) photosensitive element, a charge coupled device (CCD) photosensitive element, or an array light detector, etc.

[0028] The following specifically takes the first optical assembly including the first metasurface as an example to describe the joint training method of the first optical assembly and the neural network model.

[0029] In some embodiments of the present disclosure, the product of the metasurface modulation function of the first metasurface and the complex amplitude of the incident light field can be obtained, denoted as the first product, and the first product can be determined as the modulated output light field (i.e., the modulation result). Wherein, the distribution of the incident light field is determined by the scene represented by the training image.

[0030] For example, the light field incident to the first metasurface can be represented as a complex amplitude: ; (1) That is, the incident light field can be described in a complex exponential form of “amplitude A + phase ”, and j represents an imaginary unit, i.e., an imaginary unit in the complex exponential form.

[0031] The modulation function of the first metasurface can also be represented as a complex function: ; (2) Wherein, represents the amplitude modulation introduced by the metasurface, represents the phase modulation introduced by the metasurface.

[0032] The modulation function of the metasurface can be used to modulate the incident light field in the form of “amplitude + phase ” to obtain the modulation result .

[0033] The modulated output light field is obtained by multiplying the complex amplitudes of the incident light field and the modulation function, i.e., there is: (3) The first metasurface is a “physical layer optimizable element” The light field can be trained so as to develop in a direction favorable to subsequent task processing.

[0034] Correspondingly, by multiplying the metasurface modulation function with the incident light field, the precise regulation of the amplitude and phase of the light wave is completed at the optical front end, thereby laying a good foundation for subsequent processing. Moreover, the structure parameters of the first metasurface are associated with the incident light field, so that the first metasurface becomes a "trainable" physical layer.

[0035] In some embodiments of the present disclosure, the manner in which the input image is determined according to the modulation result can include: performing Fourier transform on the modulation result to obtain an angular spectrum, obtaining a second product of the angular spectrum and an angular spectrum transmission kernel, performing inverse Fourier transform on the second product to obtain a complex field of a sensing surface of a first sensor corresponding to the first metasurface, and determining a modulus square operation result of the complex field as the input image.

[0036] incident light field After modulation, the light field needs to propagate in free space for a distance z to reach a sensing surface, which refers to a sensing surface of a first sensor of the first metasurface. This step can be implemented by using an angular spectrum transmission kernel. The first metasurface and the first sensor can jointly constitute an imaging system.

[0037] To this end, the modulation result can be first subjected to Fourier transform to obtain the required angular spectrum That is, ; (4) where F represents Fourier transform. Through the above processing, the light field in the spatial domain can be converted to the frequency domain to facilitate subsequent analysis of the propagation law of different frequency components, etc.

[0038] Subsequently, the angular spectrum can be multiplied (second product) with an angular spectrum transmission kernel to simulate the phase change of different frequency components after propagating for a distance z.

[0039] Further, the second product can be subjected to inverse Fourier transform, thereby obtaining a complex field of the sensing surface That is, ; (5) where represents inverse Fourier transform; ; (6) ; (7) Equation (5) models the propagation of the light field of the physical layer, describes the change of the light from the first metasurface to the sensing surface, and i in equation (6) represents an imaginary unit, which is equivalent to j in equation (1).

[0040] obtaining a complex field The square operation result of the complex field determined as an input image, That is, the light intensity I(x, y), that is, the input image provided by the optical front end to the learning back end.

[0041] The input image can then be input into the neural network model to obtain the output foreground detection result, that is, the predicted label generated after the input image is processed by the neural network model. In addition, in the scheme described in the present disclosure, a foreground enhancement evaluation result corresponding to the input image, or a foreground enhancement criterion, can also be generated. The foreground enhancement evaluation result represents the degree of prominence of the frequency energy of the input image in the preset foreground enhancement frequency interval. That is, the foreground enhancement evaluation result makes the frequency components related to the foreground more prominent in the light field from the perspective of frequency energy distribution, thereby providing a more friendly input for subsequent task processing.

[0042] In some embodiments of the present disclosure, the way to obtain the foreground enhancement evaluation result can include: determining a foreground frequency region according to the scene features corresponding to the training image, obtaining a first frequency energy corresponding to the foreground frequency region, and obtaining a second frequency energy corresponding to a non-specific frequency region, the non-specific frequency region being a global frequency region or a background frequency region, the global frequency region including the foreground frequency region and the background frequency region, and determining the ratio of the first frequency energy and the second frequency energy as the foreground enhancement evaluation result.

[0043] By using the above processing method, the ratio of the foreground to the global / background frequency energy is calculated and used as the foreground enhancement evaluation result, which can quantitatively measure the enhancement effect of the optical front end on the foreground, thereby providing a clear direction for the optimization process of the first super surface, so as to actively improve the energy proportion of the key frequency of the foreground in the physical layer.

[0044] The foreground frequency region refers to the spatial frequency range related to the foreground in the task. For example, if it is necessary to detect a car in an image, the foreground frequency region can be the spatial frequency range corresponding to the car outline, texture, etc. The foreground frequency region can be determined in advance according to the scene features corresponding to the training image, such as being determined according to the specific task requirements and target visual features, etc., through analysis or prior knowledge, etc.

[0045] Taking the scenario of detecting whether a cup is full while filling it with water as an example, key visual cues for a full cup can include the outline / edge of the water surface and changes in its reflection / texture. The outline / edge of the water surface refers to the water level gradually rising until it reaches the rim of the cup, forming a clear water-cup edge. Changes in the reflection / texture of the water surface refer to the reflection on the water surface, which differs significantly from the texture and reflection patterns of the inner wall of an empty cup. Additionally, spatial frequency describes the rate of grayscale change in an image. High-frequency components correspond to rapidly changing grayscale, such as edges, textures, and details (e.g., the edge of the water surface, the reflection of water ripples), while low-frequency components correspond to slowly changing grayscale, such as large areas of background and overall outlines (e.g., the approximate shape of the cup). Accordingly, the foreground frequency region should focus on the frequency regions corresponding to the water surface edge and reflective textures. Therefore, a Fourier transform can be performed on the sample image of "water cup filling water" to observe the frequency domain distribution of areas such as "water surface edge, reflective texture". These areas are usually located in the mid-to-high frequency region, such as [0.3, 0.8] (frequency range normalized to 0~1).

[0046] For the foreground frequency region, the corresponding first frequency domain energy can be obtained, and the second frequency domain energy corresponding to the non-specific frequency region can be obtained. The non-specific frequency region is the global frequency region or the background frequency region, which is used to normalize the denominator of the energy.

[0047] In some embodiments of this disclosure, an inverse Fourier transform can be performed on the complex field to obtain the light field distribution in the frequency domain, and the result of the modulus-square operation of the light field distribution can be obtained. Then, based on the angular spectrum, the result of the modulus-square operation of the light field distribution, and a set frequency weighting function, the weighted sum of the energies of different frequency components in the foreground frequency region can be determined as the first frequency domain energy. Furthermore, based on the angular spectrum, the result of the modulus-square operation of the light field distribution, and the frequency weighting function, the weighted sum of the energies of different frequency components in a non-specific frequency region can be determined as the second frequency domain energy.

[0048] This allows for the re-fielding of the sensing surface. Perform a Fourier transform to obtain the optical field distribution in the frequency domain. You can also get it later. The result of the modulo square operation , which is the energy density of the frequency domain optical field, describes the energy magnitude of each frequency component.

[0049] Accordingly, the first frequency domain energy E1 can be expressed as: (8) The second frequency domain energy E2 can be expressed as: (9) in, Indicates the foreground frequency region. Indicates a non-specific frequency region. and That is, to represent the angle spectrum In and , This represents the frequency weighting function, which can be preset and allows for higher weighting of certain key frequencies.

[0050] E1 represents the weighted energy of the foreground frequency region, which is the sum of the weighted energy of different frequency components in the foreground frequency region. If a certain frequency component belongs to a foreground feature (such as the outline or detail of an object), then the corresponding... Higher weights can be assigned, making the corresponding energy more prominent in the molecule (the numerator of the ratio). E2 represents the weighted energy of the non-specific frequency region, that is, the sum of the weighted energies of different frequency components in the non-specific frequency region. If the non-specific frequency region is the global frequency region, the denominator of the ratio is the total weighted energy of the light field. If the non-specific frequency region is the background frequency region, the denominator of the ratio is the background weighted energy.

[0051] E1 / E2 can be used as the foreground enhancement evaluation result. During joint training, the goal is to maximize the foreground enhancement evaluation result, that is, to maximize the proportion of the first frequency domain energy, so that the generated image is more conducive to processing by the neural network model.

[0052] After the aforementioned processing, the predicted labels and foreground enhancement evaluation results were obtained. Furthermore, the comprehensive loss can be determined based on the predicted labels, the true labels in the training samples, and the foreground enhancement evaluation results.

[0053] In some embodiments of this disclosure, the neural network model includes, but is not limited to, convolutional neural network models for image classification, such as ResNet (Residual Network), Visual GeometryGroup (VGG), etc., and single-stage detectors and two-stage detectors for object detection, etc., wherein the single-stage detector may be YOLO (You Only Look Once) or Single Shot MultiBoxDetector (SSD), etc., and the two-stage detector may be Faster Region-based Convolutional Neural Network (Faster R-CNN) or Mask Region-based Convolutional Neural Network (Mask R-CNN), etc.

[0054] In some embodiments of this disclosure, the gap between the predicted label and the true label can be determined, the gap can be determined as the task loss, and the difference between 1 and the foreground enhancement evaluation result can be obtained, as well as the third product of the difference and the first hyperparameter can be obtained, the third product can be determined as the physical enhancement loss, and then the comprehensive loss can be determined based on the task loss and the physical enhancement loss.

[0055] For example, cross-entropy loss can be used to measure the gap between the predicted and true labels, thus obtaining the task loss, and the physical augmentation loss can be determined. Assuming the foreground augmentation evaluation result is used... express, The larger, 1- The smaller the value, the lower the contribution of this term to the overall loss, thus encouraging the optimization of the structural parameters of the first metasurface towards enhancing the weighted energy in the foreground frequency region.

[0056] By combining task loss and physical enhancement loss to determine the comprehensive loss, the first metasurface can actively enhance foreground features at the optical front end, reducing the computational burden on the neural network model at the learning back end and improving the model's robustness in complex scenes. Furthermore, the introduction of physical enhancement loss allows for high alignment between optical modulation and the task objective, achieving synergistic optimization of optics and algorithms.

[0057] In some embodiments of this disclosure, the sum of the task loss and the physical enhancement loss can be determined as the comprehensive loss. Alternatively, a parameter regularization term can be determined based on the current structural parameters of the first metasurface and a predetermined regularization function, and a fourth product of the parameter regularization term and the second hyperparameter can be obtained. This fourth product is then determined as the regularization constraint loss, and the sum of the task loss, the physical enhancement loss, and the regularization constraint loss can be determined as the comprehensive loss.

[0058] That is: (10) in, Indicates the overall loss. Indicates mission loss. Indicates physical enhancement loss, Indicates the first hyperparameter. This represents the second hyperparameter. This represents the regularization function; the specific function can be determined based on actual needs. This represents the parameter regularization term. This represents the loss due to regularization constraints.

[0059] In formula (10), regularization constraint loss is further introduced on the basis of task loss and physical enhancement loss. Through regularization constraint loss, overfitting of the structural parameters of the first metasurface can be effectively prevented, such as avoiding extreme structures that are physically difficult to realize or have excessively high manufacturing costs. It also improves the feasibility and generalization ability of the design, so that the joint training results not only make the task processing performance better, but also have practical processing and application value.

[0060] After determining the overall loss, the structural parameters of the first metasurface and the model parameters of the neural network model can be updated based on the overall loss. In some embodiments of this disclosure, the structural parameters and the model parameters can be updated synchronously according to the principle of minimizing the overall loss.

[0061] The ultimate optimization objective is: (11) By leveraging the aforementioned optimization objectives, the structural parameters of the first metasurface and the model parameters of the neural network model can be updated simultaneously, thus enabling end-to-end collaboration between optical modulation and intelligent analysis. The optical front end provides customized input to the learning back end, while the learning back end provides feedback on task requirements to the optical front end.

[0062] In addition, the first metasurface can be implemented in the following two ways during training: a. Fully digital simulation method: Based on electromagnetic wave propagation theory and optical system modeling, an end-to-end differentiable simulator for metasurface modulation, light field propagation and sensing surface response is built in a computing environment. In this method, all parameters can be jointly optimized by gradient descent algorithm. The training process is completed entirely in a virtual environment. The structural parameters of the first metasurface obtained by the final optimization can be used to guide physical deployment.

[0063] b. Physical hardware approach: Directly utilize a programmable metasurface entity as the optical front end, embed it into the training loop, measure the output of the metasurface under different configurations in real time, and combine it with the learning backend to perform end-to-end gradient estimation and parameter updates.

[0064] After completing the above joint training, the first metasurface can be prepared based on the structural parameters of the first metasurface obtained from the training, and the prepared first metasurface and the trained neural network model can be applied to actual task processing scenarios.

[0065] Accordingly, Figure 2 This is a flowchart illustrating an embodiment of the image processing method described in this disclosure. Figure 2 As shown, the specific implementation methods are as follows.

[0066] In step 201, a first image corresponding to the target area is generated using the first optical component.

[0067] In step 202, a neural network model is used to predict the first image to obtain the foreground detection result. The structural parameters of the first optical component and the model parameters of the neural network model are both based on... Figure 1 The result is obtained after joint training using the method shown.

[0068] After joint training, the first optical component can enhance the foreground of the target area and blur or attenuate the background area during the imaging stage, thereby reducing the processing difficulty of the neural network model, reducing the computational overhead of the neural network model, and improving the accuracy of the foreground detection results.

[0069] In addition, traditional machine vision systems typically pursue a "what you see is what you get" visual effect. However, in the solution described in this disclosure, the purpose of the first optical component is to generate an image that is "not intuitive to humans but most effective for neural network models" in order to improve the processing performance of neural network models and meet the needs of privacy-preserving imaging to a certain extent.

[0070] In some embodiments of this disclosure, the foreground region of the target area can be determined based on the first image obtained after processing by the first optical component, and the angular coordinate information corresponding to the foreground region can be determined. Then, the angular coordinate information can be sent to the driving mechanism of the second optical component, so that the driving mechanism can drive the second optical component to focus on the foreground region of the target area, and generate a second image corresponding to the foreground region through the second optical component.

[0071] The present disclosure proposes a dual-element optical structure, which, based on the first optical component and the neural network model, further adds a second optical component and a driving mechanism connected to the second optical component.

[0072] In this way, for the first image obtained after processing by the first optical component, the foreground region (region of interest) can be determined from the first image, and the angular coordinate information corresponding to the foreground region can be determined. Then, the angular coordinate information can be sent to the driving mechanism. Accordingly, the driving mechanism will drive the second optical component according to the angular coordinate information, so that the second optical component is accurately pointed to the foreground region, and the required second image can be generated by using the multispectral, polarization and other designs of the second optical component.

[0073] Specifically, determining the foreground region based on the first image can be divided into two steps: Step 1: Identify the central region corresponding to the foreground region on the first image.

[0074] First, the image can be divided into foreground (white) and background (black) by binarizing the first image.

[0075] Next, all connected white pixel regions are identified in the binary image, and their geometric properties (such as area and aspect ratio) are used to filter out obvious noise.

[0076] Finally, the remaining connected regions after filtering are identified as intermediate regions, and their image coordinates on the first image are calculated.

[0077] Alternatively, a deep learning-based approach can be used to train a semantic segmentation model to directly predict the intermediate region from the first image and calculate its image coordinates on the first image.

[0078] Step 2: Convert the image coordinates to real-world coordinates of the target area.

[0079] The dual-element optical structure enables a collaborative working mode of "wide-area detection + local fine analysis". The first optical component is responsible for quickly capturing a large-scale scene to preliminarily determine the foreground area, while the second optical component is precisely focused on the foreground area under the control of the drive mechanism, thereby further improving the detection accuracy and detail reproduction capability.

[0080] In some embodiments of this disclosure, the second optical component may include at least one of the following: at least one second metasurface, at least one second refractive optical element, at least one second diffractive optical element, and at least one second scattering medium element.

[0081] The second optical component may also include a photosensitive element. Specifically, the photosensitive element may be a CMOS photosensitive element, a CCD photosensitive element, or an array photodetector, etc.

[0082] In other embodiments of this disclosure, the second optical component can be implemented by configuring the first optical component to a specific operating mode. Specifically, when the first optical component needs to be used as the second optical component, it will automatically operate in telephoto mode.

[0083] In some embodiments of this disclosure, the field of view of the second optical component may not be greater than that of the first optical component. For example, the field of view of the first optical component may be 120°-180°, and the field of view of the second optical component may be 10°-20°.

[0084] Correspondingly, the first optical component can quickly achieve coarse positioning of the foreground area through a large field of view, ensuring that no key targets are missed, while the second optical component focuses on the foreground area with a small field of view, sacrificing breadth in exchange for higher spatial resolution and imaging quality.

[0085] In addition, the first optical component can be paired with a lower resolution, high frame rate global shutter sensor (first sensor) to quickly capture scene changes and lock the foreground area. The second optical component has a smaller field of view but higher imaging quality, so it can be paired with a high resolution, high image quality scientific-grade CMOS sensor or CCD sensor.

[0086] Furthermore, the drive mechanism can use high-precision actuators such as dual-axis voice coil motors or piezoelectric motors, which have extremely high pointing accuracy with micro-arcs and millisecond-level fast response and stability.

[0087] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure. Furthermore, for parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0088] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0089] Figure 3 This is a schematic diagram of the structural composition of Embodiment 300 of the joint training device for optical components and neural network models described in this disclosure. Figure 3 As shown, it includes: a sample acquisition module 301, an input determination module 302, a label acquisition module 303, a loss determination module 304, and a parameter update module 305.

[0090] The sample acquisition module 301 is used to acquire training samples, which include training images and ground truth labels, where the ground truth labels are the foreground detection results corresponding to the training images.

[0091] The input determination module 302 is used to obtain the modulation result of the training image after it has been modulated by the first optical component, and to determine the input image based on the modulation result.

[0092] The label acquisition module 303 is used to input the input image into the neural network model, obtain the output foreground detection result, and determine it as the predicted label.

[0093] The loss determination module 304 is used to obtain the foreground enhancement evaluation result corresponding to the input image and determine the comprehensive loss based on the predicted label, the true label and the foreground enhancement evaluation result.

[0094] The parameter update module 305 is used to update the structural parameters of the first optical component and / or the model parameters of the neural network model based on the comprehensive loss.

[0095] In some embodiments of this disclosure, the foreground enhancement evaluation result can characterize the significance of the frequency domain energy of the input image in a preset foreground enhancement frequency range.

[0096] In some embodiments of this disclosure, the first optical component may include a first metasurface, and accordingly, the structural parameters of the first metasurface and / or the model parameters of the neural network model may be updated based on the comprehensive loss.

[0097] In some embodiments of this disclosure, the input determination module 302 may determine the input image based on the modulation result by: performing a Fourier transform on the modulation result to obtain an angular spectrum, obtaining a second product of the angular spectrum and the angular spectrum transmission kernel, performing an inverse Fourier transform on the second product to obtain the complex field of the sensing surface of the first sensor corresponding to the first optical component, and determining the result of the modulus square operation of the complex field as the input image.

[0098] In some embodiments of this disclosure, the loss determination module 304 may obtain the foreground enhancement evaluation result by: determining the foreground frequency region based on the scene features corresponding to the training image, obtaining the first frequency domain energy corresponding to the foreground frequency region, and obtaining the second frequency domain energy corresponding to a non-specific frequency region, wherein the non-specific frequency region is a global frequency region or a background frequency region, the global frequency region includes the foreground frequency region and the background frequency region, and determining the ratio of the first frequency domain energy and the second frequency domain energy as the foreground enhancement evaluation result.

[0099] Specifically, in some embodiments of this disclosure, the loss determination module 304 can perform an inverse Fourier transform on the complex field to obtain the light field distribution in the frequency domain, and can obtain the result of the modulus square operation of the light field distribution. Then, based on the angular spectrum, the result of the modulus square operation of the light field distribution, and the set frequency weighting function, it can determine the weighted sum of the energy of different frequency components in the foreground frequency region as the first frequency domain energy. Based on the angular spectrum, the result of the modulus square operation of the light field distribution, and the frequency weighting function, it can determine the weighted sum of the energy of different frequency components in the non-specific frequency region as the second frequency domain energy.

[0100] In addition, in some embodiments of this disclosure, the loss determination module 304 can determine the gap between the predicted label and the real label, determine the gap as the task loss, obtain the difference between 1 and the foreground enhancement evaluation result, and obtain the third product of the difference and the first hyperparameter, determine the third product as the physical enhancement loss, and then determine the comprehensive loss based on the task loss and the physical enhancement loss.

[0101] In some embodiments of this disclosure, the loss determination module 304 can determine the sum of the task loss and the physical enhancement loss as the comprehensive loss, or it can determine the parameter regularization term based on the current structural parameters of the first optical component and a predetermined regularization term function, and obtain the fourth product of the parameter regularization term and the second hyperparameter, and determine the fourth product as the regularization constraint loss, thereby determining the sum of the task loss, the physical enhancement loss and the regularization constraint loss as the comprehensive loss.

[0102] Accordingly, in some embodiments of this disclosure, the parameter update module 305 can synchronously update the structural parameters of the first optical component and the model parameters of the neural network model according to the principle of minimizing the overall loss.

[0103] Figure 4 This is a schematic diagram of the structural composition of the first embodiment 400 of the image processing system described in this disclosure. Figure 4 As shown, it includes: a first optical component 401 and a neural network model 402.

[0104] The first optical component 401 is used to generate a first image corresponding to the target area.

[0105] Neural network model 402 is used to perform foreground detection on the first image and obtain the foreground detection result. The structural parameters of the first optical component and the model parameters of the neural network model are both based on... Figure 1 The result is obtained after joint training using the method shown.

[0106] Depending on actual needs, the image processing system described in this disclosure can be used for various task processing, and correspondingly, the foreground detection result is the corresponding task processing result.

[0107] Specifically, the image processing system described in this disclosure can be widely applied to various computer vision tasks that require separating and identifying specific targets from complex backgrounds. These include, but are not limited to: biometric recognition (such as face and gesture recognition), industrial visual inspection (such as liquid level and part positioning), security and monitoring (such as moving target detection), and robot navigation (such as obstacle recognition).

[0108] In these applications, the quality of the original image can be directly improved through the optical components described in this disclosure, providing a cleaner input for the backend neural network model, thereby systematically improving the robustness and accuracy of the entire vision task. At the same time, since preliminary foreground separation has been completed at the optical level, the requirements for backend digital computing power are reduced, making it possible to deploy a lightweight, low-power real-time vision system.

[0109] Figure 5 This is a schematic diagram of the structural composition of the second embodiment 500 of the image processing system described in this disclosure. Figure 5As shown, it includes: a first optical component 401, a neural network model 402, a post-processing module 403, a driving mechanism 404, and a second optical component 405.

[0110] The post-processing module 403 is used to determine the foreground region of the target area based on the first image, determine the angular coordinate information corresponding to the foreground region, and send the angular coordinate information to the drive mechanism 404.

[0111] The driving mechanism 404 is used to drive the second optical component 405 to focus on the foreground region.

[0112] The second optical component 405 is used to generate a second image corresponding to the foreground region based on the driving result.

[0113] In practical applications, the field of view of the second optical component 405 may not be greater than that of the first optical component 401. For example, the field of view of the first optical component may be 120°-180°, and the field of view of the second optical component may be 10°-20°.

[0114] In addition, the first optical component 401 can be paired with a low-resolution, high-frame-rate global shutter sensor to quickly capture scene changes and lock the foreground area. The second optical component 405 has a smaller field of view but higher imaging quality, so it can be paired with a high-resolution, high-image-quality scientific-grade CMOS sensor or CCD sensor.

[0115] Furthermore, the drive mechanism 404 can use high-precision actuators such as a two-axis voice coil motor or a piezoelectric motor, which has extremely high pointing accuracy with micro-arc and millisecond-level fast response and stability.

[0116] The specific workflow of the above-described apparatus and system embodiments can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated hereafter.

[0117] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0118] Figure 6 A schematic block diagram of an electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0119] likeFigure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0120] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0121] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the methods described herein by any other suitable means (e.g., by means of firmware).

[0122] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0124] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0125] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0126] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0127] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0128] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0129] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for jointly training an optical component and a neural network model, characterized in that, include: Obtain training samples, which include: training images and ground truth labels, wherein the ground truth labels are the foreground detection results corresponding to the training images; The modulation result of the training image after being modulated by the first optical component is obtained, and the input image is determined based on the modulation result; The input image is fed into a neural network model to obtain the output foreground detection result, which is then determined as the predicted label. Obtain the foreground enhancement evaluation result corresponding to the input image, and determine the comprehensive loss based on the predicted label, the true label, and the foreground enhancement evaluation result; The structural parameters of the first optical component and / or the model parameters of the neural network model are updated based on the comprehensive loss.

2. The method according to claim 1, characterized in that, The foreground enhancement evaluation result characterizes the significance of the frequency domain energy of the input image in the preset foreground enhancement frequency range.

3. The method according to claim 1, characterized in that, The first optical component includes a first metasurface; The step of updating the structural parameters of the first optical component and / or the model parameters of the neural network model based on the comprehensive loss includes: The structural parameters of the first metasurface and / or the model parameters of the neural network model are updated based on the comprehensive loss.

4. The method according to claim 1, characterized in that, Determining the input image based on the modulation result includes: Perform a Fourier transform on the modulation result to obtain the angular spectrum; Obtain the second product of the angular spectrum and the angular spectrum transmission kernel; Perform an inverse Fourier transform on the second product to obtain the complex field of the sensing surface of the first sensor corresponding to the first optical component; The result of the modulus square operation of the complex field is determined as the input image.

5. The method according to claim 4, characterized in that, The process of obtaining the foreground enhancement evaluation result corresponding to the input image includes: Based on the scene features corresponding to the training images, the foreground frequency region is determined; The first frequency domain energy corresponding to the foreground frequency region is obtained, and the second frequency domain energy corresponding to the non-specific frequency region is obtained. The non-specific frequency region is either a global frequency region or a background frequency region. The global frequency region includes the foreground frequency region and the background frequency region. The ratio of the first frequency domain energy to the second frequency domain energy is determined as the foreground enhancement evaluation result.

6. The method according to claim 5, characterized in that, The steps of obtaining the first frequency domain energy corresponding to the foreground frequency region and obtaining the second frequency domain energy corresponding to the non-specific frequency region include: Perform an inverse Fourier transform on the complex field to obtain the optical field distribution in the frequency domain, and obtain the result of the modulus square operation of the optical field distribution; Based on the angular spectrum, the result of the modulus square operation of the light field distribution, and the set frequency weighting function, the weighted sum of the energy of different frequency components in the foreground frequency region is determined as the first frequency domain energy; Based on the angular spectrum, the result of the modulus square operation of the light field distribution, and the frequency weighting function, the weighted sum of the energy of different frequency components in the non-specific frequency region is determined as the second frequency domain energy.

7. The method according to claim 1, characterized in that, The determination of the comprehensive loss based on the predicted label, the true label, and the foreground enhancement assessment result includes: Determine the gap between the predicted label and the true label, and define the gap as the task loss; Obtain the difference between 1 and the foreground enhancement evaluation result, and obtain the third product of the difference and the first hyperparameter, and determine the third product as the physical enhancement loss; The combined loss is determined based on the task loss and the physical enhancement loss.

8. The method according to claim 7, characterized in that, The determination of the overall loss based on the task loss and the physical enhancement loss includes: The sum of the task loss and the physical enhancement loss is determined as the comprehensive loss; Alternatively, a parameter regularization term can be determined based on the current structural parameters of the first optical component and a predetermined regularization function, and the fourth product of the parameter regularization term and the second hyperparameter can be obtained. The fourth product is then determined as the regularization constraint loss, and the sum of the task loss, the physical enhancement loss, and the regularization constraint loss is determined as the comprehensive loss.

9. An image processing method, characterized in that, include: A first image corresponding to the target region is generated using the first optical component; The first image is predicted using a neural network model to obtain a foreground detection result. The structural parameters of the first optical component and the model parameters of the neural network model are obtained by joint training according to the method of any one of claims 1-8.

10. The method according to claim 9, characterized in that, Also includes: The foreground region of the target area is determined based on the first image, and the angular coordinate information corresponding to the foreground region is determined. The angular coordinate information is sent to the driving mechanism of the second optical component, which drives the second optical component to focus on the foreground region and generates a second image corresponding to the foreground region through the second optical component.

11. The method according to claim 10, characterized in that, The field of view of the second optical component is no greater than that of the first optical component.

12. A joint training device for optical components and a neural network model, characterized in that, include: The module includes a sample acquisition module, an input determination module, a label acquisition module, a loss determination module, and a parameter update module. The sample acquisition module is used to acquire training samples, which include: training images and ground truth labels, wherein the ground truth labels are the foreground detection results corresponding to the training images; The input determination module is used to obtain the modulation result of the training image after it has been modulated by the first optical component, and to determine the input image based on the modulation result; The label acquisition module is used to input the input image into a neural network model, obtain the output foreground detection result, and determine it as the predicted label; The loss determination module is used to obtain the foreground enhancement evaluation result corresponding to the input image, and determine the comprehensive loss based on the predicted label, the true label and the foreground enhancement evaluation result; The parameter update module is used to update the structural parameters of the first optical component and / or the model parameters of the neural network model according to the comprehensive loss.

13. An image processing system, characterized in that, include: First optical component and neural network model; The first optical component is used to generate a first image corresponding to the target area; The neural network model is used to predict the first image and obtain the foreground detection result. The structural parameters of the first optical component and the model parameters of the neural network model are obtained by joint training according to the method of any one of claims 1-8.

14. The image processing system according to claim 13, characterized in that, Also includes: Post-processing module, second optical component, and drive mechanism; The post-processing module is used to determine the foreground region of the target region based on the first image, determine the angular coordinate information corresponding to the foreground region, and send the angular coordinate information to the driving mechanism. The driving mechanism is used to drive the second optical component to focus on the foreground region; The second optical component is used to generate a second image corresponding to the foreground region based on the driving result.

15. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

16. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

17. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the method of any one of claims 1-11.

Images